Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Squire: A polyglot application combining   Neo4j, MongoDB, Ruby and ScalaFOSDEM 2013                Alberto PerdomoGraph P...
About me
About meCo-Founder of Aentos
About meCo-Founder of AentosLiving my dream* (CNY)
About meCo-Founder of AentosLiving my dream* (CNY)Web developer
d
Features
Features★ Extensive movies & TV-shows data
Features★ Extensive movies & TV-shows data★ Info & price on streaming & rent services:iTunes, Amazon, Hulu, Netflix...
Features★ Extensive movies & TV-shows data★ Info & price on streaming & rent services:iTunes, Amazon, Hulu, Netflix...★ Soc...
Features★ Extensive movies & TV-shows data★ Info & price on streaming & rent services:iTunes, Amazon, Hulu, Netflix...★ Soc...
Features★ Extensive movies & TV-shows data★ Info & price on streaming & rent services:iTunes, Amazon, Hulu, Netflix...★ Soc...
Features★ Extensive movies & TV-shows data★ Info & price on streaming & rent services:iTunes, Amazon, Hulu, Netflix...★ Soc...
Unconventional    System Architecture*
Ingredients
Ingredients✔ MongoDB
Ingredients✔ MongoDB✔ Neo4j
Ingredients✔ MongoDB✔ Neo4j✔ Ruby
Ingredients✔ MongoDB✔ Neo4j✔ Ruby✔ Scala
⚛ System Overview ⚛                       iPad                      Client                               Recommend.       ...
API: Ruby + MongoDB
API: Ruby + MongoDB⚛ REST+JSON
API: Ruby + MongoDB⚛ REST+JSON⚛ Accounts & sessions
API: Ruby + MongoDB⚛ REST+JSON⚛ Accounts & sessions⚛ Movies & TV-Shows
API: Ruby + MongoDB⚛ REST+JSON⚛ Accounts & sessions⚛ Movies & TV-Shows⚛ Contacts
API: Ruby + MongoDB⚛ REST+JSON⚛ Accounts & sessions⚛ Movies & TV-Shows⚛ Contacts⚛ Ratings
API: Ruby + MongoDB⚛ REST+JSON⚛ Accounts & sessions⚛ Movies & TV-Shows⚛ Contacts⚛ Ratings⚛ User suggestions & recommendati...
Recommendations: Scala +        Neo4j
Recommendations: Scala +        Neo4j⚛ Movies & TV-shows (IDs & rel.)
Recommendations: Scala +        Neo4j⚛ Movies & TV-shows (IDs & rel.)⚛ Users
Recommendations: Scala +        Neo4j⚛ Movies & TV-shows (IDs & rel.)⚛ Users⚛ Ratings
Communication (RabbitMQ)                                                            iPad                  GET /suggestions...
Issues
Issues☠ API server waiting for query in Neo4j
Issues☠ API server waiting for query in Neo4j☠ Query MongoDB for add. data
Issues☠ API server waiting for query in Neo4j☠ Query MongoDB for add. data☠ Transform data
Issues☠ API server waiting for query in Neo4j☠ Query MongoDB for add. data☠ Transform data☠ Ruby server is blocking
Communication (DB)                                                      iPad              GET /suggestions                ...
Improvements
Improvements✌ No more long blocking requests inRuby server
Improvements✌ No more long blocking requests inRuby server✌ Move heavy lifting to Spray/Scala(async)
Improvements✌ No more long blocking requests inRuby server✌ Move heavy lifting to Spray/Scala(async)✌ Add cache layer
Improvements✌ No more long blocking requests inRuby server✌ Move heavy lifting to Spray/Scala(async)✌ Add cache layer
Additional Cache Layer
Additional Cache Layer* Read-Through
Additional Cache Layer* Read-Through* Stores transformed data
Additional Cache Layer* Read-Through* Stores transformed data* Implemented using spray-caching
Total Response Time40s → <1s
Delta Updates
Delta Updates* API: New ratings in queue collection(MongoDB)
Delta Updates* API: New ratings in queue collection(MongoDB)* Recommendations:Backend checks for updates in queueand appli...
Delta Updates                                                  iPad                                                 Client...
(Re)loading data in graph
(Re)loading data in graph1. Create new Neo4j instance
(Re)loading data in graph1. Create new Neo4j instance2. Import data from MongoDB
(Re)loading data in graph1. Create new Neo4j instance2. Import data from MongoDB3. Hot swap
(Re)loading data in graph1. Create new Neo4j instance2. Import data from MongoDB3. Hot swap★ Allows for massive updates in...
(Re)loading data in graph1. Create new Neo4j instance2. Import data from MongoDB3. Hot swap★ Allows for massive updates in...
Graph DB as an indexfor relations*
Graph DB as an indexfor relations*powerful queries on relations
Graph DB as an indexfor relations*powerful queries on relations
Graph DB as an indexfor relations*powerful queries on relations⁂ like full text search for strings
Design decisions: Graph
Design decisions: Graph✍ Copy subsets of data in Neo4j
Design decisions: Graph✍ Copy subsets of data in Neo4j✍ Rebuild graph with nodowntime*
Design decisions: Graph✍ Copy subsets of data in Neo4j✍ Rebuild graph with nodowntime*✍ Massive updates blazingly fast
Design decisions: Graph✍ Copy subsets of data in Neo4j✍ Rebuild graph with nodowntime*✍ Massive updates blazingly fast✍ Le...
The right tools, for us! ✌API in Ruby & MongoDB
The right tools, for us! ✌API in Ruby & MongoDBMaintanability, flexibility, libraries,
The right tools, for us! ✌API in Ruby & MongoDBMaintanability, flexibility, libraries,schema-free!, development speed
The right tools, for us! ✌Recommendations
The right tools, for us! ✌RecommendationsScala: Performance + goodinterface for Neo4j
The right tools, for us! ✌RecommendationsScala: Performance + goodinterface for Neo4jNeo4j: Graph problem
Thanks! ☃      Alberto Perdomo      @albertoperdomo
❤ Credits ❤Entering Hyperspace by Éole Wind
Squire: A polyglot application combining Neo4j, MongoDB, Ruby and Scala @ FOSDEM GraphDevRoom 2013
Squire: A polyglot application combining Neo4j, MongoDB, Ruby and Scala @ FOSDEM GraphDevRoom 2013
Squire: A polyglot application combining Neo4j, MongoDB, Ruby and Scala @ FOSDEM GraphDevRoom 2013
Upcoming SlideShare
Loading in …5
×

Squire: A polyglot application combining Neo4j, MongoDB, Ruby and Scala @ FOSDEM GraphDevRoom 2013

6,598 views

Published on

Squire is an app for iPad that provides suggestions of movies and TV shows, based on the user’s preferences. There are also some social features, for instance a user can suggest a movie or TV show to one of his/her contacts.

We have developed the backend and web services that support the Squire iPad app. The recommendations system was written in Scala and everything else was developed in Ruby. This architecture tries to combine the best features of the different languages. The Ruby API is easy to maintain, while the recommendations engine written in Scala and Neo4j is powerful, fast and able to handle heavy computations in a reasonable time.

Published in: Technology

Squire: A polyglot application combining Neo4j, MongoDB, Ruby and Scala @ FOSDEM GraphDevRoom 2013

  1. 1. Squire: A polyglot application combining Neo4j, MongoDB, Ruby and ScalaFOSDEM 2013 Alberto PerdomoGraph Processing DevRoom @albertoperdomo
  2. 2. About me
  3. 3. About meCo-Founder of Aentos
  4. 4. About meCo-Founder of AentosLiving my dream* (CNY)
  5. 5. About meCo-Founder of AentosLiving my dream* (CNY)Web developer
  6. 6. d
  7. 7. Features
  8. 8. Features★ Extensive movies & TV-shows data
  9. 9. Features★ Extensive movies & TV-shows data★ Info & price on streaming & rent services:iTunes, Amazon, Hulu, Netflix...
  10. 10. Features★ Extensive movies & TV-shows data★ Info & price on streaming & rent services:iTunes, Amazon, Hulu, Netflix...★ Social features (contacts, comments)
  11. 11. Features★ Extensive movies & TV-shows data★ Info & price on streaming & rent services:iTunes, Amazon, Hulu, Netflix...★ Social features (contacts, comments)★ Ratings
  12. 12. Features★ Extensive movies & TV-shows data★ Info & price on streaming & rent services:iTunes, Amazon, Hulu, Netflix...★ Social features (contacts, comments)★ Ratings★ Suggestions by users
  13. 13. Features★ Extensive movies & TV-shows data★ Info & price on streaming & rent services:iTunes, Amazon, Hulu, Netflix...★ Social features (contacts, comments)★ Ratings★ Suggestions by users★ System recommendations
  14. 14. Unconventional System Architecture*
  15. 15. Ingredients
  16. 16. Ingredients✔ MongoDB
  17. 17. Ingredients✔ MongoDB✔ Neo4j
  18. 18. Ingredients✔ MongoDB✔ Neo4j✔ Ruby
  19. 19. Ingredients✔ MongoDB✔ Neo4j✔ Ruby✔ Scala
  20. 20. ⚛ System Overview ⚛ iPad Client Recommend. Main API Engine (Ruby) (Scala) UsersAll data MongoDB Neo4j Ratings Movies and TV shows
  21. 21. API: Ruby + MongoDB
  22. 22. API: Ruby + MongoDB⚛ REST+JSON
  23. 23. API: Ruby + MongoDB⚛ REST+JSON⚛ Accounts & sessions
  24. 24. API: Ruby + MongoDB⚛ REST+JSON⚛ Accounts & sessions⚛ Movies & TV-Shows
  25. 25. API: Ruby + MongoDB⚛ REST+JSON⚛ Accounts & sessions⚛ Movies & TV-Shows⚛ Contacts
  26. 26. API: Ruby + MongoDB⚛ REST+JSON⚛ Accounts & sessions⚛ Movies & TV-Shows⚛ Contacts⚛ Ratings
  27. 27. API: Ruby + MongoDB⚛ REST+JSON⚛ Accounts & sessions⚛ Movies & TV-Shows⚛ Contacts⚛ Ratings⚛ User suggestions & recommendations
  28. 28. Recommendations: Scala + Neo4j
  29. 29. Recommendations: Scala + Neo4j⚛ Movies & TV-shows (IDs & rel.)
  30. 30. Recommendations: Scala + Neo4j⚛ Movies & TV-shows (IDs & rel.)⚛ Users
  31. 31. Recommendations: Scala + Neo4j⚛ Movies & TV-shows (IDs & rel.)⚛ Users⚛ Ratings
  32. 32. Communication (RabbitMQ) iPad GET /suggestions Client JSON array of data Query params in JSON Recommend. Main API Engine (Ruby) JSON array of IDs (Scala) Retrieve data RabbitMQ + JSON Collect IDs of suggestions for response Graph reset* UsersAll data MongoDB Neo4j Ratings Movies and TV shows
  33. 33. Issues
  34. 34. Issues☠ API server waiting for query in Neo4j
  35. 35. Issues☠ API server waiting for query in Neo4j☠ Query MongoDB for add. data
  36. 36. Issues☠ API server waiting for query in Neo4j☠ Query MongoDB for add. data☠ Transform data
  37. 37. Issues☠ API server waiting for query in Neo4j☠ Query MongoDB for add. data☠ Transform data☠ Ruby server is blocking
  38. 38. Communication (DB) iPad GET /suggestions Client JSON array of suggested results Redirect to Recomm. Follow redirect link Eng. w/ hashed params Recommend. Main API Engine (Ruby) Retrieve data for (Scala) response Retrieve data Collect IDs of suggestions for query params Graph reset* UsersAll data MongoDB Neo4j Ratings Movies and TV shows
  39. 39. Improvements
  40. 40. Improvements✌ No more long blocking requests inRuby server
  41. 41. Improvements✌ No more long blocking requests inRuby server✌ Move heavy lifting to Spray/Scala(async)
  42. 42. Improvements✌ No more long blocking requests inRuby server✌ Move heavy lifting to Spray/Scala(async)✌ Add cache layer
  43. 43. Improvements✌ No more long blocking requests inRuby server✌ Move heavy lifting to Spray/Scala(async)✌ Add cache layer
  44. 44. Additional Cache Layer
  45. 45. Additional Cache Layer* Read-Through
  46. 46. Additional Cache Layer* Read-Through* Stores transformed data
  47. 47. Additional Cache Layer* Read-Through* Stores transformed data* Implemented using spray-caching
  48. 48. Total Response Time40s → <1s
  49. 49. Delta Updates
  50. 50. Delta Updates* API: New ratings in queue collection(MongoDB)
  51. 51. Delta Updates* API: New ratings in queue collection(MongoDB)* Recommendations:Backend checks for updates in queueand applies them in graph
  52. 52. Delta Updates iPad Client GET /suggestions POST /rating Recommend. Main API Engine (Ruby) Retrieve data for (Scala) response Store rating + Apply updates Collect IDs of suggestionsPut rating in queue if pending Check if updates in queue Users MongoDB Neo4j Ratings Movies and TV shows
  53. 53. (Re)loading data in graph
  54. 54. (Re)loading data in graph1. Create new Neo4j instance
  55. 55. (Re)loading data in graph1. Create new Neo4j instance2. Import data from MongoDB
  56. 56. (Re)loading data in graph1. Create new Neo4j instance2. Import data from MongoDB3. Hot swap
  57. 57. (Re)loading data in graph1. Create new Neo4j instance2. Import data from MongoDB3. Hot swap★ Allows for massive updates in the primarydatabase
  58. 58. (Re)loading data in graph1. Create new Neo4j instance2. Import data from MongoDB3. Hot swap★ Allows for massive updates in the primarydatabase★ Used only for content-related data and user
  59. 59. Graph DB as an indexfor relations*
  60. 60. Graph DB as an indexfor relations*powerful queries on relations
  61. 61. Graph DB as an indexfor relations*powerful queries on relations
  62. 62. Graph DB as an indexfor relations*powerful queries on relations⁂ like full text search for strings
  63. 63. Design decisions: Graph
  64. 64. Design decisions: Graph✍ Copy subsets of data in Neo4j
  65. 65. Design decisions: Graph✍ Copy subsets of data in Neo4j✍ Rebuild graph with nodowntime*
  66. 66. Design decisions: Graph✍ Copy subsets of data in Neo4j✍ Rebuild graph with nodowntime*✍ Massive updates blazingly fast
  67. 67. Design decisions: Graph✍ Copy subsets of data in Neo4j✍ Rebuild graph with nodowntime*✍ Massive updates blazingly fast✍ Less critical pieces
  68. 68. The right tools, for us! ✌API in Ruby & MongoDB
  69. 69. The right tools, for us! ✌API in Ruby & MongoDBMaintanability, flexibility, libraries,
  70. 70. The right tools, for us! ✌API in Ruby & MongoDBMaintanability, flexibility, libraries,schema-free!, development speed
  71. 71. The right tools, for us! ✌Recommendations
  72. 72. The right tools, for us! ✌RecommendationsScala: Performance + goodinterface for Neo4j
  73. 73. The right tools, for us! ✌RecommendationsScala: Performance + goodinterface for Neo4jNeo4j: Graph problem
  74. 74. Thanks! ☃ Alberto Perdomo @albertoperdomo
  75. 75. ❤ Credits ❤Entering Hyperspace by Éole Wind

×