The document discusses Squire, a polyglot application combining Neo4j, MongoDB, Ruby, and Scala for comprehensive movie and TV show data management. It highlights features like extensive data catalogs, social interaction elements, and user-generated recommendations. The presentation details system architecture, performance improvements, and design decisions for efficient data processing and recommendations.

1. Squire: A polyglot application combining
Neo4j, MongoDB, Ruby and Scala
FOSDEM 2013 Alberto Perdomo
Graph Processing DevRoom @albertoperdomo

2. About me

3. About me
Co-Founder of Aentos

4. About me
Co-Founder of Aentos
Living my dream* (CNY)

5. About me
Co-Founder of Aentos
Living my dream* (CNY)
Web developer

7. d

9. Features

10. Features
★ Extensive movies & TV-shows data

11. Features
★ Extensive movies & TV-shows data
★ Info & price on streaming & rent services:
iTunes, Amazon, Hulu, Netflix...

12. Features
★ Extensive movies & TV-shows data
★ Info & price on streaming & rent services:
iTunes, Amazon, Hulu, Netflix...
★ Social features (contacts, comments)

13. Features
★ Extensive movies & TV-shows data
★ Info & price on streaming & rent services:
iTunes, Amazon, Hulu, Netflix...
★ Social features (contacts, comments)
★ Ratings

14. Features
★ Extensive movies & TV-shows data
★ Info & price on streaming & rent services:
iTunes, Amazon, Hulu, Netflix...
★ Social features (contacts, comments)
★ Ratings
★ Suggestions by users

15. Features
★ Extensive movies & TV-shows data
★ Info & price on streaming & rent services:
iTunes, Amazon, Hulu, Netflix...
★ Social features (contacts, comments)
★ Ratings
★ Suggestions by users
★ System recommendations

16. Unconventional
System
Architecture*

17. Ingredients

18. Ingredients
✔ MongoDB

19. Ingredients
✔ MongoDB
✔ Neo4j

20. Ingredients
✔ MongoDB
✔ Neo4j
✔ Ruby

21. Ingredients
✔ MongoDB
✔ Neo4j
✔ Ruby
✔ Scala

22. ⚛ System Overview ⚛
iPad
Client
Recommend.
Main API
Engine
(Ruby)
(Scala)
Users
All data MongoDB Neo4j Ratings
Movies and TV shows

23. API: Ruby + MongoDB

24. API: Ruby + MongoDB
⚛ REST+JSON

25. API: Ruby + MongoDB
⚛ REST+JSON
⚛ Accounts & sessions

26. API: Ruby + MongoDB
⚛ REST+JSON
⚛ Accounts & sessions
⚛ Movies & TV-Shows

27. API: Ruby + MongoDB
⚛ REST+JSON
⚛ Accounts & sessions
⚛ Movies & TV-Shows
⚛ Contacts

28. API: Ruby + MongoDB
⚛ REST+JSON
⚛ Accounts & sessions
⚛ Movies & TV-Shows
⚛ Contacts
⚛ Ratings

29. API: Ruby + MongoDB
⚛ REST+JSON
⚛ Accounts & sessions
⚛ Movies & TV-Shows
⚛ Contacts
⚛ Ratings
⚛ User suggestions & recommendations

30. Recommendations: Scala +
Neo4j

31. Recommendations: Scala +
Neo4j
⚛ Movies & TV-shows (IDs & rel.)

32. Recommendations: Scala +
Neo4j
⚛ Movies & TV-shows (IDs & rel.)
⚛ Users

33. Recommendations: Scala +
Neo4j
⚛ Movies & TV-shows (IDs & rel.)
⚛ Users
⚛ Ratings

34. Communication (RabbitMQ)
iPad
GET /suggestions Client
JSON array of data
Query params in JSON Recommend.
Main API
Engine
(Ruby)
JSON array of IDs (Scala)
Retrieve data
RabbitMQ + JSON
Collect IDs of suggestions
for response
Graph reset* Users
All data MongoDB Neo4j Ratings
Movies and TV shows

35. Issues

36. Issues
☠ API server waiting for query in Neo4j

37. Issues
☠ API server waiting for query in Neo4j
☠ Query MongoDB for add. data

38. Issues
☠ API server waiting for query in Neo4j
☠ Query MongoDB for add. data
☠ Transform data

39. Issues
☠ API server waiting for query in Neo4j
☠ Query MongoDB for add. data
☠ Transform data
☠ Ruby server is blocking

40. Communication (DB)
iPad
GET /suggestions Client JSON array of
suggested results
Redirect to Recomm. Follow redirect link
Eng. w/ hashed params
Recommend.
Main API
Engine
(Ruby)
Retrieve data for (Scala)
response
Retrieve data
Collect IDs of suggestions
for query params
Graph reset*
Users
All data MongoDB Neo4j Ratings
Movies and TV shows

41. Improvements

42. Improvements
✌ No more long blocking requests in
Ruby server

43. Improvements
✌ No more long blocking requests in
Ruby server
✌ Move heavy lifting to Spray/Scala
(async)

44. Improvements
✌ No more long blocking requests in
Ruby server
✌ Move heavy lifting to Spray/Scala
(async)
✌ Add cache layer

45. Improvements
✌ No more long blocking requests in
Ruby server
✌ Move heavy lifting to Spray/Scala
(async)
✌ Add cache layer

46. Additional Cache Layer

47. Additional Cache Layer
* Read-Through

48. Additional Cache Layer
* Read-Through
* Stores transformed data

49. Additional Cache Layer
* Read-Through
* Stores transformed data
* Implemented using spray-caching

50. Total Response Time
40s → <1s

51. Delta Updates

52. Delta Updates
* API: New ratings in queue collection
(MongoDB)

53. Delta Updates
* API: New ratings in queue collection
(MongoDB)
* Recommendations:
Backend checks for updates in queue
and applies them in graph

54. Delta Updates
iPad
Client GET /suggestions
POST /rating
Recommend.
Main API
Engine
(Ruby)
Retrieve data for (Scala)
response
Store rating + Apply updates
Collect IDs of suggestions
Put rating in queue if pending
Check if updates
in queue
Users
MongoDB Neo4j Ratings
Movies and TV shows

55. (Re)loading data in graph

56. (Re)loading data in graph
1. Create new Neo4j instance

57. (Re)loading data in graph
1. Create new Neo4j instance
2. Import data from MongoDB

58. (Re)loading data in graph
1. Create new Neo4j instance
2. Import data from MongoDB
3. Hot swap

59. (Re)loading data in graph
1. Create new Neo4j instance
2. Import data from MongoDB
3. Hot swap
★ Allows for massive updates in the primary
database

60. (Re)loading data in graph
1. Create new Neo4j instance
2. Import data from MongoDB
3. Hot swap
★ Allows for massive updates in the primary
database
★ Used only for content-related data and user

62. Graph DB as an index
for relations*

63. Graph DB as an index
for relations*
powerful queries on relations

64. Graph DB as an index
for relations*
powerful queries on relations

65. Graph DB as an index
for relations*
powerful queries on relations
⁂ like full text search for strings

66. Design decisions: Graph

67. Design decisions: Graph
✍ Copy subsets of data in Neo4j

68. Design decisions: Graph
✍ Copy subsets of data in Neo4j
✍ Rebuild graph with no
downtime*

69. Design decisions: Graph
✍ Copy subsets of data in Neo4j
✍ Rebuild graph with no
downtime*
✍ Massive updates blazingly fast

70. Design decisions: Graph
✍ Copy subsets of data in Neo4j
✍ Rebuild graph with no
downtime*
✍ Massive updates blazingly fast
✍ Less critical pieces

71. The right tools, for us! ✌
API in Ruby & MongoDB

72. The right tools, for us! ✌
API in Ruby & MongoDB
Maintanability, flexibility, libraries,

73. The right tools, for us! ✌
API in Ruby & MongoDB
Maintanability, flexibility, libraries,
schema-free!, development speed

74. The right tools, for us! ✌
Recommendations

75. The right tools, for us! ✌
Recommendations
Scala: Performance + good
interface for Neo4j

76. The right tools, for us! ✌
Recommendations
Scala: Performance + good
interface for Neo4j
Neo4j: Graph problem

77. Thanks! ☃
Alberto Perdomo
@albertoperdomo

78. ❤ Credits ❤
Entering Hyperspace by Éole Wind