This document discusses how Medium uses Neo4j as its graph database. It begins with an introduction to Medium and its goals. It then describes Medium's evaluation of different data store options before choosing Neo4j for its ability to model Medium's social network graph and enable complex queries. The rest of the document outlines Medium's architecture using Neo4j, process of migrating to Neo4j, techniques for optimizing performance, and includes a live demo of Neo4j queries on Medium's data.

1. How Medium uses Neo4j
Nathaniel Felsen
May 20th, 2015

2. About Me
Nathaniel Felsen
Data & DevOps engineer
nathanielfelsen
n@medium.com
@faitlezen

3. Agenda
• What is Medium and what problem we are solving through Neo4j ?
• Why did we pick Neo4j ?
• Our architecture
• Steps taken and obstacles encountered while going live
• Improve Neo4j’s performances
• Live Demo

4. • Easiest, fastest way to create a
beautiful story
• Seamless integration of photos,
audio & video
• Optimized for web, tablet &
mobile
Medium is a beautiful
publishing experience.

5. Medium is a home for
influential contributors.

6. Medium is a place for
important ideas.

7. • Follow, share, recommend
• Personalized story feed
• Customized daily emails &
notifications
Medium is a network that
builds audience.

8. Datastore Selection Process

9. DynamoDB
• Need to nail the schema ≠ Experimentation
• Limited ways of querying data
• Things like short path between users won’t
perform well
Pros
• Expertise
• Already used to store user info
• No maintenance
• No hardware
Cons

10. • Using Relational database to
create graphs
• Sharding
Pros
• Used by lots of people and
heavily vetted.
• Less rampup for learning
querying language.
• Strong community.
Cons

11. Pros
Flock DB
• Not maintained anymore
• 2 tiers model
• Deal with sharding in the near
future
• Expertise
• SQL Lite syntax
• Open Source / Free
Cons

12. • Not free
• No expertise in house
• Requires hardware
Pros
• Easy to start
• Easy to experiment
• Good community
• Enteprise edition: HA, Backup, Support
Cons

13. Architecture

14. Our Social Service Architecture

15. Nodes
• User
• Post
• Collection
Relationships
• Edited
• Wrote
• Published
• Recommended
• Followed
• …

16. Use queues for the
writes
Write are done to the master only
If you lose your master, you need
to wait for a new election

17. Productionising Neo4j

18. Capacity planning

19. Initial Data Import

20. Metrics / Monitoring
• Architecture
• Systems
• Neo4j
• Dataset
• Java
• Services that interact with Neo4j

21. Logs aggregation /
Indexing
• ElasticSearch
• Logstash
• Logstash forwarder
• Kibana

22. Backups
• Incremental Backup
• Full Backup

23. Runbook / Playbook

24. Getting optimal performance with
Neo4j

25. Talk to the support

26. What Neo4j is good and not as good at
Long Traversal Where NOT Dense / Super Node

27. Cypher Tricks
http://watch.neo4j.org/video/84900121

28. Tune the configuration
over time
• Java Garbage collection (stop the world)
• Neo4j settings

29. Cache Settings
Neo4J2.0and2.1

30. Neo4J2.2

31. Server Plugins & Unmanaged Extensions
• Easy to Deploy
• The server’s functionality can be extended by adding plugins.
• RESTful Web Services (JAX-RS)
• Put more logic in the code like caching
• Sharp tool

32. Demo

33. Followers who recommended a story

34. Top Recommended stories

35. People Recommended to follow

37. Collaborative Filtering

38. n@medium.com
Questions ? Feedback ?