Advertisement
Check these out next
![Tiger graph 2021 corporate overview [read only]](https://cdn.slidesharecdn.com/ss_thumbnails/tigergraph2021corporateoverviewread-only-211014093106-thumbnail.jpg?width=600&height=600&fit=bounds)
Graph Databases at Netflix
Sep. 27, 2018•0 likes•2,764 views
4 likes
Be the first to like this
Show More
views
Total views
0
On Slideshare
0
From embeds
0
Number of embeds
0
Download to read offline
Report
Data & Analytics
In the slide deck, we describe how graph databases are used at Netflix. Graph databases can be faster than relational databases for deeply-connected data - a strength of the underlying model. We have used JanusGraph on top of Cassandra. Both technologies are Open Source.
Ioannis PapapanagiotouFollow
Technology Enthusiast at Netflix, Purdue UniversityAdvertisement
Advertisement
Advertisement
Recommended
Graph-Powered Digital Asset Management with Neo4jNeo4j
Polyglot persistence @ netflix (CDE Meetup) Roopa Tangirala
Near Real-Time Netflix Recommendations Using Apache Spark Streaming with Nit...Databricks
Gremlin's AnatomyStephen Mallette
Real-Time Anomoly Detection with Spark MLib, Akka and Cassandra by Natalino BusaSpark Summit
introduction to NOSQL Databasenehabsairam
Graph Databases at Netflix
- Graph Databases @ Netflix Ioannis Papapanagiotou Cloud Database Engineering
- Data Model
- AWS Neptune
- ● JanusGraph is a scalable graph database optimized for storing and querying graphs containing hundreds of billions of vertices and edges distributed across a multi- machine cluster. ● Can support thousands of concurrent users executing complex graph traversals in real-time. ● Native Integration with open Graph APIs ○ Tinkerpop: Gremlin Graph Query Language
- Netflix integration ● Discovery (Eureka) ● Integration with our Dynamic Fast Property framework (Archaius) ● Metrics (Spectator) ● Request Tracing (Salp) ● Failure Injection Testing (FIT) ● Credential Management (Metatron)
- Artwork Display Set B Display Set A Character Movie Video Track PNGWebp Audio Track JPG Video Seg Es Sub Fr Dub Montage TEXT Track Trailer Fr Sub Person Digital Asset Management
- ● All entities (Assets, Movie, Display Sets etc.) - Vertex ● Collections are also a vertices with links to all entities within the collection ● All relations are edges ● Indexes - Composite for property key based lookups ● Entities are indexed in ElasticSearch for property based search outside of the JanusGraph Context DAM JanusGraph usage
- ● 200 M nodes in PROD cluster ● Hundred queries and updates per minute ● 70 Asset Types (Schema Definitions) ● >10 different client applications ● Test cluster with over 200 M nodes Current Stats (2017)
- Other use cases ● Authorization, user and partner management ● Distributed Tracing ● Identify Network dependencies ● Mapping Netflix infra and the relationships ○ how code gets committed to stash, built on jenkins, deployed by spinnaker
Editor's Notes
- The primary difference is that in a graph database, the relationships are stored at the individual record level, while in a relational database, the structure is defined at a higher level (the table definitions). Graph databases can be faster than relational databases for deeply-connected data - a strength of the underlying model. Graph databases take less memory for traversing deep relationships that would require multiple relational joins. Graph database schema obviates the need to manually manage indexes and reduces the write-time cost of multiple indices to cover every necessary join. If you use many-to-many relationships, in a relational database you have to introduce a JOIN table (or junction table) that holds foreign keys of both participating tables which further increases join operation costs. Those costly join operations are usually addressed by denormalizing data to reduce the number of joins necessary. Graph databases make the modeling and querying more intuitive.
- JanusGraph is OSS. It is highly concurrent and it has support for ES and Cassandra. In Cassandra, we have great plugins developed for Datastax Java Driver. JanusGraph provides the ability to indexing with ES or Solr. ES is also heavily used at Netflix.
- https://medium.com/netflix-techblog/fit-failure-injection-testing-35d8e2a9bb2
- DAM is part of the content platform engineering. CPE deals with a lot of digital entities and assets. These entities needs to be shared across many microservices. We felt that the data can be modeled very well with a graph database. DAM stores the metadata for all the assets. The metadata can be connected or not. They metadata can be of different kind and potentially connected. The assets have to be searchable and the system has to be highly available. A simplified example of how the assets are stored in the database. The artwork is related to a movie or a person. The artwork is related to a movie, a person and a character. For example, the movies can be Jessica Jones, the person Krysten Ritter, and the character Jessica Jones. DAM uses tinkerpop API and Gremlin to query the data. The data are being stored in Cassandra. The movie could have different display sets based on the language. Each display set may have different entities, such subs/dubs.
- The queries are all happening based on the vertex id All entities are a vertex
- As Netflix Studios grow these numbers will expand and the numbers will grow exponentially. Storage: 12 i2.4xlarge Cassandra + 18 r3.4xlarge ES Issues Index management. Thinking of partitioning the data. Multi-level traversals may not be very performant. Goes one step at a time. When you are doing JG stores the data in C* in binary format. It is hard to perform analytics Lack of Visualization tools
- Protego provides authorization as a service along with User and Partner Management service. Currently for some the api requirements indexes and inverted indexes are managed in C* DB. Based on future api requirements (support for TTL and entities) and relationships between some of the concepts graph db can prove to be good underlying storage. By moving to graph db managing relationships between objects will become easy as well as queries and index management to support existing and future api requirements. Salp, our distributed tracing solution, was initially developed by the Runtime platform team. The primary goal was data collection from various services and providing a proof of concept visualization. Later on, the services - Crawler, Aggregator, Backend API were added to enrich the data and allow querying. Dredge wants to identify networks dependencies. This can assist when network events happen and can significantly reduce our mean to detection. #core team would probably be the consumer of this service, which is named Kevin.
Advertisement