Downloaded 29 times
Uploaded byIoannis Papapanagiotou
Graph Databases at Netflix
The document discusses the use of JanusGraph at Netflix, detailing its capabilities as a scalable graph database for managing vast amounts of data with high concurrency. It highlights integrations with various tools and metrics frameworks, usage statistics including over 200 million nodes, and applications in areas such as authorization and infrastructure mapping. JanusGraph efficiently handles complex queries and real-time graph traversals within Netflix's cloud environment.
More Related Content
Similar to Graph Databases at Netflix
![제 23회 보아즈(BOAZ) 빅데이터 컨퍼런스 - [MBOAX] : ABSA를 활용한 소비자 반응 분석 기반 운영 효율화 대시보드 설계](https://cdn.slidesharecdn.com/ss_thumbnails/3-1boaz23rdconferencemboax-260203102709-9d519923-thumbnail.jpg?width=640&height=640&fit=bounds)
Graph Databases at Netflix
- 1. Graph Databases @Netflix Ioannis Papapanagiotou Cloud Database Engineering
- 2.
- 3.
- 4. ● JanusGraph isa scalable graph database optimized for storing and querying graphs containing hundreds of billions of vertices and edges distributed across a multi- machine cluster. ● Can support thousands of concurrent users executing complex graph traversals in real-time. ● Native Integration with open Graph APIs ○ Tinkerpop: Gremlin Graph Query Language
- 5. Netflix integration ● Discovery(Eureka) ● Integration with our Dynamic Fast Property framework (Archaius) ● Metrics (Spectator) ● Request Tracing (Salp) ● Failure Injection Testing (FIT) ● Credential Management (Metatron)
- 6. Artwork Display Set B Display SetA Character Movie Video Track PNGWebp Audio Track JPG Video Seg Es Sub Fr Dub Montage TEXT Track Trailer Fr Sub Person Digital Asset Management
- 7. ● All entities(Assets, Movie, Display Sets etc.) - Vertex ● Collections are also a vertices with links to all entities within the collection ● All relations are edges ● Indexes - Composite for property key based lookups ● Entities are indexed in ElasticSearch for property based search outside of the JanusGraph Context DAM JanusGraph usage
- 8. ● 200 Mnodes in PROD cluster ● Hundred queries and updates per minute ● 70 Asset Types (Schema Definitions) ● >10 different client applications ● Test cluster with over 200 M nodes Current Stats (2017)
- 9. Other use cases ●Authorization, user and partner management ● Distributed Tracing ● Identify Network dependencies ● Mapping Netflix infra and the relationships ○ how code gets committed to stash, built on jenkins, deployed by spinnaker
Editor's Notes
- #3 The primary difference is that in a graph database, the relationships are stored at the individual record level, while in a relational database, the structure is defined at a higher level (the table definitions). Graph databases can be faster than relational databases for deeply-connected data - a strength of the underlying model. Graph databases take less memory for traversing deep relationships that would require multiple relational joins. Graph database schema obviates the need to manually manage indexes and reduces the write-time cost of multiple indices to cover every necessary join. If you use many-to-many relationships, in a relational database you have to introduce a JOIN table (or junction table) that holds foreign keys of both participating tables which further increases join operation costs. Those costly join operations are usually addressed by denormalizing data to reduce the number of joins necessary. Graph databases make the modeling and querying more intuitive.
- #4 JanusGraph is OSS. It is highly concurrent and it has support for ES and Cassandra. In Cassandra, we have great plugins developed for Datastax Java Driver. JanusGraph provides the ability to indexing with ES or Solr. ES is also heavily used at Netflix.
- #6 https://medium.com/netflix-techblog/fit-failure-injection-testing-35d8e2a9bb2
- #7 DAM is part of the content platform engineering. CPE deals with a lot of digital entities and assets. These entities needs to be shared across many microservices. We felt that the data can be modeled very well with a graph database. DAM stores the metadata for all the assets. The metadata can be connected or not. They metadata can be of different kind and potentially connected. The assets have to be searchable and the system has to be highly available. A simplified example of how the assets are stored in the database. The artwork is related to a movie or a person. The artwork is related to a movie, a person and a character. For example, the movies can be Jessica Jones, the person Krysten Ritter, and the character Jessica Jones. DAM uses tinkerpop API and Gremlin to query the data. The data are being stored in Cassandra. The movie could have different display sets based on the language. Each display set may have different entities, such subs/dubs.
- #8 The queries are all happening based on the vertex id All entities are a vertex
- #9 As Netflix Studios grow these numbers will expand and the numbers will grow exponentially. Storage: 12 i2.4xlarge Cassandra + 18 r3.4xlarge ES Issues Index management. Thinking of partitioning the data. Multi-level traversals may not be very performant. Goes one step at a time. When you are doing JG stores the data in C* in binary format. It is hard to perform analytics Lack of Visualization tools
- #10 Protego provides authorization as a service along with User and Partner Management service. Currently for some the api requirements indexes and inverted indexes are managed in C* DB. Based on future api requirements (support for TTL and entities) and relationships between some of the concepts graph db can prove to be good underlying storage. By moving to graph db managing relationships between objects will become easy as well as queries and index management to support existing and future api requirements. Salp, our distributed tracing solution, was initially developed by the Runtime platform team. The primary goal was data collection from various services and providing a proof of concept visualization. Later on, the services - Crawler, Aggregator, Backend API were added to enrich the data and allow querying. Dredge wants to identify networks dependencies. This can assist when network events happen and can significantly reduce our mean to detection. #core team would probably be the consumer of this service, which is named Kevin.