SlideShare a Scribd company logo
1 of 49
InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/zen-pinterest-graph-storage-service
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon San Francisco
www.qconsf.com
Raghavendra Prabhu (RVP)
Zen:
Pinterest’s Graph Storage Service
“Given how robust the messenger is on
day one, it’s surprising to learn that
Pinterest built the entire product 

in three months.” — The Verge
What does it take to do this consistently?
Many different things

•Hire the best

•Culture

•Focus

Infrastructure that doesn’t get in the way
Challenge for Infrastructure
Our approach
•Make it part of the team mission statement

•Design systems with ‘move fast’ in mind

•Separation of concerns: feature vs reliability
Persistent Storage
Even with a distributed database, app needs to deal with:

•Schema design

•Fault tolerance

•Capacity management

•Performance tuning
Solution 1: UserMetaStore
Storage-as-a-Service: Key-value thrift API on top of HBase 

Features:

•Key partitioning to balance load

•Master-slave clusters, semi automatic failover

•Speculative execution

•Multi-tenancy with traffic isolation
Storage-as-a-service is a great step forward,
but can we do better?
Example: Messages Data Model
Conversation
Message 1
Message 2
Message N
User
User
Participates Contains
Realization
•These object models closely resemble a graph

•Objects are nodes, edges represent relationships

•Typical needs:

• retrieve data for a node or edge

• get all outgoing edges from a node

• get all incoming edges from a node

• count incoming or outgoing edges for a node
Enter Zen!
•Provides a graph data model instead of key-value

•Automatically creates necessary indexes

•Materializes counts for efficient querying

•Implemented on top of HBase, but can plug in other backends
Why the name Zen?
•Data model inspired by Facebook’s TAO

•But internally a very different system

•Zen:

• “evolution of Buddhism under Taoist conditions”

• “simplified version of Taoism”

• basically Pinterest’s take on the TAO idea :)
What Zen is NOT
•NOT a full fledged graph database

•NO advanced graph operations

•Basically an object-relationship data model on top of existing
databases to simplify app development
Zen API
Nodes:

• addNode, removeNode, getNode

• Node id: globally unique 64-bit integer

ID 123
Prop 1 Val 1
Prop 2 Val 2
Zen API
Edges:

• addEdge, removeEdge, getEdge

• Edge Ref: (edgeType, fromId, toId)

• Score for ordering
Edge Ref 120, 123, 4567
Prop 1 Val 1
Prop 2 Val 2
Zen API
Edge Queries:

• getEdges, countEdges, removeEdges
struct EdgeQuery {
1: required NodeId nodeId;
2: required EdgeDirection direction;
3: optional TypeId edgeType;
}
Zen API
Property Indexes

•Unique index

•Ensures a property value is unique across all nodes of a type

•Non-unique index

•Allows retrieval by property value

•Works for both nodes and edges
Zen API
Type System

•Declare node and edge types

•Specify type schema, e.g. unique and non-unique index properties

•Fully online: no deploy, no config

•Internally implemented on top of Zen itself!
Illustration: Messages on Zen
Id:1234 Id:2345 Id:3456
Type: Participates Type: Contains
Type: Conversation

Started: 12 Aug 2014 08:00

Header: “Great pin!”

Pin Id: 10001 [non-unique]
Type: User

Name: “Ben Smith” [unique]
Status: Active
Type: Message

Sent: 12 Aug 2014 08:00

Text: “Great pin!”
Zen: Current Usage
Products:

• smart feed, messages, network news, interest graph and other
upcoming features

Numbers: 

• ~10 clusters

• 100,000+ requests per second at peak

• Over 5 million HBase operations per second
Xun Liu
Internals and Production Learnings
Zen Backends
•HBase backend implemented in fall 2013

•Currently working on MySQL backend

•Other potential backends in future
HBase Data Model Overview
Data
HBase Data Model Overview
Data
col1 col2
row-key-1 val1 val2
HBase Data Model Overview
Data
col1 col2 col3
row-key-1 val1 val2
row-key-2 val3 val4
HBase Data Model Overview
Data
col1 col2 col3 col4
row-key-1 val1 val2
row-key-2 val3 val4
row-key-3 val5
Zen - Property
Data
type name score distance
12345 (node) 10 Ben Smith
12345-20-67890 (edge) 1000 1 mile
Zen - Property Index
Data
ID
<hash>-unique-10-name=ben smith 12345
<hash>-nonuniq-10-lastname=smith-12345
<hash>-nonuniq-10-lastname=smith-67890
Zen - Edge Score Index
Data
12345-out-20-1000-67890
12345-out-20-1001-67891
12345-in-30-990-67892
12345-in-30-991-67893
Zen - Edge Count
Data
Count
12345-out-20 2
12345-in-30 4
Status - Soft Delete
New Features
Built-in Cache
New Features
Zen
Cache
HBaseClient Zen HBaseClient
Cache
Before After
Namespace
New Features
Node
Namespace 1
Edge Index Node
Namespace 2
Edge Index
New Features
•Online type schema change

•Optional reverse edge

•Optional edge count

•Retrieval of subset of properties

•Descending edge score
Performance Work
Demanding work load needs special tuning 

• Inserting 1 million edges per second

• Excessive HLog (WAL) flushes
Performance Work
Batching 

• Client Side Batching — bulk edge insertion

• Zen Server Side Batching — buffer edits across clients & flush
together

• Reduced HLog (WAL) flushes by orders of magnitude
Performance Work
Memory v.s. Performance

• Bloom filter

• reduce disk seeks

• memory cost: 1 byte per row

• Block size

• the smaller block size the better random access performance

• memory cost: bigger index size
Performance Work
CPU v.s. Data Size

• Encoding

• FAST_DIFF: effective in reducing data size, cpu intensive

• PREFIX: less effective in size reduction, less cpu intensive

• Compression

• SNAPPY, LZO, GZ, etc
Performance Work
Capability to tune storage engine per special load

Zen production setup

• Dedicated Zen cluster

• Namespace in shared Zen cluster
Data Consistency
Add an edge

1. CAS create the edge row and properties

2. CAS create the unique index if any

3. Create non-unique index if any

4. Create edge score index for outgoing direction

5. Create edge score index for incoming direction

6. Increment edge count for outgoing direction

7. Increment edge count for incoming direction
Distributed transaction or not?
Data Consistency
Stay on top of data inconsistencies

• Manual rollback in Zen server

• Offline jobs (Dr Zen) to scan and fix inconsistencies

• Tools to debug and fix one-off inconsistency
Future Work
•Dr Zen (make it more efficient)

•Other backends: MySQL, etc

•Distributed transactions

•Open source!
Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/zen-
pinterest-graph-storage-service

More Related Content

More from C4Media

Streaming a Million Likes/Second: Real-Time Interactions on Live Video
Streaming a Million Likes/Second: Real-Time Interactions on Live VideoStreaming a Million Likes/Second: Real-Time Interactions on Live Video
Streaming a Million Likes/Second: Real-Time Interactions on Live VideoC4Media
 
Next Generation Client APIs in Envoy Mobile
Next Generation Client APIs in Envoy MobileNext Generation Client APIs in Envoy Mobile
Next Generation Client APIs in Envoy MobileC4Media
 
Software Teams and Teamwork Trends Report Q1 2020
Software Teams and Teamwork Trends Report Q1 2020Software Teams and Teamwork Trends Report Q1 2020
Software Teams and Teamwork Trends Report Q1 2020C4Media
 
Understand the Trade-offs Using Compilers for Java Applications
Understand the Trade-offs Using Compilers for Java ApplicationsUnderstand the Trade-offs Using Compilers for Java Applications
Understand the Trade-offs Using Compilers for Java ApplicationsC4Media
 
Kafka Needs No Keeper
Kafka Needs No KeeperKafka Needs No Keeper
Kafka Needs No KeeperC4Media
 
High Performing Teams Act Like Owners
High Performing Teams Act Like OwnersHigh Performing Teams Act Like Owners
High Performing Teams Act Like OwnersC4Media
 
Does Java Need Inline Types? What Project Valhalla Can Bring to Java
Does Java Need Inline Types? What Project Valhalla Can Bring to JavaDoes Java Need Inline Types? What Project Valhalla Can Bring to Java
Does Java Need Inline Types? What Project Valhalla Can Bring to JavaC4Media
 
Service Meshes- The Ultimate Guide
Service Meshes- The Ultimate GuideService Meshes- The Ultimate Guide
Service Meshes- The Ultimate GuideC4Media
 
Shifting Left with Cloud Native CI/CD
Shifting Left with Cloud Native CI/CDShifting Left with Cloud Native CI/CD
Shifting Left with Cloud Native CI/CDC4Media
 
CI/CD for Machine Learning
CI/CD for Machine LearningCI/CD for Machine Learning
CI/CD for Machine LearningC4Media
 
Fault Tolerance at Speed
Fault Tolerance at SpeedFault Tolerance at Speed
Fault Tolerance at SpeedC4Media
 
Architectures That Scale Deep - Regaining Control in Deep Systems
Architectures That Scale Deep - Regaining Control in Deep SystemsArchitectures That Scale Deep - Regaining Control in Deep Systems
Architectures That Scale Deep - Regaining Control in Deep SystemsC4Media
 
ML in the Browser: Interactive Experiences with Tensorflow.js
ML in the Browser: Interactive Experiences with Tensorflow.jsML in the Browser: Interactive Experiences with Tensorflow.js
ML in the Browser: Interactive Experiences with Tensorflow.jsC4Media
 
Build Your Own WebAssembly Compiler
Build Your Own WebAssembly CompilerBuild Your Own WebAssembly Compiler
Build Your Own WebAssembly CompilerC4Media
 
User & Device Identity for Microservices @ Netflix Scale
User & Device Identity for Microservices @ Netflix ScaleUser & Device Identity for Microservices @ Netflix Scale
User & Device Identity for Microservices @ Netflix ScaleC4Media
 
Scaling Patterns for Netflix's Edge
Scaling Patterns for Netflix's EdgeScaling Patterns for Netflix's Edge
Scaling Patterns for Netflix's EdgeC4Media
 
Make Your Electron App Feel at Home Everywhere
Make Your Electron App Feel at Home EverywhereMake Your Electron App Feel at Home Everywhere
Make Your Electron App Feel at Home EverywhereC4Media
 
The Talk You've Been Await-ing For
The Talk You've Been Await-ing ForThe Talk You've Been Await-ing For
The Talk You've Been Await-ing ForC4Media
 
Future of Data Engineering
Future of Data EngineeringFuture of Data Engineering
Future of Data EngineeringC4Media
 
Automated Testing for Terraform, Docker, Packer, Kubernetes, and More
Automated Testing for Terraform, Docker, Packer, Kubernetes, and MoreAutomated Testing for Terraform, Docker, Packer, Kubernetes, and More
Automated Testing for Terraform, Docker, Packer, Kubernetes, and MoreC4Media
 

More from C4Media (20)

Streaming a Million Likes/Second: Real-Time Interactions on Live Video
Streaming a Million Likes/Second: Real-Time Interactions on Live VideoStreaming a Million Likes/Second: Real-Time Interactions on Live Video
Streaming a Million Likes/Second: Real-Time Interactions on Live Video
 
Next Generation Client APIs in Envoy Mobile
Next Generation Client APIs in Envoy MobileNext Generation Client APIs in Envoy Mobile
Next Generation Client APIs in Envoy Mobile
 
Software Teams and Teamwork Trends Report Q1 2020
Software Teams and Teamwork Trends Report Q1 2020Software Teams and Teamwork Trends Report Q1 2020
Software Teams and Teamwork Trends Report Q1 2020
 
Understand the Trade-offs Using Compilers for Java Applications
Understand the Trade-offs Using Compilers for Java ApplicationsUnderstand the Trade-offs Using Compilers for Java Applications
Understand the Trade-offs Using Compilers for Java Applications
 
Kafka Needs No Keeper
Kafka Needs No KeeperKafka Needs No Keeper
Kafka Needs No Keeper
 
High Performing Teams Act Like Owners
High Performing Teams Act Like OwnersHigh Performing Teams Act Like Owners
High Performing Teams Act Like Owners
 
Does Java Need Inline Types? What Project Valhalla Can Bring to Java
Does Java Need Inline Types? What Project Valhalla Can Bring to JavaDoes Java Need Inline Types? What Project Valhalla Can Bring to Java
Does Java Need Inline Types? What Project Valhalla Can Bring to Java
 
Service Meshes- The Ultimate Guide
Service Meshes- The Ultimate GuideService Meshes- The Ultimate Guide
Service Meshes- The Ultimate Guide
 
Shifting Left with Cloud Native CI/CD
Shifting Left with Cloud Native CI/CDShifting Left with Cloud Native CI/CD
Shifting Left with Cloud Native CI/CD
 
CI/CD for Machine Learning
CI/CD for Machine LearningCI/CD for Machine Learning
CI/CD for Machine Learning
 
Fault Tolerance at Speed
Fault Tolerance at SpeedFault Tolerance at Speed
Fault Tolerance at Speed
 
Architectures That Scale Deep - Regaining Control in Deep Systems
Architectures That Scale Deep - Regaining Control in Deep SystemsArchitectures That Scale Deep - Regaining Control in Deep Systems
Architectures That Scale Deep - Regaining Control in Deep Systems
 
ML in the Browser: Interactive Experiences with Tensorflow.js
ML in the Browser: Interactive Experiences with Tensorflow.jsML in the Browser: Interactive Experiences with Tensorflow.js
ML in the Browser: Interactive Experiences with Tensorflow.js
 
Build Your Own WebAssembly Compiler
Build Your Own WebAssembly CompilerBuild Your Own WebAssembly Compiler
Build Your Own WebAssembly Compiler
 
User & Device Identity for Microservices @ Netflix Scale
User & Device Identity for Microservices @ Netflix ScaleUser & Device Identity for Microservices @ Netflix Scale
User & Device Identity for Microservices @ Netflix Scale
 
Scaling Patterns for Netflix's Edge
Scaling Patterns for Netflix's EdgeScaling Patterns for Netflix's Edge
Scaling Patterns for Netflix's Edge
 
Make Your Electron App Feel at Home Everywhere
Make Your Electron App Feel at Home EverywhereMake Your Electron App Feel at Home Everywhere
Make Your Electron App Feel at Home Everywhere
 
The Talk You've Been Await-ing For
The Talk You've Been Await-ing ForThe Talk You've Been Await-ing For
The Talk You've Been Await-ing For
 
Future of Data Engineering
Future of Data EngineeringFuture of Data Engineering
Future of Data Engineering
 
Automated Testing for Terraform, Docker, Packer, Kubernetes, and More
Automated Testing for Terraform, Docker, Packer, Kubernetes, and MoreAutomated Testing for Terraform, Docker, Packer, Kubernetes, and More
Automated Testing for Terraform, Docker, Packer, Kubernetes, and More
 

Recently uploaded

4. Cobus Valentine- Cybersecurity Threats and Solutions for the Public Sector
4. Cobus Valentine- Cybersecurity Threats and Solutions for the Public Sector4. Cobus Valentine- Cybersecurity Threats and Solutions for the Public Sector
4. Cobus Valentine- Cybersecurity Threats and Solutions for the Public Sectoritnewsafrica
 
Microservices, Docker deploy and Microservices source code in C#
Microservices, Docker deploy and Microservices source code in C#Microservices, Docker deploy and Microservices source code in C#
Microservices, Docker deploy and Microservices source code in C#Karmanjay Verma
 
Kuma Meshes Part I - The basics - A tutorial
Kuma Meshes Part I - The basics - A tutorialKuma Meshes Part I - The basics - A tutorial
Kuma Meshes Part I - The basics - A tutorialJoão Esperancinha
 
Generative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfGenerative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfIngrid Airi González
 
Landscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdfLandscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdfAarwolf Industries LLC
 
A Glance At The Java Performance Toolbox
A Glance At The Java Performance ToolboxA Glance At The Java Performance Toolbox
A Glance At The Java Performance ToolboxAna-Maria Mihalceanu
 
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...BookNet Canada
 
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better StrongerModern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better Strongerpanagenda
 
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...amber724300
 
Time Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directionsTime Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directionsNathaniel Shimoni
 
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...itnewsafrica
 
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...panagenda
 
Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...
Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...
Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...Nikki Chapple
 
A Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersA Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersNicole Novielli
 
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotesMuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotesManik S Magar
 
Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsRavi Sanghani
 
Irene Moetsana-Moeng: Stakeholders in Cybersecurity: Collaborative Defence fo...
Irene Moetsana-Moeng: Stakeholders in Cybersecurity: Collaborative Defence fo...Irene Moetsana-Moeng: Stakeholders in Cybersecurity: Collaborative Defence fo...
Irene Moetsana-Moeng: Stakeholders in Cybersecurity: Collaborative Defence fo...itnewsafrica
 
Generative AI - Gitex v1Generative AI - Gitex v1.pptx
Generative AI - Gitex v1Generative AI - Gitex v1.pptxGenerative AI - Gitex v1Generative AI - Gitex v1.pptx
Generative AI - Gitex v1Generative AI - Gitex v1.pptxfnnc6jmgwh
 
React Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App FrameworkReact Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App FrameworkPixlogix Infotech
 
Infrared simulation and processing on Nvidia platforms
Infrared simulation and processing on Nvidia platformsInfrared simulation and processing on Nvidia platforms
Infrared simulation and processing on Nvidia platformsYoss Cohen
 

Recently uploaded (20)

4. Cobus Valentine- Cybersecurity Threats and Solutions for the Public Sector
4. Cobus Valentine- Cybersecurity Threats and Solutions for the Public Sector4. Cobus Valentine- Cybersecurity Threats and Solutions for the Public Sector
4. Cobus Valentine- Cybersecurity Threats and Solutions for the Public Sector
 
Microservices, Docker deploy and Microservices source code in C#
Microservices, Docker deploy and Microservices source code in C#Microservices, Docker deploy and Microservices source code in C#
Microservices, Docker deploy and Microservices source code in C#
 
Kuma Meshes Part I - The basics - A tutorial
Kuma Meshes Part I - The basics - A tutorialKuma Meshes Part I - The basics - A tutorial
Kuma Meshes Part I - The basics - A tutorial
 
Generative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfGenerative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdf
 
Landscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdfLandscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdf
 
A Glance At The Java Performance Toolbox
A Glance At The Java Performance ToolboxA Glance At The Java Performance Toolbox
A Glance At The Java Performance Toolbox
 
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
 
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better StrongerModern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
 
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
 
Time Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directionsTime Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directions
 
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
 
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
 
Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...
Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...
Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...
 
A Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersA Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software Developers
 
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotesMuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
 
Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and Insights
 
Irene Moetsana-Moeng: Stakeholders in Cybersecurity: Collaborative Defence fo...
Irene Moetsana-Moeng: Stakeholders in Cybersecurity: Collaborative Defence fo...Irene Moetsana-Moeng: Stakeholders in Cybersecurity: Collaborative Defence fo...
Irene Moetsana-Moeng: Stakeholders in Cybersecurity: Collaborative Defence fo...
 
Generative AI - Gitex v1Generative AI - Gitex v1.pptx
Generative AI - Gitex v1Generative AI - Gitex v1.pptxGenerative AI - Gitex v1Generative AI - Gitex v1.pptx
Generative AI - Gitex v1Generative AI - Gitex v1.pptx
 
React Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App FrameworkReact Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App Framework
 
Infrared simulation and processing on Nvidia platforms
Infrared simulation and processing on Nvidia platformsInfrared simulation and processing on Nvidia platforms
Infrared simulation and processing on Nvidia platforms
 

Zen: Pinterest's Graph Storage Service

  • 1.
  • 2. InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations /zen-pinterest-graph-storage-service
  • 3. Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon San Francisco www.qconsf.com
  • 5. “Given how robust the messenger is on day one, it’s surprising to learn that Pinterest built the entire product 
 in three months.” — The Verge
  • 6.
  • 7. What does it take to do this consistently? Many different things •Hire the best •Culture •Focus Infrastructure that doesn’t get in the way
  • 9. Our approach •Make it part of the team mission statement •Design systems with ‘move fast’ in mind •Separation of concerns: feature vs reliability
  • 10. Persistent Storage Even with a distributed database, app needs to deal with: •Schema design •Fault tolerance •Capacity management •Performance tuning
  • 11. Solution 1: UserMetaStore Storage-as-a-Service: Key-value thrift API on top of HBase Features: •Key partitioning to balance load •Master-slave clusters, semi automatic failover •Speculative execution •Multi-tenancy with traffic isolation
  • 12. Storage-as-a-service is a great step forward, but can we do better?
  • 13. Example: Messages Data Model Conversation Message 1 Message 2 Message N User User Participates Contains
  • 14. Realization •These object models closely resemble a graph •Objects are nodes, edges represent relationships •Typical needs: • retrieve data for a node or edge • get all outgoing edges from a node • get all incoming edges from a node • count incoming or outgoing edges for a node
  • 15. Enter Zen! •Provides a graph data model instead of key-value •Automatically creates necessary indexes •Materializes counts for efficient querying •Implemented on top of HBase, but can plug in other backends
  • 16. Why the name Zen? •Data model inspired by Facebook’s TAO •But internally a very different system •Zen: • “evolution of Buddhism under Taoist conditions” • “simplified version of Taoism” • basically Pinterest’s take on the TAO idea :)
  • 17. What Zen is NOT •NOT a full fledged graph database •NO advanced graph operations •Basically an object-relationship data model on top of existing databases to simplify app development
  • 18. Zen API Nodes: • addNode, removeNode, getNode • Node id: globally unique 64-bit integer
 ID 123 Prop 1 Val 1 Prop 2 Val 2
  • 19. Zen API Edges: • addEdge, removeEdge, getEdge • Edge Ref: (edgeType, fromId, toId) • Score for ordering Edge Ref 120, 123, 4567 Prop 1 Val 1 Prop 2 Val 2
  • 20. Zen API Edge Queries: • getEdges, countEdges, removeEdges struct EdgeQuery { 1: required NodeId nodeId; 2: required EdgeDirection direction; 3: optional TypeId edgeType; }
  • 21. Zen API Property Indexes •Unique index •Ensures a property value is unique across all nodes of a type •Non-unique index •Allows retrieval by property value •Works for both nodes and edges
  • 22. Zen API Type System •Declare node and edge types •Specify type schema, e.g. unique and non-unique index properties •Fully online: no deploy, no config •Internally implemented on top of Zen itself!
  • 23. Illustration: Messages on Zen Id:1234 Id:2345 Id:3456 Type: Participates Type: Contains Type: Conversation Started: 12 Aug 2014 08:00 Header: “Great pin!” Pin Id: 10001 [non-unique] Type: User Name: “Ben Smith” [unique] Status: Active Type: Message Sent: 12 Aug 2014 08:00 Text: “Great pin!”
  • 24. Zen: Current Usage Products: • smart feed, messages, network news, interest graph and other upcoming features Numbers: • ~10 clusters • 100,000+ requests per second at peak • Over 5 million HBase operations per second
  • 25. Xun Liu Internals and Production Learnings
  • 26. Zen Backends •HBase backend implemented in fall 2013 •Currently working on MySQL backend •Other potential backends in future
  • 27. HBase Data Model Overview Data
  • 28. HBase Data Model Overview Data col1 col2 row-key-1 val1 val2
  • 29. HBase Data Model Overview Data col1 col2 col3 row-key-1 val1 val2 row-key-2 val3 val4
  • 30. HBase Data Model Overview Data col1 col2 col3 col4 row-key-1 val1 val2 row-key-2 val3 val4 row-key-3 val5
  • 31. Zen - Property Data type name score distance 12345 (node) 10 Ben Smith 12345-20-67890 (edge) 1000 1 mile
  • 32. Zen - Property Index Data ID <hash>-unique-10-name=ben smith 12345 <hash>-nonuniq-10-lastname=smith-12345 <hash>-nonuniq-10-lastname=smith-67890
  • 33. Zen - Edge Score Index Data 12345-out-20-1000-67890 12345-out-20-1001-67891 12345-in-30-990-67892 12345-in-30-991-67893
  • 34. Zen - Edge Count Data Count 12345-out-20 2 12345-in-30 4
  • 35. Status - Soft Delete New Features
  • 36. Built-in Cache New Features Zen Cache HBaseClient Zen HBaseClient Cache Before After
  • 37. Namespace New Features Node Namespace 1 Edge Index Node Namespace 2 Edge Index
  • 38. New Features •Online type schema change •Optional reverse edge •Optional edge count •Retrieval of subset of properties •Descending edge score
  • 39. Performance Work Demanding work load needs special tuning • Inserting 1 million edges per second • Excessive HLog (WAL) flushes
  • 40. Performance Work Batching • Client Side Batching — bulk edge insertion • Zen Server Side Batching — buffer edits across clients & flush together • Reduced HLog (WAL) flushes by orders of magnitude
  • 41. Performance Work Memory v.s. Performance • Bloom filter • reduce disk seeks • memory cost: 1 byte per row • Block size • the smaller block size the better random access performance • memory cost: bigger index size
  • 42. Performance Work CPU v.s. Data Size • Encoding • FAST_DIFF: effective in reducing data size, cpu intensive • PREFIX: less effective in size reduction, less cpu intensive • Compression • SNAPPY, LZO, GZ, etc
  • 43. Performance Work Capability to tune storage engine per special load Zen production setup • Dedicated Zen cluster • Namespace in shared Zen cluster
  • 44. Data Consistency Add an edge 1. CAS create the edge row and properties 2. CAS create the unique index if any 3. Create non-unique index if any 4. Create edge score index for outgoing direction 5. Create edge score index for incoming direction 6. Increment edge count for outgoing direction 7. Increment edge count for incoming direction
  • 46. Data Consistency Stay on top of data inconsistencies • Manual rollback in Zen server • Offline jobs (Dr Zen) to scan and fix inconsistencies • Tools to debug and fix one-off inconsistency
  • 47. Future Work •Dr Zen (make it more efficient) •Other backends: MySQL, etc •Distributed transactions •Open source!
  • 48.
  • 49. Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations/zen- pinterest-graph-storage-service