SlideShare a Scribd company logo
1 of 12
Complex graph traversal
across large, distributed datasets



Glue Conference 2010, Colorado
 Darren Wood, Lead Architect
         InfiniteGraph


           Copyright © InfiniteGraph
Scaling the [Social] Graph
      in the [Cloud]
         Darren Wood
  Lead Architect, InfiniteGraph
Graph Databases (Quickly)
• Optimized around data relationships
• Small focused API (typically not SQL)
• Typical Use Cases :
  – Social Graph Analysis
  – Catching Bad Guys (see Booth 16)
  – Fraud / Financial (more bad guys)
  – Data Intensive Science
  – Web / Advertising Analytics

                     Copyright © InfiniteGraph
Graph Databases (Almost Done)
        Vertex alice = myGraph.addVertex(new Person(“Alice”));
        Vertex bob = myGraph.addVertex(new Person(“Bob”));
        Vertex carlos = myGraph.addVertex(new Person(“Carlos”));
        Vertex charlie = myGraph.addVertex(new Person(“Charlie”));

        alice.addEdge(new Meeting(“Denver”, “5-27-10”), bob);
        bob.addEdge(new Call(timestamp), carlos);
        carlos.addEdge(new Payment(100000.00), charlie);
        bob.addEdge(new Call(timestamp), charlie);



Alice                       Bob                             Carlos             Charlie
             Meets                     Calls                            Pays



                                                                Calls

                                    Copyright © InfiniteGraph
What’s So Difficult Then ?
• Graphs grow quickly
  – Billions of phone calls / day in US
  – Emails, social media events, IP Traffic
  – Financial transactions
• Some analytics require navigation of large
  sections of the graph
• Each step (often) depends on the last
• Must distribute data and go parallel
                       Copyright © InfiniteGraph
First Some Good News…
• Graph algorithms naturally branch
• Can be automated or guided

                 Bob                            Carlos           Charlie
         Meets             Calls                         Pays

 Alice


         Calls
                 Chuck                          Dave              Eve
                            Lives                        Meets
                            With

                         Copyright © InfiniteGraph
Big Distributed Data
               (Traditional - Huge Generalization)

                             Application(s)




                              Distributed API

Processor      Processor                Processor          Processor




Partition 1    Partition 2              Partition 3       Partition ...n



                              Copyright © InfiniteGraph
Big Distributed Data
                                 (Graph)

                             Application(s)




                              Distributed API

Processor      Processor                Processor          Processor




Partition 1    Partition 2              Partition 3       Partition ...n



                              Copyright © InfiniteGraph
So What Are The Answers?
           Best Effort Partitioning

                Distributed API

  Processor                                   Processor




  Partition 1                                 Partition 2



                  Copyright © InfiniteGraph
So What Are The Answers?
                      The Look Ahead Example

                            Application

                           Distributed API

Processor                                               Processor


     A            C

                      D
              B
                                                         E

         Y
X

Partition 1                                             Partition 2

                            Copyright © InfiniteGraph
Which of These Work ?
• A carefully orchestrated combination of
  various options 
• Can be tuned (degree of look ahead)
• Healing graph can be expensive (write cost)
• This can also be tuned/configured (external
  edge thresholds)



                    Copyright © InfiniteGraph
Thankyou !
darren.wood@infinitegraph.com

   twitter.com/infinitegraph




         Copyright © InfiniteGraph

More Related Content

Similar to Gluecon InfiniteGraph Presentation: Scaling the Social Graph in the Cloud

Similar to Gluecon InfiniteGraph Presentation: Scaling the Social Graph in the Cloud (20)

RMG202 Rainmakers: How Netflix Operates Clouds for Maximum Freedom and Agilit...
RMG202 Rainmakers: How Netflix Operates Clouds for Maximum Freedom and Agilit...RMG202 Rainmakers: How Netflix Operates Clouds for Maximum Freedom and Agilit...
RMG202 Rainmakers: How Netflix Operates Clouds for Maximum Freedom and Agilit...
 
Solving Cybersecurity at Scale
Solving Cybersecurity at ScaleSolving Cybersecurity at Scale
Solving Cybersecurity at Scale
 
Drive Away Fraudsters With Driverless AI - Venkatesh Ramanathan, Senior Data ...
Drive Away Fraudsters With Driverless AI - Venkatesh Ramanathan, Senior Data ...Drive Away Fraudsters With Driverless AI - Venkatesh Ramanathan, Senior Data ...
Drive Away Fraudsters With Driverless AI - Venkatesh Ramanathan, Senior Data ...
 
Real-Time With AI – The Convergence Of Big Data And AI by Colin MacNaughton
Real-Time With AI – The Convergence Of Big Data And AI by Colin MacNaughtonReal-Time With AI – The Convergence Of Big Data And AI by Colin MacNaughton
Real-Time With AI – The Convergence Of Big Data And AI by Colin MacNaughton
 
Craftsmanship
CraftsmanshipCraftsmanship
Craftsmanship
 
Big Stream Processing Systems, Big Graphs
Big Stream Processing Systems, Big GraphsBig Stream Processing Systems, Big Graphs
Big Stream Processing Systems, Big Graphs
 
Bigdata analytics-twitter
Bigdata analytics-twitterBigdata analytics-twitter
Bigdata analytics-twitter
 
How we Learned to Stop Worrying and Solve the Distributed Graph Problem
How we Learned to Stop Worrying and Solve the Distributed Graph ProblemHow we Learned to Stop Worrying and Solve the Distributed Graph Problem
How we Learned to Stop Worrying and Solve the Distributed Graph Problem
 
David Loureiro - Presentation at HP's HPC & OSL TES
David Loureiro - Presentation at HP's HPC & OSL TESDavid Loureiro - Presentation at HP's HPC & OSL TES
David Loureiro - Presentation at HP's HPC & OSL TES
 
Venkatesh Ramanathan, Data Scientist, PayPal at MLconf ATL 2017
Venkatesh Ramanathan, Data Scientist, PayPal at MLconf ATL 2017Venkatesh Ramanathan, Data Scientist, PayPal at MLconf ATL 2017
Venkatesh Ramanathan, Data Scientist, PayPal at MLconf ATL 2017
 
Cloud is such stuff as dreams are made on
Cloud is such stuff as dreams are made onCloud is such stuff as dreams are made on
Cloud is such stuff as dreams are made on
 
Is It API Time For A New Strategy?
Is It API Time For A New Strategy?Is It API Time For A New Strategy?
Is It API Time For A New Strategy?
 
Big data and APIs for PHP developers - SXSW 2011
Big data and APIs for PHP developers - SXSW 2011Big data and APIs for PHP developers - SXSW 2011
Big data and APIs for PHP developers - SXSW 2011
 
Introduction to mobile programming with Androids.
Introduction to mobile programming with Androids. Introduction to mobile programming with Androids.
Introduction to mobile programming with Androids.
 
Performance Management in ‘Big Data’ Applications
Performance Management in ‘Big Data’ ApplicationsPerformance Management in ‘Big Data’ Applications
Performance Management in ‘Big Data’ Applications
 
How Can We Answer the Really BIG Questions?
How Can We Answer the Really BIG Questions?How Can We Answer the Really BIG Questions?
How Can We Answer the Really BIG Questions?
 
New Data Technologies, Graph Computing and Relationship Discovery in the Ente...
New Data Technologies, Graph Computing and Relationship Discovery in the Ente...New Data Technologies, Graph Computing and Relationship Discovery in the Ente...
New Data Technologies, Graph Computing and Relationship Discovery in the Ente...
 
Big data paris 2011 is cool florian douetteau
Big data paris 2011 is cool florian douetteauBig data paris 2011 is cool florian douetteau
Big data paris 2011 is cool florian douetteau
 
Scaling up business value with real-time operational graph analytics
Scaling up business value with real-time operational graph analyticsScaling up business value with real-time operational graph analytics
Scaling up business value with real-time operational graph analytics
 
GraphGen: Conducting Graph Analytics over Relational Databases
GraphGen: Conducting Graph Analytics over Relational DatabasesGraphGen: Conducting Graph Analytics over Relational Databases
GraphGen: Conducting Graph Analytics over Relational Databases
 

More from InfiniteGraph

An Introduction to Graph Databases
An Introduction to Graph DatabasesAn Introduction to Graph Databases
An Introduction to Graph Databases
InfiniteGraph
 
NoSQL Technology and Real-time, Accurate Predictive Analytics
NoSQL Technology and Real-time, Accurate Predictive AnalyticsNoSQL Technology and Real-time, Accurate Predictive Analytics
NoSQL Technology and Real-time, Accurate Predictive Analytics
InfiniteGraph
 
Vodafone xone fev142013v3 ext
Vodafone xone fev142013v3 extVodafone xone fev142013v3 ext
Vodafone xone fev142013v3 ext
InfiniteGraph
 
Dbta Webinar Realize Value of Big Data with graph 011713
Dbta Webinar Realize Value of Big Data with graph  011713Dbta Webinar Realize Value of Big Data with graph  011713
Dbta Webinar Realize Value of Big Data with graph 011713
InfiniteGraph
 
Oracle no sql overview brief
Oracle no sql overview briefOracle no sql overview brief
Oracle no sql overview brief
InfiniteGraph
 
Infinite graph nosql meetup dec 2012
Infinite graph nosql meetup dec 2012Infinite graph nosql meetup dec 2012
Infinite graph nosql meetup dec 2012
InfiniteGraph
 

More from InfiniteGraph (20)

Webinar 3/12/14: Using Social Media to Drive Value
Webinar 3/12/14: Using Social Media to Drive ValueWebinar 3/12/14: Using Social Media to Drive Value
Webinar 3/12/14: Using Social Media to Drive Value
 
NoSQL Simplified: Schema vs. Schema-less
NoSQL Simplified: Schema vs. Schema-lessNoSQL Simplified: Schema vs. Schema-less
NoSQL Simplified: Schema vs. Schema-less
 
The Value of Explicit Schema for Graph Use Cases
The Value of Explicit Schema for Graph Use CasesThe Value of Explicit Schema for Graph Use Cases
The Value of Explicit Schema for Graph Use Cases
 
Solution Use Case Demo: The Power of Relationships in Your Big Data
Solution Use Case Demo: The Power of Relationships in Your Big DataSolution Use Case Demo: The Power of Relationships in Your Big Data
Solution Use Case Demo: The Power of Relationships in Your Big Data
 
PowerOfRelationshipsInBigData_SVNoSQL
PowerOfRelationshipsInBigData_SVNoSQLPowerOfRelationshipsInBigData_SVNoSQL
PowerOfRelationshipsInBigData_SVNoSQL
 
Objectivity/DB: A Multipurpose NoSQL Database
Objectivity/DB: A Multipurpose NoSQL DatabaseObjectivity/DB: A Multipurpose NoSQL Database
Objectivity/DB: A Multipurpose NoSQL Database
 
Making sense of the Graph Revolution
Making sense of the Graph RevolutionMaking sense of the Graph Revolution
Making sense of the Graph Revolution
 
An Introduction to Graph Databases
An Introduction to Graph DatabasesAn Introduction to Graph Databases
An Introduction to Graph Databases
 
Using A Distributed Graph Database To Make Sense Of Disparate Data Stores
Using A Distributed Graph Database To Make Sense Of Disparate Data StoresUsing A Distributed Graph Database To Make Sense Of Disparate Data Stores
Using A Distributed Graph Database To Make Sense Of Disparate Data Stores
 
Turning Big Data into Smart Data with Graph Technologies
Turning Big Data into Smart Data with Graph TechnologiesTurning Big Data into Smart Data with Graph Technologies
Turning Big Data into Smart Data with Graph Technologies
 
NoSQL Technology and Real-time, Accurate Predictive Analytics
NoSQL Technology and Real-time, Accurate Predictive AnalyticsNoSQL Technology and Real-time, Accurate Predictive Analytics
NoSQL Technology and Real-time, Accurate Predictive Analytics
 
Everything Goes Better With Bacon: Revisiting the Six Degrees Problem with a ...
Everything Goes Better With Bacon: Revisiting the Six Degrees Problem with a ...Everything Goes Better With Bacon: Revisiting the Six Degrees Problem with a ...
Everything Goes Better With Bacon: Revisiting the Six Degrees Problem with a ...
 
Vodafone xone fev142013v3 ext
Vodafone xone fev142013v3 extVodafone xone fev142013v3 ext
Vodafone xone fev142013v3 ext
 
Dbta Webinar Realize Value of Big Data with graph 011713
Dbta Webinar Realize Value of Big Data with graph  011713Dbta Webinar Realize Value of Big Data with graph  011713
Dbta Webinar Realize Value of Big Data with graph 011713
 
Oracle no sql overview brief
Oracle no sql overview briefOracle no sql overview brief
Oracle no sql overview brief
 
Infinite graph nosql meetup dec 2012
Infinite graph nosql meetup dec 2012Infinite graph nosql meetup dec 2012
Infinite graph nosql meetup dec 2012
 
Oracle NoSQL DB & InfiniteGraph - Trends in Big Data and Graph Technology
Oracle NoSQL DB & InfiniteGraph - Trends in Big Data and Graph TechnologyOracle NoSQL DB & InfiniteGraph - Trends in Big Data and Graph Technology
Oracle NoSQL DB & InfiniteGraph - Trends in Big Data and Graph Technology
 
Silicon valley nosql meetup april 2012
Silicon valley nosql meetup  april 2012Silicon valley nosql meetup  april 2012
Silicon valley nosql meetup april 2012
 
NOSQL Now! Presentation, August 24, 2011: Graph Databases: Connecting the Dot...
NOSQL Now! Presentation, August 24, 2011: Graph Databases: Connecting the Dot...NOSQL Now! Presentation, August 24, 2011: Graph Databases: Connecting the Dot...
NOSQL Now! Presentation, August 24, 2011: Graph Databases: Connecting the Dot...
 
Meetup: An Introduction to InfiniteGraph, and Connecting the Dots in Big Data.
Meetup: An Introduction to InfiniteGraph, and Connecting the Dots in Big Data.Meetup: An Introduction to InfiniteGraph, and Connecting the Dots in Big Data.
Meetup: An Introduction to InfiniteGraph, and Connecting the Dots in Big Data.
 

Recently uploaded

Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native Applications
WSO2
 
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Victor Rentea
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@
 
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Victor Rentea
 

Recently uploaded (20)

Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...
 
Ransomware_Q4_2023. The report. [EN].pdf
Ransomware_Q4_2023. The report. [EN].pdfRansomware_Q4_2023. The report. [EN].pdf
Ransomware_Q4_2023. The report. [EN].pdf
 
Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native Applications
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
 
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
 
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
 
Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : Uncertainty
 
[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdf[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdf
 
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWEREMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
 
DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
DEV meet-up UiPath Document Understanding May 7 2024 AmsterdamDEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
 
ICT role in 21st century education and its challenges
ICT role in 21st century education and its challengesICT role in 21st century education and its challenges
ICT role in 21st century education and its challenges
 
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 

Gluecon InfiniteGraph Presentation: Scaling the Social Graph in the Cloud

  • 1. Complex graph traversal across large, distributed datasets Glue Conference 2010, Colorado Darren Wood, Lead Architect InfiniteGraph Copyright © InfiniteGraph
  • 2. Scaling the [Social] Graph in the [Cloud] Darren Wood Lead Architect, InfiniteGraph
  • 3. Graph Databases (Quickly) • Optimized around data relationships • Small focused API (typically not SQL) • Typical Use Cases : – Social Graph Analysis – Catching Bad Guys (see Booth 16) – Fraud / Financial (more bad guys) – Data Intensive Science – Web / Advertising Analytics Copyright © InfiniteGraph
  • 4. Graph Databases (Almost Done) Vertex alice = myGraph.addVertex(new Person(“Alice”)); Vertex bob = myGraph.addVertex(new Person(“Bob”)); Vertex carlos = myGraph.addVertex(new Person(“Carlos”)); Vertex charlie = myGraph.addVertex(new Person(“Charlie”)); alice.addEdge(new Meeting(“Denver”, “5-27-10”), bob); bob.addEdge(new Call(timestamp), carlos); carlos.addEdge(new Payment(100000.00), charlie); bob.addEdge(new Call(timestamp), charlie); Alice Bob Carlos Charlie Meets Calls Pays Calls Copyright © InfiniteGraph
  • 5. What’s So Difficult Then ? • Graphs grow quickly – Billions of phone calls / day in US – Emails, social media events, IP Traffic – Financial transactions • Some analytics require navigation of large sections of the graph • Each step (often) depends on the last • Must distribute data and go parallel Copyright © InfiniteGraph
  • 6. First Some Good News… • Graph algorithms naturally branch • Can be automated or guided Bob Carlos Charlie Meets Calls Pays Alice Calls Chuck Dave Eve Lives Meets With Copyright © InfiniteGraph
  • 7. Big Distributed Data (Traditional - Huge Generalization) Application(s) Distributed API Processor Processor Processor Processor Partition 1 Partition 2 Partition 3 Partition ...n Copyright © InfiniteGraph
  • 8. Big Distributed Data (Graph) Application(s) Distributed API Processor Processor Processor Processor Partition 1 Partition 2 Partition 3 Partition ...n Copyright © InfiniteGraph
  • 9. So What Are The Answers? Best Effort Partitioning Distributed API Processor Processor Partition 1 Partition 2 Copyright © InfiniteGraph
  • 10. So What Are The Answers? The Look Ahead Example Application Distributed API Processor Processor A C D B E Y X Partition 1 Partition 2 Copyright © InfiniteGraph
  • 11. Which of These Work ? • A carefully orchestrated combination of various options  • Can be tuned (degree of look ahead) • Healing graph can be expensive (write cost) • This can also be tuned/configured (external edge thresholds) Copyright © InfiniteGraph
  • 12. Thankyou ! darren.wood@infinitegraph.com twitter.com/infinitegraph Copyright © InfiniteGraph