Solving Problems with Graphs

  • 8,917 views
Uploaded on

Who am I and why do I feel that the world is not infinitely perfect? Which technologies should I use to rectify this situation? Enter the graph and the graph traversal.

Who am I and why do I feel that the world is not infinitely perfect? Which technologies should I use to rectify this situation? Enter the graph and the graph traversal.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
8,917
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
431
Comments
0
Likes
53

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. SOLVING PROBLEMS WITH GRAPHS MARKO A. RODRIGUEZ http://THINKAURELIUS.COM
  • 2. MARKO A. RODRIGUEZ
  • 3. MARKO A. RODRIGUEZ
  • 4. MARKO A. RODRIGUEZ
  • 5. YOU WILL NOT KNOW ME BY LOOKING WITHIN
  • 6. marko
  • 7. marko TO KNOW ME IS TO KNOW MY WORLD
  • 8. gremlin marko AND IN MY WORLD THERE ARE THINGSrussell
  • 9. gremlin a ted cre marko s ow knrussell
  • 10. gremlin a ted cre marko AND TO KNOW THESE THINGS s ow kn IS TO KNOW THEIR WORLD ... AD INFINITUMrussell
  • 11. pavel ted crea gremlin de pe nd s a ted cre marko blueprints s ow knrussell hortonworks works
  • 12. pavel ted created crea cql gremlin works de pe nd s twitter a ted cre marko s de d pe en nd d ep blueprints s titan s ow kn faunusrussell use use hadoop s s hbase hortonworks works
  • 13. pavel ted created crea cql gremlin works de pe depe nd s twitter nds a ted cre depends marko uses s de nd pe pe blueprints nd de s titan cassandra s s created epend ow d kn s depends nd faunus pe cloudera de ds de pen dependsrussell uses use use hadoop s s hbase hortonworks works
  • 14. I POSIT THAT THE SOLUTIONS TO PROBLEMS EXIST IN THE WORLD AND THAT RESOLUTION IS SOUGHT IN LINKAGE. pavel ted created crea cql gremlin works de pe depe nd s twitter nds a ted cre depends marko uses s de nd pe pe blueprints nd de s titan cassandra s s created epend ow d kn s depends nd faunus pe cloudera de ds de pen dependsrussell uses use use hadoop s s hbase hortonworks works
  • 15. PROBLEM• I want organizations with Big Graph Data to use the Aurelius Graph Cluster. pavel ted created crea cql gremlin works de pe depe nd s twitter nds a ted cre depends marko uses s de nd pe pe blueprints nd de s titan cassandra s s created epend ow d kn s depends nd faunus pe cloudera de ds de pen depends russell uses use use hadoop s s hbase hortonworks works
  • 16. PROBLEM• I want organizations with Big Graph Data to use the Aurelius Graph Cluster. pavelSOLUTION #1 crea ted created cql• Hortonworks and Cloudera gremlin use Big Data technology. works de pe depe nd s twitter nds a ted cre depends marko uses s de nd pe pe blueprints nd de s titan cassandra s s created epend ow d kn s depends nd faunus pe cloudera de ds de pen depends russell uses use use hadoop s s hbase hortonworks works
  • 17. PROBLEM• I want organizations with Big Graph Data to use the Aurelius Graph Cluster. pavelSOLUTION #1 crea ted created cql• Hortonworks and Cloudera gremlin use Big Data technology. works I know Russell. de pe depe nd s twitter nds a ted cre depends marko uses s de nd pe pe blueprints nd de s titan cassandra s s created epend ow d kn s depends nd faunus pe cloudera de ds de pen depends russell uses use use hadoop s s hbase hortonworks works
  • 18. PROBLEM• I want organizations with Big Graph Data to use the Aurelius Graph Cluster. pavelSOLUTION #1 crea ted created cql• Hortonworks and Cloudera gremlin use Big Data technology. works I know Russell. de pe depe Partnership? nd s twitter nds a ted cre depends marko uses s de nd pe pe blueprints nd de s titan cassandra s s created epend ow d kn s depends nd faunus pe cloudera de ds de pen depends russell uses use use hadoop s s hbase hortonworks works ner part r partne
  • 19. pavel ted created crea cql gremlin works de pe depe nd s twitter nds a ted cre depends marko uses s de nd pe pe blueprints nd de s titan cassandra s s created epend ow d kn s depends nd faunus pe cloudera de ds de pen dependsrussell uses use use hadoop s s hbase hortonworks works
  • 20. PROBLEM• I want organizations with Big Graph Data to use the Aurelius Graph Cluster. pavel ted created crea cql gremlin works de pe depe nd s twitter nds a ted cre depends marko uses s de nd pe pe blueprints nd de s titan cassandra s s created epend ow d kn s depends nd faunus pe cloudera de ds de pen depends russell uses use use hadoop s s hbase hortonworks works
  • 21. PROBLEM• I want organizations with Big Graph Data to use the Aurelius Graph Cluster. pavel ted createdSOLUTION #2 crea cql gremlin• Pavel and I created Gremlin. works de pe depe nd s twitter nds a ted cre depends marko uses s de nd pe pe blueprints nd de s titan cassandra s s created epend ow d kn s depends nd faunus pe cloudera de ds de pen depends russell uses use use hadoop s s hbase hortonworks works
  • 22. PROBLEM• I want organizations with Big Graph Data to use the Aurelius Graph Cluster. pavel ted createdSOLUTION #2 crea cql gremlin• Pavel and I created Gremlin. works He is a Cassandra engineer de at Twitter. pe depe nd s twitter nds a ted cre depends marko uses s de nd pe pe blueprints nd de s titan cassandra s s created epend ow d kn s depends nd faunus pe cloudera de ds de pen depends russell uses use use hadoop s s hbase hortonworks works
  • 23. PROBLEM• I want organizations with Big Graph Data to use the Aurelius Graph Cluster. pavel ted createdSOLUTION #2 crea cql gremlin• Pavel and I created Gremlin. e ts me works He is a Cassandra engineer de at Twitter. pe depe nd Meeting? s twitter nds a ted cre depends marko uses s de nd pe pe blueprints nd de s titan cassandra s s created epend ow d kn s depends nd faunus pe cloudera de ds de pen depends russell uses use use hadoop s s hbase hortonworks works
  • 24. pavel ted created crea cql gremlin works de pe depe nd s twitter nds a ted cre depends marko uses s de nd pe pe blueprints nd de s titan cassandra s s created epend ow d kn s depends nd faunus pe cloudera de ds de pen dependsrussell uses use use hadoop s s hbase hortonworks works
  • 25. pavel ted created crea cql gremlin SOFTWARE works de pe depe nd s twitter nds a ted cre depends marko uses s de nd pe pe blueprints nd de s titan cassandra s s created epend ow d kn s depends nd faunus pe cloudera de ds de pen dependsrussell uses use use hadoop s s hbase hortonworks works
  • 26. marko SOFTWARE
  • 27. marko SOFTWARE
  • 28. marko SOFTWARE
  • 29. marko
  • 30. marko pet pet mamapuppy
  • 31. marko pet pet sniffs mamapuppy
  • 32. marko chula pet pet i ffs sn sn iffs sniffs mama sniffs puppy curley biscuitmia iffs sn sn scout iffs sniffs jackson missy whettle
  • 33. marko chula pet pet i ffs sn sn iffs sniffs mama sniffs puppy curley biscuitmia males iffs sn sn scout iffs sniffs jackson missy whettle
  • 34. marko chula pet pet i ffs sn sn iffs sniffs mama sniffs sniffs puppy curley biscuit sn iffs sniffsmia males sni sni ffs s iff ffs sn iffs sn sn scout iffs sniffs jackson missy whettle
  • 35. marko chula pet pet i ffs sn sn iffs sniffs mama 1 sniffs sniffs puppy curley biscuit sn iffs sniffsmia males sni sni ffs s iff ffs sn iffs sn sn scout iffs sniffs jackson missy whettle
  • 36. marko chula pet 2 pet i ffs sn sn iffs sniffs mama 2 1 sniffs sniffs puppy curley biscuit sn iffs sniffsmia males sni sni ffs s iff ffs sn iffs sn sn scout iffs sniffs jackson missy whettle
  • 37. marko chula pet 2 pet i ffs sn sn iffs sniffs mama 2 sniffs sniffs puppy curley biscuit sn iffs 3 sniffsmia males sni sni ffs s iff ffs sn iffs sn sn scout iffs sniffs jackson missy whettle
  • 38. marko chula pet pet i ffs sn sn iffs 4 sniffs mama sniffs sniffs puppy curley biscuit sn iffs 3 sniffsmia males sni 4 sni ffs s iff ffs sn iffs sn sn scout iffs 4 4 4 sniffs jackson missy whettle
  • 39. marko chula pet pet i ffs sn sn iffs 5 4 sniffs mama sniffs sniffs puppy curley biscuit sn iffs sniffsmia males sni 4 sni 5 ffs s iff ffs sn iffs sn sn scout iffs 4 4 4 sniffs jackson missy whettle
  • 40. marko chula pet pet i ffs sn sn iffs 5 sniffs mama sniffs sniffs puppy curley biscuit sn iffs sniffsmia males sni sni 5 ffs s iff ffs sn iffs sn sn scout iffs sniffs jackson missy whettle
  • 41. marko chula pet pet i ffs sn sn iffs sniffs mama sniffs sniffs puppy curley biscuit sn iffs sniffsmia males sni sni ffs s iff ffs sn iffs sn sn scout iffs sniffs jackson missy whettle
  • 42. marko chula pet pet i ffs sn sn iffs sniffs mama sniffs sniffs puppy curley biscuit sn iffs sniffsmia males sni sni ffs s iff ffs sn iffs sn sn scout iffs sniffs jackson missy whettle
  • 43. PROBLEM • What is the best way to save this dog community from disease? marko chula pet pet i ffs sn sn iffs sniffs mama sniffs sniffs puppy curley biscuit sn iffs sniffsmia males sni sni ffs s iff ffs sn iffs sn sn scout iffs sniffs jackson missy whettle
  • 44. PROBLEM • What is the best way to save this dog community from disease? marko chula pet pet i ffs sn sn iffs sniffs mama sniffs sniffs puppy curley biscuit sn iffs sniffsmia males sni sni ffs s iff ffs sn iffs sn sn scout iffs sniffs jackson missy whettle
  • 45. PROBLEM • What is the best way to save this dog community from disease? SOLUTION marko • Inoculate the most central dog. chula pet pet i ffs sn sn e ulat iffsinoc sniffs mama sniffs sniffs puppy curley biscuit sn iffs sniffs mia males sni sni ffs s iff ffs sn iffs sn sn scout iffs sniffs jackson missy whettle
  • 46. PROBLEM • What is the best way to save this dog community from disease? SOLUTION marko • Inoculate the most central dog. chula pet pet i ffs sn sn iffs sniffs mama sniffs sniffs puppy curley biscuit sn iffs sniffsmia males sni sni ffs s iff ffs sn iffs sn sn scout iffs sniffs jackson missy whettle
  • 47. marko chula pet pet i ffs sn sn iffs sniffs mama sniffs sniffs puppy curley biscuit sn iffs sniffsmia males sni sni ffs s iff ffs sn iffs sn sn scout iffs sniffs jackson missy whettle
  • 48. marko chula DOGS pet pet i ffs sn sn iffs sniffs mama sniffs sniffs puppy curley biscuit sn iffs sniffsmia males sni sni ffs s iff ffs sn iffs sn sn scout iffs sniffs jackson missy whettle
  • 49. markoDOGS
  • 50. marko SOFTWAREDOGS
  • 51. marko SOFTWAREDOGS
  • 52. marko SOFTWAREDOGS
  • 53. marko SOFTWAREDOGS
  • 54. marko SOFTWAREDOGS
  • 55. marko
  • 56. avalanchemarko captain
  • 57. blackhawks wild panthers avalanchemarko captain
  • 58. game #1 played blackhawks #2 wild #3 panthers #4 avalanchemarko #5 captain ... #n
  • 59. game #1 played blackhawks #2 period #1 wild p1 #3 #2 p2 p3 panthers #4 #3 avalanchemarko #5 captain ... #n
  • 60. game #1 played blackhawks #2 period #1 wild p1 #3 #2 p2 p3 panthers #4 #3 ls goal goa penalty #1 #1 avalanche penaltiesmarko #5 #2 #2 captain ... #3 #n
  • 61. game #1 steve jason craig patrick ... played blackhawks #2 period #1 wild p1 #3 #2 p2 p3 panthers #4 #3 ls goal goa penalty #1 #1 avalanche penaltiesmarko #5 #2 #2 captain ... #3 #n
  • 62. game #1 steve jason craig patrick ... played blackhawks #2 period #1 received scored assisted scored wild p1 #3 #2 p2 p3 panthers #4 #3 ls goal goa penalty #1 assisted #1 avalanche penaltiesmarko #5 #2 #2 captain ... #3 #n
  • 63. PROBLEM game #1 steve jason craig patrick• Who do I draft for the 2012/2013 season? ... played blackhawks #2 period #1 received scored assisted scored wild p1 #3 #2 p2 p3 panthers #4 #3 ls goal goa penalty #1 assisted #1 avalanche penalties marko #5 #2 #2 captain ... #3 #n
  • 64. PROBLEM game #1 steve jason craig patrick• Who do I draft for the 2012/2013 season? ... played blackhawks #2 period #1 received scored assisted scored wild p1 #3 #2 p2 p3 panthers #4 #3 ls goal goa penalty #1 assisted #1 avalanche penalties marko #5 #2 #2 captain ... #3SOLUTION #n• Craig/Patrick play well together.
  • 65. PROBLEM game #1 steve jason craig patrick• Who do I draft for the 2012/2013 season? ... played blackhawks #2 period #1 received scored assisted scored wild p1 #3 #2 p2 p3 panthers #4 #3 ls goal goa penalty #1 assisted #1 avalanche penalties draft marko #5 #2 #2 captain ... #3 draftSOLUTION #n• Craig/Patrick play well together. Draft them as a pair?
  • 66. game #1 steve jason craig patrick ... played blackhawks #2 period #1 received scored assisted scored wild p1 #3 #2 p2 p3 panthers #4 #3 ls goal goa penalty #1 assisted #1 avalanche penaltiesmarko #5 #2 #2 captain ... #3 SPORTS #n
  • 67. SPORTSmarko
  • 68. SPORTSmarko SOFTWARE DOGS
  • 69. THIS IS JUSTMY SUBSET OF THE WORLD ANDMY PERSONAL PROBLEMS.
  • 70. OUR PERSONAL WORLDS ...
  • 71. ... ARE EMBEDDED WITHIN A LARGER WORLD ...
  • 72. ... OF OTHER PEOPLEAND ARTIFACTS ...
  • 73. ... AND TOGETHER WE NAVIGATE THAT WORLD ...
  • 74. ... TRYING TO BETTER OUR LIVES ...
  • 75. ... BY SOLVING OUR PROBLEMS ...
  • 76. ... BY OPTIMALLY LINKING OURSELVES WITHIN THE WORLD.
  • 77. WHY DO WE ENCODE OURSELVES?
  • 78. These are my friends. I think about these ideas. I like these books.This where I work. I am. I visit these webpages. I work on these projects. I wrote these articles. WHY ARE WE CREATING A UNIVERSAL MODEL?
  • 79. Who should I befriend? What ideas will inspire me? Where is the best place for me to live?What movie should I watch tonight with the friends Im meeting up with? What career path should I choose given my interests and expertise? Who should I fall in love with and live my life with? BECAUSE WE HAVE FAITH IN THE ALGORITHM.
  • 80. WE HAVE FAITH THAT TOGETHER, WITH COMPUTERS,WE WILL DETERMINE THE OPTIMAL EMBEDDING.
  • 81. WE WILL GENERATE ENTHRALLING CONNECTIONS ...
  • 82. THAT RESONATES US ...
  • 83. TO CREATE AND EXPERIENCE ...
  • 84. EVEN GREATER THINGS.
  • 85. HOW DO WE STORE AND PROCESS A WORLD MODEL?BILLIONS OF VERTICES TRILLIONS OF EDGES SOLVING MILLIONS OF PROBLEMS/SECOND
  • 86. AURELIUSGRAPH CLUSTER
  • 87. GRAPH COMPUTING
  • 88. GRAPH COMPUTINGDATA STRUCTURE ALGORITHM + GRAPH TRAVERSAL
  • 89. GRAPHname:marko name:Fountain Head age:32 published:1943 likes star:5 time:2002
  • 90. GRAPH KEYPROPERTIES VALUE EDGE LABEL name:marko name:Fountain Head age:32 published:1943 likes star:5 time:2002 VERTEX
  • 91. TRAVERSAL 3 2 31 3 2 3 A traversal is an algorithmic walk over a (sub)graph in order to make explicit information that is implicit within its structure. 1.) graph derivation ("my fathers father is my grandfather.") 2.) graph statistic ("many paths lead to Rome.")
  • 92. TRAVERSAL lives lives name:jen birthmonth:2 name:santa fe likes A name:russellname:marko knows C likes likes knows likes knows likes likes likes likes D name:josh B birthmonth:12 name:pavel
  • 93. TRAVERSAL lives lives name:jen birthmonth:2 name:santa fe likes A name:russellname:marko knows C likes likes knows likes knows likes likes likes likes D name:josh B birthmonth:12 name:pavel PROBLEM • What should I buy for any upcoming friends birthday?
  • 94. TRAVERSAL lives lives name:jen birthmonth:2 name:santa fe likes A name:russellname:marko knows C likes likes knows likes knows likes likes likes likes D name:josh B birthmonth:12 name:pavel g.V(name,marko)
  • 95. TRAVERSAL lives lives name:jen birthmonth:2 name:santa fe likes A name:russellname:marko knows C likes likes knows likes knows likes likes likes likes D name:josh B birthmonth:12 name:pavel g.V(name,marko).out(knows)
  • 96. TRAVERSAL lives lives name:jen birthmonth:2 name:santa fe likes A name:russellname:marko knows C likes likes knows likes knows likes likes likes likes D name:josh B birthmonth:12 name:pavel g.V(name,marko).out(knows) .filter{(it.birthmonth + currentMonth) % 12 < 2}
  • 97. TRAVERSAL lives lives name:jen birthmonth:2 name:santa fe likes A name:russellname:marko knows C likes likes knows likes knows likes likes likes likes D name:josh B birthmonth:12 name:pavel g.V(name,marko).out(knows) .filter{(it.birthmonth + currentMonth) % 12 < 2} .out(likes)
  • 98. TRAVERSAL lives lives name:jen birthmonth:2 name:santa fe likes A name:russellname:marko knows C likes likes likes x knows knows likes likes likes likes D name:josh B birthmonth:12 name:pavel g.V(name,marko).out(knows) .filter{(it.birthmonth + currentMonth) % 12 < 2} .out(likes).aggregate(x)
  • 99. TRAVERSAL lives lives name:jen birthmonth:2 name:santa fe likes A name:russellname:marko knows C likes likes likes x knows knows likes likes likes likes D name:josh B birthmonth:12 name:pavel g.V(name,marko).out(knows) .filter{(it.birthmonth + currentMonth) % 12 < 2} .out(likes).aggregate(x).in(likes)
  • 100. TRAVERSAL lives lives name:jen birthmonth:2 name:santa fe likes A name:russellname:marko knows C likes likes likes x knows knows likes likes likes likes D name:josh B birthmonth:12 name:pavel g.V(name,marko).out(knows) .filter{(it.birthmonth + currentMonth) % 12 < 2} .out(likes).aggregate(x).in(likes).out(like)
  • 101. TRAVERSAL lives lives name:jen birthmonth:2 name:santa fe likes A name:russellname:marko knows C likes likes likes x knows knows likes likes likes likes D name:josh B birthmonth:12 name:pavel g.V(name,marko).out(knows) .filter{(it.birthmonth + currentMonth) % 12 < 2} .out(likes).aggregate(x).in(likes).out(like) .except(x)
  • 102. TRAVERSAL lives lives name:jen birthmonth:2 name:santa fe likes A name:russellname:marko knows C likes likes knows likes knows likes likes likes likes D name:josh B birthmonth:12 name:pavel g.V(name,marko).out(knows) .filter{(it.birthmonth + currentMonth) % 12 < 2} .out(likes).aggregate(x).in(likes).out(like) .except(x)
  • 103. TRAVERSAL lives lives name:jen birthmonth:2 C:2 name:santa fe D:1 likes A name:russellname:marko knows C likes likes knows likes knows likes likes likes likes D name:josh B birthmonth:12 name:pavel g.V(name,marko).out(knows) .filter{(it.birthmonth + currentMonth) % 12 < 2} .out(likes).aggregate(x).in(likes).out(like) .except(x).groupCount()
  • 104. TRAVERSAL lives lives name:jen birthmonth:2 C:2 name:santa fe D:1 likes likes A name:russellname:marko knows C likes likes knows likes knows likes likes likes likes D name:josh B birthmonth:12 name:pavel g.V(name,marko).out(knows) .filter{(it.birthmonth + currentMonth) % 12 < 2} .out(likes).aggregate(x).in(likes).out(like) .except(x).groupCount()
  • 105. GRAPH COMPUTING SYSTEMS
  • 106. AURELIUS GRAPH CLUSTER TITAN FAUNUS FULGORADistributed Graph Database Graph Analytics Engine Fast Graph Processor Apache 2 Licensed
  • 107. TITANDISTRIBUTED GRAPH DATABASE
  • 108. TITAN DISTRIBUTED GRAPH DATABASERepresents the world as a single, atomic graph structure.
  • 109. TITAN DISTRIBUTED GRAPH DATABASERepresents the world as a single, atomic graph structure. Distributed over a multi-machine cluster using existing distributed data systems such as Cassandra and HBase.
  • 110. TITAN DISTRIBUTED GRAPH DATABASERepresents the world as a single, atomic graph structure. Distributed over a multi-machine cluster using existing distributed data systems such as Cassandra and HBase. Supporting numerous short-lived, topologically local, real-time traversals.
  • 111. FAUNUSGRAPH ANALYTICS ENGINE
  • 112. FAUNUS GRAPH ANALYTICS ENGINEExtracts an ephemeral snapshot of the master graph.
  • 113. FAUNUS GRAPH ANALYTICS ENGINEExtracts an ephemeral snapshot of the master graph.Leverages Hadoop as its distributed computing engine.
  • 114. FAUNUS GRAPH ANALYTICS ENGINEExtracts an ephemeral snapshot of the master graph.Leverages Hadoop as its distributed computing engine.Executing long-lived, topologically global analyses of the graph.
  • 115. FULGORAFAST GRAPH PROCESSOR
  • 116. FULGORA FAST GRAPH PROCESSORStores a compressed subset of the master graph.
  • 117. FULGORA FAST GRAPH PROCESSORStores a compressed subset of the master graph. Contained within the confines of a single high memory/CPU machine.
  • 118. FULGORA FAST GRAPH PROCESSORStores a compressed subset of the master graph. Contained within the confines of a single high memory/CPU machine.Evaluating heavily threaded, memory efficient graph and machine learning algorithms. }
  • 119. AURELIUS GRAPH CLUSTER AN INTEGRATED BIG GRAPH DATA SOLUTION TITAN FAUNUS FULGORADistributed Graph Database Graph Analytics Engine Fast Graph Processor Output Graph Output Statistics Map/Reduce Output Statistics Update Graph Other Data Analysis Tools Update Graph Apache 2 Licensed
  • 120. CREDITS PRESENTER MARKO A. RODRIGUEZ CONTRIBUTORSMATTHIAS BROCHELER STEPHEN MALLETTE DAN LAROCQUE VADAS GINTAUTAS MANY THANKS TO AURELIUS COMMUNITYTINKERPOP COMMUNITY KETRINA YIM
  • 121. RELATED MATERIAL FAITH IN THE ALGORITHM 2:COMPUTATIONAL EUDAEMONICS http://arxiv.org/abs/0904.0027 A COLLECTIVELY GENERATED MODEL OF THE WORLD http://markorodriguez.files.wordpress.com/2011/01/collective-model.pdf STRUCTURAL ABSTRACTIONS IN THE BRAIN http://thinkaurelius.com/2012/05/08/structural-abstractions-in-brains-and-graphs/GRAPHS, BRAINS, AND GREMLIN http://markorodriguez.com/2011/07/14/graphs-brains-and-gremlin/