Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

of

Adding Value through graph analysis using Titan and Faunus Slide 1 Adding Value through graph analysis using Titan and Faunus Slide 2 Adding Value through graph analysis using Titan and Faunus Slide 3 Adding Value through graph analysis using Titan and Faunus Slide 4 Adding Value through graph analysis using Titan and Faunus Slide 5 Adding Value through graph analysis using Titan and Faunus Slide 6 Adding Value through graph analysis using Titan and Faunus Slide 7 Adding Value through graph analysis using Titan and Faunus Slide 8 Adding Value through graph analysis using Titan and Faunus Slide 9 Adding Value through graph analysis using Titan and Faunus Slide 10 Adding Value through graph analysis using Titan and Faunus Slide 11 Adding Value through graph analysis using Titan and Faunus Slide 12 Adding Value through graph analysis using Titan and Faunus Slide 13 Adding Value through graph analysis using Titan and Faunus Slide 14 Adding Value through graph analysis using Titan and Faunus Slide 15 Adding Value through graph analysis using Titan and Faunus Slide 16 Adding Value through graph analysis using Titan and Faunus Slide 17 Adding Value through graph analysis using Titan and Faunus Slide 18 Adding Value through graph analysis using Titan and Faunus Slide 19 Adding Value through graph analysis using Titan and Faunus Slide 20 Adding Value through graph analysis using Titan and Faunus Slide 21 Adding Value through graph analysis using Titan and Faunus Slide 22 Adding Value through graph analysis using Titan and Faunus Slide 23 Adding Value through graph analysis using Titan and Faunus Slide 24 Adding Value through graph analysis using Titan and Faunus Slide 25 Adding Value through graph analysis using Titan and Faunus Slide 26 Adding Value through graph analysis using Titan and Faunus Slide 27 Adding Value through graph analysis using Titan and Faunus Slide 28 Adding Value through graph analysis using Titan and Faunus Slide 29 Adding Value through graph analysis using Titan and Faunus Slide 30 Adding Value through graph analysis using Titan and Faunus Slide 31 Adding Value through graph analysis using Titan and Faunus Slide 32 Adding Value through graph analysis using Titan and Faunus Slide 33 Adding Value through graph analysis using Titan and Faunus Slide 34 Adding Value through graph analysis using Titan and Faunus Slide 35 Adding Value through graph analysis using Titan and Faunus Slide 36 Adding Value through graph analysis using Titan and Faunus Slide 37 Adding Value through graph analysis using Titan and Faunus Slide 38 Adding Value through graph analysis using Titan and Faunus Slide 39 Adding Value through graph analysis using Titan and Faunus Slide 40 Adding Value through graph analysis using Titan and Faunus Slide 41 Adding Value through graph analysis using Titan and Faunus Slide 42 Adding Value through graph analysis using Titan and Faunus Slide 43 Adding Value through graph analysis using Titan and Faunus Slide 44 Adding Value through graph analysis using Titan and Faunus Slide 45 Adding Value through graph analysis using Titan and Faunus Slide 46 Adding Value through graph analysis using Titan and Faunus Slide 47 Adding Value through graph analysis using Titan and Faunus Slide 48 Adding Value through graph analysis using Titan and Faunus Slide 49 Adding Value through graph analysis using Titan and Faunus Slide 50 Adding Value through graph analysis using Titan and Faunus Slide 51
Upcoming SlideShare
Big Graph Data
Next
Download to read offline and view in fullscreen.

25 Likes

Share

Download to read offline

Adding Value through graph analysis using Titan and Faunus

Download to read offline

In this presentation we discuss how graph analysis can add value to your data and how to use open source tools like Titan and Faunus to build scalable graph processing systems.
This presentation gives an update on the development status of Titan and Faunus with a preview of what is to come.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Adding Value through graph analysis using Titan and Faunus

  1. 1. KNOWLEDGE INFORMATION DATA Adding Value Through Graph Analysis Matthias Broecheler, CTO @mbroecheler AURELIUS March V, MMXIII THINKAURELIUS.COM
  2. 2. " " " " " " " " " Communities of Interest Finding Influencers " Understanding Behavior "
  3. 3. " " " " " " " " " Information Integration Recommendation " Question Answering "
  4. 4. " " " " " " " " " Fraud Detection Risk Analysis " Market Valuation "
  5. 5. Knowledge Value Information Data
  6. 6. likes(Jane Joe, cute mamals):0.8 Knowledge userid:3552 " clicked timestamp: addid:9914 Information 93932342 " 2013-03-03 18:52:48:112; 12.123.211.192; ACCESS/TRR; http://adserve.domain.com/ render.cgi? uid=F32282DA39B&flagtru&xls=trendi Data ng ; ACTION=CLICK|DELAY=250|x=450| y=632!
  7. 7. Graph Databases & likes(Jane Joe, cute mamals):0.8 Graph Analysis Knowledge userid:3552 " clicked timestamp: addid:9914 Information 93932342 " 2013-03-03 18:52:48:112; 12.123.211.192; ACCESS/TRR; http://adserve.domain.com/ render.cgi? uid=F32282DA39B&flagtru&xls=trendi Data ng ; ACTION=CLICK|DELAY=250|x=450| y=632!
  8. 8. I Graph Foundation AURELIUS THINKAURELIUS.COM
  9. 9. name: Neptune name: Alcmene type: god type: god Vertex Property name: Saturn name: Jupiter name: Hercules type: titan type: god type: demigod name: Pluto name: Cerberus type: god type: monster Graph
  10. 10. name: Neptune name: Alcmene type: god type: god Edge brother mother name: Saturn name: Jupiter name: Hercules type: titan type: god type: demigod father father Edge battled brother Property time:12 name: Pluto name: Cerberus type: god type: monster Edge Type pet Graph
  11. 11. name: Neptune name: Alcmene type: god type: god brother mother name: Saturn name: Jupiter name: Hercules type: titan type: god type: demigod father father battled brother time:12 name: Pluto name: Cerberus type: god type: monster pet Path
  12. 12. name: Neptune name: Alcmene type: god type: god brother mother name: Saturn name: Jupiter name: Hercules type: titan type: god type: demigod father father battled brother time:12 name: Pluto name: Cerberus type: god type: monster pet Degree
  13. 13. Apache 2 Aurelius Graph Cluster TITAN FAUNUS FULGORA Map/Reduce Load Bulk Load Analysis results back into Titan Stores a massive-scale Batch processing of large Runs global graph algorithms property graph allowing real- graphs with Hadoop on large, compressed, time traversals and updates in-memory graphs
  14. 14. II Titan Graph Database AURELIUS THINKAURELIUS.COM
  15. 15. Titan Features   Numerous Concurrent Users   Many Short Transactions   read/write   Real-time Traversals (OLTP)   High Availability   Dynamic Scalability   Variable Consistency Model   ACID or eventual consistency   Real-time Big Graph Data
  16. 16. Storage Backends Partitionability Consistency Availability
  17. 17. $ ./titan-0.2.0/bin/gremlin.sh! ! ! !,,,/! (o o)! -----oOOo-(_)-oOOo-----! gremlin> g = TitanFactory.open('/tmp/titan')! ==>titangraph[local:/tmp/titan]! gremlin> v = g.V(‘name’,’Hercules’)! ==>v[4]! gremlin> v.out(‘father’).out(‘brother’).name!
  18. 18. name: Neptune name: Alcmene type: god type: god brother mother name: Saturn name: Jupiter name: Hercules type: titan type: god type: demigod father father battled brother time:12 name: Pluto name: Cerberus type: god type: monster pet gremlin> v.out(‘father’).out(‘brother’).name!
  19. 19. Vertex-Centric Indices   Sort and index edges per vertex by primary key   Primary key can be composite   Enables efficient focused traversals   Only retrieve edges that matter   Uses push down predicates for quick, index-driven retrieval
  20. 20. battled battled battled time: 1 time: 3 time: 5 mother battled v v.query()! time: 9 father fought fought
  21. 21. battled battled battled time: 1 time: 3 time: 5 mother battled v v.query()! time: 9 .direction(OUT)! father
  22. 22. battled battled battled time: 1 time: 3 time: 5 battled v v.query()! time: 9 .direction(OUT)! .labels(‘battled’)!
  23. 23. battled battled time: 1 time: 3 v v.query()! .direction(OUT)! .labels(‘battled’)! .has(‘time,T.lt,5)!
  24. 24. Titan Features I.  Data Management II.  Vertex-Centric Indices
  25. 25. Titan Features III.  Graph Partitioning IV.  Edge Compression
  26. 26. III TITAN 0.3.0 [-SNAPSHOT] AURELIUS THINKAURELIUS.COM
  27. 27. Titan Embedding   Rexster RexPro   lightweight Gremlin Server   binary protocol   Titan Gremlin Engine   Embedded Storage Backend   in-JVM method calls   Native clients   Java, Python, Clojure
  28. 28. Graph Indexing   Vertex and Edge indexing   Pluggable index provider   ElasticSearch   Lucene   Full-text search   Numeric range search   Geographic search
  29. 29. name: Neptune name: Alcmene age: 5200 age: 3300 title: God of the earth and ocean brother mother name: Jupiter name: Saturn age: 4800 name: Hercules age: 5900 title: God of the title: Divine hero heaven and skies father father battled brother time:12 location: (38.071,23.745) name: Pluto name: Cerberus age: 4900 title: Ugly beast of the title: God of the underworld underworld pet
  30. 30. name: Neptune name: Alcmene age: 5200 age: 3300 title: God of the earth and ocean brother mother name: Jupiter name: Saturn age: 4800 name: Hercules age: 5900 title: God of the title: Divine hero heaven and skies father father battled brother time:12 location: (38.071,23.745) name: Pluto name: Cerberus age: 4900 title: Ugly beast of the title: God of the underworld underworld pet g.query().has(‘age’,Cmp.GREATER_THAN,5000).vertices()!
  31. 31. name: Neptune name: Alcmene age: 5200 age: 3300 title: God of the earth and ocean brother mother name: Jupiter name: Saturn age: 4800 name: Hercules age: 5900 title: God of the title: Divine hero heaven and skies father father battled brother time:12 location: (38.071,23.745) name: Pluto name: Cerberus age: 4900 title: Ugly beast of the title: God of the underworld underworld pet g.query().has(‘title’,Txt.CONTAINS,’god’).vertices()!
  32. 32. name: Neptune name: Alcmene age: 5200 age: 3300 title: God of the earth and ocean brother mother name: Jupiter name: Saturn age: 4800 name: Hercules age: 5900 title: God of the title: Divine hero heaven and skies father father battled brother time:12 location: (38.071,23.745) name: Pluto name: Cerberus age: 4900 title: Ugly beast of the title: God of the underworld underworld pet g.query().has(‘age’,Cmp.GREATER_THAN,5000)
 has(‘title’,Txt.CONTAINS,’god’).vertices()!
  33. 33. name: Neptune name: Alcmene age: 5200 age: 3300 title: God of the earth and ocean brother mother name: Jupiter name: Saturn age: 4800 name: Hercules age: 5900 title: God of the title: Divine hero heaven and skies father father battled brother time:12 location: (38.071,23.745) name: Pluto name: Cerberus age: 4900 title: Ugly beast of the title: God of the underworld underworld pet g.query().has(‘location’,Geo.WITHIN,
 Geoshape.circle(38,23,100).edges()!
  34. 34. IV Faunus Graph Analytics AURELIUS THINKAURELIUS.COM
  35. 35. Faunus Features   Hadoop-based Graph Computing Framework   Graph Analytics   Breadth-first Traversals   Global Graph Computations   Batch Big Graph Data
  36. 36. Faunus Architecture g._()!
  37. 37. Faunus Work Flow g.V.out .out .count() hdfs://user/ubuntu/ output/job-0/ output/job-1/ graph* output/job-2/ { sideeffect* Compressed HDFS Graphs   stored in sequence files   variable length encoding   prefix compression
  38. 38. Apache 2 Aurelius Graph Cluster TITAN FAUNUS FULGORA Map/Reduce Load Bulk Load Analysis results back into Titan Stores a massive-scale Batch processing of large Runs global graph algorithms property graph allowing real- graphs with Hadoop on large, compressed, time traversals and updates in-memory graphs
  39. 39. What’s New   Faunus 0.1 released   Bulk Import / Export for Titan   loaded graph into Titan   loading derivations into Titan   RDF support   Many optimizations   vertex compression
  40. 40. Faunus Setup $ bin/gremlin.sh ! ,,,/! (o o)! -----oOOo-(_)-oOOo-----! gremlin> g = FaunusFactory.open('bin/titan-hbase.properties')! ==>faunusgraph[titanhbaseinputformat]! gremlin> g.getProperties()! ==>faunus.graph.input.format=com.thinkaurelius.faunus.formats.titan.hbase.TitanHBaseInputFormat ==>faunus.graph.output.format=org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat! ==>faunus.sideeffect.output.format=org.apache.hadoop.mapreduce.lib.output.TextOutputFormat! ==>faunus.output.location=dbpedia! ==>faunus.output.location.overwrite=true! gremlin> g._() ! 12/11/09 15:17:45 INFO mapreduce.FaunusCompiler: Compiled to 1 MapReduce job(s)! 12/11/09 15:17:45 INFO mapreduce.FaunusCompiler: Executing job 1 out of 1: MapSequence[com.thinkaurelius.faunus.mapreduce.transform.IdentityMap.Map]! 12/11/09 15:17:50 INFO mapred.JobClient: Running job: job_201211081058_0003!
  41. 41. Build a Knowledge Graph   Based on DBPedia   Graph version of Wikipedia   ~290 million edges (~1B triples) 1.  Bulk load RDF into Faunus   6 m1.xlarge 2.  Convert to property graph 3.  Bulk load into Titan   3 m1.xlarge with Cassandra 4.  OLTP+OLAP   Total Time: ~ 2 hours
  42. 42. Graph OLTP gremlin> g = TitanFactory.open('bin/cassandra.local') ! ==>titangraph[cassandrathrift:10.176.213.110]! gremlin> g.V('name','Random_walker_algorithm').both.name! ==>Random_walk! ==>Segmentation_(image_processing)! ==>Graph_(mathematics)! ==>Laplacian_matrix! ==>Graph! ==>Laplacian_matrix! ==>Electrical_network! ==>Resistor! ==>Electrical_resistance_and_conductance! ==>Ground_(electricity)! ==>Direct_current! ==>Voltage_source! ==>Precomputation! ==>Category:Computer_vision! ==>Random_Walker_(Computer_Vision)! ==>List_of_algorithms! ==>Segmentation_(image_processing)! ==>Watershed_(image_processing)! ==>Random_walker_(computer_vision)! ==>Random_Walker_(computer_vision)!
  43. 43. gremlin> g.V('name','Learning').out.out.out.out[0..10].name ! ==>Latium! ==>Roman_Kingdom! ==>Roman_Republic! ==>Roman_Empire! ==>Middle_Ages! ==>Early_modern_Europe! ==>Armenian_Kingdom_of_Cilicia! ==>Lingua_franca! ==>Vatican_City! ==>Vulgar_Latin! ==>Romance_languages!
  44. 44. Apache 2 Aurelius Graph Cluster TITAN FAUNUS FULGORA Map/Reduce Load Bulk Load Analysis results aureliusgraphs@googlegroups.com back into Titan Stores a massive-scale Batch processing of large Runs global graph algorithms property graph allowing real- graphs with Hadoop on large, compressed, time traversals and updates in-memory graphs
  45. 45. Speed of Traversal/Process The Graph Landscape Illustration only, not to scale Size of Graph
  46. 46. TINKERPOP.COM
  47. 47. Thanks! Vadas Gintautas Marko Rodriguez @vadasg @twarko Stephen Mallette Daniel LaRocque @spmallette AURELIUS THINKAURELIUS.COM
  48. 48. We are Hiring AURELIUS THINKAURELIUS.COM
  • ssusera1c366

    Mar. 12, 2018
  • AbhishekAgrawal50

    Jun. 9, 2015
  • WaelAbdelMagied

    May. 23, 2015
  • joshua.chi

    Jul. 15, 2014
  • ArnaudDemoissac

    Mar. 5, 2014
  • rigil0

    Jan. 21, 2014
  • peterverster

    Sep. 28, 2013
  • dgomez407

    Sep. 10, 2013
  • ssuser3c3394

    Aug. 3, 2013
  • MatthieuDejardins

    Apr. 10, 2013
  • mathieu-bastian

    Apr. 8, 2013
  • pdlug

    Mar. 21, 2013
  • grindaizer

    Mar. 21, 2013
  • joelvh

    Mar. 21, 2013
  • mahendram

    Mar. 20, 2013
  • Maziyarsh

    Mar. 17, 2013
  • darkseed

    Mar. 15, 2013
  • warren6332

    Mar. 13, 2013
  • dajobe

    Mar. 8, 2013
  • seralf

    Mar. 8, 2013

In this presentation we discuss how graph analysis can add value to your data and how to use open source tools like Titan and Faunus to build scalable graph processing systems. This presentation gives an update on the development status of Titan and Faunus with a preview of what is to come.

Views

Total views

17,801

On Slideshare

0

From embeds

0

Number of embeds

4,509

Actions

Downloads

318

Shares

0

Comments

0

Likes

25

×