Download as PDF, PPTX
![Don’t get fancy with byte[]
class EdgeValueWritable implements Writable {
EdgeValue edgeValue
byte[] toBytes() {
// use strings if you can help it
}
static EdgeValueWritable fromBytes(byte[] bytes) {
// use strings if you can help it
}
}
groovy](https://image.slidesharecdn.com/storingandmanipulatinggraphsinhbase-updated-120524173208-phpapp02/85/Storing-and-manipulating-graphs-in-HBase-41-320.jpg)
![Querying by vertex
def get = new Get(vertexKeyBytes)
get.addFamily(edgesFamilyBytes)
Result result = table.get(get);
result.noVersionMap.each {family, data ->
// construct edge objects as needed
// data is a Map<byte[],byte[]>
}](https://image.slidesharecdn.com/storingandmanipulatinggraphsinhbase-updated-120524173208-phpapp02/85/Storing-and-manipulating-graphs-in-HBase-42-320.jpg)
![Do implement your own comparator
public static class Comparator
extends WritableComparator {
public int compare(
byte[] b1, int s1, int l1,
byte[] b2, int s2, int l2) {
// .....
}
}
java](https://image.slidesharecdn.com/storingandmanipulatinggraphsinhbase-updated-120524173208-phpapp02/85/Storing-and-manipulating-graphs-in-HBase-59-320.jpg)
Uploaded byDan Lynn
Storing and manipulating graphs in HBase
This document discusses storing and manipulating graphs in HBase. It provides an overview of graph theory concepts and different modeling techniques for storing graphs in HBase, including the adjacency matrix and adjacency list approaches. It also covers techniques for distributed traversal and querying of graphs stored in HBase using MapReduce jobs. Key tips discussed include implementing custom comparators and using utilities like MultiScanTableInputFormat and TableMapReduceUtil.
More Related Content
Similar to Storing and manipulating graphs in HBase
![Getting Started with Apache Spark: Big Data Made Simple [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/apachesparkgettingstarted-260203175547-8361bcc3-thumbnail.jpg?width=640&height=640&fit=bounds)
Storing and manipulating graphs in HBase
- 1. Storing and ManipulatingGraphs in HBase Dan Lynn dan@fullcontact.com @danklynn
- 2. Keeps Contact InformationCurrent and Complete Based in Denver, Colorado CTO & Co-Founder
- 3. Turn Partial Contacts Into Full Contacts
- 4.
- 5.
- 6. Refresher: Graph Theory rt ex Ve
- 7. Refresher: Graph Theory Edg e
- 8.
- 9. Tweets @danklynn retweeted “#HBase rocks” follows author @xorlev
- 10. Web Links http://fullcontact.com/blog/ <a href=”...”>TechStars</a> http://techstars.com/
- 11. Why should youcare? Vertex Influence - PageRank - Social Influence - Network bottlenecks Identifying Communities
- 12.
- 13.
- 14. neo4j Very expressive querying (e.g. Gremlin)
- 15.
- 16. neo4j Data must fiton a single machine :-(
- 17.
- 18.
- 19.
- 20. FlockDB No multi-hop querysupport :-(
- 21. RDBMS (e.g. MySQL, Postgres,et al.)
- 22.
- 23. RDBMS Huge amounts ofJOINing :-(
- 25.
- 26.
- 27.
- 28.
- 29.
- 30. Adjacency Matrix 1 2 3 1 0 1 1 2 1 0 1 3 1 1 0
- 31. Adjacency Matrix Can usevectorized libraries
- 32. Adjacency Matrix Requires O(n2) memory n = number of vertices
- 33. Adjacency Matrix Hard(er) todistribute
- 34. Adjacency List 1 3 2
- 35. Adjacency List 1 2,3 2 1,3 3 1,2
- 36. Adjacency List Designin HBase e:dan@fullcontact.com p:+13039316251 t:danklynn
- 37. Adjacency List Designin HBase row key “edges” column family e:dan@fullcontact.com p:+13039316251= ... t:danklynn= ... p:+13039316251 e:dan@fullcontact.com= ... t:danklynn= ... t:danklynn e:dan@fullcontact.com= ... p:+13039316251= ...
- 38. Adjacency List Designin HBase row key “edges” column family e:dan@fullcontact.com p:+13039316251= ... t:danklynn= ... at to W e?h p:+13039316251 e:dan@fullcontact.com= ... st or t:danklynn= ... t:danklynn e:dan@fullcontact.com= ... p:+13039316251= ...
- 39. Custom Writables package org.apache.hadoop.io; publicinterface Writable { void write(java.io.DataOutput dataOutput); void readFields(java.io.DataInput dataInput); } java
- 40. Custom Writables class EdgeValueWritableimplements Writable { EdgeValue edgeValue void write(DataOutput dataOutput) { dataOutput.writeDouble edgeValue.weight } void readFields(DataInput dataInput) { Double weight = dataInput.readDouble() edgeValue = new EdgeValue(weight) } // ... } groovy
- 41. Don’t get fancywith byte[] class EdgeValueWritable implements Writable { EdgeValue edgeValue byte[] toBytes() { // use strings if you can help it } static EdgeValueWritable fromBytes(byte[] bytes) { // use strings if you can help it } } groovy
- 42. Querying by vertex defget = new Get(vertexKeyBytes) get.addFamily(edgesFamilyBytes) Result result = table.get(get); result.noVersionMap.each {family, data -> // construct edge objects as needed // data is a Map<byte[],byte[]> }
- 43. Adding edges toa vertex def put = new Put(vertexKeyBytes) put.add( edgesFamilyBytes, destinationVertexBytes, edgeValue.toBytes() // your own implementation here ) // if writing directly table.put(put) // if using TableReducer context.write(NullWritable.get(), put)
- 44. Distributed Traversal /Indexing e:dan@fullcontact.com p:+13039316251 t:danklynn
- 45. Distributed Traversal /Indexing e:dan@fullcontact.com p:+13039316251 t:danklynn
- 46. Distributed Traversal /Indexing e:dan@fullcontact.com p:+13039316251 Pi v ot v e rt ex t:danklynn
- 47. Distributed Traversal /Indexing e:dan@fullcontact.com p:+13039316251 Ma pReduce ove r out bou nd edges t:danklynn
- 48. Distributed Traversal /Indexing e:dan@fullcontact.com p:+13039316251 Em it vertexes an d edge dat a gro upe d by the piv ot t:danklynn
- 49. Distributed Traversal /Indexing Re duc e key p:+13039316251 “Ou t” vertex e:dan@fullcontact.com t:danklynn “In” vertex
- 50. Distributed Traversal /Indexing e:dan@fullcontact.com t:danklynn Re duc er em its higher-order edge
- 51. Distributed Traversal /Indexing Ite rat ion 0
- 52. Distributed Traversal /Indexing Ite rat ion 1
- 53. Distributed Traversal /Indexing Ite rat ion 2
- 54. Distributed Traversal /Indexing Reuse edges created during previ ous iterat ions Ite rat ion 2
- 55. Distributed Traversal /Indexing Ite rat ion 3
- 56. Distributed Traversal /Indexing Reuse edges created during previ ous iterat ions Ite rat ion 3
- 57. Distributed Traversal /Indexing hop s req uires on ly ite rat ion s
- 58.
- 59. Do implement yourown comparator public static class Comparator extends WritableComparator { public int compare( byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) { // ..... } } java
- 60. Do implement yourown comparator static { WritableComparator.define(VertexKeyWritable, new VertexKeyWritable.Comparator()) } java
- 61. MultiScanTableInputFormat MultiScanTableInputFormat.setTable(conf, "graph"); MultiScanTableInputFormat.addScan(conf, new Scan()); job.setInputFormatClass( MultiScanTableInputFormat.class); java
- 62. TableMapReduceUtil TableMapReduceUtil.initTableReducerJob( "graph", MyReducer.class, job); java
- 63.
- 64.
- 65. Elastic MapReduce HFi les Copy to S3 Seq uen ceFiles
- 66. Elastic MapReduce HFi les Copy to S3 Elastic MapReduce Seq uen ceFiles Seq uen ceFiles
- 67. Elastic MapReduce HFi les Copy to S3 Elastic MapReduce Seq uen ceFiles Seq uen ceFiles
- 68. Elastic MapReduce HFi les Copy to S3 Elastic MapReduce Seq uen ceFiles Seq uen ceFiles HFileOutputFormat.configureIncrementalLoad(job, outputTable) HFi les
- 69. Elastic MapReduce HFi les Copy to S3 Elastic MapReduce Seq uen ceFiles Seq uen ceFiles HFileOutputFormat.configureIncrementalLoad(job, outputTable) HFi les HBase $ hadoop jar hbase-VERSION.jar completebulkload
- 70. Additional Resources Google Pregel:BSP-based graph processing system Apache Giraph: Implementation of Pregel for Hadoop MultiScanTableInputFormat example Apache Mahout - Distributed machine learning on Hadoop
- 71.