HBaseCon 2012 | Storing and Manipulating Graphs in HBase
Report
Share
•29 likes•5,439 views
![Don’t get fancy with byte[]
class EdgeValueWritable implements Writable {
EdgeValue edgeValue
byte[] toBytes() {
// use strings if you can help it
}
static EdgeValueWritable fromBytes(byte[] bytes) {
// use strings if you can help it
}
}
groovy](https://image.slidesharecdn.com/3storingandmanipulatinggraphsinhbaseupdatedlastminute-120530160915-phpapp02/75/hbasecon-2012-storing-and-manipulating-graphs-in-hbase-41-2048.jpg?cb=1667959926)
![Querying by vertex
def get = new Get(vertexKeyBytes)
get.addFamily(edgesFamilyBytes)
Result result = table.get(get);
result.noVersionMap.each {family, data ->
// construct edge objects as needed
// data is a Map<byte[],byte[]>
}](https://image.slidesharecdn.com/3storingandmanipulatinggraphsinhbaseupdatedlastminute-120530160915-phpapp02/75/hbasecon-2012-storing-and-manipulating-graphs-in-hbase-42-2048.jpg?cb=1667959926)
![Do implement your own comparator
public static class Comparator
extends WritableComparator {
public int compare(
byte[] b1, int s1, int l1,
byte[] b2, int s2, int l2) {
// .....
}
}
java](https://image.slidesharecdn.com/3storingandmanipulatinggraphsinhbaseupdatedlastminute-120530160915-phpapp02/75/hbasecon-2012-storing-and-manipulating-graphs-in-hbase-59-2048.jpg?cb=1667959926)
HBaseCon 2012 | Storing and Manipulating Graphs in HBase
•29 likes•5,439 views
Report
Share
Download to read offline
Google’s original use case for BigTable was the storage and processing of web graph information, represented as sparse matrices. However, many organizations tend to treat HBase as merely a “web scale” RDBMS. This session will cover several use cases for storing graph data in HBase, including social networks and web link graphs, MapReduce processes like cached traversal, PageRank, and clustering and lastly will look at some lower-level modeling details like row key and column qualifier design, using FullContact’s graph processing systems as a real-world use case.
More Related Content
What's hot
What's hot(20)
Viewers also liked
Viewers also liked(20)
Similar to HBaseCon 2012 | Storing and Manipulating Graphs in HBase
Similar to HBaseCon 2012 | Storing and Manipulating Graphs in HBase(20)
More from Cloudera, Inc.
More from Cloudera, Inc.(20)
Recently uploaded
Recently uploaded(20)
HBaseCon 2012 | Storing and Manipulating Graphs in HBase
- 1. Storing and Manipulating Graphs in HBase Dan Lynn dan@fullcontact.com @danklynn
- 2. Keeps Contact Information Current and Complete Based in Denver, Colorado CTO & Co-Founder
- 3. Turn Partial Contacts Into Full Contacts
- 6. Refresher: Graph Theory rt ex Ve
- 7. Refresher: Graph Theory Edg e
- 9. Tweets @danklynn retweeted “#HBase rocks” follows author @xorlev
- 10. Web Links http://fullcontact.com/blog/ <a href=”...”>TechStars</a> http://techstars.com/
- 11. Why should you care? Vertex Influence - PageRank - Social Influence - Network bottlenecks Identifying Communities
- 12. Storage Options
- 13. neo4j
- 14. neo4j Very expressive querying (e.g. Gremlin)
- 16. neo4j Data must fit on a single machine :-(
- 17. FlockDB
- 20. FlockDB No multi-hop query support :-(
- 21. RDBMS (e.g. MySQL, Postgres, et al.)
- 23. RDBMS Huge amounts of JOINing :-(
- 29. Adjacency Matrix 1 3 2
- 30. Adjacency Matrix 1 2 3 1 0 1 1 2 1 0 1 3 1 1 0
- 31. Adjacency Matrix Can use vectorized libraries
- 32. Adjacency Matrix Requires O(n2) memory n = number of vertices
- 33. Adjacency Matrix Hard(er) to distribute
- 34. Adjacency List 1 3 2
- 35. Adjacency List 1 2,3 2 1,3 3 1,2
- 36. Adjacency List Design in HBase e:dan@fullcontact.com p:+13039316251 t:danklynn
- 37. Adjacency List Design in HBase row key “edges” column family e:dan@fullcontact.com p:+13039316251= ... t:danklynn= ... p:+13039316251 e:dan@fullcontact.com= ... t:danklynn= ... t:danklynn e:dan@fullcontact.com= ... p:+13039316251= ...
- 38. Adjacency List Design in HBase row key “edges” column family e:dan@fullcontact.com p:+13039316251= ... t:danklynn= ... at to W e?h p:+13039316251 e:dan@fullcontact.com= ... st or t:danklynn= ... t:danklynn e:dan@fullcontact.com= ... p:+13039316251= ...
- 39. Custom Writables package org.apache.hadoop.io; public interface Writable { void write(java.io.DataOutput dataOutput); void readFields(java.io.DataInput dataInput); } java
- 40. Custom Writables class EdgeValueWritable implements Writable { EdgeValue edgeValue void write(DataOutput dataOutput) { dataOutput.writeDouble edgeValue.weight } void readFields(DataInput dataInput) { Double weight = dataInput.readDouble() edgeValue = new EdgeValue(weight) } // ... } groovy
- 41. Don’t get fancy with byte[] class EdgeValueWritable implements Writable { EdgeValue edgeValue byte[] toBytes() { // use strings if you can help it } static EdgeValueWritable fromBytes(byte[] bytes) { // use strings if you can help it } } groovy
- 42. Querying by vertex def get = new Get(vertexKeyBytes) get.addFamily(edgesFamilyBytes) Result result = table.get(get); result.noVersionMap.each {family, data -> // construct edge objects as needed // data is a Map<byte[],byte[]> }
- 43. Adding edges to a vertex def put = new Put(vertexKeyBytes) put.add( edgesFamilyBytes, destinationVertexBytes, edgeValue.toBytes() // your own implementation here ) // if writing directly table.put(put) // if using TableReducer context.write(NullWritable.get(), put)
- 44. Distributed Traversal / Indexing e:dan@fullcontact.com p:+13039316251 t:danklynn
- 45. Distributed Traversal / Indexing e:dan@fullcontact.com p:+13039316251 t:danklynn
- 46. Distributed Traversal / Indexing e:dan@fullcontact.com p:+13039316251 Pi v ot v e rt ex t:danklynn
- 47. Distributed Traversal / Indexing e:dan@fullcontact.com p:+13039316251 Ma pReduce ove r out bou nd edges t:danklynn
- 48. Distributed Traversal / Indexing e:dan@fullcontact.com p:+13039316251 Em it vertexes an d edge dat a gro upe d by the piv ot t:danklynn
- 49. Distributed Traversal / Indexing Re duc e key p:+13039316251 “Ou t” vertex e:dan@fullcontact.com t:danklynn “In” vertex
- 50. Distributed Traversal / Indexing e:dan@fullcontact.com t:danklynn Re duc er em its higher-order edge
- 51. Distributed Traversal / Indexing Ite rat ion 0
- 52. Distributed Traversal / Indexing Ite rat ion 1
- 53. Distributed Traversal / Indexing Ite rat ion 2
- 54. Distributed Traversal / Indexing Reuse edges created during previ ous iterat ions Ite rat ion 2
- 55. Distributed Traversal / Indexing Ite rat ion 3
- 56. Distributed Traversal / Indexing Reuse edges created during previ ous iterat ions Ite rat ion 3
- 57. Distributed Traversal / Indexing hop s req uires on ly ite rat ion s
- 58. Tips / Gotchas
- 59. Do implement your own comparator public static class Comparator extends WritableComparator { public int compare( byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) { // ..... } } java
- 60. Do implement your own comparator static { WritableComparator.define(VertexKeyWritable, new VertexKeyWritable.Comparator()) } java
- 61. MultiScanTableInputFormat MultiScanTableInputFormat.setTable(conf, "graph"); MultiScanTableInputFormat.addScan(conf, new Scan()); job.setInputFormatClass( MultiScanTableInputFormat.class); java
- 62. TableMapReduceUtil TableMapReduceUtil.initTableReducerJob( "graph", MyReducer.class, job); java
- 65. Elastic MapReduce HFi les Copy to S3 Seq uen ceFiles
- 66. Elastic MapReduce HFi les Copy to S3 Elastic MapReduce Seq uen ceFiles Seq uen ceFiles
- 67. Elastic MapReduce HFi les Copy to S3 Elastic MapReduce Seq uen ceFiles Seq uen ceFiles
- 68. Elastic MapReduce HFi les Copy to S3 Elastic MapReduce Seq uen ceFiles Seq uen ceFiles HFileOutputFormat.configureIncrementalLoad(job, outputTable) HFi les
- 69. Elastic MapReduce HFi les Copy to S3 Elastic MapReduce Seq uen ceFiles Seq uen ceFiles HFileOutputFormat.configureIncrementalLoad(job, outputTable) HFi les HBase $ hadoop jar hbase-VERSION.jar completebulkload
- 70. Additional Resources Google Pregel: BSP-based graph processing system Apache Giraph: Implementation of Pregel for Hadoop MultiScanTableInputFormat: (code to appear on GitHub) Apache Mahout - Distributed machine learning on Hadoop