Download to read offline
Uploaded byKoverse, Inc.
Accumulo design
Apache Accumulo is a scalable key-value store based on Google's Bigtable, designed to manage large volumes of data across multiple commodity servers with features that support fast lookups, highly consistent writes, and dynamic column management. It is suitable for various use cases, particularly in organizations needing to analyze disparate data sources with varying sensitivity levels, while also supporting real-time user requests. Key features include iterators, batch scanners, and secondary indices to enhance data discovery and analysis.
Accumulo design
- 1. APACHE ACCUMULO From adesign perspective
- 2. SCALABLE KEY-VALUE STORE BASEDON GOOGLE'S BIGTABLE
- 3. BIGTABLE FEATURES • Distributesdata across many commodity servers • Sorts data by key for fast lookup of values by key • Scan across multiple key value pairs • Highly consistent writes to single row • Support for MapReduce jobs
- 4.
- 5. Row ID ColFam Col Qual Timestamp Value Bob Email id0023 20120301 Hey joe, can you send ... Bob Email id0024 20120302 Re: next Thursday ... Bob UserPrefs Background 20130101 Grey Fred Email id0001 20080302 Welcome to gmail ... Sarah Email id0004 20130201 Hi again ... Sara Videos ytid009 20100303 nsu736:)jdudjd k$:)378;'$$)
- 6. Tablet servers HDFSDataNodes Commit Layer Replication Layer
- 7. SINCE 2006 • SeveralBigTable implementations • Apache Hbase • Apache Cassandra • Apache Accumulo • others …
- 8. BIGTABLE IS BIGTABLERIGHT?
- 9.
- 10. HBASE • Open sourceApache project started by developers at Powerset, bought by Microsoft • Now used at Facebook, StumbleUpon, other big web sites • Fast reads • Row-oriented API • Each column family has it's own set of files
- 11.
- 12. CASSANDRA • Apache projectstarted at Facebook • Combines elements of BigTable and Amazon's Dynamo into one system • Used at Netflix, other web sites • Fast writes • Tunable consistency
- 13. Tablet servers Commit andReplication Layer
- 14. CONSISTENCY • Highly consistentmeans: writes in one place • Eventually consistent: writes in > one place • Writes in > one place: network partition tolerance • Partition tolerance: geographically distributed servers • *Google uses Spanner to synchronize multiple dbs
- 15. Tablet servers Data CenterA Data Center B
- 16. Data Center AData Center B Tablet servers
- 17. OVERVIEW • Both highlyscalable • Used to build web applications that can serve millions of users at once • Serves as a low-latency persistence layer for real time service of requests • Available in single data center or cross data center options
- 18. USE CASE • Mostdata comes from users • Schema defined by the application • Data builds up over time
- 19.
- 20.
- 21. ACCUMULO • Can supportthe web application use-case • But what are those other extra features for?
- 22. ACCUMULO ‘EXTRAS’ • DynamicColumn Families • ColumnVisibility • Key-value oriented API • Iterators • Batch Scanners
- 23. BIG ORGANIZATIONS • Missionsother than internet services • Various disparate operational systems that generate data • Desire to look across and analyze that data • Desire to deliver results to their own population
- 24. USE CASE ISDISCOVERING AND ANALYZING ALL DATA
- 25. ISSUES • Scale • Unknown/ multiple schema • Support for analysis without data movement • Varying levels of sensitivity in the same system • Support a high number of low-latency user requests
- 26.
- 27.
- 28.
- 29. NO CONTROL OVEROR MANY DIFFERENT SCHEMA?
- 30. MAP EXISTING FIELDSTO COLUMNSDYNAMICALLY
- 31.
- 32. VARYING LEVELS OFDATA SENSITIVITY?
- 33.
- 34. DATA MODEL Key Value Row ID Column Time stamp FamilyQualifier Visibility
- 35. Row ID ColFam Col Qual Col Vis Timestamp Value Bob Email id0023 personal comms 20120301 Hey joe, can you send ... Bob Email id0024 personal comms 20120302 Re: next Thursday ... Bob UserPrefs Background prefs 20130101 Grey Fred Email id0001 personal comms 20080302 Welcome to gmail ... Sarah Email id0004 personal comms 20130201 Hi again ... Sara Videos ytid009 public post 20100303 nsu736:)jdu djdk $:)378;'$$)
- 36. DATA OFVARYING SENSITIVITY LEVELSCAN BE PHYSICALLY CO-LOCATED
- 37. FRAMEWORKS LIKE HADOOP MAPREDUCE LOVE IT WHEN DATA IS ALLTOGETHER
- 38.
- 39.
- 40. SECONDARY INDICES • Application-createddata: known • Pre-existing data? unknown
- 41.
- 42. SECONDARY INDICES RowID ColQual Value RID00001 age 54 RID00001 name bob RID00002 name fred RID00003 age 43 RID00003 height 5’9” RID00003 name harry RID00004 name carl RID00005 name evan RowID Col Fam Col Qual 43 age RID00003 54 age RID00001 5’9” height RID00003 bob name RID00001 carl name RID00004 evan name RID00005 fred name RID00002 harry name RID00003
- 43.
- 44.
- 45. RowID Col QualValue RID00001 age 54 RID00001 name bob RID00002 name fred RID00003 age 43 RID00003 height 5’9” RID00003 name harry RID00004 name carl RID00005 name evan Batch Scanner
- 46.
- 47.
- 48.
- 49. SHUFFLE-SORTED? • Between Mapand Reduce phases is shuffle-sort • Sorting by key is necessary so all the values for a given key end up next to each other … • BigTable also sorts keys …
- 50.
- 51.
- 52.
- 53.
- 54.