Apache Accumulo Overview

1. 11 Apache Accumulo Overview Bill Havanki Solutions Architect, Cloudera Government Solutions

3. 3 A Quick History 3

5. 5 ©2014 Cloudera, Inc. All rights reserved. Google BigTable • Began development in 2004 • Built on Google File System • Non-relational • Byte-oriented and schemaless • Stores data in the petabyte range • Research paper published in 2006 5

7. 7 ©2014 Cloudera, Inc. All rights reserved. From Cloudbase to Accumulo • Started in 2008 as National Security Agency project • Submitted to Apache Incubator in 2011 (and renamed) • Top-level project in 2012 7

8. 8 Storage Model 8

14. 14 ©2014 Cloudera, Inc. All rights reserved. It’s all bytes All key and value data are stored as bytes except timestamp is a long There are no built-in data types but lexicoders help with common types Key components are usually UTF-8 strings 14

15. 15 ©2014 Cloudera, Inc. All rights reserved. Some rows for you 15 row cf cq cv ts value bhavanki job employer 2013-09-01 Cloudera bhavanki personal beer 2013-09-15 Omission bhavanki personal house NOMUGGL 2014-01-25 Ravenclaw brees job employer 2013-10-01 White Cliffs brees personal house NOMUGGL 2014-01-01 Hufflepuff

17. 17 ©2014 Cloudera, Inc. All rights reserved. Locality Groups You can identify sets of one or more column families as locality groups Data in a locality group is stored together for improved read performance 17

19. 19 ©2014 Cloudera, Inc. All rights reserved. Tablets Tablets maps to data files in HDFS 19 employees;Semployees;Semployees;Hemployees;H employees;~employees;~ rfile 2rfile 2rfile 1rfile 1 rfile 3rfile 3

21. 21 Loading and Querying 21

23. 23 ©2014 Cloudera, Inc. All rights reserved. Java Client API Read using scanners Scanner s = conn.createScanner(“employees”, new Authorizations()); s.setRange(“alice”, “eve”); s.setColumnFamily(“personal”); for (Entry<Key, Value> e : s) employeeIds.add(e.getKey().getRow()); 23

24. 24 ©2014 Cloudera, Inc. All rights reserved. Java Client API Read access via iterator pattern • server-side system iterators handle timestamps, authorization checks, and lots more • iterators almost always wrap other iterators, forming a chain • you can define your own, client-side or server-side 24

25. 25 ©2014 Cloudera, Inc. All rights reserved. Java Client API Scanners fetch sorted rows from one range Batch scanners fetch unsorted rows from multiple ranges in parallel Isolated scanners ensure that you do not see a row mid-change 25

28. 28 ©2014 Cloudera, Inc. All rights reserved. Shell Command-line / manual access to Accumulo data • scan, insert, delete • iterator management • table management (creation, deletion, cloning) • user and authorization management • table splitting and merging • ... more 28

29. 29 ©2014 Cloudera, Inc. All rights reserved. Bulk Import Got lots of data to import quickly? • Use MR job to format data using AccumuloFileOutputFormat • Import files using shell Trade off latency / availablity for throughput 29

30. 30 Daemons 30

31. 31 ©2014 Cloudera, Inc. All rights reserved. Tablet Server Serves tablets (table data) • writes data to walog, memtable; deals with compaction • serves data for reads from files, memtable • handles recovery from walogs in case of server failure Most client calls go to tablet servers 31

32. 32 ©2014 Cloudera, Inc. All rights reserved. Master • assigns tablets to tablet servers • detects tablet server failures and reassigns tablets • balances tablet assignments over time • coordinates table operations Multiple supported for failover, only one active 32

33. 33 ©2014 Cloudera, Inc. All rights reserved. Everybody Else in Accumulo Garbage Collector (GC) - identifies and deletes files in HDFS that are no longer needed Tracer - listens for and stores distributed trace messages using a special table 33

35. 35 ©2014 Cloudera, Inc. All rights reserved. Everybody Else outside Accumulo • HDFS (as part of Apache Hadoop) • stores tablet files • stores write-ahead logs (1.5+) • MapReduce (Hadoop) • bulk import • batch processing • Apache ZooKeeper 35

36. 36 Getting Started a.k.a. the Pitch 36

38. 38 ©2014 Cloudera, Inc. All rights reserved. Making Steps 1 and 2 Easier Use a complete, pre-packaged Hadoop distribution ... like CDH! a leading commercial distribution centered on Apache Hadoop •many ecosystem components •configured / updated to work together 38

40. 40 ©2014 Cloudera, Inc. All rights reserved. Making Step 3 Easier Standard Apache Accumulo installation is via tarball • no longer shipping RPM / DEB / ... Using CDH/CM you can use: • a tarball, RPM or DEB with Accumulo packaged for CDH • a parcel (like RPM / ZIP) for easier upgrades • 1.4.4 and 1.4.5 available now • 1.6.0 soon 40

41. 41 ©2014 Cloudera, Inc. All rights reserved. Where to Go for More • http://accumulo.apache.org/ • http://www.cloudera.com/content/cloudera/en/products-and-service • http://www.cloudera.com/content/cloudera/en/products-and-service • http://www.cloudera.com/content/cloudera/en/products-and- services/cdh/accumulo.html 41

Apache Accumulo Overview

Apache Accumulo Overview

Recommended

More Related Content

What's hot

What's hot(20)

Viewers also liked

Viewers also liked(20)

Similar to Apache Accumulo Overview

Similar to Apache Accumulo Overview(20)

Recently uploaded

Recently uploaded(20)

Apache Accumulo Overview