Knowing how to design a table or a set of tables that fit your application's requirements is one of the most difficult hurdles developers face when starting to work with Apache Accumulo. In this talk we'll review common strategies in table designs and their appropriateness for various types of applications. Attendees will learn how and why to choose the right strategy to avoid problems such as bad performance or inconsistency. – Speaker – Aaron Cordova Co-founder and CTO, Koverse Aaron has built multiple, large-scale, big data systems that are used by the intelligence, defense, finance and healthcare industries. Aaron co-founded Koverse Inc. Prior to that, Aaron spent five years as a researcher for the National Security Agency (NSA) where he developed and deployed into operations dozens of advanced analytical techniques. He is the founder of Apache Accumulo, a scalable and secure data store on top of Apache Hadoop and the author of the recently released O’Reilly book, Accumulo: Application Development, Table Design, and Best Practices. — More Information — For more information see http://www.accumulosummit.com/

1. Choosing a Table Design
Strategy
Aaron Cordova, Koverse
Accumulo Summit 2017

3. Proposed Metric for Comparing
Designs
Query Speed
Concurrency
Query PowerAtomicity
Ingest Speed
Design

4. Dimensions
Query speed – how
fast to queries that
the table was
designed for run
01
Concurrency – how
many users can
query at once
02
Query Power – how
many different
queries are
supported without
resorting to table
scans
03
Atomicity – how
easy is it to change a
value without
creating
inconsistency
04
Ingest Speed – how
fast can new data be
ingested
05

5. Data
Relationships
Aren’t
Perfectly
Modeled by
Hierarchies

6. Accumulo Data Model (Abbr.)
Row A
Family A
Qualifier A Value A
Qualifier B Value B
Family B
Qualifier C Value C
Qualifier D Value D

7. Single Table Strategy
User A
Orders
OID1_Item_A $5
OID1_Item_B $7
OID2_Item_A $5
Details
Address 123 main st.
Phone 555-1212

8. Query
Patterns,
Single
Table
Easy and fast to get all of a user’s
information
Could relatively easily scan for just orders
or just user details, across multiple users,
using locality groups
Looking for who bought a particular item
would require a table scan
Looking for which items are over a certain
price requires a table scan
Notice how dimensions start to get
‘doubled up’ in a particular component –
Order ID (OID) and Item name in this case

9. Single Table Profile
0
1
2
3
4
5
6
7
8
9
10
Query Speed
Concurrency
Query PowerAtomicity
Ingest Speed

10. Table With Some Inversion
User A
Orders
OID1_Item_A $5
OID1_Item_B $7
OID2_Item_A $5
Details
Address 123 main st.
Phone 555-1212
Phone_555-
1212
Users User A
Item A Users User A 2

11. Query
Patterns,
Table with
Inversion
Easy and fast to get all of a user’s
information
Could relatively easily scan for just
orders or just user details, across
multiple users, using locality groups
Looking for who bought a particular
item is now fast
Looking for which items are over a
certain price requires a table scan
Updating a user’s phone number must
be done in two places – still
manageable

12. Table With Inversion Profile
0
1
2
3
4
5
6
7
8
9
10
Query Speed
Concurrency
Query PowerAtomicity
Ingest Speed

13. Secondary Index Strategy
Record A Fields
User ID User A
Order ID 1
Items
Item A, $5
Item B, $7
Record B Fields
User ID User A
Order ID 2
Items Item_A, $5
Record C Fields
User ID User A
Address
123 main
st.
Phone 555-1212
$5 Items
Record A
Record B
$7 Items Record A
1 Order ID Record A
123 main
st.
Address Record C
555-1212 Phone Record C
Item A Items
Record A
Record B
Item B Items Record B
User A User ID
Record A
Record B
Record C
Record Table Index Table

14. Query
Patterns,
Secondary
Index
Can now quickly look up records
containing any value in any field
Every query has two steps: index
look followed by record lookups
Indexing everything will decrease
ingest speed
Some updates may be costly as
values may appear in many records
Each query still involves few
machines, preserving concurrency

15. Secondary Index Strategy
0
1
2
3
4
5
6
7
8
9
10
Query Speed
Concurrency
Query PowerAtomicity
Ingest Speed

16. Embedded Index Strategy
Record A Fields
User ID User A
Order ID 1
Items
Item A, $5
Item B, $7
Record B Fields
User ID User A
Order ID 2
Items Item_A, $5
Record C Fields
User ID User A
Address
123 main
st.
Phone 555-1212
User A User ID Record A
1 Order ID Record A
$5 Items Record A
$7 Items Record A
Record Table Index Table
User A User ID
Record B
Record C
123 Main
st.
Address Record C
2 Order ID Record B
555-1212 Phone Record C
$5 Items Record C
$7 Items Record C

17. Query
Patterns,
Embedded
Index
Can now quickly look up records
containing any value in any field
Every query has two steps: index look
followed by record lookups, within each
Tablet Server
Indexing everything will decrease ingest
speed
Index entries located with documents,
updates can be done atomically in more
cases
Each query involves many machines,
sacrificing concurrency

18. Embedded Index Strategy
0
1
2
3
4
5
6
7
8
9
10
Query Speed
Concurrency
Query PowerAtomicity
Ingest Speed

19. Embedded Index w/ Rounds Strategy
Record A Fields
User ID User A
Order ID 1
Items
Item A, $5
Item B, $7
Record B Fields
User ID User A
Order ID 2
Items Item_A, $5
Record C Fields
User ID User A
Address
123 main
st.
Phone 555-1212
User A User ID Record A
1 Order ID Record A
$5 Items Record A
$7 Items Record A
User A User ID
Record B
Record C
123 Main
st.
Address Record C
2 Order ID Record B
555-1212 Phone Record C
$5 Items Record C
$7 Items Record CRecord A Fields
User ID User A
Order ID 1
Items
Item A, $5
Item B, $7
Record B Fields
User ID User A
Order ID 2
User A User ID Record A
1 Order ID Record A
$5 Items Record A
$7 Items Record A
User A User ID
Record B
Record C
123 Main
st.
Address Record C
2 Order ID Record B
555-1212 Phone Record C
1
2

20. Query
Patterns,
Embedded
Index
Can now quickly look up records
containing any value in any field
Every query has two steps: index look
followed by record lookups, within each
Tablet Server
Indexing everything will decrease ingest
speed
Index entries located with documents,
updates can be done atomically in more
cases
Each query involves many machines,
sacrificing concurrency

21. Embedded Index w/ Rounds Strategy
0
1
2
3
4
5
6
7
8
9
10
Query Speed
Concurrency
Query PowerAtomicity
Ingest Speed

22. Comparison
0
5
10
0
5
10
0
5
10
0
5
10
0
5
10
Single Some Invert Secondary Index Embed Index Embed Index
+ Round

23. Recommended Approach
Query Power
how many types
of queries do I
need to support
Atomicity
how frequent
are updates
Concurrency
how many
simultaneous
users
Ingest Speed
how many
machines to
keep up with
ingest

24. Questions?

25. Thanks!
Aaron Cordova, Koverse Inc.
@aaroncordova