The document discusses secondary indexing in HBase, addressing challenges in efficiently performing lookups deeper than row keys and proposing various implementations like Omid and Percolator. It highlights the need for flexible indexing strategies tailored to specific data use cases, pointing out considerations for built-in vs. external libraries, index cardinality, and client-side management of indexes. The author emphasizes the importance of allowing users to choose their indexing methods while maintaining ease of use and performance.

1. Jesse Yates
Salesforce.com
Secondary Indexing
the discussion so far….
9/11/12 HBase Pow-wow

2. What is it?

3. Problem
• HBase rows are multi-dimensional
– Only sorted on the row key
• How do you efficiently lookup deeper into the
row key?

4. Example
Row Family Qualifier Timestamp value
1 Name First 0 Babe
1 Name Last 0 Ruth
How do we find all people with the last name ‘Ruth’?
Full table scan!

5. Indexing!
Row Family Qualifier Timestamp Value
Ruth Name Last 0 1
Store the property we need to search
for as the primary key
• pointer back to the primary row
• fast lookup - O(lg(n))

6. Use Cases
• Point lookups
– Volume of data influences usefulness of index
• Let user decide if they need to use an index
• Scan lookup
– WHERE age > 16

7. Implementations

8. Omid
Full transactional support
Centralized oracle

9. Lily
WAL implementation on top of HBase
100-500 writes/sec

10. Percolator
Full transactions
Distributed, optimistic locking
~10 sec latencies possible

11. Culvert
Async
Dead project, incomplete

12. http://jyates.github.com/2012/07/0
9/consistent-enough-secondary-
indexes.html
Client-side coordinated index
Use timestamps to coordinate
Not yet implemented

13. Trend Micro Implementation
Still just POC
???

14. Solr/Lucene
Standard Lucene library bolted on HBase
Not commonly used
Lots of formats/codecs already written

15. Considerations for HBase
What do we need to do?

16. Built-in vs.
external library vs.
semi-supported (e.g. security)

17. Which should I use??
• HBase experts write a single ‘right’ impl
• Officially endorse a ‘correct’ version
• What changes do we need to make
• How close to the core is the project
– Written in everywhere
– hbase-index module
– External library

18. Async vs.
Synchronous vs.
Transactional

19. Key Observation
“Secondary indexing is inherently an easier
problem than full transactions… secondary
index updates are idempotent.”
- Lars Hofhansl

20. Async vs. Synchronous vs.Transactional
• We don’t need full transactions
– Transactions are slow
– Transactions fail with increasing probability as
number of servers increases
• Optionally async or sync
– Async
• Inherently ‘dirty’ index
• How does index cleanup work?
– Inherently different for each type

21. Locality

22. Where’s my data?
• Extra columns vs. index table
• HBase Region-pinning
– Has to be best-effort or will decrease availability
– Helps minimize RPC overhead
– Cross-table region-pinning
– Needs a coprocessor hook to be useful
• HDFS block allocation
– Keep index and data blocks on same HDFS node

23. Index Cardinality

24. How much data are we talking?
“Seems like there are 3 categories of sparseness:
1. sparse indexes (like ipAddress) where a per-table
approach is more efficient for reads
1. dense indexes (like eventType) where there are likely
values of every index key on each region
1. very dense indexes (like male/female) where you
should just be doing a table scan anyway”
- Matt Corgan (9/10/12)

25. Impact on implementation
• Need a lot of knowledge of data to pick the
right kind of index
– User knows their data, let them do the hard work
of picking indexes

26. Pluggability

27. Everyone’s got an impl already
• We need to make HBase flexible enough to
support (most) current indexing formats with
minimal overhead for switching
– Lucene style Codec/CodecProvider?

28. Client-interface

29. What should it look like?
• Minimal changes to the top-level interfaces
– Add a single new flag?
– Configuration based?
• Enough that the user gets to be smart about
what should be used
– We can’t get all cases right – just provide building
blocks
• Automatically use an index?
• Scanner/Filter style use?

30. Properties for the client
• Should the user even see the index lookups?
• ACID?
• Ordering of results?
– Support the current sorted order?
– Batch lookup?
• Implications on current features
– Replication
– splitting

31. Schema(less)
• Schema enforced?
– Rigid usage of index matching an expected schema?
– Schema table? Reserved schema columns?.META.?
• Schema-less
– Let the user apply whatever they think and use only
what actually works
• Best-effort
– Use client-hinted schema and try to apply all the
known indexes

32. My random thoughts….
• Client-side managed indexes are efficient
– Minimal RPC overhead
• Cleanup is async to client and rarely misses
– Solves the cross-region/server problem
• Region-pinning is a nice-to-have optimization
– Scales without concern for locality
– Flexible enough to support custom codecs
– Can be built to provide server-side optimizations
• Locality aware indexes to minimize RPCs

33. Discussion!