Turning Your Iterators Up to 11: New Features for Complex Iterator Implementations
•
0 likes•161 views
1) The document discusses new features for complex iterator implementations in Apache Accumulo, including stateful iterators, deep copying iterators, multi-threading iterators, and implementing the YieldingKeyValue interface for long running scans. 2) It provides examples of using iterators for pre-fetching data and implementing processing pipelines. 3) The document also covers best practices for debugging, testing, and monitoring iterators using metrics.
More Related Content
What's hot
Recently uploaded
Recently uploaded (20)
Turning Your Iterators Up to 11: New Features for Complex Iterator Implementations
- 2. Lifecycle • init(source, options, env) • seek(range, cfs, includeCfs) • while hasTop() • getTopKey(), getTopValue() • next() Turning your Iterators up to 1110/16/2017 2
- 4. Stateful • You can keep state in an iterator • If tablet moves, then state is lost – Rebalance or tserver process is stopped – Can use external caches for persistent state • Preserve state in deepCopy • There is no close method (yet) Turning your Iterators up to 1110/16/2017 4
- 5. Re-seek after Teardown • Iterators may be “torn down” • seek(new Range(lastKeyReturned, false, …), …) • How can you tell the difference from the initial seek? • Use start range inclusive flag • Keep track of last key returned and compare • Form of last key returned might be used, but this could be fragile Turning your Iterators up to 1110/16/2017 5
- 6. Multi-threading an iterator • Can be used to execute multiple parts of the range concurrently, or to pre-fetch results • Use static thread pools • Remember returned keys must be sorted • source.deepCopy per thread Turning your Iterators up to 1110/16/2017 6
- 7. deepCopy • Creates a copy of an iterator that can be used independently • Make sure you • Keep track of copies • Reuse copies if possible • One copy per thread Turning your Iterators up to 1110/16/2017 7
- 12. Our Use Case • Boolean queries against a document store • Documents are composed of field, value tuples • Iterator takes a boolean query with a range, and returns documents that match the query Turning your Iterators up to 1110/16/2017 12
- 13. Data Structure Turning your Iterators up to 11 Document Table … Shard_1 Global Index fv -> fn -> shard_x … Field Index fn,fv -> doc_n … Documents Doc_n -> fn, fv … 10/16/2017 13
- 14. Data Structure • Global Index • Maps values to field names and then to shards • Field Index • Within a shard, maps field name,values to doc ids • Documents: indexed by shard id + doc id • Within a shard, maps doc id to field name,value tuples Turning your Iterators up to 1110/16/2017 14
- 16. Metrics • Emanate real-time metrics from the tservers • See Timely • What to count • deepCopies • next and seek calls • Specific iterators • All of our metrics are tagged with the query id Turning your Iterators up to 1110/16/2017 16
- 17. Debugging iterators • Unit Tests: • Iterator test framework • Minicluster • Scan options • Include ids with your scans that you can track • Logging Turning your Iterators up to 1110/16/2017 17
- 18. Debugging iterators • Metrics • Include ids • Use table context on a cloned table • Allows testing a separate deployment at scale • Attaching to tservers • Do not to stop process when you hit a breakpoint Turning your Iterators up to 1110/16/2017 18
- 19. Future • Prioritization of scans • Set at scan time • Change priority dynamically • Invoking a close method • Automatically once hasTop is false • Invoked externally when client has left Turning your Iterators up to 1110/16/2017 19