The Presto/Accumulo connector has been in production for over 18 months. It's been successful overall, but we have had some pain points along the way with some tech debt as well as design decisions. During this session, we'll briefly review the Accumulo connector for Presto as well as the use case that drove the initial implementation. We'll discuss the pain points we have experienced with the connector, and the latest features and changes to the connector to improve query performance, ingestion, and ease of use. – Speaker – Adam Shook Principal Consultant, Datacatessen Adam Shook is Founder and Principal Consultant at Datacatessen, a boutique big data solutions company specializing in data architecture and engineering. Shook graduated with a B.S. in Computer Science from the University of Maryland Baltimore County (UMBC) and took a job building a new high-performance graphics engine for a game studio. Looking for new challenges, he enrolled in the Computer Science graduate program at UMBC focusing on distributed computing technologies. Shook has worked on developing a wide variety of data applications and analytics deployed on large-scale production data platforms in both the commercial and government spaces. He is involved in developing and instructing graduate and undergraduate courses at UMBC, preparing young minds to work with big data. He spends what little free time he has playing video games and homebrewing. — More Information — For more information see http://www.accumulosummit.com/

1. Presto/Accumulo
Lessons Learned
Adam Shook Datacatessen
Datacatessen

2. Abstract
The Presto-Accumulo connector has been in
production for over 18 months. It's been successful
overall, but we have had some pain points along
the way with initial design decisions and tech debt.
During this session, we'll briefly review the
Accumulo connector for Presto and discuss the use
case that led to its development. We'll discuss the
pain points we have experienced with the
connector, and the latest features and changes to
the connector to improve query performance,
ingestion, and ease of use.
Datacatessen

3. Outline
• Presto-Accumulo Overview
• Use Case Review
• Lessons Learned and New Features
Datacatessen

4. Presto-Accumulo Review
• Open-source and built by Facebook
– MPP OLAP engine with pluggable storage
• ANSI SQL for NoSQL
• Aim is to accelerate relational OLTP use cases by abstracting
away common Accumulo design patterns
• Load data using SQL or Java
• Supports predicate pushdown via advanced indexes and
metrics
• Queries ranging from milliseconds to seconds
• Available since Presto 0.153
– See https://github.com/bloomberg/presto for the latest
features
Datacatessen

5. Client Coordinator
WorkerWorker Worker Worker
Accumulo
Coordinator leverages
indexes and optimizations
to gather Ranges to scan
Each worker is given a
subset of the Ranges to
read from Accumulo in
parallel via
BatchScanners
Workers pull data from
Accumulo, converting it
into Presto’s internal
object model
Accumulo’s job is done,
Presto takes over to
shuffle data as needed
and complete the query
Presto/Accumulo Workflow
Datacatessen

6. Bloomberg Use Case
• Surveillance application for Compliance Officer to review
events
• Web application uses JDBC to execute SQL queries against
Presto
• Ingest is done via Storm topology using the
PrestoBatchWriter
• Use case heavily relies on event time index to retrieve any
recent events
• Query performance ranges from 10 milliseconds to ~10
seconds for most common queries with table size in TBs
and thousands of tablets
• Presto deployed on Mesos
Datacatessen

7. LESSONS LEARNED
And the stuff done to make it better
Datacatessen

8. Dropping/Updating Data
• No trivial way to age off or update data using
the PrestoBatchWriter
– Delete mutations were ignored
– Updating was manual as indexes and metrics
needed to be dropped/decremented
– Unable to use AgeOffIterator
Datacatessen

9. Dropping/Updating Data
• Led to API improvements within the
PrestoBatchWriter to delete and update
– Supports delete mutations
– Properly handles deleting index entries, decrementing
metrics entries, and creating new index/metric entries
– Explicit API calls to change the values of columns
• Want to implement DELETE so we can drop data
via SQL
• LL: Need a clean API to delete stuff
Datacatessen

10. Arbitrary Row Batching
• Connector packs 50,000 row IDs into a split
– Split is the unit of parallelism in Presto
– Resulted in a variable number of splits, frequently
creating too few splits and not leveraging the
distributed nature of Presto
– Caused problems with concurrent queries over
larger data sets
Datacatessen

11. Arbitrary Row Batching
• Created a formula for determining the number of
splits per batch (with min/max)
– Number of splits is a function of desired number of
parallel queries and number of concurrent splits per
worker
• r: number of rows to be scanned
• s: splits per worker
• w: number of workers
• r / s / w
• LL: Need to properly generate splits for maximum
parallelization
Datacatessen

12. Bottleneck in Index Retrieval
• Connector would regularly spend several
seconds fetching row IDs from the index
– Process was single-threaded and non-distributed
– Regularly retrieving more rows than necessary
due to rows being filtered via predicates
Datacatessen
Row IDs where
user='adam'
Row IDs where
date='2017-10-16'
Row IDS where
user='adam' AND date='2017-10-16'

13. Bottleneck in Index Retrieval
• Three new features/optimizations
– ThreadPool to fetch row IDs in parallel
– Composite indexes
– Distributing the index lookup to Workers
• LL: More parallelism and more indexes make
faster queries
Datacatessen
'alice' and 'wendy'
'alice' and 'erin’ 'erin' and 'olivia' 'oscar' and 'wendy'
From Coordinator
To Worker

14. No Query History
• Presto’s Coordinator regularly purges past
queries, and they are lost on a restart of the
Coordinator
– Presto released an EventListener API that will
provide metrics and metadata about a query to all
implementors
Datacatessen

15. No Query History
• Implemented an AccumuloEventListener that
archives all queries in a Presto-Accumulo table
– Table is queryable via SQL
– Very helpful in generating usage reports
– History is persisted between restarts
• LL: Need proper storage of query history
Datacatessen

16. No Visibility or Timestamps
• Scope of the use case expanded to require
information within the full visibility label
– No way to access this information from a Presto
table’s schema
Datacatessen

17. No Visibility or Timestamps
• Added support for hidden visibility and
timestamp columns
– Doesn’t clutter up the forward facing DDL
– Get them for ‘free’ for all non-row ID columns (don’t
have to explicitly define them)
– Available via SELECT
• <column_name>_vis
• <column_name>_ts
• LL: Need to expose visibility and timestamp
information
Datacatessen

18. Large Ad-hoc Queries
• “Big” scans (millions of rows) are causing
problems
– Occupying scan threads on TabletServers
– Blocked other queries that needed near real-time
responses
Datacatessen

19. Large Ad-hoc Queries
• Still not really solved today
– We reject queries that will scan more than 50
million rows
– Been brain storming ideas such as reading Rfiles
• LL: Don’t use the Accumulo connector for
OLAP queries
Datacatessen

20. Index Hotspotting
• Indexing 1.0 was a quick win
– Basic reverse index
– Caused all of the problems you want to avoid with
indexes
– Low cardinality columns caused very wide rows
– Timestamp columns are monotonic increasing
– Key distribution was all over the place due to different
data types in the same table
– Deleting columns requires configuring RegexFilters
and compacting/merging the tables
Datacatessen

21. Index Hotspotting
• Large refactoring effort for Indexing 2.0
– Split index table into one table per indexed column
– Merged metrics into index table (separate locality
group)
– Added configurable IndexStorage methods that would
encode/decode to shard or post-fix random data
• LL: Support multiple index strategies based on
the type of data being stored
Datacatessen

22. Feature Summary
• PrestoBatchWriter improvements (deletions)
• Smarter resource allocation
• Composite Indexes
• Accumulo query archive (ROW types)
• Distributed Index Lookup
• Hidden visibility and timestamp columns
• Error on reading too much data
• Index hotspotting
Datacatessen

23. Questions?
Datacatessen