Talk Abstract Bulk ingest enables Accumulo to import externally-prepared data into existing tables. Unlike ingest via batch writers, much of the work of organizing data can be left to external processing frameworks such as MapReduce and scaled independently of the Accumulo cluster itself. This reduces the work required of the tablet servers to support ingest, freeing resources to support other operations. Under the hood, bulk ingest involves a number a moving parts and accounting for a variety of failure scenarios. This talk covers the components of the bulk ingest process in-depth and describes past, current and future implementations of this capability. Attendees will leave this session with an understanding of bulk ingest that will enable troubleshooting, capacity estimation and performance management. Speaker Eric Newton Senior Software Developer, SWComplete Eric Newton has been a programmer for over 30 years, and has worked on Accumulo since 2009. He has been an open-source contributor and consumer since 1988. Through the years, his distributed communications systems work has included Air Traffic Control, Systems Monitoring and Databases. Eric has started 3 of his own companies and helped several other businesses start.

1. Building Bulk Import
Eric Newton
SW Complete, Inc.

2. Ingest 101
The life of a mutation:
○ Send to server
○ Write to Write-Ahead Log
○ Store in memory
○ Write memory to a file
○ Merge, re-write as needed
At least two writes, small in-memory sort

3. Bulk Ingest
● Accumulo: heavy ingest
● Use Map-Reduce efficiency
● Pre-sort incoming data
● Hand whole sorted files to accumulo
● One write
● Larger sorts

4. Version 1
importDirectory(String dir, String failDir)
Client computes the servers that need the file:
■ Analysis of file
■ Moves directory under Accumulo
■ Retry logic (to handle splits, failures)

5. Version 1: problems
● Limited to the client’s computational power
● Permission: client had to be all-knowing
● Files could be added to servers many times
● Defer file collection while bulk importing
● Clients can fail

6. Version 1.1
● Clients hand bulk imports to the master
● Fixed permission problems
● Added a bulk import test to the Random
Walk test suit

7. The Test
Create a sorted file with a lot of 1’s:
12345678 -> 1
Create an identical file, with lots of -1’s:
12345678 -> -1
Add a summation iterator over the table
Verify: every entry should be zero:
12345678 -> 0

8. Random Walk 101
Randomly:
○ Import files in random order
○ Split the files into random sizes
○ Split tablets and random points
○ Kill tablet servers (agitate)
Under load
At scale

9. Version 2.0
● Master only coordinates file processing
● Distribute work to tablet servers
● Master distributes files to tablet servers
● Tablet serves
○ Analyze files for assignments
○ Retry
○ Communicate with destination tablet servers

10. FilesFilesFiles
Command Flow
Client Master
Tablet
Server
Tablet
Server
Tablet
Server
Tablet
Server

11. Problems
Problems solved:
○ Permissions controlled by Master
○ File processing is distributed
Bulk import tested heavily
Consistency
○ Not so much

12. Version 3.0
Problem: file imported more than once
○ Repeated reloading
○ RPC timeouts
○ Tablet migration
○ Tablet split
Add flags to metadata table to prevent:
○ file garbage collection
○ repeated imports

13. Version 3.0
● Problem Solved
○ Reduced Name Node ops
○ Reduced trash laying around from failed imports
○ Imports not repeated
● Does it stand up to the Random Walk Test?
○ Not so much

14. Death by Slow Thread
“Please take this file”
“OK… looks good, never saw this before”
Sleep
“Hey, Please take this file, again”
“OK… looks good, never saw this before”
“Thanks!”
Compact!
Wakeup! Time to import that file from the 1st
request!

15. Zookeeper to the Rescue
Add a 3rd
party negotiator
● Define a session
● Add a file only while session is active
● Store session in zookeeper
● Get agreement about the session at each critical point,
including metadata table updates
Session guides clean-up of markers
Session closes only after all agree

16. Session
“Take this file, for session 123”
“OK, working on session 123”
“Never saw this file before”
Sleep
“Repeat, file for session 123”
“Never saw this file before”
“Anybody working on session 123?”

17. Session
“Yes, I’m processing session 123!”
“OK, finish up.”
Double check on session 123, import file
“Anybody working on session 123?”
“What’s session 123?”
Remove markers in metadata

18. Bulk Import
● Problem solved:
○ Distributed processing
○ Permissions
○ Files imported once, and only once
○ Markers are cleaned up in the face of failures
● Performance
○ Not so much

19. Master as Bottleneck
Bulk import thousands of files
Every 15 minutes
Master renames files and puts them under /accumulo
Bottleneck
One master competes for NN ops with N tablet servers

20. Master, Go Faster
Add configurable thread pool
Push more move requests to the NN
Compete more fiercely for resources

21. Bulk Import
More efficient data ingest
Easy: just a file to the right tablets
Hard: consistency in a distributed system
Testing is your friend
Nothing prepares you for large-scale problems!