Most large organizations using Accumulo, such as Bloomberg, have two or more data centers distributed geographically. Replication was recently added to Accumulo providing automated eventual consistency between these data centers. On the other hand clients are still left with a need for an automated way to handle a failover from one Accumulo cluster to another. A client application needs the ability to continue processing and operating regardless of what data center. An operations manager of an Accumulo data center needs the capability to control which Accumulo data center should be used by clients. At any given point in time a data center failover could be triggered by routine maintenance, system upgrade, or any other unforeseen event. To solve these use cases, the Accumulo Client Failover (ACF) API was developed. This presentation will cover the motivation, design, and use of ACF. We will show how ACF is used by an Accumulo client, and how it can be used alongside an existing third-party Accumulo client library. – Speaker –  Noe Detore Platform Architect , Miner & Kasch Mr. Detore has over fourteen years of software engineering experience. His past 5 years, as a senior cloud java developer, have focused on the entire architecture of cloud computing from utility, ingest, storage, M/R, query and UI. Before cloud Mr. Detore gained his J2EE experience using Spring and other java frameworks to develop rest services, ESB, and web applications. — More Information — For more information see http://www.accumulosummit.com/

1. Noe Detore
Automated data center selection for a client
Accumulo Client Failover
Copyright 2016 Bloomberg Finance L.P. All rights reserved.

2. Abstract
Most large organizations using Accumulo, such as Bloomberg, have two or more data centers distributed
geographically. Replication was recently added to Accumulo providing automated eventual consistency
between these data centers. On the other hand clients are still left with a need for an automated way to
handle a failover from one Accumulo cluster to another. A client application needs the ability to continue
processing and operating regardless of data center outages or failovers.
An operations manager of an Accumulo data center needs the capability to control which Accumulo data
center should be used by clients. At any given point in time a data center failover could be triggered by routine
maintenance, system upgrade, or any other unforeseen event.
To solve these use cases, the Accumulo Client Failover (ACF) API was developed. This presentation will cover
the motivation, design, and use of ACF. We will show how ACF is used by an Accumulo client, and how it can be
used alongside an existing third-party Accumulo client library.

3. Outline
•Accumulo and multiple data centers
•What about clients?
•Accumulo Client Failover (ACF)
•How is ACF used
•ACF used with 3rd party library

4. Multi Data Center Support
• Accumulo 1.7 supports eventually consistent replication
– Write-ahead-log (WAL) files are sent when closed
– Multiple configurations supported

5. • Clients write to multiple clusters
• Use NiFi to deliver the same ingestion to multiple DCs
Alternative Architectures

6. Accumulo Users & Multiple DCs
• No automated way to know active DC
• Not able to failover from one DC to other
• Maintenance outages require coordination with
all clients
• No automated failover for catastrophic event

7. Accumulo Client Failover (ACF)
Goals
• A client library that can handle arbitrary failover of Accumulo clusters
• Provide failover capable implementations of Accumulo interfaces
– Connector, Instance, Scanner, BatchScanner, BatchWriter
– Use an eventingmodel to notify clients of an active/preferred
Accumulo cluster
• Upon receiving a failover event, client code will be interrupted so that it
can do cleanup before continuing on with the new Accumulo cluster

8. Replication with Failover

9. ACF Initial Release
• Failover is controlled and triggered by ZooKeeper
• Current Accumulo instance information from znode
• Update of znode triggers failover
• Watcher on znode creates FailoverEvent when changed
• Ultimately results in reader or writer throwing a
FailoverException for a client to handle

10. • FailoverController triggers FailoverEvent
• FailoverHandler and FailoverFlag
• FailFailoverInstance like ZooKeeperInstance with a handle
• FailoverConnector type behaves in a similar fashion
– Once notified will notify all readers/writers it created
• FailoverScanner, FailoverBatchScanner,
FailoverBatchWriter…
– Will throw a FailoverException on the next method call
ACF Design

11. ACF Code Example
Connector con = new ZooKeeperFailoverController(znode,zookeepers)
.getInstance().getConnector(principal,token);
while(true) {
try {
Scanner s = con.createScanner(table, auths);
s.setRange(range);
for(Entry<Key, Value> e : s) System.out.println(e);
break;
}catch(FailoverException e) {
System.println(“failover, going to try again”);
}
}

12. Retryable
• An API to abstract away the while loop
• Retry any actions in case they fail
• Allows users to focus solely on user code
retry(() -> {
Scanner s = con.createScanner(table, auths);
s.setRange(range);
for(Entry<Key, Value> e : s) System.out.println(e);
})

13. Third Party APIs
GeoWave
dataStore = new AccumuloDataStore(new BasicAccumuloOperations(connector));
retry(() -> { IndexWriter indexWriter = dataStore.createWriter(adapter, index));
for (SimpleFeature sf : features) {indexWriter.write(sf);}
retry(() -> { TripleFileWriter tripleFileWriter = new TripleFileWriter(connector));
File rowFile = getDataFile(String.valueOf(SCALE)+version+"r.txt");
File colFile = getDataFile(String.valueOf(SCALE)+version+"c.txt");}

14. Questions
?