Talk Abstract Apache ZooKeeper plays a central role within the Accumulo architecture. Its quorum consistency model supports an overall Accumulo architecture with no single points of failure. Beyond that, Accumulo leverages ZooKeeper to store and communication configuration information for users and tables, as well as operational states of processes and tablets. For most Accumulo users, ZooKeeper is a black box full of goodness. Unfortunately, operational challenges mean we often have to delve into the dark depths to decipher what's going on when something goes wrong. In this talk, we will cover some basics about ZooKeeper's role, what it's good at and what it's not. Then we will discuss ways to debug what's stored inside of ZooKeeper, including how to overcome challenges with ZooKeeper's sometimes difficult ACL model. Speaker Michael Allen Security Architect, Sqrrl Michael Allen is Sqrrl's security architect. Before joining the team, Michael finished up 9 years working for PGP Corporation (and, post-acquisition, Symantec) in a variety of roles developing encryption software. In addition to encryption systems, Michael has extensive experience working with Java and Java-based web applications. He holds an MS in computer science from UC Santa Cruz, and a BA in Computer Science from Pomona College. When he's not making things up at work, Michael makes things up with other actors performing improvisational theater.

1. Securely explore your data
ZOOKEEPER,
ACCUMULO
AND YOU
Michael Allen
Architect
Sqrrl Data, Inc.

2. THE CASE OF THE DEAD TSERVER
Why when I close my laptop does
my tablet server die?

3. ACCUMULO CLUSTER
Worker
Hadoop
DataNode
Accumulo
Tserver
Worker
Hadoop
DataNode
Accumulo
Tserver
Accumulo
Master
Zookeeper
Cluster
Hadoop
NameNod
e

4. ZOOKEEPER CLUSTER
Zookeeper
Cluster

5. ZOOKEEPER CLUSTER
Zookeeper
Cluster

6. ZOOKEEPER CLUSTER
Zookeeper
Node
Zookeeper
Node
Zookeeper
Node

7. ZOOKEEPER CLUSTER
Zookeeper
Node
Zookeeper
Node
Zookeeper
Node
Zookeeper
Node
Zookeeper
Node

8. WHAT'S ZOOKEEPER GOOD AT?
Small amounts of
data
Consistent across
cluster
Fast reads
(in memory)
Total order of
operations
Zab

9. ZOOKEEPER DATA
Zookeeper Node

10. ZOOKEEPER DATA
Zookeeper Node
/
/accumulo
/accumulo/instances

11. ZOOKEEPER DATA
Zookeeper Node
/
/accumulo
/accumulo/instances
834c234-cd2731

12. ZOOKEEPER CLIENTS
Zookeeper Node
Zookeeper
Client
2181/tcp
Random long client ID and random password
Keep alive ping

13. EPHEMERAL NODES
Zookeeper Node
Ephemeral node

14. SEQUENTIAL NODES
Zookeeper Node
5
Sequential nodes
6

15. WATCHES
Zookeeper Node
5 6

16. ACCUMULO DATA IN ZOOKEEPER
/
accumulo
cf7c5ecd-0c3c-4d5c-aed2-36d9ca076976
user
monitor
root_tablet
gc table_locks hdfs_reservationsnamespaces recoveryfate
tservers tables
next_filetracers
config
dead bulk_failed_copyqmasters
instances

17. tservers
zlock-0000000000
server1.companyco.com:9997
TSERVERS
zlock-0000000000
server2.compnyco.com:9997

18. config
gc.cycle.delay
CONFIG
general.rpc.timeout general.server.message.size.max

19. tables
+r
TABLES
!0 1 2 3 4
state conf flush-id compact-id compact-cancel-id name namespace

20. root_tablet
last_location
ROOT_TABLET
location dir walogs

21. users
root
USERS
mallen jvines afuchs

22. USERS

23. USERS

24. users
root
ZOOKEEPER ACLS
mallen jvines afuchs
'digest,'accumulo:SkvnZlrIQ19GNd7eLDXGKg0Esgw=: cdrwa

25. DIGEST SCHEME REALLY MEANS
PASSWORDS

26. UH OH...
I forgot the password I used!
But...
I do have access to zkServer.sh

27. ROOTING YOUR ZOOKEEPER
1. Create an identity

28. ZK-DIGEST.SH
#!/bin/bash
if [ -z ${ZOOKEEPER_HOME} ]; then
echo "Set $ZOOKEEPER_HOME before running this script"
exit 4747
fi
if [ -z ${JAVA_HOME} ]; then
echo "Set $JAVA_HOME before running this script"
exit 4747
fi
if [ $# -eq 0 ]; then
echo "usage: zk-digest.sh <digest string>"
echo ""
echo " Utility to produce authentication digests, such as you might see in ZooKeeper node ACL entries"
echo ""
echo " Example: zk-digest.sh sqrrl:secret"
exit 4747
fi
ZK_CLASSPATH="
${ZOOKEEPER_HOME}/build/classes:
${ZOOKEEPER_HOME}/build/lib/*.jar:
${ZOOKEEPER_HOME}/lib/slf4j-log4j12-1.6.1.jar:
${ZOOKEEPER_HOME}/lib/slf4j-api-1.6.1.jar:
${ZOOKEEPER_HOME}/lib/netty-3.2.2.Final.jar:
${ZOOKEEPER_HOME}/lib/log4j-1.2.15.jar:
${ZOOKEEPER_HOME}/lib/jline-0.9.94.jar:
${ZOOKEEPER_HOME}/zookeeper-3.4.5.jar:
${ZOOKEEPER_HOME}/src/java/lib/*.jar:
${ZOOKEEPER_HOME}/conf
"
${JAVA_HOME}/bin/java -Dzookeeper.log.dir="."
-Dzookeeper.root.logger="INFO,CONSOLE"
-cp "${ZK_CLASSPATH}"
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.local.only=false
org.apache.zookeeper.server.auth.DigestAuthenticationProvider $*

29. ROOTING YOUR ZOOKEEPER
1. Create an identity
zk-digest.sh super:secret
super:secret->super:lK75jTNcA+U9vtVEw5vB51mj/w4=

30. ROOTING YOUR ZOOKEEPER
1. Create an identity
2. Edit zk-server.sh

31. ROOTING YOUR ZOOKEEPER
1. Create an identity
2. Edit zk-server.shnohup $JAVA "-Dzookeeper.log.dir=${ZOO_LOG_DIR}" "-Dzookeeper.root.logger=${ZOO_LOG4J_PROP}"
-cp "$CLASSPATH" $JVMFLAGS $ZOOMAIN "$ZOOCFG" > "$_ZOO_DAEMON_OUT" 2>&1 < /dev/null &

32. ROOTING YOUR ZOOKEEPER
1. Create an identity
2. Edit zk-server.shnohup $JAVA "-Dzookeeper.log.dir=${ZOO_LOG_DIR}" "-Dzookeeper.root.logger=${ZOO_LOG4J_PROP}"
"-Dzookeeper.DigestAuthenticationProvider.superDigest=super:lK75jTNcA+U9vtVEw5vB51mj/w4= "
-cp "$CLASSPATH" $JVMFLAGS $ZOOMAIN "$ZOOCFG" > "$_ZOO_DAEMON_OUT" 2>&1 < /dev/null &

33. ROOTING YOUR ZOOKEEPER
1. Create an identity
2. Edit zk-server.sh
3. Reboot Zookeeper

34. ROOTING YOUR ZOOKEEPER

35. THE CASE OF THE DEAD TSERVER
Why when I close my laptop does
my tablet server die?

36. THE CASE OF THE DEAD TSERVER
tservers
zlock-0000000000
server1.companyco.com:9997

37. THE CASE OF THE DEAD TSERVER
tservers
zlock-0000000000
server1.companyco.com:9997

38. QUESTIONS?