This document discusses running Apache Accumulo, a distributed key-value store, on Amazon's Elastic Compute Cloud (EC2). It provides details on instance types and configurations for Accumulo and Hadoop on EC2, including software, ports, and security groups. It also covers scaling the Accumulo cluster on EC2 by provisioning new instances, handling failures automatically, and scaling down by removing instances. The document shows Accumulo can scale to handle over a million writes per second on EC2 and that clients see no errors when machines fail or are added.

1. Accumulo in the Cloud
Amazon’s EC2

2. Apache Accumulo

3. BigTable

4. Java

5. Apache Hadoop

6. Advanced Features

7. Cell-level security

8. Time
Row Column Visibility Value
Stamp
family |
Joe photos:vacation - photo425.jpg
friends
Joe photos:expo work - photo648.jpg
photos:
Joe friends - photo772.jpg
bachelor_party

9. Time
Row Column Visibility Value
Stamp
family |
Joe photos:vacation - photo425.jpg
friends
Joe photos:expo work - photo648.jpg
photos:
Joe friends - photo772.jpg
bachelor_party
FRIENDS

10. Time
Row Column Visibility Value
Stamp
family |
Joe photos:vacation - photo425.jpg
friends
Joe photos:expo work - photo648.jpg
photos:
Joe friends - photo772.jpg
bachelor_party
FAMILY

11. Time
Row Column Visibility Value
Stamp
family |
Joe photos:vacation - photo425.jpg
friends
Joe photos:expo work - photo648.jpg
photos:
Joe friends - photo772.jpg
bachelor_party
WORK

12. Combiners:

13. Combiners:
Server-side Computation

14. Time
Row Column Visibility Value
Stamp
storeA sales:shoes acct 20120305 100
storeA sales:shoes acct 20120303 550
storeA sales:shoes acct 20120302 300
storeA sales:cameras acct 20120305 100
storeA sales:cameras acct 20120303 200
COMBINER: SUM()

15. Time
Row Column Visibility Value
Stamp
storeA sales:shoes acct - 950
storeA sales:cameras acct - 300
COMBINER: SUM()

16. Time
Row Column Visibility Value
Stamp
storeA sales:shoes acct - 950
storeA sales:cameras acct - 300
storeA sales:shoes acct 200120306 150
COMBINER: SUM()

17. Time
Row Column Visibility Value
Stamp
storeA sales:shoes acct - 1100
storeA sales:cameras acct - 300
COMBINER: SUM()

18. Combiners execute

19. Query time

20. Asynchronously in
background

21. Insert / updates are fast

22. 10s of thousands per
second per server

23. Accumulo Leading brand
0 7500 15000 22500 30000
Inserts
Updates
WRITE PERFORMANCE
PROJECT
APPROXIMATE
DATE CLIENT
DATE NAME

24. Amazon
Elastic Compute Cloud

25. Really fast provisioning

26. On-demand hardware

27. Scale up/down in minutes

28. Independence
DIFFERENT MACHINES
DATA CENTERS
GEOGRAPHIC LOCATIONS

29. Spot Instances

30. Set max price
for instances

31. They’re yours
until you’re outbid

32. SPOT INSTANCE PRICING HISTORY

34. Are Big Data and Cloud a
match made in heaven?

35. MATCH
Big Data Cloud
Always need more hardware, Add more machines as
not alway easy to predict needed

36. MATCH
Big Data Cloud
Need so much hardware,
Sells exclusively commodity
have to use lots of
virtual servers that blow up
commodity machines and
occasionally
fault-tolerant software

37. MATCH
Big Data Cloud
Having lots of independent
I/O (IOPs and bandwidth) is
Offers disks local to the
important, both for
instance
MapReduce and servicing
requests

38. MISMATCH
Big Data Cloud
Having lots of independent
I/O (IOPs and bandwidth) is
Offers shared storage over
important, both for
ethernet (SAN)
MapReduce and servicing
requests

39. MISMATCH
Big Data Cloud
Large volumes of data have Create an entire cluster from
‘mass’, get harder to move nothing, bring all the data in,
around process, write it all out

40. MISMATCH, BUT OK
Big Data Cloud
Software benefits from lots
Heavily reliance on
of independent hardware,
virtualization
and using all the hardware

41. Scalable administration
and elasticity

42. When a machine fails

43. Is admin intervention
required?

44. Will clients see
exceptions?

45. When adding a machine

46. Must data move before it
can service requests?

47. Must processes be
restarted?

48. Accumulo
FAILOVER AUTOMATIC
RE-REPLICATION AUTOMATIC
DATA REPLICATED ASYNCHRONOUSLY
REQUEST-LOAD BALANCED INDEPENDENTLY
NEW MACHINES DISCOVERED AUTOMATICALLY

49. Some Other NoSQL dbs ...
NOT SO MUCH

50. Your DB needs to scale in
the human dimension too.

51. Accumulo in EC2

52. Instance Types

53. 64-bit machines

54. no 32-bit t1.micros,
m1.small, or m1.medium

55. about 2-4GB RAM / core

56. about 2 disks / core

57. m1.larges

58. 7.5GB RAM for 4 cores*
*EC2 COMPUTE UNITS

59. 2 x 420GB disks

60. m1.xlarge

61. 15GB RAM for 8 cores

62. 4 x 414GB disks

63. m2.xlarge

64. 17.1GB RAM for 6.5 cores

65. only 1 x 420GB disk

66. Bigger instances have
more RAM, CPU ...

67. Not more disks

68. EBS is an option ...

69. RAID is not necessary

70. Lose some independence

71. Exceptions

72. HDFS NameNode

73. Might get a machine with
more memory

74. 68GB is the current max

75. Use RAID across multiple
NameNode disks

76. Where to place instances?

77. Region = geographic area

78. Availability Zone
= Data Center

79. Can span multiple AZs

80. Tested on four AZs
in East US region

81. Spanning regions?

82. Cross-site/WAN replication

83. AMIs

84. Ubuntu
Maverick (10.10) x86_64

85. CentOS

86. Cloudera Hadoop

87. Cloudera’s Whirr
HTTPS://CCP.CLOUDERA.COM/DISPLAY/CDHDOC/WHIRR+INSTALLATION

88. OS Config

89. No swapping

90. Swappiness = 0

91. Up open file limits to 64k

92. Software

93. Hadoop

94. Only need HDFS

95. MapReduce is fine though

96. Version 0.20

97. Cloudera CDH3u2

98. or MapR!

99. ZooKeeper

100. Version 3.3.1 or greater

101. Java

102. Sun Java JDK 1.6

103. OpenJDK 1.6

104. Accumulo 1.3.5 or 1.4.0

105. wget
HTTP://INCUBATOR.APACHE.ORG/ACCUMULO/DOWNLOADS

106. Configuration

107. Use internal IP addresses
for important machines

108. HDFS NameNode

109. Accumulo Master

110. MapReduce JobTracker

111. EC2 Security Groups

112. a.k.a. the Firewall

113. 2181,2888,3888 ZOOKEEPER
Ports 4560 ACCUMULO MONITOR
9000 HDFS
9001 JOBTRACKER
9997 TABLET SERVER
9999 MASTER SERVER
11224 ACCUMULO LOGGER
12234 ACCUMULO TRACER
50010 DATANODE DATA
50020 DATANODE METADATA
50060 TASKTRACKERS
50070 NAMENODE HTTP MONITOR
50075 DATANODE HTTP MONITOR
50091 ACCUMULO
50095 ACCUMULO HTTP MONITOR

114. Create SSH key on
Accumulo Master

115. Distribute to tablet servers

116. Follow ordinary Hadoop
and Accumulo config steps

117. Scaling Up

118. Provision new instances

119. Only need to be identically
configured

120. Could store config files
into an AMI

121. Start HDFS data node

122. Start Accumulo loggers
and Tablet Server

123. Tablet servers assigned
tablets immediately

124. HDFS blocks can be
rebalanced

125. Some new blocks will go
to new machines

126. Scaling Up 20
40
100
200
400

127. Scaling Up 20
40
ABOUT 85% INCREASE IN WRITE RATE 100
EACH TIME CLUSTER SIZE DOUBLED
200
400

128. Scaling Up 20
40
HIT 1 MILLION WRITES PER SECOND 100
AT 100 M1.LARGES WITH 50 CLIENTS
200
400

129. m1.large NameNode
served 400 node cluster

130. Running ‘at scale’

131. Some machines will be
lost

132. Everything keeps working

133. Accumulo auto-recovers
using write-ahead logs

134. Occasionally provision
replacement machines

135. On your schedule

136. Clients see no exceptions /
errors

137. Watch writes and reads
scale on monitor page
HTTP://ACCUMULO-MONITOR:50095

139. See a list of failed
machines too

140. How many machines?

141. May need up to 2x VMs vs.
bare metal

142. Scaling Down

143. Identify a set of machines
to remove

144. Stop tablet servers,
loggers

145. accumulo admin stop <node>
IN VERSION 1.4

146. Decommission HDFS
datanodes
HTTP://DEVELOPER.YAHOO.COM/HADOOP/TUTORIAL/MODULE2.HTML

147. NameNode will
re-replicate blocks

148. Remaining machines have
enough storage?

149. Can lower replication
factor if necessary

150. When decommissioning is
done, terminate instances

151. Scale Up again!

152. Repeat.

153. Details
HTTP://WWW.ACCUMULODATA.COM/EC2.HTML

154. Questions