1. Castle: Reinventing
Storage for Big Data
Tom Wilkie
Founder & VP Engineering

2. Before the Flood
1990
Small databases
BTree indexes
BTree File systems
RAID
Old hardware

3. Two Revolutions
2010
Distributed, shared-nothing databases
Write-optimised indexes Write-optimised indexes
BTree file systems BTree file systems
RAID ... RAID
New hardware New hardware

4. Bridging the Gap
2011
Distributed, shared-nothing databases
Castle Castle
...
New hardware New hardware

5. With Big Data, how
do I...
S HO TS
SN AP

6. What’s in the
Castle?

7. Shared memory interface
keys
Userspace
Acunu Kernel
values
In-kernel
async, shared
memory ring workloads
interface
shared buffers
userspace
Streaming interface
range key buffered key buffered
queries insert value insert get value get
interface
kernelspace
Doubling Arrays
insert Bloom filters
queues key
get x
arrays
range arrays
queries management
mapping layer
key
doubling array
insert merges
Arrays
key Version tree
insert btree
key
get
btree
range
modlist btree
mapping layer
queries value arrays
Cache
"Extent" layer
extent block
extent cache
freespace
allocator
prefetcher
manager
& mapper
cacheing layer
flusher
block mapping &
page cache
Linux Kernel
Block layer Memory manager
MM layers
linux's block &

8. Shared memory interface
Castle
keys
Userspace
Acunu Kernel
userspace
interface
values
In-kernel
async, shared
memory ring workloads
shared buffers
kernelspace
Streaming interface
interface
range key buffered key buffered
queries insert value insert get value get
Doubling Arrays • Opensource (GPLv2, MIT
doubling array
mapping layer
for user libraries)
insert Bloom filters
queues key
get
arrays x
range arrays
queries management
key
•
insert merges
http://bitbucket.org/acunu
Arrays
mapping layer
•
modlist btree
key Version tree
Loadable Kernel Module,
insert btree
key
get
btree
targeting CentOS’s 2.6.18
range
queries value arrays
•
Cache
block mapping &
http://www.acunu.com/
cacheing layer
"Extent" layer
prefetcher
extent block
extent cache
blogs/andy-twigg/why-
freespace
allocator
manager
flusher
& mapper
page cache
acunu-kernel/
linux's block &
Linux Kernel
MM layers
Block layer Memory manager

9. The Interface
Shared memory interface
keys
Userspace
Acunu Kernel
userspace
interface
values
In-kernel
async, shared
memory ring workloads
shared buffers
kernelspace
Streaming interface
interface
range key buffered key buffered
queries insert value insert get value get
Doubling Arrays
ubling array
apping layer
insert Bloom filters
queues key
get
arrays x
range
queries
castle_{back,objects}.c
arrays
management

10. interface
userspac
values
In-kernel
async, shared
memory ring workloads
shared buffers
kernelspace
interface
Doubling Array Streaming interface
range key buffered key buffered
queries insert value insert get value get
Doubling Arrays
doubling array
mapping layer
insert Bloom filters
queues key
get
arrays x
range arrays
queries management
key
insert merges
Arrays
mapping layer
modlist btree
key Version tree
insert btree
key
get
btree
range
queries value arrays
castle_{da,bloom}.c

11. Range Query
Update
(Size Z)
O(logB N) O(Z/B)
B-Tree random IOs random IOs
B = “block size”, say 8KB at 100 bytes/entry ~= 100 entries

12. Doubling Array
Inserts
2 2 9
9
Buffer arrays in memory
until we have > B of them

13. Doubling Array
Inserts
11 2 9 2 8 9 11
8 8 11
etc...
Similar to log-structured merge trees (LSM), cache-
oblivious lookahead array (COLA), ...

14. Demo
https://acunu-videos.s3.amazonaws.com/dajs.html

15. Range Query
Update
(Size Z)
O(logB N) O(Z/B)
B-Tree random IOs random IOs
O((log N)/B)
Doubling Array sequential IOs
B = “block size”, say 8KB at 100 bytes/entry ~= 100 entries

16. Doubling Array
Queries
query(k)
• Add an index to each array to do lookups
• query(k) searches each array independently

17. Doubling Array
Queries
query(k)
• Bloom Filters can help exclude arrays from
search
• ... but don’t help with range queries

18. 8KB @ 100MB/s, w/ 8ms seek 100 / 5
= 100 IOs/s = 20 updates/s
~ log (2^30)/log 100
= 5 IOs/update
Range Query
Update
(Size Z)
O(logB N) O(Z/B)
B-Tree random IOs random IOs
O((log N)/B) O(Z/B)
Doubling Array sequential IOs sequential IOs
~ log (2^30)/100 8KB @ 100MB/s 13k / 0.2
= 0.2 IOs/update = 13k IOs/s = 65k updates/s
B = “block size”, say 8KB at 100 bytes/entry ~= 100 entries

19. interface
userspac
values
In-kernel
async, shared
memory ring workloads
shared buffers
kernelspace
interface
Doubling Array Streaming interface
range key buffered key buffered
queries insert value insert get value get
Doubling Arrays
doubling array
mapping layer
insert Bloom filters
queues key
get
arrays x
range arrays
queries management
key
insert merges
Arrays
mapping layer
modlist btree
key Version tree
insert btree
key
get
btree
range
queries value arrays
castle_{da,bloom}.c

20. Doubling Arrays
doubling array
mapping layer
“Mod-list” B-Tree
insert Bloom filters
queues key
get
arrays x
range arrays
queries management
key
insert merges
Arrays
mapping layer
modlist btree
key Version tree
insert btree
key
get
btree
range
queries value arrays
Cache
block mapping &
cacheing layer
"Extent" layer
prefetcher
extent block
extent cache
freespace
allocator
manager
flusher
& mapper
page cache
castle_{btree,versions}.c
&

21. Copy-on-Write BTree
Idea:
• Apply path-copying [DSST] to
the B-tree
Problems:
• Space blowup: Each update may
rewrite an entire path
• Slow updates: as above
A log file system makes updates sequential, but relies on
random access and garbage collection (achilles heel!)

22. Range
Update Space
Query
CoW B- O(logB Nv) O(Z/B)
O(N B logB Nv)
Tree random IOs random IOs
Nv = #keys live (accessible) at version v

23. “BigTable” snapshots
v1
• Inserts produce arrays
1 a 1 b

24. “BigTable” snapshots
v1 v2
• Inserts produce arrays
2
1 a 2
1 b 1 c • Snapshots increment ref
counts on arrays
• Merges product more
arrays, decrement ref
count on old arrays

25. “BigTable” snapshots
v1 v2
• Inserts produce arrays
1 a 1 b • Snapshots increment ref
counts on arrays
• Merges product more
1 a b c
arrays, decrement ref
count on old arrays

26. “BigTable” snapshots
v1 v2
• Inserts produce arrays
1 a 1 b • Snapshots increment ref
counts on arrays
• Merges product more
1 a b c
arrays, decrement ref
count on old arrays
• Space blowup

27. Range
Update Space
Query
CoW B- O(logB Nv) O(Z/B)
O(N B logB Nv)
Tree random IOs random IOs
“BigTable” O((log N)/B) O(Z/B)
O(VN)
style DA sequential IOs sequential IOs
Nv = #keys live (accessible) at version v

28. “Mod-list” BTree
Idea:
• Apply fat-nodes [DSST] to the
B-tree
• ie insert (key, version, value)
tuples, with special operations
Problems:
• Similar performance to a BTree
If you limit the #versions, can be constructed
sequentially, and embedded into a DA

29. Range
Update Space
Query
CoW B- O(logB Nv) O(Z/B)
O(N B logB Nv)
Tree random IOs random IOs
“BigTable” O((log N)/B) O(Z/B)
O(VN)
LevelDB
style DA sequential IOs sequential IOs
“Mod-list” O((log N)/B) O(Z/B)
CASTLE
in a DA sequential IOs sequential IOs
O(N)
Nv = #keys live (accessible) at version v

30. Stratified BTree
v1 v2 v2 v1 v2 v1 v0 v1 v0 v1 v0 v1
Problem:
newer older Embedded “Mod-
merge (duplicates removed) list” #versions limit
Solution:
k1 k2 k3 k4 k5
v1 v0 v2 v1 v0 v2 v1 v0 v2 v1
Version-split arrays
v-split during merges
k1 k2 k3 k4 k5
v0 entries here are
{v2} v0 v2 v2 v0 v2 v0
duplicates
k1 k2 k4 k5
v1 v2
{v1,v0} v1 v0 v1 v0 v1 v0 v1

31. Doubling Arrays
doubling array
mapping layer
“Mod-list” B-Tree
insert Bloom filters
queues key
get
arrays x
range arrays
queries management
key
insert merges
Arrays
mapping layer
modlist btree
key Version tree
insert btree
key
get
btree
range
queries value arrays
Cache
block mapping &
cacheing layer
"Extent" layer
prefetcher
extent block
extent cache
freespace
allocator
manager
flusher
& mapper
page cache
castle_{btree,versions}.c
&

32. Arrays
mapping layer
modlist btree
key Version tree
insert btree
Disk Layout: RDA
key
get
btree
range
queries value arrays
Cache
block mapping &
cacheing layer
"Extent" layer
prefetcher
extent block
extent cache
freespace
allocator
manager
flusher
& mapper
page cache
linux's block &
Linux Kernel
MM layers
Block layer Memory manager
castle_{cache,extent,freespace,rebuild}.c

33. Disk Layout: RDA
random duplicate allocation
4 2 1 4 5 2 5 3 1 3
7 10 7 6 8 9 9 10 6 8
15 12 14 11 13 14 11 12 13 15
16 16

34. Performance
Comparison

35. Small random inserts
Inserting 3 billion rows
Acunu powered Cassandra -
‘standard’ Cassandra -

36. Insert latency
While inserting 3 billion rows
Acunu powered Cassandra x
‘standard’ Cassandra +

37. Small random range queries
Performed immediately after inserts
Acunu powered Cassandra -
‘standard’ Cassandra -

38. Performance summary
Standard Acunu Benefits
inserts rate ~32k/s ~45k/s >1.4x
95% latency ~32s ~0.3s >100x
gets rate ~100/s ~350/s >3.5x
95% latency ~2s ~0.5s >4x
range queries ~0.4/s ~40/s >100x
95% latency ~15s ~2s >7.5x

39. • Castle: like BDB, but for Big Data
• DA: transforms random IO into
sequential IO
• Snapshots & Clones: addressing
real problems with new workloads
• 2 orders of magnitude better
performance and predictability

40. Questions?
Tom Wilkie
@tom_wilkie
tom@acunu.com
http://bitbucket.org/acunu
http://www.acunu.com/download
http://www.acunu.com/insights

41. References
[LSM] The Log-Structured Merge-Tree (LSM-Tree)
Patrick O'Neil, Edward Cheng, Dieter Gawlick,
Elizabeth O'Neil Stratified B-trees and versioned dictionaries, - Andy
http://staff.ustc.edu.cn/~jpq/paper/flash/1996-The Twigg, Andrew Byde, Grzegorz Miłoś, Tim Moreton,
%20Log-Structured%20Merge-Tree%20%28LSM- John Wilkes, Tom Wilkie, HotStorage’11
Tree%29.pdf http://www.usenix.org/event/hotstorage11/tech/
final_files/Twigg.pdf
[COLA] Cache-Oblivious Streaming B-trees,
Michael A. Bender et al [RDA] Random duplicate storage strategies for
http://www.cs.sunysb.edu/~bender/newpub/ load balancing in multimedia servers, 2000, Joep
BenderFaFi07.pdf Aerts and Jan Korst and Sebastian Egner
http://www.win.tue.nl/~joep/IPL.ps
[DSST] Making Data Structures Persistent - J. R.
Driscoll, N. Sarnak, D. D. Sleator, R. E. Tarjan, Making Apache, Apache Cassandra, Cassandra, Hadoop, and
Data Structures Persistent, Journal of Computer the eye and elephant logos are trademarks of the
and System Sciences,Vol. 38, No. 1, 1989 Apache Software Foundation.
http://www.cs.cmu.edu/~sleator/papers/making-
data-structures-persistent.pdf