This document summarizes a talk given by Noa Resare about Spotify's use of Cassandra for persistent storage. It introduces Spotify and its service needs, discusses why Cassandra was chosen, how it has performed, and some lessons learned. Key points covered include how Cassandra solved Spotify's storage problems through sharding and replication, its fast write speeds, flexibility for playlists, and potential for upgrades without downtime. Challenges with backups and solutions using incremental backups and a separate backup datacenter are also summarized.

1. Cassandra at Spotify
28th of March 2012

2. About this talk

3. About this talk
An introduction Spotify, to our service and
our persistent storage needs

4. About this talk
An introduction Spotify, to our service and
our persistent storage needs
What Cassandra brings

5. About this talk
An introduction Spotify, to our service and
our persistent storage needs
What Cassandra brings
What we have learned

6. About this talk
An introduction Spotify, to our service and
our persistent storage needs
What Cassandra brings
What we have learned
What I would have liked to have known a year
ago

7. About this talk
An introduction Spotify, to our service and
our persistent storage needs
What Cassandra brings
What we have learned
What I would have liked to have known a year
ago
Not a comparison between different NoSQL
solutions

8. About this talk
An introduction Spotify, to our service and
our persistent storage needs
What Cassandra brings
What we have learned
What I would have liked to have known a year
ago
Not a comparison between different NoSQL
solutions
The real reason: yes, we are hiring.

9. Noa Resare

10. Noa Resare
Stockholm, Sweden

11. Noa Resare
Stockholm, Sweden
Service Reliability Engineering

12. Noa Resare
Stockholm, Sweden
Service Reliability Engineering
noa@spotify.com

13. Noa Resare
Stockholm, Sweden
Service Reliability Engineering
noa@spotify.com
@blippie

14. Spotify — all music, all the time

15. Spotify — all music, all the time
A better user experience than file sharing.

16. Spotify — all music, all the time
A better user experience than file sharing.
Native desktop and mobile clients.

17. Spotify — all music, all the time
A better user experience than file sharing.
Native desktop and mobile clients.
Custom backend, built for performance and
scalability.

18. Spotify — all music, all the time
A better user experience than file sharing.
Native desktop and mobile clients.
Custom backend, built for performance and
scalability.

19. Spotify — all music, all the time
A better user experience than file sharing.
Native desktop and mobile clients.
Custom backend, built for performance and
scalability.
13 markets. More than ten million users.

20. Spotify — all music, all the time
A better user experience than file sharing.
Native desktop and mobile clients.
Custom backend, built for performance and
scalability.
13 markets. More than ten million users.
3 datacenters.

21. Spotify — all music, all the time
A better user experience than file sharing.
Native desktop and mobile clients.
Custom backend, built for performance and
scalability.
13 markets. More than ten million users.
3 datacenters.
Tens of gigabits of data pushed per
datacenter.

22. Spotify — all music, all the time
A better user experience than file sharing.
Native desktop and mobile clients.
Custom backend, built for performance and
scalability.
13 markets. More than ten million users.
3 datacenters.
Tens of gigabits of data pushed per
datacenter.
Backend systems that support a large set of
innovative features.

23. Innovative features in practice

24. Innovative features in practice
Playlist

25. Innovative features in practice
Playlist
A named list of tracks

26. Innovative features in practice
Playlist
A named list of tracks
Keep multiple devices in sync

27. Innovative features in practice
Playlist
A named list of tracks
Keep multiple devices in sync
Support nested playlists

28. Innovative features in practice
Playlist
A named list of tracks
Keep multiple devices in sync
Support nested playlists
Offline editing, pubsub

29. Innovative features in practice
Playlist
A named list of tracks
Keep multiple devices in sync
Support nested playlists
Offline editing, pubsub
Scale. More than half a billion lists currently
in the system

30. Innovative features in practice
Playlist
A named list of tracks
Keep multiple devices in sync
Support nested playlists
Offline editing, pubsub
Scale. More than half a billion lists currently
in the system
About 10 kHz on peak traffic.

31. Innovative features in practice
Playlist
A named list of tracks
Keep multiple devices in sync
Support nested playlists
Offline editing, pubsub
Scale. More than half a billion lists currently
in the system
About 10 kHz on peak traffic.
Result: accidentally implemented VCS

32. Suggested solutions

33. Suggested solutions
Flat files

34. Suggested solutions
Flat files
We don’t need ACID

35. Suggested solutions
Flat files
We don’t need ACID
Linux page cache kicks ass.

36. Suggested solutions
Flat files
We don’t need ACID
Linux page cache kicks ass.
(Not really)

37. Suggested solutions
Flat files
We don’t need ACID
Linux page cache kicks ass.
(Not really)
SQL

38. Suggested solutions
Flat files
We don’t need ACID
Linux page cache kicks ass.
(Not really)
SQL
Tried and true. Facebook does this

39. Suggested solutions
Flat files
We don’t need ACID
Linux page cache kicks ass.
(Not really)
SQL
Tried and true. Facebook does this
Simple Key-Value store

40. Suggested solutions
Flat files
We don’t need ACID
Linux page cache kicks ass.
(Not really)
SQL
Tried and true. Facebook does this
Simple Key-Value store
Tokyo cabinet, some experience

41. Suggested solutions
Flat files
We don’t need ACID
Linux page cache kicks ass.
(Not really)
SQL
Tried and true. Facebook does this
Simple Key-Value store
Tokyo cabinet, some experience
Clustered Key-Value store

42. Suggested solutions
Flat files
We don’t need ACID
Linux page cache kicks ass.
(Not really)
SQL
Tried and true. Facebook does this
Simple Key-Value store
Tokyo cabinet, some experience
Clustered Key-Value store
Evaluated a lot, end game contestants HBase
and Cassandra

43. Enter Cassandra

44. Enter Cassandra
Solves a large subset of storage related
problems

45. Enter Cassandra
Solves a large subset of storage related
problems
Sharding, replication

46. Enter Cassandra
Solves a large subset of storage related
problems
Sharding, replication
No single point of failure

47. Enter Cassandra
Solves a large subset of storage related
problems
Sharding, replication
No single point of failure
Free software

48. Enter Cassandra
Solves a large subset of storage related
problems
Sharding, replication
No single point of failure
Free software
Active community, commercial backing

49. Enter Cassandra
Solves a large subset of storage related
problems
Sharding, replication
No single point of failure
Free software
Active community, commercial backing
66 + 18 + 9 + 28 production nodes

50. Enter Cassandra
Solves a large subset of storage related
problems
Sharding, replication
No single point of failure
Free software
Active community, commercial backing
66 + 18 + 9 + 28 production nodes
About twenty nodes for various testing
clusters

51. Enter Cassandra
Solves a large subset of storage related
problems
Sharding, replication
No single point of failure
Free software
Active community, commercial backing
66 + 18 + 9 + 28 production nodes
About twenty nodes for various testing
clusters
Datasets ranging from 8T to a few gigs.

52. Cassandra, winning!

53. Cassandra, winning!
Major upgrades without service interruptions
(in theory)

54. Cassandra, winning!
Major upgrades without service interruptions
(in theory)
Crazy fast writes

55. Cassandra, winning!
Major upgrades without service interruptions
(in theory)
Crazy fast writes
Not just because you have a hardware RAID
card that is good at lying to you

56. Cassandra, winning!
Major upgrades without service interruptions
(in theory)
Crazy fast writes
Not just because you have a hardware RAID
card that is good at lying to you
Uses the knowledge that sequential is I/O
faster than random I/O

57. Cassandra, winning!
Major upgrades without service interruptions
(in theory)
Crazy fast writes
Not just because you have a hardware RAID
card that is good at lying to you
Uses the knowledge that sequential is I/O
faster than random I/O
In case of inconsistencies, knows what to do

58. Cassandra, winning!
Major upgrades without service interruptions
(in theory)
Crazy fast writes
Not just because you have a hardware RAID
card that is good at lying to you
Uses the knowledge that sequential is I/O
faster than random I/O
In case of inconsistencies, knows what to do
Cross datacenter replication support

59. Cassandra, winning!
Major upgrades without service interruptions
(in theory)
Crazy fast writes
Not just because you have a hardware RAID
card that is good at lying to you
Uses the knowledge that sequential is I/O
faster than random I/O
In case of inconsistencies, knows what to do
Cross datacenter replication support
Tinker friendly

60. Cassandra, winning!
Major upgrades without service interruptions
(in theory)
Crazy fast writes
Not just because you have a hardware RAID
card that is good at lying to you
Uses the knowledge that sequential is I/O
faster than random I/O
In case of inconsistencies, knows what to do
Cross datacenter replication support
Tinker friendly
Readable code

61. Cassandra flexibility for Playlist

62. Cassandra flexibility for Playlist
The main use cases for playlist:

63. Cassandra flexibility for Playlist
The main use cases for playlist:
Get me all changes since version N of playlist P

64. Cassandra flexibility for Playlist
The main use cases for playlist:
Get me all changes since version N of playlist P
Apply the following changes on top of version
M of playlist Q

65. Cassandra flexibility for Playlist
The main use cases for playlist:
Get me all changes since version N of playlist P
Apply the following changes on top of version
M of playlist Q
This translates to CFs head and change

66. Cassandra flexibility for Playlist
The main use cases for playlist:
Get me all changes since version N of playlist P
Apply the following changes on top of version
M of playlist Q
This translates to CFs head and change
Asymmetric sizes

67. Cassandra flexibility for Playlist
The main use cases for playlist:
Get me all changes since version N of playlist P
Apply the following changes on top of version
M of playlist Q
This translates to CFs head and change
Asymmetric sizes
Neat trick: read change with level=ONE,
fallback to LOCAL_QUORUM

68. Cassandra flexibility for Playlist
The main use cases for playlist:
Get me all changes since version N of playlist P
Apply the following changes on top of version
M of playlist Q
This translates to CFs head and change
Asymmetric sizes
Neat trick: read change with level=ONE,
fallback to LOCAL_QUORUM

69. Let me tell you a story

70. Let me tell you a story
Latest stable kernel from Debian Squeeze
2.6.32-5

71. Let me tell you a story
Latest stable kernel from Debian Squeeze
2.6.32-5
What happens after 209 days of uptime?

72. Let me tell you a story
Latest stable kernel from Debian Squeeze
2.6.32-5
What happens after 209 days of uptime?
Load average around 120.

73. Let me tell you a story
Latest stable kernel from Debian Squeeze
2.6.32-5
What happens after 209 days of uptime?
Load average around 120.
No CPU activity reported by top

74. Let me tell you a story
Latest stable kernel from Debian Squeeze
2.6.32-5
What happens after 209 days of uptime?
Load average around 120.
No CPU activity reported by top
Mattias de Zalenski:
log((209 days) / (1 nanoseconds)) / log(2) = 54.0034557
(2^54) nanoseconds = 208.499983 days
Somewhere nanosecond values are shifted ten bits?

75. Let me tell you a story
Latest stable kernel from Debian Squeeze
2.6.32-5
What happens after 209 days of uptime?
Load average around 120.
No CPU activity reported by top
Mattias de Zalenski:
log((209 days) / (1 nanoseconds)) / log(2) = 54.0034557
(2^54) nanoseconds = 208.499983 days
Somewhere nanosecond values are shifted ten bits?
Downtime for payment

76. Let me tell you a story
Latest stable kernel from Debian Squeeze
2.6.32-5
What happens after 209 days of uptime?
Load average around 120.
No CPU activity reported by top
Mattias de Zalenski:
log((209 days) / (1 nanoseconds)) / log(2) = 54.0034557
(2^54) nanoseconds = 208.499983 days
Somewhere nanosecond values are shifted ten bits?
Downtime for payment
Downtime for account creation

77. Let me tell you a story
Latest stable kernel from Debian Squeeze
2.6.32-5
What happens after 209 days of uptime?
Load average around 120.
No CPU activity reported by top
Mattias de Zalenski:
log((209 days) / (1 nanoseconds)) / log(2) = 54.0034557
(2^54) nanoseconds = 208.499983 days
Somewhere nanosecond values are shifted ten bits?
Downtime for payment
Downtime for account creation
No downtime for cassandra backed systems

78. Backups

79. Backups
A few terabytes of live data, many nodes.
Painful.

80. Backups
A few terabytes of live data, many nodes.
Painful.
Inefficient. Copy of on disk structure, at least
3 times the data

81. Backups
A few terabytes of live data, many nodes.
Painful.
Inefficient. Copy of on disk structure, at least
3 times the data
Non-compacted. Possibly a few tens of old
versions.

82. Backups
A few terabytes of live data, many nodes.
Painful.
Inefficient. Copy of on disk structure, at least
3 times the data
Non-compacted. Possibly a few tens of old
versions.
Pulling data off nodes evict hot data from
page cache.

83. Backups
A few terabytes of live data, many nodes.
Painful.
Inefficient. Copy of on disk structure, at least
3 times the data
Non-compacted. Possibly a few tens of old
versions.
Pulling data off nodes evict hot data from
page cache.
Initially, only full backups (pre 0.8)

84. Our solution to backups

85. Our solution to backups
NetworkTopologyStrategy is cool

86. Our solution to backups
NetworkTopologyStrategy is cool
Separate datacenter for backups with RF=1

87. Our solution to backups
NetworkTopologyStrategy is cool
Separate datacenter for backups with RF=1
Beware: tricky

88. Our solution to backups
NetworkTopologyStrategy is cool
Separate datacenter for backups with RF=1
Beware: tricky
Once removed from production performance
considerations

89. Our solution to backups
NetworkTopologyStrategy is cool
Separate datacenter for backups with RF=1
Beware: tricky
Once removed from production performance
considerations
Application level incremental backups

90. Our solution to backups
NetworkTopologyStrategy is cool
Separate datacenter for backups with RF=1
Beware: tricky
Once removed from production performance
considerations
Application level incremental backups
This week, cassandra level incremental
backups

91. Our solution to backups
NetworkTopologyStrategy is cool
Separate datacenter for backups with RF=1
Beware: tricky
Once removed from production performance
considerations
Application level incremental backups
This week, cassandra level incremental
backups
Still some issues: lots of SSTables

92. Solid state is a game changer

93. Solid state is a game changer
Asymmetrically sized datasets

94. Solid state is a game changer
Asymmetrically sized datasets
I Can Haz superlarge SSD?

95. Solid state is a game changer
Asymmetrically sized datasets
I Can Haz superlarge SSD?
No.

96. Solid state is a game changer
Asymmetrically sized datasets
I Can Haz superlarge SSD?
No.
With small disks, on disk data structure size
matters a lot

97. Solid state is a game changer
Asymmetrically sized datasets
I Can Haz superlarge SSD?
No.
With small disks, on disk data structure size
matters a lot
Our plan:

98. Solid state is a game changer
Asymmetrically sized datasets
I Can Haz superlarge SSD?
No.
With small disks, on disk data structure size
matters a lot
Our plan:
Leveled compaction strategy, new in 1.0

99. Solid state is a game changer
Asymmetrically sized datasets
I Can Haz superlarge SSD?
No.
With small disks, on disk data structure size
matters a lot
Our plan:
Leveled compaction strategy, new in 1.0
Hack cassandra to have configurable datadirs
per keyspace.

100. Solid state is a game changer
Asymmetrically sized datasets
I Can Haz superlarge SSD?
No.
With small disks, on disk data structure size
matters a lot
Our plan:
Leveled compaction strategy, new in 1.0
Hack cassandra to have configurable datadirs
per keyspace.
Our patch is integrated in Cassandra 1.1

101. Some unpleasant surprises

102. Some unpleasant surprises
Immaturity.

103. Some unpleasant surprises
Immaturity.
Has anyone written nodetool -h ring?

104. Some unpleasant surprises
Immaturity.
Has anyone written nodetool -h ring?
Broken on disk bloom filters in 0.8. Very
painful upgrade to 1.0

105. Some unpleasant surprises
Immaturity.
Has anyone written nodetool -h ring?
Broken on disk bloom filters in 0.8. Very
painful upgrade to 1.0
Small disk, high load, very possible to get
into an Out Of Disk condition

106. Some unpleasant surprises
Immaturity.
Has anyone written nodetool -h ring?
Broken on disk bloom filters in 0.8. Very
painful upgrade to 1.0
Small disk, high load, very possible to get
into an Out Of Disk condition
Logging is lacking

107. Lessons learned from backup datacenter

108. Lessons learned from backup datacenter
Asymmetric cluster sizes are painful.

109. Lessons learned from backup datacenter
Asymmetric cluster sizes are painful.
60 production nodes, 6 backup nodes

110. Lessons learned from backup datacenter
Asymmetric cluster sizes are painful.
60 production nodes, 6 backup nodes
Repairs that replicate all data 10 times

111. Lessons learned from backup datacenter
Asymmetric cluster sizes are painful.
60 production nodes, 6 backup nodes
Repairs that replicate all data 10 times
The workaround: manual repairs

112. Lessons learned from backup datacenter
Asymmetric cluster sizes are painful.
60 production nodes, 6 backup nodes
Repairs that replicate all data 10 times
The workaround: manual repairs
Remove sstables from broken node (to free up
space)

113. Lessons learned from backup datacenter
Asymmetric cluster sizes are painful.
60 production nodes, 6 backup nodes
Repairs that replicate all data 10 times
The workaround: manual repairs
Remove sstables from broken node (to free up
space)
Start it to have it take writes while repopulating

114. Lessons learned from backup datacenter
Asymmetric cluster sizes are painful.
60 production nodes, 6 backup nodes
Repairs that replicate all data 10 times
The workaround: manual repairs
Remove sstables from broken node (to free up
space)
Start it to have it take writes while repopulating
Snapshot and move SSTables from 4 evenly
spaced nodes

115. Lessons learned from backup datacenter
Asymmetric cluster sizes are painful.
60 production nodes, 6 backup nodes
Repairs that replicate all data 10 times
The workaround: manual repairs
Remove sstables from broken node (to free up
space)
Start it to have it take writes while repopulating
Snapshot and move SSTables from 4 evenly
spaced nodes
Do a full compaction

116. Lessons learned from backup datacenter
Asymmetric cluster sizes are painful.
60 production nodes, 6 backup nodes
Repairs that replicate all data 10 times
The workaround: manual repairs
Remove sstables from broken node (to free up
space)
Start it to have it take writes while repopulating
Snapshot and move SSTables from 4 evenly
spaced nodes
Do a full compaction
Do a repair and hope for the best

117. Spot the bug

118. Spot the bug
Hector java cassandra driver:

119. Spot the bug
Hector java cassandra driver:
private AtomicInteger counter = new AtomicInteger();
private Server getNextServer() {
counter.compareAndSet(16384, 0);
return servers[counter.getAndIncrement() % servers.length];
}

120. Spot the bug
Hector java cassandra driver:
private AtomicInteger counter = new AtomicInteger();
private Server getNextServer() {
counter.compareAndSet(16384, 0);
return servers[counter.getAndIncrement() % servers.length];
}
Race condition

121. Spot the bug
Hector java cassandra driver:
private AtomicInteger counter = new AtomicInteger();
private Server getNextServer() {
counter.compareAndSet(16384, 0);
return servers[counter.getAndIncrement() % servers.length];
}
Race condition
java.lang.ArrayIndexOutOfBoundsException

122. Spot the bug
Hector java cassandra driver:
private AtomicInteger counter = new AtomicInteger();
private Server getNextServer() {
counter.compareAndSet(16384, 0);
return servers[counter.getAndIncrement() % servers.length];
}
Race condition
java.lang.ArrayIndexOutOfBoundsException
After close to 2**31 requests

123. Spot the bug
Hector java cassandra driver:
private AtomicInteger counter = new AtomicInteger();
private Server getNextServer() {
counter.compareAndSet(16384, 0);
return servers[counter.getAndIncrement() % servers.length];
}
Race condition
java.lang.ArrayIndexOutOfBoundsException
After close to 2**31 requests
Took a few days

124. Thrift payload size limits
Communication with Cassandra is based on
thrift
Large mutations, larger than 15MiB
Thrift drops the underlying TCP connection
Hector considers the connection drop a node
specific problem
Retries on all cassandra nodes
Effectively shutting down all cassandra traffic

125. Conclusions

126. Conclusions
In the 0.6-1.0 timeframe, developers and
operations engineers are needed

127. Conclusions
In the 0.6-1.0 timeframe, developers and
operations engineers are needed
You need to keep an eye on bugs created, be
part of the community

128. Conclusions
In the 0.6-1.0 timeframe, developers and
operations engineers are needed
You need to keep an eye on bugs created, be
part of the community
Exotic stuff (such a asymmetrically sized
datacenters) is tricky

129. Conclusions
In the 0.6-1.0 timeframe, developers and
operations engineers are needed
You need to keep an eye on bugs created, be
part of the community
Exotic stuff (such a asymmetrically sized
datacenters) is tricky
Lots of things gets fixed. You need to keep
up with upstream

130. Conclusions
In the 0.6-1.0 timeframe, developers and
operations engineers are needed
You need to keep an eye on bugs created, be
part of the community
Exotic stuff (such a asymmetrically sized
datacenters) is tricky
Lots of things gets fixed. You need to keep
up with upstream
You need to integrate with monitoring and
graphing

131. Conclusions
In the 0.6-1.0 timeframe, developers and
operations engineers are needed
You need to keep an eye on bugs created, be
part of the community
Exotic stuff (such a asymmetrically sized
datacenters) is tricky
Lots of things gets fixed. You need to keep
up with upstream
You need to integrate with monitoring and
graphing
Consider it a toolkit for constructing
solutions.

132. Questions? Answers.