Making sure your Data Model will work on the production cluster after 6 months as well as it does on your laptop is an important skill. It's one that we use every day with our clients at The Last Pickle, and one that relies on tools like the cassandra-stress. Knowing how the data model will perform under stress once it has been loaded with data can prevent expensive re-writes late in the project. In this talk Christopher Batey, Consultant at The Last Pickle, will shed some light on how to use the cassandra-stress tool to test your own schema, graph the results and even how to extend the tool for your own use cases. While this may be called premature optimisation for a RDBS, a successful Cassandra project depends on it's data model. About the Speaker Christopher Batey Consultant / Software Engineer, The Last Pickle Christopher (@chbatey) is a part time consultant at The Last Pickle where he works with clients to help them succeed with Apache Cassandra as well as a freelance software engineer working in London. Likes: Scala, Haskell, Java, the JVM, Akka, distributed databases, XP, TDD, Pairing. Hates: Untested software, code ownership. You can checkout his blog at: http://www.batey.info

1. Cassandra stress
Christopher Batey
@chbatey
The Last Pickle

4. Overview
● Importance of early realistic testing
● What is cassandra stress
● How you use it

5. Why do Cassandra
projects fail?

6. How cassandra projects roll
● New shiny project
● Cassandra is a good fit
● Build application against a single node cassandra cluster
● Go to production
● Fail

7. Cassandra is not an agile database
● Schema migrations :(
● Data migrations :(
● Must know queries up front

8. You must iterate on your schema early
● Know your queries
● Know your data
● Test them against a realistic cluster

9. Cassandra stress
cassandra-stress

10. Where
<install>/tools/bin/cassandra-stress
<install>/tools/bin/cassandra-stressd
<install>/tools/bin/compaction-stress

11. Resources

12. Commands
cassandra-stress help
---Commands---
read : Multiple concurrent reads - the cluster must first be populated by a write test
write : Multiple concurrent writes against the cluster
mixed : Interleaving of any basic commands
user : Interleaving of user provided queries, with configurable ratio and distribution
help : Print help for a command or option
10

13. Command help
cassandra-stress help write
Usage: write n=? [no-warmup] [truncate=?] [cl=?]
OR
Usage: write duration=? [no-warmup] [truncate=?] [cl=?]
err<? (default=0.02) Run until the standard error of the mean is below this fraction
n>? (default=30) Run at least this many iterations before accepting uncertainty convergence
n<? (default=200) Run at most this many iterations before accepting uncertainty convergence
no-warmup Do not warmup the process
truncate=? (default=never) Truncate the table: never, before performing any work, or before each iteration
cl=? (default=LOCAL_ONE) Consistency level to use
n=? Number of operations to perform
duration=? Time to run in (in seconds, minutes or hours)

14. Example
cassandra-stress write n=100 no-warmup truncate=once cl=ONE

15. Output
Op rate : 654 op/s [WRITE: 654 op/s]
Partition rate : 654 pk/s [WRITE: 654 pk/s]
Row rate : 654 row/s [WRITE: 654 row/s]
Latency mean : 59.5 ms [WRITE: 59.5 ms]
Latency median : 64.0 ms [WRITE: 64.0 ms]
Latency 95th percentile : 79.3 ms [WRITE: 79.3 ms]
Latency 99th percentile : 84.0 ms [WRITE: 84.0 ms]
Latency 99.9th percentile : 84.9 ms [WRITE: 84.9 ms]
Latency max : 84.9 ms [WRITE: 84.9 ms]
Total partitions : 100 [WRITE: 100]
Total errors : 0 [WRITE: 0]
Total GC count : 0
Total GC memory : 0.000 KiB
Total GC time : 0.0 seconds
Avg GC time : NaN ms
StdDev GC time : 0.0 ms
Total operation time : 00:00:00

16. Options
---Options---
-pop : Population distribution and intra-partition visit order
-insert : Insert specific options relating to various methods for batching and splitting partition
-rate : Thread count, rate limit or automatic mode (default is auto)
-mode : Thrift or CQL with options
-errors : How to handle errors when encountered during stress
-sample : Specify the number of samples to collect for measuring latency
-schema : Replication settings, compression, compaction, etc.
-node : Nodes to connect to
-log : Where to log progress to, and the interval at which to do it
-transport : Custom transport factories
-port : The port to connect to cassandra nodes on
-sendto : Specify a stress server to send this command to

17. Options help
cassandra-stress help -mode
Usage: -mode native [unprepared] cql3 [compression=?] [port=?] [user=?] [password=?] [auth-provider=?] [maxPending=?]
[connectionsPerHost=?] [protocolVersion=?]
user=? username
password=? password
unprepared force use of unprepared statements
compression=? (default=none)
port=? (default=9046)
auth-provider=? Fully qualified implementation of com.datastax.driver.core.AuthProvider
maxPending=? (default=) Maximum pending requests per connection
connectionsPerHost=? (default=) Number of connections per host
protocolVersion=? (default=NEWEST_SUPPORTED) CQL Protocol Version

18. User mode
● YAML profile

19. Scenario: tracking your staff
● Get all the activities for a staff member
● Get the latest event for a staff member
● Is Cassandra a good fit?

20. Defining your schema
table: staff_activities
table_definition: |
CREATE TABLE staff_activities (
name text,
when timeuuid,
what text,
PRIMARY KEY(name, when)
)

21. Column metadata
columnspec:
- name: name
size: uniform(5..10) # The names of the staff members are between 5-10 characters
population: uniform(1..10) # 10 possible staff members to pick from
- name: when
cluster: uniform(20..500) # Staff members do between 20 and 500 activities
- name: what
size: normal(10..100,50)

22. Column metadata
- name: name
size: uniform(5..10)
population: uniform(1..10)

23. Column metadata
- name: when
cluster: uniform(20..500)

24. Column metadata
- name: what
size: NORMAL(0..500,250,10)

25. Distributions
cassandra-stress print
Usage: print dist=DIST(?)
dist=DIST(?): A mathematical distribution
EXP(min..max)
EXTREME(min..max,shape)
QEXTREME(min..max,shape,quantas)
GAUSSIAN(min..max,stdvrng)
GAUSSIAN(min..max,mean,stdev)
UNIFORM(min..max)
FIXED(val)
SEQ(min..max)

26. Distributions
cassandra-stress print dist="UNIFORM(0..100)"
% of samples Range % of total
10.0 10 10.0
20.0 20 20.0
30.0 30 30.0
40.0 40 40.0
50.0 50 50.0
60.0 60 60.0
70.0 70 70.0
80.0 80 80.0
90.0 90 90.0
95.0 95 95.0
99.0 99 99.0
100.0 100 100.0

27. Distributions
cassandra-stress print dist="FIXED(100)"
% of samples Range % of total
10.0 100 100.0
20.0 100 100.0
30.0 100 100.0
40.0 100 100.0
50.0 100 100.0
60.0 100 100.0
70.0 100 100.0
80.0 100 100.0
90.0 100 100.0
95.0 100 100.0
99.0 100 100.0
100.0 100 100.0

28. Distributions
cassandra-stress print dist="FIXED(100)"
% of samples Range % of total
10.0 100 100.0
20.0 100 100.0
30.0 100 100.0
40.0 100 100.0
50.0 100 100.0
60.0 100 100.0
70.0 100 100.0
80.0 100 100.0
90.0 100 100.0
95.0 100 100.0
99.0 100 100.0
100.0 100 100.0

29. Distributions
cassandra-stress print "dist=NORMAL(0..500,250,10)"
% of samples Range % of total
10.0 237 47.4
20.0 241 48.2
30.0 244 48.8
40.0 247 49.4
50.0 250 50.0
60.0 252 50.4
70.0 255 51.0
80.0 258 51.6
90.0 262 52.4
95.0 266 53.2
99.0 273 54.6
100.0 500 100.0

30. Column metadata
columnspec:
- name: name
size: uniform(5..10) # The names of the staff members are between 5-10 characters
population: uniform(1..10) # 10 possible staff members to pick from
- name: when
cluster: uniform(20..500) # Staff members do between 20 and 500 activities
- name: what
size: normal(10..100,50)

31. Insertion of data
insert:
# we only update a single partition in any given insert
partitions: fixed(1)
# we want to insert a single row per partition and we have between 20 and 500
# rows per partition
select: fixed(1)/500
batchtype: UNLOGGED

32. Queries
queries:
events:
cql: select * from staff_activities where name = ?
latest_event:
cql: select * from staff_activities where name = ? LIMIT 1

33. Running
cassandra-stress user profile=./tlp-users.yaml
duration=1h
"ops(insert=1,latest_event=1,events=1)"
truncate=once
cl=LOCAL_QUORUM

34. Throughput mode (default)
● Keeps increasing the number of clients (threads) until no performance gain

35. Limit mode
● You specify a target TPS e.g. 10,000 writes / seconds
● Replaced with fixed/throttle mode in trunk

36. Graphing

37. Graphing

38. Graphing - adding to existing graph

39. Other features

40. Daemon mode
client

41. Overriding protocol version
cassandra-stress -mode native protocolVersion=X

42. Coordinated omission
● Limit mode replaced with throttle and fixed mode
http://psy-lob-saw.blogspot.co.uk/2016/07/fixing-co-in-cstress.html

43. Summary
● Not very flexible for inserting data
● No LWT support
● Distributed mode limited

44. Questions?
Christopher Batey
@chbatey
The Last Pickle

45. Coordinated omission

46. Coordinated omission

47. Coordinated omission