The document discusses the evaluation process for selecting Apache Cassandra as a database, emphasizing the importance of scalability, security, and good performance metrics. It outlines the steps taken during the assessment, including creating data models, deploying infrastructure, and testing for failure scenarios to ensure robustness. The findings and experiences learned throughout the testing phase are documented to aid in future database selections and project implementations.

1. Database Evaluation: Selecting Apache Cassandra

2. Introduction
2© 2015. All Rights Reserved.
@ChrisMonosmith
cmmonosmith@gmail.com
github.com/cmmonosmith

3. © 2015. All Rights Reserved. 3
Introduction
software engineer at vertafore, >6 years
gotten hands in almost every project in east lansing, enhancing or maintaining
attended some talks, 2014 cassandra summit, first time speaking outside of the office

4. Prelude
© 2015. All Rights Reserved. 4
first a little history

5. Prelude
© 2015. All Rights Reserved. 5
monolith farm
Big Oracle databases with VPDs, PL/SQL
WebLogic clusters, Large web applications

6. A New Adventure
© 2015. All Rights Reserved. 6

7. A New Adventure
© 2015. All Rights Reserved. 7
green field development
guidelines (next slides)

8. A New Adventure
8© 2015. All Rights Reserved.
0.001%
99.999%
Up Down
Modern goals for a modern system

9. A New Adventure
9© 2015. All Rights Reserved.
scalable, get it? but easy, intuitive

10. A New Adventure
10© 2015. All Rights Reserved.
maximum security, but not in a way that impedes us or our performance

11. A New Adventure
11© 2015. All Rights Reserved.
“money is irrelevant to the evaluation.”
money is always relevant…

12. Choose Your Party
© 2015. All Rights Reserved. 12
“where do we start?”
“find something to evaluate”

13. Choose Your Party
© 2015. All Rights Reserved. 13
etc…
limited exposure to nosql or non relational databases
Google “nosql”

14. Choose Your Party
© 2015. All Rights Reserved. 14
There are so many systems, and they all excel at everysomething
How do you choose what makes the cut?

15. Choose Your Party
Consider Your Data Model And Goals
© 2015. All Rights Reserved. 15
evaluation guidelines: good response times, good throughput
99th percentiles should also be “good”

16. Choose Your Party
© 2015. All Rights Reserved. 16
we care about entities and relationships

17. Choose Your Party
17© 2015. All Rights Reserved.
we also care about the history of these entities

18. Choose Your Party
© 2015. All Rights Reserved. 18
document stores and key-value stores were not on our keep list.
Scary blog posts about data modelling relationships with document stores like MongoDB

19. Choose Your Party
© 2015. All Rights Reserved. 19
document stores and key-value stores were not on our keep list.
Scary blog posts about data modelling relationships with document stores like MongoDB

20. Choose Your Party
© 2015. All Rights Reserved. 20
column stores and data abstraction layers looked to be worth our time

21. Choose Your Party
© 2015. All Rights Reserved. 21
from here we took a deeper dive into documentation
we eventually dropped hbase because it seemed to be for a different scale
we also eventually dropped datomic because it was very new

22. Choose Your Party
The Incumbent
© 2015. All Rights Reserved. 22
in a manner of speaking, everything would be measured against oracle

23. Level Select
© 2015. All Rights Reserved. 23

24. Level Select
Choose An Environment That Is Advantageous
© 2015. All Rights Reserved. 24

25. Level Select
25© 2015. All Rights Reserved.
as powerful a machine as this might be…
other processes, limited cores and memory
cluster with cassandra cluster manager can’t take advantage of optimized write path in cassandra due to extra disk seeks

26. Level Select
26© 2015. All Rights Reserved.
In-house Virtual machines
may or may not give you more flexibility, depending on who manages them

27. Level Select
27© 2015. All Rights Reserved.
the cloud. e.g. AWS, Microsoft Azure
we used one called Skytap
consider: operating systems, cores, memory, and network interfaces, security
cost: pay a little now to save a lot later (migration)

28. Level Select
Something You Can Control
© 2015. All Rights Reserved. 28
can’t know all activities up front, need root to mess with stuff

29. Level Select
Follow Documented Best Practices
Trust the Experts
© 2015. All Rights Reserved. 29
Setup Virtual Machines: OS/Kernel, JVM, Utilities
Again, factor in project requirements: Do you need a cluster?
there’s a reason the default cassandra config is what it is

30. Save Early and Often
© 2015. All Rights Reserved. 30

31. Save Early and Often
© 2015. All Rights Reserved. 31
Just good life advice
whether saving for retirement, saving a game, saving an essay, committing code…
also this project

32. Save Early and Often
© 2015. All Rights Reserved. 32
in OOP, developers gain efficiency through re-use. doesn’t stop at code
game developers re-use assets, like tree models, to make forests
what we did was use VM snapshots to build our world

33. Save Early and Often
© 2015. All Rights Reserved. 33
to do this, we needed help
part of building your world is knowing what it should look like
Knowledge on the business side helps here
Create projections for Customer base, Data volume
User profile variance (i.e. partition width in C* terms)
Analytics may or may not be obvious…
Keep reporting in the back of your mind

34. Save Early and Often
© 2015. All Rights Reserved. 34
Different databases will require independent models
you’ll have to work around unique limitations
Ask for help! DataStax helped us a lot
now let’s talk a little more about snapshots

35. Save Early and Often
© 2015. All Rights Reserved. 35
install your database

36. Save Early and Often
36© 2015. All Rights Reserved.

37. Save Early and Often
© 2015. All Rights Reserved. 37
Create your schema

38. Save Early and Often
38© 2015. All Rights Reserved.

39. Save Early and Often
© 2015. All Rights Reserved. 39
Build your cluster

40. Save Early and Often
40© 2015. All Rights Reserved.

41. Save Early and Often
© 2015. All Rights Reserved. 41
Build your data set. maybe encrypt it

42. Save Early and Often
42© 2015. All Rights Reserved.

43. Save Early and Often
© 2015. All Rights Reserved. 43
Add application servers

44. Save Early and Often
44© 2015. All Rights Reserved.

45. Save Early and Often
© 2015. All Rights Reserved. 45
Add stress test servers

46. Save Early and Often
46© 2015. All Rights Reserved.

47. Save Early and Often
© 2015. All Rights Reserved. 47
Add health monitoring services/servers

48. Save Early and Often
48© 2015. All Rights Reserved.

49. Save Early and Often
Do Anything Yourself At Most Once
© 2015. All Rights Reserved. 49
Replace manual steps with
Snapshots, obviously
Scripts
Small services or executables
Existing tools

50. Bring Your Gear
© 2015. All Rights Reserved. 50
tools

51. Bring Your Gear
Check Out The Jepsen Series
by Kyle Kingsbury
https://aphyr.com/tags/jepsen
© 2015. All Rights Reserved. 51
Jepsen series has lots of info and examples about different techs performance in terms of CAP theorem and data loss in failure/partition scenarios
very succinctly points out the difficulties of distributed systems

52. © 2015. All Rights Reserved. 52
microservice
Bring Your Gear
microservice
microservice
m
icroservice
microservice
We built a representative microservice prototype
One REST-ish API, many data layers
DataStax Java Driver
Oracle JDBC
HTTP endpoints
Creating representative data sets
Simulated user requests

53. Apache JMeter
Bring Your Gear
© 2015. All Rights Reserved. 53
A sample service needs sample clients
Used an existing tool: JMeter (introduce jmeter: load testing and performance)
Tools/automation team member built JMeter projects/test suites
Included a variety of load types: slow, bursty, firehose, read- vs write-heavy
important: Executable via command line
important: Wrote all results to disk

54. Bring Your Gear
© 2015. All Rights Reserved. 54
microservice
OpsCenter
Generate lots of useful data… need more tools
consistent format: microservice, jmeter opscenter
Do some math, write some CSVs
Find averages, mins, maxes…
Percentiles are great

55. Bring Your Gear
After a few minutes with Excel…
© 2015. All Rights Reserved. 55

56. Bring Your Gear
© 2015. All Rights Reserved. 56
these are some cassandra numbers
average and percentile tell a story together

57. Fight Some Dragons
57© 2015. All Rights Reserved.
This is the fun part

58. Fight Some Dragons
58© 2015. All Rights Reserved.

59. Fight Some Dragons
59© 2015. All Rights Reserved.
came in on a saturday. doesn’t happen often, outside of releases
partly because this was great fun
partly because it took so long… couldn’t run every test at once
also some anomalies… will get to that later

60. Fight Some Dragons
© 2015. All Rights Reserved. 60
have account for bad situations

61. Fight Some Dragons
© 2015. All Rights Reserved. 61
what happens when a party member gets knocked out?
drop a node during a test

62. Fight Some Dragons
© 2015. All Rights Reserved. 62
what happens when you resurrect the fallen comrade?
restore node in cluster during test

63. Fight Some Dragons
© 2015. All Rights Reserved. 63
what happens when a new fighter joins your party?
add an additional node
does it help, or just get in the way?

64. Fight Some Dragons
“Failure Is Always An Option”
- Adam Savage
© 2015. All Rights Reserved. 64
The Goal of every test is To Achieve Some Sort Of Failure,
or you’ve only learned part of the lesson it has to teach

65. Fight Some Dragons
© 2015. All Rights Reserved. 65
When will this thing topple over?
How does that compare to our volume/throughput projections?
Is there room for growth? Can we easily scale?
every db works to some extent, even using a simple text file as a database
remember “the goal is always to achieve some sort of failure”

66. Fight Some Dragons
Find Where It Fails Here,
Not In Production
© 2015. All Rights Reserved. 66
this will probably save you time and money and stress
and those Saturdays where you don’t get to just have fun
when you’ve done this, you’ve achieved a major goal of the evaluation

67. Collect Your Loot
© 2015. All Rights Reserved. 67
every good adventure ends with treasure

68. Collect Your Loot
© 2015. All Rights Reserved. 68
take good notes - on everything
design journal - documented our findings incrementally
ended up with tremendous volume

69. Collect Your Loot
© 2015. All Rights Reserved. 69
it’s like drawing a map
lets you retrace your steps, point out pitfalls, help out newcomers

70. Collect Your Loot
© 2015. All Rights Reserved. 70
a series of technical meetings and presentations
teaching is a great way to learn - taught about cassandra
lots of q&a with different people, DBAs, etc
now you have knowledge, metrics, charts… just line that up with your goals

71. Collect Your Loot
© 2015. All Rights Reserved. 71
Those metrics and charts were real “gems”
We brought our findings to management, and…
we work with very reasonable people

72. Easter Eggs
© 2015. All Rights Reserved. 72
During our testing, some of our VMs behaved strangely
Sporadic poor performance, slow startup, just fine later
We had the metrics to clearly illustrate this!
Hard to reproduce, but they eventually did
They pushed out a fix within a couple days
It never happened again after their fix

73. Easter Eggs
© 2015. All Rights Reserved. 73
teamwork!

74. Replay
Consider Your Data Model And Goals
© 2015. All Rights Reserved. 74
evaluation guidelines

75. Replay
Choose An Environment That Is Advantageous
© 2015. All Rights Reserved. 75

76. Replay
Follow Documented Best Practices
Trust the Experts
© 2015. All Rights Reserved. 76

77. Replay
Do Anything Yourself At Most Once
© 2015. All Rights Reserved. 77
snapshots! tools!

78. Replay
© 2015. All Rights Reserved. 78
“Failure Is Always An Option”
- Adam Savage
and find those failures before you deploy anything

79. Questions or Comments
© 2015. All Rights Reserved. 79

80. Thanks
80© 2015. All Rights Reserved.
Go see Robert Johnson!
robert johnson, online analytics processing

81. Thanks
81© 2015. All Rights Reserved.
We’re Hiring!
tinyurl.com/VertaforeEastLansing
we’re hiring

82. Thanks
82© 2015. All Rights Reserved.
@ChrisMonosmith
cmmonosmith@gmail.com
github.com/cmmonosmith
me me me

83. Thank you