Hailo has leveraged Cassandra to build one of the most successful startups in European history. This presentations looks at how Hailo grew from a simple MySQL-backed infrastructure to a resilient Cassandra-backed system running in three data centers globally. Topics covered include: the process of migration, experience running multi-DC on AWS, common data modeling patterns and security implications for achieving PCI compliance.

1. #CASSANDRA13
Cassandra at Hailo
David Gardner | Architect @ Hailo
CASSANDRASUMMIT2013

2. #CASSANDRA13 CASSANDRASUMMIT2013

3. #CASSANDRA13 CASSANDRASUMMIT2013
What is this talk about?

4. #CASSANDRA13 CASSANDRASUMMIT2013

5. #CASSANDRA13 CASSANDRASUMMIT2013

6. #CASSANDRA13 CASSANDRASUMMIT2013
• 1,352 changed files with 235,413 additions and 47,487 deletions
• 7,429 commits
• 1,653 tickets completed
https://github.com/apache/cassandra/compare/cassandra-0.6.0...cassandra-1.2
https://github.com/apache/cassandra/blob/trunk/CHANGES.txt
0.6 to 1.2

7. #CASSANDRA13 CASSANDRASUMMIT2013
Cassandra adoption at Hailo from three perspectives:
1. Development
2. Operational
3. Management
What this talk is about

8. #CASSANDRA13 CASSANDRASUMMIT2013
What is Hailo?
Hailo is the taxi app. Use Hailo to get a taxi wherever you are, whenever you want.

9. #CASSANDRA13 CASSANDRASUMMIT2013

10. #CASSANDRA13 CASSANDRASUMMIT2013
• The world’s highest-rated taxi app - over 7,000 five-star reviews
• Over 300,000 registered passengers
• A Hailo hail is accepted around the world every 5 seconds
• Hailo is growing (30%+) every month
• Became the largest taxi network in all of Ireland within two months
of launch
What is Hailo?

11. #CASSANDRA13 CASSANDRASUMMIT2013
The history
The story behind Cassandra adoption at Hailo

12. #CASSANDRA13 CASSANDRASUMMIT2013
Hailo launched in London in November 2011
• Launched on AWS
• Two PHP/MySQL web apps plus a Java backend
• Mostly built by a team of 3 or 4 backend engineers
• MySQL multi-master for single AZ resilience

13. #CASSANDRA13 CASSANDRASUMMIT2013
Why Cassandra?
• A desire for greater resilience – “become a utility”
Cassandra is designed for high availability
• Plans for international expansion around a single consumer app
Cassandra is good at global replication
• Expected growth
Cassandra scales linearly for both reads and writes
• Prior experience
I had experience with Cassandra and could recommend it

14. #CASSANDRA13 CASSANDRASUMMIT2013
The path to adoption
• Largely unilateral decision by developers – a result of a startup
culture
• Replacement of key consumer app functionality, splitting up the
PHP/MySQL web app into a mixture of global PHP/Java services
backed by a Cassandra data store
• Launched into production in XYZ– originally just powering North
American expansion, before gradually switching over Dublin and
London

15. #CASSANDRA13 CASSANDRASUMMIT2013
Development perspective

16. #CASSANDRA13 CASSANDRASUMMIT2013
“Cassandra just works”
Dom Wong, Senior Engineer

17. #CASSANDRA13 CASSANDRASUMMIT2013
Use cases
1. Entity storage
2. Time series data

18. #CASSANDRA13 CASSANDRASUMMIT2013
CF = customers
126007613634425612:
createdTimestamp: 1370465412
email: dave@cruft.co
givenName: Dave
familyName: Gardner
locale: en_GB
phone: +447911111111

19. #CASSANDRA13 CASSANDRASUMMIT2013
Considerations for entity storage
• Do not read the entire entity, update one property and then write
back a mutation containing every column
• Only mutate columns that have been set
• This avoids read-before-write race conditions

20. #CASSANDRA13 CASSANDRASUMMIT2013

21. #CASSANDRA13 CASSANDRASUMMIT2013

22. #CASSANDRA13 CASSANDRASUMMIT2013
CF = comms
2013-06-01:
55374fa0-ce2b-11e2-8b8b-0800200c9a66: {“to”:”dave@c…
a48bd800-ce2b-11e2-8b8b-0800200c9a66: {“to”:”foo@ex…
b0e15850-ce2b-11e2-8b8b-0800200c9a66: {“to”:”bar@ho …
bfac6c80-ce2b-11e2-8b8b-0800200c9a66: {“to”:”baz@fo…

23. #CASSANDRA13 CASSANDRASUMMIT2013
CF = comms
dave@cruft.co:
13b247f0-ce2c-11e2-8b8b-0800200c9a66: {“to”:”dave@c…
20f70a40-ce2c-11e2-8b8b-0800200c9a66: {“to”:”dave@c…
2b44d3b0-ce2c-11e2-8b8b-0800200c9a66: {“to”:”dave@c…
338a22f0-ce2c-11e2-8b8b-0800200c9a66: {“to”:”dave@c…

24. #CASSANDRA13 CASSANDRASUMMIT2013

25. #CASSANDRA13 CASSANDRASUMMIT2013

26. #CASSANDRA13 CASSANDRASUMMIT2013
Considerations for time series storage
• Choose row key carefully, since this partitions the records
• Think about how many records you want in a single row
• Denormalise on write into many indexes

27. #CASSANDRA13 CASSANDRASUMMIT2013
Client libraries
• Astyanax (Java)
• phpcassa (PHP)
• github.com/carloscm/gossie (Go)

28. #CASSANDRA13 CASSANDRASUMMIT2013
Analytics
• With Cassandra we lost the ability to carry out analytics
eg: COUNT, SUM, AVG, GROUP BY
• We use Acunu Analytics to give us this abilty in real time, for pre-
planned query templates
• It is backed by Cassandra and therefore highly available, resilient
and globally distributed
• Integration is straightforward

29. #CASSANDRA13 CASSANDRASUMMIT2013
AQL
SELECT
SUM(accepted),
SUM(ignored),
SUM(declined),
SUM(withdrawn)
FROM Allocations
WHERE timestamp BETWEEN '1 week ago' AND 'now’
AND driver='LON123456789’
GROUP BY timestamp(day)

30. #CASSANDRA13 CASSANDRASUMMIT2013

31. #CASSANDRA13 CASSANDRASUMMIT2013

32. #CASSANDRA13 CASSANDRASUMMIT2013
Challenges

33. #CASSANDRA13 CASSANDRASUMMIT2013
10 Average years experience
per team member
MySQL Cassandra

34. #CASSANDRA13 CASSANDRASUMMIT2013

35. #CASSANDRA13 CASSANDRASUMMIT2013
Lessons learned

36. #CASSANDRA13 CASSANDRASUMMIT2013
Have an advocate
• Get someone who will sell the vision internally
• Make an effort to get everyone on board

37. #CASSANDRA13 CASSANDRASUMMIT2013
Learn the theory
• Teach each team member the fundamentals
• CQL can encourage an SQL mindset, but it’s important to
understand the underlying data model
• Make a real effort to share knowledge – keep in mind the gulf in
experience for most team members between their old world and the
new world (SQL vs NoSQL)
• Peer review data models

38. #CASSANDRA13 CASSANDRASUMMIT2013
Operational perspective

39. #CASSANDRA13 CASSANDRASUMMIT2013
“Allows a team of 2 to achieve things they wouldn’t
have considered before Cassandra existed”
Chris Hoolihan, Operations Engineer

40. #CASSANDRA13 CASSANDRASUMMIT2013

41. #CASSANDRA13 CASSANDRASUMMIT2013
2 clusters
6 machines per region
3 regions
(stats cluster pending addition
of third DC)
Operational
Cluster
Stats
Cluster
ap-southeast-1 us-east-1 eu-west-1
us-east-1 eu-west-1

42. #CASSANDRA13 CASSANDRASUMMIT2013
AWS VPCs with Open
VPN links
3 AZs per region
m1.large machines
Provisoned IOPS EBS
Operational
Cluster
Stats
Cluster
~ 600GB/node
~ 100GB/node

43. #CASSANDRA13 CASSANDRASUMMIT2013
Backups
• SSTable snapshot
• Used to upload to S3, but this was taking >6 hours and consuming
all our network bandwidth
• Now take EBS snapshot of the SSTable snapshots

44. #CASSANDRA13 CASSANDRASUMMIT2013
Encryption
• Requirement for NYC launch
• We use dmcrypt to encrypt the entire EBS volume
• Chose dmcrypt because it is uncomplicated
• Our tests show a 1% performance hit in disk performance, which
concurs with what Amazon suggest

45. #CASSANDRA13 CASSANDRASUMMIT2013
Datastax Ops Centre
• We run the free version
• Offers up easily accessible “one screen” overviews of the activity of
the entire cluster
• Big fans – an easy win

46. #CASSANDRA13 CASSANDRASUMMIT2013

47. #CASSANDRA13 CASSANDRASUMMIT2013
Multi DC
• Something that Cassandra makes trivial
• Would have been very difficult to accomplish active-active inter-DC
replication with a team of 2 without Cassandra
• Rolling repair needed to make it safe (we use LOCAL_QUORUM)
• We schedule “narrow repairs” on different nodes in our cluster each
night

48. #CASSANDRA13 CASSANDRASUMMIT2013
Compression
• Our stats cluster was running at ~1.5TB per node
• We didn’t want to add more nodes
• With compression, we are now back to ~600GB
• Easy to accomplish
• `nodetool upgradesstables` on a rolling schedule

49. #CASSANDRA13 CASSANDRASUMMIT2013
Lessons learned

50. #CASSANDRA13 CASSANDRASUMMIT2013

51. #CASSANDRA13 CASSANDRASUMMIT2013
Management perspective

52. #CASSANDRA13 CASSANDRASUMMIT2013
“The days of the quick and dirty are over”
Simon Veingard, EVP Operations

53. #CASSANDRA13 CASSANDRASUMMIT2013
Technically, everything is fine…
• Our COO feels that C* is “technically good and beautiful”, a
“perfectly good option”
• Our EVPO says that C* reminds him of a time series database in
use at Goldman Sachs that had “very good performance”
…but there are concerns

54. #CASSANDRA13 CASSANDRASUMMIT2013
People who can
attempt to query
MySQL
People who can
attempt to
query Cassandra

55. #CASSANDRA13 CASSANDRASUMMIT2013

56. #CASSANDRA13 CASSANDRASUMMIT2013
Lessons learned

57. #CASSANDRA13 CASSANDRASUMMIT2013
Keep the business informed
• Pre-launch, we were tasked with increasing resiliency
• Cassandra addressed immediate business needs, but the trade offs
involved should have been communicated more clearly

58. #CASSANDRA13 CASSANDRASUMMIT2013
Sing from the same hymn sheet
• A senior founding engineer had doubts about the adoption of
Cassandra until very recently
• In the presence of business doubt, this lack of consistency
amongst developers exacerbated the concerns
• We should have made more effort to make bilateral decisions on
adoption – I don’t think this would have been hard to achieve

59. #CASSANDRA13 CASSANDRASUMMIT2013

60. #CASSANDRA13 CASSANDRASUMMIT2013
Provide solutions
• There are many options for ad-hoc querying of Cassandra
• We underestimated the impact of not having a good solution for
this from the very beginning

61. #CASSANDRA13 CASSANDRASUMMIT2013
The future

62. #CASSANDRA13 CASSANDRASUMMIT2013
Cassandra at Hailo
• We will continue to invest in Cassandra as we expand globally
• We will hire people with experience running Cassandra
• We will focus on expanding our reporting facilities

63. #CASSANDRA13 CASSANDRASUMMIT2013

64. #CASSANDRA13
Thank you
CASSANDRASUMMIT2013