Downloaded 14 times
Uploaded byDataStax Academy
C* Summit EU 2013: No Whistling Required: Cabs, Cassandra, and Hailo
The document discusses the adoption of Cassandra by Hailo, a leading taxi app, detailing its motivations, implementation, and outcomes from development, operational, and management perspectives. Hailo chose Cassandra for its high availability, resilience, and ability to scale internationally, which has significantly aided their growth and operational efficiency. The talk emphasizes the importance of having advocates for technology adoption, understanding fundamental principles of usage, and maintaining communication with management about trade-offs.
More Related Content
Similar to C* Summit EU 2013: No Whistling Required: Cabs, Cassandra, and Hailo
![Coded Agents – with UiPath SDK + LangGraph [Virtual Hands-on Workshop]](https://cdn.slidesharecdn.com/ss_thumbnails/codedagentsdeck-251215155422-5497c599-thumbnail.jpg?width=640&height=640&fit=bounds)
C* Summit EU 2013: No Whistling Required: Cabs, Cassandra, and Hailo
- 1. Cabs, Cassandra, andHailo David Gardner, Architect at Hailo #CASSANDRAEU CASSANDRASUMMITEU
- 2.
- 3.
- 4. 0.6 to 1.2 • 1,352 changed files with 235,413 additions and 47,487 deletions • 7,429 commits • 1,653 tickets completed https://github.com/apache/cassandra/compare/cassandra-0.6.0...cassandra-1.2 https://github.com/apache/cassandra/blob/trunk/CHANGES.txt #CASSANDRAEU CASSANDRASUMMITEU
- 5. What this talkis about Cassandra adoption at Hailo from three perspectives: 1. Development 2. Operational 3. Management #CASSANDRAEU CASSANDRASUMMITEU
- 6. What is Hailo? Hailois The Taxi Magnet. Use Hailo to get a cab wherever you are, whenever you want. #CASSANDRAEU CASSANDRASUMMITEU
- 7.
- 8.
- 9.
- 10. What is Hailo? • The world’s highest-rated taxi app – over 11,000 five-star reviews • Over 500,000 registered passengers • A Hailo hail is accepted around the world every 4 seconds • Hailo operates in 15 cities on 3 continents from Tokyo to Toronto in nearly 2 years of operation #CASSANDRAEU CASSANDRASUMMITEU
- 11. Hailo is growing • Hailo is a marketplace that facilitates over $100M in run-rate transactions and is making the world a better place for passengers and drivers • Hailo has raised over $50M in financing from the world's best investors including Union Square Ventures, Accel, the founder of Skype (via Atomico), Wellington Partners (Spotify), Sir Richard Branson, and our CEO's mother, Janice #CASSANDRAEU CASSANDRASUMMITEU
- 12. The history The storybehind Cassandra adoption at Hailo #CASSANDRAEU CASSANDRASUMMITEU
- 13. Hailo launched inLondon in November 2011 • Launched on AWS • Two PHP/MySQL web apps plus a Java backend • Mostly built by a team of 3 or 4 backend engineers • MySQL multi-master for single AZ resilience #CASSANDRAEU CASSANDRASUMMITEU
- 14. Why Cassandra? • Adesire for greater resilience – “become a utility” Cassandra is designed for high availability • Plans for international expansion around a single consumer app Cassandra is good at global replication • Expected growth Cassandra scales linearly for both reads and writes • Prior experience I had experience with Cassandra and could recommend it #CASSANDRAEU CASSANDRASUMMITEU
- 15. The path toadoption • Largely unilateral decision by developers – a result of a startup culture • Replacement of key consumer app functionality, splitting up the PHP/MySQL web app into a mixture of global PHP/Java services backed by a Cassandra data store • Launched into production in September 2012 – originally just powering North American expansion, before gradually switching over Dublin and London #CASSANDRAEU CASSANDRASUMMITEU
- 16. One year on... • Further breakdown of functionality into Go/Java SOA • Migrating all online databases to Cassandra #CASSANDRAEU CASSANDRASUMMITEU
- 17.
- 18. “Cassandra just works” DomW, Senior Engineer #CASSANDRAEU CASSANDRASUMMITEU
- 19. Use cases 1. Entitystorage 2. Time series data #CASSANDRAEU CASSANDRASUMMITEU
- 20. CF = customers 126007613634425612: createdTimestamp: email: givenName: familyName: locale: phone: #CASSANDRAEU 1370465412 dave@cruft.co Dave Gardner en_GB +447911111111 CASSANDRASUMMITEU
- 21. Considerations for entitystorage • Do not read the entire entity, update one property and then write back a mutation containing every column • Only mutate columns that have been set • This avoids read-before-write race conditions #CASSANDRAEU CASSANDRASUMMITEU
- 22.
- 23. CF = stats_db 2013-06-01: 55374fa0-ce2b-11e2-8b8b-0800200c9a66: a48bd800-ce2b-11e2-8b8b-0800200c9a66: b0e15850-ce2b-11e2-8b8b-0800200c9a66: bfac6c80-ce2b-11e2-8b8b-0800200c9a66: #CASSANDRAEU {“action”:”… {“action”:”… {“action”:”… {“action”:”… CASSANDRASUMMITEU
- 24. CF = stats_db LON123456: 13b247f0-ce2c-11e2-8b8b-0800200c9a66: 20f70a40-ce2c-11e2-8b8b-0800200c9a66: 2b44d3b0-ce2c-11e2-8b8b-0800200c9a66: 338a22f0-ce2c-11e2-8b8b-0800200c9a66: #CASSANDRAEU {“action”:”… {“action”:”… {“action”:”… {“action”:”… CASSANDRASUMMITEU
- 25.
- 26. Considerations for timeseries storage • Choose row key carefully, since this partitions the records • Think about how many records you want in a single row • Denormalise on write into many indexes #CASSANDRAEU CASSANDRASUMMITEU
- 27. Client libraries • Gossie(Go) • Astyanax (Java) • phpcassa (PHP) #CASSANDRAEU CASSANDRASUMMITEU
- 28. Analytics • With Cassandrawe lost the ability to carry out analytics eg: COUNT, SUM, AVG, GROUP BY • We use Acunu Analytics to give us this abilty in real time, for preplanned query templates • It is backed by Cassandra and therefore highly available, resilient and globally distributed • Integration is straightforward #CASSANDRAEU CASSANDRASUMMITEU
- 29.
- 30. AQL SELECT SUM(accepted), SUM(ignored), SUM(declined), SUM(withdrawn) FROM Allocations WHERE timestampBETWEEN '1 week ago' AND 'now’ AND driver='LON123456789’ GROUP BY timestamp(day) #CASSANDRAEU CASSANDRASUMMITEU
- 31.
- 32.
- 33. “Allows a teamof 2 to achieve things they wouldn’t have considered before Cassandra existed” Chris H, Operations Engineer #CASSANDRAEU CASSANDRASUMMITEU
- 34.
- 35. us-east-1 eu-west-1 us-east-1 eu-west-1 Operational Cluster 6 machines perregion Stats Cluster 3 clusters 3 regions (stats cluster is a long story) ap-southeast-1 #CASSANDRAEU CASSANDRASUMMITEU
- 36.
- 37. 3 AZs perregion m1.large machines ~ 1TB/node Stats Cluster AWS VPCs with Open VPN links Provisoned IOPS EBS #CASSANDRAEU Operational Cluster ~ 200GB/node CASSANDRASUMMITEU
- 38. Backups • SSTable snapshot • Used to upload to S3, but this was taking >6 hours and consuming all our network bandwidth • Now take EBS snapshot of the data volumes #CASSANDRAEU CASSANDRASUMMITEU
- 39. Encryption • Requirement forNYC launch • We use dmcrypt to encrypt the entire EBS volume • Chose dmcrypt because it is uncomplicated • Our tests show a 1% performance hit in disk performance, which concurs with what Amazon suggest #CASSANDRAEU CASSANDRASUMMITEU
- 40. Datastax Ops Centreis a quick win #CASSANDRAEU CASSANDRASUMMITEU
- 41. Multi DC • Somethingthat Cassandra makes trivial • Would have been very difficult to accomplish active-active inter-DC replication with a team of 2 without Cassandra • Rolling repair needed to make it safe (we use LOCAL_QUORUM) • We schedule “narrow repairs” on different nodes in our cluster each night #CASSANDRAEU CASSANDRASUMMITEU
- 42. Compression • Our statscluster was running at ~1.5TB per node • We didn’t want to add more nodes • With compression, we are now back to ~600GB • Easy to accomplish • `nodetool upgradesstables` on a rolling schedule #CASSANDRAEU CASSANDRASUMMITEU
- 43.
- 44. “The days ofthe quick and dirty are over” Simon V, EVP Operations #CASSANDRAEU CASSANDRASUMMITEU
- 45. Technically, everything isfine… • Our COO feels that C* is “technically good and beautiful”, a “perfectly good option” • Our EVPO says that C* reminds him of a time series database in use at Goldman Sachs that had “very good performance” …but there are concerns #CASSANDRAEU CASSANDRASUMMITEU
- 46. People who can attemptto query MySQL People who can attempt to query Cassandra #CASSANDRAEU CASSANDRASUMMITEU
- 47.
- 48.
- 49. There might bea gulf in experience #CASSANDRAEU CASSANDRASUMMITEU
- 50. 10 Average years experience perteam member MySQL Cassandra #CASSANDRAEU CASSANDRASUMMITEU
- 51. Lesson learned • Havean advocate - get someone who will sell the vision internally • Learn the theory - teach each team member the fundamentals • Make an effort to get everyone on board #CASSANDRAEU CASSANDRASUMMITEU
- 52. Things can driftinto failure #CASSANDRAEU CASSANDRASUMMITEU
- 53.
- 54.
- 55.
- 56.
- 57.
- 58. Lesson learned • Bepro-active with Cassandra, even if it seems to be running smoothly • Peer-review data models, take time to think about them • Big rows are bad - use cfstats to look for them • Mixed workloads can cause problems - use cfhistograms and look out for signs of data modeling problems • Think about the compaction strategy for each CF #CASSANDRAEU CASSANDRASUMMITEU
- 59.
- 60. Lessons learned • EBSis nearly always the cause of Amazon outages • EBS is a single point of failure (it will fail everywhere in your cluster) • EBS is slow • EBS is expensive • EBS is unnecessary! #CASSANDRAEU CASSANDRASUMMITEU
- 61. Management need toknow the trade offs #CASSANDRAEU CASSANDRASUMMITEU
- 62. Lessons learned • Keepthe business informed – explain the tradeoffs in simple terms • Sing from the same hymn sheet • Make sure there solutions in place for every use case from the beginning #CASSANDRAEU CASSANDRASUMMITEU
- 63. People who can attemptto query MySQL #CASSANDRAEU People who can attempt to query Cassandra CASSANDRASUMMITEU
- 64.
- 65. We like Cassandra • Solid design • HA characteristics • Easy multi-DC setup • Simplicity of operation #CASSANDRAEU CASSANDRASUMMITEU
- 66. Lessons for successfuladoption • Have an advocate, sell the dream • Learn the fundamentals, get the best out of Cassandra • Invest in tools to make life easier • Keep management in the loop, explain the trade offs #CASSANDRAEU CASSANDRASUMMITEU
- 67. The future • Wewill continue to invest in Cassandra as we expand globally • We will hire people with experience running Cassandra • We will focus on expanding our reporting facilities • We aspire to extend our network (1M consumer installs, wallet) beyond cabs • We will continue to hire the best engineers in London, NYC and Asia #CASSANDRAEU CASSANDRASUMMITEU
- 68.