Ben Bromhead is the co-founder and CTO of Instaclustr, which provides Cassandra-as-a-Service. Instaclustr manages 50+ Cassandra nodes for customers. Early on, Instaclustr encountered issues like a Cassandra bug causing assertion errors for large column names and had to perform an emergency migration for a customer whose self-managed cluster was down for 48 hours. Migrations and real-world usage revealed new challenges compared to initial perfect test scenarios.

1. From the front lines
Saving stranded clusters
#CassandraSummit

2. Who am I and what do I do?
• Ben Bromhead
• Co-founder and CTO of Instaclustr -> www.instaclustr.com
• Instaclustr provides Cassandra-as-a-Service in the cloud.
• Currently in AWS, Google Cloud in private beta with more to come.
• We currently manage 50+ nodes for various customers, who do
various things with it.

3. In the beginning…
• Well designed schemas
• No data migrations
• Everything was perfect and happy
• Then we got customers

4. Our first C* patch
• CASSANDRA-6521
• Cassandra wouldn’t check the length of a column name in a range
predicate for slice operations.
• So for large column names it would throw an assertion error.
• Which would in turn tie up threads, causing the node to be
unresponsive and eventually the whole cluster.

5. Our first C* patch
• What was the size of the column name that would cause this issue?
• Around 130kb
• wat…

6. Our first migration
• Receive frantic phone call
• Self managed cluster has been down for 48 hours, for a company
that gets 25 million monthly unique views.
• They are hurting

7. Our first migration
• The cluster was running a very early version of C* 2.0
• Update/patch the old cluster, get everything back online
• Start the migration process…

8. Our first migration
• Bulkload manages to kill their new cluster with us in about 5
minutes.
• Open logs
• Read 1 live and 38456 tombstoned cells (see
tombstone_warn_threshold)
• For every column family
• wat…

9. Conclusion
• Everything is awesome
• Then reality occurs
• It’s actually way more fun
• Want to make C* even better? We are hiring!