I will talk about the reasons and our experience of migration of our main data storage from Cassandra to DynamoDB. Tips, tricks, pitfalls, and results.

1. Twilio
Alexander Malinovsky
Migration from Cassandra
to DynamoDB

2. © 2020 TWILIO INC. ALL RIGHTS RESERVED.
Conversations API
A single API for scalable, multiparty
conversations across channels
https://www.twilio.com/conversations-api

3. © 2020 TWILIO INC. ALL RIGHTS RESERVED.
High level architecture of Conversations API

4. © 2020 TWILIO INC. ALL RIGHTS RESERVED.
Current load
33 Cassandra boxes in production

5. © 2020 TWILIO INC. ALL RIGHTS RESERVED.
Reasons of selecting Cassandra as main data storage
High resiliency

6. © 2020 TWILIO INC. ALL RIGHTS RESERVED.
Reasons of selecting Cassandra as main data storage
Partitioning out of the box and easiness to scale up(for reads and for writes)

7. © 2020 TWILIO INC. ALL RIGHTS RESERVED.
Reasons of selecting Cassandra as main data storage
Knowledge and experience

8. © 2020 TWILIO INC. ALL RIGHTS RESERVED.
What’s wrong with Cassandra?
Big operational overhead

9. © 2020 TWILIO INC. ALL RIGHTS RESERVED.
What’s wrong with Cassandra?
Uneven partitions for big customers

10. © 2020 TWILIO INC. ALL RIGHTS RESERVED.
What’s wrong with Cassandra?
Tombstone issues

11. © 2020 TWILIO INC. ALL RIGHTS RESERVED.
What’s wrong with Cassandra?
Bugs
https://thelastpickle.com/blog/2018/07/05/undetectable-tombstones-in-apache-
cassandra.html

12. © 2020 TWILIO INC. ALL RIGHTS RESERVED.
What’s wrong with Cassandra?

13. © 2020 TWILIO INC. ALL RIGHTS RESERVED.
Which alternatives we considered

14. © 2020 TWILIO INC. ALL RIGHTS RESERVED.
How DynamoDB solves our problems
• Completely cloud solution. All maintenance on AWS
• Don’t need to worry about internals on how DynamoDB stores data, following generic
guidelines provides almost constant operation time:
• AWS support is very responsive and responds and solves issues pretty quickly

15. © 2020 TWILIO INC. ALL RIGHTS RESERVED.
How DynamoDB solves sharding issue
1. Use unique identifier as table’s partition key to avoid hot partitions and DynamoDB errors.
2. Use GSI(global secondary index) for creating secondary lookups.
3. Use caching layer for queries which requeire strong consistency.
https://www.dynamodbguide.com/global-secondary-indexes

16. © 2020 TWILIO INC. ALL RIGHTS RESERVED.
© 2020 TWILIO INC. ALL RIGHTS RESERVED.
Migration process
Dialog with product managers first!

17. © 2020 TWILIO INC. ALL RIGHTS RESERVED.
Table schema and secondary index keys

18. © 2020 TWILIO INC. ALL RIGHTS RESERVED.
Dualhead writes
Fail of insert in table fails whole
operation. Retry mechanism to prevent
intermittent hiccups.

19. © 2020 TWILIO INC. ALL RIGHTS RESERVED.
Data migration from Cassandra to DynamoDB
RCU – read capacity units
WCU – write capacity units
On Demand mode – no limit on RCU and WCU and price is based on amount of requests
Capacity control and scaling of DynamoDB tables

20. © 2020 TWILIO INC. ALL RIGHTS RESERVED.
Data migration from Cassandra to DynamoDB
• Script to migrate data from Cassandra to DynamoDB using Cassandra Token function mechanism
• @DynamoDBVersionAttribute and conflict resolution for existing records

21. © 2020 TWILIO INC. ALL RIGHTS RESERVED.
Data migration from Cassandra to DynamoDB
Script to check that all data in DynamoDB is actual with Cassandra
…

22. © 2020 TWILIO INC. ALL RIGHTS RESERVED.
Monitoring and switching off reads from Cassandra to DynamoDB
Feature flags for configuring reads and writes

23. © 2020 TWILIO INC. ALL RIGHTS RESERVED.
Monitoring and switching off reads from Cassandra to DynamoDB
Logging and rollbar as useful tool for catching invalid situations
https://rollbar.com/

24. © 2020 TWILIO INC. ALL RIGHTS RESERVED.
Monitoring and switching off reads from Cassandra to DynamoDB
Datadog and additional dashboards for understanding the whole picture and configuring DDB
provisioned capacity
https://www.datadoghq.com/

25. © 2020 TWILIO INC. ALL RIGHTS RESERVED.
Switching off the writes and cleaning up the resources
Refactoring of DAOs to be able to make the switch staged

26. © 2020 TWILIO INC. ALL RIGHTS RESERVED.
Switching off the writes and cleaning up the resources
Switching of the writes
Removing unneeded tables and potential scale down

27. © 2020 TWILIO INC. ALL RIGHTS RESERVED.
© 2020 TWILIO INC. ALL RIGHTS RESERVED.
Outcomes
No costs on DB maintenance, more time on feature development
1. DB maintanance(security fixes, boxes redeployment)
2. DB backups
3. Additional services to maintain
SLA is still the same 99,99%

28. © 2020 TWILIO INC. ALL RIGHTS RESERVED.
© 2020 TWILIO INC. ALL RIGHTS RESERVED.
Outcomes
More granular scaling approach
Costs reduction in 5-6 times

29. © 2020 TWILIO INC. ALL RIGHTS RESERVED.
© 2020 TWILIO INC. ALL RIGHTS RESERVED.
Outcomes
Less tables to maintain in DB which improved consistency
From 23 tables in Cassandra to 12 tables in DynamoDB

30. © 2020 TWILIO INC. ALL RIGHTS RESERVED.
© 2020 TWILIO INC. ALL RIGHTS RESERVED.
Q&A
malinovskiyalex13@gmail.com