Disney+ Hotstar is the fastest growing branch of Disney+. Join Disney+ Hotstar Architect Vamsi Subhash and senior data engineer Balakrishnan Kaliyamoorthy to learn… How Disney+ Hotstar architected their systems to handle massive data loads Why they chose to replace both Redis and Elasticsearch Their requirements for massively scalable data infrastructure and evolving data models How they migrated their data to Scylla Cloud, ScyllaDB’s fully managed NoSQL database-as-a-service, without suffering downtime

1. Balakrishnan Kaliyamoorthy
Vamsi Subhash Achanta
Disney+ Hotstar: Scaling
NoSQL for Millions of
Video On-Demand Users

2. About Disney+ Hotstar
https://en.wikipedia.org/wiki/Hotstar
- Disney+ Hotstar is an Indian subscription video on-demand streaming service
- By 2019, we had over 300 million active users monthly
- World record of concurrent viewership: 25.3 million

3. Use Case: Continue Watching

4. Use Case: Continue Watching

5. Use Case: Continue Watching
WEB
IOS

6. Use Case: Continue Watching

7. Use Case: Continue Watching

8. Pre-Scylla Architecture
Event
processor
API server
Send watch
event
Client
Kafka
Redis Elasticsearch
Process event
Save to Data-stores
Buffer read from
Data-stores
Serve response
(500GB) (20TB)
Key: 30 bytes
Value: 5-10 KBs

9. Data Model
● Redis
● Elasticsearch
User-Id_Content-Id: {
“Field 1” : …
“Field 2” : …
“Field 3” : …
“Field 4” : …
}
User-Id List<Content-Ids>

10. Challenges Faced
Multiple data-stores and data-models
Huge data in order of
TBs
Expensive

11. New Data Model
● User Table
● User-Content Table
User-Id (PK) List<Content-Id>
User-Id (PK) Content-Id (SK) Timestamp Field 3 Field 4

12. Possible Candidates

13. Why Scylla ?
● Less cost for writes
● Low latency (writes + reads) given the same throughput

14. Why Scylla ?
● Less cost for writes
● Low latency

15. Migrating to Scylla
● Redis to Scylla
● Elastic-search to Scylla
● Scylla open-source to Scylla cloud

16. Redis to ScyllaDB Migration
RDB file
Redis
Take Redis
Snapshot
Convert to
CSV
Use
COPY <table> FROM
<csv-file>
WITH DELIMITER ‘,’
CHUNKSIZE=1

17. Cutover
Processor API Server
Write to
Redis
Write to
Elasticsearch
Read from Redis
Read from ES
Write to
Scylla
Read from Scylla

18. Scylla OSS to Cloud
● SSTableloader
● Spark migrator

19. SSTableloader Based Migration
Scylla OSS Scylla DB
Snapshot
Scylla OSS Scylla DB
Snapshot Scylla cloud
Scylla OSS Scylla DB
Snapshot
nodetool
snapshot
<table> -t
"snapshot1"
nodetool
snapshot
<table> -t
"snapshot2"
nodetool
snapshot
<table> -t
"snapshot2"
sstableloader -d
<host-addresss>
<auth>
snapshot1
sstableloader -d
<host-addresss>
<auth>
snapshot2
sstableloader -d
<host-addresss>
<auth>
snapshot3

20. SSTableloader Based Migration
● Cons
○ SSTable migration slowed down when we have a secondary / composite key.

21. Spark Scylla Migrator
Scylla
opensource
New single-node
Scylla opensource
Backup snapshot to
S3
Take Scylla backups
and move to S3
Restore the backup
In new scylla
open-source cluster
Read via spark
Migrate data to
Scylla-cloud cluster
Scylla cloud
Write to
Scylla cloud
Unirestore

22. Recommendations
● Create new cluster in order to avoid any down-time / impact during spark based
migration.
● Increase parallelism until destination load reaches 100% and we don’t see any write
timeouts.
● Blog : https://www.scylladb.com/2019/03/12/deep-dive-into-the-scylla-spark-migrator/

23. Future plans
● We are looking to move more use cases to Scylla cloud cluster instead of our existing
DBs.
● Use-cases :
○ Watchlist - bucket where user saves content he likes to watch-later
○ Offline recommendation metadata

24. User Watch Store
● Time Series
● Append only writes
● Aggregate on reads
● Expiration
W R

25. User Watch Store

26. User Watch Store
● Time range query on user’s watch
○ Aggregate duplicated watches (watch time / video position / etc.)
○ Expiration of the watches (in some time window)
● Near real-time serving
○ Volume: 1.5M rpm writes at peak
○ API serving online requests

27. Issues at Scale
● P50 > 1s!

28. Issues at Scale

29. Discovered Anti-patterns

30. Discovered Anti-patterns

31. Compactions

32. Lessons Learned

33. Mitigation
● Avoid Queue Like Anti-pattern
○ Exclude tombstone with time range in query
○ Cache friendly with more static time range
Remember the split point,
Query from the it!
UTC
00:00

34. Optimizations: Compaction
● Use consistency to CL=ALL for deletes
● Set GC_GRACE_SECONDS to 0
● Schedule daily once major compactions during off peak hours -> to reduce
tombstones without latency impact during the day time.

35. United States
2445 Faber St, Suite #200
Palo Alto, CA USA 94303
Israel
Maskit 4
Herzliya, Israel 4673304
www.scylladb.com
@scylladb
Thank You!