Disney+ Hotstar: Scaling NoSQL for Millions of Video On-Demand Users

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Balakrishnan Kaliyamoorthy
Vamsi Subhash Achanta
Disney+ Hotstar: Scaling
NoSQL for Millions of
Video On-Demand Users

About Disney+ Hotstar
https://en.wikipedia.org/wiki/Hotstar
- Disney+ Hotstar is an Indian subscription video on-demand st...

Use Case: Continue Watching

Use Case: Continue Watching

Use Case: Continue Watching
WEB
IOS

Use Case: Continue Watching

Use Case: Continue Watching

Pre-Scylla Architecture
Event
processor
API server
Send watch
event
Client
Kafka
Redis Elasticsearch
Process event
Save to...

$Data Model ● Redis ● Elasticsearch User-Id_Content-Id: { “Field 1” : … “Field 2” : … “Field 3” : … “Field 4” : … } User-Id...$

Challenges Faced
Multiple data-stores and data-models
Huge data in order of
TBs
Expensive

New Data Model
● User Table
● User-Content Table
User-Id (PK) List<Content-Id>
User-Id (PK) Content-Id (SK) Timestamp Fiel...

Possible Candidates

Why Scylla ?
● Less cost for writes
● Low latency (writes + reads) given the same throughput

Why Scylla ?
● Less cost for writes
● Low latency

Migrating to Scylla
● Redis to Scylla
● Elastic-search to Scylla
● Scylla open-source to Scylla cloud

Redis to ScyllaDB Migration
RDB file
Redis
Take Redis
Snapshot
Convert to
CSV
Use
COPY <table> FROM
<csv-file>
WITH DELIMI...

Cutover
Processor API Server
Write to
Redis
Write to
Elasticsearch
Read from Redis
Read from ES
Write to
Scylla
Read from ...

Scylla OSS to Cloud
● SSTableloader
● Spark migrator

SSTableloader Based Migration
Scylla OSS Scylla DB
Snapshot
Scylla OSS Scylla DB
Snapshot Scylla cloud
Scylla OSS Scylla D...

SSTableloader Based Migration
● Cons
○ SSTable migration slowed down when we have a secondary / composite key.

Spark Scylla Migrator
Scylla
opensource
New single-node
Scylla opensource
Backup snapshot to
S3
Take Scylla backups
and mo...

Recommendations
● Create new cluster in order to avoid any down-time / impact during spark based
migration.
● Increase par...

Future plans
● We are looking to move more use cases to Scylla cloud cluster instead of our existing
DBs.
● Use-cases :
○ ...

User Watch Store
● Time Series
● Append only writes
● Aggregate on reads
● Expiration
W R

User Watch Store

User Watch Store
● Time range query on user’s watch
○ Aggregate duplicated watches (watch time / video position / etc.)
○ ...

Issues at Scale
● P50 > 1s!

Issues at Scale

Discovered Anti-patterns

Discovered Anti-patterns

Compactions

Lessons Learned

Mitigation
● Avoid Queue Like Anti-pattern
○ Exclude tombstone with time range in query
○ Cache friendly with more static ...

Optimizations: Compaction
● Use consistency to CL=ALL for deletes
● Set GC_GRACE_SECONDS to 0
● Schedule daily once major ...

United States
2445 Faber St, Suite #200
Palo Alto, CA USA 94303
Israel
Maskit 4
Herzliya, Israel 4673304
www.scylladb.com
...