Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
NETFLIX’S BIG LEAP
FROM
ORACLE TO C*
ROOPA TANGIRALA
Engineering Manager
Netflix
WHO AM I?
Engineering Manager @ Netflix
Twitter - @roopatangirala
Email roopa.tangirala@gmail.com
Linkedin - https://www.l...
OVERVIEW
• Brief History
• Set up
• Migration Strategy
• Migration Challenges
• Example of Real use cases
• Lessons learnt
1997
DATACENTER
BACKEND
ORACLE DATAMODEL LIMITATIONS
NO HORIZONTAL SCALING
LICENSE COST
EVERY TWO WEEKS!!
2008
MOVE TO CLOUD
REQUIREMENTS
• HIGHLY AVAILABLE
• MULTI DATACENTER SUPPORT
• PREDICTABLE PERFORMANCE AT SCALE
WHY C* ?
• Massively scalable architecture
• Multi-datacenter, multi-directional replication
• Linear scale performance
• ...
BACKEND NOW
MICRO SERVICES
• Horizontal, Homogenous, Commoditized
DOWNTIME
ALMOST DAILY PUSHES
ACTIVE ACTIVE
GLOBAL PRESSENCE
MIGRATION STRATEGY
BABY STEPS
NEW FEATURES FIRST TO CLOUD
DATA MODEL REVIEW
keyspace
column family
Row
column
•name
•value
•timestamp

DB/Schema
Table
Row
column
• name
• value
O
...
SCHEMALESS DESIGN
• Row-oriented
• Number of columns/Names can differ
namexyz Paul zip 95123
nameabc Adam zip 94538 sex Ma...
UNDERSTAND WRITE PATH
client
Commit
log (Disk)
Memtable
(memory)
sstable sstable sstable
Flush
UNDERSTAND READ PATH
client
memtable
sstable
sstable
sstable
Row cache/key cache
LOGIC IN APPLICATION
• Stored procedures
• Functions
• Triggers
• Referential integrity constraints
DATACENTER AWS
ORACLE
C*
APP
DUAL WRITESREADS
CONSISTENCY
CHECKER
WRITES
READS
WRITES
FORKLIFT
MAIN APPROACH
• DUAL WRITES
• FORKLIFT OLD DATASET
• CONSISTENCY CHECKER
• MIGRATE READS TO C* FIRST
• SWITCH OFF WRITES T...
Relationships – Better in App Layer
ORACLE SEQUENCE
• USE UUID for Unique keys
• Distributed Sequence Generator for Ordering
• C* counters – not so much
Heavy Transactional Use Case
RDBMS
CHALLENGES
SECURITY
DENORMALIZE
DENORMALIZE
DENORMALIZE
DENORMALIZE
DENORMALIZE
Roman Riding
Model Around Queries
Engineering Effort
Know limitations
SOURCE OF TRUTHF TRUTH
REAL EXAMPLES
API
• High concurrency
• Range scans
• ~1MB of data
• Caused heap issues at Cassandra level
• Very high read latency
Range scan replaced with
inverted index
0000/odp/{ui}/pathEvaluator_2 active Scripts_1 Text datafalseallocation
0000
/xbox...
Inverted Index considerations
• Column name can be used a row key placeholder
• Hotspots!!
• Sharding
VIEWING HISTORY
Growth of Viewing History
47
Problem
Growth Pattern: “Hockey-stick”
Retention Policy: Retain forever
Access Pattern: Retrieve all for a
customer
Sc...
Goals
Small Storage Footprint
Consistent Read/Write Performance
Infinite Scalability
49
Old Data Model
50
Old Data Model - Pros
Simple Wide Rows
No additional processing
Fast Write
Fast Read for Majority
51
Old Data Model - Cons
Read latency increases as number of viewing records
increases
Cassandra Internal
• Number of SSTab...
New Data Model
Split Viewing History into two Column Families
1. Recent
• Small records with frequent updates
• Cassandra ...
New Data Model cont’d
54
RESULTS
55
Think Data Archival
Data stores grow exponentially
Have a process in place to archive data
• Moving to a separate column...
Cinematch Rating Service
• First Model
• Second Model
Movie_id12345 BLOB
Movie_id4355 BLOB
545 1 545 4 545 534512345 4 545...
WHAT WORKED?
12345
4355
343 674
5
Name
3
4542443 242 Name
BLOB
4 BLOB52
BLOB vs COLUMN/VALUE
Try out column/value approach first and hopefully
it satisfies avg/95/99th
Column value pro’s:
• Wr...
LESSONS LEARNT
• Get the model right
• Baby Steps
• Huge Engineering effort
• Think Compression
• Performance test
• DB ju...
WE ARE HIRING !! – CHECK OUT JOBS.NETFLIX.COM
Upcoming SlideShare
Loading in …5
×

Netflix's Big Leap from Oracle to Cassandra

1,585 views

Published on

Talk which I gave at cassandra Summit 2015

Published in: Technology
  • Be the first to comment

Netflix's Big Leap from Oracle to Cassandra

  1. 1. NETFLIX’S BIG LEAP FROM ORACLE TO C* ROOPA TANGIRALA Engineering Manager Netflix
  2. 2. WHO AM I? Engineering Manager @ Netflix Twitter - @roopatangirala Email roopa.tangirala@gmail.com Linkedin - https://www.linkedin.com/pub/roopa-tangirala/3/960/2b
  3. 3. OVERVIEW • Brief History • Set up • Migration Strategy • Migration Challenges • Example of Real use cases • Lessons learnt
  4. 4. 1997
  5. 5. DATACENTER
  6. 6. BACKEND
  7. 7. ORACLE DATAMODEL LIMITATIONS
  8. 8. NO HORIZONTAL SCALING
  9. 9. LICENSE COST
  10. 10. EVERY TWO WEEKS!!
  11. 11. 2008
  12. 12. MOVE TO CLOUD
  13. 13. REQUIREMENTS • HIGHLY AVAILABLE • MULTI DATACENTER SUPPORT • PREDICTABLE PERFORMANCE AT SCALE
  14. 14. WHY C* ? • Massively scalable architecture • Multi-datacenter, multi-directional replication • Linear scale performance • Transparent fault detection and recovery • Flexible, dynamic schema data • Guaranteed data safety • Tunable data consistency
  15. 15. BACKEND NOW
  16. 16. MICRO SERVICES • Horizontal, Homogenous, Commoditized
  17. 17. DOWNTIME
  18. 18. ALMOST DAILY PUSHES
  19. 19. ACTIVE ACTIVE
  20. 20. GLOBAL PRESSENCE
  21. 21. MIGRATION STRATEGY
  22. 22. BABY STEPS
  23. 23. NEW FEATURES FIRST TO CLOUD
  24. 24. DATA MODEL REVIEW keyspace column family Row column •name •value •timestamp  DB/Schema Table Row column • name • value O R A C L E CASSANDRA
  25. 25. SCHEMALESS DESIGN • Row-oriented • Number of columns/Names can differ namexyz Paul zip 95123 nameabc Adam zip 94538 sex Male namecde XYZ
  26. 26. UNDERSTAND WRITE PATH client Commit log (Disk) Memtable (memory) sstable sstable sstable Flush
  27. 27. UNDERSTAND READ PATH client memtable sstable sstable sstable Row cache/key cache
  28. 28. LOGIC IN APPLICATION • Stored procedures • Functions • Triggers • Referential integrity constraints
  29. 29. DATACENTER AWS ORACLE C* APP DUAL WRITESREADS CONSISTENCY CHECKER WRITES READS WRITES FORKLIFT
  30. 30. MAIN APPROACH • DUAL WRITES • FORKLIFT OLD DATASET • CONSISTENCY CHECKER • MIGRATE READS TO C* FIRST • SWITCH OFF WRITES TO DC
  31. 31. Relationships – Better in App Layer
  32. 32. ORACLE SEQUENCE • USE UUID for Unique keys • Distributed Sequence Generator for Ordering • C* counters – not so much
  33. 33. Heavy Transactional Use Case RDBMS
  34. 34. CHALLENGES
  35. 35. SECURITY
  36. 36. DENORMALIZE DENORMALIZE DENORMALIZE DENORMALIZE DENORMALIZE
  37. 37. Roman Riding
  38. 38. Model Around Queries
  39. 39. Engineering Effort
  40. 40. Know limitations
  41. 41. SOURCE OF TRUTHF TRUTH
  42. 42. REAL EXAMPLES
  43. 43. API • High concurrency • Range scans • ~1MB of data • Caused heap issues at Cassandra level • Very high read latency
  44. 44. Range scan replaced with inverted index 0000/odp/{ui}/pathEvaluator_2 active Scripts_1 Text datafalseallocation 0000 /xbox/{ui}/pathEvaluator_ 1 active Scripts_1 Text datafalseallocation 0000/tvui/{ui}/pathEvaluator_1 active Scripts_1 Text datafalseallocation active_Scripts_idx 1, 2 Idx_1 /tvui/{ui}/pathEvaluator 2/odp/{ui}/pathEvaluator 2/odp/{ui}/pathEvaluator 2/odp/{ui}/pathEvaluator scripts client 1 2
  45. 45. Inverted Index considerations • Column name can be used a row key placeholder • Hotspots!! • Sharding
  46. 46. VIEWING HISTORY
  47. 47. Growth of Viewing History 47
  48. 48. Problem Growth Pattern: “Hockey-stick” Retention Policy: Retain forever Access Pattern: Retrieve all for a customer Scaling and performance challenges as the data grows 48
  49. 49. Goals Small Storage Footprint Consistent Read/Write Performance Infinite Scalability 49
  50. 50. Old Data Model 50
  51. 51. Old Data Model - Pros Simple Wide Rows No additional processing Fast Write Fast Read for Majority 51
  52. 52. Old Data Model - Cons Read latency increases as number of viewing records increases Cassandra Internal • Number of SSTables • Compaction • Read Repair Lesson learned : Wide Row Limitation 52
  53. 53. New Data Model Split Viewing History into two Column Families 1. Recent • Small records with frequent updates • Cassandra tuning : compaction, read repair, etc. 2. Compressed • Large historical records with rare updates • Rollup • Compression • Cassandra: rare compaction, no read repair 53
  54. 54. New Data Model cont’d 54
  55. 55. RESULTS 55
  56. 56. Think Data Archival Data stores grow exponentially Have a process in place to archive data • Moving to a separate column family • Moving to a separate cluster (non SSD) • Setting right expectations w.r.t latencies with historical data Cassandra TTL’s
  57. 57. Cinematch Rating Service • First Model • Second Model Movie_id12345 BLOB Movie_id4355 BLOB 545 1 545 4 545 534512345 4 545 2 454355 2 66 2 67 4
  58. 58. WHAT WORKED? 12345 4355 343 674 5 Name 3 4542443 242 Name BLOB 4 BLOB52
  59. 59. BLOB vs COLUMN/VALUE Try out column/value approach first and hopefully it satisfies avg/95/99th Column value pro’s: • Write payload can be smaller • Query by specific columns • Read path does not require reading the entire row Blob considerations • Read percentage is very high (90’s) • Read latencies are very important • All the data is read most of the time
  60. 60. LESSONS LEARNT • Get the model right • Baby Steps • Huge Engineering effort • Think Compression • Performance test • DB just for Data
  61. 61. WE ARE HIRING !! – CHECK OUT JOBS.NETFLIX.COM

×