Cassandra Summit 2014: Launching PlayStation 4 with Apache Cassandra

1.
Launching PS4 with Cassandra

2.
Introduction • AlexanderFilipchik – Staff Software Engineer at SNEI • Dustin Pham – Staff Software Engineer at SNEI

3.
Agenda • Journeytowards Cassandra • Cassandra-backed PS4 Features • Ops-y Stuff • Lessons learned

4.
Journey towards Cassandra

5.
Challenges • SmallTeam • Legacy Support • Hardware Deadline • Scaling @ Peak Time

6.
Why Cassandra •Strong community • Horizontally scalable architecture • Good performance • Cost effective • New adventure J 6

7.
PS4 Features backedby Cassandra

8.
Cassandra-backed PS4 features • What’s New • Video Library • My Library • PS Now • Notifications • LiveArea • Store catalog • Pre-order • PS Plus • Recommendations • Remote Download • Share • Authentication • +more

9.

10.

11.

12.

13.

14.

15.

16.

17.

18.

19.

20.

21.
Cassandra-backed PS4 features • What’s New • Video Library • My Library • PS Now • Notifications • LiveArea • Store catalog • Pre-order • PS Plus • Recommendations • Remote Download • Share • Authentication • + more

22.
Ops-y Stuff

23.
Infrastructure • Hostedin cloud and physical DCs • Several hundred nodes and growing • Cluster by feature • Vnodes and Assigned token clusters • Astyanax Client

24.
Stats for PS4cloud nodes • Data throughput: Gigabytes / sec • Cassandra read/writes: > 200,000 / sec • Data size: tens of terabytes • 10M PS4 and 80M PS3 sold 24

25.
Clusters • Clusterper Read/Write pattern initially • Now use cluster per feature • Seeds referenced by DNS names • Size Tiered compaction • Manual compactions for some CFs 25

26.
A typical node • m2.4xl + i2.2xl • 2 ephemeral disks (~ 2 x 800 GB) • Commit log on root partition • Topology managed in the topology file managed by chef 26

27.
AWS • Nodesare interleaved between AZs – Replication factor spreads data across AZ’s – Minimizes downtime due to AZ outage Availability Zone A Availability Zone C

28.
Eph1 Disk Layout Eph0 Pre-Launch Launch Current ü 2 Ephemerals in a RAID 0 ü Higher throughput (io spreads into 2 devices for reading & writing) ü If you lose 1 device, you loose the array ! ü 2 Ephemerals in a RAID 1 ü Higher throughput for reading (io spreads into 2 devices), but not for writing ü If you lose 1 device, the array continues up in degraded mode. ü ½ the available space Eph0 ü 2 individual Ephemerals ü Higher throughput (io spreads into 2 devices for reading & writing) ü You lose 1 device, Cassandra stops (configurable) ü No RAID overhead Eph0 AWS m2.4xl RAID 0 Eph1 AWS m2.4xl RAID 1 Eph1 AWS m2.4xl

29.
Cluster Resizing

30.
Thrift Payload Size thri%_framed_transport_size_in_mb thri%_max_message_length_in_mb

31.
Bouncing Nodes phi_convict_threshold

32.
Inter-DC Latency

33.
Monitor system health • Nagios • Kibana/Elasticsearch • Graphite • AWS Cloudwatch • App level monitoring • Opscenter

35.
App level metrics

36.
Lessons Learned

37.
Fun with AstyanaxClient • Cross DC Latencies – Several second latencies in JP and EE data centers – Astyanax configs to ensure local datacenters used • Imbalanced node traffic – Hashing algorithm (MD5 vs Murmur3) • DNS Caching in the JVM – Stale seed nodes

38.
A tale of2 Nodes

39.
Cluster lessons •A single bad node can raise app latencies significantly • Taking out an entire cassandra cluster is easy (not so fun) – Compressing data before sending to cassandra helps a lot. • Corrupted SStable resulted in cascading failure

41.
• Monitoring –Memtable flush frequency – Hinted handoffs – Garbage collection – Compactions – Histograms

42.
• VPNs area dangerous bottle neck • Easier to rebuild a node than to fix • Backup data – Replication factor helps but does not account for data corruption

43.
• Denormalization costs • Disk is cheap but EC2s are not • TTL on almost everything • Adjust gc_grace_period based off TTL times • Transactions ? Be creative • Load test with real data

44.
• Replication strategy: – Read / Write pattern – Data is source of truth or not – Data locality – User Level data vs App level data • Cluster wide commands should be staggered – Global repair L

45.
Tokens • Vnodesvs Assigned Tokens – Increased chattiness on gossip protocol with vnodes – Perceived slowness on repair and cleanup operations on vnodes enabled cluster – Astyanax client does not like vnodes…

46.
Compactions • Compactionsare your worst enemy – larger disk usage = high cpu & longer compactions • Leveled compaction vs sized compaction – Start up time – Cpu tradeoff – IO tradeoff • Updates + Removals eat up disks

47.
We are hiring… sonyentertainmentnetwork.com/careers

Cassandra Summit 2014: Launching PlayStation 4 with Apache Cassandra

More Related Content

What's hot

Viewers also liked

Similar to Cassandra Summit 2014: Launching PlayStation 4 with Apache Cassandra

More from DataStax Academy

Recently uploaded

Cassandra Summit 2014: Launching PlayStation 4 with Apache Cassandra