20 sec. Set theme for talk. Not about distributed systems aspects. Experiences with Single node performance issues during SSD migration.
1 min. multi tenant. TODO : say explicitly “focus on diagram at the bottom”
2 min “Tackling these GC issues, gave us some intuitions for benchmarking java based applications on SSD”. In end-end correlation, “First of all, given Big data on SSD is relatively new, we need tools to correlate patterns across the stack and make sense out of it. For example – young gen pauses due to linux page stealing. Line up page scan data and system time components in young gen pause from gc log”
Voldemort on Solid State Drives
Voldemort on SolidState DrivesVinoth Chandar, Lei Gao, Cuong TranLinkedin Corp, Mountain View, CA
What is Voldemort?• Distributed key-value storage system • Based on Amazon Dynamo• Open source • https://github.com/voldemort/voldemort• Widely used • Several TB of data @ Linkedin • Eharmony, Nokia, JIVE Software etc.• We are on SSD!
SSD Migration• Improve average latency • 20 ms to 2 ms• Cluster expansion & data restoration • Improved 10x• 95th & 99th latencies shot up • 30ms to 130ms and 240ms to 380ms• GC Issues !! Client operations BDB Cleaner Retention Job BDB Cache App1 App2 .. AppN Server JVM Heap Linux
Benchmarking for SSD• Java based systems• Need for End-End Correlation • Linux page stealing • Use AlwaysPreTouch• Run Workload causing Fragmentation • Promotion failure more likely • Defragmentation costly• Run Workload with High memory churn • cost of CMS Initial Mark • High CMS initiating fraction• Factor in higher IO Parallelism • Sequential IO benchmarks no longer use single thread
Takeaways• GC impact significant • Big heaps = Big pauses• Longer runs to account for SSD GC • SSD is log structured • Worst case 2 ms• Special attention to 95th & 99th • Fragmentation eats up gains