More Related Content

Recently uploaded(20)


Scott Andreas - Garbage, Garbage Everywhere: GC Strategies for Event Processing Systems on the JVM, Boundary Tech Talks 11/17/11

  1. Garbage, Garbage Everywhere GC Strategies for Event Processing Systems on the JVM C. Scott Andreas Pizza, Beer, and Tech Talks November 17, 2011
  2. What’s ESP / CEP? • Event Stream Processing Selecting events on dimensions among a stream of moving data, maintaining them for a brief period, emitting aggregations. • Complex Event Processing Identifying correlations between events, predicting trends, and programmatically reacting to emergent trends.
  3. ESP and Network Analytics • Packet flows are event streams with many dimensions. • Blast them into the engine, select over the stream, emit aggregations based on queries. • Ipfix data flows in, JSON comes out.
  4. Back of the Envelope • 500 Mbps / sec data comes into the JVM xxx Mbps / sec data goes out of the JVM • This memory must be allocated, retained for processing, freed, and collected. • Actual allocation rates far higher than data in / out (Memory also used for deserializing, aggregations, etc).
  5. Opening the Grimoire • GC Tuning to the rescue • Oracle guide with 84 -XX: options • Stas’s guide: 830 options (OpenJDK debug builds)
  6. Moment of Pause • Don’t touch the knobs unless you need to • Server defaults are a decent place to start for local development • Defaults shipped with Cassandra decent for bimodal GC profiles • Basic rule of thumb: if you’re aggressively tuning garbage collection, you can trade hours of frustration for ~10% gain
  7. Generational Garbage Collection • Modern JVMs divide heap space up into multiple “generations.” • Most applications have a lot of objects which live for a very short time, and a lot which live (nearly) forever. • Generational collection enables the JVM to collect unused memory more efficiently by avoiding unnecessarily scanning heap / object graphs for references or free regions.
  8. first attempt: “deploy the g1”
  9. Low-Pause Collector: The G1 v% => threshold violations. lower is better.
  10. G1 Collector • Hundreds of tiny 1ms collections / second rather than ParNew’s ~100 - 200ms larger collections. • Capable of meeting ambitious pause targets. • Powered by a gang of threads working in parallel • ...cooperating to chew through CPU like it’s free.
  11. second attempt: you’re gonna laugh
  12. the unsafe
  13. Unsafe • A OpenJDK/HotSpot class exposing direct access to the underlying VM, OS, and memory. • This includes the ability to allocate, manage, and free memory. • Perhaps we can outsmart the JVM and do a better job than it!
  14. Learned While Astray • Finalization occurs in a single thread. • Jumping from native finalization back into Java is expensive. • Attempting to outsmart the garbage collector by creating hundreds of thousands of tiny ByteBuffers is...a thing. • Java’s collectors are very good at collecting garbage. Your home-grown in-app GC go-kart is probably not.
  15. returning to earth :: attempt 3[a]
  16. Lessons from Science • Your rate of “freeing” must be equal to or exceed your rate of object allocation on the heap. • High rates of allocation speed up heap fragmentation, which compounds the problem. • Creating less garbage reduces your rate of allocation (and freeing). • This means less work for the garbage collector.
  17. best way to help out the gc ::
  19. Breaking out YourKit
  20. attempt 3[b]: responsible tuning of the old hat
  21. Optimizing for Infant Mortality default newgen ratios in java 6 • Java 6 AMD64 (server) defaults to allocating 1/3 of heap to the new gen, 2/3 to the old gen. • ESP/CEP workloads place tremendous pressure on the newgen. The vast majority of objects survive less than five seconds. • Experiment: Allocate 80% of heap to the new gen, set a higher tenuring threshold, and lean hard on the ParNew collector.
  22. CMS Collector • Guardian of the tenured generation, favorite workhorse for years. • Primarily parallel, easier on the CPU than the G1. • ...But contains a significant pause phase, is less suited to meeting low pause targets.
  23. ParNew Collector • Designed for the small, but works great in the large. Excellent throughput, parallel collection. • Can collect ~5GB in ~200ms on a quad-core Xeon w/HT. • 200ms pause every several seconds favorable compared to less frequent multi-second pauses and promotion failures.
  24. Explosions in the Barrel
  25. “real-time” and the jvm
  26. Real Time and the JVM • Real Time Ability to meet specific targets with low variance is critical to the bare minimum functionality of the product (e.g., air bags). • “Soft” Real Time Ability to meet targets important but not critical. Value of system’s functionality is diminished but not eliminated by delay.
  27. Real Time and “The Pause” • To what extent can a system which can endure pauses of unpredictable duration be considered “real-time”? • Is it sufficient to mitigate the frequency and duration of pauses for a system to still deliver value as “soft real-time”? • Is the alternative worth the cost?
  28. what does your app sound like?
  29. Garbage, Garbage Everywhere GC Strategies for Event Processing Systems on the JVM C. Scott Andreas Pizza, Beer, and Tech Talks November 17, 2011