0
How long can you afford to Stop The World?Strategies to overcome long application pause times caused by Java GCBerlin, May...
The information and evaluations expressed in this presentation are based on the author‘spersonal experiences and knowledge...
• Developing software for about 20 years• More than 10 years experience in the Enterprise Java world (JDK 1.2)• Prior Jest...
Agenda• Motivation and Scope• Summary of Java Memory Management Basics / GC Analysis• Discussion of Different Strategies t...
Motivation• The demand to process large quantities of data in memory is steadilyincreasing:– More and more data to process...
Motivation• Java GC can heavily impact application performance, especially in terms oflatency / responsiveness (multi-seco...
Motivation / Scope• There are multiple strategies to overcome/minimize application pausetime issues related to Garbage Col...
Out of Scope - HotSpot GC Tuning DetailsBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture[REF_02] Devoxx FR 201...
Scope• Ensure all attendees are aware of enough of the memory managementbasics in order to at least understand– The reason...
Automatic Memory Management Basics• Responsibilities of Automatic Memory Management• Basic Garbage Collector Algorithms / ...
Responsibilities of Automatic Memory Management• Service provided by a “managed runtime” (e.g. the Java Virtual Machine)in...
Garbage Collectors – Classification (1)• Serial versus Parallel– Serial: Only one GC task at a time (only single CPU core ...
Garbage Collectors – Classification (2)BERLIN, May 22. 2013 | Eric Hubert - Strategy & ArchitectureMostly-[REF_03] Memory ...
Garbage Collectors – Classification (3)• Reference Counting / Tracing– Ref. Counting: No longer in practical use due to re...
Garbage Collectors – Simplified View (Tracing)• Find and reclaim unreachable objects> Trace the heap starting at the roots...
Garbage Collection – Basic Algorithms (1)• Copy/Scavenge– Copy all live objects starting from the roots in a single pass o...
Garbage Collection – Basic Algorithms (2)• Mark / Sweep / (Compact)– Mark any object reachable as live– Scan heap for obje...
Garbage Collection – Basic Algorithms (3)• Mark / Sweep / (Compact)BERLIN, May 22. 2013 | Eric Hubert - Strategy & Archite...
Garbage Collection – Basic Algorithms (4)• Mark / Compact– Reachable objects are marked– Compacting step relocates the rea...
Concept of Generations / Generational GC• Incorporate this typical object lifetime structure into GC– Different heap areas...
Concept of Generations / Generational GC• Generations are of new and survived objects• Heap divided in zones by age of the...
Concept of Generations / Generational GC• Focus collection efforts on young generation– Normally live objects represent on...
Common Triggers of Full GC• Completely JVM implementation specific, more specifically it also dependson selected GC algori...
Analysis of GC behavior / Information Sources (1)• GC traces from JVM-XX:+PrintGC (same as -verbose:gc )-XX:+PrintGCDetail...
Analysis of GC behavior / Information Sources (2)• Example of STW Pause caused by Full GC using Parallel GC[Full GC [PSYou...
Analysis of GC behavior / Information Sources (3)• Standard Java GC-related Management Beans (JMX)– Can be used for (remot...
Analysis of GC behavior / Information Sources (4)• Standard Memory Management Beans– java.lang:type=Memory• Metrics: HeapM...
Analysis of GC behavior / Information Sources (4)• Custom GC Management Bean– MajorCollectionCount– MajorCollectionTime– M...
GC Analysis – Command Line Tools1) jstat/jstad/jpsUsage: jstat -help|-optionsjstat -<option> [-t] [-h<lines>] <vmid> [<int...
GC Analysis – GUI Tools (1)Gcviewer – GC trace analyzer• Originally developed by tagtraum industries (only maintained unti...
GC Analysis – GUI Tools (2)HPjmeter• GC log analyzer and monitoring (the latter for HP UX)• Download: www.hp.com/go/hpjmet...
GC Analysis – GUI Tools (3)HPjmeterBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
GC Analysis – GUI Tools (4)JConsole• Part of JDK• Can be used for monitoring of local (jvmid/pid) or remote (JMX RMI) JVMs...
GC Analysis – GUI Tools (5)JConsoleBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
GC Analysis – GUI Tools (6)Visual VM with Visual GC Plugin• Part of JDK• Based on jvmstat (local monitoring via jvmid, rem...
GC Analysis – GUI Tools (7)Visual VM with Visual GC PluginBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
GC Analysis – GUI Tools (8)IBM GCMV• Eclipse RCP Application (can also be installed as plugin in Eclipse)• Loads gc log fi...
GC Analysis – GUI Tools (9)IBM GCMVBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
GC Analysis – GUI Tools (10)JHiccup• Small Java tool from Azul Systems to demonstrate application hiccups(primarily caused...
GC Analysis – GUI Tools (11)JHiccup – Example Graphs of Telco App• More details/examples later in this presentation …• Dow...
GC Tuning – Memory Performance TriangleBERLIN, May 22. 2013 | Eric Hubert - Strategy & ArchitectureMemory footprintThrough...
Strategies to overcome GC Pause Time issues1. Tuning of the JVM Runtime Behavior2. Reduce memory footprint of the applicat...
1. Tuning of the JVM Runtime Behavior (1)• Structured Approach - Preconditions– Knowledge about Java Memory Management• Un...
1. Tuning of the JVM Runtime Behavior (2)• If motivator is concrete performance issue (e.g. large pause time)– First ensur...
1. Tuning of the JVM Runtime Behavior (3)• Where are the large pauses?  typically old gen• Start to tune young gen!• Firs...
Young Old1. Tuning of the JVM Runtime Behavior (4)• Generational Oracle HotSpot JVM (6 Collector choices/combinations)BERL...
1. Tuning of the JVM Runtime Behavior (5)• JVM attempts to use reasonable defaults in all areas, but also offers a largenu...
1. Tuning of the JVM Runtime Behavior (6)• AssessmentBERLIN, May 22. 2013 | Eric Hubert - Strategy & ArchitecturePros Cons...
2. Reduce Memory Footprint of the application• Sometimes easier said than actually done• Generally one should avoid too mu...
2. Reduce Memory Footprint of the application• AssessmentBERLIN, May 22. 2013 | Eric Hubert - Strategy & ArchitecturePros ...
3. More Powerful Hardware (RAM/CPU cores)• Very much depending on starting situation whether more computingresources can h...
3. More Powerful Hardware (RAM/CPU cores)• AssessmentBERLIN, May 22. 2013 | Eric Hubert - Strategy & ArchitecturePros Cons...
4. Distribute Memory Processing to multiple JVMs• Sometimes easy, sometimes harder• If we are talking about a mostly state...
4. Distribute Memory Processing to multiple JVMs• AssessmentBERLIN, May 22. 2013 | Eric Hubert - Strategy & ArchitecturePr...
5. Custom Off-Heap Memory Management (1)• sun.misc.Unsafe(internal implementation, dangerous, non-portable, and volatile)•...
5. Custom Off-Heap Memory Management (2)BERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecturestatic {Method directB...
Custom Off-Heap Memory Management (3)• Projects and products using this strategy:– Oracle Coherence– GigaSpaces, (to be va...
5. Custom Off-Heap Memory Management (4)• Assessment:BERLIN, May 22. 2013 | Eric Hubert - Strategy & ArchitecturePros Cons...
6. JVM with more efficient Memory Management (1)The Ultimate JVM GC Tuning Guidejava -Xmx40gZINGBERLIN, May 22. 2013 | Eri...
6. JVM with more efficient Memory Management (2)• Azul Zing Practical Evaluation / Comparison against Oracle HotSpot JVM• ...
6. JVM with more efficient Memory Management (3)• Oracle HotSpot 1.6.0_43-b01, 64bit – 1 GB MaxHeap - ParallelGC:-Xms768m ...
6. JVM with more efficient Memory Management (4)BERLIN, May 22. 2013 | Eric Hubert - Strategy & ArchitectureOracle HotSpot...
6. JVM with more efficient Memory Management (5)• Oracle Hotspot 1.6.0_43-b01, 64bit – 4 GB MaxHeap, ParallelGC-Xms2048m –...
6. JVM with more efficient Memory Management (6)BERLIN, May 22. 2013 | Eric Hubert - Strategy & ArchitectureOracle Hotspot...
6. JVM with more efficient Memory Management (7)• Oracle Hotspot 1.6.0_43-b01, 64bit – 4 GB MaxHeap, CMS-Xms4096m -Xmx4096...
6. JVM with more efficient Memory Management (8)BERLIN, May 22. 2013 | Eric Hubert - Strategy & ArchitectureOracle Hotspot...
6. JVM with more efficient Memory Management (9)• Oracle Hotspot 1.6.0_43-b01, 64bit – 2 GB MaxHeap, CMS (tuned):-Xms2g -X...
6. JVM with more efficient Memory Management (10)BERLIN, May 22. 2013 | Eric Hubert - Strategy & ArchitectureOracle Hotspo...
6. JVM with more efficient Memory Management (11)• Azul Zing 1.6.0_33-ZVM_5.5.3.0-b5, 64bit – 10 GB MaxHeap, C4:-Xmx10gBER...
6. JVM with more efficient Memory Management (12)BERLIN, May 22. 2013 | Eric Hubert - Strategy & ArchitectureAzul Zing 1.6...
6. JVM with more efficient Memory Management (12)• The efficiency of memory and thread management is up to each JVMimpleme...
6. JVM with more efficient Memory Management (13)• Comparison of peak mremap rates for 16 GB of remaps• Zing has a custom ...
6. JVM with more efficient Memory Management (14)BERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture[REF_08] Unde...
6. JVM with more efficient Memory Management (15)• AssessmentBERLIN, May 22. 2013 | Eric Hubert - Strategy & ArchitectureP...
Future Perspectives (1)• Garbage Collection and performance on virtualized environments isamong the hot future topics• Ora...
Future Perspectives (2)• No interest of Linux Kernel community to integrate Azul’s improvements(consequently they gave up ...
Discussion / QuestionsBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
References• [REF_01] Systems Computing - Understanding CPU Caching and Performanceby Jon "Hannibal" Stokes• [REF_02] Death...
Further Reading• Official Oracle JVM options documentation (subset)• Official Oracle JVM GC Tuning Documentation• Java Gar...
Upcoming SlideShare
Loading in...5
×

How long can you afford to Stop The World?

8,657

Published on

Vortrag von Eric Hubert vor der Java Usergroup Berlin-Brandenburg zur Optimierung des Garbage Collection-Verhaltens der Java Virtual Machine.

Published in: Technology
1 Comment
17 Likes
Statistics
Notes
No Downloads
Views
Total Views
8,657
On Slideshare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
74
Comments
1
Likes
17
Embeds 0
No embeds

No notes for slide

Transcript of "How long can you afford to Stop The World?"

  1. 1. How long can you afford to Stop The World?Strategies to overcome long application pause times caused by Java GCBerlin, May 22. 2013 | Eric Hubert - Strategy & Architecture
  2. 2. The information and evaluations expressed in this presentation are based on the author‘spersonal experiences and knowledge.They do not necessarily reflect the views of Jesta Digital.The author makes no warranties of any kind regarding the accuracy and veracity of informationand data provided.No one shall rely on any of the published test results which are inherently environment-specific.Readers are strongly encouraged to conduct own testing in their specific environment whichmay or may not show different results.All mentioned trademarks are property of their respective owners.BERLIN, May 22. 2013 | Eric Hubert - Strategy & ArchitectureDisclaimer
  3. 3. • Developing software for about 20 years• More than 10 years experience in the Enterprise Java world (JDK 1.2)• Prior Jesta worked for debis Systemhaus, T-Systems and Adesso• Working for Jesta since 2007 (formerly Jamba!, Fox Mobile)• Currently leading „Strategy & Architecture“ team focused on– Strategic development of platform infrastructure and middleware– Automation of software build-, packaging-, testing- , deployment-, release-and application monitoring processes– Close collaboration with cross-functional teams and central systemadministration/operations team• Contact: eric.hubert@jestadigital.com, XING, Linked inBERLIN, May 22. 2013 | Eric Hubert - Strategy & ArchitectureAbout me
  4. 4. Agenda• Motivation and Scope• Summary of Java Memory Management Basics / GC Analysis• Discussion of Different Strategies to overcome GC Pause Time issues• Future Perspectives• Open DiscussionBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  5. 5. Motivation• The demand to process large quantities of data in memory is steadilyincreasing:– More and more data to process and analyze in shorter times(near-time/real time business requirements)– Availability of commodity servers with up to 2 TB of RAM(over the past decades available memory grew ≈ 100x every 10 years)– Memory is still by far the fastest storage technologyBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  6. 6. Motivation• Java GC can heavily impact application performance, especially in terms oflatency / responsiveness (multi-second pause times on multi GB heaps)• The runtime of most GC algorithms is proportional to the size of the liveset of objects  the larger the Heap the larger the pause timesBERLIN, May 22. 2013 | Eric Hubert - Strategy & ArchitectureStorage Technology Random Access LatencyRegisters 1-3 nsCPU L1 Cache 2-8 nsCPU L2 Cache 5-12 nsMemory (RAM) 10-60 nsHigh-speed network 10,000-30,000ns (10-30µs)Solid State Disk (SSD) Drives 70,000-120,000ns (70-120µs)Hard Disk Drives 3,000,000-10,000,000ns (3-10ms)[REF_01] – Random Access Latencies of Storage Technologies
  7. 7. Motivation / Scope• There are multiple strategies to overcome/minimize application pausetime issues related to Garbage Collection• Nevertheless most talks, blog posts and other information sources centeron JVM tuning(choice of collector algorithms, hints to improve promotion betweengenerations etc.)• Most people know at least the most frequently used JVM GC tuningarguments, but only some know basics of the automated memorymanagement and alternative strategies• OOME causes, memory leaks, heap analysis etc. out of scope• Not going to stress you with excessive JVM tuning options eitherBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  8. 8. Out of Scope - HotSpot GC Tuning DetailsBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture[REF_02] Devoxx FR 2012 „Death by Pauses“ by Frank Pavageau
  9. 9. Scope• Ensure all attendees are aware of enough of the memory managementbasics in order to at least understand– The reason of long GC pauses– How to verify an application unresponsiveness was caused by GC• The main goal of my talk is to provide you with a broader view onstrategies to solve application pause times due to GC activity• Will not deep dive into any of those strategies, but explain each approach,discuss the pros and cons (as well as limitations and side- effects)• If applicable will provide pointers to information sources covering moredetailsBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  10. 10. Automatic Memory Management Basics• Responsibilities of Automatic Memory Management• Basic Garbage Collector Algorithms / Important Terms• Concept of Generations• Common Triggers of Full GC• Analysis of GC behavior / Information Sources• Memory Performance TriangleBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  11. 11. Responsibilities of Automatic Memory Management• Service provided by a “managed runtime” (e.g. the Java Virtual Machine)in which the program executes– Assisted allocation– Managed access to objects and their fields– Automatic de-allocation of objects (Garbage Collection)• Ensures that objects remain as long as they are in use• Deems objects with no incoming references from other live objects asgarbage• Ensures that objects that are no longer required are thrown away to freeup the memory they occupy for new objects• Ensures any finalize method is run before the object is thrown awayBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  12. 12. Garbage Collectors – Classification (1)• Serial versus Parallel– Serial: Only one GC task at a time (only single CPU core used)– Parallel: Multiple GC tasks are performed in parallel (multiple CPU core usage)• Stop-The-World (STW) versus (Mostly) Concurrent– STW: app threads are suspended during whole GC– Concurrent: app threads are executed while GC tasks are performed• Incremental– Performs a garbage collection operation or phase as a series of smalleroperations with gaps in betweenBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  13. 13. Garbage Collectors – Classification (2)BERLIN, May 22. 2013 | Eric Hubert - Strategy & ArchitectureMostly-[REF_03] Memory Management in the Java HotSpot VM
  14. 14. Garbage Collectors – Classification (3)• Reference Counting / Tracing– Ref. Counting: No longer in practical use due to reference cycle problematic– Tracing: Currently most common; either single phase copy or multiple phases(mark and optionally sweep and/or compact)• Copying versus Non-compacting versus CompactingBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  15. 15. Garbage Collectors – Simplified View (Tracing)• Find and reclaim unreachable objects> Trace the heap starting at the roots(thread stacks, static fields, operands of executed expression)> Visits every live object> Anything not visited is unreachable> Therefore garbage• If you can follow a chain of references from a root to a particular object,then that object is "strongly" referenced. It will not be collected.• Referenced objects are also called „live objects“ or “live set”BERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  16. 16. Garbage Collection – Basic Algorithms (1)• Copy/Scavenge– Copy all live objects starting from the roots in a single pass operation from asource space to a target space and reclaim source space (effectively a moveoperation)• At the beginning all objects are in source space and all references point to sourcespace• Start at the roots, copy any reachable object to target space and correct referenceswhile doing so• At the end of copy all objects are in target space and all references point to targetspace; source space can be completely cleared– Amount of work is generally linear to the „live set“BERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  17. 17. Garbage Collection – Basic Algorithms (2)• Mark / Sweep / (Compact)– Mark any object reachable as live– Scan heap for objects not marked live (traced in a kind of free-list)(the sweep step is generally linear to the entire heap size, not just the live set)– Over time, memory fragments• Slower allocation• Longer sweep phases• Risk not having large enough contiguous space for allocation of large objects; canresult in OOME– Compaction moves (relocates) live objects together to reclaim contiguousempty space; all object references need to be corrected (remap); compactingis an expensive /time consuming operation– A mark/sweep collector would not be a good choice for young generation, asit will not gain efficiency from the sparsenessBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  18. 18. Garbage Collection – Basic Algorithms (3)• Mark / Sweep / (Compact)BERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture[REF_04] Mark-Sweep-Compact – Keith D. Gregory
  19. 19. Garbage Collection – Basic Algorithms (4)• Mark / Compact– Reachable objects are marked– Compacting step relocates the reachable (marked) objects either towards thebeginning of the heap area (in-place compaction) or to another location(evacuating compaction)– Mark and compact work are both linear to live set, while sweep work is linearto heap size– Consequently, a mark/compact collector is linear to live set only, giving itsimilar efficiency characteristics to copying collectors– Examle: Azul C4BERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  20. 20. Concept of Generations / Generational GC• Incorporate this typical object lifetime structure into GC– Different heap areas for objects with different lifetime– Mostly different GC algorithms for objects with different lifetimeBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture[REF_05] The Art of Garbage Collection Turing – Angelika Langer & Klaus Kreft
  21. 21. Concept of Generations / Generational GC• Generations are of new and survived objects• Heap divided in zones by age of the objectsBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architectureeden survivor tenuredAllocation of objects ineden space(experienced no GC )2 alternately usedcopy target spaces(experienced several GCs)Objects survivedmultiple GCsYoung (nursery) generation tenured generation(old generation)objectscollected byMinor andFull GCobjectscollected onlybyFull GCobject lifetime[REF_06] Based on Java 7 Garbage Collector G1 by Antons Kranga
  22. 22. Concept of Generations / Generational GC• Focus collection efforts on young generation– Normally live objects represent only relatively small percentage of space– Promote objects living long enough to older generations• Tends to be much more efficient; great way to keep up with high allocationrate• Only collect older generation as it fills up• Requires a “Remembered set”: a way to track all references into the younggeneration from the outside• Usually want to keep surviving objects in young generation for a whilebefore promoting them to the old generation:– Immediate promotion can dramatically reduce generational filter efficiency– Waiting too long to promote can dramatically increase copying workBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  23. 23. Common Triggers of Full GC• Completely JVM implementation specific, more specifically it also dependson selected GC algorithms• “Common“ triggers in Oracle HotSpot JVM are:– Old generation or permanent generation filled to a certain percentage– Calling System.gc() (unless JVM option -XX:+DisableExplicitGC is set)– Not enough free space in survivor space to copy objects from eden space– Space extends or shrinkage (also applies to PermGen)• Verification via gc logs and/or Java MBeansBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  24. 24. Analysis of GC behavior / Information Sources (1)• GC traces from JVM-XX:+PrintGC (same as -verbose:gc )-XX:+PrintGCDetails-XX:+PrintGCTimeStamps-XX:+PrintGCDateStamps (since JDK 6 Update 6)-Xloggc:logs/gc.log-XX:GCLogFileSize=50M-XX:NumberOfGCLogFiles=3-XX:+UseGCLogFileRotationAddition diagnose options (trouble shooting / tuning)-XX:+PrintTenuringDistribution-XX:+PrintHeapAtGC-XX:+TraceClassLoading / -XX:+TraceClassUnloading-XX:+PrintGCApplicationStoppedTime (Warning: misleading – all JVM safepoints)-XX:+PrintSafepointStatistics (can be used for verification)BERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  25. 25. Analysis of GC behavior / Information Sources (2)• Example of STW Pause caused by Full GC using Parallel GC[Full GC [PSYoungGen: 8523K->0K(188160K)][PSOldGen: 574126K->428865K(575808K)] 582650K->428865K(763968K)[PSPermGen: 115404K->115404K(246144K)], 4.8381260 secs]• Example of STW Pauses caused by non-concurrent phases of CMS GC[GC [1 CMS-initial-mark: 1477934K(1835008K)] 1521654K(2053504K),0.0902490 secs][1 CMS-remark: 3717734K(4023936K)] 3810187K(4177280K), 1.0523700 secs]• Examples of STW Pauses caused by fallback of CMS GC to Serial Old GC(concurrent mode failure): 1784934K->1309805K(1926784K), 3.6729840 secs]1927090K->1309805K(2080128K), [CMS Perm : 94690K->93690K(131072K)],3.7968250 secs]Hint: -XX:CMSInitiatingOccupancyFraction=<xx> and -XX:+UseCMSInitiatingOccupancyOnlyGC ParNew (promotion failed): 153344K->153344K(153344K), 0.1724000secs]1574.221: [CMS: 2786273K->2531770K(4023936K), 5.5668010 secs]2926282K->2531770K(4177280K), [CMS Perm : 94733K->92808K(131072K)],5.7397890 secs]BERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  26. 26. Analysis of GC behavior / Information Sources (3)• Standard Java GC-related Management Beans (JMX)– Can be used for (remote) real-time monitoring of GC behavior and memoryusage– MBean names are Garbage Collector specific we use custom code for normalization to streamline monitoring config– java.lang:type=GarbageCollector,name=<collector name>• Young Gen: Copy, ParNew, PS Scavenge, G1 Young Generation• Old Gen: MarkSweepCompact , PS MarkSweep, ConcurrentMarkSweep,G1 Old Generation• Metrics: CollectionCount, CollectionTime, LastGcInfo (Composite)BERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  27. 27. Analysis of GC behavior / Information Sources (4)• Standard Memory Management Beans– java.lang:type=Memory• Metrics: HeapMemoryUsage (Composite: init, committed, usage, max)– java.lang:type=MemoryPool,name=<space name>• Eden: Eden Space, Par Eden Space, PS Eden Space, G1 Eden• Survivor: Survivor Space, Par Survivor Space, PS Survivor Space, G1 Survivor• Old: Tenured Gen, PS Old Gen, CMS Old Gen, G1 Old Gen• Perm: Perm Gen, PS Perm Gen, G1 Perm Gen• Metrics: Usage (init, committed, max, used)BERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  28. 28. Analysis of GC behavior / Information Sources (4)• Custom GC Management Bean– MajorCollectionCount– MajorCollectionTime– MinorCollectionCount– MinorCollectionTime– CumulatedCollectionTime– LastMajorCollectionDuration– LastMajorCollectionMemoryReduction– LastMajorCollectionStartTime– LastMajorCollectionEndTime– TenuredCollector– YoungCollector– Uptime• Warning: CollectionTime <> STW Pause Time for Concurrent CollectorsBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  29. 29. GC Analysis – Command Line Tools1) jstat/jstad/jpsUsage: jstat -help|-optionsjstat -<option> [-t] [-h<lines>] <vmid> [<interval> [<count>]]Example output – Live Server instance:jstat -gc 30731 1s 10S0C S1C S0U S1U EC EU OC OU3520.0 3520.0 2593.7 0.0 28224.0 2688.8 261728.0 184199.8PC PU YGC YGCT FGC FGCT GCT43172.0 25851.9 168136 978.259 1021 4.195 982.454– Can only be used to calculate averages or with short update intervals– Use option „-gcutil“ if you rather want to see space usage percentages– jps helps to determine vmid (but mostly maps to PID anyway)– jstatd is required for remote usage of jstatBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  30. 30. GC Analysis – GUI Tools (1)Gcviewer – GC trace analyzer• Originally developed by tagtraum industries (only maintained until 2008)• Fork: https://github.com/chewiebug/GCViewer/downloadsBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  31. 31. GC Analysis – GUI Tools (2)HPjmeter• GC log analyzer and monitoring (the latter for HP UX)• Download: www.hp.com/go/hpjmeterBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  32. 32. GC Analysis – GUI Tools (3)HPjmeterBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  33. 33. GC Analysis – GUI Tools (4)JConsole• Part of JDK• Can be used for monitoring of local (jvmid/pid) or remote (JMX RMI) JVMservice:jmx:rmi:///jndi/rmi://<remote-machine>:<port>/jmxrmior if behind firewall and using custom jmx rmi proxy:service:jmx:rmi://<remote-machine>:<proxyport>/jndi/rmi://<remote-machine>:<port>/jmxrmi• Integrated MBean browser• Shows active JVM options, GC info and more runtime information• Can be used to verify memory/gc behavior in realtimeBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  34. 34. GC Analysis – GUI Tools (5)JConsoleBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  35. 35. GC Analysis – GUI Tools (6)Visual VM with Visual GC Plugin• Part of JDK• Based on jvmstat (local monitoring via jvmid, remote requires jstad)• Many other plugins available (also MBean browser)• Shows active JVM options and other runtime information (not GC algos)• Can be used to verify memory/gc behavior in realtime• Very detailed view including information regarding survivor space usageas well as age information (histogram – not available for all algorithms)BERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  36. 36. GC Analysis – GUI Tools (7)Visual VM with Visual GC PluginBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  37. 37. GC Analysis – GUI Tools (8)IBM GCMV• Eclipse RCP Application (can also be installed as plugin in Eclipse)• Loads gc log file similar to gcviewer and provides statistics and graphs• nice capability to zoom into pause time graph area• Mainly written for IBM J9, but most parts also work for Oracle JVMs• Update Site:http://download.boulder.ibm.com/ibmdl/pub/software/isa/isa410/production/BERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  38. 38. GC Analysis – GUI Tools (9)IBM GCMVBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  39. 39. GC Analysis – GUI Tools (10)JHiccup• Small Java tool from Azul Systems to demonstrate application hiccups(primarily caused by GC, or any other JVM safepoint/OS jitter etc.)• Either run from Command Line (script wrapping java command) or asjavaagent• Writes logfiles which can later be loaded in Excel to render nice diagramsby hitting a button (macros need to be active)• Possibility to compare percentile values against expectations (SLA)BERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  40. 40. GC Analysis – GUI Tools (11)JHiccup – Example Graphs of Telco App• More details/examples later in this presentation …• Download: http://www.azulsystems.com/downloads/jHiccupBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  41. 41. GC Tuning – Memory Performance TriangleBERLIN, May 22. 2013 | Eric Hubert - Strategy & ArchitectureMemory footprintThroughput Latency
  42. 42. Strategies to overcome GC Pause Time issues1. Tuning of the JVM Runtime Behavior2. Reduce memory footprint of the application3. More powerful hardware (more RAM/CPU cores)4. Distribute processing to multiple JVMs (with Remote Communication)5. Custom Off-heap memory management6. Switch to JVM implementation with more efficient Memory ManagementBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  43. 43. 1. Tuning of the JVM Runtime Behavior (1)• Structured Approach - Preconditions– Knowledge about Java Memory Management• Understanding how the memory is organized in the JVM to be tuned• Knowing the options to change GC related runtime behavior and their limits– Effect of Garbage Collector Choice– Effect of Memory Space Sizing– Effect of other collector-specific configuration switches– Knowledge about GC Analysis• Knowing what to measure• Knowing how to measure• Knowing how to interpret metrics– Knowledge about GC Tuning• Know at least how to approach a tuning– Know your Operational Requirements– Have one or multiple concrete Tuning Goals prior to any modification!BERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  44. 44. 1. Tuning of the JVM Runtime Behavior (2)• If motivator is concrete performance issue (e.g. large pause time)– First ensure the problem is really GC-related!– Verify current GC configuration and analyze current GC behavior– Evaluate your chances of improvement by runtime configuration tuning• Verify your hardware and OS resources• Verify object allocation rate• Verify occupancy of tenured generation after Full GC• Set those measures in relation to your goals• Decide whether to proceed– Use a comparable test environment with comparable workload to replicateyour issue (automate tests!)– Capture baseline data, do small changes at a time and compare with baselineBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  45. 45. 1. Tuning of the JVM Runtime Behavior (3)• Where are the large pauses?  typically old gen• Start to tune young gen!• First verify reasons of promotion (young  old), depending on outcome– Think of increasing new size (decrease NewRatio value); do it stepwise andverify result– Think of increasing survivor spaces (SurvivorRatio)– Think of increasing age threshold to avoid too early promotion to old gen• verify total heap size, think of decreasing it (if possible)• Proceed with old gc tuning• Switch collector, try to use CMS (-XX:+UseConcMarcSweepGC)BERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  46. 46. Young Old1. Tuning of the JVM Runtime Behavior (4)• Generational Oracle HotSpot JVM (6 Collector choices/combinations)BERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture(G1 Young Generation) G1 (G1 Old Generation)-XX:+UseG1GCSerial Young(DefNew : Copy)-XX:+UseSerialGCConcurrent OldConcurrentMarkSweep-XX:+UseConcMarkSweepGCSerial Old(MarkSweepCompact)Parallel Young(ParNew : ParNew)-XX:+UseParNewGCParallel Old(PS OldGen - PS MarkSweep)-XX:+UseParallelOldGCParallel Scavenge(PSYoungGen : PS Scavenge)-XX:+UseParallelGCFallback-XX:-UseParNewGCFallback
  47. 47. 1. Tuning of the JVM Runtime Behavior (5)• JVM attempts to use reasonable defaults in all areas, but also offers a largenumber of feature switches (currently more than 600)• Useful resources to gather details:– Official Oracle JVM documentation (lists about 90 options)http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html– Use JVM build-in listing functionsjava -XX:+UnlockDiagnosticVMOptions -XX:+PrintFlagsFinalFor adventurous guys add (Please don’t use any of those in production!)-XX:+UnlockExperimentalVMOptions– Choose one of the unofficial „complete references“ (gathered from source), e.g.http://www.pingtimeout.fr/2012/05/jvm-options-complete-reference.htmlBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  48. 48. 1. Tuning of the JVM Runtime Behavior (6)• AssessmentBERLIN, May 22. 2013 | Eric Hubert - Strategy & ArchitecturePros Conscan drastically improve performance(e.g. reduce maximum pause timesand/or improve throughput)quite a lot of knowledge aboutmemory management andimplementation specific switchesrequiredrelatively quickly to apply (dependingon knowledge and experience)danger to optimize for the moment(many variants: load, functionalityused, software changes)needs to be carefully monitored andrepeated with each redeployment /changed useheavily implementation dependent /can change with each minor JVMversion update
  49. 49. 2. Reduce Memory Footprint of the application• Sometimes easier said than actually done• Generally one should avoid too much premature optimizations• Rather frequently use heap dumps with memory analyzer or memoryprofiler and verify proper data structure usage in development iterations• Look out for usage of wrong scopes, e.g. mistakenly declared variableswithin loops although not needed (unnecessary allocation pressure)• Only load the amount of data in memory you need to process (e.g. fromsome persistent store)• For large amounts of objects carefully select data structures (verifyoverhead – fixed and per entry)BERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  50. 50. 2. Reduce Memory Footprint of the application• AssessmentBERLIN, May 22. 2013 | Eric Hubert - Strategy & ArchitecturePros Conshigh memory savings possible(e.g. reduction of allocation rate and/orlong-lived objects by up to >50%) alsoresulting in much smaller pause timesrather high, consistent effortcan have bigger positive impact thanany runtime tuningcan negatively impact execution time ifnot properly appliedcan introduce bugs if existing codeneeds to be changed
  51. 51. 3. More Powerful Hardware (RAM/CPU cores)• Very much depending on starting situation whether more computingresources can help to solve GC issues– e.g. application is heavily CPU bound and not enough CPU cycles to properlyrun GC concurrently– or maximum heap sizes should be increased, but not enough RAM• The VM implementation and the chosen GC algorithms have a big impactas well• If live set > 1 or 2 GB and currently using parallel GC on only two CPUcores, increasing the heap size, switching to CMS and increasing thenumber of CPU cores (thus GC threads) can have a large effectBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  52. 52. 3. More Powerful Hardware (RAM/CPU cores)• AssessmentBERLIN, May 22. 2013 | Eric Hubert - Strategy & ArchitecturePros Consnot much working effort to realize involved costsmay only mask underlying problemsuntil a later stage (e.g. increased load)
  53. 53. 4. Distribute Memory Processing to multiple JVMs• Sometimes easy, sometimes harder• If we are talking about a mostly stateless application with a rather smallamount of long lived objects horizontal scaling is quite easy (using properloadbalancing and failover)• Long-lived data needs to be somehow partitioned/sharded in order toimprove efficiency (either manually or by using products supportingdistributed memory structures aka. DataGrids – like Hazelcast, Infinispan,Terracotta, GridGain, Coherence etc.)BERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  54. 54. 4. Distribute Memory Processing to multiple JVMs• AssessmentBERLIN, May 22. 2013 | Eric Hubert - Strategy & ArchitecturePros Consdepending on nature of applicationJava heap usage per instance can bedrastically reducedif existing application, distribution mayneed some re-architectureif memory issues are the only reason tomassively scale horizontally oneshouldn‘t forget about increasedcomplexity, operational overhead andtotal memory overhead
  55. 55. 5. Custom Off-Heap Memory Management (1)• sun.misc.Unsafe(internal implementation, dangerous, non-portable, and volatile)• java.nio.ByteBuffer#allocateDirect (since JDK 1.4)• Maximum size to be set with –XX:MaxDirectMemorySize=• You have to use some serialization/deserialization mechanism• Java‘s default serialization/deserialization is not very fast• Two sub strategies:– Dynamic size and merging: no memory wasted, but suffers fragmentation(synchronization at allocation/deallocation)– Fixed size buffer allocation: no fragmentation, but memory wasted• Proper cleanup not quite elegant to achieve (relies on finalizer )BERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  56. 56. 5. Custom Off-Heap Memory Management (2)BERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecturestatic {Method directBufferCleanerX = null;Method directBufferCleanerCleanX = null;boolean v;try {directBufferCleanerX =Class.forName("java.nio.DirectByteBuffer").getMethod("cleaner");directBufferCleanerX.setAccessible(true);directBufferCleanerCleanX =Class.forName("sun.misc.Cleaner").getMethod("clean");directBufferCleanerCleanX.setAccessible(true);v = true;} catch (Exception e) {v = false;}CLEAN_SUPPORTED = v;directBufferCleaner = directBufferCleanerX;directBufferCleanerClean = directBufferCleanerCleanX;}Lucene/Elastic Search Code:within inner class in interfaceorg.apache.lucene.store.bytebuffer.ByteBufferAllocatorforked to other projects like JBoss Netty
  57. 57. Custom Off-Heap Memory Management (3)• Projects and products using this strategy:– Oracle Coherence– GigaSpaces, (to be validated)– Hazelcast (Enterprise Edition)– GridGain– Terracotta BigMemory– Lucene / Elastic Search• Open Frameworks to use this strategy– Apache DirectMemory (Serialization via Protostuff)– FST - Fast SerializationBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  58. 58. 5. Custom Off-Heap Memory Management (4)• Assessment:BERLIN, May 22. 2013 | Eric Hubert - Strategy & ArchitecturePros Conscan reduce GC overhead / max pausetimesquite tricky to get right, extremeimplementations end up in own GC„useful“ usage limited to simple/flatdata structures (key-value) access;usage as medium speed tier in Cachesoff-heap allocation is a lot slower thanon Java-heap allocatoineither memory waste or fragmentationstandard heap analysis tooling does notapply, second set of tooling required
  59. 59. 6. JVM with more efficient Memory Management (1)The Ultimate JVM GC Tuning Guidejava -Xmx40gZINGBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  60. 60. 6. JVM with more efficient Memory Management (2)• Azul Zing Practical Evaluation / Comparison against Oracle HotSpot JVM• Preparation:– selected real software system as part of our platform showingsome GC issues in production– setup test environment (single JVM instance) on VM with 16 GB, 8 cores– created load test using real data captured from live systems– single test run designed to last about one and a half hours• Test Conduction:– incrementally increased load (concurrent users) until Oracle Hotspot withsome default memory configuration showing severe issues– changed memory sizing as well as GC algorithm in order to demonstrate issuesknown from real live– switched to untuned Azul ZingBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  61. 61. 6. JVM with more efficient Memory Management (3)• Oracle HotSpot 1.6.0_43-b01, 64bit – 1 GB MaxHeap - ParallelGC:-Xms768m -Xmx1024m -XX:PermSize=128m -XX:MaxPermSize=128mBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  62. 62. 6. JVM with more efficient Memory Management (4)BERLIN, May 22. 2013 | Eric Hubert - Strategy & ArchitectureOracle HotSpot 1.6.0_43-b01, 64bit – 1 GB MaxHeap - ParallelGC:
  63. 63. 6. JVM with more efficient Memory Management (5)• Oracle Hotspot 1.6.0_43-b01, 64bit – 4 GB MaxHeap, ParallelGC-Xms2048m –Xmx4096m -XX:PermSize=128m -XX:MaxPermSize=128mBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  64. 64. 6. JVM with more efficient Memory Management (6)BERLIN, May 22. 2013 | Eric Hubert - Strategy & ArchitectureOracle Hotspot 1.6.0_43-b01, 64bit – 4 GB MaxHeap, ParallelGC
  65. 65. 6. JVM with more efficient Memory Management (7)• Oracle Hotspot 1.6.0_43-b01, 64bit – 4 GB MaxHeap, CMS-Xms4096m -Xmx4096m -XX:+UseConcMarkSweepGC //PermGen unchangedBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  66. 66. 6. JVM with more efficient Memory Management (8)BERLIN, May 22. 2013 | Eric Hubert - Strategy & ArchitectureOracle Hotspot 1.6.0_43-b01, 64bit – 4 GB MaxHeap, CMS
  67. 67. 6. JVM with more efficient Memory Management (9)• Oracle Hotspot 1.6.0_43-b01, 64bit – 2 GB MaxHeap, CMS (tuned):-Xms2g -Xmx2g -Xmn256m -XX:SurvivorRatio=4 -XX:+UseConcMarkSweepGC -XX:ParallelGCThreads=16-XX:CMSInitiatingOccupancyFraction=80 -XX:+UseCMSInitiatingOccupancyOnly // PermGen unchangedBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  68. 68. 6. JVM with more efficient Memory Management (10)BERLIN, May 22. 2013 | Eric Hubert - Strategy & ArchitectureOracle Hotspot 1.6.0_43-b01, 64bit – 2 GB MaxHeap, CMS (tuned):
  69. 69. 6. JVM with more efficient Memory Management (11)• Azul Zing 1.6.0_33-ZVM_5.5.3.0-b5, 64bit – 10 GB MaxHeap, C4:-Xmx10gBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  70. 70. 6. JVM with more efficient Memory Management (12)BERLIN, May 22. 2013 | Eric Hubert - Strategy & ArchitectureAzul Zing 1.6.0_33-ZVM_5.5.3.0-b5, 64bit – 10 GB MaxHeap, C4
  71. 71. 6. JVM with more efficient Memory Management (12)• The efficiency of memory and thread management is up to each JVMimplementation• Azul Systems offers an highly optimized commercial JVM called Zing which isdesigned for low latency use with large (multi-GB heaps: 1 – > 300 GB)• It uses a special read barrier (Loaded Value Barrier) to support concurrentcompaction, concurrent remapping, and concurrent incremental updatetracing• Zing uses Generational GC, but the same base algorithm for both young andold gen: C4 (Continuously Concurrent Compacting Collector (C4)• Zing is built on top of a proprietary Loadable Linux Kernel Module(multiple Linux distributions supported: RedHat, CentOs, SLES, Ubuntu, etc.)BERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  72. 72. 6. JVM with more efficient Memory Management (13)• Comparison of peak mremap rates for 16 GB of remaps• Zing has a custom memory and thread management implementation andadds a production monitoring and management platformBERLIN, May 22. 2013 | Eric Hubert - Strategy & ArchitectureActiveThreadsStockLinuxModifiedLinuxSpeedup0 43.58 GB/sec (360ms) 4734.85 TB/sec (3us) >100,000x1 3.04 GB/sec (5s) 1488.10 TB/sec (11us) >480,000x2 1.82 GB/sec (8s) 1166.04 TB/sec (14us) >640,000x4 1.19 GB/sec (13s) 913.74 TB/sec (18us) >750,000x8 897.65 MB/sec (18s) 801.28 TB/sec (20us) >890,000x12 736.65 MB/sec (21s) 740.52 TB/sec (22us) >1,000,000x[REF_07] C4: The Continuously Concurrent Compacting Collector
  73. 73. 6. JVM with more efficient Memory Management (14)BERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture[REF_08] Understanding Zing LX Memory Use
  74. 74. 6. JVM with more efficient Memory Management (15)• AssessmentBERLIN, May 22. 2013 | Eric Hubert - Strategy & ArchitecturePros Consincredibly low pause times not available in a free, unsupported form;license costs as with other supported JVMsno GC tuning required if notaiming for microsecondpause timesrequires Azul Linux Kernel Module (dependent onLinux distribution and Kernel ABI/signature changepolicy some update restrictions and increasedoperational effort possible)predictable worst case pausetimesrequires more memory to work efficiently (reservedfor Zing usage even if JVM is not running)supports multi-GB largeheapsa delay in which features in the major Oracle JavaHotspot JVM releases are available in Zing, which isaimed to be further reduced in the futuresophisticated monitoring inproduction(certain CPU requirements, but fulfilled by allmodern commodity server CPUs)
  75. 75. Future Perspectives (1)• Garbage Collection and performance on virtualized environments isamong the hot future topics• Oracle currently busy with GC „convergence“(merging sources of former Bea JRockit and Sun Hotspot JVMs andtooling), stabilization and performance improvements of G1 (futurestandard?, only one GC „framework“ instead of currently three)• General Goals according to talks to vendors like Oracle and IBM:– Solve linear scaling problem (200ms @ 1GB → 20s @ 100GB)(partly caused as result of deferring of “expensive operations”)– Scale using result-based, concurrent and incremental (through partitionedheap) garbage collection– More flexible utilization of hardware (different data stores, SSD etc.)BERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  76. 76. Future Perspectives (2)• No interest of Linux Kernel community to integrate Azul’s improvements(consequently they gave up on this)• Future will show to what extend Azul Systems will actively participate inthe OpenJDK project and how this will influence upcoming Java SEversions• The gap between Azul Zing and all other JVM implementations in terms ofmemory model efficiency seems to be large and not likely to be closedanytime soonBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  77. 77. Discussion / QuestionsBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  78. 78. References• [REF_01] Systems Computing - Understanding CPU Caching and Performanceby Jon "Hannibal" Stokes• [REF_02] Death By Pauses - Devoxx France 2012by Frank Pavageau• [REF_03] Memory Management in the Java HotSpot Virtual Machineby Sun Microsystems• [REF_04] Java Reference Objectsby Keith D. Gregory• [REF_05] The Art of Garbage Collection Tuningby Angelika Langer & Klaus Kreft• [REF_06] Java7 Garbage Collector G1by Antons Kranga• [REF_07] C4: The Continuously Concurrent Compacting Collectorby Azul Systems (Gil Tene, Balaji Iyengar, Michael Wolf• [REF_08] Understanding Zing LX Memory Useby Azul SystemsBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  79. 79. Further Reading• Official Oracle JVM options documentation (subset)• Official Oracle JVM GC Tuning Documentation• Java Garbage Collection Analysis and Tuning• JavaOne 2012 - Gil Tene - Azul Systems - Understanding GC• Alexey Ragozin - HotSpot JVM GC Options Cheat Sheet (v2)• Alexey Ragozin - Understanding GC pauses in JVM, HotSpots minor GC• Alexey Ragozin - Understanding GC pauses in JVM, HotSpots CMS collector• Alexey Ragozin - Surviving 16GiB heap and greater• Java OutOfMemoryError – Eine Tragödie in sieben Akten• JavaOne 2012 - G1 Garbage Collector Tuning• How to Monitor Java Garbage Collection | CUBRID Blog• Everything I ever learned about jvm performance tuning (twitter)• Displaying Java’s Memory Pool Statistics with VisualVMBERLIN, May 22. 2013 | Eric Hubert - Strategy & Architecture
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×