The document discusses strategies for addressing Metaspace OutOfMemoryErrors (OOM) when redeploying web applications. It recommends monitoring Tomcat logs and memory usage, analyzing heap dumps to find classloader leaks and duplicate classes, and making configuration changes like closing Quartz schedulers and Log4j contexts. Specific issues addressed include leaks caused by third-party JARs like log4j-web and Xerces, and weak references in WeakHashMaps. Stress testing and troubleshooting on a production system found the root cause was an outdated Xerces JAR producing classloader leaks.

1. Fight with
Metaspace
OOM

2. Hello!
(Leon Chen)
Director of Enterprise Architect
WT Microelectronics
leon.chen@wtmec.com
leon.jchen@gmail.com
2

3. Leon Chen
○ Master of NTHU CS
○ More than 20 years in Java/JEE, as programmer, architect,
developer leader and consultant, in finance/telecom domain.
■ WT Microelectronics
■ Oracle
■ Ericsson
■ Shin Kong Life Insurance Co., Ltd.
■ Smarteam Co., Ltd.
■ Industrial Technology Research Institute
3

4. • Java GC , Java Developer Day 2014
http://www.slideshare.net/leonjchen/java-gc-javadeveloperdaytw
• JVM , OOM , JCConf 2015
https://www.slideshare.net/leonjchen/jvm-oom-jcconf-2015
4

5. Fight with Metaspace OOM
web application
re-deploy
Metaspace OOM?
restart Tomcat / Application Server?
5
!

6. 1.
What is
Metaspace?
6

7. Before Java 8: PermGen
○ Put in PermGen space of heap
○ Class metadata
○ Method of a class (include bytecodes)
○ Names of the classes
○ Constant pool Information
○ Objects array and type arras associated with a class
○ Internal objects crated by the JVM
○ Information used for optimization by the compilers (JITs)
7

8. After Java 8: Metaspace
○ Remove PermGen space
○ Use native memory
○
○ … OS
○ MaxMetaspaceSize
● -XX:MaxMetaspaceSize=1G
8

9. 9
From: https://stuefe.de/posts/metaspace/what-is-metaspace/

10. 10
From: https://stuefe.de/posts/metaspace/what-is-metaspace/

11. 2.
Fight!
11

12. 12
Tuning Methodology
Monitoring
Analyze
Change

13. Monitoring: Watch Tomcat’s Log
○ Watch system out / catalina.out during undeploy/redeploy
web application
13
07-Sep-2019 21:25:15.168 WARNING [localhost-startStop-2]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
web application [test-web] appears to have started a thread named [Log4j2-TF-4-
Scheduled-2] but has failed to stop it. This is very likely to create a memory leak.
Stack trace of thread:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

14. 14

15. Monitoring: VisualVM
15

16. 16
Metaspace +
CompressedClassSpaceSize
CompressedClassSpaceSize default
= 1G

17. 17
From: https://stuefe.de/posts/metaspace/what-is-metaspace/

18. Analysis: Memory Analyzer Tool (MAT)
18
Duplicate
Classes
Reason:
• Thread not
close
• Referenced by
App Server /
JVM
Physical Memory

19. 19

20. 20

21. 21

22. 22

23. Analysis: Memory Analyzer Tool (MAT)
○ JMAP to dump heap
● jmap -dump:format=b,live,file=<filename>.bin <pid>
○ MAT
● Duplicate Classes
● Path to GC Roots
■ exclude all phantom/weak/soft etc. references
● List objects
■ with outgoing references (target object uses what objects)
■ with incoming references (what objects use target object)
23

24. Quartz
24

25. Monitoring / Analysis
25

26. Change: Close Quartrz Scheduler
○ ServletContextListener.contextDestroyed()
26
Scheduler defaultScheduler =
StdSchedulerFactory.getDefaultScheduler();
defaultScheduler.shutdown(true);

27. Change: Close Quartrz Scheduler
○ Spring
27
<bean id="testFactoryBean"
class="org.springframework…..SchedulerFactoryBean">
...
<property name="waitForJobsToCompleteOnShutdown"
value="true" />
...
</bean>

28. Log4j2
28

29. Monitoring / Analysis
29

30. Change: Add log4j-web.jar
○ https://logging.apache.org/log4j/2.x/faq.html
○ Not working! !
○ Is log4j-web.jar worked? "
30

31. Decompile log4j-web.jar
31
JD-GUI

32. Change log4j2.xml
32

33. Change log4j.xml
○ But failed to see the log …
33
!

34. Decompile log4j-web.jar
34

35. Google StatusLogger
35

36. Change log4j2.xml
36

37. Log4j2 shutdown
37

38. Log4j2 re-initial!!!
38
!

39. Check: LoggerContext
39
Eclipse:
Open Type
(Ctrl + Shift + T)
(Cmd + Shift + T)

40. Check: LoggerContext
40
Eclipse:
Insert break point

41. Check: LoggerContext
41
Start tomcat in
Eclipse
(debug mode)
Shutdown tomcat by
command line

42. Check: LoggerContext
42

43. 43

44. Log4j2 shutdown
Spring shutdown
Spring shutdown log
Log4j2 initial
Log4j2 + Spring
44

45. https://issues.apache.org/jira/browse/LOG4J2-1419
Memory leak on Tomcat shutdown
45

46. https://logging.apache.org/log4j/2.x/manual/webapp.html
46

47. 47

48. Log4j2 + Spring
BUT
48

49. Stress Testing
49
!

50. Heap Dump and Analyze
50
Path To GC Roots |
Exclude all phantom/weak/soft etc. references
!

51. Heap Dump and Analyze
51
Path To GC Roots | with all references

52. Reference
○ Strong reference
Car car = new Car ();
○ Weak reference
WeakReference<Car> weakCar = new WeakReference<>(car);
○ Soft reference
SoftReference<Car> softCar = new SoftReference<>(car);
○ Phantom reference
PhantomReference<Car> phaCar = new PhantomReference<>(car, referenceQueue);
52

53. Weak Reference
○ Look dude, I am creating this object as a weak reference. Even
though I need it, feel free to garbage collect it if you run out of
memory. I know this object can be GC'd any time and am
prepared to deal with it.
- http://www.programmr.com/blogs/what-every-java-developer-should-know-strong-and-weak-
references
○ Don’t care Weak References
53

54. Soft References
○ All soft references to softly-reachable objects are guaranteed to have
been cleared before the virtual machine throws an
OutOfMemoryError.
● https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/ref/SoftReference.html
○ That’s a lie. It was true when soft references were first introduced in
java 1.2, but from java 1.3.1 the jvm property
-XX:SoftRefLRUPolicyMSPerMB was introduced. It defaults to 1000
(milliseconds), meaning that if there’s only 10MB available heap, the
garbage collector will free references that have been used more than
10s ago.
- https://blog.shiftleft.io/understanding-jvm-soft-references-for-great-good-and-building-a-cache-244a4f7bb85d 54

55. Phantom Reference
○ Use Cases:
● It can be used instead of a finalize method, guaranteeing that the
object is not resurrected during finalization
● Detect exactly when an object has been removed from memory
○ If the garbage collector determines at a certain point in time that the
referent of a phantom reference is phantom reachable (neither
strongly, softly, nor weakly reachable, it has been finalized, and some
phantom reference refers to it), then at that time or at some later
time it will enqueue the reference..
https://stackoverflow.com/questions/53822132/java-phantomreference-vs-finalize
https://en.wikipedia.org/wiki/Phantom_reference 55
ReferenceQueue<Object> referenceQueue = new
ReferenceQueue<>();
PhantomReference<Object> phantomReference = new
PhantomReference<>(object, referenceQueue);

56. Heap Dump and Analyze
56
Path To GC Roots | exclude weak references

57. -XX:SoftRefLRUPolicyMSPerMB=0
, Metaspace OOM
GO PRODUCTION!
57
!

58. …
58

59. 59
!

60. 60

61. 61

62. java.util.WeakHashMap$Entry
62
!

63. Weak Reference
○ Look dude, I am creating this object as a weak reference. Even
though I need it, feel free to garbage collect it if you run out of
memory. I know this object can be GC'd any time and am
prepared to deal with it.
- http://www.programmr.com/blogs/what-every-java-developer-should-know-strong-and-weak-
references
○ Don’t care Weak References ???
63

64. java.util.WeakHashMap
○ Implementation note: The value objects in a WeakHashMap
are held by ordinary strong references. Thus care should be
taken to ensure that value objects do not strongly refer to
their own keys, either directly or indirectly, since that will
prevent the keys from being discarded.
- https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/WeakHashMap.html
64

65. 65

66. 66

67. 67

68. How to solve …. ?
68

69. JAXB memory leak?
○ -Dcom.sun.xml.bind.v2.bytecode.ClassTailor.noOptimize=true
○ https://stackoverflow.com/questions/44830943/metaspace-
memory-leak
○ https://stackoverflow.com/questions/33255578/old-jaxb-and-
jdk8-metaspace-outofmemory-issue/33431431Constant pool
Information
○ NO …
69

70. JAXB memory leak?
○ https://java.jiderhamn.se/2012/02/26/classloader-leaks-v-
common-mistakes-and-known-offenders/
● …JAXB Reference Implementation shipped with JDK 1.6+ …
(We are using Java 8, should be no problem …)
● …it seems that if for example you have a version of Xerces
inside your application, the factory method
(… check …)
70

71. 71

72. 72

73. 73

74. : xerceImpl-2.9.1.jar (2008) implementation
deprecated Sun API
xerceImpl
… …
74

75. Production jar
side effect?
reproduce…
75
!

76. Plan
1. Production DF_CACHE
test case
2. Dev xerceImpl.jar
1 test case
76
!

77. 77

78. 78

79. :
PRODUCTION runtime,
com.sun.xml.internal.bind.DatatypeConverterImpl.
getDatatypeFactory()
But... How ???
79
!

80. 80

81. Java Grey’s-anatomy
https://github.com/oldmanpushcart/greys-anatomy
Alibaba
81

82. 82

83. 83
stack com.sun.xml.internal.bind.DatatypeConverterImpl getDatatypeFactory

84. 3 …
84

85. 85
!

86. 86

87. 87
monitor JVM class (com.sun…..),
options unsafe true

89. test case
xerceImpl
PRODUCTION
DONE!
89
!

90. Summarize
○ Metaspace
○ Monitoring: Tomcat's Log, VisualVM
○ Analysis: Memory Analyzer Tool
○ Quartz: Close scheduler
○ Log4j2 + Spring: Decompile, Open Source troubleshooting
○ Stress Testing: Weakreference/SoftReference
○ Production Troubleshooting: xerceImpl,
java.util.WeakHashMap. Grey’s-anatomy 90

91. Tools
○ Memory Analyzer Tool
https://www.eclipse.org/mat/
○ VisualVM
https://visualvm.github.io/
○ Java Decompiler (JD, JD-GUI)
http://java-decompiler.github.io/
○ Gceasy
https://gceasy.io/
○ Java Grey’s-Anatomy
https://github.com/oldmanpushcart/greys-anatomy 91

92. Reference
○ https://blogs.oracle.com/jonthecollector/presenting-the-permanent-
generation
○ http://www.programmr.com/blogs/what-every-java-developer-
should-know-strong-and-weak-references
○ https://blog.shiftleft.io/understanding-jvm-soft-references-for-great-
good-and-building-a-cache-244a4f7bb85d
○ https://stuefe.de/posts/metaspace/what-is-metaspace/
92

93. 93
If you want the job
done right
hire a professional

94. Enterprise Architect
https://github.com/twjug/jobs/issues/8
Senior/Junior Java Developer
https://github.com/twjug/jobs/issues/9

95. THANKS!
Any question?
You can find me at
leon.jchen@gmail.com/leon.chen@wtmec.com
95

96. CREDITS
Special thanks to all the people who made and released these awesome
resources for free:
○ Presentation template by SlidesCarnival
○ Photographs by Unsplash
96