1. Mattias Jiderhamn – java.jiderhamn.se
Join the war on
Mattias Jiderhamn
ClassLoader leaks

2. Mattias Jiderhamn – java.jiderhamn.se
java.lang.OutOfMemoryError:
PermGen space
:-(
Metaspace

3. Mattias Jiderhamn – java.jiderhamn.se
Local dev env

4. Mattias Jiderhamn – java.jiderhamn.se
Continuous Deploy
.war
Crashing

5. Mattias Jiderhamn – java.jiderhamn.se
Agenda
What does it mean?
Why does it happen?
Where does it leak?
How to avoid?
OSS examples
Training!

6. Mattias Jiderhamn – java.jiderhamn.se
What?

7. Mattias Jiderhamn – java.jiderhamn.se
JVM memory
Stack
Heap
PermGen /
Metaspace

8. Mattias Jiderhamn – java.jiderhamn.se
JVM memory
Per thread
Local variables and
method parameters
Stack

9. Mattias Jiderhamn – java.jiderhamn.se
JVM memory
Heap
Young generation
Old generation /
tenured space
Eden
space
Survivor
space
Object instances

10. Mattias Jiderhamn – java.jiderhamn.se
JVM memory
Permanent Generation
java.lang.Class instances
etc
Named before JEE and
class unloading
Renamed Metaspace in Java 8
Aka Method area
PermGen /
Metaspace

11. Mattias Jiderhamn – java.jiderhamn.se
java.lang.OutOfMemoryError:
PermGen space / Metaspace
= Too many classes are loaded
What?

12. Mattias Jiderhamn – java.jiderhamn.se
Why?

13. Mattias Jiderhamn – java.jiderhamn.se
Reason for OOME
1. Your application is too large
-XX:MaxPermSize=256M
Java 8: Metaspace auto increase by default
2. java.lang.Class instances could not
be garbage collected after redeploy
java.lang.OutOfMemoryError:
PermGen space / Metaspace

14. Mattias Jiderhamn – java.jiderhamn.se
Reference types
• Strong (i.e. normal) reference
–Never GC:ed if reachable
• Soft reference
–GC:ed before OutOfMemoryError
• Weak reference (WeakHashMap)
–GC:ed when no Strong or Soft refs
• Phantom reference
–… won’t prevent GC

15. Mattias Jiderhamn – java.jiderhamn.se
Example
Foo
WeakHashMap
Map m = new WeakHashMap();
Foo myFoo = new Foo();
m.put(myFoo, ”bar”);
myFoo = null;

16. Mattias Jiderhamn – java.jiderhamn.se
GC reachability
GC roots

17. Mattias Jiderhamn – java.jiderhamn.se
Redeploy
Application Server
ClassLoader
app.war
ClassLoader
app.war’
ClassLoader
app.war’’

18. Mattias Jiderhamn – java.jiderhamn.se
ClassLoader refs
ClassLoader
Class ClassClass
Instance

19. Mattias Jiderhamn – java.jiderhamn.se
How leaks happen
Application Server
ClassLoader
Class ClassClass
Instance
GC root

20. Mattias Jiderhamn – java.jiderhamn.se
Where?

21. Mattias Jiderhamn – java.jiderhamn.se
Application Server
• Application Server bugs
• Logging frameworks
– Apache Commons Logging
Unless LogFactory.release()
– Log4j - some configs
Unless LogManager.shutdown()
– java.util.logging custom Level
• Bean Validation API (JSR 303)
• Unified Expression Language (javax.el)
?

22. Mattias Jiderhamn – java.jiderhamn.se
GC roots
• Class loaded by system ClassLoader
– static field in JDK classes (java.* etc)
• Live thread
– Stack – local vars, method params
–java.lang.Thread instance
• Object held as synchronization monitor
• JNI references
• JVM specials…

23. Mattias Jiderhamn – java.jiderhamn.se
System classes
• java.sql.DriverManager
• Bean introspection cache, shutdown hooks,
custom default Authenticator, custom
security Provider, custom MBeans, custom
ThreadGroup, custom property editor, …
• Reference to contextClassLoader of first caller

24. Mattias Jiderhamn – java.jiderhamn.se
DriverManager

25. Mattias Jiderhamn – java.jiderhamn.se
DriverManager
Application Server
app.war
mysql-jdbc.jar
JRE
DriverManager
ClassLoader
com.mysql.jdbc.Driver
1) …
… or 2)
DriverManager.deregisterDriver(…)

26. Mattias Jiderhamn – java.jiderhamn.se
Context shutdown
public class MyCleanupListener implements
javax.servlet.ServletContextListener {
...
/** Called when application is undeployed */
public void contextDestroyed(
ServletContextEvent servletContextEvent) {
DriverManager.deregisterDriver(…);
}
}

27. Mattias Jiderhamn – java.jiderhamn.se
Threads
• Thread stack
– Local variables
– Method parameters
• MyThread extends Thread
MyRunnable implements Runnable
• contextClassLoader +
AccessControlContext inherited
– Example HTTP 1.1 Keep-Alive-Timer

28. Mattias Jiderhamn – java.jiderhamn.se
Context shutdown
public class MyCleanupListener implements
javax.servlet.ServletContextListener {
...
/** Called when application is undeployed */
public void contextDestroyed(
ServletContextEvent servletContextEvent) {
DriverManager.deregisterDriver(…);
// Stop threads here!
}
}

29. Mattias Jiderhamn – java.jiderhamn.se
Stopping threads
public class MyThread extends Thread {
public void run() {
while(true) { // Bad idea!
// Do something
}
}
}

30. Mattias Jiderhamn – java.jiderhamn.se
Stopping threads
public class MyThread extends Thread {
private boolean running = true;
public void run() {
while(running) { // Until stopped
// Do something
}
}
public void shutdown() {
running = false;
}
}
private volatile boolean running = true;
Heinz Kabutz / Java Specialists’ - The Law of the Blind Spot

31. Mattias Jiderhamn – java.jiderhamn.se
Threads
• Thread stack
– Local variables
– Method parameters
• MyThread extends Thread
MyRunnable implements Runnable
• contextClassLoader +
AccessControlContext inherited
– Example HTTP 1.1 Keep-Alive-Timer
• ThreadLocal

32. Mattias Jiderhamn – java.jiderhamn.se
ThreadLocal
WeakHashMap<Thread, ?>
ThreadLocal
Thread Foo

33. Mattias Jiderhamn – java.jiderhamn.se
ThreadLocal
ThreadLocal JavaDoc:
”Each thread holds an implicit
reference to its copy of a thread-
local variable as long as the thread
is alive and the ThreadLocal
instance is accessible; …”

34. Mattias Jiderhamn – java.jiderhamn.se
ThreadLocal
ThreadLocalMap
Entry
Thread
ThreadLocal Foo
put()
set()

35. Mattias Jiderhamn – java.jiderhamn.se
ThreadLocal
Pooled threads:
•Threads may outlive ClassLoader
•ThreadLocal → ThreadGlobal!
(?)

36. Mattias Jiderhamn – java.jiderhamn.se
ThreadLocal
ThreadLocalMap
Entry
Thread
ThreadLocal Foo
ClassClassLoader
static
Class

37. Mattias Jiderhamn – java.jiderhamn.se
ThreadLocal
ThreadLocalMap
Entry
Thread
ThreadLocal Foo
ClassClassLoader
Stale entry

38. Mattias Jiderhamn – java.jiderhamn.se
ThreadLocal
ThreadLocalMap JavaDoc:
”However, since reference queues
are not used, stale entries are
guaranteed to be removed only
when the table starts running out
of space”

39. Mattias Jiderhamn – java.jiderhamn.se
ThreadLocal
•Custom value (incl references)
–static ThreadLocal = leak
–otherwise = unpredicted GC
•Custom ThreadLocal = no leak

40. Mattias Jiderhamn – java.jiderhamn.se
ThreadLocal
try {
myThreadLocal.set(foo);
…
}
finally {
myThreadLocal.remove();
}
Always clear your ThreadLocals!

41. Mattias Jiderhamn – java.jiderhamn.se
Known offenders
Apache ActiveMQ
Apache Axis
Apache Batik
Apache Commons Pool / DBCP
Apache CXF
AWT Toolkit
Bean Validation API / JSR 303
CGLIB (Hibernate / Spring / JBoss /
Apache Geronimo)
dom4j
EclipseLink
GeoTools
Google Guice
Groovy
GWT / javax.imageio
Hessian
iCal4J
Infinispan
IntrospectionUtils
JarURLConnection
Java Advanced Imaging (JAI)
Javassist
Java Cryptography Architecture (JCA)
Java Server Faces 2
javax.security.auth.Policy/login.Configuration
JGroups
Logback
JAXB
Mozilla Rhino
MVEL
OpenOffice Java Uno RunTime (JURT)
Oracle JDBC
RMI
Serialization (<= JDK 1.4.2)
Spring framework
Unified Expression Language / javax.el
URLConnection + HTTP 1.1
XML parsing (DocumentBuilderFactory)

42. Mattias Jiderhamn – java.jiderhamn.se
How?

43. Mattias Jiderhamn – java.jiderhamn.se
Leak Prevention Lib
• Application server independent
– Version 2 core module is pure Java SE!
• Covers a lot more than Tomcat
– System class references avoided/cleared
– Threads are stopped
– ThreadLocals are cleared
– Known offenders handled
• Logs warnings
• Apache 2 licensed
– You may modify and redistribute

44. Mattias Jiderhamn – java.jiderhamn.se
Leak Prevention Lib
Zero runtime dependencies
No required config
Maven pom.xml (Servlet 3.0+):
<dependency>
<groupId>se.jiderhamn.classloader-leak-prevention</groupId>
<artifactId>classloader-leak-prevention-servlet3</artifactId>
<version>2.0.2</version>
</dependency>

45. Mattias Jiderhamn – java.jiderhamn.se
Tomcat
Bugzilla #48895
ThreadLocalMap
Thread
Cleanup
Thread
remove()
Unsafe!
get()
set()
remove()
Removed in 6.0.27
Tomcat 7.0.6+ renews the thread pool
whenever an application is redeployed.
Disposable threads for lifecycle events.
Bugzilla #49159

46. Mattias Jiderhamn – java.jiderhamn.se
Leak Prevention Lib
set(null) ThreadLocalMap
Entry
Thread
ThreadLocal Foo
Stale
Cleanup
Thread

47. Mattias Jiderhamn – java.jiderhamn.se
Open Source
examples

48. Mattias Jiderhamn – java.jiderhamn.se
Bean Validation API
Version 1.0.0.GA
javax.validation.Validation
Application Server
validation-api-1.0.0.GA.jar
app.war
hibernate-validator.jar

49. Mattias Jiderhamn – java.jiderhamn.se
javax.el API
javax.el.BeanELResolver

50. Mattias Jiderhamn – java.jiderhamn.se
Apache CXF
org.apache.cxf.transport.http.CXFAuthenticator
CXF-5442

51. Mattias Jiderhamn – java.jiderhamn.se
OpenOffice JURT
com.sun.star.lib.util.AsynchronousFinalizer

52. Mattias Jiderhamn – java.jiderhamn.se
Training!
Gear up!

53. Mattias Jiderhamn – java.jiderhamn.se
Free tool
Eclipse Memory Analyzer
aka MAT
eclipse.org/mat

54. Mattias Jiderhamn – java.jiderhamn.se
Heap dump
Run application server with
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/temp

55. Mattias Jiderhamn – java.jiderhamn.se
Links
Join the fight!
java.jiderhamn.se @mjiderhamn
github.com/mjiderhamn
• ClassLoader Leak Prevention Library
• Step by step guide: acquire heap
dump and analyze for leaks using
Eclipse Memory Analyzer (MAT)
• List of known offenders and links to bug reports
• Submit reports and pull requests!
???