Successfully reported this slideshow.

ClassLoader Leaks

29,740 views

Published on

Are you tired of java.lang.OutOfMemoryError: PermGen space? Then this talk is for you! We'll begin with a crash course in the Java memory model in order to understand what the error message means. Then we'll look at different causes of the error and how to avoid them. We may glance at a few interesting mistakes from the Open Source world. Last but not least you'll learn how you can get rid of java.lang.OutOfMemoryError: PermGen space once and for all.

Published in: Technology
  • Be the first to comment

ClassLoader Leaks

  1. 1. Mattias Jiderhamn – java.jiderhamn.se Join the war on Mattias Jiderhamn ClassLoader leaks
  2. 2. Mattias Jiderhamn – java.jiderhamn.se java.lang.OutOfMemoryError: PermGen space :-( Metaspace
  3. 3. Mattias Jiderhamn – java.jiderhamn.se Local dev env
  4. 4. Mattias Jiderhamn – java.jiderhamn.se Continuous Deploy .war Crashing
  5. 5. Mattias Jiderhamn – java.jiderhamn.se Agenda What does it mean? Why does it happen? Where does it leak? How to avoid? OSS examples Training!
  6. 6. Mattias Jiderhamn – java.jiderhamn.se What?
  7. 7. Mattias Jiderhamn – java.jiderhamn.se JVM memory Stack Heap PermGen / Metaspace
  8. 8. Mattias Jiderhamn – java.jiderhamn.se JVM memory Per thread Local variables and method parameters Stack
  9. 9. Mattias Jiderhamn – java.jiderhamn.se JVM memory Heap Young generation Old generation / tenured space Eden space Survivor space Object instances
  10. 10. Mattias Jiderhamn – java.jiderhamn.se JVM memory Permanent Generation java.lang.Class instances etc Named before JEE and class unloading Renamed Metaspace in Java 8 Aka Method area PermGen / Metaspace
  11. 11. Mattias Jiderhamn – java.jiderhamn.se java.lang.OutOfMemoryError: PermGen space / Metaspace = Too many classes are loaded What?
  12. 12. Mattias Jiderhamn – java.jiderhamn.se Why?
  13. 13. Mattias Jiderhamn – java.jiderhamn.se Reason for OOME 1. Your application is too large -XX:MaxPermSize=256M Java 8: Metaspace auto increase by default 2. java.lang.Class instances could not be garbage collected after redeploy java.lang.OutOfMemoryError: PermGen space / Metaspace
  14. 14. Mattias Jiderhamn – java.jiderhamn.se Reference types • Strong (i.e. normal) reference –Never GC:ed if reachable • Soft reference –GC:ed before OutOfMemoryError • Weak reference (WeakHashMap) –GC:ed when no Strong or Soft refs • Phantom reference –… won’t prevent GC
  15. 15. Mattias Jiderhamn – java.jiderhamn.se Example Foo WeakHashMap Map m = new WeakHashMap(); Foo myFoo = new Foo(); m.put(myFoo, ”bar”); myFoo = null;
  16. 16. Mattias Jiderhamn – java.jiderhamn.se GC reachability GC roots
  17. 17. Mattias Jiderhamn – java.jiderhamn.se Redeploy Application Server ClassLoader app.war ClassLoader app.war’ ClassLoader app.war’’
  18. 18. Mattias Jiderhamn – java.jiderhamn.se ClassLoader refs ClassLoader Class ClassClass Instance
  19. 19. Mattias Jiderhamn – java.jiderhamn.se How leaks happen Application Server ClassLoader Class ClassClass Instance GC root
  20. 20. Mattias Jiderhamn – java.jiderhamn.se Where?
  21. 21. Mattias Jiderhamn – java.jiderhamn.se Application Server • Application Server bugs • Logging frameworks – Apache Commons Logging Unless LogFactory.release() – Log4j - some configs Unless LogManager.shutdown() – java.util.logging custom Level • Bean Validation API (JSR 303) • Unified Expression Language (javax.el) ?
  22. 22. Mattias Jiderhamn – java.jiderhamn.se GC roots • Class loaded by system ClassLoader – static field in JDK classes (java.* etc) • Live thread – Stack – local vars, method params –java.lang.Thread instance • Object held as synchronization monitor • JNI references • JVM specials…
  23. 23. Mattias Jiderhamn – java.jiderhamn.se System classes • java.sql.DriverManager • Bean introspection cache, shutdown hooks, custom default Authenticator, custom security Provider, custom MBeans, custom ThreadGroup, custom property editor, … • Reference to contextClassLoader of first caller
  24. 24. Mattias Jiderhamn – java.jiderhamn.se DriverManager
  25. 25. Mattias Jiderhamn – java.jiderhamn.se DriverManager Application Server app.war mysql-jdbc.jar JRE DriverManager ClassLoader com.mysql.jdbc.Driver 1) … … or 2) DriverManager.deregisterDriver(…)
  26. 26. Mattias Jiderhamn – java.jiderhamn.se Context shutdown public class MyCleanupListener implements javax.servlet.ServletContextListener { ... /** Called when application is undeployed */ public void contextDestroyed( ServletContextEvent servletContextEvent) { DriverManager.deregisterDriver(…); } }
  27. 27. Mattias Jiderhamn – java.jiderhamn.se Threads • Thread stack – Local variables – Method parameters • MyThread extends Thread MyRunnable implements Runnable • contextClassLoader + AccessControlContext inherited – Example HTTP 1.1 Keep-Alive-Timer
  28. 28. Mattias Jiderhamn – java.jiderhamn.se Context shutdown public class MyCleanupListener implements javax.servlet.ServletContextListener { ... /** Called when application is undeployed */ public void contextDestroyed( ServletContextEvent servletContextEvent) { DriverManager.deregisterDriver(…); // Stop threads here! } }
  29. 29. Mattias Jiderhamn – java.jiderhamn.se Stopping threads public class MyThread extends Thread { public void run() { while(true) { // Bad idea! // Do something } } }
  30. 30. Mattias Jiderhamn – java.jiderhamn.se Stopping threads public class MyThread extends Thread { private boolean running = true; public void run() { while(running) { // Until stopped // Do something } } public void shutdown() { running = false; } } private volatile boolean running = true; Heinz Kabutz / Java Specialists’ - The Law of the Blind Spot
  31. 31. Mattias Jiderhamn – java.jiderhamn.se Threads • Thread stack – Local variables – Method parameters • MyThread extends Thread MyRunnable implements Runnable • contextClassLoader + AccessControlContext inherited – Example HTTP 1.1 Keep-Alive-Timer • ThreadLocal
  32. 32. Mattias Jiderhamn – java.jiderhamn.se ThreadLocal WeakHashMap<Thread, ?> ThreadLocal Thread Foo
  33. 33. Mattias Jiderhamn – java.jiderhamn.se ThreadLocal ThreadLocal JavaDoc: ”Each thread holds an implicit reference to its copy of a thread- local variable as long as the thread is alive and the ThreadLocal instance is accessible; …”
  34. 34. Mattias Jiderhamn – java.jiderhamn.se ThreadLocal ThreadLocalMap Entry Thread ThreadLocal Foo put() set()
  35. 35. Mattias Jiderhamn – java.jiderhamn.se ThreadLocal Pooled threads: •Threads may outlive ClassLoader •ThreadLocal → ThreadGlobal! (?)
  36. 36. Mattias Jiderhamn – java.jiderhamn.se ThreadLocal ThreadLocalMap Entry Thread ThreadLocal Foo ClassClassLoader static Class
  37. 37. Mattias Jiderhamn – java.jiderhamn.se ThreadLocal ThreadLocalMap Entry Thread ThreadLocal Foo ClassClassLoader Stale entry
  38. 38. Mattias Jiderhamn – java.jiderhamn.se ThreadLocal ThreadLocalMap JavaDoc: ”However, since reference queues are not used, stale entries are guaranteed to be removed only when the table starts running out of space”
  39. 39. Mattias Jiderhamn – java.jiderhamn.se ThreadLocal •Custom value (incl references) –static ThreadLocal = leak –otherwise = unpredicted GC •Custom ThreadLocal = no leak
  40. 40. Mattias Jiderhamn – java.jiderhamn.se ThreadLocal try { myThreadLocal.set(foo); … } finally { myThreadLocal.remove(); } Always clear your ThreadLocals!
  41. 41. Mattias Jiderhamn – java.jiderhamn.se Known offenders Apache ActiveMQ Apache Axis Apache Batik Apache Commons Pool / DBCP Apache CXF AWT Toolkit Bean Validation API / JSR 303 CGLIB (Hibernate / Spring / JBoss / Apache Geronimo) dom4j EclipseLink GeoTools Google Guice Groovy GWT / javax.imageio Hessian iCal4J Infinispan IntrospectionUtils JarURLConnection Java Advanced Imaging (JAI) Javassist Java Cryptography Architecture (JCA) Java Server Faces 2 javax.security.auth.Policy/login.Configuration JGroups Logback JAXB Mozilla Rhino MVEL OpenOffice Java Uno RunTime (JURT) Oracle JDBC RMI Serialization (<= JDK 1.4.2) Spring framework Unified Expression Language / javax.el URLConnection + HTTP 1.1 XML parsing (DocumentBuilderFactory)
  42. 42. Mattias Jiderhamn – java.jiderhamn.se How?
  43. 43. Mattias Jiderhamn – java.jiderhamn.se Leak Prevention Lib • Application server independent – Version 2 core module is pure Java SE! • Covers a lot more than Tomcat – System class references avoided/cleared – Threads are stopped – ThreadLocals are cleared – Known offenders handled • Logs warnings • Apache 2 licensed – You may modify and redistribute
  44. 44. Mattias Jiderhamn – java.jiderhamn.se Leak Prevention Lib Zero runtime dependencies No required config Maven pom.xml (Servlet 3.0+): <dependency> <groupId>se.jiderhamn.classloader-leak-prevention</groupId> <artifactId>classloader-leak-prevention-servlet3</artifactId> <version>2.0.2</version> </dependency>
  45. 45. Mattias Jiderhamn – java.jiderhamn.se Tomcat Bugzilla #48895 ThreadLocalMap Thread Cleanup Thread remove() Unsafe! get() set() remove() Removed in 6.0.27 Tomcat 7.0.6+ renews the thread pool whenever an application is redeployed. Disposable threads for lifecycle events. Bugzilla #49159
  46. 46. Mattias Jiderhamn – java.jiderhamn.se Leak Prevention Lib set(null) ThreadLocalMap Entry Thread ThreadLocal Foo Stale Cleanup Thread
  47. 47. Mattias Jiderhamn – java.jiderhamn.se Open Source examples
  48. 48. Mattias Jiderhamn – java.jiderhamn.se Bean Validation API Version 1.0.0.GA javax.validation.Validation Application Server validation-api-1.0.0.GA.jar app.war hibernate-validator.jar
  49. 49. Mattias Jiderhamn – java.jiderhamn.se javax.el API javax.el.BeanELResolver
  50. 50. Mattias Jiderhamn – java.jiderhamn.se Apache CXF org.apache.cxf.transport.http.CXFAuthenticator CXF-5442
  51. 51. Mattias Jiderhamn – java.jiderhamn.se OpenOffice JURT com.sun.star.lib.util.AsynchronousFinalizer
  52. 52. Mattias Jiderhamn – java.jiderhamn.se Training! Gear up!
  53. 53. Mattias Jiderhamn – java.jiderhamn.se Free tool Eclipse Memory Analyzer aka MAT eclipse.org/mat
  54. 54. Mattias Jiderhamn – java.jiderhamn.se Heap dump Run application server with -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/temp
  55. 55. Mattias Jiderhamn – java.jiderhamn.se Links Join the fight! java.jiderhamn.se @mjiderhamn github.com/mjiderhamn • ClassLoader Leak Prevention Library • Step by step guide: acquire heap dump and analyze for leaks using Eclipse Memory Analyzer (MAT) • List of known offenders and links to bug reports • Submit reports and pull requests! ???

×