SlideShare a Scribd company logo
1 of 90
JavaOne


          Efficient Memory and Thread
          Management in Highly Parallel
          Applications



          Phillip Koza
          IBM
JavaOne
 IBM CONFIDENTIAL

 Agenda

      Java memory overview
      Causes of excessive java heap memory usage
      Estimating memory usage of objects
      Runtime memory usage estimation
      Track and control memory usage
      Minimize usage of memory
      Efficiently manage threads
      Summary




                                                    2
JavaOne
 IBM CONFIDENTIAL

 Java memory usage overview

   Many different forms of memory usage by the Java Virtual
   Machine (JVM)
   OS and C runtime
   Native heap
   Method area
    Storage for objects that describe classes and methods




                                                             3
JavaOne
 IBM CONFIDENTIAL

 Java memory usage overview

   JVM stack
   Java heap
    This is where instances of java classes and arrays are
     allocated
   Excessive use of java heap memory will cause performance
    issues and ultimately, java.lang.OutOfMemoryError
   This presentation covers monitoring and controlling java
    heap memory usage:
     Better performance
    Higher application availability




                                                               4
JavaOne
 IBM CONFIDENTIAL

 Java memory usage overview

   The memory usage of a Java object is composed of:
    Overhead: This stores the reference to the class object and
     various flags. Can range from 8 – 24 bytes depending on JVM
     and 32 bit vs. 64 bit
    Memory for primitive fields
    Memory for reference fields
    Alignment bytes. In some (most) JVMs, the size of all objects
     must be a multiple of 8. Because of this, an Integer object
     uses as much memory as a Long
    Arrays use an extra 4-8 bytes to store the size of the array




                                                                    5
JavaOne
 IBM CONFIDENTIAL

 Causes of excessive memory usage

    Lack of insight into the memory usage of common and
    custom objects
    Overuse of delegation in class design
     i.e. too many objects!
    Too many threads
     This applies to the java heap usage of a thread. Most threads
      will use some amount of java heap memory.
    Incorrect use of collection classes




                                                                 6
JavaOne
 IBM CONFIDENTIAL

 Causes of excessive memory usage

    Memory leaks
     “Forgotten” or “lost” reference: application is holding a
      reference, but has forgotten about it or doesn’t know where it is
    Infrequent or delayed cleanup of no longer needed objects
    Excessive use of finalizers
     There is no guarantee when a finalizer will be run or that it will
      be run at all. An object that has a finalizer will not be garbage
      collected until its finalizer is run




                                                                     7
JavaOne
 IBM CONFIDENTIAL

 Excessive memory usage symptoms

   Degradation in performance
   High Garbage Collection (GC) costs
    GC must run more frequently when large amounts of memory
     are being used
    GC must sweep over and analyze more memory to find that
     which can be garbage collected
    JVM can thrash when memory utilization exceeds 85-90% of
     max memory




                                                            8
JavaOne
 IBM CONFIDENTIAL

 Excessive memory usage symptoms

   Java.lang.OutOfMemoryError (OOM)
    In most JVMs, by default this only kills the thread attempting to
     allocate the memory that triggered the error
    Can lead to threads hanging or behaving strangely: “Zombie
     Threads”
    JVM may crash – but perhaps not immediately!




                                                                   9
JavaOne
 IBM CONFIDENTIAL

 Long term object memory usage

    Most JVM memory allocation and GC is generational
      Default in Oracle HotSpot
      Default in IBM J9 1.6.0 build 2.6 and later, and in WepSphere App
       Server V8 and later
    In generational GC, objects are initially allocated in the “young”
     generation
    If the object survives enough garbage collections, it is moved to
     the “old” or “tenured” generation
    GC’ing the tenured generation is more expensive
    Reducing the amount of data that is put in the tenured generation
     can increase performance dramatically!




                                                                           10
JavaOne
 IBM CONFIDENTIAL

 Long term object memory usage

    Hard to say for sure what objects will become tenured, but
     likely candidates are objects:
     Stored in static structures
     Passed between threads
     Stored in member variable collections
    By definition, long term objects tie up memory for longer
    periods of time, increasing the odds of OOM
    It is still possible to run out memory with no or very few long
     term objects – but this is unlikely!




                                                                 11
JavaOne
 IBM CONFIDENTIAL

 Long term object memory usage

    How to minimize the amount of long term memory:
     Delay allocating objects until you need them
     Immediately delete objects that are no longer needed or
      unlikely to be needed again
     Convert objects to a more memory-efficient form if they won’t
      be needed for a while
     Design your objects to be memory efficient from the start!




                                                                   12
JavaOne
 IBM CONFIDENTIAL

 Estimating memory usage

    Memory usage will depend on JVM vendor and 32 bit vs. 64
     bit
    For 64 bit, must determine if references will be compressed
     Oracle HotSpot VM:
       Enabled by default in 6.0.23 & later
       Enabled by default in Java 7 if –Xmx is less than 32 GB
       Can be disabled via –XX:-UseCompressedOops
     IBM: Enabled via –Xcompressedrefs if –Xmx is less than 30GB




                                                                  13
JavaOne
 IBM CONFIDENTIAL

 Estimating memory usage

    Two ways to estimate memory usage:
     Runtime.totalMemory() – Runtime.freeMemory()
     java.lang.instrument.Instrumentation.getObjectSize()




                                                             14
JavaOne
 IBM CONFIDENTIAL

 Estimating memory usage

    The classic way to determine the memory use of an object
     (and the only way prior to Java 5) is to use the Runtime
     memory methods
    First determine how much memory is used at the start of the
     test via Runtime.totalMemory() – Runtime.freeMemory()
    Then allocate objects of a given type and see how much
     memory is now being used
    The difference between the before and after memory usage
     should be the memory used by the allocated objects




                                                              15
JavaOne
 IBM CONFIDENTIAL

 Estimating memory usage

    Unfortunately, this can be inaccurate at times. To be accurate, the
     garbage collector must run before we measure the before memory
     use
    It’s not possible to guarantee the garbage collector is run, and even
     if it is, it might not collect all the garbage objects
    So, call System.gc() at the start of the test many times to increase
     the odds it will run
    gc() problems can be minimized by allocating a large number of
     objects of the same type and then taking the average
    Note: Escape analysis may decide to allocate your objects on the
     stack if they don’t “escape” the method you are allocating them in




                                                                       16
JavaOne
 IBM CONFIDENTIAL

 Code sample
    public interface ObjectFactoryInterface {
        public Object makeObject( int index);
    }
    public class ObjectFactory implements ObjectFactoryInterface {
        public Object makeObject( int index) {
            return new Object();
        }
    }
    public class IntegerFactory implements ObjectFactoryInterface {
        public Object makeObject( int index) {
            return new Integer(123456);
        }
    }
    public class LongFactory implements ObjectFactoryInterface {
        public Object makeObject( int index) {
            return new Long(123456L);
        }
    }




                                                                      17
JavaOne
 IBM CONFIDENTIAL

 Code sample

   public class FloatFactory implements ObjectFactoryInterface {
       public Object makeObject( int index) {
           return new Float(123.456);
       }
   }
   public class DoubleFactory implements ObjectFactoryInterface {
       public Object makeObject( int index) {
           return new Double(123.456);
       }
   }
   public class BigDecimalFactory implements ObjectFactoryInterface {
       public Object makeObject( int index) {
           return new BigDecimal(“123.456”);
       }
   }




                                                                        18
JavaOne
 IBM CONFIDENTIAL

 Code sample
   public class StringFactory implements ObjectFactoryInterface {
       public Object makeObject( int index) {
           // Make a string from the provided index. This is necessary so that
           // each String object will be new and will create a new character array
           // (Strings can be interred and String character arrays can be shared
           // if they represent the same value)
           return Integer.valueOf(index).toString();
       }
   }
   public class HashMapDefaultFactory implements ObjectFactoryInterface {
       public Object makeObject(int index) {
           return new HashMap();
       }
   }
   public class IntArrayEmptyFactory implements ObjectFactoryInterface {
       public Object makeObject(int index) {
           return new int[0];
       }
   }




                                                                                     19
JavaOne
 IBM CONFIDENTIAL

 Code sample
    public class MeasureMemoryUsage {


     public static void main(String[] argv) {
         System.out.println("Measuring Memory Usage: ");
         printMemUsage( new ObjectFactory());
         printMemUsage( new IntegerFactory());
         printMemUsage( new LongFactory());
         printMemUsage( new FloatFactory());
         printMemUsage( new DoubleFactory());
         printMemUsage( new StringFactory());
         printMemUsage( new BigDecimalFactory());
         printMemUsage( new HashMapDefaultFactory());
         printMemUsage( new IntArrayEmptyFactory());
     }
     private static void printMemUsage(ObjectFactoryInterface objFactory) {
         System.out.println(objFactory.getClass().getName() + " used " + estimateMemUsage(objFactory) + " bytes per
         Object");
     }




                                                                                                             20
JavaOne
 IBM CONFIDENTIAL

 Code sample

    private static long estimateMemUsage(ObjectFactoryInterface objFactory) {
        final int numObjects = 5000;
        Object[] objArray = new Object[numObjects];
        runGc();
        long beforeMemUsed = getMemUsed();
        System.out.println("Before memory used: " + beforeMemUsed);
        for ( int i = 0; i < numObjects; i++) {
            objArray[i] = objFactory.makeObject(10000000+i);
        }
        long afterMemUsed = getMemUsed();
        System.out.println("After memory used: " + afterMemUsed);
        return Math.round(((double)(afterMemUsed - beforeMemUsed)) / (double)numObjects);
    }




                                                                                            21
JavaOne
 IBM CONFIDENTIAL

 Code sample
    private static long getMemUsed() {
            Runtime rTime = Runtime.getRuntime();
            return rTime.totalMemory() - rTime.freeMemory();
        }
            private static void runGc() {
            // Run GC 25 times to increase the odds it is really done
            for ( int i=0; i < 25; i++) {
                System.gc();
                try {
                    Thread.sleep(200);
                }
                catch (InterruptedException e) {
                }
            }
        }
    }




                                                                        22
JavaOne
 IBM CONFIDENTIAL

 Estimating memory usage

   Using java.lang.instrument.Instrumentation.getObjectSize()
    Create an instrument agent class:
     Must have a method called “premain” with the following
      signature: public static void premain(String args,
      Instrumentation inst)
    The JVM will pass an object that implements the
     Instrumentation interface to the premain method
    We can then use this object to invoke the Instrumentation
     interface method getObjectSize(object) , passing the object
     we want to estimate the memory for




                                                                23
JavaOne
 IBM CONFIDENTIAL

 Estimating memory usage

    Must package the agent class into a jar with a Manifest file
     (for example, manifestFile.txt) with the following line:
      Premain-class: <packagename>.<agentclassname>
    To run a program that can use the getObjectSize method,
     must specify the agent on the command line via
      -javaagent:<agentjarname>
    Note that getObjectSize doesn’t include the memory of
     objects referenced by the specified object. Only the
     memory for the reference will be included.
    To get a deep estimate, reflection would need to be used




                                                              24
JavaOne
 IBM CONFIDENTIAL

 Code sample
   import java.lang.instrument.*;
   public class MemInstrumentAgent {
       private static volatile Instrumentation instrObj;
       public static void premain(String args, Instrumentation instrParam) {
           instrObj = instrParam;
       }
       public static long getObjectSize(Object obj) {
           if (instrObj != null)
            return instrObj.getObjectSize(obj);
       else
           throw new IllegalStateException(“Instrumentation agent not initialized!”);
       }
   }




                                                                                        25
JavaOne
 IBM CONFIDENTIAL

 Code sample

   jar –cmf manifestFile.txt instrAgent.jar
     MemInstrumentAgent.class
   public class testMemoryUse {
       public static void main(String args[]) {
        Integer intObj = new Integer(100);
       System.out.println(“Memory estimate for Integer: “ +
       MemInstrumentAgent.getObjectSize(intObj));
   }
   java –javaagent:instrAgent.jar –cp . testMemoryUse




                                                          26
JavaOne
 IBM CONFIDENTIAL

 Estimating memory usage

  Following results are for:
    HotSpot 6.0_29 –XX-UseTLAB


    IBM J9, 1.6.0 build 2.4
      Note: -Xgcpolicy:gencon is the default as of 1.6.0
   build 2.6. Also the default in WebSphere AppServer V8
   and later.




                                                      27
JavaOne
 IBM CONFIDENTIAL

 Estimating memory usage
                 HotSpot32   HotSpot64   HotSpot         IBM32        IBM64    IBM64
                                                                               CompRefs
                                         CompOops
    Object               8          16              16           12       24         16

    Integer             16          24              16           16       32         16

    Long                16          24              24           24       32         24

    Float               16          24              16           16       32         16

    Double              16          24              24           24       32         24

    String              56          82              65           64       90         65
    (8 chars)
    BigDecimal          96         148             123       104         156        115
    (123.456)
    HashMap            120         216             128       128         224        128
    (16,0.75)
    int[0]              16          24              16           16       24         16

    int[100]           416         424             416       416         424        416




                                                                                          28
JavaOne
 IBM CONFIDENTIAL

 Estimating memory usage

    Which is better?
    Using the Runtime methods is the simplest
    Instrumentation is more accurate, so it is preferred if you
     don’t need a deep memory estimate
    For deep estimates, Instrumentation + Reflection is the
     ultimate solution, but is more complex and slower




                                                               29
JavaOne
 IBM CONFIDENTIAL

 Runtime memory usage estimation

    Your goal is to write a method to estimate the memory
     usage for objects of each class for which you will tracking
     the memory usage
     Any referenced classes will need a separate method
    Since each class needs its own estimation method, you
     don’t need deep memory estimates when testing to
     determine how much your objects will use
    Have utility classes that have pre-calculated the memory
     usage of all primitives and standard objects




                                                             30
JavaOne
 IBM CONFIDENTIAL

 Runtime memory usage estimation

    Need methods/constants for the following:
     Object overhead
     Alignment policy
     Size of a reference
     Size of a primitive (boolean, char, byte, short, int, long,
      float, double)
     Size of primitive arrays
     Size of any basic common objects used (String, Integer,
      Long, BigDecimal, Timestamp, etc)




                                                               31
JavaOne
 IBM CONFIDENTIAL

 Runtime memory usage estimation

    For collections, measure the size of an empty collection
     Then determine the overhead for an entry added to the
      collection
    It is better to overestimate then underestimate!
     Better to use a little less than is available then to run out!
     You can always refine the estimates to make them better
     And anyway, the JVM is probably using more memory than
      you think!




                                                                       32
JavaOne
 IBM CONFIDENTIAL

 Runtime memory usage estimation

     Have a MemoryUsage interface with one method:
         long getMemoryUsage()
     Have each class whose memory you are going to track
      implement this interface
     Each implementation estimates the memory usage of its
      primitives and object references
      Primitives, object overhead, and object reference memory use can
       be a static final for that class
     For any non-null object references, invoke the appropriate
      method to get the memory estimate for that object
      This includes arrays




                                                                   33
JavaOne
 IBM CONFIDENTIAL

 Runtime memory usage estimation

     Round the final result up to a multiple of 8
     If recursive references are possible, you need to remember if
      you’ve already visited an object when computing the estimate
     Shared objects should only be counted once, or not at all
     For potential “flyweight” objects (interred strings, cached
      Integers, etc.) :
       Assume not a flyweight unless you know you’ve used the
        value before or will probably use it again
         For example, a String that stores a frequently used file name




                                                                          34
JavaOne
 IBM CONFIDENTIAL

 Runtime memory usage estimation

     Most custom objects quickly devolve down to objects
      containing primitives and basic common objects
     You don’t need to do this for every class – just for those
      that you are going to track the memory use
     You can easily check the accuracy of your estimates
      against reality using the techniques described earlier
     The memory estimation is quick




                                                               35
JavaOne
 IBM CONFIDENTIAL


  Runtime memory usage estimation :
  Code Sample
   public class WorkUnit implements MemoryUsage
   {
     private static final int primitiveUsage = 16;
     private static final int thisOhead = 4;
     private static final int refsUsage = 2 * 4;
     private static final int fixedOhead = primitiveUsage + thisOhead + refsUsage;
     long userid;
     String originatingSystem;
     Request requestObj;
     long memUsage = -1;




                                                                                     36
JavaOne
 IBM CONFIDENTIAL


  Runtime memory usage estimation :
  Code Sample
     long getMemoryUsage()
     {
         if (memUsage == -1) {
           int estimate = fixedOhead;
           estimate +=
             MemUtility.getStringMemUsage(originatingSystem.length());
           estimate += requestObj.getMemoryUsage();
           memUsage = MemUtility.roundUsage(estimate);
        }
     }
   } // WorkUnit




                                                                         37
JavaOne
 IBM CONFIDENTIAL


  Runtime memory usage estimation :
  Code Sample
   public class Request implements MemoryUsage
   {
     private static final int primitiveUsage = 16;
     private static final int thisOhead = 4;
     private static final int refsUsage = 2 * 4;
     private static final int fixedOhead = primitiveUsage + thisOhead + refsUsage;
     long requestId;
     String request;
     String discountCode;
     long memUsage = -1;




                                                                                     38
JavaOne
 IBM CONFIDENTIAL


  Runtime memory usage estimation :
  Code Sample
      long getMemoryUsage()
      {
          if (memUsage == -1) {
            int estimate = fixedOhead;
            estimate +=
               MemUtility.getStringMemUsage(request.length());
            if (discountCode != null) {
               estimate +=
                 MemUtility.getStringMemUsage(discountCode.length());
            }
            memUsage = MemUtility.roundUsage(estimate);
         }
      }
   } // Request



                                                                        39
JavaOne
 IBM CONFIDENTIAL


  Runtime memory usage estimation :
  Code Sample
   public class MemUtility
   {
       static long getStringMemoryUsage( int length)
       {
          return 40 + (length << 1);
       }
       static long roundUsage( long usage)
       {
          long usageDiv8 = usage >> 3;
          if ((usageDiv8 << 3) == usage)
             return usage;
          else
             return (usageDiv8 + 1) << 3;
       }

   }



                                                       40
JavaOne
 IBM CONFIDENTIAL

 Track and Control Memory Usage

     Have one logical pool of memory which consists of all the
      heap memory that is used by the objects whose memory
      usage will be controlled
      Avoid separate pools!
     Memory usage can be controlled “statically” or “dynamically”
     Controlling memory usage statically means we reserve the
      memory up front, when the application starts, or when a new
      thread starts
      Note that you don’t need to actually allocate the memory up front,
       you just reserve it




                                                                      41
JavaOne
 IBM CONFIDENTIAL

 Track and Control Memory Usage

     Dynamic tracking and control:
      Keep track of the total used by all threads
        Get the estimated memory usage of an object from its getMemoryUsage
         method and increment the total amount used
        When the object is no longer needed, the memory use is decremented
         from the total amount used
     Only track long term memory usage
     Don’t need to “pay” for the memory dynamically if it was
      accounted for statically
     Each thread maintains a local total that tracks its
      memory usage




                                                                          42
JavaOne
 IBM CONFIDENTIAL

 Track and Control Memory Usage

     The global total amount used is a globally
      accessible long
     All increments and decrements of the global total
      are synchronized
     Updates of the global total are buffered to minimize
      synchronization costs




                                                       43
JavaOne
 IBM CONFIDENTIAL

 Track and Control Memory Usage

     The absolute limit on the memory that can be used is the
      JVM max memory specified with the –Xmx parameter
     But dynamically tracked memory can’t have all of that
     Memory should be reserved statically for objects or buffers if:
         The application must have them
         There are not very many
         They will be used for a long time
         Static size, or a static upper limit (reserve the upper limit)




                                                                           44
JavaOne
 IBM CONFIDENTIAL

 Track and Control Memory Usage

     Examples of buffers to reserve memory for statically:
      I/O buffers
      LIFO page caches
     Need to reserve a certain amount of memory for “everything
      else” – i.e. what we are not accounting for statically or
      dynamically
      This can be one sum, a per-thread value * # of threads, or both
     Also need to reserve memory to prevent the JVM from
      thrashing
      Don’t want to allow the heap to grow past 85-90% of the JVM max
       memory
      Recommend reserve 15% of the JVM max memory




                                                                     45
JavaOne
 IBM CONFIDENTIAL

 Track and Control Memory Usage

     So, the amount of memory available for dynamically
      tracked memory is: max memory – (staticBuffers +
      thrashingOverhead + everythingElse)
     Let’s call this “globalMaxDynamicMemory”
     Every time a thread increments the global total, the
      global total is compared against
      globalMaxDynamicMemory
     If the global max is exceeded, the increment request is
      denied




                                                             46
JavaOne
 IBM CONFIDENTIAL

 Track and Control Memory Usage

     If an increment of the global total is denied, the
      requesting thread must handle it.
     A thread can handle this in a number of ways:
      Wait and periodically retry until more memory becomes
       available
      Reduce some of its existing registered memory usage –
       throw away unneeded objects, etc
      Write some of its data to disk
      Throw an exception, i.e. give up
      Get another thread to give up some memory (possible, but
       more complicated!)




                                                            47
JavaOne
 IBM CONFIDENTIAL

 Track and Control Memory Usage

     If a thread reduces its registered memory usage by the
      amount of the denied request, it doesn’t need to re-
      submit the request
     To prevent one thread from “hogging” all the memory,
      can have heuristics, such as one thread cannot have
      more than 90% of the dynamically tracked memory if
      there is more than one thread
     The decrement of a thread’s memory use should be in
      finally blocks to ensure the memory is released if the
      thread gets an uncaught exception




                                                          48
JavaOne
 IBM CONFIDENTIAL

 Soft references

     Use of soft references to control your memory use really
      only works for objects you can get again or don’t really
      need
     This may be acceptable for something like a web page
      image – if it becomes reclaimed by the GC, you just
      read the image from disk again
     Doesn’t work for objects where the only copy is in
      memory!
     No way to control the order in which softly reachable
      objects are reclaimed




                                                              49
JavaOne
 IBM CONFIDENTIAL

 Minimize Memory Usage

    Prefer arrays over more complex structures
    Collections:
     Avoid empty collections
     Size collections appropriately
     Choose correct collection
    Class structure impact on memory usage
     Avoid storing data in a separate class if all instances of your
      class need that data, and it is not needed outside of that
      class
     Store info specific to only a subset of class instances in a
      derived class or in a separate class




                                                                 50
JavaOne
 IBM CONFIDENTIAL

 Minimize Memory Usage

    Convert objects to byte arrays
     Objects that won’t be needed for a while
     Objects that are going to be written to a stream soon
    Example: objects in a hash map when most won’t be
     used for a while
     Convert to byte array
     Convert back when fetched
     Can have a mix of converted/unconverted data: convert a
      byte array once it has been fetched once
     However, the hash map won’t be able to use generics if it
      has a mix of converted/unconverted data




                                                              51
JavaOne
 IBM CONFIDENTIAL

 Minimize Memory Usage

    Cleanup unneeded objects ASAP
      Optimize for mainline processing, not exceptional processing
      For example, if possible, delete requested info as soon as send it
       back to the user – don’t wait for the confirmation response
        If response is negative, read the requested info back in
        Of course, this is only worthwhile if failures are rare!
    Use finally blocks liberally!
      Make sure any allocated memory is freed
      And if it’s dynamically tracked, be sure to decrement the memory
       usage from the global total!




                                                                      52
JavaOne
 IBM CONFIDENTIAL

 Minimize Memory Usage

    Object pooling
     Doesn’t reduce your memory usage, but reduces the cost to
      allocate and GC it
     Only worth it for simple, multi-purpose objects, such as primitive
      arrays
       Byte arrays are the best candidate as they can be used to store any kind
        of data, and don’t have to be an exact match
       Relatively easy to have a pool of byte arrays of various sizes that can be
        used to store other objects
       Store large objects (i.e. lobs) as a series of smaller byte array buffers
        instead of one large one
     If you find your pooling is getting complicated – stop! Leave
      complex memory mgmt to the JVM




                                                                             53
JavaOne
 IBM CONFIDENTIAL

 Balance and control threads

    Feed work to a thread via an ArrayBlockingQueue or your own
    synchronized buffering scheme, with a limited size
    When a thread input queue is empty, or an output queue is full, the
     thread will have to wait
      This keeps threads from getting too far ahead of each other
      The capacity of the queue/buffer determines how far ahead threads
       are allowed to get relative to each other
    To balance, keep track of the times producers and consumers must
     wait to put/take data from the queue
      Lots of consumer waits = need more producer threads
      Lots of producer waits = need more consumer threads




                                                                       54
JavaOne
 IBM CONFIDENTIAL

 Balance and control threads

    The capacity of a java blocking queue is specified as a
     number of objects
    Unless all objects placed in the queue have the same size,
     the memory usage of the queue can vary widely
    Can run out of memory if don’t limit or keep track of the
     memory usage of blocking queues




                                                                 55
JavaOne
 IBM CONFIDENTIAL

 Balance and control threads

    The solution is decorate the blocking queue with a logical
     queue
    The logical queue buffers objects in lists and adds these list
     buffers to the queue when the buffers are full
    The lists are considered full when they reach a certain
     amount of estimated memory usage
    So the queue is a queue of lists, each of which will use a
     similar amount of memory
    The maximum amount of memory the queue can use is then
     bufferSize * (queueCapacity)




                                                                  56
JavaOne
 IBM CONFIDENTIAL

 Balance and control threads

    The maximum amount of memory usage for each inter-
     thread queue is statically reserved when each queue is
     created, and released when those threads stop
    This has the additional benefit of reducing the number of
     puts/gets to the underlying queue, which reduces the
     synchronization expense
    Also bases the amount of queued-up work for a thread on
     the amount of memory that work consumes, instead of a
     number of requests
    This is probably a better metric, since requests that
     consume more memory probably take longer to process




                                                                 57
JavaOne
 IBM CONFIDENTIAL

 Waiting/Flushing

    A thread needs to flush a buffered output queue if it is idle
    or has to wait
    Waiting indefinitely is dangerous!
     Try to NEVER wait on anything indefinitely
     If you are waiting for another thread to do something, and it
      dies, you will be waiting forever
     So, use loops+timeouts whenever possible
       When wake up, check if anyone wants you to stop, process a more
        urgent request, etc.
       If not, re-issue request




                                                                      58
JavaOne
 IBM CONFIDENTIAL

  Handle OutOfMemoryError

    Despite your best efforts, your app may still run out of
     memory
    There are many reasons for this:
     Tracked memory usage estimates are too low
     Untracked memory usage was larger or was utilized longer
      than expected
     Short term memory usage burst overwhelmed the GC
     Memory Leaks
     Too many threads, etc




                                                                 59
JavaOne
 IBM CONFIDENTIAL

  Handle OutOfMemoryError

    If an OOM error occurs, you will probably need a heap dump
     to determine why!
    IBM JVM enables heap dumps on OOM by default
     To disable, set environment variable
      IBM_HEAPDUMP_OUTOFMEMORY=FALSE
    Oracle HotSpot disables heap dumps on OOM by default
     To enable: -XX+HeapDumpOnOutOfMemoryError
     To see the value of all flags, specify: -XX:+PrintFlagsFinal




                                                                     60
JavaOne
 IBM CONFIDENTIAL

  Handle OutOfMemoryError

    When an OutOfMemoryError occurs, this is only guaranteed
     to stop the thread that triggered it
    This can lead to “zombie threads”
     Threads that were dependent on the thread that died can hang
     The death of the thread getting the OOM might be sufficient to
      avoid another OOM, but not enough to keep the JVM from
      thrashing, thus preventing remaining threads from doing
      anything
    Need a mechanism to bring down related threads




                                                                 61
JavaOne
 IBM CONFIDENTIAL

  Handle OutOfMemoryError

    Bring down related threads with stop request objects
    Bring down the JVM if the thread that dies is a “critical”
     thread – i.e. it is essential for your application
    Or if zombie threads are a big problem, can have your
     uncaught exception handler bring down the entire JVM if the
     exception is OOM




                                                                  62
JavaOne
 IBM CONFIDENTIAL

 Stopping

      Requesting stop via flag objects
      Have a stop request object that indicates thread should stop
       Classes check regularly, and always before sleeping, waiting,
        I/O, putting/getting from a blocking queue, etc.
       Works the same regardless of whether the class is running as
        a thread or not
       Stop request object can indicate different types of stop: stop
        immediate, stop gracefully (finish work in progress), etc.




                                                                     63
JavaOne
 IBM CONFIDENTIAL

 Stopping

      Have a stop request object that indicates thread should stop
       Can have a single stop request object for all related threads
         But this only allows all of them to be stopped at once
       For finer granularity of stop control, have a higher level stop
        request object with references to thread stop request objects
       Setting of stop request object must be synchronized – but this
        does not occur very often




                                                                      64
JavaOne
 IBM CONFIDENTIAL

 Stopping

     Have a request stop object that indicates thread
      should stop
      Reading of the stop request object will occur
       frequently
      Reading stop request object does not need to be
       synchronized for most applications
         It is not critical to see a stop request instantaneously, and
          synchronization doesn’t guarantee this anyway
         The stop request should be a boolean or enum, so updates
          of it are atomic




                                                                     65
JavaOne
 IBM CONFIDENTIAL

 Stopping

     Detecting a thread has actually stopped
      Could use Thread.isAlive() to detect if a thread has
       stopped
         This requires a handle to the thread
         Presumes the class is running as a thread
         Won’t work for higher abstractions




                                                         66
JavaOne
 IBM CONFIDENTIAL

 Stopping

      Detecting a thread has actually stopped
       Better to make this abstract
       Separate Runnable class from class that does the real work
       Runnable class invokes the work class
       The Runnable class has an “isRunning” object that tracks the
        run status
       All threads use this common Runnable class




                                                                  67
JavaOne
 IBM CONFIDENTIAL

 Stopping

     Detecting a thread has actually stopped
      If a graceful stop is requested, threads must wait for
       all of their producer threads to stop before they can
       stop
        This allows in-flight data to be processed
      Unexpected thread termination
       Just set the stop request object to bring down all related
        threads




                                                                     68
JavaOne
 IBM CONFIDENTIAL

 Stopping

      Returning control to the user
       Controller or highest level thread must make sure all threads
        are truly stopped before returning
       Strange things can happen if you try to restart and some
        threads from a previous incarnation are still running!
       Could call thread.join()
         But must have handle to all threads, and can’t be responsive to
          other requests
       Better for the controller to query the isRunning flag of all thread
        classes and wait for all to be stopped.




                                                                            69
JavaOne
 IBM CONFIDENTIAL

 Example: buffering, flushing, stopping

    work()
    {
      while (!checkForStopRequests()) {
       int maxBufSize = 102400;
       int curBufMemUsage = 0;
       int outBufCapacity = 10;
       ArrayList<WorkUnit> outBuf = new ArrayList<WorkUnit>(outBufCapacity);
       ArrayList<WorkUnit> inBuf = inQueue.poll(200,TimeUnit.MILLISECONDS);
       // If have to wait for input, flush output
       if (inBuf == null) {
          if (outBuf.size() > 0) {
               flushOutput();
          }
       }




                                                                               70
JavaOne
 IBM CONFIDENTIAL

   Example: buffering, flushing, stopping

            else {
                for (WorkUnit workUnit : inBuf) {
                  // process workUnit
                  …
                  int unitMemUsage = workUnit.getMemoryUsage();
                  if (curBufMemUsage + unitMemUsage > maxBufSize) {
                    flushOutput(outBuf);
                    curBufMemUsage = 0;
                  }
                  else {
                     curBufMemUsage += unitMemUsage;
                     outBuf.add(workUnit);
                 }
            }
        }
    }




                                                                      71
JavaOne
 IBM CONFIDENTIAL

   Example: buffering, flushing, stopping

    flushOutput(ArrayList<WorkUnit> outBuf) {
       boolean success = false;
       while (!success && !checkForStopRequests()) {
         success = outQueue.offer(outBuf, 200,TimeUnit.MILLISECONDS);
       }
    }


    Boolean checkForStopRequests() {
      // check for stop requests. Basic handling is:
      if (stop requested) {
         threadRunnableObj.isRunning = false;
         return true; // stop this thread
      }
      return false;
    }




                                                                        72
JavaOne
 IBM CONFIDENTIAL

 Uncaught exception handling

    Proper handling for uncaught exceptions is absolutely
     critical !
     Uncaught exceptions are the leading cause of unexpected
      thread termination
     If any threads are dependent on others, if one goes down, the
      remaining threads can hang!
     Hung threads can be hard to detect
     The best way to handle this is to prevent it!




                                                                73
JavaOne
 IBM CONFIDENTIAL

 Uncaught exception handling

    By default, uncaught exceptions are handled by the default
     uncaught exception handler
     This just writes the stack trace to standard error
     This is rarely acceptable!
       Stack trace may be lost
       If class tracks its run status, this will be inaccurate after an uncaught
        exception: run status will still be ‘running’
       What about threads that are dependent on this thread?




                                                                            74
JavaOne
 IBM CONFIDENTIAL

 Uncaught exception handling
    Ensure you have handling for all uncaught exceptions
    Keep it simple: do what must be done, but no more
      Complex handling for uncaught exceptions could trigger other uncaught
        exceptions, which can be extremely difficult to debug!
      And since uncaught exceptions don’t occur often, the handler probably has not
        been tested as much as other modules
    All handlers, at a minimum, should:
      Log the error and stack trace somewhere durable and secure
      Alert the user so the error will be noticed
    If the thread class has a run status, the run status should be set to stopped,
     error, or some equivalent
      Requires access to the thread class state




                                                                                75
JavaOne
 IBM CONFIDENTIAL

 Uncaught exception handling

    Ideally, the handler should also alert dependent threads
    Can notify controller or parent, which can:
        Just restart the thread that got the uncaught exception
          Not recommended! Any related threads could be in an indeterminate state
        Stop all related threads (safest)
          Set the stopping request object for the related threads
          Since all threads are monitoring, this will bring them all down
          Threads won’t hang!

    Controller/parent doesn’t need an explicit message to know a
     thread has died, if it is monitoring the run status of the threads
      In fact, the controller doesn’t even need to get involved –
        Just have the uncaught exception handler set the stop request object to stop




                                                                                     76
JavaOne
 IBM CONFIDENTIAL

 Uncaught exception handling

    Where to handle?
    Three primary possibilities:
     Custom uncaught exception handler
     Catch Throwable() in run() method
     Finally block in run() method




                                          77
JavaOne
 IBM CONFIDENTIAL

 Uncaught exception handling

    Custom uncaught exception handler
    If a uncaught exception handler is declared and is in scope,
    the default uncaught exception handler is NOT invoked
    Determine scope:
     Per JVM
     Per Thread group
     Per Runnable class




                                                              78
JavaOne
 IBM CONFIDENTIAL

 Uncaught exception handling

    Custom uncaught exception handler – per JVM scope
    Logging the error and alerting the user is about all that can
     be done
      Since this is per-JVM, it does not have access to any specific
       thread classes or thread groups
    Set via the static Thread method
     setDefaultUncaughtExceptionHandler
      Thread.setDefaultUncaughtExceptionHandler(
       Thread.UncaughtExceptionHandler ueh)
        ueh is a class that must implement uncaughtException(Thread thr,
         Throwable the)




                                                                      79
JavaOne
 IBM CONFIDENTIAL

 Uncaught exception handling

    Custom uncaught exception handler – per Thread group
     Can guarantee standard handling for a group of threads by having
      their Thread instance use the same custom ThreadGroup class
     Custom ThreadGroup class extends ThreadGroup and overrides the
      ThreadGroup uncaughtException method
     Allows for common code for the thread group, but doesn’t have access
      to individual Runnable classes
     Could have the stop object for the threads in the group registered here
       set to true if an uncaught exception occurs in any thread in the group




                                                                                 80
JavaOne
 IBM CONFIDENTIAL

 Uncaught exception handling

    Custom uncaught exception handler – per Thread class
     Can have access to the Runnable class, so the handler can log
      details about the class structure, set run status, etc. before
      exiting
     Guaranteed to be invoked
    Set via the Thread method setUncaughtExceptionHandler
     setUncaughtExceptionHandler(
      Thread.UncaughtExceptionHandler ueh)
       ueh is a class that must implement uncaughtException(Thread thr,
        Throwable the)




                                                                     81
JavaOne
 IBM CONFIDENTIAL

 Uncaught exception handling

    Catch Throwable() in run() method
     Not recommended as the sole mechanism for handling
      uncaught exceptions
       If you do this, do it outside any loops in the thread run method, i.e.
        at the highest level possible.
       Otherwise, the exception will not trigger the thread to exit, which can
        cause very strange thread conditions!
         Thread can be “alive”, but constantly cycling through errors, or hanging




                                                                                82
JavaOne
 IBM CONFIDENTIAL

 Uncaught exception handling

    Finally block in run() method
      Have handled flag that is set to true at end of try block and every catch
       block
      !handled = uncaught exception
      Advantage: Can have handling customized to the invoked class
        Can set run status and stop request
      Disadvantages
        Every catch block must set “handled” flag or the checked exception will be
         treated as unhandled
        An uncaught exception handler that logs the exception is still necessary,
         since the finally block doesn’t have access to the exception




                                                                                 83
JavaOne
 IBM CONFIDENTIAL

 Uncaught exception handling

   Critical threads
    Some threads may be critical to your application
    If they stop unexpectedly, the application will not
     function
    Best to stop the entire application when this occurs
    The uncaught exception handling for the thread
     should call System.exit(<non-zero value>)




                                                           84
JavaOne
 IBM CONFIDENTIAL

 Example using finally block
   public class bulletproofRunnable implements Runnable {
    boolean isRunning;
    Worker worker;
     boolean handled;
     bulletproofRunnable(Worker worker) {
         this.worker = worker;
         handled = false;
     }




                                                            85
JavaOne
 IBM CONFIDENTIAL

 Example using finally block
       run() {
         try {
           isRunning = true;
          worker.work();
           handled = true;
         }
         finally {
            if (!handled) {
               // uncaught exception!
               isRunning = false;
               worker.setStopRequest(true);
               // have uncaught exception handler log error and stack trace to error file
            }
         }
       }

   }




                                                                                            86
JavaOne
 IBM CONFIDENTIAL

 Example using custom handler

   public class bulletproofRunnable implements Runnable,
    Thread.uncaughtExceptionHandler {
    boolean isRunning;
    Worker worker;
    bulletproofRunnable(Worker worker) {
        this.worker = worker;
    }




                                                           87
JavaOne
 IBM CONFIDENTIAL

 Example using custom handler

       public void uncaughtException {Thread thr, Throwable the)
           isRunning = false;
           worker.setStopRequest(true);
           // log error and stack trace to error file
       }
   }
   Add to where the actual thread is created:
   bulletproofThread.setUncaughtExceptionHandler(bulletproofRunnable);




                                                                         88
JavaOne
 IBM CONFIDENTIAL

 Summary

      Excessive memory usage can have a large adverse affect on
       application performance
      Estimating memory usage of the objects used by your application
       is the first step to managing the application memory usage
      Tracking and controlling long-term memory usage is essential to
       avoid OOM
      Many techniques exist to minimize memory usage
      Limiting thread input/output queue sizes by their memory usage
       balances thread memory use and helps avoid OOM
      A framework for stopping threads and consistent handling of
       uncaught exceptions is essential to avoid hanging when OOM
       errors or other unexpected exceptions occur




                                                                    89
JavaOne       Thank You




          Phillip Koza
          pkoza@us.ibm.com

More Related Content

What's hot

What's hot (20)

Java Performance and Profiling
Java Performance and ProfilingJava Performance and Profiling
Java Performance and Profiling
 
Shall we play a game?
Shall we play a game?Shall we play a game?
Shall we play a game?
 
Diagnosing Your Application on the JVM
Diagnosing Your Application on the JVMDiagnosing Your Application on the JVM
Diagnosing Your Application on the JVM
 
Introduction to the IBM Java Tools
Introduction to the IBM Java ToolsIntroduction to the IBM Java Tools
Introduction to the IBM Java Tools
 
IBM InterConnect: Java vs JavaScript for Enterprise WebApps
IBM InterConnect: Java vs JavaScript for Enterprise WebAppsIBM InterConnect: Java vs JavaScript for Enterprise WebApps
IBM InterConnect: Java vs JavaScript for Enterprise WebApps
 
IBM Monitoring and Diagnostic Tools - GCMV 2.8
IBM Monitoring and Diagnostic Tools - GCMV 2.8IBM Monitoring and Diagnostic Tools - GCMV 2.8
IBM Monitoring and Diagnostic Tools - GCMV 2.8
 
The JVM is your friend
The JVM is your friendThe JVM is your friend
The JVM is your friend
 
High performance network programming on the jvm oscon 2012
High performance network programming on the jvm   oscon 2012 High performance network programming on the jvm   oscon 2012
High performance network programming on the jvm oscon 2012
 
Virtualization aware Java VM
Virtualization aware Java VMVirtualization aware Java VM
Virtualization aware Java VM
 
Garbage First and you
Garbage First and youGarbage First and you
Garbage First and you
 
Are you ready for cloud-native java JavaCro2019
Are you ready for cloud-native java JavaCro2019Are you ready for cloud-native java JavaCro2019
Are you ready for cloud-native java JavaCro2019
 
FPGAs in the cloud? (October 2017)
FPGAs in the cloud? (October 2017)FPGAs in the cloud? (October 2017)
FPGAs in the cloud? (October 2017)
 
Quarkus - a shrink ray to your Java Application
Quarkus - a shrink ray to your Java ApplicationQuarkus - a shrink ray to your Java Application
Quarkus - a shrink ray to your Java Application
 
WebSphere Technical University: Introduction to the Java Diagnostic Tools
WebSphere Technical University: Introduction to the Java Diagnostic ToolsWebSphere Technical University: Introduction to the Java Diagnostic Tools
WebSphere Technical University: Introduction to the Java Diagnostic Tools
 
Java profiling Do It Yourself (jug.msk.ru 2016)
Java profiling Do It Yourself (jug.msk.ru 2016)Java profiling Do It Yourself (jug.msk.ru 2016)
Java profiling Do It Yourself (jug.msk.ru 2016)
 
JavaOne 2015: From Java Code to Machine Code
JavaOne 2015: From Java Code to Machine CodeJavaOne 2015: From Java Code to Machine Code
JavaOne 2015: From Java Code to Machine Code
 
Exploring Java Heap Dumps (Oracle Code One 2018)
Exploring Java Heap Dumps (Oracle Code One 2018)Exploring Java Heap Dumps (Oracle Code One 2018)
Exploring Java Heap Dumps (Oracle Code One 2018)
 
ObjectLayout: Closing the (last?) inherent C vs. Java speed gap
ObjectLayout: Closing the (last?) inherent C vs. Java speed gapObjectLayout: Closing the (last?) inherent C vs. Java speed gap
ObjectLayout: Closing the (last?) inherent C vs. Java speed gap
 
IMCSummit 2015 - Day 2 IT Business Track - 4 Myths about In-Memory Databases ...
IMCSummit 2015 - Day 2 IT Business Track - 4 Myths about In-Memory Databases ...IMCSummit 2015 - Day 2 IT Business Track - 4 Myths about In-Memory Databases ...
IMCSummit 2015 - Day 2 IT Business Track - 4 Myths about In-Memory Databases ...
 
Lessons Learned from Deploying Apache Spark as a Service on IBM Power Systems...
Lessons Learned from Deploying Apache Spark as a Service on IBM Power Systems...Lessons Learned from Deploying Apache Spark as a Service on IBM Power Systems...
Lessons Learned from Deploying Apache Spark as a Service on IBM Power Systems...
 

Viewers also liked

JavaOne 2014: Java Debugging
JavaOne 2014: Java DebuggingJavaOne 2014: Java Debugging
JavaOne 2014: Java Debugging
Chris Bailey
 
Towards JVM Dynamic Languages Toolchain
Towards JVM Dynamic Languages ToolchainTowards JVM Dynamic Languages Toolchain
Towards JVM Dynamic Languages Toolchain
Attila Szegedi
 
An Introduction To Java Profiling
An Introduction To Java ProfilingAn Introduction To Java Profiling
An Introduction To Java Profiling
schlebu
 

Viewers also liked (20)

Everything I Ever Learned About JVM Performance Tuning @Twitter
Everything I Ever Learned About JVM Performance Tuning @TwitterEverything I Ever Learned About JVM Performance Tuning @Twitter
Everything I Ever Learned About JVM Performance Tuning @Twitter
 
Resource management in java bof6823 - java one 2012
Resource management in java   bof6823 - java one 2012Resource management in java   bof6823 - java one 2012
Resource management in java bof6823 - java one 2012
 
JavaOne 2012 CON3978 Scripting Languages on the JVM
JavaOne 2012 CON3978 Scripting Languages on the JVMJavaOne 2012 CON3978 Scripting Languages on the JVM
JavaOne 2012 CON3978 Scripting Languages on the JVM
 
Whats Next for JCA?
Whats Next for JCA?Whats Next for JCA?
Whats Next for JCA?
 
JavaOne BOF 5957 Lightning Fast Access to Big Data
JavaOne BOF 5957 Lightning Fast Access to Big DataJavaOne BOF 5957 Lightning Fast Access to Big Data
JavaOne BOF 5957 Lightning Fast Access to Big Data
 
JPA Performance Myths -- JavaOne 2013
JPA Performance Myths -- JavaOne 2013JPA Performance Myths -- JavaOne 2013
JPA Performance Myths -- JavaOne 2013
 
Hands-on Performance Tuning Lab - Devoxx Poland
Hands-on Performance Tuning Lab - Devoxx PolandHands-on Performance Tuning Lab - Devoxx Poland
Hands-on Performance Tuning Lab - Devoxx Poland
 
Java profiling Do It Yourself
Java profiling Do It YourselfJava profiling Do It Yourself
Java profiling Do It Yourself
 
JVM Multitenancy (JavaOne 2012)
JVM Multitenancy (JavaOne 2012)JVM Multitenancy (JavaOne 2012)
JVM Multitenancy (JavaOne 2012)
 
JavaOne 2014: Java Debugging
JavaOne 2014: Java DebuggingJavaOne 2014: Java Debugging
JavaOne 2014: Java Debugging
 
Profiler Guided Java Performance Tuning
Profiler Guided Java Performance TuningProfiler Guided Java Performance Tuning
Profiler Guided Java Performance Tuning
 
Debugging Your Production JVM
Debugging Your Production JVMDebugging Your Production JVM
Debugging Your Production JVM
 
Towards JVM Dynamic Languages Toolchain
Towards JVM Dynamic Languages ToolchainTowards JVM Dynamic Languages Toolchain
Towards JVM Dynamic Languages Toolchain
 
Hotspot Garbage Collection - The Useful Parts
Hotspot Garbage Collection - The Useful PartsHotspot Garbage Collection - The Useful Parts
Hotspot Garbage Collection - The Useful Parts
 
Java Performance Tuning
Java Performance TuningJava Performance Tuning
Java Performance Tuning
 
Java Performance Tuning
Java Performance TuningJava Performance Tuning
Java Performance Tuning
 
Real Life Java EE Performance Tuning
Real Life Java EE Performance TuningReal Life Java EE Performance Tuning
Real Life Java EE Performance Tuning
 
Windows Debugging Tools - JavaOne 2013
Windows Debugging Tools - JavaOne 2013Windows Debugging Tools - JavaOne 2013
Windows Debugging Tools - JavaOne 2013
 
An Introduction To Java Profiling
An Introduction To Java ProfilingAn Introduction To Java Profiling
An Introduction To Java Profiling
 
Three Key Concepts for Understanding JSR-352: Batch Programming for the Java ...
Three Key Concepts for Understanding JSR-352: Batch Programming for the Java ...Three Key Concepts for Understanding JSR-352: Batch Programming for the Java ...
Three Key Concepts for Understanding JSR-352: Batch Programming for the Java ...
 

Similar to Efficient Memory and Thread Management in Highly Parallel Java Applications

A begineers guide of JAVA - Getting Started
 A begineers guide of JAVA - Getting Started A begineers guide of JAVA - Getting Started
A begineers guide of JAVA - Getting Started
Rakesh Madugula
 
Security Applications For Emulation
Security Applications For EmulationSecurity Applications For Emulation
Security Applications For Emulation
Silvio Cesare
 
Scotas - Oracle Open World Sao Pablo
Scotas - Oracle Open World Sao PabloScotas - Oracle Open World Sao Pablo
Scotas - Oracle Open World Sao Pablo
Julian Arocena
 

Similar to Efficient Memory and Thread Management in Highly Parallel Java Applications (20)

Efficient Memory and Thread Management in Highly Parallel Java Applications
Efficient Memory and Thread Management in Highly Parallel Java ApplicationsEfficient Memory and Thread Management in Highly Parallel Java Applications
Efficient Memory and Thread Management in Highly Parallel Java Applications
 
Javasession10
Javasession10Javasession10
Javasession10
 
Size of in java
Size of in javaSize of in java
Size of in java
 
Memory Heap Analysis with AppDynamics - AppSphere16
Memory Heap Analysis with AppDynamics - AppSphere16Memory Heap Analysis with AppDynamics - AppSphere16
Memory Heap Analysis with AppDynamics - AppSphere16
 
Performance Tuning - Memory leaks, Thread deadlocks, JDK tools
Performance Tuning -  Memory leaks, Thread deadlocks, JDK toolsPerformance Tuning -  Memory leaks, Thread deadlocks, JDK tools
Performance Tuning - Memory leaks, Thread deadlocks, JDK tools
 
Java programing considering performance
Java programing considering performanceJava programing considering performance
Java programing considering performance
 
Java virtual machine
Java virtual machineJava virtual machine
Java virtual machine
 
A begineers guide of JAVA - Getting Started
 A begineers guide of JAVA - Getting Started A begineers guide of JAVA - Getting Started
A begineers guide of JAVA - Getting Started
 
Ahieving Performance C#
Ahieving Performance C#Ahieving Performance C#
Ahieving Performance C#
 
Heapoff memory wtf
Heapoff memory wtfHeapoff memory wtf
Heapoff memory wtf
 
performance optimization: Memory
performance optimization: Memoryperformance optimization: Memory
performance optimization: Memory
 
Memory Leak In java
Memory Leak In javaMemory Leak In java
Memory Leak In java
 
Troubleshooting XLConnect - OutofMemoryError(java) GC overhead limit exceeded
Troubleshooting XLConnect - OutofMemoryError(java) GC overhead limit exceededTroubleshooting XLConnect - OutofMemoryError(java) GC overhead limit exceeded
Troubleshooting XLConnect - OutofMemoryError(java) GC overhead limit exceeded
 
Simple tweaks to get the most out of your jvm
Simple tweaks to get the most out of your jvmSimple tweaks to get the most out of your jvm
Simple tweaks to get the most out of your jvm
 
Security Applications For Emulation
Security Applications For EmulationSecurity Applications For Emulation
Security Applications For Emulation
 
ContainerWorkloadwithSemeru.pdf
ContainerWorkloadwithSemeru.pdfContainerWorkloadwithSemeru.pdf
ContainerWorkloadwithSemeru.pdf
 
Simple tweaks to get the most out of your JVM
Simple tweaks to get the most out of your JVMSimple tweaks to get the most out of your JVM
Simple tweaks to get the most out of your JVM
 
Scotas - Oracle Open World Sao Pablo
Scotas - Oracle Open World Sao PabloScotas - Oracle Open World Sao Pablo
Scotas - Oracle Open World Sao Pablo
 
MacRuby for Fun and Profit
MacRuby for Fun and ProfitMacRuby for Fun and Profit
MacRuby for Fun and Profit
 
Jvm internal detail
Jvm internal detailJvm internal detail
Jvm internal detail
 

Recently uploaded

Recently uploaded (20)

Easier, Faster, and More Powerful – Notes Document Properties Reimagined
Easier, Faster, and More Powerful – Notes Document Properties ReimaginedEasier, Faster, and More Powerful – Notes Document Properties Reimagined
Easier, Faster, and More Powerful – Notes Document Properties Reimagined
 
Portal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russePortal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russe
 
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdfSimplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
 
Working together SRE & Platform Engineering
Working together SRE & Platform EngineeringWorking together SRE & Platform Engineering
Working together SRE & Platform Engineering
 
ECS 2024 Teams Premium - Pretty Secure
ECS 2024   Teams Premium - Pretty SecureECS 2024   Teams Premium - Pretty Secure
ECS 2024 Teams Premium - Pretty Secure
 
Oauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoftOauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoft
 
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsContinuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
 
What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024
 
State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!
 
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfIntroduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
 
Overview of Hyperledger Foundation
Overview of Hyperledger FoundationOverview of Hyperledger Foundation
Overview of Hyperledger Foundation
 
Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджера
 
1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT
1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT
1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT
 
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdfLinux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
 
ERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage Intacct
 
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptxWSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
 
AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101
 
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
 
Using IESVE for Room Loads Analysis - UK & Ireland
Using IESVE for Room Loads Analysis - UK & IrelandUsing IESVE for Room Loads Analysis - UK & Ireland
Using IESVE for Room Loads Analysis - UK & Ireland
 
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
 

Efficient Memory and Thread Management in Highly Parallel Java Applications

  • 1. JavaOne Efficient Memory and Thread Management in Highly Parallel Applications Phillip Koza IBM
  • 2. JavaOne IBM CONFIDENTIAL Agenda  Java memory overview  Causes of excessive java heap memory usage  Estimating memory usage of objects  Runtime memory usage estimation  Track and control memory usage  Minimize usage of memory  Efficiently manage threads  Summary 2
  • 3. JavaOne IBM CONFIDENTIAL Java memory usage overview  Many different forms of memory usage by the Java Virtual Machine (JVM)  OS and C runtime  Native heap  Method area  Storage for objects that describe classes and methods 3
  • 4. JavaOne IBM CONFIDENTIAL Java memory usage overview  JVM stack  Java heap  This is where instances of java classes and arrays are allocated  Excessive use of java heap memory will cause performance issues and ultimately, java.lang.OutOfMemoryError  This presentation covers monitoring and controlling java heap memory usage:  Better performance  Higher application availability 4
  • 5. JavaOne IBM CONFIDENTIAL Java memory usage overview  The memory usage of a Java object is composed of:  Overhead: This stores the reference to the class object and various flags. Can range from 8 – 24 bytes depending on JVM and 32 bit vs. 64 bit  Memory for primitive fields  Memory for reference fields  Alignment bytes. In some (most) JVMs, the size of all objects must be a multiple of 8. Because of this, an Integer object uses as much memory as a Long  Arrays use an extra 4-8 bytes to store the size of the array 5
  • 6. JavaOne IBM CONFIDENTIAL Causes of excessive memory usage  Lack of insight into the memory usage of common and custom objects  Overuse of delegation in class design  i.e. too many objects!  Too many threads  This applies to the java heap usage of a thread. Most threads will use some amount of java heap memory.  Incorrect use of collection classes 6
  • 7. JavaOne IBM CONFIDENTIAL Causes of excessive memory usage  Memory leaks  “Forgotten” or “lost” reference: application is holding a reference, but has forgotten about it or doesn’t know where it is  Infrequent or delayed cleanup of no longer needed objects  Excessive use of finalizers  There is no guarantee when a finalizer will be run or that it will be run at all. An object that has a finalizer will not be garbage collected until its finalizer is run 7
  • 8. JavaOne IBM CONFIDENTIAL Excessive memory usage symptoms  Degradation in performance  High Garbage Collection (GC) costs  GC must run more frequently when large amounts of memory are being used  GC must sweep over and analyze more memory to find that which can be garbage collected  JVM can thrash when memory utilization exceeds 85-90% of max memory 8
  • 9. JavaOne IBM CONFIDENTIAL Excessive memory usage symptoms  Java.lang.OutOfMemoryError (OOM)  In most JVMs, by default this only kills the thread attempting to allocate the memory that triggered the error  Can lead to threads hanging or behaving strangely: “Zombie Threads”  JVM may crash – but perhaps not immediately! 9
  • 10. JavaOne IBM CONFIDENTIAL Long term object memory usage  Most JVM memory allocation and GC is generational  Default in Oracle HotSpot  Default in IBM J9 1.6.0 build 2.6 and later, and in WepSphere App Server V8 and later  In generational GC, objects are initially allocated in the “young” generation  If the object survives enough garbage collections, it is moved to the “old” or “tenured” generation  GC’ing the tenured generation is more expensive  Reducing the amount of data that is put in the tenured generation can increase performance dramatically! 10
  • 11. JavaOne IBM CONFIDENTIAL Long term object memory usage  Hard to say for sure what objects will become tenured, but likely candidates are objects:  Stored in static structures  Passed between threads  Stored in member variable collections  By definition, long term objects tie up memory for longer periods of time, increasing the odds of OOM  It is still possible to run out memory with no or very few long term objects – but this is unlikely! 11
  • 12. JavaOne IBM CONFIDENTIAL Long term object memory usage  How to minimize the amount of long term memory:  Delay allocating objects until you need them  Immediately delete objects that are no longer needed or unlikely to be needed again  Convert objects to a more memory-efficient form if they won’t be needed for a while  Design your objects to be memory efficient from the start! 12
  • 13. JavaOne IBM CONFIDENTIAL Estimating memory usage  Memory usage will depend on JVM vendor and 32 bit vs. 64 bit  For 64 bit, must determine if references will be compressed  Oracle HotSpot VM:  Enabled by default in 6.0.23 & later  Enabled by default in Java 7 if –Xmx is less than 32 GB  Can be disabled via –XX:-UseCompressedOops  IBM: Enabled via –Xcompressedrefs if –Xmx is less than 30GB 13
  • 14. JavaOne IBM CONFIDENTIAL Estimating memory usage  Two ways to estimate memory usage:  Runtime.totalMemory() – Runtime.freeMemory()  java.lang.instrument.Instrumentation.getObjectSize() 14
  • 15. JavaOne IBM CONFIDENTIAL Estimating memory usage  The classic way to determine the memory use of an object (and the only way prior to Java 5) is to use the Runtime memory methods  First determine how much memory is used at the start of the test via Runtime.totalMemory() – Runtime.freeMemory()  Then allocate objects of a given type and see how much memory is now being used  The difference between the before and after memory usage should be the memory used by the allocated objects 15
  • 16. JavaOne IBM CONFIDENTIAL Estimating memory usage  Unfortunately, this can be inaccurate at times. To be accurate, the garbage collector must run before we measure the before memory use  It’s not possible to guarantee the garbage collector is run, and even if it is, it might not collect all the garbage objects  So, call System.gc() at the start of the test many times to increase the odds it will run  gc() problems can be minimized by allocating a large number of objects of the same type and then taking the average  Note: Escape analysis may decide to allocate your objects on the stack if they don’t “escape” the method you are allocating them in 16
  • 17. JavaOne IBM CONFIDENTIAL Code sample public interface ObjectFactoryInterface { public Object makeObject( int index); } public class ObjectFactory implements ObjectFactoryInterface { public Object makeObject( int index) { return new Object(); } } public class IntegerFactory implements ObjectFactoryInterface { public Object makeObject( int index) { return new Integer(123456); } } public class LongFactory implements ObjectFactoryInterface { public Object makeObject( int index) { return new Long(123456L); } } 17
  • 18. JavaOne IBM CONFIDENTIAL Code sample public class FloatFactory implements ObjectFactoryInterface { public Object makeObject( int index) { return new Float(123.456); } } public class DoubleFactory implements ObjectFactoryInterface { public Object makeObject( int index) { return new Double(123.456); } } public class BigDecimalFactory implements ObjectFactoryInterface { public Object makeObject( int index) { return new BigDecimal(“123.456”); } } 18
  • 19. JavaOne IBM CONFIDENTIAL Code sample public class StringFactory implements ObjectFactoryInterface { public Object makeObject( int index) { // Make a string from the provided index. This is necessary so that // each String object will be new and will create a new character array // (Strings can be interred and String character arrays can be shared // if they represent the same value) return Integer.valueOf(index).toString(); } } public class HashMapDefaultFactory implements ObjectFactoryInterface { public Object makeObject(int index) { return new HashMap(); } } public class IntArrayEmptyFactory implements ObjectFactoryInterface { public Object makeObject(int index) { return new int[0]; } } 19
  • 20. JavaOne IBM CONFIDENTIAL Code sample public class MeasureMemoryUsage { public static void main(String[] argv) { System.out.println("Measuring Memory Usage: "); printMemUsage( new ObjectFactory()); printMemUsage( new IntegerFactory()); printMemUsage( new LongFactory()); printMemUsage( new FloatFactory()); printMemUsage( new DoubleFactory()); printMemUsage( new StringFactory()); printMemUsage( new BigDecimalFactory()); printMemUsage( new HashMapDefaultFactory()); printMemUsage( new IntArrayEmptyFactory()); } private static void printMemUsage(ObjectFactoryInterface objFactory) { System.out.println(objFactory.getClass().getName() + " used " + estimateMemUsage(objFactory) + " bytes per Object"); } 20
  • 21. JavaOne IBM CONFIDENTIAL Code sample private static long estimateMemUsage(ObjectFactoryInterface objFactory) { final int numObjects = 5000; Object[] objArray = new Object[numObjects]; runGc(); long beforeMemUsed = getMemUsed(); System.out.println("Before memory used: " + beforeMemUsed); for ( int i = 0; i < numObjects; i++) { objArray[i] = objFactory.makeObject(10000000+i); } long afterMemUsed = getMemUsed(); System.out.println("After memory used: " + afterMemUsed); return Math.round(((double)(afterMemUsed - beforeMemUsed)) / (double)numObjects); } 21
  • 22. JavaOne IBM CONFIDENTIAL Code sample private static long getMemUsed() { Runtime rTime = Runtime.getRuntime(); return rTime.totalMemory() - rTime.freeMemory(); } private static void runGc() { // Run GC 25 times to increase the odds it is really done for ( int i=0; i < 25; i++) { System.gc(); try { Thread.sleep(200); } catch (InterruptedException e) { } } } } 22
  • 23. JavaOne IBM CONFIDENTIAL Estimating memory usage Using java.lang.instrument.Instrumentation.getObjectSize()  Create an instrument agent class:  Must have a method called “premain” with the following signature: public static void premain(String args, Instrumentation inst)  The JVM will pass an object that implements the Instrumentation interface to the premain method  We can then use this object to invoke the Instrumentation interface method getObjectSize(object) , passing the object we want to estimate the memory for 23
  • 24. JavaOne IBM CONFIDENTIAL Estimating memory usage  Must package the agent class into a jar with a Manifest file (for example, manifestFile.txt) with the following line: Premain-class: <packagename>.<agentclassname>  To run a program that can use the getObjectSize method, must specify the agent on the command line via -javaagent:<agentjarname>  Note that getObjectSize doesn’t include the memory of objects referenced by the specified object. Only the memory for the reference will be included.  To get a deep estimate, reflection would need to be used 24
  • 25. JavaOne IBM CONFIDENTIAL Code sample import java.lang.instrument.*; public class MemInstrumentAgent { private static volatile Instrumentation instrObj; public static void premain(String args, Instrumentation instrParam) { instrObj = instrParam; } public static long getObjectSize(Object obj) { if (instrObj != null) return instrObj.getObjectSize(obj); else throw new IllegalStateException(“Instrumentation agent not initialized!”); } } 25
  • 26. JavaOne IBM CONFIDENTIAL Code sample jar –cmf manifestFile.txt instrAgent.jar MemInstrumentAgent.class public class testMemoryUse { public static void main(String args[]) { Integer intObj = new Integer(100); System.out.println(“Memory estimate for Integer: “ + MemInstrumentAgent.getObjectSize(intObj)); } java –javaagent:instrAgent.jar –cp . testMemoryUse 26
  • 27. JavaOne IBM CONFIDENTIAL Estimating memory usage Following results are for: HotSpot 6.0_29 –XX-UseTLAB IBM J9, 1.6.0 build 2.4 Note: -Xgcpolicy:gencon is the default as of 1.6.0 build 2.6. Also the default in WebSphere AppServer V8 and later. 27
  • 28. JavaOne IBM CONFIDENTIAL Estimating memory usage HotSpot32 HotSpot64 HotSpot IBM32 IBM64 IBM64 CompRefs CompOops Object 8 16 16 12 24 16 Integer 16 24 16 16 32 16 Long 16 24 24 24 32 24 Float 16 24 16 16 32 16 Double 16 24 24 24 32 24 String 56 82 65 64 90 65 (8 chars) BigDecimal 96 148 123 104 156 115 (123.456) HashMap 120 216 128 128 224 128 (16,0.75) int[0] 16 24 16 16 24 16 int[100] 416 424 416 416 424 416 28
  • 29. JavaOne IBM CONFIDENTIAL Estimating memory usage  Which is better?  Using the Runtime methods is the simplest  Instrumentation is more accurate, so it is preferred if you don’t need a deep memory estimate  For deep estimates, Instrumentation + Reflection is the ultimate solution, but is more complex and slower 29
  • 30. JavaOne IBM CONFIDENTIAL Runtime memory usage estimation  Your goal is to write a method to estimate the memory usage for objects of each class for which you will tracking the memory usage  Any referenced classes will need a separate method  Since each class needs its own estimation method, you don’t need deep memory estimates when testing to determine how much your objects will use  Have utility classes that have pre-calculated the memory usage of all primitives and standard objects 30
  • 31. JavaOne IBM CONFIDENTIAL Runtime memory usage estimation  Need methods/constants for the following:  Object overhead  Alignment policy  Size of a reference  Size of a primitive (boolean, char, byte, short, int, long, float, double)  Size of primitive arrays  Size of any basic common objects used (String, Integer, Long, BigDecimal, Timestamp, etc) 31
  • 32. JavaOne IBM CONFIDENTIAL Runtime memory usage estimation  For collections, measure the size of an empty collection  Then determine the overhead for an entry added to the collection  It is better to overestimate then underestimate!  Better to use a little less than is available then to run out!  You can always refine the estimates to make them better  And anyway, the JVM is probably using more memory than you think! 32
  • 33. JavaOne IBM CONFIDENTIAL Runtime memory usage estimation  Have a MemoryUsage interface with one method: long getMemoryUsage()  Have each class whose memory you are going to track implement this interface  Each implementation estimates the memory usage of its primitives and object references  Primitives, object overhead, and object reference memory use can be a static final for that class  For any non-null object references, invoke the appropriate method to get the memory estimate for that object  This includes arrays 33
  • 34. JavaOne IBM CONFIDENTIAL Runtime memory usage estimation  Round the final result up to a multiple of 8  If recursive references are possible, you need to remember if you’ve already visited an object when computing the estimate  Shared objects should only be counted once, or not at all  For potential “flyweight” objects (interred strings, cached Integers, etc.) :  Assume not a flyweight unless you know you’ve used the value before or will probably use it again  For example, a String that stores a frequently used file name 34
  • 35. JavaOne IBM CONFIDENTIAL Runtime memory usage estimation  Most custom objects quickly devolve down to objects containing primitives and basic common objects  You don’t need to do this for every class – just for those that you are going to track the memory use  You can easily check the accuracy of your estimates against reality using the techniques described earlier  The memory estimation is quick 35
  • 36. JavaOne IBM CONFIDENTIAL Runtime memory usage estimation : Code Sample public class WorkUnit implements MemoryUsage { private static final int primitiveUsage = 16; private static final int thisOhead = 4; private static final int refsUsage = 2 * 4; private static final int fixedOhead = primitiveUsage + thisOhead + refsUsage; long userid; String originatingSystem; Request requestObj; long memUsage = -1; 36
  • 37. JavaOne IBM CONFIDENTIAL Runtime memory usage estimation : Code Sample long getMemoryUsage() { if (memUsage == -1) { int estimate = fixedOhead; estimate += MemUtility.getStringMemUsage(originatingSystem.length()); estimate += requestObj.getMemoryUsage(); memUsage = MemUtility.roundUsage(estimate); } } } // WorkUnit 37
  • 38. JavaOne IBM CONFIDENTIAL Runtime memory usage estimation : Code Sample public class Request implements MemoryUsage { private static final int primitiveUsage = 16; private static final int thisOhead = 4; private static final int refsUsage = 2 * 4; private static final int fixedOhead = primitiveUsage + thisOhead + refsUsage; long requestId; String request; String discountCode; long memUsage = -1; 38
  • 39. JavaOne IBM CONFIDENTIAL Runtime memory usage estimation : Code Sample long getMemoryUsage() { if (memUsage == -1) { int estimate = fixedOhead; estimate += MemUtility.getStringMemUsage(request.length()); if (discountCode != null) { estimate += MemUtility.getStringMemUsage(discountCode.length()); } memUsage = MemUtility.roundUsage(estimate); } } } // Request 39
  • 40. JavaOne IBM CONFIDENTIAL Runtime memory usage estimation : Code Sample public class MemUtility { static long getStringMemoryUsage( int length) { return 40 + (length << 1); } static long roundUsage( long usage) { long usageDiv8 = usage >> 3; if ((usageDiv8 << 3) == usage) return usage; else return (usageDiv8 + 1) << 3; } } 40
  • 41. JavaOne IBM CONFIDENTIAL Track and Control Memory Usage  Have one logical pool of memory which consists of all the heap memory that is used by the objects whose memory usage will be controlled  Avoid separate pools!  Memory usage can be controlled “statically” or “dynamically”  Controlling memory usage statically means we reserve the memory up front, when the application starts, or when a new thread starts  Note that you don’t need to actually allocate the memory up front, you just reserve it 41
  • 42. JavaOne IBM CONFIDENTIAL Track and Control Memory Usage  Dynamic tracking and control:  Keep track of the total used by all threads  Get the estimated memory usage of an object from its getMemoryUsage method and increment the total amount used  When the object is no longer needed, the memory use is decremented from the total amount used  Only track long term memory usage  Don’t need to “pay” for the memory dynamically if it was accounted for statically  Each thread maintains a local total that tracks its memory usage 42
  • 43. JavaOne IBM CONFIDENTIAL Track and Control Memory Usage  The global total amount used is a globally accessible long  All increments and decrements of the global total are synchronized  Updates of the global total are buffered to minimize synchronization costs 43
  • 44. JavaOne IBM CONFIDENTIAL Track and Control Memory Usage  The absolute limit on the memory that can be used is the JVM max memory specified with the –Xmx parameter  But dynamically tracked memory can’t have all of that  Memory should be reserved statically for objects or buffers if:  The application must have them  There are not very many  They will be used for a long time  Static size, or a static upper limit (reserve the upper limit) 44
  • 45. JavaOne IBM CONFIDENTIAL Track and Control Memory Usage  Examples of buffers to reserve memory for statically:  I/O buffers  LIFO page caches  Need to reserve a certain amount of memory for “everything else” – i.e. what we are not accounting for statically or dynamically  This can be one sum, a per-thread value * # of threads, or both  Also need to reserve memory to prevent the JVM from thrashing  Don’t want to allow the heap to grow past 85-90% of the JVM max memory  Recommend reserve 15% of the JVM max memory 45
  • 46. JavaOne IBM CONFIDENTIAL Track and Control Memory Usage  So, the amount of memory available for dynamically tracked memory is: max memory – (staticBuffers + thrashingOverhead + everythingElse)  Let’s call this “globalMaxDynamicMemory”  Every time a thread increments the global total, the global total is compared against globalMaxDynamicMemory  If the global max is exceeded, the increment request is denied 46
  • 47. JavaOne IBM CONFIDENTIAL Track and Control Memory Usage  If an increment of the global total is denied, the requesting thread must handle it.  A thread can handle this in a number of ways:  Wait and periodically retry until more memory becomes available  Reduce some of its existing registered memory usage – throw away unneeded objects, etc  Write some of its data to disk  Throw an exception, i.e. give up  Get another thread to give up some memory (possible, but more complicated!) 47
  • 48. JavaOne IBM CONFIDENTIAL Track and Control Memory Usage  If a thread reduces its registered memory usage by the amount of the denied request, it doesn’t need to re- submit the request  To prevent one thread from “hogging” all the memory, can have heuristics, such as one thread cannot have more than 90% of the dynamically tracked memory if there is more than one thread  The decrement of a thread’s memory use should be in finally blocks to ensure the memory is released if the thread gets an uncaught exception 48
  • 49. JavaOne IBM CONFIDENTIAL Soft references  Use of soft references to control your memory use really only works for objects you can get again or don’t really need  This may be acceptable for something like a web page image – if it becomes reclaimed by the GC, you just read the image from disk again  Doesn’t work for objects where the only copy is in memory!  No way to control the order in which softly reachable objects are reclaimed 49
  • 50. JavaOne IBM CONFIDENTIAL Minimize Memory Usage  Prefer arrays over more complex structures  Collections:  Avoid empty collections  Size collections appropriately  Choose correct collection  Class structure impact on memory usage  Avoid storing data in a separate class if all instances of your class need that data, and it is not needed outside of that class  Store info specific to only a subset of class instances in a derived class or in a separate class 50
  • 51. JavaOne IBM CONFIDENTIAL Minimize Memory Usage  Convert objects to byte arrays  Objects that won’t be needed for a while  Objects that are going to be written to a stream soon  Example: objects in a hash map when most won’t be used for a while  Convert to byte array  Convert back when fetched  Can have a mix of converted/unconverted data: convert a byte array once it has been fetched once  However, the hash map won’t be able to use generics if it has a mix of converted/unconverted data 51
  • 52. JavaOne IBM CONFIDENTIAL Minimize Memory Usage  Cleanup unneeded objects ASAP  Optimize for mainline processing, not exceptional processing  For example, if possible, delete requested info as soon as send it back to the user – don’t wait for the confirmation response  If response is negative, read the requested info back in  Of course, this is only worthwhile if failures are rare!  Use finally blocks liberally!  Make sure any allocated memory is freed  And if it’s dynamically tracked, be sure to decrement the memory usage from the global total! 52
  • 53. JavaOne IBM CONFIDENTIAL Minimize Memory Usage  Object pooling  Doesn’t reduce your memory usage, but reduces the cost to allocate and GC it  Only worth it for simple, multi-purpose objects, such as primitive arrays  Byte arrays are the best candidate as they can be used to store any kind of data, and don’t have to be an exact match  Relatively easy to have a pool of byte arrays of various sizes that can be used to store other objects  Store large objects (i.e. lobs) as a series of smaller byte array buffers instead of one large one  If you find your pooling is getting complicated – stop! Leave complex memory mgmt to the JVM 53
  • 54. JavaOne IBM CONFIDENTIAL Balance and control threads  Feed work to a thread via an ArrayBlockingQueue or your own synchronized buffering scheme, with a limited size  When a thread input queue is empty, or an output queue is full, the thread will have to wait  This keeps threads from getting too far ahead of each other  The capacity of the queue/buffer determines how far ahead threads are allowed to get relative to each other  To balance, keep track of the times producers and consumers must wait to put/take data from the queue  Lots of consumer waits = need more producer threads  Lots of producer waits = need more consumer threads 54
  • 55. JavaOne IBM CONFIDENTIAL Balance and control threads  The capacity of a java blocking queue is specified as a number of objects  Unless all objects placed in the queue have the same size, the memory usage of the queue can vary widely  Can run out of memory if don’t limit or keep track of the memory usage of blocking queues 55
  • 56. JavaOne IBM CONFIDENTIAL Balance and control threads  The solution is decorate the blocking queue with a logical queue  The logical queue buffers objects in lists and adds these list buffers to the queue when the buffers are full  The lists are considered full when they reach a certain amount of estimated memory usage  So the queue is a queue of lists, each of which will use a similar amount of memory  The maximum amount of memory the queue can use is then bufferSize * (queueCapacity) 56
  • 57. JavaOne IBM CONFIDENTIAL Balance and control threads  The maximum amount of memory usage for each inter- thread queue is statically reserved when each queue is created, and released when those threads stop  This has the additional benefit of reducing the number of puts/gets to the underlying queue, which reduces the synchronization expense  Also bases the amount of queued-up work for a thread on the amount of memory that work consumes, instead of a number of requests  This is probably a better metric, since requests that consume more memory probably take longer to process 57
  • 58. JavaOne IBM CONFIDENTIAL Waiting/Flushing  A thread needs to flush a buffered output queue if it is idle or has to wait  Waiting indefinitely is dangerous!  Try to NEVER wait on anything indefinitely  If you are waiting for another thread to do something, and it dies, you will be waiting forever  So, use loops+timeouts whenever possible  When wake up, check if anyone wants you to stop, process a more urgent request, etc.  If not, re-issue request 58
  • 59. JavaOne IBM CONFIDENTIAL Handle OutOfMemoryError  Despite your best efforts, your app may still run out of memory  There are many reasons for this:  Tracked memory usage estimates are too low  Untracked memory usage was larger or was utilized longer than expected  Short term memory usage burst overwhelmed the GC  Memory Leaks  Too many threads, etc 59
  • 60. JavaOne IBM CONFIDENTIAL Handle OutOfMemoryError  If an OOM error occurs, you will probably need a heap dump to determine why!  IBM JVM enables heap dumps on OOM by default  To disable, set environment variable IBM_HEAPDUMP_OUTOFMEMORY=FALSE  Oracle HotSpot disables heap dumps on OOM by default  To enable: -XX+HeapDumpOnOutOfMemoryError  To see the value of all flags, specify: -XX:+PrintFlagsFinal 60
  • 61. JavaOne IBM CONFIDENTIAL Handle OutOfMemoryError  When an OutOfMemoryError occurs, this is only guaranteed to stop the thread that triggered it  This can lead to “zombie threads”  Threads that were dependent on the thread that died can hang  The death of the thread getting the OOM might be sufficient to avoid another OOM, but not enough to keep the JVM from thrashing, thus preventing remaining threads from doing anything  Need a mechanism to bring down related threads 61
  • 62. JavaOne IBM CONFIDENTIAL Handle OutOfMemoryError  Bring down related threads with stop request objects  Bring down the JVM if the thread that dies is a “critical” thread – i.e. it is essential for your application  Or if zombie threads are a big problem, can have your uncaught exception handler bring down the entire JVM if the exception is OOM 62
  • 63. JavaOne IBM CONFIDENTIAL Stopping  Requesting stop via flag objects  Have a stop request object that indicates thread should stop  Classes check regularly, and always before sleeping, waiting, I/O, putting/getting from a blocking queue, etc.  Works the same regardless of whether the class is running as a thread or not  Stop request object can indicate different types of stop: stop immediate, stop gracefully (finish work in progress), etc. 63
  • 64. JavaOne IBM CONFIDENTIAL Stopping  Have a stop request object that indicates thread should stop  Can have a single stop request object for all related threads  But this only allows all of them to be stopped at once  For finer granularity of stop control, have a higher level stop request object with references to thread stop request objects  Setting of stop request object must be synchronized – but this does not occur very often 64
  • 65. JavaOne IBM CONFIDENTIAL Stopping Have a request stop object that indicates thread should stop Reading of the stop request object will occur frequently Reading stop request object does not need to be synchronized for most applications  It is not critical to see a stop request instantaneously, and synchronization doesn’t guarantee this anyway  The stop request should be a boolean or enum, so updates of it are atomic 65
  • 66. JavaOne IBM CONFIDENTIAL Stopping Detecting a thread has actually stopped Could use Thread.isAlive() to detect if a thread has stopped  This requires a handle to the thread  Presumes the class is running as a thread  Won’t work for higher abstractions 66
  • 67. JavaOne IBM CONFIDENTIAL Stopping  Detecting a thread has actually stopped  Better to make this abstract  Separate Runnable class from class that does the real work  Runnable class invokes the work class  The Runnable class has an “isRunning” object that tracks the run status  All threads use this common Runnable class 67
  • 68. JavaOne IBM CONFIDENTIAL Stopping Detecting a thread has actually stopped If a graceful stop is requested, threads must wait for all of their producer threads to stop before they can stop This allows in-flight data to be processed  Unexpected thread termination  Just set the stop request object to bring down all related threads 68
  • 69. JavaOne IBM CONFIDENTIAL Stopping  Returning control to the user  Controller or highest level thread must make sure all threads are truly stopped before returning  Strange things can happen if you try to restart and some threads from a previous incarnation are still running!  Could call thread.join()  But must have handle to all threads, and can’t be responsive to other requests  Better for the controller to query the isRunning flag of all thread classes and wait for all to be stopped. 69
  • 70. JavaOne IBM CONFIDENTIAL Example: buffering, flushing, stopping work() { while (!checkForStopRequests()) { int maxBufSize = 102400; int curBufMemUsage = 0; int outBufCapacity = 10; ArrayList<WorkUnit> outBuf = new ArrayList<WorkUnit>(outBufCapacity); ArrayList<WorkUnit> inBuf = inQueue.poll(200,TimeUnit.MILLISECONDS); // If have to wait for input, flush output if (inBuf == null) { if (outBuf.size() > 0) { flushOutput(); } } 70
  • 71. JavaOne IBM CONFIDENTIAL Example: buffering, flushing, stopping else { for (WorkUnit workUnit : inBuf) { // process workUnit … int unitMemUsage = workUnit.getMemoryUsage(); if (curBufMemUsage + unitMemUsage > maxBufSize) { flushOutput(outBuf); curBufMemUsage = 0; } else { curBufMemUsage += unitMemUsage; outBuf.add(workUnit); } } } } 71
  • 72. JavaOne IBM CONFIDENTIAL Example: buffering, flushing, stopping flushOutput(ArrayList<WorkUnit> outBuf) { boolean success = false; while (!success && !checkForStopRequests()) { success = outQueue.offer(outBuf, 200,TimeUnit.MILLISECONDS); } } Boolean checkForStopRequests() { // check for stop requests. Basic handling is: if (stop requested) { threadRunnableObj.isRunning = false; return true; // stop this thread } return false; } 72
  • 73. JavaOne IBM CONFIDENTIAL Uncaught exception handling  Proper handling for uncaught exceptions is absolutely critical !  Uncaught exceptions are the leading cause of unexpected thread termination  If any threads are dependent on others, if one goes down, the remaining threads can hang!  Hung threads can be hard to detect  The best way to handle this is to prevent it! 73
  • 74. JavaOne IBM CONFIDENTIAL Uncaught exception handling  By default, uncaught exceptions are handled by the default uncaught exception handler  This just writes the stack trace to standard error  This is rarely acceptable!  Stack trace may be lost  If class tracks its run status, this will be inaccurate after an uncaught exception: run status will still be ‘running’  What about threads that are dependent on this thread? 74
  • 75. JavaOne IBM CONFIDENTIAL Uncaught exception handling  Ensure you have handling for all uncaught exceptions  Keep it simple: do what must be done, but no more  Complex handling for uncaught exceptions could trigger other uncaught exceptions, which can be extremely difficult to debug!  And since uncaught exceptions don’t occur often, the handler probably has not been tested as much as other modules  All handlers, at a minimum, should:  Log the error and stack trace somewhere durable and secure  Alert the user so the error will be noticed  If the thread class has a run status, the run status should be set to stopped, error, or some equivalent  Requires access to the thread class state 75
  • 76. JavaOne IBM CONFIDENTIAL Uncaught exception handling  Ideally, the handler should also alert dependent threads  Can notify controller or parent, which can:  Just restart the thread that got the uncaught exception  Not recommended! Any related threads could be in an indeterminate state  Stop all related threads (safest)  Set the stopping request object for the related threads  Since all threads are monitoring, this will bring them all down  Threads won’t hang!  Controller/parent doesn’t need an explicit message to know a thread has died, if it is monitoring the run status of the threads  In fact, the controller doesn’t even need to get involved –  Just have the uncaught exception handler set the stop request object to stop 76
  • 77. JavaOne IBM CONFIDENTIAL Uncaught exception handling  Where to handle?  Three primary possibilities:  Custom uncaught exception handler  Catch Throwable() in run() method  Finally block in run() method 77
  • 78. JavaOne IBM CONFIDENTIAL Uncaught exception handling  Custom uncaught exception handler  If a uncaught exception handler is declared and is in scope, the default uncaught exception handler is NOT invoked  Determine scope:  Per JVM  Per Thread group  Per Runnable class 78
  • 79. JavaOne IBM CONFIDENTIAL Uncaught exception handling  Custom uncaught exception handler – per JVM scope  Logging the error and alerting the user is about all that can be done  Since this is per-JVM, it does not have access to any specific thread classes or thread groups  Set via the static Thread method setDefaultUncaughtExceptionHandler  Thread.setDefaultUncaughtExceptionHandler( Thread.UncaughtExceptionHandler ueh)  ueh is a class that must implement uncaughtException(Thread thr, Throwable the) 79
  • 80. JavaOne IBM CONFIDENTIAL Uncaught exception handling  Custom uncaught exception handler – per Thread group  Can guarantee standard handling for a group of threads by having their Thread instance use the same custom ThreadGroup class  Custom ThreadGroup class extends ThreadGroup and overrides the ThreadGroup uncaughtException method  Allows for common code for the thread group, but doesn’t have access to individual Runnable classes  Could have the stop object for the threads in the group registered here  set to true if an uncaught exception occurs in any thread in the group 80
  • 81. JavaOne IBM CONFIDENTIAL Uncaught exception handling  Custom uncaught exception handler – per Thread class  Can have access to the Runnable class, so the handler can log details about the class structure, set run status, etc. before exiting  Guaranteed to be invoked  Set via the Thread method setUncaughtExceptionHandler  setUncaughtExceptionHandler( Thread.UncaughtExceptionHandler ueh)  ueh is a class that must implement uncaughtException(Thread thr, Throwable the) 81
  • 82. JavaOne IBM CONFIDENTIAL Uncaught exception handling  Catch Throwable() in run() method  Not recommended as the sole mechanism for handling uncaught exceptions  If you do this, do it outside any loops in the thread run method, i.e. at the highest level possible.  Otherwise, the exception will not trigger the thread to exit, which can cause very strange thread conditions!  Thread can be “alive”, but constantly cycling through errors, or hanging 82
  • 83. JavaOne IBM CONFIDENTIAL Uncaught exception handling  Finally block in run() method  Have handled flag that is set to true at end of try block and every catch block  !handled = uncaught exception  Advantage: Can have handling customized to the invoked class  Can set run status and stop request  Disadvantages  Every catch block must set “handled” flag or the checked exception will be treated as unhandled  An uncaught exception handler that logs the exception is still necessary, since the finally block doesn’t have access to the exception 83
  • 84. JavaOne IBM CONFIDENTIAL Uncaught exception handling Critical threads Some threads may be critical to your application If they stop unexpectedly, the application will not function Best to stop the entire application when this occurs The uncaught exception handling for the thread should call System.exit(<non-zero value>) 84
  • 85. JavaOne IBM CONFIDENTIAL Example using finally block public class bulletproofRunnable implements Runnable { boolean isRunning; Worker worker; boolean handled; bulletproofRunnable(Worker worker) { this.worker = worker; handled = false; } 85
  • 86. JavaOne IBM CONFIDENTIAL Example using finally block run() { try { isRunning = true; worker.work(); handled = true; } finally { if (!handled) { // uncaught exception! isRunning = false; worker.setStopRequest(true); // have uncaught exception handler log error and stack trace to error file } } } } 86
  • 87. JavaOne IBM CONFIDENTIAL Example using custom handler public class bulletproofRunnable implements Runnable, Thread.uncaughtExceptionHandler { boolean isRunning; Worker worker; bulletproofRunnable(Worker worker) { this.worker = worker; } 87
  • 88. JavaOne IBM CONFIDENTIAL Example using custom handler public void uncaughtException {Thread thr, Throwable the) isRunning = false; worker.setStopRequest(true); // log error and stack trace to error file } } Add to where the actual thread is created: bulletproofThread.setUncaughtExceptionHandler(bulletproofRunnable); 88
  • 89. JavaOne IBM CONFIDENTIAL Summary  Excessive memory usage can have a large adverse affect on application performance  Estimating memory usage of the objects used by your application is the first step to managing the application memory usage  Tracking and controlling long-term memory usage is essential to avoid OOM  Many techniques exist to minimize memory usage  Limiting thread input/output queue sizes by their memory usage balances thread memory use and helps avoid OOM  A framework for stopping threads and consistent handling of uncaught exceptions is essential to avoid hanging when OOM errors or other unexpected exceptions occur 89
  • 90. JavaOne Thank You Phillip Koza pkoza@us.ibm.com

Editor's Notes

  1. C runtime is for those JVMs written in C. Native heap is used for things like direct byte buffers and JNI calls.
  2. All types of memory usage can cause an Out Of Memory Error except for OS and C runtime memory usage
  3. 1. So how to java heap objects use memory?
  4. 1. OK, that’s how java uses memory. What are the typical reasons your app used more than expected? Different collections have different memory usage characteristics. For more information, attend Chris Bailey’s talk “From Java code to Java heap”, Wednesday 8:30am – 9:30am
  5. Using finalizers can lead to a problem similar to a memory leak
  6. 1. So what are the indications your app is using too much memory? Degradation in performance = both responsiveness and throughput suffer Allocating memory takes time. But allocating memory is fairly cheap compared to the impact on the garbage collector of excessive memory use! JVM thrashes as it gets harder and harder for the GC to find memory to free. Eventually, the JVM spends almost all of its time doing GC
  7. Typically when you get an OOM, it is too late! Before this error is thrown, it is likely the JVM was thrashing, spending all of its time doing GC For these reasons, you do NOT want to wait until you get an OOM to worry about your memory usage
  8. 1. Now I’d like to discuss the performance impact of long-lived objects GC’ing tenured generation typically requires a full or major collection
  9. To run out of memory with no long term objects, your application would need to be allocating very large objects or a large number of objects in a short amount of time.
  10. 1. So you’re going to want to control your memory usage. But before you can control your memory usage, you need a way to determine how much your objects will use
  11. Lots of issues with GC Escape analysis was added in Java 6
  12. We define a agent class that defines the premain method, then we have our own getObjectSize method that invokes the instrumentation interface object getObjectSize method.
  13. We jar it up, and then we can use our agent class to get memory estimates
  14. TLAB = Thread Local Allocation Blocks. –UseTLAB disables this feature
  15. IBM JVM uses from 4-8 bytes more for many common objects without compressed 64 bit references.
  16. 1. Using one of the two preceding techniques, you can estimate the amount of memory objects use in your JVM during testing
  17. 1. Now, you need to translate this knowledge into methods that will efficiently estimate the memory usage of objects while your application is running!
  18. Alignment policy: All objects aligned on 8 byte boundaries, or Only fields that must be, such as a long =&gt; Easiest to just assume all objects must be aligned on 8 byte boundaries
  19. 1. Now to calculate the memory usage of your app’s objects: A null pointer uses the same amount of memory as a real pointer!
  20. If possible, calculate the estimate on demand, when you need the estimate to track the memory usage . If an object and its referenced objects are immutable after the memory is estimated, store the estimate in the object so it won’t need to be calculated again
  21. 1. This may sound like a lot of work, but it really isn’t! The benefit is worth the effort! As mentioned earlier, long term memory usage is what is really expensive, so only track the memory usage of long term objects. The memory estimation is quick: just CPU cost for the method calls to get the memory use of referenced objects, and then sum up and return the individual cost components
  22. This example assumes your application only runs 32 bit.
  23. 1. Now that we can calculate the memory usage of an object, we can use that to track and control the memory usage of objects at runtime
  24. Only track long term memory usage: this means only track the usage of objects that are being stored in a collection or some other buffer for later use
  25. Dynamically tracked memory only gets what is left over after we have accounted for everything else. One example is memory we reserve statically
  26. A thread that reduced its memory usage because of a denied request just doesn’t deduct the memory it just freed. In effect, it’s a swap of one memory use for another. That guarantees it will “get” the memory and not some other thread.
  27. 1. Tracking and controlling memory usage is useful, but the best thing is to avoid allocating memory in the first place. Avoid empty collections by allocating them on demand
  28. 1. Now that we’ve got our memory usage under control, we’re going to switch gears here and discuss how to keep our threads under control!
  29. 1. Now, there’s an issue with the standard ArrayBlockingQueue implementations:
  30. Of course, this only works if the buffer size is larger than most objects.
  31. Probably true that larger requests take a longer time to process
  32. There will be enough other synchronization boundaries crossed to guarantee the stop request object modification will be seen: . Puts/gets to blocking queues, synchronized access to other shared structures, etc. Synchronizing on write but not on read is known as Asymmetrical Locking
  33. 1. Now, how do you know a thread has actually stopped?
  34. This guarantees all threads have a common mechanism for tracking run status. This can also be used to guarantee common handling for uncaught exceptions
  35. ArrayList is created with a relatively small capacity, because each element will be a buffer full of WorkUnit objects. Statically reserve outBufCapacity * maxBufSize bytes of memory (1 MB in this example).
  36. 1. The last thing we are going to discuss today is handling uncaught exceptions. Hung threads are hard to detect: Are they hung, or just in-between bursts of traffic? Proper monitoring statistics can help detect this, but It can take a while before you can be sure. Auto-aborting threads that are perceived to be hung will inevitably abort too soon in certain situations. So typically this requires human intervention.
  37. Don’t let the default happen to you!
  38. This is a good choice for a “default” exception handler
  39. Simplest design is to have ueh be the Runnable class
  40. This can be useful in a thread with a finally block that could trigger uncaught exceptions itself. If an uncaught exception occurs in the finally block, it will mask the original uncaught exception. You could catch Throwable and have it log the exception. You could re-throw the exception, or also have the logic to set the stop request object here. You’d still need some form of uncaught exception handling for any that occur in the finally block.
  41. The best strategy is to use a combination of all of these
  42. If the ThreadGroup uncaught exception handler is guaranteed to be invoked, then can have a separate critical thread ThreadGroup. All critical threads are part of this group. The group handler calls system.exit().