Garbage Collection, Tuning And Monitoring JVM In EBS 11i And R12
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Garbage Collection, Tuning And Monitoring JVM In EBS 11i And R12

on

  • 15,088 views

 

Statistics

Views

Total Views
15,088
Views on SlideShare
15,073
Embed Views
15

Actions

Likes
2
Downloads
541
Comments
0

2 Embeds 15

http://www.linkedin.com 12
https://www.linkedin.com 3

Accessibility

Categories

Upload Details

Uploaded via as Microsoft Word

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Garbage Collection, Tuning And Monitoring JVM In EBS 11i And R12 Document Transcript

  • 1. Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12Siddharth GandhiNov 2008Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12 1
  • 2. ContentsGarbage Collection, Tuning and Monitoring JVM in EBS 11i and R12..........................................................1 Introduction.............................................................................................................................................3 First Steps................................................................................................................................................4 Memory management.............................................................................................................................4 Garbage Collection Concepts...................................................................................................................5 Definition.............................................................................................................................................5 Generations.........................................................................................................................................5 Types of Collectors...............................................................................................................................6 Measurement......................................................................................................................................6 Few notable JVM parameters..................................................................................................................8 Monitor the JVM....................................................................................................................................10 Future changes - JRockit........................................................................................................................13 References and Quotes..........................................................................................................................14 Appendix................................................................................................................................................15Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12 2
  • 3. IntroductionIn many of our support tickets we might have encountered performance and memory issues with themiddle-tier and the ubiquitous java.lang.OutOfMemoryError(s) statement in our stack trace.We might be encountering these performance issues for many reasons impacted by hardware(specifications, networking etc) or software (OS, application versions/patches/parameters) or just byexternal parameters like user behavior, current number of active users, bandwidth etc.In the Oracle Application ‘development’ space, plugging in these performance and memory issues willmajorly boil down to JVM tuning, stopping memory leaks, JDBC optimizations and few others.In this whitepaper, we will brief upon • Garbage Collection, • Tuning JVM for better performance, • Understanding few of the JVM tuning parameters provided in Jserv.properties and opmn.xml file • Monitoring JVMThe document focuses only on the OACoreGroup JVM and not considers the JVM settings for otherOracle services like Forms & Reports, Discoverer, Configurator etc. Also assumed is that the reader hassome familiarity with Java and E-Business Suite 11i and Release 12Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12 3
  • 4. First StepsQuoting Sun’s documentation, Out of Memory (OOM) errors are usually thrown…“… when the Java Virtual Machine cannot allocate an object because it is out of memory, and no morememory could be made available by the garbage collector.”And so follows the usual fix of increasing the JVM heap size,java -Xms1024M -Xmx1024M ….But in this quick-fix world, we comfortably forget the following two important questions, 1. Why Java Virtual Machine is out of memory? 2. Why garbage collector is not able to make more memory available?Let’s remember the fact that physical memory is limited and on the other hand there are easier, faster andcheaper approaches to consider then upgrading the hardware.So before we proceed to tune our JVMs, let’s brush our basics for memory management and role ofGarbage Collection (GC)Memory management“Memory management is the process of recognizing when allocated objects are no longer needed, de-allocating (freeing) the memory used by such objects, and making it available for subsequent allocation“Explicit Vs. Automatic Memory ManagementJava employs automatic memory management unlike C and few other programming languages wherememory management is developer responsibility. Explicit management many a times results intounexpected or erroneous program behavior and crashes.Whereas automatic management is handled by a program called a garbage collector (GC). GCautomatically frees all memory no longer referenced.There are basically two ways you can alter the JVM’s behavior to improve performance. Firstly, you caninfluence memory use by changing how memory is allocated and organized by the JVM. Secondly, youcan influence how garbage collection is performed.Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12 4
  • 5. Garbage Collection ConceptsDefinitionObjects that are referenced are said to be live. Objects that are no longer referenced are considereddead and are termed garbage. The process of finding and reclaiming the space used by these objects isknown as garbage collection.The timing of garbage collection is up to the garbage collector.GenerationsMemory in the Java HotSpot virtual machine is organized into three generations: 1. Young generation, 2. Old generation(or Tenured), and 3. Permanent generation (or Perm)This generational garbage collection, takes advantage of the observation that, most programs create: • Many objects that have short lives (for example, iterators and local variables). • Some objects that have very long lifetimes (for example, high level persistent objects)The young generation consists of an "eden space" and two "survivor spaces." The VM initially assigns allobjects to the Eden space, and most objects die there. When the young generation area fills up, it causesMinor Garbage Collection. Any objects survived after Minor GC is moved to Tenured Generation.And when Tenured Generation area fills up it causes Major Garbage Collection i.e. Full GC.Major GC is slow and expensive as compared to Minor GC since it involves evaluating all live objects.The permanent generation holds objects that the JVM finds convenient to have the garbage collectormanage, such as objects describing classes and methods, as well as the classes and methodsthemselves.Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12 5
  • 6. Figure 1Types of CollectorsTo reclaim memory, HotSpot has four different collectors as of J2SE 5.0 (and the option to enable them): • The default or serial collector • Throughput collector (-XX:+UseParallelGC) • Concurrent low pause collector (-XX:+UseConcMarkSweepGC) • Incremental collector (-XX:+UseTrainGC)The default collector is a serial collector that pauses the application during minor and major collections. Ifthe host machine has a single CPU, the serial collector will likely be as fast or faster as the othercollectors.The throughput collector uses the same collection mechanism as the serial collector for major collections,but implements a parallel minor collector for the young generation space.On the other hand, the concurrent low-pause collector attempts to perform most of the work of a majorcollection without interrupting your application (pauses are low/short).Finally the incremental collector, by careful book-keeping, collects just a portion of the tenured generationat each minor collection, trying to spread the large pause of a major collection over many minorcollections. However, the incremental collector is deprecated and will eventually be removed.Which collector works best for your application depends on the load of the application and the hostsystem. However, unless you have a system with multiple CPUs, tuning memory configuration rather thangarbage collection is likely to yield performance gains.MeasurementBefore we consider the metrics, let’s understand various parameters, on which it’s evaluated.Throughput is the percentage of total time not spent in garbage collection, considered over long periodsof time.Pauses are the times when an application appears unresponsive because garbage collection is occurring.Usually tolerable, but in an interactive graphics program even short pauses may negatively affect the userexperience.Promptness is the time between when an object becomes dead and when the memory becomesavailable, an important consideration for distributed systems, including remote method invocation (RMI).Footprint is the working set of a process, measured in pages and cache lines. On systems with limitedphysical memory or many processes, footprint may dictate scalability.Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12 6
  • 7. Throughput and footprint are best measured using metrics particular to the application. On the otherhand, pauses due to garbage collection are easily estimated by inspecting the diagnostic output of thevirtual machine itself.Now let’s understand that how and what garbage collection details are generated in the E-Business Suite.Table 1 below briefs how and where to set the parameters in the appropriate file to generate the GC dataand view it for specific E-Business Suite version Table 1Suite Generate the GC data View the output {file location}11i Provide the command line argument -verbose:gc in the file $IAS_ORACLE_HOME/Apache/Jserv/l $IAS_ORACLE_HOME/Apache/Jserv/etc/jserv.properties. For ogs/jvm/OACoreGroup.<core#>.stdout instance, wrapper.bin.parameters=-verbose:gc -hotspot -Xmx1024M -Xms128M -XX:MaxPermSize=128M or value of the s_jvm_options context variable in the file $APPL_TOP/admin/<CONTEXT>.xml <jvm_options oa_var=”s_jvm_options” osd=”Solaris”>- verbose:gc -Xmx512M -Xms128M -XX:MaxPermSize=128M </jvm_options>R12 Set the argument -verbose:gc in the start-parameters, java- $LOG_HOME/ora/10.1.3/opmn/OC4J~ options element in the file oacore~default_group_* $ORA_CONFIG_HOME/10.1.3/opmn/conf/opmn.xml. For instance, <ias-component id="OC4J"> <process-type id="home" module-id="OC4J" status="enabled"> <module-data> <category id="start-parameters"> <data id=”java-options” value=”- server -verbose:gc -Xmx512M -Xms128M. . ."/> ... </category> ... </module-data> ... </process-type> ... </ias-component>Following are the sample GC details that are generated after setting the value -verbose:gc [GC 325407K->83000K(776768K), 0.2300771 secs] [GC 325816K->83372K(776768K), 0.2454258 secs] [Full GC 267628K->83769K(776768K), 1.8479984 secs]Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12 7
  • 8. But what does these values means and how to measure them? Figure 21) Here we see two minor collections, and a2) Major collection3) and 4) Indicate the combined size of live objects before and after garbage collection, respectively. Afterminor collections the count includes objects that arent necessarily alive but cant be reclaimed, eitherbecause they are directly alive, or because they are within or referenced from the tenured generation.5) Is the total available space, not counting the space in the permanent generation6) The minor collection took about 24 secs to runSimply said, the first statement of this output tells that we had 325407Kb used, but that after about quarterof a second of work, GC has freed 242407KB so you now have 776768KB used.Note: Additional flags like, -XX:+PrintGCDetails or -XX:+PrintGCTimeStamps can be used to print moreinformation about the collections, for instance,[GC 111.042: [DefNew: 8128K->8128K(8128K), 0.0000505 secs]111.042: [Tenured: 18154K->2311K(24576K), 0.1290354secs]Few notable JVM parametersFew key options that should be considered for useful GC logging are: Table 2Option Meaning-verbose:gc Enable verbose GC logging.-XX:+PrintGCDetails Provide detailed statistics about collections. Give details about young generation survivor spaces and how many objects are-XX:+PrintTenuringDistribution promoted to the tenured space.Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12 8
  • 9. Following are few of the options for controlling how the JVM allocates and organizes memory (refer Sun’sdocumentation for details) Table 3Option Meaning-Xms Configure the initial Java heap size.-Xmx Configure the maximum Java heap size. Instructs the VM to analyze the current operating environment and attempt to optimize settings for memory-intensive applications. This option also enables implicit garbage-XX:+AggressiveHeap collection and adaptive memory sizing. It is intended for machines with large amounts of memory and multiple CPUs.-XX:NewRatio Configure the ratio of new/old generation sizes-XX:NewSize Configure the initial size of the young generation space.-XX:MaxNewSize Configure the maximum size of the young generation space.Now let’s look at some of the default JVM parameters provided in one of the configuration files. Forinstance, the sample context file $APPL_TOP/admin/$CONTEXT_NAME.xml contains the followingoptions,<jvm_options oa_var=”s_jvm_options” osd=”Solaris”>-verbose:gc -Xmx512M -Xms128M -XX:MaxPermSize=128M -XX:NewRatio=2-XX:+PrintGCTimeStamps -XX:+UseTLAB </jvm_options> Table 4Option Meaning-Xms128M Initiate with 128MB heap size-Xmx512M Grow JVM heap size to maximum size of 512 MB-XX:NewRatio=2 If the size of young generation is 10 MB then the size of old generation should be around 20MB-XX:MaxPermSize=128M Maximum grow able size of permanent generation to 128M-XX:+UseTLAB Uses thread-local object allocation blocks. This improves concurrency by reducing contention on the shared heap lock.But all this tuning might go for a toss, if the developer calls System.gc() since this forces a major collection.Application can be prohibited to make this operational by providing -XX:+DisableExplicitGCFew tips for JVM tuning in E-Business Suite  Upgrade to highest possible certified JDK version (JRE 6 is latest certified with both 11i and R12). To know current version of Java, search for the parameter s_jdktop in the file $APPL_TOP/admin/ $CONTEXT_NAME.xml <JDK_TOP oa_var=”s_jdktop”>/oracle/apps/11i/bin/java/1.4/</JDK_TOP>  For OACoreGroup consider no more than 100 active users per JVM (Refer Appendix for SQL Script)  There should be no more than 1 active JVM/OC4J instance per CPU. Ideally, there should be 1 JVM per 2 CPUs  Size your maximum heap size as either 512 MB or 1 GB. If you start with 512 MB and find that more memory is required, increase to a maximum of 1 GB. If more memory than 1 GB is required, use multiple JVM’s of 512MB or 1024 MB. (For multiple JVM’s, edit <oacore_nprocsGarbage Collection, Tuning and Monitoring JVM in EBS 11i and R12 9
  • 10. oa_var=”s_oacore_nprocs”>2</oacore_nprocs>) This is to reduce GC time since more heap size means more time to garbage collect  Setting -Xms and -Xmx to the same value avoid the need to memory allocation during runtime  With multi-CPU servers using JDK 1.4 and higher, the -XX:+UseConcMarkSweepGC -XX: +UseParNewGC parameters have been seen to improve performance and GC throughput on some environments.  Minor GC should be occurring at interval long enough to allow many objects to die young (i.e. lot of objects should die between two minor GC).  Review the frequency of collections, especially major collections (i.e. Full GC)  If full GCs are too frequent, consider increasing Xms and Xmx  Throughput is inversely proportion to amount of memory. Bigger the heap, more the time for GC, meaning low throughput.  Unless you have problems with pauses, try granting as much memory as possible to VM (512 is good start and fine tune as per load testing results)  100 users per OACore JVM has generally proven to be a practical and performant number  If heap usage is steadily increasing, then there may be a memory leak which require tweaking the application code  Do not use -XX:+UseParallelGC with -XX:+UseConcMarkSweepGC  Last but not the least, try experimenting with various JVM parameters referred in Table 2 and Table 3 above and many more available at Sun’s documentation site (refer References section), to arrive at the optimal solutionMonitor the JVMThe basic text based monitoring tool available to us are the log files which contains the data produced bythe JVM (using –verbose:gc option). For the details of these log files and how to understand the heap dataavailable in these files refer the Measurement section above.Apart for this text based approach, there are many other visual tools available to us like JConsole,VisualVM etcThe following section will discuss one of them, JConsole.JConsole provide information on performance and resource consumption of applications running on theJava platform using Java Management Extension (JMX) technology. This tool comes in JDK 1.5 andprovides graphical view of the JVM heap, usage, threads, classes and other statistics in real time.Refer Figure 3 below.You can run JConsole either locally (from the server that you are monitoring as localhost) or remotely viaa network connection with Oracle Applications. The safest and easier option is to run locally since usingJConsole remotely is more complex, insecure and it may introduce unacceptable risk for a productionenvironment.The JConsole interface in JDK 5 is composed of the following six tabs: 1. Summary displays summary information on the JVM and monitored values. 2. Memory displays information about memory use. 3. Threads displays information about thread use. 4. Classes displays information about class loading. 5. MBeans displays information about MBeans. 6. VM displays information about the JVM.Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12 10
  • 11. For instance, to check for memory leaks in your application, you can start a load test to create a constantload on the application. If the functionality youre testing during the load test does not contain a memoryleak, the memory usage should stay between a specific upper and lower limit.Connecting LocallyUsing JConsole with Oracle E-Business Suite 11iAdd the following lines to your jserv.properties file present under $IAS_TOP/Apache/Jserv/etc/ ,save thechanges and bounce Apache serverNote: To make this change permanent, refer Metalink note 270519.1 about AutoConfigwrapper.bin.parameters=-Dcom.sun.management.jmxremotewrapper.bin.parameters=-Dcom.sun.management.jmxremote.ssl=falsewrapper.bin.parameters=-Dcom.sun.management.jmxremote.authenticate=false Figure 3Connecting RemotelyFirstly, authentication is enabled by default so you will either need to configure your username/passwordsor could choose to defeat authentication entirely by using the jserv.properties value below.Note: The following change allows everyone to attach to your JVM processes wrapper.bin.parameters=-Dcom.sun.management.jmxremote.authenticate=falseGarbage Collection, Tuning and Monitoring JVM in EBS 11i and R12 11
  • 12. The second issue to overcome is that each JVM needs its own TCP port to accept connections. Onepossible approach is to customize the $IAS_ORACLE_HOME/Apache/Apache/bin/java.sh JVM startupscript and provide a port number range from which you allocate a TCP port number.The following sample script provided by Steven Chan but may give you some ideas as to how you couldovercome this issue in your own environment. Add this modification just before the existing lineexec $JSERVJAVA $JAVA_ADDITIONAL_ARGS $ARGV 1>> $STDOUTLOG 2>> $STDERRLOG#### author Steven Chan#### start of modification for JConsole remote monitoring#### Not supported or recommended by Oracle Corporation######## If there is a clash of port numbers calculated, then#### the JVM will not startup. Script therefore needs to be#### improved to check port not already in use######## Use port number range 9600 - 9650 as an illustrationif [ "$JDK_VERSION" = "1.5" ] ; then mJMXPORTMIN=9600 mJXMPORTMAX=9650 mNUMPORTS=`expr ${mJXMPORTMAX} - ${mJMXPORTMIN}` mRANDOM=$(((($RANDOM*${mNUMPORTS})/32767)+1)) mJMXPORT=`expr ${mRANDOM} + ${mJMXPORTMIN}` JAVA_ADDITIONAL_ARGS=" -DCLIENT_PROCESSID=$$ -Dcom.sun.management.jmxremote.port=${mJMXPORT}"fi#### end of modificationAfter starting your JVMs we then need to identify the TCP port they are listening on. This can bedetermined by using the following command:netstat -a | grep tcp | grep 96The JConsole executable is present in $JDK_HOME/bin, where JDK_HOME is the directory where theJDK is installed. Once JConsole is started a connection dialog box opens, with tab lists as Local,Remote and ‘Advanced. Check that under Remote tab Host or IP field value should be localhost (forlocal connection) or the IP address of the remote machine. For the remote connection, we additionallyneed to provide the port we wish to connect to and also the username/password if appropriate. Then clickon Connect button and it will show the details of any JVMs running on the local/remote system startedwith the same user ID as JConsole, along with their process ID and class/argument information and it canbe monitored.Note: JConsole works with the JRockit JVM which will be part of Fusion stack in future. But it’srecommended to use the tools that are packaged as JRockit Mission Control for monitoring JRockit JVMGarbage Collection, Tuning and Monitoring JVM in EBS 11i and R12 12
  • 13. Future changes - JRockitAcquisition of BEA and introduction of BEA Weblogic Application Server and JRockit JVMAll the discussion above has been restricted to Sun HotSpot JVM. But with the acquisition of BEA,JRockit JVM will also be used across the Oracle Fusion Middleware and Oracle Applications productfamilies.This will bring additional changes to JVM tuning for E-Business Suite associated with JRocketFor instance, the Table 4 below lists options that work the same or similarly on both Sun Hotspot and theJRockit JVM but have different names depending upon the JVM you are running. Table 4Sun Hotspot Option Name JRockit JVM Option Function Name-XX:+AggressiveHeap -XXaggressive:memory Configures the memory system for memory-intensive workloads and sets an expectation to enable large amounts of memory resources to ensure high throughput. The JRockit JVM will also use large pages, if available.-verbose:gc -Xverbose:memory Prints out log information about the memory system-Xmn, -XXNewSize, -Xns Sets the size of the young generation-XXMaxNewSize-XX:+UseConcMarkSweepGC -Xgc:singlecon Sets the garbage collector to use a concurrent strategy-XX:+UseParallelGC -Xgc:parallel Sets the garbage collector to use a parallel strategyNote: The certified plugs-ins for Forms is Jinitiator and Sun only.Garbage Collection, Tuning and Monitoring JVM in EBS 11i and R12 13
  • 14. References and QuotesGeneralhttp://java.sun.com/j2se/reference/whitepapers/memorymanagement_whitepaper.pdfhttp://java.sun.com/docs/hotspot/gc1.4.2/example.htmlhttp://blogs.oracle.com/stevenChan/http://java.sun.com/docs/hotspot/gc1.4.2/http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.htmlhttp://www.oracle.com/technology/software/products/jrockit/FAQ.htmlhttp://java.sun.com/developer/technicalArticSE/jconsole.htmlhttp://java.sun.com/j2se/1.5.0/docs/guide/management/jconsole.htmlhttp://java.sun.com/javase/technologies/hotspot/vmoptions.jsphttp://edocs.bea.com/jrockit/geninfo/diagnos/migrate.htmlOn upgrading the JDK in E-Business Suite:Note 246105.1- Upgrading to J2SE 1.4.2 with Oracle Applications 11iNote 304099.1- Using J2SE Version 5.0 with Oracle E-Business Suite 11iNote 401561.1- Using J2SE Version 6.0 with Oracle E-Business Suite 11iNotes on JVM tuning with respect to E-Business SuiteNote 244040.1- Oracle E-Business Suite Recommended Performance PatchesNote 370583.1- Basic troubleshooting of JVM consuming CPU or too many JDBC connections in AppsNote 244040.1- Oracle E-Business Suite Recommended Performance PatchesGarbage Collection, Tuning and Monitoring JVM in EBS 11i and R12 14
  • 15. AppendixScript to determine "active users" for OACoreGroup {run as APPS user}REMREM SQL to count number of Apps 11i users , refer Metalink Note: Note:427759.1REM Run as APPS userREMselect Number of user sessions : || count( distinct session_id) How_many_user_sessions from icx_sessions icxwhere disabled_flag != Yand PSEUDO_FLAG = Nand (last_connect + decode(FND_PROFILE.VALUE(ICX_SESSION_TIMEOUT), NULL,limit_time,0,limit_time,FND_PROFILE.VALUE(ICX_SESSION_TIMEOUT)/60)/24) > sysdate and counter < limit_connects;REMREM END OF SQLREMTags for this document:JVM TuningJava.lang.OutOfMemoryErrorJRockitJConsoleGarbage CollectionGarbage Collection, Tuning and Monitoring JVM in EBS 11i and R12 15