1
Contents
LOAD & PERFORMANCE TESTING .................................................... ERROR! BOOKMARK NOT DEFINED.
1 ST...
4.1.4 Pause Target ..........................................................................................................
3.6 AUTOMATICALLY RECOVERING LEAKED CONNECTIONS ..................................... ERROR! BOOKMARK NOT DEFINED.
3.7 LIM...
5.3 PARTITIONING........................................................................................... ERROR! BOOKMAR...
10.4.1 Enable SecureFiles for SOAINFRA schema ....................................... Error! Bookmark not defined.
10.4.2 ...
3.4 ANALYSIS OF SOA SERVER MEMORY USAGE ................................................... ERROR! BOOKMARK NOT DEFINED.
4...
APPENDIX D: SQL QUERIES .............................................................. ERROR! BOOKMARK NOT DEFINED.
1 EM C...
6 EFFECTS............................................................................................... ERROR! BOOKMARK N...
1.1 DETERMINING PERFORMANCE GOALS AND OBJECTIVES – CURRENT & FUTURE.... ERROR! BOOKMARK NOT DEFINED.
1.2 MEASURING PERFORM...
Exhibit 34: BPEL EJBs - 3 ................................................................... Error! Bookmark not defined....
Exhibit 74: free ................................................................................. Error! Bookmark not def...
Snippets
Snippet 1: Sample Work Manager - 1 .............................................. Error! Bookmark not defined.
Sn...
Tables
Table 1: AuditLevel........................................................................... Error! Bookmark not ...
JVM	Tuning	
SOA Suite 11g ideally uses two JVMs to function.
Admin Server JVM: On this JV, Weblogic Admin Console and the ...
Exhibit 1: JVM Tuning Tradeoffs

Let us understand how this tradeoff works.

1 Pause Times and Throughput
With the JVM one...
Exhibit 2: JVM's Garbage Collection and Throughput

1.2 Defragmentation Pauses and Throughput
The mark and sweep garbage c...
1. jRockit
2. HotSpot
Other than Solaris, jRokit is better option to run WLS. I do not have any data to prove this bias
bu...
-Xms:<size>[g|G|m|M|k|K]

Sets the initial and minimum heap size.

-Xmx:<size>[g|G|m|M|k|K]

Sets the maximum heap size.

...
4.1.3 Nursery Size
Nursery is an area of the heap where new objects are allocated. When the nursery becomes full
it is gar...
4.2.1 Lazy Unlocking
This option facilitates synchronized Java code to run faster when the contention on the locks is
low....
4.3.1 Compaction
Compaction of objects is the process of moving objects closer to each other in the heap, thus
reducing th...
synchronizing with other threads. When the TLA gets full the thread simply requests a new TLA.
The objects allocated in a ...
5.1 Garbage Collection Strategies
5.1.1 Serial GC Strategy
Serial GC strategy performs well in Single processor machine si...
•

-XX:+CMSIncrementalMode

Incremental concurrent collection must be specified with concurrent strategy option.
java –ser...
Here is one output from -XX:+PrintGCDetails
[GC [DefNew: 64289K->875K(64286K), 0.0397641 secs] 196554K->133321K(262284K),
...
m

or M

Megabyte (MB)

k

or K

Kilobyte (KB)

One should set the minimum and maximum heap size as same value to avoid
un...
attempt to keep garbage collection pauses shorter than the specified value. These
adjustments may cause the garbage collec...
For BPEL engine’s managed server parallel GC is sufficient, so will not discuss the details of
concurrent GC strategy.

6 ...
The PermSpace is normally not a target of tuning efforts, usually more space must be allocated
in the -XX:PermSize/-XX:Max...
•
•
•

Mapped Buffers, created by mapping a region of a file into memory
(using java.nio.channels.FileChannel.map)
mmap fi...
•

Analysis of the composites, are they looping, are there a lot of synchronous BPEL
processes running for long periods of...
3. Performance tuning of jRockit:
http://docs.oracle.com/cd/E13150_01/jrockit_jvm/jrockit/geninfo/diagnos/part_02.html
4. ...
Upcoming SlideShare
Loading in …5
×

BPEL PM 11g performance tuning - 3

1,529 views

Published on

This is third chapter of Oracle Fusion Middleware BPEL PM 11g Performance Tuning. This chapter covers JVM (HotSpot as well as jRokit) tuning.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,529
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

BPEL PM 11g performance tuning - 3

  1. 1. 1
  2. 2. Contents LOAD & PERFORMANCE TESTING .................................................... ERROR! BOOKMARK NOT DEFINED. 1 STRESS, LOAD, SOAK, SPIKE TESTS LOAD PROFILES ........................................ ERROR! BOOKMARK NOT DEFINED. 2 LNP TESTING LIFE CYCLE.......................................................................... ERROR! BOOKMARK NOT DEFINED. 3 EXAMPLES OF LNP TESTING OBJECTIVES ...................................................... ERROR! BOOKMARK NOT DEFINED. 3.1 RESPONSE TIME......................................................................................... ERROR! BOOKMARK NOT DEFINED. 3.2 RELIABILITY............................................................................................... ERROR! BOOKMARK NOT DEFINED. 3.3 CONFIGURATION SIZING .............................................................................. ERROR! BOOKMARK NOT DEFINED. 3.4 CAPACITY PLANNING .................................................................................. ERROR! BOOKMARK NOT DEFINED. 3.5 REGRESSION ............................................................................................. ERROR! BOOKMARK NOT DEFINED. 3.6 BOTTLENECK IDENTIFICATION ...................................................................... ERROR! BOOKMARK NOT DEFINED. 3.7 SCALABILITY .............................................................................................. ERROR! BOOKMARK NOT DEFINED. BPEL PM: INTRODUCTION ............................................................... ERROR! BOOKMARK NOT DEFINED. 1 BPEL PM PROCESS TYPES ....................................................................... ERROR! BOOKMARK NOT DEFINED. 1.1 BY INTERFACE ........................................................................................... ERROR! BOOKMARK NOT DEFINED. 1.2 BY DURABILITY .......................................................................................... ERROR! BOOKMARK NOT DEFINED. 2 TERMINOLOGY ...................................................................................... ERROR! BOOKMARK NOT DEFINED. 3 DIFFERENCE BETWEEN ASYNC & SYNC BPEL .............................................. ERROR! BOOKMARK NOT DEFINED. 4 SOA 11G CONFIGURATION FILES AND LOG FILES LOCATION ............................. ERROR! BOOKMARK NOT DEFINED. 5 DEFAULT DATA SOURCES ......................................................................... ERROR! BOOKMARK NOT DEFINED. 6 PERFORMANCE OBJECTIVE ....................................................................... ERROR! BOOKMARK NOT DEFINED. 7 WHAT TO TUNE .................................................................................... ERROR! BOOKMARK NOT DEFINED. 8 SYSTEM SETUP ...................................................................................... ERROR! BOOKMARK NOT DEFINED. 9 IMPORTANT POINTS ............................................................................... ERROR! BOOKMARK NOT DEFINED. 10 REFERENCE......................................................................................... ERROR! BOOKMARK NOT DEFINED. JVM TUNING .................................................................................................................................... 15 1 PAUSE TIMES AND THROUGHPUT ....................................................................................................... 16 1.1 GARBAGE COLLECTION VS. THROUGHPUT ................................................................................................... 16 1.2 DEFRAGMENTATION PAUSES AND THROUGHPUT ......................................................................................... 17 2 PERFORMANCE AND MEMORY FOOTPRINT ........................................................................................... 17 3 JVM CHOICE ................................................................................................................................. 17 4 JROCKIT’S TUNING .......................................................................................................................... 18 4.1 BOARDING INTO SHIP – STEP 1 ................................................................................................................. 18 4.1.1 Heap Size ......................................................................................................................................... 18 4.1.2 Garbage Collection.......................................................................................................................... 19 4.1.3 Nursery Size ................................................................................................................................... 20 2
  3. 3. 4.1.4 Pause Target ................................................................................................................................... 20 4.2 LEAVING THE SHORE – STEP 2 ................................................................................................................... 20 4.2.1 Lazy Unlocking ................................................................................................................................ 21 4.2.2 Call Profiling .................................................................................................................................... 21 4.2.3 Large Pages ..................................................................................................................................... 21 4.3 IN THE HIGH SEAS – STEP 3 ...................................................................................................................... 21 4.3.1 Compaction ..................................................................................................................................... 22 4.3.2 Thread Local Area size .................................................................................................................... 22 5 HOTSPOT’S TUNING ........................................................................................................................ 23 5.1 GARBAGE COLLECTION STRATEGIES ........................................................................................................... 24 5.1.1 Serial GC Strategy ........................................................................................................................... 24 5.1.2 Parallel GC Strategy ........................................................................................................................ 24 5.1.3 Parallel GC strategy ......................................................................................................................... 27 6 WHERE TO SPECIFY JVM AND ITS PARAMETERS ..................................................................................... 29 7 OUTOFMEMORYERROR ................................................................................................................... 29 7.1 JAVA.LANG.OUTOFMEMORYERROR: JAVA HEAP SPACE ................................................................................ 29 7.2 JAVA.LANG.OUTOFMEMORYERROR: PERMGEN SPACE ................................................................................. 29 7.3 JAVA.LANG.OUTOFMEMORYERROR: UNABLE TO CREATE NEW NATIVE THREAD................................................. 30 7.4 JAVA.LANG.OUTOFMEMORYERROR: REQUESTED XXX BYTES FOR CHUNKPOOL::ALLOCATE. OUT OF SWAP SPACE ... 30 7.5 ANALYSIS .............................................................................................................................................. 31 8 REFERENCE ................................................................................................................................... 32 WEBLOGIC SERVER TUNING ............................................................ ERROR! BOOKMARK NOT DEFINED. 1 THREAD MANAGEMENT/WORK MANAGER ................................................. ERROR! BOOKMARK NOT DEFINED. 1.1 REQUEST CLASSES ..................................................................................... ERROR! BOOKMARK NOT DEFINED. 1.2 CONSTRAINTS ........................................................................................... ERROR! BOOKMARK NOT DEFINED. 1.3 WORK MANGER TYPES............................................................................... ERROR! BOOKMARK NOT DEFINED. 1.3.1 Default Thread Pool ........................................................................... Error! Bookmark not defined. 1.3.2 SOAWorkManager ............................................................................. Error! Bookmark not defined. 2 NETWORK I/O ...................................................................................... ERROR! BOOKMARK NOT DEFINED. 2.1 MUXERS .................................................................................................. ERROR! BOOKMARK NOT DEFINED. 2.2 CHUNK PARAMETERS ................................................................................. ERROR! BOOKMARK NOT DEFINED. 2.3 CONNECTION BACKLOG BUFFERING .............................................................. ERROR! BOOKMARK NOT DEFINED. 2.4 CACHED CONNECTIONS .............................................................................. ERROR! BOOKMARK NOT DEFINED. 3 TUNING DATA SOURCE CONNECTION POOLS ................................................ ERROR! BOOKMARK NOT DEFINED. 3.1 STATEMENT CACHE .................................................................................... ERROR! BOOKMARK NOT DEFINED. 3.1.1 Usage Restrictions for Statement Cache ........................................... Error! Bookmark not defined. 3.2 CONNECTION TESTING OPTIONS FOR A DATA SOURCE ..................................... ERROR! BOOKMARK NOT DEFINED. 3.2.1 Automatic Testing .............................................................................. Error! Bookmark not defined. 3.3 ENABLING CONNECTION CREATION RETRIES................................................... ERROR! BOOKMARK NOT DEFINED. 3.4 ENABLING CONNECTION REQUESTS TO WAIT FOR A CONNECTION ..................... ERROR! BOOKMARK NOT DEFINED. 3.5 XA TRANSACTION TIMEOUT ........................................................................ ERROR! BOOKMARK NOT DEFINED. 3
  4. 4. 3.6 AUTOMATICALLY RECOVERING LEAKED CONNECTIONS ..................................... ERROR! BOOKMARK NOT DEFINED. 3.7 LIMITING STATEMENT PROCESSING TIME ...................................................... ERROR! BOOKMARK NOT DEFINED. 3.8 PINNED TO THREAD PROPERTY .................................................................... ERROR! BOOKMARK NOT DEFINED. 3.9 GRIDLINK ................................................................................................. ERROR! BOOKMARK NOT DEFINED. 3.10 SINGLE CLIENT ACCESS NAME.................................................................... ERROR! BOOKMARK NOT DEFINED. 3.11 LOGGING LAST RESOURCE ......................................................................... ERROR! BOOKMARK NOT DEFINED. 4 DATABASE MODE .................................................................................. ERROR! BOOKMARK NOT DEFINED. 5 REFERENCE .......................................................................................... ERROR! BOOKMARK NOT DEFINED. BPEL ENGINE TUNING ..................................................................... ERROR! BOOKMARK NOT DEFINED. 1 REAL TIME VIEWING OF PERFORMANCE METRICS .......................................... ERROR! BOOKMARK NOT DEFINED. 1.1 FUSION MIDDDLEWARE CONTROL ................................................................ ERROR! BOOKMARK NOT DEFINED. 1.2 DYNAMIC MEMORY SERVICE ....................................................................... ERROR! BOOKMARK NOT DEFINED. 2 BPEL ENGINE THREADING MODEL............................................................. ERROR! BOOKMARK NOT DEFINED. 2.1 DISPATCHER MAXIMUM REQUEST DEPTH...................................................... ERROR! BOOKMARK NOT DEFINED. 2.2 INVOKEMESSAGES IN CACHE ....................................................................... ERROR! BOOKMARK NOT DEFINED. 3 TRANSACTION TIMEOUT .......................................................................... ERROR! BOOKMARK NOT DEFINED. 3.1 SYNCMAXWAITTIME ................................................................................. ERROR! BOOKMARK NOT DEFINED. 3.2 BPEL EJBS ............................................................................................... ERROR! BOOKMARK NOT DEFINED. 3.3 GLOBAL TRANSACTION TIMEOUT AT WEBLOGIC DOMAIN LEVEL ........................ ERROR! BOOKMARK NOT DEFINED. 3.4 WEBSERVICE READ AND CONNECTION TIMEOUT ............................................. ERROR! BOOKMARK NOT DEFINED. 3.5 RELATIONSHIP AMONG TIMEOUT PARAMETERS .............................................. ERROR! BOOKMARK NOT DEFINED. 3.6 WLS TIMEOUT PARAMETERS ....................................................................... ERROR! BOOKMARK NOT DEFINED. 3.6.1 Complete Message Timeout .............................................................. Error! Bookmark not defined. 3.6.2 Idle Connection Timeout: .................................................................. Error! Bookmark not defined. 3.6.3 Tunneling Client Timeout................................................................... Error! Bookmark not defined. 3.6.4 Post Timeout ...................................................................................... Error! Bookmark not defined. 3.6.5 Duration ............................................................................................. Error! Bookmark not defined. 3.6.6 Tunneling Client Timeout................................................................... Error! Bookmark not defined. 4 BPEL PM AUDIT AND LOGGING ................................................................ ERROR! BOOKMARK NOT DEFINED. 4.1 AUDIT TUNING .......................................................................................... ERROR! BOOKMARK NOT DEFINED. 4.1.1 SOA Infrastructure level ..................................................................... Error! Bookmark not defined. 4.1.2 BPEL Engine Level .............................................................................. Error! Bookmark not defined. 4.1.3 AuditStorePolicy ................................................................................ Error! Bookmark not defined. 4.1.4 AuditFlushByteThreshold................................................................... Error! Bookmark not defined. 4.1.5 AuditFlushEventThreshold ................................................................. Error! Bookmark not defined. 4.1.6 AsyncAuditPersisterThreads .............................................................. Error! Bookmark not defined. 4.1.7 AsyncAuditQueueSize ........................................................................ Error! Bookmark not defined. 4.2 LOGGING TUNING...................................................................................... ERROR! BOOKMARK NOT DEFINED. 5 BPEL ENGINE DATABASE SCHEMA ............................................................. ERROR! BOOKMARK NOT DEFINED. 5.1 TWO SCHEMAS ......................................................................................... ERROR! BOOKMARK NOT DEFINED. 5.2 PURGE ..................................................................................................... ERROR! BOOKMARK NOT DEFINED. 4
  5. 5. 5.3 PARTITIONING........................................................................................... ERROR! BOOKMARK NOT DEFINED. 5.3.1 Partial Partitioning ............................................................................. Error! Bookmark not defined. 5.3.2 Complete Partitioning ........................................................................ Error! Bookmark not defined. 6 MDS SCHEMA TUNING ........................................................................... ERROR! BOOKMARK NOT DEFINED. 6.1 PURGING ................................................................................................. ERROR! BOOKMARK NOT DEFINED. 6.2 STATISTIC COLLECTION ............................................................................... ERROR! BOOKMARK NOT DEFINED. 6.3 OPTIMIZING PATH_FULLNAME TABLE ........................................................... ERROR! BOOKMARK NOT DEFINED. 7 TECHNOLOGY ADAPTERS ......................................................................... ERROR! BOOKMARK NOT DEFINED. 7.1 FILE AND FTP ADAPTER .............................................................................. ERROR! BOOKMARK NOT DEFINED. 7.1.1 Threads .............................................................................................. Error! Bookmark not defined. 7.1.2 Inbound Throttling ............................................................................. Error! Bookmark not defined. 7.1.3 Outbound Throttling .......................................................................... Error! Bookmark not defined. 7.1.4 Outbound Performance Best Practices.............................................. Error! Bookmark not defined. 7.1.5 Tuning number of files to be processed at a time ............................. Error! Bookmark not defined. 7.2 DATABASE ADAPTER .................................................................................. ERROR! BOOKMARK NOT DEFINED. 7.2.1 Basic considerations .......................................................................... Error! Bookmark not defined. 7.2.2 Existence Checking............................................................................. Error! Bookmark not defined. 7.2.3 Throttling ........................................................................................... Error! Bookmark not defined. 7.2.4 Adapter Threads ................................................................................ Error! Bookmark not defined. 7.3 JMS ADAPTER .......................................................................................... ERROR! BOOKMARK NOT DEFINED. 7.3.1 Receive Threads ................................................................................. Error! Bookmark not defined. 7.4 AQ ADAPTER ............................................................................................ ERROR! BOOKMARK NOT DEFINED. 7.4.1 Dequeue Threads ............................................................................... Error! Bookmark not defined. 7.4.2 Dequeue Threads and Number of database connection................... Error! Bookmark not defined. 7.5 MQ ADAPTER ........................................................................................... ERROR! BOOKMARK NOT DEFINED. 7.5.1 Dequeue Threads ............................................................................... Error! Bookmark not defined. 7.6 SOCKET ADAPTER ...................................................................................... ERROR! BOOKMARK NOT DEFINED. 8 BPEL PM – MISCELLANEOUS PROPERTIES ................................................... ERROR! BOOKMARK NOT DEFINED. 8.1 INSTANCE KEY BLOCK SIZE........................................................................... ERROR! BOOKMARK NOT DEFINED. 8.2 AUTOMATIC RECOVERY ATTEMPTS ............................................................... ERROR! BOOKMARK NOT DEFINED. 8.3 DISABLE BPEL MONITORS AND SENSORS ...................................................... ERROR! BOOKMARK NOT DEFINED. 9 BEST PRACTICES: BPEL PROCESS/COMPOSITE/COMPONENT............................. ERROR! BOOKMARK NOT DEFINED. 9.1 LARGE PAYLOADS ...................................................................................... ERROR! BOOKMARK NOT DEFINED. 9.2 PERSIST ONLY THE FAULTED BPEL INSTANCES................................................. ERROR! BOOKMARK NOT DEFINED. 9.3 DO NOT VALIDATE INCOMING XML .............................................................. ERROR! BOOKMARK NOT DEFINED. 9.4 IDEMPOTENENCY ....................................................................................... ERROR! BOOKMARK NOT DEFINED. 9.5 NONBLOCKINGINVOKE................................................................................ ERROR! BOOKMARK NOT DEFINED. 10 DATABASE TUNING .............................................................................. ERROR! BOOKMARK NOT DEFINED. 10.1 INITIALIZATION PARAMETERS ..................................................................... ERROR! BOOKMARK NOT DEFINED. 10.2 HUGE PAGES .......................................................................................... ERROR! BOOKMARK NOT DEFINED. 10.3 DB CONTENTION..................................................................................... ERROR! BOOKMARK NOT DEFINED. 10.3.1 Wait events ...................................................................................... Error! Bookmark not defined. 10.4 SECUREFILES........................................................................................... ERROR! BOOKMARK NOT DEFINED. 5
  6. 6. 10.4.1 Enable SecureFiles for SOAINFRA schema ....................................... Error! Bookmark not defined. 10.4.2 Enabling Encryption ......................................................................... Error! Bookmark not defined. 10.5 SQL PLAN .............................................................................................. ERROR! BOOKMARK NOT DEFINED. 11 REFERENCE......................................................................................... ERROR! BOOKMARK NOT DEFINED. EM FUSION MIDDLEWARE CONTROL AND WLS ADMIN CONSOLE..... ERROR! BOOKMARK NOT DEFINED. 1 2 3 4 5 6 7 8 9 10 11 12 STUCK THREAD MAX TIME....................................................................... ERROR! BOOKMARK NOT DEFINED. STUCK THREAD TIMER INTERVAL ............................................................... ERROR! BOOKMARK NOT DEFINED. TUNE DUMP CONFIGURATION ................................................................... ERROR! BOOKMARK NOT DEFINED. DATA DISPLAY OPTION FOR EM CONSOLE ................................................... ERROR! BOOKMARK NOT DEFINED. CACHE TIMEOUT FOR DISCOVERY ............................................................... ERROR! BOOKMARK NOT DEFINED. PERMSIZE OF ADMIN SERVER ................................................................... ERROR! BOOKMARK NOT DEFINED. DMS’S FREQUENCY................................................................................ ERROR! BOOKMARK NOT DEFINED. UN-TARGET DMS FROM SERVERS ............................................................. ERROR! BOOKMARK NOT DEFINED. BPEL RECOVERY CONSOLE OPTION ............................................................. ERROR! BOOKMARK NOT DEFINED. INDEX CREATION.................................................................................. ERROR! BOOKMARK NOT DEFINED. DB MANUAL STATS COLLECTION ............................................................. ERROR! BOOKMARK NOT DEFINED. REFERENCE......................................................................................... ERROR! BOOKMARK NOT DEFINED. DATA COLLECTION .......................................................................... ERROR! BOOKMARK NOT DEFINED. 1 BEFORE LNP TEST .................................................................................. ERROR! BOOKMARK NOT DEFINED. 1.1 INFRASTRUCTURE ...................................................................................... ERROR! BOOKMARK NOT DEFINED. 1.2 JVM-HOTSPOT-<<MANAGEDSERVER>> AND JVM-JROCKIT-<<MANAGEDSERVER>> ....... ERROR! BOOKMARK NOT DEFINED. 1.3 WLS-THREADING-<<MANAGEDSERVER>> ................................................... ERROR! BOOKMARK NOT DEFINED. 1.4 WLS-NETWORKIO-<<MANAGEDSERVER>> .................................................. ERROR! BOOKMARK NOT DEFINED. 1.5 WLS-DATASOURCE-<<MANAGEDSERVER>>................................................. ERROR! BOOKMARK NOT DEFINED. 1.6 BPEL PM-<<MANAGEDSERVER>> .............................................................. ERROR! BOOKMARK NOT DEFINED. 1.7 TECHNOLOGY ADAPTERS............................................................................. ERROR! BOOKMARK NOT DEFINED. 1.8 BPEL PM – COMPOSITE ............................................................................ ERROR! BOOKMARK NOT DEFINED. 1.9 DATABASE ................................................................................................ ERROR! BOOKMARK NOT DEFINED. 2 DURING LNP TEST ................................................................................. ERROR! BOOKMARK NOT DEFINED. 2.1 CPU AND RAM USAGE AND MEMORY AVAILABILITY ........................................ ERROR! BOOKMARK NOT DEFINED. 2.2 JVM’S HEAP USAGE, GARBAGE COLLECTION AND ANY MEMORY LEAK IF ANY ........ ERROR! BOOKMARK NOT DEFINED. 2.3 DATABASE CONNECTIONS AVAILABILITY AND USAGE ........................................ ERROR! BOOKMARK NOT DEFINED. 2.4 AVAILABILITY OF PERIPHERAL RESOURCES LIKE JMS QUEUES, AQ AND THEIR USAGE PATTERN .... ERROR! BOOKMARK NOT DEFINED. 3 AFTER LNP TEST .................................................................................... ERROR! BOOKMARK NOT DEFINED. 3.1 GET FULL DUMP OF FEW IMPORTANT TABLES.................................................. ERROR! BOOKMARK NOT DEFINED. 3.2 EXECUTE SQL QUERIES AS AND WHEN REQUIRED ............................................ ERROR! BOOKMARK NOT DEFINED. 3.3 ANALYSIS OF AWR REPORT ........................................................................ ERROR! BOOKMARK NOT DEFINED. 6
  7. 7. 3.4 ANALYSIS OF SOA SERVER MEMORY USAGE ................................................... ERROR! BOOKMARK NOT DEFINED. 4 REFERENCE .......................................................................................... ERROR! BOOKMARK NOT DEFINED. LNP TEST CASE DESIGN AND LOAD TEST SELECTION ......................... ERROR! BOOKMARK NOT DEFINED. 1 LNP TEST CASE DESIGN ........................................................................... ERROR! BOOKMARK NOT DEFINED. 2 DESIGN OF EXPERIMENT .......................................................................... ERROR! BOOKMARK NOT DEFINED. 3 REFERENCE .......................................................................................... ERROR! BOOKMARK NOT DEFINED. APPENDIX A: WEBLOGIC SERVER OVERVIEW ................................... ERROR! BOOKMARK NOT DEFINED. 1 2 3 4 5 6 7 8 9 WLS CONFIGURATION ............................................................................ ERROR! BOOKMARK NOT DEFINED. DOMAIN ............................................................................................. ERROR! BOOKMARK NOT DEFINED. SERVER ............................................................................................... ERROR! BOOKMARK NOT DEFINED. ADMINISTRATION SERVER ....................................................................... ERROR! BOOKMARK NOT DEFINED. MANAGED SERVER ................................................................................ ERROR! BOOKMARK NOT DEFINED. ADMINISTRATION SERVER TO MANAGED SERVER INTERACTION ........................ ERROR! BOOKMARK NOT DEFINED. CLUSTER .............................................................................................. ERROR! BOOKMARK NOT DEFINED. NODE MANAGER................................................................................... ERROR! BOOKMARK NOT DEFINED. MACHINE ............................................................................................ ERROR! BOOKMARK NOT DEFINED. APPENDIX B: AUDITING IN BPEL PM ................................................ ERROR! BOOKMARK NOT DEFINED. 1 AUDIT LEVELS ....................................................................................... ERROR! BOOKMARK NOT DEFINED. 2 ORDER OF PRECEDENCE FOR AUDIT LEVEL SETTINGS ........................................ ERROR! BOOKMARK NOT DEFINED. APPENDIX C: ANTI PATTERNS .......................................................... ERROR! BOOKMARK NOT DEFINED. 1 2 3 4 5 6 7 8 9 10 SYNCHRONOUS – ASYNCHRONOUS ............................................................ ERROR! BOOKMARK NOT DEFINED. OVER USE OF ASYNCHRONOUS PROCESSES ................................................... ERROR! BOOKMARK NOT DEFINED. OVER USE OF DURABLE PROCESSES ............................................................. ERROR! BOOKMARK NOT DEFINED. NO FAULT HANDLING ............................................................................. ERROR! BOOKMARK NOT DEFINED. SYNCHRONOUS FAULT HANDLING .............................................................. ERROR! BOOKMARK NOT DEFINED. TO MANY RETRIES .................................................................................. ERROR! BOOKMARK NOT DEFINED. CHATTING BPEL PROCESS (CALL BACK) ....................................................... ERROR! BOOKMARK NOT DEFINED. OVER USE OF FLOWN ............................................................................. ERROR! BOOKMARK NOT DEFINED. LOOPS AND MORE LOOPS......................................................................... ERROR! BOOKMARK NOT DEFINED. SYNCHRONOUS AND ASYNCHRONOUS PROCESSES ON SAME MANAGED SERVER/CLUSTER ... ERROR! BOOKMARK NOT DEFINED. 11 DURABLE AND TRANSIENT PROCESSES ON SAME MANAGED SERVER/CLUSTER...... ERROR! BOOKMARK NOT DEFINED. 12 STICKY LOAD BALANCER ......................................................................... ERROR! BOOKMARK NOT DEFINED. 13 NOT KEEPING ASPECT RATIO ................................................................... ERROR! BOOKMARK NOT DEFINED. 7
  8. 8. APPENDIX D: SQL QUERIES .............................................................. ERROR! BOOKMARK NOT DEFINED. 1 EM CONSOLE SQL QUERIES ..................................................................... ERROR! BOOKMARK NOT DEFINED. 1.1 RECOVERY CONSOLE QUERIES ...................................................................... ERROR! BOOKMARK NOT DEFINED. 1.2 RECENT FAULT AND REJECTED MESSAGES QUERY ............................................. ERROR! BOOKMARK NOT DEFINED. 1.3 RECENT COMPOSITE INSTANCE QUERY........................................................... ERROR! BOOKMARK NOT DEFINED. 1.4 INSTANCE TAB PAGE QUERY ......................................................................... ERROR! BOOKMARK NOT DEFINED. 1.5 INSTANCE TAB PAGE SEARCH QUERY BASED ON NAME VS TITLE QUERY ................ ERROR! BOOKMARK NOT DEFINED. 1.6 FAULT AND REJECTED MESSAGE TAB PAGE QUERIES ......................................... ERROR! BOOKMARK NOT DEFINED. 1.6.1 Parent query ...................................................................................... Error! Bookmark not defined. 1.6.2 Child query ......................................................................................... Error! Bookmark not defined. 2 MISCELLANEOUS ................................................................................... ERROR! BOOKMARK NOT DEFINED. 2.1 STORED PROCEDURE TO CONVERT BLOB IN STRING .......................................... ERROR! BOOKMARK NOT DEFINED. 2.2 QUERY TO FIND PERCENTAGE OF FREE SPACE .................................................. ERROR! BOOKMARK NOT DEFINED. 2.3 QUERY TO FIND THE WAIT EVENTS FOR LGWR USING ITS SID ........................... ERROR! BOOKMARK NOT DEFINED. 2.4 QUERY TO MONITOR ‘REDO BUFFER ALLOCATION RETRIES’................................ ERROR! BOOKMARK NOT DEFINED. 2.5 SQL STATEMENT TO RECLAIM SPACE AFTER PURGING....................................... ERROR! BOOKMARK NOT DEFINED. 2.6 QUERY TO FIND OUT TOTAL SESSIONS ON A DATABASE ..................................... ERROR! BOOKMARK NOT DEFINED. 2.7 QUERY TO FIND OUT UTILIZATION OF PROCESSES AND SESSIONS IN A DATABASE ... ERROR! BOOKMARK NOT DEFINED. 2.8 FIND OUT THE PROCESS INSTANCE FROM A CONVERSATION ID WHEN THERE IS NO INSTANCE NUMBER SHOWING IN THE LOG FILE (BPEL INSTANCE ID FOR A TIMES OUT ITEM) ....................................... ERROR! BOOKMARK NOT DEFINED. 2.9 QUERY TO GET AUDIT DETAILS FROM AUDIT_DETAILS TABLE .............................. ERROR! BOOKMARK NOT DEFINED. 2.10 QUERY TO GET AUDIT DETAILS FROM AUDIT_TRAIL TABLE ............................... ERROR! BOOKMARK NOT DEFINED. 2.11 QUERY TO GET XML MESSAGE WITH THE GIVEN INSTANCE ID ......................... ERROR! BOOKMARK NOT DEFINED. 2.12 QUERY TO GET XML MESSAGE WITH A GIVEN INSTANCE NAME........................ ERROR! BOOKMARK NOT DEFINED. 2.13 QUERY TO GET PAYLOAD SIZE OF MESSAGE ................................................... ERROR! BOOKMARK NOT DEFINED. 2.14 QUERY TO GET EXECUTION TIME OF BPEL INSTANCES .................................... ERROR! BOOKMARK NOT DEFINED. 2.15 QUERY TO GET THE EXECUTION TIME OF BPEL INSTANCES AND TO FIND THE PARENT THAT HAS INITIALIZED THE COMPOSITE ....................................................................................................... ERROR! BOOKMARK NOT DEFINED. 2.16 QUERY TO IDENTIFY ALL THE FAULTS FOR THE MESSAGES THAT WERE SITTING IN BPEL ENGINE LEVEL RECOVERY AS UNDELIVERED INVOKES ........................................................................................ ERROR! BOOKMARK NOT DEFINED. APPENDIX E: BIG OR LARGE OR HUGE PAGES ................................... ERROR! BOOKMARK NOT DEFINED. 1 2 3 4 LINUX ................................................................................................. ERROR! BOOKMARK NOT DEFINED. WINDOWS ........................................................................................... ERROR! BOOKMARK NOT DEFINED. SOLARIS .............................................................................................. ERROR! BOOKMARK NOT DEFINED. REFERENCE: ......................................................................................... ERROR! BOOKMARK NOT DEFINED. APPENDIX F: ORA-01438: VALUE LARGER THAN SPECIFIED PRECISION ALLOWED .... ERROR! BOOKMARK NOT DEFINED. 5 WHAT IS THE ERROR IN LOGS? .................................................................. ERROR! BOOKMARK NOT DEFINED. 8
  9. 9. 6 EFFECTS............................................................................................... ERROR! BOOKMARK NOT DEFINED. 7 CAUSE ................................................................................................ ERROR! BOOKMARK NOT DEFINED. 8 SOLUTION ............................................................................................ ERROR! BOOKMARK NOT DEFINED. APPENDIX G: LOBS IN THE SOAINFRA SCHEMA ................................ ERROR! BOOKMARK NOT DEFINED. APPENDIX H: AWR, ADDM, & ASH REPORTS .................................... ERROR! BOOKMARK NOT DEFINED. 1 AWR REPORT ...................................................................................... ERROR! BOOKMARK NOT DEFINED. 2 ADDM REPORT .................................................................................... ERROR! BOOKMARK NOT DEFINED. 3 ASH REPORT ........................................................................................ ERROR! BOOKMARK NOT DEFINED. 4 AWR REPORT ANALYSIS ......................................................................... ERROR! BOOKMARK NOT DEFINED. 4.1 SQL STATEMENTS ORDERED BY ELAPSED TIME ............................................... ERROR! BOOKMARK NOT DEFINED. 4.2 SQL STATEMENTS ORDERED BY CPU TIME..................................................... ERROR! BOOKMARK NOT DEFINED. 4.3 SQL STATEMENTS ORDERED BY GETS ............................................................ ERROR! BOOKMARK NOT DEFINED. 4.4 SQL STATEMENTS ORDERED BY READS .......................................................... ERROR! BOOKMARK NOT DEFINED. 4.5 SQL STATEMENTS ORDERED BY EXECUTIONS .................................................. ERROR! BOOKMARK NOT DEFINED. 4.6 SQL STATEMENTS ORDERED BY PARSE CALLS ................................................. ERROR! BOOKMARK NOT DEFINED. 5 REFERENCE .......................................................................................... ERROR! BOOKMARK NOT DEFINED. APPENDIX I: MONITORING SCRIPTS ................................................. ERROR! BOOKMARK NOT DEFINED. 1 DATABASE MONITORING ......................................................................... ERROR! BOOKMARK NOT DEFINED. 2 JMS MONITORING ................................................................................ ERROR! BOOKMARK NOT DEFINED. 3 AQ MONITORING .................................................................................. ERROR! BOOKMARK NOT DEFINED. APPENDIX J: HOW TO MONITOR SOA SERVER MEMORY USAGE ....... ERROR! BOOKMARK NOT DEFINED. 1 2 3 4 SETUP: JCONSOLE OR VISUALVM (INSTALLED LOCALLY)................................... ERROR! BOOKMARK NOT DEFINED. SETUP: JVISUALVM (INSTALLED AT REMOTE MACHINE) ................................... ERROR! BOOKMARK NOT DEFINED. SETUP: JROCKIT MISSION CONTROL (INSTALLED AT REMOTE MACHINE) .............. ERROR! BOOKMARK NOT DEFINED. REFERENCE .......................................................................................... ERROR! BOOKMARK NOT DEFINED. APPENDIX K: HEAP DUMP FILES ANALYSIS: JROCKIT AND HOTSPOT JVMS ....... ERROR! BOOKMARK NOT DEFINED. 1 EXAMPLE ANALYSIS OF A HEAP DUMP FILE USING ECLIPSE MEMORY ANALYZER ... ERROR! BOOKMARK NOT DEFINED. 2 REFERENCE .......................................................................................... ERROR! BOOKMARK NOT DEFINED. APPENDIX L: CAPACITY PLANNING................................................... ERROR! BOOKMARK NOT DEFINED. 1 CAPACITY PLANNING FOR BPEL PM .......................................................... ERROR! BOOKMARK NOT DEFINED. 9
  10. 10. 1.1 DETERMINING PERFORMANCE GOALS AND OBJECTIVES – CURRENT & FUTURE.... ERROR! BOOKMARK NOT DEFINED. 1.2 MEASURING PERFORMANCE METRICS .......................................................... ERROR! BOOKMARK NOT DEFINED. 1.3 IDENTIFYING BOTTLENECKS ......................................................................... ERROR! BOOKMARK NOT DEFINED. 1.4 IMPLEMENTING A CAPACITY MANAGEMENT PLAN .......................................... ERROR! BOOKMARK NOT DEFINED. 2 REFERENCE .......................................................................................... ERROR! BOOKMARK NOT DEFINED. Exhibits Exhibit 1: Stress, Load, Soak, Spike Tests Load Profiles ................... Error! Bookmark not defined. Exhibit 2: LnP Testing Life Cycle and SDLC ....................................... Error! Bookmark not defined. Exhibit 3: BPEL Processes ................................................................. Error! Bookmark not defined. Exhibit 4: JVM Tuning Tradeoffs ................................................................................................... 16 Exhibit 5: JVM's Garbage Collection and Throughput .................................................................. 17 Exhibit 6: jRockit Attributes .......................................................................................................... 23 Exhibit 7: HostSpot Attributes ...................................................................................................... 28 Exhibit 8: WLS internal applications at startup - 1 ......................... Error! Bookmark not defined. Exhibit 9: WLS internal applications at startup - 2 ......................... Error! Bookmark not defined. Exhibit 10: WLS internal applications at startup - 3 ....................... Error! Bookmark not defined. Exhibit 11: WLS internal applications at startup - 4 ........................ Error! Bookmark not defined. Exhibit 12: WLS internal applications at startup - 5 ........................ Error! Bookmark not defined. Exhibit 13: WLS internal applications at startup - 6 ........................ Error! Bookmark not defined. Exhibit 14: SOAWorkManager - 1 .................................................... Error! Bookmark not defined. Exhibit 15: SOAWorkManager - 2 .................................................... Error! Bookmark not defined. Exhibit 16: SOAWorkManager - 3 .................................................... Error! Bookmark not defined. Exhibit 17: WLS Muxers ................................................................... Error! Bookmark not defined. Exhibit 18: FMC Login....................................................................... Error! Bookmark not defined. Exhibit 19: FMC Farm Summary....................................................... Error! Bookmark not defined. Exhibit 20: FMC Performance Summary - 1 ..................................... Error! Bookmark not defined. Exhibit 21: FMC Performance Summary - 2 ..................................... Error! Bookmark not defined. Exhibit 22: FMC Performance Summary - 3 ..................................... Error! Bookmark not defined. Exhibit 23: Dynamic Memory Service .............................................. Error! Bookmark not defined. Exhibit 24: BPEL Engine Threading Model ....................................... Error! Bookmark not defined. Exhibit 25: Dispatcher Maximum Request Depth - 1....................... Error! Bookmark not defined. Exhibit 26: Dispatcher Maximum Request Depth - 2....................... Error! Bookmark not defined. Exhibit 27: InvokeMessages In Cache .............................................. Error! Bookmark not defined. Exhibit 28: syncMaxWaitTime - 1 .................................................... Error! Bookmark not defined. Exhibit 29: syncMaxWaitTime - 2 .................................................... Error! Bookmark not defined. Exhibit 30: syncMaxWaitTime - 3 .................................................... Error! Bookmark not defined. Exhibit 31: syncMaxWaitTime – 4 .................................................... Error! Bookmark not defined. Exhibit 32: BPEL EJBs - 1 ................................................................... Error! Bookmark not defined. Exhibit 33: BPEL EJBs - 2 ................................................................... Error! Bookmark not defined. 10
  11. 11. Exhibit 34: BPEL EJBs - 3 ................................................................... Error! Bookmark not defined. Exhibit 35: Global transaction timeout at WLS Domain Level - 1 .... Error! Bookmark not defined. Exhibit 36: Global transaction timeout at WLS Domain Level - 2 .... Error! Bookmark not defined. Exhibit 37: WLS Timeout parameters - 1 ......................................... Error! Bookmark not defined. Exhibit 38: WLS Timeout parameters - 2 ......................................... Error! Bookmark not defined. Exhibit 39: WLS Timeout parameters - 3 ......................................... Error! Bookmark not defined. Exhibit 40: Tunneling Client Timeout - 1.......................................... Error! Bookmark not defined. Exhibit 41: Tunneling Client Timeout - 2.......................................... Error! Bookmark not defined. Exhibit 42: Tunneling Client Timeout - 3.......................................... Error! Bookmark not defined. Exhibit 43: SOA Infrastructure level audit tuning - 1 ....................... Error! Bookmark not defined. Exhibit 44: SOA Infrastructure level audit tuning - 2 ....................... Error! Bookmark not defined. Exhibit 45: AuditConfig - 1 ............................................................... Error! Bookmark not defined. Exhibit 46: AuditConfig - 2 ............................................................... Error! Bookmark not defined. Exhibit 47: AuditConfig - 3 ............................................................... Error! Bookmark not defined. Exhibit 48: AuditConfig - 4 ............................................................... Error! Bookmark not defined. Exhibit 49: AuditConfig - 5 ............................................................... Error! Bookmark not defined. Exhibit 50: Payload Validation - 1 .................................................... Error! Bookmark not defined. Exhibit 51: Payload Validation - 2 .................................................... Error! Bookmark not defined. Exhibit 52: AuditStorePolicy - 1 ....................................................... Error! Bookmark not defined. Exhibit 53: AuditStorePolicy - 2 ....................................................... Error! Bookmark not defined. Exhibit 54: Logging Tuning - 1 .......................................................... Error! Bookmark not defined. Exhibit 55: Logging Tuning - 2 .......................................................... Error! Bookmark not defined. Exhibit 56: Logging Tuning - 3 .......................................................... Error! Bookmark not defined. Exhibit 57: Logging Tuning - 4 .......................................................... Error! Bookmark not defined. Exhibit 58: MDS Purging - 1 ............................................................. Error! Bookmark not defined. Exhibit 59: MDS Purging - 2 ............................................................. Error! Bookmark not defined. Exhibit 60: Stuck Thread Timer Interval - 1...................................... Error! Bookmark not defined. Exhibit 61: : Stuck Thread Timer Interval - 2.................................... Error! Bookmark not defined. Exhibit 62: : Stuck Thread Timer Interval - 3.................................... Error! Bookmark not defined. Exhibit 63: Data Display Option for EM Console - 1 ........................ Error! Bookmark not defined. Exhibit 64: : Data Display Option for EM Console - 2 ...................... Error! Bookmark not defined. Exhibit 65: : Data Display Option for EM Console - 3 ...................... Error! Bookmark not defined. Exhibit 66: Un-Target DMS from servers - 1 .................................... Error! Bookmark not defined. Exhibit 67: Un-Target DMS from servers - 2 .................................... Error! Bookmark not defined. Exhibit 68: : Un-Target DMS from servers - 3 .................................. Error! Bookmark not defined. Exhibit 69: Un-Target DMS from servers - 4 .................................... Error! Bookmark not defined. Exhibit 70: Un-Target DMS from servers - 5 .................................... Error! Bookmark not defined. Exhibit 71: BPEL recovery console option - 1................................... Error! Bookmark not defined. Exhibit 72: BPEL recovery console option - 2................................... Error! Bookmark not defined. Exhibit 73: vmstat ............................................................................ Error! Bookmark not defined. 11
  12. 12. Exhibit 74: free ................................................................................. Error! Bookmark not defined. Exhibit 75: pmap .............................................................................. Error! Bookmark not defined. Exhibit 76: top .................................................................................. Error! Bookmark not defined. Exhibit 77: sar-B ............................................................................... Error! Bookmark not defined. Exhibit 78: meminfo ......................................................................... Error! Bookmark not defined. Exhibit 79: mpstat ............................................................................ Error! Bookmark not defined. Exhibit 80: mpstat 2 4 ...................................................................... Error! Bookmark not defined. Exhibit 81: df-m ................................................................................ Error! Bookmark not defined. Exhibit 82: WebLogic Homes ........................................................... Error! Bookmark not defined. Exhibit 83: Synchronous – Asynchronous - 1 ................................... Error! Bookmark not defined. Exhibit 84: Synchronous – Asynchronous - 2 ................................... Error! Bookmark not defined. Exhibit 85: Database Monitoring ..................................................... Error! Bookmark not defined. Exhibit 86: JMS Monitoring .............................................................. Error! Bookmark not defined. Exhibit 87: AQ Monitoring ............................................................... Error! Bookmark not defined. Exhibit 88: Setup jConsole or visualVM (installed locally) - 1 .......... Error! Bookmark not defined. Exhibit 89: Setup jConsole or visualVM (installed locally) - 2 .......... Error! Bookmark not defined. Exhibit 90: Setup jConsole or visualVM (installed locally) - 3 .......... Error! Bookmark not defined. Exhibit 91: Setup jConsole or visualVM (installed locally) - 4 .......... Error! Bookmark not defined. Exhibit 92: Heap Dump Files analysis - 1 ......................................... Error! Bookmark not defined. Exhibit 93: Heap Dump Files analysis - 2 ......................................... Error! Bookmark not defined. Exhibit 94: Heap Dump Files analysis - 3 ......................................... Error! Bookmark not defined. Exhibit 95: Heap Dump Files analysis - 4 ......................................... Error! Bookmark not defined. Exhibit 96: Heap Dump Files analysis - 5 ......................................... Error! Bookmark not defined. Exhibit 97: Heap Dump Files analysis - 6 ......................................... Error! Bookmark not defined. 12
  13. 13. Snippets Snippet 1: Sample Work Manager - 1 .............................................. Error! Bookmark not defined. Snippet 2: Sample Work Manager - 2 .............................................. Error! Bookmark not defined. Snippet 3: Sample config.xml........................................................... Error! Bookmark not defined. Snippet 4: Sample startWeblogic.sh ................................................ Error! Bookmark not defined. Snippet 5: Using setNull In a Prepared Statement .......................... Error! Bookmark not defined. Snippet 6: Sample thread waiting for DB Call to finish.................... Error! Bookmark not defined. Snippet 7: composite.xml ................................................................ Error! Bookmark not defined. Snippet 8: Drop Histrogram ............................................................. Error! Bookmark not defined. Snippet 9: Set Table Preferences ..................................................... Error! Bookmark not defined. Snippet 10: Single Threaded Model - jca file ................................... Error! Bookmark not defined. Snippet 11: Partitioned Threaded Model - jca file........................... Error! Bookmark not defined. Snippet 12: Tuning number of files to be processed at a time ....... Error! Bookmark not defined. Snippet 13: JMS Adapter - Receive Thread - composite.xml ........... Error! Bookmark not defined. Snippet 14: AQ Adapter - Dequeue Threads - composite.xml......... Error! Bookmark not defined. Snippet 15: common properties schema - 1.................................... Error! Bookmark not defined. Snippet 16: common properties schema - 2.................................... Error! Bookmark not defined. Snippet 17: bpel.xml ........................................................................ Error! Bookmark not defined. Snippet 18: composite.xml .............................................................. Error! Bookmark not defined. Snippet 19: SecureFiles .................................................................... Error! Bookmark not defined. Snippet 20: BasicFiles to SecureFiles ............................................... Error! Bookmark not defined. 13
  14. 14. Tables Table 1: AuditLevel........................................................................... Error! Bookmark not defined. Table 2: BPEL Engine - AuditLevel .................................................... Error! Bookmark not defined. Table 3: AuditStorePolicy ................................................................. Error! Bookmark not defined. Table 4: SOA Schemas ...................................................................... Error! Bookmark not defined. Table 5: SOAINFRA Tables ................................................................ Error! Bookmark not defined. Table 6: MDS Schema tables............................................................ Error! Bookmark not defined. Table 7: CUBE_INSTANCE................................................................. Error! Bookmark not defined. Table 8: CUBE_SCOPE ...................................................................... Error! Bookmark not defined. Table 9: COMPOSITE_INSTANCE ...................................................... Error! Bookmark not defined. Table 10: COMPOSITE_INSTANCE_FAULT ....................................... Error! Bookmark not defined. Table 11: AUDIT_TRAIL .................................................................... Error! Bookmark not defined. Table 12: AUDIT_DETAILS ................................................................ Error! Bookmark not defined. Table 13: DLV_MESSAGE ................................................................. Error! Bookmark not defined. Table 14: DLV_SUBSCRIPTION ......................................................... Error! Bookmark not defined. Table 15: DOCUMENT_CI_REF ......................................................... Error! Bookmark not defined. Table 16: DOCUMENT_DLV_MSG_REF ............................................ Error! Bookmark not defined. Table 17: WORK_ITEM ..................................................................... Error! Bookmark not defined. Table 18: XML_DOCUMENT ............................................................. Error! Bookmark not defined. Table 19: XML_DOCUMENT_REF ..................................................... Error! Bookmark not defined. Table 20: HEADERS_PROPERTIES ..................................................... Error! Bookmark not defined. Table 21: CLUSTER_MASTER ............................................................ Error! Bookmark not defined. Table 22: CLUSTER_NODE ................................................................ Error! Bookmark not defined. Table 23: Partial Partitioning ........................................................... Error! Bookmark not defined. Table 24: Complete Partitioning ...................................................... Error! Bookmark not defined. Table 25: MDS_NAMESAPCES.......................................................... Error! Bookmark not defined. Table 26: MDS_PARTITIONS ............................................................ Error! Bookmark not defined. Table 27: MDS_PATHS ..................................................................... Error! Bookmark not defined. Table 28: File & FTP Adapter - Inbound Throttling .......................... Error! Bookmark not defined. Table 29: : File & FTP Adapter - Outbound Throttling ..................... Error! Bookmark not defined. Table 30: : File & FTP Adapter - Outbound Best Practices............... Error! Bookmark not defined. Table 31: completionPersistPolicy property .................................... Error! Bookmark not defined. Table 32: validateSchema value ...................................................... Error! Bookmark not defined. Table 33: Idempotency value........................................................... Error! Bookmark not defined. Table 34: nonBlockingInvoke value ................................................ Error! Bookmark not defined. Table 35: Database initialization parameters .................................. Error! Bookmark not defined. Table 36: Contention tables ............................................................. Error! Bookmark not defined. Table 37: Tables under potential of index contention .................... Error! Bookmark not defined. 14
  15. 15. JVM Tuning SOA Suite 11g ideally uses two JVMs to function. Admin Server JVM: On this JV, Weblogic Admin Console and the EM Fusion Middleware Control are deployed. Though, this JVM does not affect performance of BPEL PM but if this JVM is leading to reduced performance of WebLogic Admin Console and EM Fusion Middleware Control, it leads to irritation and frustration during monitoring and deployment. SOA Managed Server: On this JVM entire SOA Suite and B2B product stack are deployed. This JVM is critical for overall performance of BPEL PM. The JVM tuning depends upon number of environmental factors and objectives of tuning. Since each deployment of BPEL PM is unique, so no sure shot formula can be derived but generally speaking following parameters should be considered while tuning JVM: • • • • • • • • • • • Which JVM (HotSpot or jRokit) Hardware configuration Amount of RAM available Number of Servers in cluster What else is running on servers Number of partitions on the servers Number of BPEL Processes/Composites deployed Types of interfaces (Synchronous vs. Asynchronous and Durable vs. Transient) Number of interface instances per second or minute Number of activities in an interface Type of activities in an interface Tuning the JVMs should not be speculative but a measured and calibrated. One should rely on Memory and GC analyzing tools like jConsole, visualVM, Flight Control etc. in conjunction with LnP Testing activities. Finer tuning should take help of thread dump and GC activities. JVM tuning leads to trade-offs among garbage collection pause time, application throughput, and memory footprint. From JVM metrics purpose: GC pause time is defined as typical duration for which application threads are paused when GC is happening. During pause time, application seems to be unresponsive. Throughput is measured in terms of time spent doing garbage collection vs. the time spent outside of garbage collection, considered over long periods of time. Memory footprints refer to amount of memory space utilized by JVM. 15
  16. 16. Exhibit 1: JVM Tuning Tradeoffs Let us understand how this tradeoff works. 1 Pause Times and Throughput With the JVM one can choose between garbage collection pauses per unit of time and application throughput. 1.1 Garbage Collection vs. Throughput The trade-off between garbage collection pauses and application throughput is partially caused by the mostly concurrent garbage collection strategy that enables brief garbage collection pauses. There are primarily two strategies for garbage collection. First, during garbage collection, Java threads are paused. In this strategy frequent garbage collection increases number of pauses which results in reduced application throughput. Second, during garbage collection, Java threads are not paused completely, they run during part of garbage collection duration. In this strategy garbage collection strategy requires more work due to new objects created and changes in references change. This causes longer duration for garbage collection execution. Both of the strategies results in reduced application throughput if garbage collection happens frequently but frequent garbage collection also results in increased application throughput. 16
  17. 17. Exhibit 2: JVM's Garbage Collection and Throughput 1.2 Defragmentation Pauses and Throughput The mark and sweep garbage collection algorithm can cause heap fragmentation when portion of memory too small for allocated objects are freed between blocks of live objects (similar to disk defragmentation on windows machines). Defragmentation or compacting the heap reduces this fragmentation. Fragmentation has a negative effect on the application throughput because of difficult object allocation and frequent garbage collections. The JRockit JVM does partial compaction of the heap in each garbage collection while Java threads are paused. Moving objects in memory takes time and is directly proportional to number of objects to be moved. Reducing the amount of defragmentation reduces the compaction pause time, but increased fragmentation results in reduced throughput. 2 Performance and Memory Footprint There is a trade-off for having a memory footprint and both application throughput and garbage collection pauses. A large heap reduces the garbage collection frequency and fragmentation, improving the throughput of the application; however, a large heap increases the memory footprint of the Java process. This also results in increase in bookkeeping effort with respect to live and dead objects and references and compaction duration. Over all application throughput and Garbage collection pause time increases. 3 JVM Choice When it comes to JVM, there are two popular choices: 17
  18. 18. 1. jRockit 2. HotSpot Other than Solaris, jRokit is better option to run WLS. I do not have any data to prove this bias but my experience and conversation with peers. On the basis of prefix JVM option it can be divided into three parts, JVM Options which starts with –X, -XX and others. 1. JVM Options that begin with -X are non-standard. These are not guaranteed to be supported on all JVM implementations and subject to change without notice in future releases of the Java. 2. JVM Options that begin with -XX are not stable and are not recommended for casual use. These options are subject to change without notice. 3. Apart from above mentioned there are several other options which are stable and standard across different implementations of JVM. Fusion Middleware SOA Suite (BPEL Engine) is multithreaded application which can host both long running processes and short one. Depending upon the processes hosted in a BPEL Engine, one need to tune JVM. jRockit is preferred option to run WLS on all platforms other than Solaris on which HotSpot is first choice. 4 jRockit’s Tuning 4.1 Boarding Into ship – Step 1 In this step we need to understand how to set Heap size, choosing Garbage Collection strategy, Nursery Size, and Garbage Collection pauses and their implications on performance. 4.1.1 Heap Size Heap is the memory area where Java objects reside. A large heap decreases the garbage collection frequency but may results in logger garbage collection cycle. Typically a heap should be at least twice the size of the live objects in the heap, so around half of the heap will be freed at each garbage collection. For Managed Server on which only BPEL engine is running heap size can be set as large as the available memory in system allow, as long as this doesn’t cause paging. On the server on which Admin server is also running with Managed Server, heap size of one should be accommodating both. There are two parameters to set Heap Size. 18
  19. 19. -Xms:<size>[g|G|m|M|k|K] Sets the initial and minimum heap size. -Xmx:<size>[g|G|m|M|k|K] Sets the maximum heap size. Where: g or G Gigabyte (GB) m or M Megabyte (MB) k or K Kilobyte (KB) If unit is not specified then byte (B). One should set the minimum and maximum heap size as same value to avoid unnecessary JVM effort to manage fluctuating heap size. For example a managed server running on a machine with 8 GB RAM memory could be started with the following settings: java -Xms:8000m -Xmx:8000m MyManagedServer This starts the JVM with a heap of 8000 MB and allows the heap to grow up to 8000 MB☺. 4.1.2 Garbage Collection Garbage collection uses system resources like any other process. By tuning the garbage collection one can decide how and when these resources will be used. JRockit JVM has three garbage collection strategies. The default strategy optimizes application throughput, second one optimizes pauses during garbage collection, and third one focuses on very short and deterministic garbage collection pauses. Garbage collection is optimized for application throughput. This is the default garbage collection strategy. -XgcPrio:throughput -XgcPrio:pausetime Garbage collection is optimized for short garbage collection pauses. Garbage collection is optimized for very short and deterministic garbage collection pauses. This option is only available in JRockit Real Time. -XgcPrio:deterministic JVM on which managed server (on which BPEL engine is running) is running should choose garbage collection strategy which optimizes application throughput. If BPEL engine is hosting short running synchronous processes only and load on server is constant (not varying within short interval of time), deterministic strategy might be better option. 19
  20. 20. 4.1.3 Nursery Size Nursery is an area of the heap where new objects are allocated. When the nursery becomes full it is garbage collected separately in a young collection. The nursery size determines the frequency and duration of young collections. A larger nursery decreases the frequency of young collection but increases the duration. In the JRockit JVM R27.3.0 and later the nursery size is adjusted automatically to optimize for application throughput if one uses -XgcPrio:throughput. For other garbage collection strategies and older versions of the JVM one needs to tune the nursery size manually. Typically the nursery size should be as large as possible while maintaining reasonably short young collection pauses. Depending on the number of processes/composites deployed in BPEL engine, nursery size can be anything from a few megabytes up to about half of the heap size. Nursery size should not be more than half of Heap size. 4.1.4 Pause Target -XgcPrio:pausetime and -XgcPrio:deterministic use a pause target for optimizing the pause times while keeping the application throughput as high as possible. • -XpauseTarget:<time> A higher pause target usually turns into higher application throughput, so one should set the pause target as high as BPEL Engine and hence deployed processes/composites can tolerate. For example a BPEL engine where transactions normally take 50 ms and time out after 350 ms could be started with the following settings: java -XgcPrio:deterministic -XpauseTarget:150 MyManagedServer These options will start up the JVM with garbage collection optimized for very short pauses with a pause target of 150 ms. This leaves a 200 ms margin before time-out for 350 ms transactions that are interrupted by a 150 ms garbage collection pause. 4.2 Leaving the shore – Step 2 To be able to tune JVM for better application throughput one must first have a way of assessing the throughput. A common way of measuring the application throughput is to time the execution of a pre-defined set of test cases. Optimally the test cases should simulate several different use cases and be as close to real scenarios as possible. Also, one test run should take at least a few minutes, so that the JVM has time to warm up. Before entering into this phase, one should be ready with some sample BPEL processes/composites which can model production grade real life ones. 20
  21. 21. 4.2.1 Lazy Unlocking This option facilitates synchronized Java code to run faster when the contention on the locks is low. But this option is available in JRockit JVM R27.3 and later -XXlazyUnlocking On all platforms except IA64, lazy unlocking is enabled by default for R27.5 for all garbage collection strategies except for deterministic. This option is intended for applications with many unshared locks. Since BPEL Engine and deployed processes/composites shares short lived locks, specifying this option may hit performance. 4.2.2 Call Profiling Call profiling records useful runtime statistics specific to the application. This option is supported in the JRockit JVM R27.3.0 and later versions. -XXcallProfiling Call Profiling option may increase performance because JVM can act on those statistics. 4.2.3 Large Pages The JRockit JVM can use large pages for the Java heap. To use large pages, one must first configure operating system for large pages. -XlargePages By default the JVM will continue to run without large pages if large pages cannot be acquired when -XlargePages is enabled. -XlargePages:exitOnFailure=true Use extended option to override default behavior and force the JVM to exit if enough large pages can’t be acquired. This extended option is supported in JRockit JVM R27.5 and later releases. This option tells the JRockit to use large pages, for the Java heap. Large pages allow application to more effectively use the translation look-aside buffer (TLB) in the processor, if available To configure large pages on operating system, refer Appendix E. 4.3 In the High Seas – Step 3 In most of the cases step 1 and 2 will be sufficient for tuning of JVM but in some cases, one needs to look into few more details. 21
  22. 22. 4.3.1 Compaction Compaction of objects is the process of moving objects closer to each other in the heap, thus reducing the fragmentation and making object allocation easier for the JVM. JRockit JVM compacts a part of the heap at each garbage collection. Compaction may leads to long garbage collection pauses. 4.3.1.1 Compaction Ratio The compaction ratio determines what percent of the heap will be compacted during garbage collection. -XXcompactRatio:<percentage> As a start, one can try lowering the compaction ratio to 1 and measure the pause. If pause is within tolerable limit, gradually increase compaction ratio till it start affecting BPEL engine and hence deployed processes/composites. A good value for the compact ratio is usually between 1 and 20, sometimes even higher. Compaction ratio must be in tune with compact set limit. Low value of compaction ratio may increase the fragmentation and the amount of free space that is too small to be used for object allocation. 4.3.1.2 Compact Set Limit The compact set limit sets a limit for how many references there can be to objects within the compaction area. If the number of references exceeds this limit, the compaction is canceled. -XXcompactSetLimit:<references> As a start, one can try setting the compact set limit at 10,000 and measure the pause. If pause is within tolerable limit, gradually increase the compact set limit as long as the compaction times stay low. A normal value for the compact set limit is usually between 100,000 and several million, while lower values are used when the pause time limits are very low. Setting the compact set limit too low may stop compaction from being done altogether. Running without any compaction at all may lead to increasing fragmentation, which will in the end force the JVM to perform a full compaction of the whole heap at once, which may take several seconds. -XXcompactSetLimit has no effect when -XgcPrio:deterministic XgcPrio:pausetime is used. or - 4.3.2 Thread Local Area size Thread local area (TLA) is a chunk of free memory reserved on the heap or the nursery and given to a thread for its exclusive use. A thread can allocate small objects in its own TLA without 22
  23. 23. synchronizing with other threads. When the TLA gets full the thread simply requests a new TLA. The objects allocated in a TLA are accessible to all Java threads and are not considered “thread local” in any way after they have been allocated. -XXtlaSize:min=<size>,preferred=<size> The “min” value is the minimum TLA size, while the “preferred” value is a preferred size. This means that TLAs will be of the “preferred” size whenever possible, but may be as small as the “min” size. Typically the preferred TLA size can be up to twice the size of the largest commonly used object size in the application. Adjusting the min size may have an effect on garbage collection performance, but is seldom necessary. A normal value for the min size is 2 KB. Increasing the TLA size is beneficial for BPEL Engine and hence deployed processes/composites (since BPEL engine is a multi-threaded applications and each thread allocates a lot of objects and size of objects is large). Increasing the TLA size too much may however cause more fragmentation and more frequent garbage collections. Exhibit 3: jRockit Attributes 5 HotSpot’s Tuning HotSpot claims that it is “ergonomic”. Based upon the platform configuration, it will select a compiler, Java heap configuration, and garbage collector for best results. But still human intervention in terms of its tuning is required because of staggering number of variations possible. HotSpot tuning focusses around Garbage Collection strategies and mode for server and client. This book will be focusing on server class machines only. Keeping this assumption, JVM startup command will contain following options in all scenarios: • • This option specifies that JVM is running in server mode -d64: This option specifies that JVM is running on 64 bit architecture -server: 23
  24. 24. 5.1 Garbage Collection Strategies 5.1.1 Serial GC Strategy Serial GC strategy performs well in Single processor machine since it is single threaded. Because of its very nature of algorithm, Serial GC strategy cannot take advantage of multiprocessor hardware. One can assume that any enterprise deployment of BPEL engine will be running on multi-processor and multi core machines, so we will not explore Serial GC strategy further. • -XX:+UseSerialGC 5.1.2 Parallel GC Strategy The parallel GC strategy, as name suggest performs minor GC collections in parallel to reduce GC overhead. Parallel GC strategy focusses on throughput of application by reducing pause time. • -XX:+UseParallelGC HotSpot JVM also has one more parallel GC strategy which also runs compaction in parallel to reduce fragmentation of Heap. • -XX:+UseParallelOldGC Without parallel compaction, major collections are performed using single thread, which can limit scalability. Parallel GC strategy (with or without compaction) is designed for small to medium scale applications. The concurrent GC strategy, as name suggests does most of its work while application is running, so minimizes pause time. This strategy focuses on response time rather than over all throughput. • -XX:+UseConcMarkSweepGC Concurrent collections can be performed incrementally as well. During a concurrent phase the garbage collector thread use one or more processors. The incremental option decreases the impact of long concurrent collections by periodically stopping the concurrent collection to yield back the processor to the application. This mode divides the work done concurrently by the collector into small chunks of time which are scheduled between young generation collections. This feature is useful when applications require small pause times and machine has small numbers of processors (e.g., 1 or 2). 24
  25. 25. • -XX:+CMSIncrementalMode Incremental concurrent collection must be specified with concurrent strategy option. java –server –d64 -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode To collect statistics and analyze, one should supply following option while starting HotSpot JVM. • -verbose:gc Verbose option prints heap and GC information at each collection. For example, here is output from a server application: [GC 383801K->94089K(999253K), 0.1903221 secs] [GC 325816K->83372K(776768K), 0.2454258 secs] [Full GC 267628K->83769K(776768K), 1.8479984 secs] Here we see two minor collections followed by one major collection. Explanation for first line (Minor collection): The numbers before and after the arrow (e.g. 383801K ->94089K) indicate the combined size of live objects before and after garbage collection, respectively. After minor collections the size includes some objects that are garbage (no longer alive) but that cannot be reclaimed. These objects are either contained in the tenured generation, or referenced from the tenured or permanent generations. The next number in parentheses (e.g. 999253K) is the committed size of the heap: the amount of space usable for java objects without requesting more memory from the operating system. Note that this number does not include one of the survivor spaces, since only one can be used at any given time, and also does not include the permanent generation, which holds metadata used by the virtual machine. The last item (e.g. 0.1903221 secs) indicates the time taken to perform the collection. The third line is similar to first two with only difference mentioning that this is full (major) collection. • -XX:+PrintGCDetails PrintGCDetails option prints amount of young generation, heap recovered and time spent in doing so. 25
  26. 26. Here is one output from -XX:+PrintGCDetails [GC [DefNew: 64289K->875K(64286K), 0.0397641 secs] 196554K->133321K(262284K), 0.0399680 secs] It indicates that the minor collection recovered about 98.6% ((64289-875)/64289) of the young generation and took 0.0397641 seconds. The usage of the entire heap was reduced to about 32% ((196554-875)/196554) and that there was some slight additional overhead for the collection (over and above the collection of the young generation) as indicated by the final time of 0.0399680 seconds • -XX:+PrintGCTimeStamps PrintGCTimeStamps option appends time stamp at the start of each collection. Here is one output from -XX:+PrintGCTimeStamps 99.032: [GC 99.032: [DefNew: 8136K->8136K(8136K), 0.0000210 secs]99.032: [Tenured: 18264K->2524K(24576K), 0.899311 secs] 45289K->2571K(33782K), 0.1294000 secs] The collection starts about 99.032 seconds into the execution of the application. The minor collection starts at about the same time. It indicates that the minor collection does not recovered anything ((8136-8136)/ 8136) of the young generation and took 0.0000210 seconds. The tenured generation usage was reduced to about 14% (100- ((18264-2524)/ 18264)) and took 0. 899311 seconds. The usage of the entire heap was reduced to about 6% (100- ((45289-2571)/45289)) and that there was some slight additional overhead for the collection (over and above the collection of the young generation) as indicated by the final time of 0.1294000 seconds Like jRockit, HotSpot also has mechanism to define Heap size. There are two parameters to set Heap Size. -Xms:<size>[g|G|m|M|k|K] Sets the initial and minimum heap size. -Xmx:<size>[g|G|m|M|k|K] Sets the maximum heap size. Where: g or G Gigabyte (GB) 26
  27. 27. m or M Megabyte (MB) k or K Kilobyte (KB) One should set the minimum and maximum heap size as same value to avoid unnecessary JVM effort to manage fluctuating heap size. For example a managed server running on a machine with 8 GB RAM memory could be started with the following settings: java -Xms:8000m -Xmx:8000m MyManagedServer This starts the JVM with a heap of 8000 MB and allows the heap to grow up to 8000 MB☺. Garbage Collection (minor as well as major collections) occurs when generations (young or tenured portion of heap) fill up, throughput is inversely proportional to the amount of memory available. Total available memory is the most important factor affecting garbage collection performance. 5.1.3 Parallel GC strategy JVM sets priority of targets at execution time. Priority 1: Maximum pause time target Priority 2: Throughput target Priority 3: Minimum footprint target On a machine with P processors, parallel collector uses P garbage collector threads; but, this number can be adjusted with following option • XX:ParallelGCThreads=<N> In parallel collector minor collection pauses are shorter. Because there are multiple garbage collector threads participating in the minor collection there is a small possibility of fragmentation due to promotions from the young generation to the tenured generation during the collection. Each garbage collection thread reserves a part of the tenured generation for promotions and the division of the available space into these "promotion buffers" can cause a fragmentation effect. Reducing the number of garbage collector threads will reduce this fragmentation effect as will increasing the size of the tenured generation. The maximum pause time target is specified with following option • -XX:MaxGCPauseMillis=<N> This is a hint to HotSpot JVM that pause times of <N> milliseconds or less are desired. By default there is no maximum pause time target. If a pause time target is specified, the heap size and other garbage collection related parameters are adjusted in an 27
  28. 28. attempt to keep garbage collection pauses shorter than the specified value. These adjustments may cause the garbage collector to reduce the overall throughput of the application. The throughput target is specified with following option: • -XX:GCTimeRatio=<N> It sets the ratio of garbage collection time to application time to 1 / (1 + <N>) For example, -XX:GCTimeRatio=29 sets a goal of 1/30 or 3.33% of the total time in garbage collection. Default value on Solaris is 99. Maximum heap footprint is specified using the • -Xmx:<size> The collector has an implicit goal of minimizing the size of the heap as long as the other goals are being met. If more than 98% of the total time is spent in garbage collection and less than 2% of the heap is recovered, an OutOfMemoryError is thrown. This feature prevents applications from running for an extended period of time while making little or no progress because the heap is too small. HotSpot provides option to suppress this error using • -XX:-UseGCOverheadLimit Do not use UseGCOverheadLimit, it hides the issue rather than solving. Exhibit 4: HostSpot Attributes 28
  29. 29. For BPEL engine’s managed server parallel GC is sufficient, so will not discuss the details of concurrent GC strategy. 6 Where to specify JVM and its parameters In a typical setup, to specify which JDK to be used, look for setDomainEnv.sh at $MW_HOME/user_projects/domains/[Domain]/bin/setDomainEnv.sh To specify JVM related parameters look for $MW_HOME/user_projects/domains/[Domain]/bin/setSOADomainEnv.sh 7 OutOfMemoryError Out of memory is one of the most common error for an JVM. At any time the JVM may run out of memory, which effectively crashes the server and requires a restart. There are many causes for out of memory exceptions. From too small of a heap defined, memory leak, poor quality code, not enough Perm Gen space defined, poor garbage collection settings, not enough physical memory being available on the machine where the JVM runs, etc. Most common reasons of out of memory error are following: • • • • java.lang.OutOfMemoryError: Java heap space java.lang.OutOfMemoryError: PermGen space java.lang.OutOfMemoryError: unable to create new native thread java.lang.OutOfMemoryError: requested xxx bytes for ChunkPool::allocate. Out of swap space 7.1 java.lang.OutOfMemoryError: Java heap space This indicates that the maximum defined heap space in -Xmx has been exceeded. It could be that the maximum heap is too small, the system is overloaded, there is a memory leak, Eden Space (Young Gen) is not allocated at an ideal percentage of the maximum heap, or the garbage collection scheme is not optimal for the applications running in the JVM. 7.2 java.lang.OutOfMemoryError: PermGen space This only applies to the HotSpot JDK only. The PermSpace contains: • Class information (not objects) • Constant strings and dynamic String.intern() objects • Hotspot compiled code • PermSpace is independent of Young/Old Space • It lives outside the region specified by Xms/Xmx 29
  30. 30. The PermSpace is normally not a target of tuning efforts, usually more space must be allocated in the -XX:PermSize/-XX:MaxPermSize parameters. There are a few exceptions though: • Applications that do dynamic class loading and unloading might suffer from a too small PermSpace • A PermSpace can trigger unnecessary Full GC activity with the Concurrent-Mark-Sweep (CMS) collector • An oversized PermSpace unnecessarily increases memory footprint which in turn increases the likelihood to hit the 2 or 4 g process size limit in 32 bit mode 7.3 java.lang.OutOfMemoryError: unable to create new native thread There is an upper limit on the number of operating system threads that can be open by one process, not to be confused with the number of file handle descriptors or jvm threads. When that os thread limit is reached, the java.lang.OutOfMemoryError: unable to create new native thread error is thrown. Possible solutions: • • • • • Set kernel parameter maxdsiz to a higher value Reduce the current heap size Check the kernel values: ulimit –a If the NPROC soft limit is lower than the hard limit, increase it as needed: ulimit -u <new value>, check the Operating System documentation to make changes permanent at the OS configuration files Need to reduce the JVM stack size and the OS stack size both, set the -xss in java options and set ulimit -s on the OS level 7.4 java.lang.OutOfMemoryError: requested xxx bytes for ChunkPool::allocate. Out of swap space This indicates HotSpot JVM ran out of native memory in the space reserved in native heap (Cheap) for compiling java methods. In most of the cases, application reaching the 4 GB address space limit for 32 bit processes, rather than to a shortage of installed memory or swap space. On Windows and on Linux, there can be 2 GByte and 3 GByte process address space limits. Reasons for reaching this limit might be: • • • • Memory leaks in calls to native libraries Loading many native libraries that fill up process address space Suboptimal JVM configurations due to changed memory requirements for an application Direct Buffers that are allocated outside of the garbage-collected heap (by java.nio.ByteBuffer.allocateDirect) 30
  31. 31. • • • Mapped Buffers, created by mapping a region of a file into memory (using java.nio.channels.FileChannel.map) mmap files using native implementations Any native resources that will tie up the address space Following actions might help to solve this error: • • • • • • • Is there any native memory leak in application? Does the code of application can be optimized for memory usage? Decrease the Java heap (-Xms/-Xmx) Decrease the permanent generation space (-XX:MaxPermSize) Decrease the stack size for Java threads (-Xss) Decrease the number of threads Increase HotSpot's code cache space For Java 6 the default minimum code cache size is 500 KB and the reserved code cache size is 32 MB. Sometimes these values are too small. One could try increasing the values by setting specific JVM options, example: -XX:CodeCacheMinimumFreeSpace=2M -XX:ReservedCodeCacheSize=64M Since process space is limited, running a demanding application under high load often involves finding an appropriate trade-off between native heap availability, Java heap availability, and threads. Reducing the Java heaps (-Xmx or -XX:MaxPermSize) will make more native heap available, but at the expense of reducing the number of Java objects or classes that can be accommodated. Reducing the stack size (-Xss) will also reduce the load on the native heap, but will reduce the available call stack depth, which might cause heavily recursive code to fail. Reducing the number of threads is usually only possible by reducing the load on the application. If the failure is not the result of a memory leak, and the application cannot be further optimized to reduce memory needs, then one might consider a move to a 64-bit JVM. However, this will inevitably use more memory than the same application on a 32-bit JVM, and the system memory must be sized appropriately. 7.5 Analysis In case of java.lang.OutOfMemoryError log files are generally unreliable and not of much use. One should try to understand, what is happening on the system at the time of error. From a SOA suite perspective one should look for answers of following questions: • Is the BPEL Audit Trail level set at Development causing additional overhead in the JVM when each Assign is being written to the audit trail? This should always be set to Production level or less on systems under load. • Are there large payload files being processed at the time of the error? • Is the composite set to do XML Validation which will add processing overhead? 31
  32. 32. • Analysis of the composites, are they looping, are there a lot of synchronous BPEL processes running for long periods of time each consuming memory or any other application misuse of memory? One should collect the heap usage and garbage collection statistics for analysis of java.lang.OutOfMemoryError. A memory leak is evident if the used heap goes up at an ever increasing rate with garbage collections having little or no impact on decreasing the used heap.. In both the JRockit and HotSpot JDKs one can pass parameters to dump the heap to a binary heap dump file at the time of out of memory: -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/oracle Note that the heap dump file will be a little larger than the maximum defined heap size so make sure that the defined HeapDumpPath has enough disk space to accommodate the file(s). On JRockit one can also produce an out of memory diagnostic file: -Djrockit.oomdiagnostics=true -Djrockit.oomdiagnostics.filename=/home/oracle This is a text file that will give additional information about what is happening with the memory. JRockit also has these: -XX:+|-HeapDiagnosticsOnOutOfMemoryError -XX:HeapDiagnosticsPath -XX:HeapDumpSegmentSize -XX:+|-HeapDumpOnCtrlBreak -XX:HeapDumpSegmentSize To force heap dumps on demand for JRockit: Apply -XX:+HeapDumpOnCtrlBreak to the start parameters of the JVM and use CTRL-BREAK to cause a heap dump, or in JRockit Mission Control. This will create a jrockit_<pid>.hprof heapdump file in the defined directory. 8 Reference 1. Comprehensive list of documents for jRockit: http://docs.oracle.com/cd/E13150_01/jrockit_jvm/jrockit/webdocs/index.html 2. Understanding of jRockit: http://docs.oracle.com/cd/E13150_01/jrockit_jvm/jrockit/geninfo/diagnos/part_01.html 32
  33. 33. 3. Performance tuning of jRockit: http://docs.oracle.com/cd/E13150_01/jrockit_jvm/jrockit/geninfo/diagnos/part_02.html 4. jRockit’s Command Line reference: http://docs.oracle.com/cd/E13150_01/jrockit_jvm/jrockit/jrdocs/refman/index.html 5. HotSpot’s option list: http://docs.oracle.com/javase/1.4.2/docs/tooldocs/windows/java.html 6. Comprehensive list of documents for HotSpot: http://www.oracle.com/technetwork/java/javase/tech/index-jsp-136373.html 7. HotSpot tuning: http://www.oracle.com/technetwork/java/javase/gc-tuning-6140523.html 8. Java SE 6 HotSpot Virtual Machine Garbage Collection Tuning: http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html 9. HotSpot VM options: http://www.oracle.com/technetwork/java/javase/tech/vmoptionsjsp-140102.html 33

×