Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Double the Performance of Oracle SOA Suite 11g? Absolutely! (whitepaper)


Published on

April 2013 - Collaborate/IOUG

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Double the Performance of Oracle SOA Suite 11g? Absolutely! (whitepaper)

  1. 1. Middleware 1 Session #187 DDOOUUBBLLEE TTHHEE PPEERRFFOORRMMAANNCCEE OOFF OORRAACCLLEE SSOOAA SSUUIITTEE 1111GG?? AABBSSOOLLUUTTEELLYY!! Ahmed Aboulnaga, Raastech, Inc. ABSTRACT Tuning Oracle SOA Suite 11g is an extremely vast topic that alone can span a book. Some changes are obvious, and others are quite hidden but can yield drastic performance improvements. One change will result in an immediate 32% performance improvement of your Oracle SOA Suite 11g infrastructure. Another will result in a further 24% performance improvement. This paper will cover some of the more impactful tuning recommendations to maximize the performance of your infrastructure, covering areas surrounding JVM, JTA, auditing, HTTP references, and more. TARGET AUDIENCE This paper is intended primarily for Oracle SOA Suite 11g administrators, but will benefit designers and developers as well. EXECUTIVE SUMMARY The reader will be able to:  Improve the overall performance of the Oracle SOA Suite 11g infrastructure  Learn about hard-to-find parameters designed to drastically improve the performance Oracle SOA Suite 11g  Gain insight on SCA design-time considerations to alleviate performance issues BACKGROUND Out of the box, Oracle SOA Suite 11g performs reasonably well. However, as activity grows and the complexities of your integrations increase, the default settings will not be sufficient to sustain the load and the infrastructure may not perform adequately. In this paper, we will present a few key areas that will yield considerable performance improvement; some of them obvious, and some of them quite unique. Typically in most Oracle SOA Suite 11g installations, the JVM heap size is rarely a concern as long as it is sized appropriately and is running PS3 ( or higher. On the midtier, disk I/O is also rarely a concern. However, Oracle SOA Suite 11g is very write intensive to the database, so database tuning is an area that should always be looked at by a competent DBA. As long as your heap and OS memory are sized appropriately, SWAP usage typically remains low. CPU utilization, likewise, is usually seldom a problem under load. MONITORING THE INFRASTRUCTURE Whenever a tuning exercise is conducted, typically there needs to be a way to measure baseline performance. Simply speaking, instance monitoring can be done by reviewing the actual instance flow through Enterprise Manager Fusion Middleware Control. Not only is this cumbersome, but the timestamps shown only display in the magnitude of seconds, as shown in the following screenshot.
  2. 2. Middleware 2 Session #187 Alternatively, running SQL queries against the MDS database schema is a much more preferred way. Not only are more granular timings providing, but grouping and aggregated statistics can be outputted. Observe the following SQL statement. SELECT DOMAIN_NAME PARTITION, COMPONENT_NAME, DECODE(STATE,'1','RUNNING','5','COMPLETED','6', 'FAULTED','9','STALE') STATE, TO_CHAR(AVG((TO_NUMBER(SUBSTR(TO_CHAR(MODIFY_DATE- CREATION_DATE),12,2))*60*60) + (TO_NUMBER(SUBSTR(TO_CHAR(MODIFY_DATE-CREATION_DATE),15,2))*60) + TO_NUMBER(SUBSTR(TO_CHAR(MODIFY_DATE-CREATION_DATE),18,4))),'999990.000') AVG, TO_CHAR(MIN((TO_NUMBER(SUBSTR(TO_CHAR(MODIFY_DATE-CREATION_DATE),12,2))*60*60) + (TO_NUMBER(SUBSTR(TO_CHAR(MODIFY_DATE-CREATION_DATE),15,2))*60) + TO_NUMBER(SUBSTR(TO_CHAR(MODIFY_DATE-CREATION_DATE),18,4))),'999990.000') MIN, TO_CHAR(MAX((TO_NUMBER(SUBSTR(TO_CHAR(MODIFY_DATE-CREATION_DATE),12,2))*60*60) + (TO_NUMBER(SUBSTR(TO_CHAR(MODIFY_DATE-CREATION_DATE),15,2))*60) + TO_NUMBER(SUBSTR(TO_CHAR(MODIFY_DATE-CREATION_DATE),18,4))),'999990.000') MAX, COUNT(1) COUNT FROM CUBE_INSTANCE GROUP BY DOMAIN_NAME, COMPONENT_NAME, STATE ORDER BY COMPONENT_NAME, STATE The query may look complicated, but only to format the output data in a clear and easily read format. The following screenshot shows the output of this query.
  3. 3. Middleware 3 Session #187 Another tool provided by Oracle SOA Suite 11g is the DMS Spy Servlet. By accessing its URL at http://<host>:<soaport>/dms/Spy, realtime metrics can be returned. The following screenshot shows the output of the DMS Spy Servlet. The WebLogic Server Administration Console provides numerous mechanisms to monitor performance. One of the main areas includes viewing the JVM runtime heap. By simply navigating to Home > Monitoring Dashboard, heap usage on the managed servers can be viewed in realtime.
  4. 4. Middleware 4 Session #187 It is important not to neglect other areas such as server disk space, CPU utilization, memory and I/O usage, as well as regularly reviewing the operating system message logs, open files, and running processes. These are often overlooked and server issues are one of leading causes of performance degradation. TUNING CONSIDERATIONS This section covers some key areas of tuning that should be considered in any Oracle SOA Suite 11g installation. The tuning options are endless, and this section covers some of the areas that will yield the most considerable performance improvements. The tuning recommendations documented here are based off of the recommendations from Oracle SOA Suite 11g Administrator’s Handbook, published in August 2012 by Packt Publishing. TUNING THE JAVA VIRTUAL MACHINE (JVM) JRockit is proven to be far superior in performance compared to Sun JDK, resulting in a 32% overall performance improvement. So consider switching from Sun JDK to JRockit JDK. A 64-bit JVM is recommended to avoid the 4GB (Linux) or 3GB (Windows) addressing space limitations. Once installed, converting to JRockit JDK is simple: 1. Edit $MW_HOME/wlserver_10.3/common/bin/ OLD: JAVA_HOME="/u01/app/oracle/jdk1.6.0_26" NEW: JAVA_HOME="/u01/app/oracle/jrockit1.6.0_29" OLD: JAVA_VENDOR=Sun NEW: JAVA_VENDOR=Oracle 2. Edit $MW_HOME/user_projects/domains/[Domain]/bin/ OLD: BEA_JAVA_HOME="" NEW: BEA_AVA_HOME="/u01/app/oracle/jrockit1.6.0_29" OLD: SUN_JAVA_HOME="/u01/app/oracle/jdk1.6.0_26" NEW: SUN_JAVA_HOME="" 3. Edit $MW_HOME/user_projects/domains/[Domain]/bin/ OLD: PORT_MEM_ARGS="-Xms768m -Xmx1536m" NEW: PORT_MEM_ARGS="-Xms1536m -Xmx1536m -Xgcprio:throughput -XXtlasize:min=16k,preferred=128k,wasteLimit=8k" TUNING THE JAVA TRANSACTION API (JTA) Ensure that the JTA timeout setting is not set to a value too low so that transactions are not rolled back unnecessarily. The recommended value is 120-300, but can be set higher as needed. This will reduce the occurrence of the following exception in the logs: Unexpected exception while enlisting XAConnection java.sql.SQLException: Transaction rolled back: Transaction timed out after 301 seconds For example, if an end to end flow is taking longer than the JTA timeout setting, it will rollback. If retries are involved, the retry counter is reset and may thus result in an infinite number of retries. To set the JTA: 1. Login to the WebLogic Server Administration Console 2. Click on soa_domain > JTA 3. Set the Timeout Settings to 300 TUNING THE LOG LEVEL Loggers should typically be set to ERROR in production environments, and set to varying levels of NOTIFICATION or TRACE on non-production environments to assist in troubleshooting. Logs are typically located under the
  5. 5. Middleware 5 Session #187 $MW_HOME/user_projects/domains/[Domain]/servers/[soa_server]/logs directory. Excessive logging results in considerably more disk space usage and also impacts performance. To set the logger levels: 1. Login to Enterprise Manager Fusion Middleware Control 2. Right-click on soa-infra, navigate to Logs > Log Configuration 3. Select the dropdown View With Persistent Log Level State 4. Apply the following log configuration settings: Log Levels - oracle.bpm ERROR:1 (SEVERE) Log Levels - oracle.integration ERROR:1 (SEVERE) Log Levels - oracle.sdp ERROR:1 (SEVERE) Log Levels - oracle.sdpinternal ERROR:1 (SEVERE) Log Levels - oracle.soa ERROR:1 (SEVERE) Log Levels - oracle.sysman ERROR:1 (SEVERE) Log Levels - oracle.wsm ERROR:1 (SEVERE) TUNING THE AUDIT STORE LOGGING On all non-development environments, set compositeInstanceStateEnabled to “false”. The first record of the following screenshot is shown with the composite instance state disabled while the second one with it enabled. Though keeping it enabled helps identify the state of the composite (e.g., running, completed), it also results in an overall 24% performance hit! This is because the dashboard must query the COMPOSITE_INSTANCE, CUBE_INSTANCE, and MEDIATOR_INSTANCE tables to maintain the actual state. So consider keeping this setting set to “true” in development environments and “false” in all others: 1. Login to Enterprise Manager Fusion Middleware Control 2. Right-click on soa-infra, navigate to SOA Administration > Common Properties 3. Uncheck Capture Composite Instance State 4. Also uncheck Payload Validation in non-development environments TUNING THE AUDIT LEVEL Reducing the audit level from Development to Production results in a 46% performance improvement. Changing the audit level from Production to Off results in a further 61% performance improvement. Thus, it is recommended to keep the audit levels set to Production in all non-development environments as follows: 1. Login to Enterprise Manager Fusion Middleware Control 2. Right-click on soa-infra, navigate to SOA Administration > Common Properties 3. Set Audit Level to “Production”
  6. 6. Middleware 6 Session #187 TUNING THE AUDIT CONFIGURATION POLICY Auditing in SOA Suite 11g has severe performance implications. Perhaps the largest composite instance performance gain can be attained by setting the audit policy to deferred, which results in all audit data being committed to the database asynchronously. Therefore, set the audit configuration to Deferred instead of the default Immediate setting: 1. Login to Enterprise Manager Fusion Middleware Control 2. Right-click on soa-infra, navigate to SOA Administration > Common Properties > More SOA Infra Advanced Configuration Properties 3. Click on Audit Config, and set the following properties as shown in this screenshot: TUNING THE BPEL SERVICE ENGINE There are many BPEL Service Engine properties that can be tuned, and the following table highlights some of the keys ones that will help the most: Parameter Name Default Recommended Setting Description Audit Trail 50000 20000 Messages with values greater than this threshold are saved in the AUDIT_DETAILS table instead of AUDIT_TRAIL. Dispatcher System Threads 2 10 Cleaning and housekeeping work for the engine. Dispatcher Engine Threads 30 100 Responsible for picking and processing instances. Set higher if you have a lot of processes with many dehydration points. Dispatcher Invoke Threads 20 70 Instantiates new instances in the engine. Tune higher value if there are multiple branches/flows for faster processing. Basically throttles the load setting of your BPEL Engine. AuditStorePolicy SyncLocal Transaction AsyncsyncLocal Transaction Will change the audit data storage strategy to save audit data asynchronously, using a separate local transaction. SyncMaxWaitTime 45 300 Determines the timeout for request-response operations. TUNING THE MEDIATOR SERVICE ENGINE There are many Mediator Service Engine properties that can be tuned, and the following table highlights some of the key ones. Not only will this improve overall throughput, but, for example, increasing the Mediator worker threads can result in a 30% performance improvement for asynchronous services. Parameter Name Default Recommended Setting Description Metrics Level Enabled Disabled Disable this unless DMS metrics collection is explicitly required. Parallel Worker Threads 4 20 Set appropriately; small (4) versus large environments (2). Parallel Maximum Rows Retrieved 200 600 Set appropriately; small (200) versus large environments (600). Parallel Locker Thread Sleep 2 1 Bad parameter. Always set to 1.
  7. 7. Middleware 7 Session #187 IMPLEMENTING BPEL IN MEMORY PERSISTENCE To improve the performance of certain types of BPEL processes, consider using inMemoryOptimization which prevents in-flight instances from writing to the database. Though this improves performance, the restrictions are that the WAIT and ONMESSAGE activities cannot be used, long running processes are at risk of being unable to rollback in the event of a server crash, and limited audit information is displayed on the screen. To reduce the number of times the BPEL process writes to the dehydration store, perform the following: 1. Add the following to the BPEL component in composite.xml to enable inMemoryOptimization: <BPELProcess src="YourProcess.bpel" id="YourProcess"> <configurations> <property name="inMemoryOptimization">true</property> </configurations> </BPELProcess> IMPLEMENTING HTTP TIMEOUTS It is strongly recommended to set HTTP timeouts at design time. This is to reduce consumption of threads in the event of poor performing external services. There are two parameters that help control this. HTTP Connection Timeout is based on whether SOA Suite is able to establish a connection with the remote server. HTTP Read Timeout is based on how long the actual remote service takes to respond. To set these properties at design time: 1. Add the following two properties in the reference section of composite.xml: <reference name="Register"> < port=" location="RegisterWS.svc.wsdl"> <property name="oracle.webservices.httpConnTimeout" type="xs:string" many="false">10000</property> <property name="oracle.webservices.httpReadTimeout" type="xs:string" many="false">10000</property> </> </reference> Note that these can also be manipulated at runtime through Enterprise Manager Fusion Middleware Control. USING MEDIATOR SEQUENTIAL ROUTING RULES Mediator executes routing rules either sequentially or in parallel, as dictated during design time. Sequential routing rules are superior in performance compared to parallel. This is because of the poor design that Oracle has chosen for avoiding the starving of threads. The problem with sequential routing rules though is that they are synchronous in nature and fault policies do not take effect. So consider using them for all end-to-end synchronous Mediator flows: 1. Edit all *.mplan files for your Mediator operations and change the executionType as follows: OLD: executionType="direct" NEW: executionType="queued" Considering this is a design time setting and that sequential routing rules will not provide true asynchronicity, test them out to understand the behavior first. Implementing sequential routing rules may improve performance of Mediator instances anywhere from 4% to 509%.
  8. 8. Middleware 8 Session #187 SUMMARY Tuning Oracle SOA Suite 11g is a very lengthy exercise, as there are a near unlimited number of areas that can be addressed, even expanding to the WebLogic Server, Oracle Database, and the underlying operating system. The tuning considerations provided in this paper were meant to highlight core areas of tuning that would yield considerable performance improvements in most environments. Some of the key takeaways of this paper include:  Upgrading to PS3 ( addresses memory instability issues  Moving from Sun JDK to JRockit results in a 32% boost  Optimizing the audit level makes a huge difference, resulting in a 46% to 92% performance improvement  Increasing the Mediator worker threads results in a 30% performance improvement for async services  Mediator sequential routing rules may improve performance from 4% to 509%  Don’t forget to tune the BPEL Service Engine  Consider parallel garbage collection, but don't have statistics The majority of the recommendations here are pulled from Oracle SOA Suite 11g Administrator’s Handbook, published in August 2012 by Packt Publishing. Chapter 4 of the book, titled Tuning Oracle SOA Suite 11g for Optimum Performance, includes hundreds of other recommendations. REFERENCES  “Oracle SOA Suite 11g Administrator’s Handbook,” Ahmed Aboulnaga & Arun Pareek (Packt Publishing, 2012)  “Performance tuning Oracle SOA Suite 11g,” Ahmed Aboulnaga, April 22, 2011