OTM Performance Review and Benchmarking


Published on

More and more clients are looking to understand the capabilities of the OTM/G-Log architecture and configuration in order better tune OTM. Usually, this is required because of poor OTM performance or as preparation for significant changes to OTM configuration, volume, or platform. The client may be experience poor performance throughout the entire system or for a very specific use cases. The primary objective of a Performance Tuning Exercise is to understand how OTM is being utilized and to recommend solution to improve the performance of OTM.

We recommend and will take the audience through a “ground-up” performance tuning exercise, starting with hardware and infrastructure, moving to Java and App server tuning, then to OTM technical tuning and finally to the OTM functional tuning (data, agents, etc).

These audits may identify hardware constraints at each tier, networking, or other infrastructure constraints causing sub-optimal system performance. Simply stated, the performance audit will identify all bottlenecks in the system if they exist.

In many cases the largest performance is impacts are not hardware, but rather how the data is configured within the application. So as part of the exercise we will analyze database performance, individual SQL queries, OTM Queues, bulk planning parameters, agents, rates and the settlement process.

Understanding the methods which will best identify these bottlenecks will help you avoid performance issues early in your project and save considerable time and expense as you near go-live. This presentation will guide you through the steps necessary to better understand what is impacting performance and how to best handle it. It will provide lessons learned and tools that are available to you better manage and maintain a healthy OTM environment.

Presented by Chris Plough at MavenWire

Published in: Business, Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

OTM Performance Review and Benchmarking

  1. 1. OTM Performance Review and Benchmarking OTM SIG User Conference ‘08
  2. 2. Agenda Performance Analysis Methodologies Our Holistic Bottom-Up Approach Benchmarking Other Options Q&A
  3. 3. Methodologies The “Top-Down” Approach Great for determining the root-cause of specific performance issues. The “Bottom-Up” Approach Great for doing a site review and identifying issues and bottlenecks (current and potential). The Holistic Approach Ensure that all components (technical, functional and external) are taken into consideration.
  4. 4. MavenWire’s Holistic Bottom-Up Approach Our Approach Hardware / Platforms Operating System Java Tuning Application Server Tuning OTM Thread Tuning OTM Diag Servlets Agents Rates Itineraries Logs Benchmarking
  5. 5. Hardware / Platforms CPU and Hardware Platform Matters! CPU Speed – Not a Good Indicator of Performance Other factors (cores, memory bandwidth, on-chip cache) necessitate benchmarking OTM Requires both high multi-threading and high single-thread performance Lots of cores and high per-core performance Performance of Current Platforms Linux / x86-64 Windows / x86-64 (note: memory limitations) Solaris HP-UX / PA-RISC Note: HP-UX / Itanium currently unknown AIX / POWER
  6. 6. Operating System / Stats Review system performance under production load for the previous 2 weeks Utilize System Tools to Monitor sar / kSar top / prstat / topas / etc Utilize Tools to Trend Nagios / Munin / etc
  7. 7. Oracle DB The following have improved DB performance Ensure you’re updating DB stats regularly Patched to Partitioning is enabled CURSOR_SHARING from EXACT to SIMILAR OPTIMIZER_MODE = CHOOSE STATISTICS_LEVEL = ALL In some cases the following helped OPTIMIZER_FEATURES_ENABLE = 9.2.0 Otherwise – Tune it like a normal Oracle DB Standard DBA skill set and best practices Utilize tools like the Oracle DB Statspack Increase memory (PGA / SGA / etc) allocation Pin frequently used packages/procedures to memory Decrease storage IO Wait (more spindles, etc) Separate out indexes, tablespaces, logs
  8. 8. Java Tuning OTM is HIGHLY Dependent on JVM Performance JRockit Performs considerably faster than other JVMs Many Current-Generation JVMs Self-Tune (including JRockit) Platform Specific Parameters Allocate as much Memory as Possible 2-3GB depending on platform For both Web and App server Monitor Garbage Collection Most frequent JVM performance issue OTM v6.0 will utilize a 64-bit JVM NOT a silver bullet!
  9. 9. Application Server Tuning Shouldn’t need to tune Apache or Tomcat WebLogic vs. OAS OTM has been running on WL since 1999 May no longer be an issue with BEA acquisition WebLogic Tuning Utilize the WebLogic Console http://otmapp.mavenwire.com:7001/console Number of Execute Threads $OTM_HOME/weblogic/config/gc3domain/config.xml.template ExecuteThreads Should be 70-90, depending on load Percentage of Socket Reader Threads $OTM_HOME/weblogic/config/gc3domain/config.xml.template ThreadPoolPercentSocketReaders May have to increase to 75 DB Connection Pool Now Tuned within the OTM Application
  10. 10. OTM Tuning Ensure You’re Running the Latest RU Thread Tuning Take your time – verify results Tune iteratively – avoid contention If you don’t know what a thread pool does – ask first Pay attention to Queue Size and Wait Time Temporarily Tune Threads via the EventDiagServlet Careful about killing threads! Permanently Set Threads via the glog.properties file OTM Thread Tuning No Bottleneck? Check the MediatorDiagServlet
  11. 11. OTM Diagnostic Servlets OTM Event Diag Servlet http://otmweb.mavenwire.com/GC3/glog.webserver.event.E ventDiagServlet OTM Topic Queue Assignments Servlet http://otmweb.mavenwire.com/GC3/glog.webserver.event.T opicQueueAssignmentsServlet OTM Mediator Diag Servlet http://otmweb.mavenwire.com/GC3/glog.webserver.mediat or.MediatorDiagServlet OTM Object Lock Diag Servlet http://otmweb.mavenwire.com/GC3/glog.webserver.synch. object.ObjectLockDiagServlet OTM Connection Pool Diag Servlet http://otmweb.mavenwire.com/GC3/glog.webserver.dataso urce.CPDiagServlet OTM Bean Cache Servlet http://otmweb.mavenwire.com/GC3/glog.webserver.beanca che.BeanCacheServlet
  12. 12. Agents Try to model agents so only one agent fires for any given event Review your saved conditions for optimal performance Leverage other functionality than Agents – Auto Assignment Rules, Contact Notifications
  13. 13. Rates Ensure that rate offerings that are expired are also inactive If you have rate offerings active ensure that rate records are associated to them. Avoid redundancy in your rate structure. If you rate offering can only handle once piece of equipment there is no need to also define it at the rate record level.
  14. 14. Itineraries The more itineraries valid for a given move the longer OTM will take to plan. Test using the order planning action “Show Routing Options” Simply your itineraries where ever possible
  15. 15. Logs To much logging turned on will impact performance. Turn logging on as needed to troubleshoot Log features that can impact performance SQL RatingEngine RatingEngineDetail RatingEngineDebug Persistence
  16. 16. MavenWire Benchmark Suite Suite of Benchmarks compiled to allow comparative hardware / platform testing Does not require OTM installation, decreasing setup complexity and testing time Freely available to the OTM Community All Benchmarks utilized are Open Source Software (OSS), allowing for free use and modification as necessary Full Details, including download, installation, runtime and comparison data available at: http://www.otmfaq.com/forums/blogs/chrisplough/
  17. 17. Benchmarking - VolanoMark VolanoMark Java-based benchmark that simulates high transactional and multi-threaded load Reflects the performance of the following OTM activities Web UI, Agents, Integration, General Workflow, General OTM Activities (not including optimization and planning based) Higher numbers are better
  18. 18. Benchmarking - DaCapo DaCapo Java-based benchmark that simulates highly computational, algorithmic, single-threaded processing Reflects the performance of the following OTM activities Optimization and Planning / Bulk Planning Lower numbers are better
  19. 19. Benchmarking – Soap Stone Soap Stone Java-based benchmark that tests data throughput between servers and replicates application protocols, such as HTTP, RMI and RAW. Reflects the throughput and protocols utilized between the various OTM Tiers Browser / Web: HTTP Web / App: RMI App / DB: RAW Higher numbers are better
  20. 20. Benchmarking – Hammerora Hammerora Benchmark based on the TPC-C and TPC-H benchmarks. Reflects the performance and scalability of the DB Tier Lower numbers are better
  21. 21. Other Options Web Tier Load Balancing SSL Accelerator WAN / Web App Accelerator App Tier OTM SCA DB Tier Oracle RAC
  22. 22. Online Resources Performance kSar http://ksar.atomique.net/ Nagios http://www.nagios.org/ Munin http://munin.projects.linpro.no/ Benchmarking Full Replication Details http://www.otmfaq.com/forums/blogs/chrisplough/ VolanoMark http://www.volano.com/benchmarks.html DaCapo http://dacapobench.org/ Soap Stone http://soap-stone.sourceforge.net/ Hammerora http://hammerora.sourceforge.net/
  23. 23. Q & A and Discussion Questions?
  24. 24. Thank You! chris.plough@mavenwire.com 866.343.4870 x701 www.MavenWire.com