WSI33 - Advanced Performance Tactics for IBM WebSphere Application Server

IBM European WebSphere
Technical Conference
14 – 18 November 2008, Barcelona, Spain
© 2008 IBM Corporation
Conference materials may not be reproduced in whole or in part without the prior written permission of IBM.
Advanced Performance Tactics forAdvanced Performance Tactics for
IBM WebSphere Application ServerIBM WebSphere Application Server
Session Number: WSI33Session Number: WSI33 Hendrik van RunHendrik van Run –– hvanrun@uk.ibm.comhvanrun@uk.ibm.com

WebSphere Technical Conference and Transaction & Messaging Technical Conference
2
AgendaAgenda
Threading
HttpSession Manager
EJB Container
JMS Performance Tuning
Dynamic Cache Service
High Availability Manager
JVM Tuning
64 bit WebSphere
Hardware

3
ThreadingThreading
Channel Framework TuningChannel Framework Tuning
Channel Framework
Non-blocking I/O in WAS 6.0 and higher
Accepts very large numbers of concurrent in-bound requests
Unlikely to see “connection refused” errors at high loading
Stabilizes the Web Container even under bursty conditions
Web Container thread tuning
Tuning practices unchanged with Channel Framework
Defines the maximum concurrent activity executing in the Web Container
Limited by CPU, memory, remote resources, etc.
Tune to drive CPU to maximum
• Avoid exceeding more threads than CPU can usefully engage
• 50-75 threads about the “normal”
Channel framework working with the Web Container
Handles requests exceeding available container threads
Response time will increase after throughput saturation point
However, web container will still do useful work and accept connections

4
AgendaAgenda
Threading
HttpSession Manager
EJB Container
JVM Tuning
64 bit WebSphere
Hardware

5
HttpSession ManagerHttpSession Manager
MemoryMemory--toto--Memory replicationMemory replication
Simplified configuration
Terminology changes
No more replicas and partitions, instead we have client and servers in a replication
domain
Topology changes
Replication domain is no longer collection of replicas
Integration with WLM
Provides hot failover in peer-to-peer mode

6
PeerPeer--toto--Peer Memory replicationPeer Memory replication

7
Client Server Memory replicationClient Server Memory replication

8
AgendaAgenda
Threading
HttpSession Manager
EJB Container
JVM Tuning
64 bit WebSphere
Hardware

9
EJB ContainerEJB Container
Tuning the EJB CacheTuning the EJB Cache
Cache size = Maximum concurrent active instances
Entity Beans required per transaction * maximum expected concurrent transactions
Add maximum active Stateless Session Bean instances
Don’t be afraid of large EJB caches if you have sufficient heap
Benchmark caches routinely sized at 30K+ entries
If the cache fills up, the container tries to passivate based on LRU
Passivation triggers disk I/O – slow
Remember, some EJBs may be long lived
Stateful Session Beans
In WAS V6.0x
Lazy initialization of EJBs
Faster server startup with large EJB applications

10
AgendaAgenda
Threading
HttpSession Manager
EJB Container
JVM Tuning
64 bit WebSphere
Hardware

11
JMS Performance TuningJMS Performance Tuning
Reliability levelsReliability levels
BEST_EFFORT_NONPERSISTENT
Messages are never written to disk
throw away messages if memory cache over-runs
EXPRESS_NONPERSISTENT
Messages are written asynchronously to persistent storage if memory cache
overruns, but are not kept over server restarts
No acknowledgement that the ME has received the message
RELIABLE_NONPERSISTENT
Same as Express_Nonpersistent, except, we have a low level
acknowledgement message that the client code waits for, before returning to
the application with an OK or not OK response
RELIABLE_PERSISTENT
Messages are written asynchronously to persistent storage during normal
processing, and stay persisted over server restarts.
If the server fails, messages might be lost if they are only held in the cache at
the time of failure.
ASSURED_PERSISTENT
Highest degree of reliability where assured delivery is supported
High
Reliability
High
Performance

12
Data BuffersData Buffers
Each Messaging Engine has two data buffers to store messages
Discardable data buffer (only used for Best Effort Non-persistent)
Cached data buffer (all other messages)
Messages are discarded from the Databuffer using FIFO
Messages from Cached data buffer are still available from the database
Messages from Discardable data buffer are lost when discarded!
Data storeMessaging Engine
Message control structure
Message itself
D
C
iscardable data buffer
ached data buffer
JDBC

13
Sizing the data buffersSizing the data buffers
Default size is 320 KB for both data buffers
This is very small; likely that messages will be discarded from the buffers
Discarded messages from the Discardable data buffer will throw an exception
Look for com.ibm.ws.sib.msgstore.OutOfCacheSpace in SystemOut.log
Discarded messages from the Cache data buffer will not throw an exception
But messages might need to be retrieved from the database, hurting performance
Actively monitor PMI data
Configure data buffers using custom properties for the SIBus
sib.msgstore.discardableDataBufferSize
sib.msgstore.cachedDataBufferSize
PMI: Performance Modules > SIB Service > SIB Messaging Engines > Messaging Engine > Storage
Management > Cache
CacheStoredDiscardCount (for Cache data buffer)
CacheNotStoredDiscardCount (for Discardable data buffer)

14
WAS 6.1 Flat File Data Store OptionWAS 6.1 Flat File Data Store Option
WAS 6.1 introduces new Messaging Engine store options
Flat file (new)
JDBC (traditional)
Flat file advantages
Easier to setup
Faster than database engine in many cases
Flat file HA participation
Place file on highly available, shared drive
• GPFS, NFSv4, etc.
Use IBM File System Locking Protocol Test for verification
http://www.ibm.com/support/docview.wss?uid=swg21215152

15
AgendaAgenda
Threading
HttpSession Manager
EJB Container
JVM Tuning
64 bit WebSphere
Hardware

16
Dynamic Cache ServiceDynamic Cache Service
OverviewOverview
Consider Dynamic Caching during application design
Part of design rather than retrofitting later
Organize page layout and design for caching
Servlet/JSP caching to hold repetitive, expensive page elements
Stateful elements vs. general use elements
API Caching via Distributed Map services
Leverage Dynamic Caching services within the application
Leverage the caching structure
Multiple internal caches supported beginning in WAS 6.0x
Split objects into different caches
Distribute cache data more efficiently

17
OverviewOverview
Two types of caching function
Policy Based (defined via the cachespec.xml file)
• Servlet/JSP
• Commands
• Web Service
• Web Service Client (JAXRPC)
API Based
• Distributed Map
• Cacheable Servlet
• Cacheable Command
Distribution features
Distribute cache contents to peer instances
Push content to HTTP servers and other “edge” components
Dynamic Cache Service has limitations for eXtreme Transaction Processing applications
Extremely scalable applications with large number of transactions acting on vast amounts of data
WebSphere Extreme Scale provides much better capability than the Dynamic Cache Service
Covered in more detail in session WSI35 – “WebSphere Extreme Scale Customer Scenarios and
Use Cases”

18
Features in WebSphere 6.0/6.1Features in WebSphere 6.0/6.1
Dynamic Caching Features
Struts and Tiles Caching Support
Dynamic Content Provider
• Allows cacheable fragment to contain non-cacheable content
• An exit calls out when fragment retrieved to get dynamic content
Servlet and Object Cache Instances
• Multiple cached instances of the same servlet/object in a single server
Distributed Object Cache enhancements
Enhanced Cache Monitor tool
• Support new caching features, including multiple caches
Disk offload
• Allows cache to overflow to the hard disk
• Also, may save cache to disk when stopping WAS instance
Expanded policy definitions
• Including “static” caching
WAS 6.1 significantly improved disk cache offload performance!
This improvement has been backported to WAS 6.0.2 and WAS 5.1.1
http://www.ibm.com/support/docview.wss?uid=swg24013097

19
AgendaAgenda
Threading
HttpSession Manager
EJB Container
JVM Tuning
64 bit WebSphere
Hardware

20
High Availability ManagerHigh Availability Manager
WAS 6.0 introduced the High Availability (HA) manager
HA Manager is enabled by default and typically required by the WAS runtime
WAS 6.0.2.5 and higher allow disabling of this feature
• Normally not recommended!
HA manager is a collaborative approach to failure management
Instances in regular communication (“heartbeats”)
On failure, other instances assume tasks of failed instance
Examples of services depending on HA manager
• Transaction log recovery service
• Service Integration Bus (Default JMS Provider)
• Services using Distribution and Consistency Service
− Memory-to memory replication of sessions
− Dynamic Cache Service replication
“Core groups” concept
Set of instances working together to support HA manager
By default, all instances belong to default core group
Core group bridging
Allows communication between core groups
Example: EJB location, etc.

21
High Availability ManagerHigh Availability Manager
Performance ConsiderationsPerformance Considerations
HA manager requires synchronization at start up
CPU intensive as the core group size increases
Recommendations
Keep the core group size to 50 or less
High performance sites might require smaller core groups
• Instances may be marked down because of delayed heartbeat response
Migrating from WebSphere 5.x with very large instance deployments
Be aware of this issue
Re-allocate large environments to smaller core groups, if required
Disable HA manager only as a “last option”
• Many features may use HA manager functionality
• Future features also may leverage HA manager

22
AgendaAgenda
Threading
HttpSession Manager
EJB Container
JVM Tuning
64 bit WebSphere
Hardware

23
JVM TuningJVM Tuning
HeapsizeHeapsize
Sizing the Java heap appropriately is key to good WebSphere performance
Always keep the JVM(s) within the physical memory of the server
Determine good maximum heap size through testing
Enable verbose:gc and analyse the native_stderr.log
• Use “IBM Pattern Modeling and Analysis Tool for Java Garbage Collector”
− Tool available through IBM Support Assistant
http://www.ibm.com/software/support/isa/
• Other tools can be used as well
Good starting point for maximum heap size
• 512 MB for WebSphere Application Server
• 1024 MB for WebSphere Process Server
• 1024 MB for WebSphere Portal Server
Determining minimum heapsize is usually easier
Production systems set minimum heap size lower than the maximum
• Allows the JVM to determine the optimal heap size
• Can result in a more efficient object table
• Gives headroom for emergencies
Some “Burst” scenarios might require setting the minimum equal to the maximum
• Avoids resizing the heap completely

24
JVM OptionsJVM Options
WAS 6.0
IBM Java 1.4.2 JVM
32 and 64 bit editions
Flat memory model (all editions)
Generational model (selected 64 bit editions)
WAS 6.1
Introduction of IBM Java 5.0 JVM
Flat or generational garbage collection
Support for shared classes
WAS 7.0
Introduction of IBM Java 6.0 JVM
Flat or generational garbage collection
Compressed References Technology provides enhanced 64 bit performance

25
IBM 1.4.2 JVM (and earlier)IBM 1.4.2 JVM (and earlier) –– Controlling Memory FragmentationControlling Memory Fragmentation
Pinned and dosed objects are unmovable during compaction
Example: Native calls generate pinned objects
Heap fragmentation could become an issue
Especially true with large objects
OutOfMemory occurs even if there appears to be sufficient free heap available
kCluster is an area of storage for class blocks
Default size = 1280 (entries) x 256 (bytes)
Can be reset via JVM argument –Xk<size>
pCluster is an area for pinned objects
16KB by default
Newly created pCluster are 2KB in size
Can be reset using –Xp<iiii>[K][,<oooo>[K]]
• iiii = initial pCluster size
• oooo = size of subsequent overflow pClusters

26
IBM Java 5.0 JVMIBM Java 5.0 JVM
IBM implementation of Java 5.0 runtime
Built to the Java 5.0 specifications
IBM Java 5.0 JVM manages JNI objects differently
No pinned or dosed objects
Reduces fragmentation
Two memory models available in IBM Java 5.0 JVM
Flat model: similar to IBM Java 1.4.2 (Sovereign) model
Generational: similar idea as Sun and HP generational collectors
Simplified model and tuning
Variety of garbage collection policies
Optimizations for pause times or processing time
Sub-pool support to reduce thread contention
Improved Just-in-Time (JIT) compiler
Longer warm-up for optimized code
No longer disabled for remote debugging
Faster WAS startup times with remote debugger attached

27
IBM Java 5.0 JVMIBM Java 5.0 JVM –– shared classesshared classes
IBM Java 5.0 introduced support for shared classes
Multiple WAS JVMs can share common WebSphere classes
Offers significant startup speed improvements and memory savings
Cache persistent beyond the lifetime of any JVM (but lost on system shutdown)
WebSphere Application Server 6.1 has shared classes enabled by default
Except for platforms that do not use the IBM Java 5.0 runtime (Sun Solaris and HP/UX)
–Xshareclasses:name=webspherev61_%g,groupAccess,nonFatal -Xscmx50M
• Name of the cache is webspherev61_%g
• Cache can be accessed by all users in the same group
• Use of the shared classes is nonFatal (i.e. the JVM will start without it in case of problems)
• Maximum size of the cache is 50 MB
Cache can be cleared manually using a script
• $WAS_HOME/bin/clearClassCache.[bat|sh]
Cache can be disabled per WebSphere Application Server process if required
Include –Xshareclasses:none on Generic JVM arguments settings

28
IBM Java 6.0 JVMIBM Java 6.0 JVM
IBM Java 6.0 introduces Compressed Reference (CR) technology
Reduces the width of 64 bit Java heap references to 32 bits
Implements an efficient bit shifting algorithm to accomplish this
Can be used for heap sizes up to 28 GB
WebSphere Application Server 7.0 benefits from CR technology
Reduces 64 bit WebSphere memory footprint back to 32 bit equivalent
• Typical memory footprint of 64 bit WebSphere 6.1/6.0 was 60-70% larger than 32 bit
Reduces performance overhead of 64 bit WebSphere
• Performance is generally around 95% compared to 32 bit WebSphere
• Minimal performance cost is due to reference compression/decompression
Full details to be provided in a whitepaper
“IBM WebSphere Application Server WAS V7 64-bit performance – Introducing
WebSphere Compressed Reference Technology”
http://www.ibm.com/software/webservers/appserv/was/performance.html

29
IBM Java 6.0 JVMIBM Java 6.0 JVM –– shared classesshared classes
IBM Java 6.0 further extends the support for shared classes
Shared cache information can now be persisted to the filesystem
Cache can survive a system restart
Reduces startup time of WebSphere processes, even after a reboot!
•–Xshareclasses:persistent
enables persistent cache (default except on z/OS)
•–Xshareclasses:nonpersistent
disables persistent cache
Shared cache can now also stored Ahead of Time (AOT) compiled code
The Just in Time (JIT) compiler generates AOT compiled code (native code)
• Speeds up execution of application code
• Compilation can now be avoided if the AOT compiled code is in the shared cache
IBM Java 5.0 only allowed for sharing the static class data

30
AgendaAgenda
Threading
HttpSession Manager
EJB Container
JVM Tuning
64 bit WebSphere
Hardware

31
WAS 64 bit supportWAS 64 bit support
OverviewOverview
64 bit WAS available
6.0.1
• Linux on Power
• Linux on x86-64 Platforms
6.0.2
• 6.0.1 platforms
• AIX on Power
• Windows Server 2003 Enterprise x64 Edition on x86-64
6.1 and 7.0
• 6.0.2 platforms plus Solaris/HP-UX
Allows WAS JVM to grow well beyond 32bit process size boundaries
Note that typical Java application requires 60-70% more heap on WAS 64 bit
Typically lower performance due to larger heap and lower cache efficiency
• The exception is WAS 7.0 64 bit on IBM platforms
• Compressed Reference yields performance of ~95% of 32 bit WAS for heaps up to 25 GB
Applications experiencing greatest benefit
Memory constrained
• Extra memory supports better caching strategy
Computationally expensive code
• Security algorithms, etc.

32
AgendaAgenda
Threading
HttpSession Manager
EJB Container
JVM Tuning
64 bit WebSphere
Hardware

33
HardwareHardware
General ConsiderationsGeneral Considerations
Maximize L2/L3 CPU caches
Especially important for 64 bit systems
• Orderable option for some platforms
• Typically inexpensive
• Not upgradeable/add-on
Increases effective statement pipeline

34
HardwareHardware
Power5/Power6 architecturePower5/Power6 architecture
Two cores per chip
Supports two-way Symmetric Multi Threading (SMT) per core
Supported on AIX 5.3
Operating system “sees” 4 CPUs per chip
Recommendation is to enable SMT for OLTP WebSphere applications
• SMT provides a significant performance enhancement when many threads are executing similar code
• More information on SMT and Power 5 available here
http://www-941.ibm.com/collaboration/wiki/display/WikiPtype/POWER5+Architecture
Partitioning capabilities are very powerful
Static LPARs
Dynamic LPARs
Dynamic Micro-Partitioning
Beware of performance impact of Dynamic Micro-Partitioning
Fractional amount of physical CPU resources can be assigned to LPARs
Processor cache of physical CPU will be flushed frequently
Frequent flushing of processor cache can significantly reduce performance for applications that are sensitive to the
efficiency of these caches, for example:
• WebSphere Application Server
• DB2 Universal Database

35
HardwareHardware
Power5/Power6 architecturePower5/Power6 architecture –– continuedcontinued
Large Page support on Power 4 or higher
16 MB memory pages instead of default 4 KB
10-15% better performance improvement for processes that use a lot of memory
• For example WebSphere Application Server
Large page support requires all memory pages in a 256 MB segment to be large pages
• Requires a change to OS configuration plus reboot
Medium Page support on Power5+ and higher
64 KB memory pages instead of default 4 KB
Almost the same performance boost as 16 MB pages
No need to change OS configuration
Full details in WebSphere information center
• http://publib.boulder.ibm.com/infocenter/wasinfo/v6r1/topic/com.ibm.websphere.nd.do
c/info/ae/ae/tprf_tuneaix.html
• http://publib.boulder.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=/com.ibm.webs
phere.nd.doc/info/ae/ae/tprf_tuneaix.html

36
SummarySummary
WebSphere performance spans many areas and features
High-end function and features provide performance
when used appropriately
Hardware and OS selection play important roles in performance

37
ReferencesReferences
“Improving HttpSession Performance with Smart Serialization”, Brown, Botzum
http://www.ibm.com/developerworks/websphere/library/bestpractices/httpsession_perfor
mance_serialization.html
“IBM WebSphere Application Server WAS V7 64-bit performance – Introducing
WebSphere Compressed Reference Technology”, Cox et al
http://www.ibm.com/software/webservers/appserv/was/performance.html
IBM Support Assistant
http://www.ibm.com/software/support/isa/
Many thanks to Stacy Joines, Bill Hines, Jessica Man, Dipak Patel, Leigh
Williamson, Paul Edlund, and others for their contributions to this presentation.

38
Other SessionsOther Sessions
WSI32 – WebSphere Performance Fundamentals
WSI35 – WebSphere Extreme Scale Customer Scenarios and Use Cases

39
Further ReadingFurther Reading
Performance Analysis for Java Web Sites
Joines, Willenborg, Hygh
IBM WebSphere Deployment and Advanced Configuration
Barcia, et al
IBM WebSphere System Administration
Williamson, et al
WebSphere Application Server: Step by Step
Turaga, et al
Persistence in the Enterprise
Barcia, Hambrick, et al

IBM European WebSphere
Technical Conference
14 – 18 November 2008, Barcelona, Spain
Conference materials may not be reproduced in whole or in part without the prior written permission of IBM.
Questions ?Questions ?
Thank you for attending! Please complete your session evaluationThank you for attending! Please complete your session evaluation, s, session number is WSI33ession number is WSI33..

WSI33 - Advanced Performance Tactics for IBM WebSphere Application Server

More Related Content

What's hot

Similar to WSI33 - Advanced Performance Tactics for IBM WebSphere Application Server

More from Hendrik van Run

Recently uploaded

WSI33 - Advanced Performance Tactics for IBM WebSphere Application Server