This document provides an overview of performance tuning for software systems. It discusses setting reasonable performance goals, understanding your system environment and usage patterns, identifying common performance issues, and taking a cyclical approach to ongoing monitoring and optimization. The key points are: set achievable performance goals based on peak demand, understand dependencies, data flows, and where load lies; monitor baseline performance and "red flags" like disk I/O, memory, and CPU usage; revisit performance regularly and after changes; and consider virtual environments, add-ons, OS patches, and hardware drivers.

1. ID114
Wrestling the Snake:
Performance Tuning 101
Wes Morgan | Senior Software Engineer
© 2012 IBM Corporation

2. Agenda
■ Why Are We Here?
■ Setting A Reasonable Goal
■ It's a Cycle, Folks
■ Understand Your Environment
■ A word about add on products
■ Take a Picture of “Everyday Life”
■ Common “Red Flags”
■ Thinking about SANs
■ Dealing with the Virtual World
■ A Quick Word About the Operating System
2 | © 2012 IBM Corporation

3. Why Are We Here?
■ Supply and Demand
■ “Throwing hardware at it” no longer an option
■ Stretching existing systems
■ Making upgrades tactical
■ Need numbers to justify actions
■ Know what to look for
3 | © 2012 IBM Corporation

4. Setting A Reasonable Goal
■ We aren't going to Unicorn-and-Rainbow Land
─ Even the best designs can be stressed
■ Changing demand
■ New initiatives, new software, new users
■ Most reasonable goal: Meet peak demand with (some) room to spare
4 | © 2012 IBM Corporation

5. It's a Cycle, Folks
■ You can't do this once and call it finished
■ As system load changes, we must adjust and adapt
─ Remember that load can change in volume, footprint, complexity or all of the above
■ Revisit performance on a regular basis
─ Know what numbers are important for your enterprise
─ Know who supports what in your organization
─ Automate reports (monthly, bi-monthly)
─ Catch trends before they become problems
■ Revisit performance on any significant change
─ New software releases
─ Changing OS (or OS version)
─ Adding/moving/removing virtual machines
─ Networking changes in your enterprise
5 | © 2012 IBM Corporation

6. Understand Your Environment!
■ Know your dependencies
─ Directories (Active Directory, LDAP)
─ Third-party products (Domino add-ins, Portal adjuncts)
─ Back-end components (WCM, DB2, etc.)
■ Know your data flow
─ What's handling your data, where, and when?
■ Know your networking
─ Are you traversing firewalls?
─ Must you use proxy servers?
■ Know where your load lies
─ What do clients do?
─ What do servers handle?
6 | © 2012 IBM Corporation

7. Understand Your “Everyday Life”
■ Gather baseline performance statistics
─ perfpmr (AIX), platform statistics (Domino), perfmon (Windows)
■ First Question: What does “routine” work look like?
■ Make sure that your baseline includes at least one “peak use” time
─ Monday morning, 9 am?
■ Use this baseline to understand where your system “pushes”
■ Identify potential “pressure points”
─ e.g. Peak use of particular facilities, but not yet in the “danger zone”
■ Track back to application-layer activities
─ System backups
─ Domino agents
─ Third-party scripts
7 | © 2012 IBM Corporation

8. Environment, Data Flow and Performance
Authentication
Application Storage
Third-Party
Firewall
Component
8 | © 2012 IBM Corporation

9. A word about add on products
■ What is an “add on” product?
─ Any IBM or third party product or component designed to link into a Domino server.
─ Can be a silent killer of resources
■ “Off the shelf” application
─ Know the impact before you begin
─ Know what the “best practices” are
■ API Tool Kit
─ Provide multiple ways to do the same task
─ With great flexibility comes great responsibility
9 | © 2012 IBM Corporation

10. A word about add on products – Blackberry
Enterprise Server®
■ Research In Motion's answer to mobile connectivity
■ Does not run on your mail server but can have a huge impact
■ NBES thread count can starve the hand held or overload the mail server
10 | © 2012 IBM Corporation

11. A word about add on products – Applications
■ IBM Lotus Enterprise Integrator (LEI) / Domino Enterprise Connection
Services (DECS)
─ Provide the interface to back-end database systems
─ Can drive a high disk and network load
■ Extension Manager / NSFHooks
─ Test, Test, Test
─ Use LotusScript when available
11 | © 2012 IBM Corporation

12. Common “Red Flags” - Disk
■ Disk I/O
─ Huge contributor to performance problems
─ Disk Queue Length
– # of pending requests (regardless of their size)
– Every OS vendor agrees – disk queues longer than 2.0 indicate a bottleneck
– Look for both peak queue length and average queue length
─ Disk utilization
– Usually expressed as a percentage
– Effects of high utilization are obvious
– High utilization can contribute to high queue lengths
12 | © 2012 IBM Corporation

13. Common “Red Flags” - Memory
■ Paging
─ High paging can indicate RAM starvation
─ Also increases disk activity – can send you 'over the edge' with disk latency
─ Set paging files to at least 1.5x RAM as baseline
■ Swapping
─ High swapping can indicate process thrashing
─ Can be triggered by high volume of short-term activity (e.g. scripts, agents)
■ NOTE: No real consensus on “bad numbers” for these
─ Examine these in comparison to your healthy “baseline”
─ Look for spikes or high plateaus
■ Virtual servers add to the complexity here!
13 | © 2012 IBM Corporation

14. Common “Red Flags” - CPU
■ Looking for repeated spikes or high plateaus
■ Easy for one process to effectively hang your system
■ May need to go to per-process views to find culprit
■ First cut – match against known actions (agents, scripts, backups)
14 | © 2012 IBM Corporation

15. Thinking About SANs
■ They aren't infinite
─ Know what shares your SAN (or SAN slice)
─ Know general SAN operations (e.g. caching, backups, etc.)
─ Remember that another system's I/O can affect your performance
■ They aren't magic
─ Applications/OS still see them as a single disk
─ Disk queueing and utilization concerns still apply
─ Demand can differ drastically among applications
15 | © 2012 IBM Corporation

16. Dealing With The Virtual World
■ Regardless of vendor, you're still sharing one set of physical resources
■ Still subject to “interference” from other virtual machines
■ Still need to provision virtual machines as fully as “real hardware”
■ Compare your performance to other virtual machines on the same host
■ Be careful with moving virtual machines
─ Pay attention to overall application architecture
─ Can lose fault-tolerance/failover
■ Remember, you now have TWO potential bottleneck points
─ Within your virtual machine
─ The virtual host itself
– Example: Output queue length on VM network interface
─ ASK ABOUT OVERCOMMITTING THE VIRTUAL HOST!!!
16 | © 2012 IBM Corporation

17. A Word About OS Patches
■ Be very, very careful
■ Some patches can introduce problems all their own
■ Use a test environment whenever possible
■ Keep abreast of new patches
■ Be sure to spot-check performance numbers after patch installation
17 | © 2012 IBM Corporation

18. What About BIOS and Drivers?
■ Keep system BIOS as close to “latest and greatest” as possible
─ Some vendors release new BIOS versions almost monthly
─ May address/optimize important subsystems
─ May resolve particular performance issues
■ Watch for new device drivers as dictated by your performance
─ If you're disk-heavy, keep an eye out for SAN/controller drivers
─ If you're network-heavy, watch for advanced network drivers
■ Partner with your hardware team!
18 | © 2012 IBM Corporation

19. In Conclusion...
■ No “one right answer” across systems or applications
■ Performance tuning driven by your users, your data, your network
■ It's a repeating cycle – get used to it
■ Partner with network/hardware teams for best effect
19 | © 2012 IBM Corporation

20. Questions
&
Answers
I'll be in the Meet the Developers Lab all week...
Don't forget the Session Evaluation...
THANKS FOR BEING HERE!
Twitter: @wesmorgan1
20 | © 2012 IBM Corporation

21. Legal disclaimer
© IBM Corporation 2012. All Rights Reserved.
The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained in this publication,
it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are subject to change by IBM without notice.
IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing contained in this publication is intended to, nor shall have
the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software.
References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced
in this presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any
way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other
results.
IBM, the IBM logo, Lotus, Lotus Notes, Notes, Domino, Quickr, Sametime, WebSphere, UC2, PartnerWorld and Lotusphere are trademarks of International Business Machines Corporation in the United
States, other countries, or both. Unyte is a trademark of WebDialogs, Inc., in the United States, other countries, or both.
21 | © 2012 IBM Corporation