Performance case studies Common Europe june 2012

IBM Power Systems

Performance Case Studies
Examples of performance analysis

Dawn May - dmmay@us.ibm.com

© 2012 IBM Corporation

IBM Power Systems

Viewing
Collection Services Data

General Analysis Review

2 © 2012 IBM Corporation

IBM Power Systems

• In this example, we are not looking at a reported
problem. Rather, we are surveying Collection Services
data to see if anything interesting shows up.

• Open the Collection Services content package
• Select “CPU Utilization and Waits Overview”

• Select the COMMON2 collection library
• Select Q071123119 for the collection

• Display this


• This chart is a basic system overview of CPU
IBM Power Systems
utilization and the more common wait conditions.
Tooltips
• Use the Tooltips tool from the tool box. As you move
this over various areas of the chart it will tell you what
metric you are looking at and more explicit information
on that point.

• Note the drop in CPU consumption in interval 11.
Also note operating system contention time just prior
to that in interval 10 be related.


• Using the selection tool, click on the CPU utilization
IBM Power Systems
line at the low point.
Selection tool
• Note that you can select multiple points or a range.
When you drill down after selecting points or a range,
that information is remembered and used as input to
future charts.

Selected point


• Since we are wondering why the CPU has dropped,
IBM Power Systems
lets look at wait data.

• Go to the Select Action window where you will find a
list of possible actions.

• Select the Waits Overview chart.


• By using the fly-over, we can see that most of our wait
IBM Power Systems
time is due to disk page faulting.

• There are several next steps that could be taken, and
those steps may be based upon experience,
knowledge of the workload running on the system,
etc.


IBM Power Systems • Using the selection tool, click on the large orange area
that reflects Disk Page Faults Time.

• Select Waits by Job or Task.


• Use the Tooltips tool to look at the pages faults for the
IBM Power Systems
QRWTSRVR/QUSER/436662 jobs.

• Note that by clicking on the Disk Page Faults Time on the prior
chart, this chart is sorted by Disk Page Faults Time.


• One point of interest here is that we are looking at a couple of jobs
IBM Power Systems
that have thousands of seconds of wait time. How so? A guess is
the server is multi-threaded.

• Click on the first job with the selection tool and then select All Waits
by Thread or Task from the Select Action window.


• Yes, there are four threads spending nearly all their
IBM Power Systems
time waiting on page faults. But how do you know
that for sure?
• The interval size is 900 seconds. Most of that is spent
waiting, not much left over for running. We can see
that by looking at wait data.


• Use the Zoom Region tool from the tool bar and make a
IBM Power Systems
Zoom Region Tool rectangle over the very left hand part of the chart.

• Do this a couple times to zoom into the leftmost data.

• (If you have trouble with the zoom region, use the Reset
Zoom tool to start over).


• This data was hidden from view due to the large time
IBM Power Systems

Reset Zoom Tool • You can see each thread did run for almost a second.

• You can press the reset zoom tool button to get back
to where this chart started.


• What more can we find out? Collection Services does not know what data this server is
working with (you would need to use Disk Watcher or Job Watcher for that). We can find out
IBM Power Systems
what this server is and who the client is.

• Click on the arrow for the History and go back to
Waits Overview.

• From the Select Action window, select
Waits by Server Type.


• This is the DDM/DRDA server job.
IBM Power Systems
• Press Done.

• When back at the Waits Overview chart, select the
Waits by Job Current User Profile chart.


• Now we can see the user that this server job is doing
IBM Power Systems
work for is VCPANYLT.

• From the History menu, select Home to go
back to the main Investigate Data panel.


• Once again, take the CPU Utilization and Waits
IBM Power Systems
Overview on the same collection we have been
working with.

• Select interval 10 we identified with Operating System
Contention time earlier in this lab by clicking on the
bar.

• Drill down into Contention Waits Overview


• From this chart we can see that the blue bar is
IBM Power Systems
“Machine Level Gate Serialization Time” and the pink
is “Semaphore Contention Time”. Semaphore
contention is often a normal wait condition, so we're
not really interested in that. We are, however
interested in what would be causing the machine level
gate serialization.

• Drill down into Waits by Job or Task to see if we can
figure out what jobs are contributing to this contention.


IBM Power Systems • We have identified the QRWTSRVR jobs as the
jobs with the machine level gate serialization.
However, Collection Services cannot tell us
more. To understand this machine level gate
serialization would require Job Watcher data
that has more information including holders and
call stacks.

• Press Done to return to the main Investigate
Data panel. Or, you can use the History arrow
to go “Home”.


IBM Power Systems

This concludes the exercise using the Performance Data Investigator
with Collection Services data.

We have barely scratched the surface of the capabilities of the
Performance Investigator and what information you can discover
by looking at Collection Services data.

We hope you found this interesting, useful, and realize that you do
not need to be a performance expert to benefit from the
performance data available to you when using the Performance
Investigator.


IBM Power Systems

Viewing Waits
with
Job Watcher

Example of Machine Level Gate Serialization


IBM Power Systems Start with the IBM Systems Director Navigator for i
Expand the “IBM i Management” tree
Performance Tasks Select Performance category.
Click on “Investigate Data”.


• Open the Job Watcher content package
IBM Power Systems
• Select the “CPU Utilization and Waits Overview”

• Select the COMMON collection library
• Select DAWNJW2 for the collection

• Display this (the next chart may take a minute to two to display)


IBM Power Systems
• You will see the following chart

• Click on the “Full Zoom Out” icon


IBM Power Systems
• Look for unusual patterns as a way to start the
investigation

• Here, note the drop in CPU utilization just before 8:52,
along with a corresponding increase in wait
information.

• Let's zoom into that timeframe using the zoom region
tool. The zoom region will let you draw a box around
the timeframe you are interested in.


By zooming in, we can see that “Operating System Contention Time”
IBM Power Systems
is a significant wait contributor during the time when CPU utilization
dropped.

Use the tooltips tool to see the information for the Operating System
Contention time.

You will also note that there are gaps in the graph between some of
the stacked bars. With Job Watcher, it is possible for a collection
interval to take longer to complete than the Job Watcher definition
specifies. When these “long” collection intervals occur, they will show
up as gaps in the graph.


Using the selection tool (arrow), click on the first bar with significant
IBM Power Systems
Operating System Contention. Also select the last bar with significant
Operating System Contention.

Drill into Contention Waits Overview once the two data points have
been selected.

By selecting specific data points in the graph, all future drill-downs will
now be limited to the timeframe which has been selected.

Using the zoom tool (as we did a few steps earlier) does NOT select
data points and does not limit the scope of drill-downs.


• Machine Level Gate Serialization now shows up as
IBM Power Systems
a wait type
• Use the flyover tool to display the wait information
• Return to using the selection tool (arrow)


• Select an interval to investigate further by clicking on
IBM Power Systems
a bar as shown below

• Drill down into All Waits by Thread or Task sorted by
Machine Level Gate Serialization


• You will see the following chart
IBM Power Systems

• Use the zoom tool to get a closer view


• You will get the following chart when you zoom in.
IBM Power Systems

• Use the flyover tool to display information about the
machine gate serialization waits.

• Return to using the selection tool.


• Select a thread to investigate further.
IBM Power Systems

• Drill into All Waits for One Thread or Task


• Select the interval to investigate further.
IBM Power Systems

• Drill into Interval Details for One Thread or Task


Here you see this thread is waiting for the QAUDJRN
IBM Power Systems journal at 8:51:05.

In the call stack you will see an entry that shows the
job is creating an audit journal entry.

Note that access to the audit journal is serialized by a
“gate”. So why is this job blocked and waiting to
create the audit record?


If the audit journal information was still available, you
IBM Power Systems
could look at it.

This screen capture shows the audit journal entries from
the matching time period.
Display Journal Entries
● NR is Next Receiver
Journal . . . . . . : QAUDJRN ● PR is Previous . . . :
Library . . . Receiver
QSYS
Largest sequence number on this screen . . . . . . : 00000000000088885894
Type options, press Enter.
5=Display entire entry

Opt Sequence Code Type Object Library Job Time
88885883 T GS BEIJINGA 8:51:02
88885884 T SK QSYSARB 8:51:02
88885885 J NR QDBSRV02 8:51:02
88885886 J PR QDBSRV02 8:51:06
88885887 T GS BEIJINGA 8:51:07
88885888 T GS BEIJINGA 8:51:07
88885889 T GS BEIJINGA 8:51:07
88885890 T SK QSYSARB 8:51:07
88885891 T GS BEIJINGA 8:51:07
88885892 T GS BEIJINGA 8:51:07
88885893 T GS BEIJINGA 8:51:07
88885894 T GS BEIJINGA 8:51:07
More...
F3=Exit F12=Cancel


IBM Power Systems

This exercise shows how a normal system function for going to a new journal
receiver affected the CPU utilization of the system for a short period of time.

In this scenario, the next steps would be to evaluate what information is being
captured in the security audit journal to ensure you are not auditing
information you do not need.

This exercise also shows how powerful the Job Watcher capabilities are for
understanding the details of what is happening on the system.

This is something only IBM i can do!


IBM Power Systems

If you were to start this lab over, the graph which is displayed after doing the “full
zoom out” will show other potentially interesting timeframes in the data.

At about 9:04 and a bit after 9:26 there are additional spikes in operating system
contention time.

Throughout the graph there are several drops in CPU utilization.

Feel free to examine this Job Watcher data further if time allows.

With the Job Watcher data and the Performance Data Investigator, you can learn
quite a bit about the performance of your IBM i.


IBM Power Systems

Viewing Waits
with

Job Watcher

Example of Object Lock Contention


IBM Power Systems

Job Watcher: CPU Utilization and Waits Overview

Look at the run/wait
signature for the
entire collection

Drill down into the
details for that wait
bucket

Look for the wait time
that appears to be the most
pervasive throughout the
collection. In this case, it
Is Lock Contention Time

IBM Power Systems

Seizes and Locks Waits Overview 
All Waits by Thread or Task …
Look at all the waits
by thread or task
for that wait type


IBM Power Systems

All Waits by Thread or Task 
All Waits for One Thread or Task Select the job with the
object lock contention time.
Look at all waits
for that one thread or
task


IBM Power Systems

All Waits for One Thread or Task 
Interval Details
Display the interval
details for that
thread or task

Select an interval
where the wait is
displayed by
clicking on it


IBM Power Systems

Interval Details
The information about the object
waited on and who is holding
the lock to that object can be found
here. The call stack is below.
The call stack can give an idea
of where to look to
find the root cause of the problem.
Very powerful!!


IBM Power Systems

Special notices
This document was developed for IBM offerings in the United States as of the date of publication. IBM may not make these offerings available in
other countries, and the information is subject to change without notice. Consult your local IBM business contact for information on the IBM
offerings available in your area.
Information in this document concerning non-IBM products was obtained from the suppliers of these products or other public sources. Questions
on the capabilities of non-IBM products should be addressed to the suppliers of those products.
IBM may have patents or pending patent applications covering subject matter in this document. The furnishing of this document does not give
you any license to these patents. Send license inquires, in writing, to IBM Director of Licensing, IBM Corporation, New Castle Drive, Armonk, NY
10504-1785 USA.
All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives
only.
The information contained in this document has not been submitted to any formal IBM test and is provided "AS IS" with no warranties or
guarantees either expressed or implied.
All examples cited or described in this document are presented as illustrations of the manner in which some IBM products can be used and the
results that may be achieved. Actual environmental costs and performance characteristics will vary depending on individual client configurations
and conditions.
IBM Global Financing offerings are provided through IBM Credit Corporation in the United States and other IBM subsidiaries and divisions
worldwide to qualified commercial and government clients. Rates are based on a client's credit rating, financing terms, offering type, equipment
type and options, and may vary by country. Other restrictions may apply. Rates and offerings are subject to change, extension or withdrawal
without notice.
IBM is not responsible for printing errors in this document that result in pricing or information inaccuracies.
All prices shown are IBM's United States suggested list prices and are subject to change without notice; reseller prices may vary.
IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply.
Any performance data contained in this document was determined in a controlled environment. Actual results may vary significantly and are
dependent on many factors including system hardware configuration and software design and configuration. Some measurements quoted in this
document may have been made on development-level systems. There is no guarantee these measurements will be the same on generally-
available systems. Some measurements quoted in this document may have been estimated through extrapolation. Users of this document
should verify the applicable data for their specific environment.

Revised September 26, 2006


IBM Power Systems

Special notices (cont.)
IBM, the IBM logo, ibm.com AIX, AIX (logo), AIX 5L, AIX 6 (logo), AS/400, BladeCenter, Blue Gene, ClusterProven, DB2, ESCON, i5/OS, i5/OS (logo), IBM Business
Partner (logo), IntelliStation, LoadLeveler, Lotus, Lotus Notes, Notes, Operating System/400, OS/400, PartnerLink, PartnerWorld, PowerPC, pSeries, Rational, RISC
System/6000, RS/6000, THINK, Tivoli, Tivoli (logo), Tivoli Management Environment, WebSphere, xSeries, z/OS, zSeries, Active Memory, Balanced Warehouse,
CacheFlow, Cool Blue, IBM Systems Director VMControl, pureScale, TurboCore, Chiphopper, Cloudscape, DB2 Universal Database, DS4000, DS6000, DS8000,
EnergyScale, Enterprise Workload Manager, General Parallel File System, , GPFS, HACMP, HACMP/6000, HASM, IBM Systems Director Active Energy Manager,
iSeries, Micro-Partitioning, POWER, PowerExecutive, PowerVM, PowerVM (logo), PowerHA, Power Architecture, Power Everywhere, Power Family, POWER Hypervisor,
Power Systems, Power Systems (logo), Power Systems Software, Power Systems Software (logo), POWER2, POWER3, POWER4, POWER4+, POWER5, POWER5+,
POWER6, POWER6+, POWER7, System i, System p, System p5, System Storage, System z, TME 10, Workload Partitions Manager and X-Architecture are trademarks
or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are
marked on their first occurrence in this information with a trademark symbol (® or ™), these symbols indicate U.S. registered or common law trademarks owned by IBM at
the time this information was published. Such trademarks may also be registered or common law trademarks in other countries.

A full list of U.S. trademarks owned by IBM may be found at: http://www.ibm.com/legal/copytrade.shtml.

Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or
other countries.
AltiVec is a trademark of Freescale Semiconductor, Inc.
AMD Opteron is a trademark of Advanced Micro Devices, Inc.
InfiniBand, InfiniBand Trade Association and the InfiniBand design marks are trademarks and/or service marks of the InfiniBand Trade Association.
Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered
trademarks of Intel Corporation or its subsidiaries in the United States and other countries.
IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency which is now part of the Office of Government Commerce.
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates.
Linear Tape-Open, LTO, the LTO Logo, Ultrium, and the Ultrium logo are trademarks of HP, IBM Corp. and Quantum in the U.S. and other countries.
Linux is a registered trademark of Linus Torvalds in the United States, other countries or both.
Microsoft, Windows and the Windows logo are registered trademarks of Microsoft Corporation in the United States, other countries or both.
NetBench is a registered trademark of Ziff Davis Media in the United States, other countries or both.
SPECint, SPECfp, SPECjbb, SPECweb, SPECjAppServer, SPEC OMP, SPECviewperf, SPECapc, SPEChpc, SPECjvm, SPECmail, SPECimap and SPECsfs are
trademarks of the Standard Performance Evaluation Corp (SPEC).
The Power Architecture and Power.org wordmarks and the Power and Power.org logos and related marks are trademarks and service marks licensed by Power.org.
TPC-C and TPC-H are trademarks of the Transaction Performance Processing Council (TPPC).
UNIX is a registered trademark of The Open Group in the United States, other countries or both.

Other company, product and service names may be trademarks or service marks of others.

Revised December 2, 2010


Performance case studies Common Europe june 2012

More Related Content

Similar to Performance case studies Common Europe june 2012

More from COMMON Europe

Recently uploaded

Performance case studies Common Europe june 2012