IBM Power SystemsPerformance Case Studies       Examples of performance analysisDawn May - dmmay@us.ibm.com               ...
IBM Power Systems                        Viewing               Collection Services Data               General Analysis Rev...
IBM Power Systems                    •   In this example, we are not looking at a reported                        problem....
•   This chart is a basic system overview of CPUIBM Power Systems                        utilization and the more common w...
•   Using the selection tool, click on the CPU utilizationIBM Power Systems                         line at the low point....
•   Since we are wondering why the CPU has dropped,IBM Power Systems                        lets look at wait data.       ...
•   By using the fly-over, we can see that most of our waitIBM Power Systems                        time is due to disk pa...
IBM Power Systems   •   Using the selection tool, click on the large orange area                        that reflects Disk...
• Use the Tooltips tool to look at the pages faults for theIBM Power Systems                         QRWTSRVR/QUSER/436662...
•   One point of interest here is that we are looking at a couple of jobsIBM Power Systems                        that hav...
•   Yes, there are four threads spending nearly all theirIBM Power Systems                        time waiting on page fau...
•   Use the Zoom Region tool from the tool bar and make aIBM Power SystemsZoom Region Tool        rectangle over the very ...
•   This data was hidden from view due to the large timeIBM Power Systems Reset Zoom Tool    •   You can see each thread d...
•      What more can we find out? Collection Services does not know what data this server is       working with (you would...
•   This is the DDM/DRDA server job.IBM Power Systems                    •   Press Done.                    •   When back ...
•   Now we can see the user that this server job is doingIBM Power Systems                                           work ...
•   Once again, take the CPU Utilization and WaitsIBM Power Systems                        Overview on the same collection...
•   From this chart we can see that the blue bar isIBM Power Systems                        “Machine Level Gate Serializat...
IBM Power Systems   •   We have identified the QRWTSRVR jobs as the                        jobs with the machine level gat...
IBM Power Systems          This concludes the exercise using the Performance Data Investigator             with Collection...
IBM Power Systems                    Viewing Waits                         with                     Job Watcher Example of...
IBM Power Systems   Start with the IBM Systems Director Navigator for i                    Expand the “IBM i Management” t...
•   Open the Job Watcher content packageIBM Power Systems                    •   Select the “CPU Utilization and Waits Ove...
IBM Power Systems                    •   You will see the following chart                    •   Click on the “Full Zoom O...
IBM Power Systems                    •   Look for unusual patterns as a way to start the                        investigat...
By zooming in, we can see that “Operating System Contention Time”IBM Power Systems                    is a significant wai...
Using the selection tool (arrow), click on the first bar with significantIBM Power Systems                    Operating Sy...
• Machine Level Gate Serialization now shows up asIBM Power Systems                    a wait type                    • Us...
• Select an interval to investigate further by clicking onIBM Power Systems                    a bar as shown below       ...
• You will see the following chartIBM Power Systems                    • Use the zoom tool to get a closer view30         ...
• You will get the following chart when you zoom in.IBM Power Systems                    • Use the flyover tool to display...
• Select a thread to investigate further.IBM Power Systems                    • Drill into All Waits for One Thread or Tas...
• Select the interval to investigate further.IBM Power Systems                    • Drill into Interval Details for One Th...
Here you see this thread is waiting for the QAUDJRNIBM Power Systems   journal at 8:51:05.                    In the call ...
If the audit journal information was still available, youIBM Power Systems                                           could...
IBM Power Systems This exercise shows how a normal system function for going to a new journal   receiver affected the CPU ...
IBM Power SystemsIf you were to start this lab over, the graph which is displayed after doing the “full    zoom out” will ...
IBM Power Systems                    Viewing Waits                        with                    Job Watcher        Examp...
IBM Power Systems Job Watcher: CPU Utilization and Waits Overview                                Look at the run/wait     ...
IBM Power SystemsSeizes and Locks Waits Overview        All Waits by Thread or Task …                                    ...
IBM Power SystemsAll Waits by Thread or Task         All Waits for One Thread or Task      Select the job with the       ...
IBM Power SystemsAll Waits for One Thread or Task         Interval Details                               Display the inte...
IBM Power SystemsInterval Details                      The information about the object                        waited on a...
IBM Power SystemsSpecial notices This document was developed for IBM offerings in the United States as of the date of publ...
IBM Power SystemsSpecial notices (cont.)IBM, the IBM logo, ibm.com AIX, AIX (logo), AIX 5L, AIX 6 (logo), AS/400, BladeCen...
Upcoming SlideShare
Loading in …5
×

Performance case studies Common Europe june 2012

590 views

Published on

COMMON Europe Congress 2012 - Vienna

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
590
On SlideShare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
22
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Performance case studies Common Europe june 2012

  1. 1. IBM Power SystemsPerformance Case Studies Examples of performance analysisDawn May - dmmay@us.ibm.com © 2012 IBM Corporation
  2. 2. IBM Power Systems Viewing Collection Services Data General Analysis Review2 © 2012 IBM Corporation
  3. 3. IBM Power Systems • In this example, we are not looking at a reported problem. Rather, we are surveying Collection Services data to see if anything interesting shows up. • Open the Collection Services content package • Select “CPU Utilization and Waits Overview” • Select the COMMON2 collection library • Select Q071123119 for the collection • Display this3 © 2012 IBM Corporation
  4. 4. • This chart is a basic system overview of CPUIBM Power Systems utilization and the more common wait conditions. Tooltips • Use the Tooltips tool from the tool box. As you move this over various areas of the chart it will tell you what metric you are looking at and more explicit information on that point. • Note the drop in CPU consumption in interval 11. Also note operating system contention time just prior to that in interval 10 be related.4 © 2012 IBM Corporation
  5. 5. • Using the selection tool, click on the CPU utilizationIBM Power Systems line at the low point. Selection tool • Note that you can select multiple points or a range. When you drill down after selecting points or a range, that information is remembered and used as input to future charts. Selected point5 © 2012 IBM Corporation
  6. 6. • Since we are wondering why the CPU has dropped,IBM Power Systems lets look at wait data. • Go to the Select Action window where you will find a list of possible actions. • Select the Waits Overview chart.6 © 2012 IBM Corporation
  7. 7. • By using the fly-over, we can see that most of our waitIBM Power Systems time is due to disk page faulting. • There are several next steps that could be taken, and those steps may be based upon experience, knowledge of the workload running on the system, etc.7 © 2012 IBM Corporation
  8. 8. IBM Power Systems • Using the selection tool, click on the large orange area that reflects Disk Page Faults Time. • Select Waits by Job or Task.8 © 2012 IBM Corporation
  9. 9. • Use the Tooltips tool to look at the pages faults for theIBM Power Systems QRWTSRVR/QUSER/436662 jobs. • Note that by clicking on the Disk Page Faults Time on the prior chart, this chart is sorted by Disk Page Faults Time.9 © 2012 IBM Corporation
  10. 10. • One point of interest here is that we are looking at a couple of jobsIBM Power Systems that have thousands of seconds of wait time. How so? A guess is the server is multi-threaded. • Click on the first job with the selection tool and then select All Waits by Thread or Task from the Select Action window.10 © 2012 IBM Corporation
  11. 11. • Yes, there are four threads spending nearly all theirIBM Power Systems time waiting on page faults. But how do you know that for sure? • The interval size is 900 seconds. Most of that is spent waiting, not much left over for running. We can see that by looking at wait data.11 © 2012 IBM Corporation
  12. 12. • Use the Zoom Region tool from the tool bar and make aIBM Power SystemsZoom Region Tool rectangle over the very left hand part of the chart. • Do this a couple times to zoom into the leftmost data. • (If you have trouble with the zoom region, use the Reset Zoom tool to start over).12 © 2012 IBM Corporation
  13. 13. • This data was hidden from view due to the large timeIBM Power Systems Reset Zoom Tool • You can see each thread did run for almost a second. • You can press the reset zoom tool button to get back to where this chart started.13 © 2012 IBM Corporation
  14. 14. • What more can we find out? Collection Services does not know what data this server is working with (you would need to use Disk Watcher or Job Watcher for that). We can find out IBM Power Systems what this server is and who the client is.• Click on the arrow for the History and go back to Waits Overview.• From the Select Action window, select Waits by Server Type. 14 © 2012 IBM Corporation
  15. 15. • This is the DDM/DRDA server job.IBM Power Systems • Press Done. • When back at the Waits Overview chart, select the Waits by Job Current User Profile chart.15 © 2012 IBM Corporation
  16. 16. • Now we can see the user that this server job is doingIBM Power Systems work for is VCPANYLT. • From the History menu, select Home to go back to the main Investigate Data panel.16 © 2012 IBM Corporation
  17. 17. • Once again, take the CPU Utilization and WaitsIBM Power Systems Overview on the same collection we have been working with. • Select interval 10 we identified with Operating System Contention time earlier in this lab by clicking on the bar. • Drill down into Contention Waits Overview17 © 2012 IBM Corporation
  18. 18. • From this chart we can see that the blue bar isIBM Power Systems “Machine Level Gate Serialization Time” and the pink is “Semaphore Contention Time”. Semaphore contention is often a normal wait condition, so were not really interested in that. We are, however interested in what would be causing the machine level gate serialization. • Drill down into Waits by Job or Task to see if we can figure out what jobs are contributing to this contention.18 © 2012 IBM Corporation
  19. 19. IBM Power Systems • We have identified the QRWTSRVR jobs as the jobs with the machine level gate serialization. However, Collection Services cannot tell us more. To understand this machine level gate serialization would require Job Watcher data that has more information including holders and call stacks. • Press Done to return to the main Investigate Data panel. Or, you can use the History arrow to go “Home”.19 © 2012 IBM Corporation
  20. 20. IBM Power Systems This concludes the exercise using the Performance Data Investigator with Collection Services data. We have barely scratched the surface of the capabilities of the Performance Investigator and what information you can discover by looking at Collection Services data. We hope you found this interesting, useful, and realize that you do not need to be a performance expert to benefit from the performance data available to you when using the Performance Investigator.20 © 2012 IBM Corporation
  21. 21. IBM Power Systems Viewing Waits with Job Watcher Example of Machine Level Gate Serialization21 © 2012 IBM Corporation
  22. 22. IBM Power Systems Start with the IBM Systems Director Navigator for i Expand the “IBM i Management” treePerformance Tasks Select Performance category. Click on “Investigate Data”.22 © 2012 IBM Corporation
  23. 23. • Open the Job Watcher content packageIBM Power Systems • Select the “CPU Utilization and Waits Overview” • Select the COMMON collection library • Select DAWNJW2 for the collection • Display this (the next chart may take a minute to two to display) © 2012 IBM Corporation
  24. 24. IBM Power Systems • You will see the following chart • Click on the “Full Zoom Out” icon © 2012 IBM Corporation
  25. 25. IBM Power Systems • Look for unusual patterns as a way to start the investigation • Here, note the drop in CPU utilization just before 8:52, along with a corresponding increase in wait information. • Lets zoom into that timeframe using the zoom region tool. The zoom region will let you draw a box around the timeframe you are interested in.25 © 2012 IBM Corporation
  26. 26. By zooming in, we can see that “Operating System Contention Time”IBM Power Systems is a significant wait contributor during the time when CPU utilization dropped. Use the tooltips tool to see the information for the Operating System Contention time. You will also note that there are gaps in the graph between some of the stacked bars. With Job Watcher, it is possible for a collection interval to take longer to complete than the Job Watcher definition specifies. When these “long” collection intervals occur, they will show up as gaps in the graph.26 © 2012 IBM Corporation
  27. 27. Using the selection tool (arrow), click on the first bar with significantIBM Power Systems Operating System Contention. Also select the last bar with significant Operating System Contention. Drill into Contention Waits Overview once the two data points have been selected. By selecting specific data points in the graph, all future drill-downs will now be limited to the timeframe which has been selected. Using the zoom tool (as we did a few steps earlier) does NOT select data points and does not limit the scope of drill-downs.27 © 2012 IBM Corporation
  28. 28. • Machine Level Gate Serialization now shows up asIBM Power Systems a wait type • Use the flyover tool to display the wait information • Return to using the selection tool (arrow)28 © 2012 IBM Corporation
  29. 29. • Select an interval to investigate further by clicking onIBM Power Systems a bar as shown below • Drill down into All Waits by Thread or Task sorted by Machine Level Gate Serialization29 © 2012 IBM Corporation
  30. 30. • You will see the following chartIBM Power Systems • Use the zoom tool to get a closer view30 © 2012 IBM Corporation
  31. 31. • You will get the following chart when you zoom in.IBM Power Systems • Use the flyover tool to display information about the machine gate serialization waits. • Return to using the selection tool.31 © 2012 IBM Corporation
  32. 32. • Select a thread to investigate further.IBM Power Systems • Drill into All Waits for One Thread or Task32 © 2012 IBM Corporation
  33. 33. • Select the interval to investigate further.IBM Power Systems • Drill into Interval Details for One Thread or Task33 © 2012 IBM Corporation
  34. 34. Here you see this thread is waiting for the QAUDJRNIBM Power Systems journal at 8:51:05. In the call stack you will see an entry that shows the job is creating an audit journal entry. Note that access to the audit journal is serialized by a “gate”. So why is this job blocked and waiting to create the audit record?34 © 2012 IBM Corporation
  35. 35. If the audit journal information was still available, youIBM Power Systems could look at it. This screen capture shows the audit journal entries from the matching time period. Display Journal Entries ● NR is Next Receiver Journal . . . . . . : QAUDJRN ● PR is Previous . . . : Library . . . Receiver QSYS Largest sequence number on this screen . . . . . . : 00000000000088885894 Type options, press Enter. 5=Display entire entry Opt Sequence Code Type Object Library Job Time 88885883 T GS BEIJINGA 8:51:02 88885884 T SK QSYSARB 8:51:02 88885885 J NR QDBSRV02 8:51:02 88885886 J PR QDBSRV02 8:51:06 88885887 T GS BEIJINGA 8:51:07 88885888 T GS BEIJINGA 8:51:07 88885889 T GS BEIJINGA 8:51:07 88885890 T SK QSYSARB 8:51:07 88885891 T GS BEIJINGA 8:51:07 88885892 T GS BEIJINGA 8:51:07 88885893 T GS BEIJINGA 8:51:07 88885894 T GS BEIJINGA 8:51:07 More... F3=Exit F12=Cancel35 © 2012 IBM Corporation
  36. 36. IBM Power Systems This exercise shows how a normal system function for going to a new journal receiver affected the CPU utilization of the system for a short period of time.In this scenario, the next steps would be to evaluate what information is being captured in the security audit journal to ensure you are not auditing information you do not need.This exercise also shows how powerful the Job Watcher capabilities are for understanding the details of what is happening on the system.This is something only IBM i can do!36 © 2012 IBM Corporation
  37. 37. IBM Power SystemsIf you were to start this lab over, the graph which is displayed after doing the “full zoom out” will show other potentially interesting timeframes in the data.At about 9:04 and a bit after 9:26 there are additional spikes in operating system contention time.Throughout the graph there are several drops in CPU utilization.Feel free to examine this Job Watcher data further if time allows.With the Job Watcher data and the Performance Data Investigator, you can learn quite a bit about the performance of your IBM i.37 © 2012 IBM Corporation
  38. 38. IBM Power Systems Viewing Waits with Job Watcher Example of Object Lock Contention38 © 2012 IBM Corporation
  39. 39. IBM Power Systems Job Watcher: CPU Utilization and Waits Overview Look at the run/wait signature for the entire collection Drill down into the details for that wait bucket Look for the wait timethat appears to be the most pervasive throughout the collection. In this case, it Is Lock Contention Time 39 © 2012 IBM Corporation
  40. 40. IBM Power SystemsSeizes and Locks Waits Overview  All Waits by Thread or Task … Look at all the waits by thread or task for that wait type40 © 2012 IBM Corporation
  41. 41. IBM Power SystemsAll Waits by Thread or Task  All Waits for One Thread or Task Select the job with the object lock contention time. Look at all waits for that one thread or task41 © 2012 IBM Corporation
  42. 42. IBM Power SystemsAll Waits for One Thread or Task  Interval Details Display the interval details for that thread or task Select an interval where the wait is displayed by clicking on it42 © 2012 IBM Corporation
  43. 43. IBM Power SystemsInterval Details The information about the object waited on and who is holding the lock to that object can be found here. The call stack is below. The call stack can give an idea of where to look to find the root cause of the problem. Very powerful!!43 © 2012 IBM Corporation
  44. 44. IBM Power SystemsSpecial notices This document was developed for IBM offerings in the United States as of the date of publication. IBM may not make these offerings available in other countries, and the information is subject to change without notice. Consult your local IBM business contact for information on the IBM offerings available in your area. Information in this document concerning non-IBM products was obtained from the suppliers of these products or other public sources. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM may have patents or pending patent applications covering subject matter in this document. The furnishing of this document does not give you any license to these patents. Send license inquires, in writing, to IBM Director of Licensing, IBM Corporation, New Castle Drive, Armonk, NY 10504-1785 USA. All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. The information contained in this document has not been submitted to any formal IBM test and is provided "AS IS" with no warranties or guarantees either expressed or implied. All examples cited or described in this document are presented as illustrations of the manner in which some IBM products can be used and the results that may be achieved. Actual environmental costs and performance characteristics will vary depending on individual client configurations and conditions. IBM Global Financing offerings are provided through IBM Credit Corporation in the United States and other IBM subsidiaries and divisions worldwide to qualified commercial and government clients. Rates are based on a clients credit rating, financing terms, offering type, equipment type and options, and may vary by country. Other restrictions may apply. Rates and offerings are subject to change, extension or withdrawal without notice. IBM is not responsible for printing errors in this document that result in pricing or information inaccuracies. All prices shown are IBMs United States suggested list prices and are subject to change without notice; reseller prices may vary. IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply. Any performance data contained in this document was determined in a controlled environment. Actual results may vary significantly and are dependent on many factors including system hardware configuration and software design and configuration. Some measurements quoted in this document may have been made on development-level systems. There is no guarantee these measurements will be the same on generally- available systems. Some measurements quoted in this document may have been estimated through extrapolation. Users of this document should verify the applicable data for their specific environment. Revised September 26, 200644 © 2012 IBM Corporation
  45. 45. IBM Power SystemsSpecial notices (cont.)IBM, the IBM logo, ibm.com AIX, AIX (logo), AIX 5L, AIX 6 (logo), AS/400, BladeCenter, Blue Gene, ClusterProven, DB2, ESCON, i5/OS, i5/OS (logo), IBM BusinessPartner (logo), IntelliStation, LoadLeveler, Lotus, Lotus Notes, Notes, Operating System/400, OS/400, PartnerLink, PartnerWorld, PowerPC, pSeries, Rational, RISCSystem/6000, RS/6000, THINK, Tivoli, Tivoli (logo), Tivoli Management Environment, WebSphere, xSeries, z/OS, zSeries, Active Memory, Balanced Warehouse,CacheFlow, Cool Blue, IBM Systems Director VMControl, pureScale, TurboCore, Chiphopper, Cloudscape, DB2 Universal Database, DS4000, DS6000, DS8000,EnergyScale, Enterprise Workload Manager, General Parallel File System, , GPFS, HACMP, HACMP/6000, HASM, IBM Systems Director Active Energy Manager,iSeries, Micro-Partitioning, POWER, PowerExecutive, PowerVM, PowerVM (logo), PowerHA, Power Architecture, Power Everywhere, Power Family, POWER Hypervisor,Power Systems, Power Systems (logo), Power Systems Software, Power Systems Software (logo), POWER2, POWER3, POWER4, POWER4+, POWER5, POWER5+,POWER6, POWER6+, POWER7, System i, System p, System p5, System Storage, System z, TME 10, Workload Partitions Manager and X-Architecture are trademarksor registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms aremarked on their first occurrence in this information with a trademark symbol (® or ™), these symbols indicate U.S. registered or common law trademarks owned by IBM atthe time this information was published. Such trademarks may also be registered or common law trademarks in other countries.A full list of U.S. trademarks owned by IBM may be found at: http://www.ibm.com/legal/copytrade.shtml.Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/orother countries.AltiVec is a trademark of Freescale Semiconductor, Inc.AMD Opteron is a trademark of Advanced Micro Devices, Inc.InfiniBand, InfiniBand Trade Association and the InfiniBand design marks are trademarks and/or service marks of the InfiniBand Trade Association.Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registeredtrademarks of Intel Corporation or its subsidiaries in the United States and other countries.IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency which is now part of the Office of Government Commerce.Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates.Linear Tape-Open, LTO, the LTO Logo, Ultrium, and the Ultrium logo are trademarks of HP, IBM Corp. and Quantum in the U.S. and other countries.Linux is a registered trademark of Linus Torvalds in the United States, other countries or both.Microsoft, Windows and the Windows logo are registered trademarks of Microsoft Corporation in the United States, other countries or both.NetBench is a registered trademark of Ziff Davis Media in the United States, other countries or both.SPECint, SPECfp, SPECjbb, SPECweb, SPECjAppServer, SPEC OMP, SPECviewperf, SPECapc, SPEChpc, SPECjvm, SPECmail, SPECimap and SPECsfs aretrademarks of the Standard Performance Evaluation Corp (SPEC).The Power Architecture and Power.org wordmarks and the Power and Power.org logos and related marks are trademarks and service marks licensed by Power.org.TPC-C and TPC-H are trademarks of the Transaction Performance Processing Council (TPPC).UNIX is a registered trademark of The Open Group in the United States, other countries or both.Other company, product and service names may be trademarks or service marks of others. Revised December 2, 2010 45 © 2012 IBM Corporation

×