Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

IBM i Job wait accounting job watcher june 2012


Published on

COMMON Europe Congress 2012 - Vienna

Published in: Technology, Business
  • Be the first to comment

IBM i Job wait accounting job watcher june 2012

  1. 1. IBM Power SystemsIBM i Job Watcher and Wait Accounting Improve your Job Performance with Run-Wait AnalysisDawn May - © 2012 IBM Corporation
  2. 2. IBM Power SystemsAgenda What is Wait Accounting? Why is Wait Accounting important? How to Use Wait Accounting – General information – Collection Services – Job Watcher2 © 2012 IBM Corporation
  3. 3. IBM Power SystemsWhat is Wait Accounting? Wait Accounting is the ability to determine what a job is doing when it is not running – When a job is not running, it is waiting • But what is it waiting for? – Waits may be normal, some waits are not normal • Job Wait Accounting helps to determine if a wait condition is a problem IBM i has instrumented most of the wait conditions – Wait information is collected by Collection Services and Job Watcher i Exclusive!3 © 2012 IBM Corporation
  4. 4. IBM Power SystemsWhat is Wait Accounting?  Wait Accounting is used to understand what is happening when a job is waiting – Wait information is tracked for each job, thread and task on system  A job spends its time in one of three states – CPU • Time spent dispatched to the processor, active/running – CPU queuing • Ready to be processed, waiting for a processor to become available – Wait • Waiting for something or someone, blocked or idle © 2012 IBM Corporation
  5. 5. IBM Power SystemsBasics of Waiting Two basic types of waits – Idle: waiting for a work request • Typically not indicative of a problem  E.g., waiting for the “Enter” key to be pressed on a 5250 display session • If a problem, usually external to the machine  E.g., slow arrival of work requests due to communications problem • Possible, but not typical in batch jobs  E.g., waiting for an entry to be placed on a data queue – Blocked: waits that occur while performing a work request Blocked waits are the ones we want to take a closer look at ● “Outside of CPU usage and CPU queue time, blocked waits are the reason jobs/threads take as long as they do to complete their work”5 © 2012 IBM Corporation
  6. 6. IBM Power SystemsRun/Wait Signature Typical batch job run/wait signature CPU CPU queue Wait Elapsed time Interactive job run/wait signature Idle CPU CPU queue Wait Elapsed time6 © 2012 IBM Corporation
  7. 7. IBM Power SystemsWhy is Wait Accounting important? Wait time is easy to compute, but rarely scrutinized For batch-type work: Waits = Elapsed time – CPU time For interactive-type work: Waits = Elapsed time – CPU time – Idle time▪ Historically, little instrumentation has existed to measure and provide detail on waits ● i Wait Accounting provides the instrumentation to understand and analyze waits7 © 2012 IBM Corporation
  8. 8. IBM Power SystemsBasic example Batch job with total run time of 6 hours Run/wait signature CPU CPU queue Wait 120 min 70 min 170 min Elapsed time 6 hours (360 mins) ● Potential to run in 3 hrs 10 min if wait time could be eliminated ▪ Wait analysis and reduction can be a powerful and cost- effective way to improve response time and throughput8 © 2012 IBM Corporation
  9. 9. IBM Power SystemsDetailing wait time Determine the components of time spent waiting Elapsed time CPU CPU queue Wait Record Disk reads Disk writes Journal locks9 © 2012 IBM Corporation
  10. 10. IBM Power SystemsDetailing wait time - metrics Then get metrics related to these components of wait time Total Disk reads Disk writes Record Journal count Locks 3,523 17,772 355 5,741 Total time 42 sec 73 sec 45 sec 44 sec 0.012 sec 0.004 sec 0.126 sec 0.007 sec Avg time per wait▪ Even at this level, we can already see possible questions to ask: ● How many of the reads are page faults? Could memory/pool changes help? ● What programs are causing reads? Could they be reduced or made async? ● What programs are causing writes? Could they be reduced or made async? ● What DB2 files are involved with the record locks? ● What files are being journaled? Are journals needed and optimally configured?10 © 2012 IBM Corporation
  11. 11. IBM Power SystemsExpected vs. Unexpected Waits Some waits are “expected” and others “unexpected” – In most cases, “normal” and “abnormal” are more accurate descriptors • Example: a record lock may be normal, but one that lasts for a very long duration is abnormal – Regardless of the type of wait, it is always better if wait time can be minimized or eliminated – There are a few block points on the system that are almost always considered abnormal11 © 2012 IBM Corporation
  12. 12. IBM Power SystemsHolders versus Waiters IBM i keeps track of who is holding a resource, and if applicable, who is waiting to access that resource – A Holder is the job/thread/task that is holding the serialized resource – A Waiter is the job/thread/task that wants to access the serialized resource IBM i also maintains call stacks for every job/thread/task The combination of wait buckets, information regarding holders/waiters, and call stacks can provide a very powerful solution for analyzing wait conditions Job Watcher is the tool to accomplish this – Collection services can tell you about waits, but not about holders/waiter nor call stacks12 © 2012 IBM Corporation
  13. 13. IBM Power SystemsWait Accounting - “Wait Buckets” Licensed Internal Code (LIC) has identified points where waits occur and assigned a numeric identifier to each point, also known as a “block point” This level of detail poses challenges to efficiently evaluate the wait time in a job – Difficult to fully understand hundreds of block points – Huge amounts of storage would be needed to track counts and times for every block point in every job For this reason, block points are assigned to categories commonly referred to as “wait buckets” “Mapping” refers to how block points are assigned to buckets13 © 2012 IBM Corporation
  14. 14. IBM Power Systems32 Wait Buckets as Defined at 6.1 and 7.1 – Time dispatched on a CPU – Object lock contention – CPU queuing – Ineligible waits – Reserved – Main storage pool overcommitment – Other waits – Classic Java user including locks – Disk page faults – Classic Java JVM – Disk non fault reads – Classic Java other – Disk space usage contention – Socket accepts (idle) – Disk op-start contention – Socket transmits – Disk writes – Socket receives – Disk other – Socket other – Journaling – IFS – Semaphore contention – PASE – Mutex contention – Data queue receives – Machine level gate serialization – Idle / waiting for work – Seize contention – Synchronization Token contention – Database record lock contention – Abnormal contention14 © 2012 IBM Corporation
  15. 15. IBM Power SystemsWait Bucket Example Database Record Lock Contention – Bucket 16 – Several different causes for waits in this bucket • Read • Update • Weak • Transfer • Check • Conflict Exit15 © 2012 IBM Corporation
  16. 16. IBM Power Systems Collection Services As of 6.1 Collection Services collects all 32 wait buckets When Create Performance Data (CRTPFRDTA) runs, data is written to files – QAPMJOBWT - wait bucket data file • Information that accounts for the time a job, task, or thread spends waiting ● Each job, task, or thread that consumed CPU during the interval • Counts and clock times for each bucket • Current wait: time waiting and bucket to be used when wait is satisfied – QAPMJOBWTD - wait bucket description file • Descriptions of the counter sets found in the above file – in unicode – QAPMJOBWTG – wait conditions not available in the QAPMJOBWT file • One record for each job, task, or thread that did not consume CPU during the interval Note: The Information Center also describes ‘buckets’ as ‘groups’ or ‘sets’ 16 © 2012 IBM Corporation
  17. 17. IBM Power SystemsUnderstanding “Time Dispatched on a CPU”Time dispatched on a CPU (Bucket 1) – Thread or task has been assigned to a processor and is NOT waiting – Complicated by certain features • Hardware Multi Threading (HMT) ● Allows multiple threads/tasks to be assigned to a single physical processor ● Causes bucket 1 time to be greater than actual CPU time • Background assisting tasks ● Promote their CPU usage back into the client job/thread ● Causes client thread’s bucket 1 time to be smaller than measured CPU time • LPAR shared/partial processors ● Bucket 1 records time dispatched to the virtual processor ● Bucket 1 time may be greater than CPU time because it may include time the thread/task is waiting for the physical processor behind the virtual processor Bucket 1 does NOT equal CPU Time17 © 2012 IBM Corporation
  18. 18. IBM Power SystemsUnderstanding “CPU Queuing”CPU Queuing (Wait Bucket 2) – Thread or task has been assigned to a processor and is waiting for the CPU to become available • Too much work on the partition causing threads to need to wait for the processors • Spiky workloads • Shared processors Latency due to hypervisor sharing the physical processors among multiple partitions18 © 2012 IBM Corporation
  19. 19. IBM Power SystemsWaits that Applications may use Disk waits Semaphores, Mutexes, Synchronization Tokens Journaling Database record locks Object locks Sockets19 © 2012 IBM Corporation
  20. 20. IBM Power SystemsCollection Services vs Job Watcher Collection Services and Job Watcher both collect wait information – Both provide graphs that show waits – Collection Services runs by default in 6.1 and later – Job Watcher data is generally collected when additional information is necessary to analyze a problem Job Watcher graphs have better drill-down capabilities – Display graphs that have the data sorted by wait types – Collection Services can accomplish this via table views Job Watcher can also collect call stacks and SQL statements – Provides additional information for detailed analysis20 © 2012 IBM Corporation
  21. 21. IBM Power SystemsIBM Systems Director Navigator for i Job Watcher Job Watcher GUI is included with IBM i Performance Tools, Job Watcher Feature From GO LICPGM: – 5761PT1 *COMPATIBLE IBM Performance Tools for i5/OS – 5761PT1 *COMPATIBLE Performance Tools - Manager Feature – 5761PT1 *COMPATIBLE Performance Tools - Agent Feature  5761PT1 *COMPATIBLE Performance Tools - Job Watcher21 © 2012 IBM Corporation
  22. 22. IBM Power SystemsIBM Systems Director Navigator for i IBM Systems Director Navigator for i is the Web console for managing IBM i – Much of the function that exists in System i Navigator, but with a browser user interface22 © 2012 IBM Corporation
  23. 23. IBM Power SystemsPrerequisites Users need to be authorized to use the Investigate Data and Collections performance tasks Include users on the QPMCCDATA authorization list Edit Authorization List Object . . . . . . . : QPMCCDATA Owner . . . . . . . : QSYS Library . . . . . : QSYS Primary group . . . : *NONE Type changes to current authorities, press Enter. Object List User Authority Mgt *PUBLIC *EXCLUDE QSYS *ALL X PDI01 *USE PDI02 *USE PDI03 *USE PDI04 *USE PDI05 *USE PDI06 *USE PDI07 *USE PDI08 *USE PDI09 *USE More...23 © 2012 IBM Corporation
  24. 24. IBM Power SystemsPrerequisites, continued To see the Job Watcher definitions through the GUI, users need authority to the QAPYJWDFN file in QUSRSYS – File is shipped with *PUBLIC *EXCLUDE24 © 2012 IBM Corporation
  25. 25. IBM Power SystemsHow do I run Job Watcher?25 © 2012 IBM Corporation
  26. 26. IBM Power SystemsIBM-Supplied Job Watcher Definitions Several pre-defined Job Watcher definitions are available – The main difference is the sample intervals • Q1SEC* – 1 second intervals with call stacks, another to include SQL • Q10SEC* –10 second intervals with call stacks, another to include SQL • Q5SEC* – 5 second intervals with call stacks, another to include SQL26 © 2012 IBM Corporation
  27. 27. IBM Power SystemsAdding a Job Watcher Definition27 © 2012 IBM Corporation
  28. 28. IBM Power SystemsStarting Job Watcher28 © 2012 IBM Corporation
  29. 29. IBM Power SystemsHow Do I Run Job Watcher with the Commands? CL Commands also available starting with 6.1 – Add Job Watcher Definition (ADDJWDFN) to define the collection • Identifies the performance data that is to be collected – Remove Job Watcher Definition (RMVJWDFN) to remove a definition (Note: Job Watcher Definitions can only be displayed through the GUI) – Start Job Watcher (STRJW) to start the collection – End Job Watcher (ENDJW) to end the collection (optionally) Prior to 6.1, the iDoctor product is required to collect Job Watcher data29 © 2012 IBM Corporation
  30. 30. IBM Power SystemsJob Watcher Definition Parameters Collection interval – Interval between retrieval of job/task data – Considerations for frequency of sampling and amount of data collected Additional data categories – Additional types of data to include in the collection • For example, call stacks, Java, sockets and TCP data – Allows granularity on the data collected and frequency Wait-based call stack data – A special case where call stacks can be collected for certain jobs and/or tasks which have been in an abnormal or conflict wait Job name – The jobs that will be included in the collection Task name – The tasks that will be included in the collection TDE number – The Task Dispatching Elements (TDE) that will be included in the collection Current user profile – The current user profile of the jobs which will be included in the collection30 © 2012 IBM Corporation
  31. 31. IBM Power SystemsJob Watcher Definition Parameters Subsystem – The subsystem of the jobs which will be included in the collection Current storage pool – The current system pool of the jobs and/or tasks which will be included in the collection Force record write – When data records are written to the database files – At the end of the interval is necessary for real time analysis Include inactive jobs/tasks – Whether detailed data for inactive jobs and tasks will be collected during the first interval To file ASP threshold – Refers to the ASP where the Job Watcher database files are located – The maximum percent of this ASP that can be consumed System ASP threshold  The maximum percent of the system ASP that can be consumed31 © 2012 IBM Corporation
  32. 32. IBM Power SystemsJob Watcher Definition - Conditional parameters Provides a way to limit the size of the data collection – Can specify parameters to conditionally collect data • Average, Percentage, Rate, Direct comparisons – Can specify the type of conditional collection • Per interval checks in every interval ● Data will be collected for intervals in which the condition was satisfied • Trigger checks until the condition is satisfied ● Once the condition is met, data will unconditionally be collected • Until Met collects data unconditionally until the condition is met ● Once the condition is met, the collection ends – Timeout option • How long the collection should run without the condition being met – Consecutive occurrence count – History size – User exit program • Program that will be called at the time the condition is satisfied32 © 2012 IBM Corporation
  33. 33. IBM Power SystemsStart Job Watcher Parameters Definition – Name of the definition that identifies the collection parameters Collection – The name you want the collection to be known by Library – Library where the database files will be created Text description – Description associated with the database file members Replace data – Whether to overwrite existing database file members with the specified name Collection Interval – Can override what is specified on the definition End collection – Determines when data collection will end – Multiple end options may be specified • Elapsed time, storage consumed, number of intervals ● Whichever occurs first will end the collection Job name, Task name, TDE number – Can override what is specified on the definition33 © 2012 IBM Corporation
  34. 34. IBM Power SystemsManaging Job Watcher Collections34 © 2012 IBM Corporation
  35. 35. IBM Power SystemsManaging Job Watcher Collections …35 © 2012 IBM Corporation
  36. 36. IBM Power SystemsHow do I analyze Job Watcher data? Look for trends in the data Look for presence of waits – Drill down into wait details Display call stacks for running or waiting jobs36 © 2012 IBM Corporation
  37. 37. IBM Power SystemsJob Watcher Content Package37 © 2012 IBM Corporation
  38. 38. IBM Power SystemsCPU Utilization and Waits Overview  Both Collection Services and Job Watcher have a “CPU Utilization and Waits Overview” graph as a general starting point for wait analysis CPU Utilization and Waits Overview38 © 2012 IBM Corporation
  39. 39. IBM Power SystemsIBM Systems Lab Services and Training Mainframe Systems Our Mission and Profile  Support the IBM Systems Agenda and accelerate the adoption of new Power Systems products and solutions  Maximize performance of our clients’ existing IBM systems  Deliver technical training, conferences, and other services tailored to System x & Bladecenter meet client needs  Team with IBM Service Providers to optimize the deployment of IBM solutions (GTS, GBS, SWG Lab Services and our IBM Business System Storage Partners) Our Competitive Advantage IT Infrastructure Optimization  Leverage relationships with the IBM development labs to build deep technical skills and exploit the expertise of our developers  Combined expertise of Lab Services and the Training for Systems team Data Center Services  Skills can be deployed worldwide to assure all client needs can be met Successful worldwide history: Training Services 17 years in Americas, 9 years in Europe/Middle East/Africa, 5 years in Asia Pacific © 2012 IBM Corporation
  40. 40. IBM Power Systems IBM Systems Lab Services and Training Power Services Key Offerings Americas, WW Contacts  High Availability Services on Power Systems (including Advanced Copy Services for PowerHA™ on IBM i) Mark Even, 507-253-1313  Systems Director Services IBM i  PowerCare Services Frank Kriss  Performance and Scalability services (including system,, 507-253-1354 application, and database tuning) IBM i, High Availability  Virtualization Services for AIX® on Power Systems™ Karen Anderson  Application and database modernization consulting (SOA, 972-561-6337 implementation) IBM i Vouchers  Linux® on Power consulting, custom application Stephen Brandenburg development, implementation, and optimization services, 301-803-6199 PowerVouchers, Virtualization Program, AIX  Security on Power consulting and implementation services SWOT/SWAT, AIX  System consolidation and migration service  Allen Johnston High Performance Computing consulting and, 704-340-9165 implementation services PowerCare  SAP® on IBM i consulting Cindy Mestad  Power Blades on BladeCenter (including VIOS on i and, 507-253-6143 blades running IBM i implementation) Power Performance and Scalability Center  Smart Analytics services (including DB2® Web Query implementation and consulting)  Public, private, customized and self-paced virtual training  Power Systems Technical © 2012 IBM Corporation
  41. 41. IBM Power SystemsPerformance and Scalability Services The IBM i Performance and Scalability Services Center can provide facilities and hardware IN ROCHESTER to assist you in testing hardware or software changes  “Traditional” benchmarks  Release-to-release upgrades  Assess application performance when migrating to a new release of IBM I  Stress test your system  Determine impact of application changes  Proofs of Concept (e.g. HA alternatives; SSD analysis, external storage, etc.)  Evaluate application scalability  Capacity planning … all with the availability of Lab Services IBM i experts and development personnel To request any of these services, submit at: • © 2012 IBM Corporation
  42. 42. IBM Power Systems Back Up and Reference Information42 © 2012 IBM Corporation
  43. 43. IBM Power SystemsReference Information for Job Watcher iDoctor versus IBM i 6.1 Performance Tools – “Web Power” – Introduction to Job Watcher Green Screen Commands – End-to-End Performance Management on IBM i – a Redbooks publication – i Can … Tell You Why You Are Waiting – © 2012 IBM Corporation
  44. 44. IBM Power SystemsSpecial notices This document was developed for IBM offerings in the United States as of the date of publication. IBM may not make these offerings available in other countries, and the information is subject to change without notice. Consult your local IBM business contact for information on the IBM offerings available in your area. Information in this document concerning non-IBM products was obtained from the suppliers of these products or other public sources. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM may have patents or pending patent applications covering subject matter in this document. The furnishing of this document does not give you any license to these patents. Send license inquires, in writing, to IBM Director of Licensing, IBM Corporation, New Castle Drive, Armonk, NY 10504-1785 USA. All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. The information contained in this document has not been submitted to any formal IBM test and is provided "AS IS" with no warranties or guarantees either expressed or implied. All examples cited or described in this document are presented as illustrations of the manner in which some IBM products can be used and the results that may be achieved. Actual environmental costs and performance characteristics will vary depending on individual client configurations and conditions. IBM Global Financing offerings are provided through IBM Credit Corporation in the United States and other IBM subsidiaries and divisions worldwide to qualified commercial and government clients. Rates are based on a clients credit rating, financing terms, offering type, equipment type and options, and may vary by country. Other restrictions may apply. Rates and offerings are subject to change, extension or withdrawal without notice. IBM is not responsible for printing errors in this document that result in pricing or information inaccuracies. All prices shown are IBMs United States suggested list prices and are subject to change without notice; reseller prices may vary. IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply. Any performance data contained in this document was determined in a controlled environment. Actual results may vary significantly and are dependent on many factors including system hardware configuration and software design and configuration. Some measurements quoted in this document may have been made on development-level systems. There is no guarantee these measurements will be the same on generally- available systems. Some measurements quoted in this document may have been estimated through extrapolation. Users of this document should verify the applicable data for their specific environment. Revised September 26, 200644 © 2012 IBM Corporation
  45. 45. IBM Power Systems Special notices (cont.)IBM, the IBM logo, AIX, AIX (logo), AIX 6 (logo), AS/400, Active Memory, BladeCenter, Blue Gene, CacheFlow, ClusterProven, DB2, ESCON, i5/OS, i5/OS(logo), IBM Business Partner (logo), IntelliStation, LoadLeveler, Lotus, Lotus Notes, Notes, Operating System/400, OS/400, PartnerLink, PartnerWorld, PowerPC, pSeries,Rational, RISC System/6000, RS/6000, THINK, Tivoli, Tivoli (logo), Tivoli Management Environment, WebSphere, xSeries, z/OS, zSeries, AIX 5L, Chiphopper, Chipkill,Cloudscape, DB2 Universal Database, DS4000, DS6000, DS8000, EnergyScale, Enterprise Workload Manager, General Purpose File System, , GPFS, HACMP,HACMP/6000, HASM, IBM Systems Director Active Energy Manager, iSeries, Micro-Partitioning, POWER, PowerExecutive, PowerVM, PowerVM (logo), PowerHA, PowerArchitecture, Power Everywhere, Power Family, POWER Hypervisor, Power Systems, Power Systems (logo), Power Systems Software, Power Systems Software (logo),POWER2, POWER3, POWER4, POWER4+, POWER5, POWER5+, POWER6, POWER7, pureScale, System i, System p, System p5, System Storage, System z, TivoliEnterprise, TME 10, TurboCore, Workload Partitions Manager and X-Architecture are trademarks or registered trademarks of International Business MachinesCorporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with atrademark symbol (® or ™), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Suchtrademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at "Copyright and trademarkinformation" at Power Architecture and wordmarks and the Power and logos and related marks are trademarks and service marks licensed by is a registered trademark of The Open Group in the United States, other countries or both.Linux is a registered trademark of Linus Torvalds in the United States, other countries or both.Microsoft, Windows and the Windows logo are registered trademarks of Microsoft Corporation in the United States, other countries or both.Intel, Itanium, Pentium are registered trademarks and Xeon is a trademark of Intel Corporation or its subsidiaries in the United States, other countries or both.AMD Opteron is a trademark of Advanced Micro Devices, Inc.Java and all Java-based trademarks and logos are trademarks of Sun Microsystems, Inc. in the United States, other countries or both.TPC-C and TPC-H are trademarks of the Transaction Performance Processing Council (TPPC).SPECint, SPECfp, SPECjbb, SPECweb, SPECjAppServer, SPEC OMP, SPECviewperf, SPECapc, SPEChpc, SPECjvm, SPECmail, SPECimap and SPECsfs aretrademarks of the Standard Performance Evaluation Corp (SPEC).NetBench is a registered trademark of Ziff Davis Media in the United States, other countries or both.AltiVec is a trademark of Freescale Semiconductor, Inc.Cell Broadband Engine is a trademark of Sony Computer Entertainment Inc.InfiniBand, InfiniBand Trade Association and the InfiniBand design marks are trademarks and/or service marks of the InfiniBand Trade Association.Other company, product and service names may be trademarks or service marks of others. Revised February 9, 2010 45 © 2012 IBM Corporation