Optimizing z/OS Batch

1. Large Systems Update 2011Optimizing z/OS BatchMartin PackerIBM Worldwide Banking Center of Excellencemartin_packer@uk.ibm.com

2. NoticesThis information was developed for products and services offered in the U.S.A.Note to U.S. Government Users Restricted Rights — Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the users responsibility to evaluate and verify the operation of any non-IBM product, program, or service.IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing, IBM Corporation, North Castle Drive Armonk, NY 10504-1785 U.S.A.The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you.This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice.Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk.IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you.Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental.COPYRIGHT LICENSE:This information contains sample application programs in source language, which illustrates programming techniques on various operating platforms. You may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs. You may copy, modify, and distribute these sample programs in any form without payment to IBM for the purposes of developing, using, marketing, or distributing application programs conforming to IBMs application programming interfaces.2 © 2011 IBM Corporation

4. AbstractBatch performance optimization remains a hot topic for many customers, whether merging workloads, supporting growth, removing cost or extending the online day.This presentation outlines a structured methodology for optimizing the batch window, incorporating techniques written about in a Redbook written by experts from around the world. This methodology is well-structured and draws on information every installation should have access to. © 2011 IBM Corporation

7. About The Redbook Written by a worldwide team in Summer 2011: –Alex Louwe-Kooijmans (ITSO Poughkeepsie) –Jan van Cappelle (IBM Netherlands) –Martin Packer (IBM United Kingdom) –Tomohiko Kaneki (IBM Japan) Follows on from 2009 Redbook “Batch Modernization on z/OS” SG24-7779 –Tries hard not to repeat its material Meanwhile in the other room –Other residents wrote about modern batch containers –They were “beyond the firewall” © 2011 IBM Corporation

9. Problem Statements Context is about turning an initial problem statement (or question) into a better one. Initial problem statements look like: 1)“Between 10:00 am and 4:00 pm we had bad online transaction response times.” 2)“The time window to run my batch is too short.” 3)“The processor utilization is too high.” Initial questions look like: 1)“When we merge company A with company B will there still be enough time to run our batch jobs?” 2) “Our batch schedule is so complex we don’t know how it works anymore. We are worried that when there are problems we won’t be to figure out how it works.” 3) “Can you help us reduce our batch processing costs by 20%?” 4) “There is no specific department responsible for our batch schedule, every department has it own standards and, no one thinks about the overall corporate perspective and requirements.” 5) “We want to clean up our batch schedule but do not know where to start. Can you help?” © 2011 IBM Corporation

10. Better Problem Statements We think the following are better versions of the 3 problem statements you just saw: 1)“Between 10:00am and 4:00pm online transaction response times exceeded the service level agreement by 50%. This is causing 100 user complaints a day. We have identified that batch running in the online day is causing a processor capacity shortage.” 2)“We have between 8:00pm and 6:00am to run our batch in. In particular we have to get the replenishment application completed by 3:15am. Today we finish around 2:30am except for when we have a problem...” 3)“Batch is driving our processor capacity requirements (or our software bill because it drives the peak of our Rolling 4 Hour Average). We need to reduce this peak by 25% to help manage our costs.” © 2011 IBM Corporation

12. Roles ... Translating initial problem statements into better ones takes the right people – Initial statements tend to come from the business • Client business expert talks to IBM representative • Sometimes, though, IT acts proactively – IBM Batch Subject-Matter-Expert (SME) needs to help improve the problem statement – SME also needs to shape the project • And maybe assemble a team with other technical SMEs in © 2011 IBM Corporation

13. A Good Business Case Because Batch Optimization projects are extensive a good business case is essentialSome drivers include: Processor usage is too high and a reduction in processor usage lowers costs. – The reduction of million service units (MSU) lowers costs. • When batch processing has bottlenecks it can impact the business clients and there is a risk of not – meeting Service Level Agreements (SLAs). When SLAs are not met, it impacts the business clients and you might have to pay a penalty for • not meeting your SLA. This represents a risk of profit loss. The client wants to simplify the batch schedule. – The advantages are lower maintenance costs, fewer skilled operators are needed, and there is • less risk in your batch schedule. The client wants to reduce people time by cleaning up the batch schedule. – When your batch schedule is cleaned up you need less time to maintain the schedule and less • time to develop additional function, because the schedule is now easier to understand. The client wants to reduce hardware resources. – Reducing your hardware resources such as disk space and network usage lowers your costs. • client wants to reduce errors in their batch schedule. The – When your schedule is cleaned up and simplified the probability of errors is reduced and it is • easier to meet the SLA. © 2011 IBM Corporation

14. But Consider The Costs, Too Solutions can have financial impacts, introduce risk, or require a lot of effort. For example: – The removal of 30,000 obsolete jobs from a schedule of 300,000 jobs. When removing obsolete jobs you also have to review and correct any dependencies. – The removal or changes to 10% incorrect dependencies in a schedule of 300,000 jobs. – The installation of BatchPipes/MVS to create more parallelism. But this requires you to acquire and install this program product. – Required hardware upgrades in order to meet the SLA. A good business case would, of course, have far more benefits than costs © 2011 IBM Corporation

16. Analysis Methodology At this level the methodology applies equally well to batch on all platforms Comprises 3 steps: 1) Initialization • Standard Project Management “kick off” • We emphasise it because of the complexity 2) Analysis • The interesting part :-) 3) Implementation • Again, similar to “standard” Project Management • Focus on implementing and measuring • Emphasised because of the complexity © 2011 IBM Corporation

18. Setting The Strategy An important first step – Sets the technical direction based on • System conditions • Proclivities of the Batch workload Woven from 7 strategies: – Ensuring the system is properly configured – Implementing Data In Memory (DIM) – Optimizing I/O – Increasing parallelism – Reducing the impact of failures – Increasing operational effectiveness – Improving application efficiency These strategies described in more detail in: – SG24-2557 “Parallel Sysplex Batch Performance” – SG24-7779 “Batch Modernization on z/OS” © 2011 IBM Corporation

19. Setting The Strategy – In The z/OS EnvironmentIn the book we take platform-neutral “7 strategies” and make them more z/OS specificSome of the strategies can have resource implications: - We advocate the use of RMF and other SMF to understand the resource pictureSome require trawling through eg Operations logs - Especially if you want to know when certain events occurApplication efficiency (Strategy 7) is especially difficult to assess - Best left to “Understanding the batch landscape” © 2011 IBM Corporation

20. 7 Strategies – Some z/OS Examples1) Ensuring the system is properly configured:  Ensure the WLM setup is appropriate for batch2) Implementing Data In Memory:  Using VSAM LSR buffering adequately3) Optimizing I/O:  Install enough disk I/O bandwidth4) Increasing Parallelism:  Use BatchPipes/MVS5) Reducing Impact Of Failures:  Use DFSORT / ICETOOL capabilities to “cleanse” data6) Increasing Operational Effectiveness:  Use Tivoli Workload Scheduler to automate batch7) Application Efficiency:  DB2 for z/OS SQL Tuning © 2011 IBM Corporation

21. Understanding The Batch Landscape (or “Batchscape”) Complexity is the biggest obstacle to managing batch – Tens or hundreds of thousands of jobs – Many hundreds of thousands of data sets and database objects – etc... Understanding the batch landscape is important – and difficult Try to understand: – Naming conventions – and their degree of observance – Job functions – for the important jobs – Application ownership boundaries – Users of database objects – Critical path and end points – Reliability – Job times … amongst other elements of the batch landscape You probably already do understand the systems and database managers and networks... To prepare for the next stage you need to derive a short list of jobs to examine in detail – Probably no more than 30 to start with – Easily extensible later – Whether “critical” or “typical” they have to be worth tuning © 2011 IBM Corporation

23. Understanding The Batch Landscape – In The z/OS EnvironmentMain sources of information are: - Tivoli Workload Scheduler • Application Description database - SMF Type 30 Job- and Step-End Records • Give names, runtime and CPU information • Useful for characterising jobs - SMF Type 101 Accounting Trace • Correlation ID is usually Job Name • For IMS its more complicated than just Correlation ID • Connection Type (QWHCATYP) can be used to discard non-batch records • Also gives very good time breakdown below step levelTheres lots of other information which helps build the total Batch Landscape view - System-Level - Subsystem-Level - WLM Service Class Level © 2011 IBM Corporation

25. Selecting The Jobs To Tune Jobs for detailed study might be chosen for a variety of reasons: – CPU Intensive – Long Running – On “Critical Path” – Problematic – Typical of their type – In important applications Can only realistically look at a few tens of jobs in depth – But these are the ones that are really important to tune Be prepared to add additional jobs to your initial list © 2011 IBM Corporation

26. Deriving Detailed Tuning Actions Remember detailed tuning actions have to fit into the overall strategy – But without these actions there is no progress When considering a system there are many resources to tune or provision – Yes, System-Level counts as a “Detailed” tuning action Likewise database managers Jobs are the obvious things to tune: – Speeding up individual steps – Breaking up jobs for parallelism – Removing job dependencies • Especially spurious, scheduler-induced ones © 2011 IBM Corporation

27. Speeding Up Individual Steps Generally easier and lower risk than the 2 “break up” scenarios Long-running steps are readily identifiable. – Rough guide: Steps > 10% of job’s run time worth considering Examples of how you might tune a step’s run time are: – Tune the application code • Might also reduce processor consumption – Tune data set access – Split the step into clones • running against a subset of the data – Tune DB2 aspects • For example a locking problem • Use DB2 Accounting Trace (SMF 101) Many steps comprise more than one phase. Most well-known example is a sorting step. – A sorting step has three phases: 1) Input phase 2) Work phase 3) Output phase © 2011 IBM Corporation

28. Breaking Up Jobs For Parallelism / Removing Job Dependencies Though the mechanics are different the analysis is similar Reasons why one step runs after another include: – One step writes a data set another step needs to process – One step accesses DB2 in a way that requires another step to be scheduled later – Someone added function to a pre-existing job by simply adding steps – The original design didn’t consider whether steps were independent or dependent on each other As you can see the word “step” could be replaced by “job” in the above examples – Or even “step phase” © 2011 IBM Corporation

29. Dependency Analysis Dependencies are either: – In the schedule (Job-level) – Step-serialised (Step-level) For data sets use Life Of A Data Set (LOADS) analysis: – Some patterns suggest a dependency: • A writer followed by a reader • A writer followed by a writer • A reader followed by a reader • A reader followed by a writer For DB2 etc its much more difficult © 2011 IBM Corporation

31. Governance We define governance as: – “... the formal process and policies for controlling, directing, and making batch decisions”We go on to say: “Governance empowers you with consistent management, cohesive policies, guidance, and decision rights for your batch actions and solutions. You can use these processes to verify your batch performance.” An important part of it is creating the Batch Design Authority (BDA) To create governance: 1) Appoint the Batch Design Authority. 2) Revise the standards and guidelines. 3) Communicate the standards and guidelines. 4) Set up a baseline and monitor. 5) Start making improvements. 6) Control and perform quality assurance. © 2011 IBM Corporation

33. Antipatterns Not to be confused with “sins” – Difference depends on whether the “sin” matters Pattern: – A named solution to a recurring problem in a contexts Anti-pattern: – Similar to a pattern but leading to negative consequences Of course its not always clear which it is – In the book we have at least one example of this © 2011 IBM Corporation

34. Antipatterns The antipatterns we wrote about in the book are: – Incorrect usage of a special resource – Incorrect usage of a start time – Invalid relationship – Obsolete dummy jobs or dummy operations – Actual starttime not equal to endtime last predecessor – Workload Manager unhelpfully set up for batch – Redundant copies of files – Redundant image copies – Batch prevented from effectively using sysplex-wide resources We couldve written about many more, given an infinite number of monkeys © 2011 IBM Corporation

36. Tools We wrote about – Tivoli Workload Scheduler for z/OS (TWS) tools – z/OS tools – Batch Schedule Performance Optimization (BSPO) Particularly for the z/OS tools we sought to avoid duplicating what was in SG24-7779 © 2011 IBM Corporation

37. Tivoli Workload Scheduler Tools Operational Efficiency: – Query Current Plan (QCP) and Modify Current Plan (MCP) show current Production status – Manage Parallelism by defining parallel servers on the “workstation” Monitoring: – Job errors – Jobs delayed beyond latest start time – Jobs running unexpectedly long Forecasting: – Long Term Plan Current Plan Critical Path Analysis Automatic Job Restart Promoting to a “higher” WLM Service Class ... © 2011 IBM Corporation

41. Batch Schedule Performance OptimizationWhen a customer complains that: –The Batch Schedule does not fit any more in the time line –Critical endpoints are not met –The batch schedule is to complexThen it is time to visit the client and analyze what is wrongAfter the first quick analyses we can advise performing some steps of the seven stepstrategy or we advise to bring the batch schedule under governance and after thatoptimize the batch schedule using BSPO © 2011 IBM Corporation

42. Batch Schedule Performance OptimizationBSPO is: – You have your batch schedule under governance – You start optimizing your batch schedule using the anti-pattern versus pattern strategy. – An Anti-pattern is a do not. • e.g. do not use superfluous relationships. • A pattern is how things have to be. – e.g. only use those relationships you really need. • You can use Source2VALUE™ tooling to find the anti patterns you have to change in a pattern. – The Source2VALUE™ tooling delivers lists with anti patterns. • When you have the list with anti-patterns you start to change them in the to be situation. – Source2VALUE™ is a product of Omnext BV • Has an agreement with IBM to work together on BSPO. © 2011 IBM Corporation

43. More information on zEnterprise IBM zEnterprise landing page: http:// www.ibm.com/systems/z/hardware/zenterprise/index.html IBM zEnterprise 114 (z114): http://www.ibm.com/systems/z/hardware/zenterprise/z114.html IBM zEnterprise Events Landing Page: http://www.ibm.com/systems/breakthrough IBM software for zEnterprise: http://www.ibm.com/software/os/systemz/announcements IBM System Storage: http://www.ibm.com/systems/storage/product/z.html IBM Global Financing: http://www.ibm.com/financing/us/lifecycle/acquire/zenterprise/ IBM Services for zEnterprise: http://www.ibm.com/services/us/gts/zenterprise/index.html IBM zEnterprise / System z Redbooks Portal: http://www.redbooks.ibm.com/portals/systemz43 © 2011 IBM Corporation