Memory Matters in 2011

IBM System z Technical University – Vienna , Austria – May 2-6NoticesThis information was developed for products and services offered in the U.S.A.Note to U.S. Government Users Restricted Rights — Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the users responsibility to evaluate and verify the operation of any non-IBM product, program, or service.IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing, IBM Corporation, North Castle Drive Armonk, NY 10504-1785 U.S.A.The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you.This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice.Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk.IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you.Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental.COPYRIGHT LICENSE:This information contains sample application programs in source language, which illustrates programming techniques on various operating platforms. You may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs. You may copy, modify, and distribute these sample programs in any form without payment to IBM for the purposes of developing, using, marketing, or distributing application programs conforming to IBMs application programming interfaces. © 2011 IBM Corporation

IBM System z Technical University – Vienna , Austria – May 2-6Trademarks This presentation contains trade-marked IBM products and technologies. Refer to the following Web site: http://www.ibm.com/legal/copytrade.shtml © 2011 IBM Corporation

IBM System z Technical University – Vienna , Austria – May 2-6AbstractMemory - hardware, z/OS and associated middleware - continues to be anexciting subject, with enhancements coming thick and fast. Meanwhile itsusage and management is often poorly understood.This presentation brings performance practitioners and capacity plannersup to date with topics on both real and virtual memory, with DB2 discussedas a key memory-consuming product.There will also be an opportunity to discuss what instrumentation andcapabilities are needed for the future. (Martin is in active discussions withhardware and software development organisations on both the function andinstrumentation fronts.) © 2011 IBM Corporation

IBM System z Technical University – Vienna , Austria – May 2-6Topics  System-Level – Paging Subsystem Design – Dump Space – 1 MB (Large) Pages – z/OS R.10 64-Bit Common – ADDR64 – System Area Above 2GB  Coupling Facility  DB2  Conclusion © 2011 IBM Corporation

IBM System z Technical University – Vienna , Austria – May 2-6Systems That The “New” z/OS R.8 Algorithm Might Impact  Design point is for “zero paging” considerations – Most production systems  Systems likely to page: – Small production systems – Development and Test systems • Is “Development” really so low priority these days?  A potential mitigator for paging systems: – Frequently-referenced pages generally “more important”  Pragmatic choice: – Whether to allow some memory constraint – Even more pragmatic choice: • Whether to configure for some unusual peak – (See later foils on dumping) © 2011 IBM Corporation

IBM System z Technical University – Vienna , Austria – May 2-6Paging Subsystem Design  Zero paging is not a reason to skimp on your paging subsystem – Even more memory to dump when you have to • At least be able to dump your big address spaces eg DBM1 • It’s preferable to be able to dump the entire system  Rule of Thumb: – Have your free paging space be at least 1.5 times your real storage • Gives you a chance at containing "big dump" cases  Another Rule of Thumb: – Dont let your page data sets get above 30% full • Contiguous Slot Algorithm degrades above that point  These Two Rules of Thumb complement each other – But consider them a starting point in your design discussions  Consider the value of PAV now that ASM supports it – But still consider the importance of proper I/O design © 2011 IBM Corporation

IBM System z Technical University – Vienna , Austria – May 2-6Paging Subsystem Design …  It’s possible to have two page data sets on the same volume – Optimism that performance is OK with PAVS enabled • IOSQ observed to be 0 • Disc observed to be non-trivial but probably the case with 1 – Paging I/O not cached – Motivation: To use large volumes exclusively for paging  APAR OA20749 allows larger page data sets – Limit up from 4 GB to 44.9 GB • 64K cylinders – Available with z/OS R.8 onwards – 1 large page data set can have 2 PAVs • 2 smaller ones can have 4  In a not-so-recent customer case 2107 greatly outperformed 2105 – And ASM was observed to route paging I/O towards the 2107 data sets  HyperPAVs work properly for page data sets as of R.10 © 2011 IBM Corporation

IBM System z Technical University – Vienna , Austria – May 2-6Dump Space  Dumping managed at system level by CHNGDUMP (CD) operator command – CD SDUMP,BUFFERS=nnnM • Target value for real frames – CD SDUMP,MAXSPACE=nnnnnnnnM • Maximum concurrent virtual space – Defaults to 500MB • If you run out dumps can’t be taken • Partial dumps a possibility – Likely to impede diagnostics  Dumping much faster if all in memory than a combination of paging space and memory – A trade off between availability and memory provisioning  Subsystems may be smart in what they dump and in what order – To maximise the likelihood of getting adequate diagnostics © 2011 IBM Corporation

IBM System z Technical University – Vienna , Austria – May 2-6Dumping Enhancements in z/OS R.12 Batch page-in I/O operations during SDUMP capture eliminates much of the I/O delay Data captured will no longer look recently referenced – This data will be paged-out before other potentially more important data Certain components now exploit a more efficient capture method in their SDUMP exits – For example GRS for SDATA GRSQ data, IOS for CTRACE data, and configuration dataspaces © 2011 IBM Corporation

IBM System z Technical University – Vienna , Austria – May 2-61MB (Large) Pages ● Issue: Translation Lookaside Buffer (TLB) Coverage shrinking as % of memory size ● 1 MB Pages allow for a single TLB entry to fulfill many more address translations than 4KB pages ● 1 MB Pages will provide exploiters with better TLB coverage ● Only fixed above the bar virtual can be backed by 1 MB pages ● No paging ● Mixed 4 KB and 1 MB running the rule ● OA25485: NEW FUNCTION - FACILITY CLASS FOR RSM LARGE PAGE ACCESS ● Allows non-Authorised applications to use IARV64 REQUEST=GETSTOR PAGEFRAMESIZE=1MEG or MAX ● Facility Class is IARRSM.LRGPAGES ● Exploitation by Websphere Application Server V7 ● Java heap ● Java 6 SR 1 or later with -X1p1M flag ● Buffer Pools in DB2 10 ● OA32001: LFAREA PARAMETER INCONSISTENT WITH VALIDITY CHECKS ● Corrects calculation if LFAREA=nn% used to be % of >2GB real © 2011 IBM Corporation

IBM System z Technical University – Vienna , Austria – May 2-6 z/OS R.12 Large Page Support Nucleus – Note: Nucleus is in 31-bit storage – Expected to improve performance for z/OS itself Large Page Coalesce support – During page storage shortage situations available large pages converted to 4KB pages – Later 4KB pages will be coalesced to form 1MB pages – OA31116: VARIOUS LARGE PAGE COALESCE AND RECOVERY FIXES • Implements this and fixes some other problems • For Release 10 and 11 also © 2011 IBM Corporation

IBM System z Technical University – Vienna , Austria – May 2-6z/OS R.10 64-Bit Common  IARV64 REQUEST=GETCOMMON  Allows sharing above the bar across every address space – In a much simpler fashion than Shared 64-Bit  Controlled by HVCOMMON  Any address space can access storage without explicitly having to request access  To limit overlays only allowed if: – Authorized code – System keys 0-7  Storage can be DREF, Fixed – Unlike Shared  Storage needs to be explicitly freed  Requires explicit exploitation – To achieve 31-Bit Common VSCR © 2011 IBM Corporation

IBM System z Technical University – Vienna , Austria – May 2-6RMF Support For 64-Bit Common  Type 71: – Number of 64-bit common memory objects allocated – 64-bit common memory frames backed in real storage – 64-bit fixed common memory frames – 64-bit common memory auxiliary storage slots – Additional information on shared memory objects: • Number of 64-bit shared memory objects allocated • High virtual shared memory frames backed in real storage © 2011 IBM Corporation

IBM System z Technical University – Vienna , Austria – May 2-6RMODE 64 For Non-Executable Modules New with z/OS R.11 The LOAD macro supports new keywords to identify an area above 2G in virtual for a directed load: – eg ADDR64 keyword as the analog of ADDR The system does not verify that the module truly can be relocated above 2G – For example, it may have a 4-byte address constant to another part of itself – This is consistent with current LOAD with ADDR behavior The system does not verify that the module is truly non-executable. – If attempt is made to transfer flow of execution to the module, results are unpredictable • Also true if any executable code is placed above 2G by any mechanism © 2011 IBM Corporation

IBM System z Technical University – Vienna , Austria – May 2-6USEZOSV1R9RULES Setting to NO changes GETMAIN / STORAGE OBTAIN behaviour – Generally a safe thing to do – However may result in ABEND0Cx, overlay of storage, or other problems • Documented in APAR OA27291 Setting to NO has been observed to improve DB2 startup behaviour: – Particularly the opening of a large number of VSAM data sets – Note also DB2 APAR PM17542 and z/OS R.12 MEMDSENQMGMT: • Uses in-memory structures to decrease ENQ time for large numbers of data sets USEZOSV1R9RULES introduced with z/OS Release 10 © 2011 IBM Corporation

IBM System z Technical University – Vienna , Austria – May 2-6System Area Above 2GB A new system area will be created in the address space 64-bit map – Analogue of (E)LSQA Memory objects allocated in the System Area will start at x’8_00000000’ An authorized program can issue IARV64 REQUEST=GETSTOR, LOCALSYSAREA=NO|YES to indicate that the memory object should be allocated from the System Area above 2GB In fork processing, during the 64-bit copy phase, memory objects that are allocated in the system area will not be copied to the child space © 2011 IBM Corporation

IBM System z Technical University – Vienna , Austria – May 2-6Coupling Facility Memory  Much more static usage than most z/OS LPARs – Though varying traffic suggests different demands for memory  “White space” considerations apply – Duplexing obviously lowers the requirement  Main categories: – Dump – Structures • Structure sizing and management is an important topic – Available  Instrumentation: – Type 74 Subtype 4 – (Type 70 Subtype 1 just gives memory at ICF LPAR level) – Exploiter-level statistics © 2011 IBM Corporation

IBM System z Technical University – Vienna , Austria – May 2-6Coupling Facility Memory - Structures  Each structure type has different memory management considerations – e.g “False Lock Contention” avoidance in lock structures – Not just structure type but exploiter • e.g how VSAM RLS Cache works vs DB2 Group Buffer Pools  Structures occasionally need to increase in size when upgrading to a later CFLEVEL  Use CFSIZER website to size CF structures – http://www.ibm.com/systems/z/cfsizer/ – Most IBM Product structures catered for • Given a product name it tells you which structures you want – And suggests a size • Produces an “initial” configuration – For example DB2 structures are likely to need tuning > In fact CFSIZER probably suggests to you too few GBPs • Has good Help © 2011 IBM Corporation

IBM System z Technical University – Vienna , Austria – May 2-6Coupling Facility Memory – Structures ...  4 Potential sizes: – Current size - R744SSIZ – Maximum size (MAXSIZE) - R744SMAS – Initial size (INITSIZE) - R744QSIZ – Minimum Size (MINSIZE) - R744SMIS  AUTOALTER can change current size – For all structure types – e.g Directory Entry / Data Element ratio for cache structures • Increases the size of the portion thats constrained  AUTOALTER is NOT perceived as a “workload spike” handler – Its really for gradual workload growth  Current usage and limits for contents of structure in 74-4 – Also Lock Structure Contention / False Contention numbers – True understanding requires exploiter instrumentation • e.g DB2 Statistics and Accounting traces © 2011 IBM Corporation

IBM System z Technical University – Vienna , Austria – May 2-6DB2  DBM1 Address Space is biggest user of real memory – IRLM, MSTR and DIST address spaces generally smaller • Rare to see virtual storage issues with these 3  Main DBM1 virtual storage usage in 2 areas: – Buffer Pools • In 64-Bit Virtual in V8 • WLM Automatic Buffer Pool Management in V9 – Thread-Related Storage • Used for callers of DB2 services i.e. applications • In 31-Bit Virtual in V8  Other areas generally smaller – But not always • e.g. EDM Pool – Sizes usually related to applications • But not linear with concurrent threads – e.g. Prepared Statement Cache, RID Pool, Sort Pool © 2011 IBM Corporation

IBM System z Technical University – Vienna , Austria – May 2-6DB2 Memory Effectiveness Instrumentation  SMF 100 Records – Give virtual memory view of buffer pools • Virtual pools, dataspace pools (and hiperpools) – And thresholds • All these are variable using ALTER BUFFERPOOL command • Effectiveness of buffer pools • Group Buffer Pools – Reinforced by SMF 101 application-level reporting – Some other areas • e.g Logging, EDM Pool © 2011 IBM Corporation

IBM System z Technical University – Vienna , Austria – May 2-6DB2 Virtual Storage Instrumentation  IFCID 225 breaks down into eg – Long-Term used • eg Buffer pools – Used unrelated to threads – Thread related storage • All threads – Answers general questions about scalability – Actual groupings are GETMAINed, Variable, Stack and Fixed – Also gives REAL memory usage  IFCID 225 is enhanced significantly in DB2 Versions 8, 9 and 10 – For example to show real memory usage • Very recent APAR in V10 to show real memory above the Bar  IFCID 217 allows you to break down thread-related storage into – eg Blue, Red and Green threads – Potentially answers questions about scalability based on detailed thread growth "by colour"  Lots of tuning knobs for DB2 virtual storage © 2011 IBM Corporation

IBM System z Technical University – Vienna , Austria – May 2-6 Client H DB2 Virtual Storage SYSA Subsystem DB2A July 14, 2004 1200 1000 800 MB 600 400 200 0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 Hour Virtual Pools EDM Pool Free EDM Pool Used GBP Castout Comp Dicts Other GETMAINed Agt Local Non-Sys Ag Local Sys Pipe Mgr RDS Ops RID Pool PSC Control Blocks PSC Statements Trace Tables Other Variable Fixed Stack © 2011 IBM Corporation

IBM System z Technical University – Vienna , Austria – May 2-6 Client H DB2 DBM1 Thread Memory - Headline Numbers SYSA Subsystem DB2A 700 800 600 700 500 600 500 Threads 400 MB 400 300 300 200 200 100 100 0 0 Hour Thread Storage Threads © 2011 IBM Corporation

IBM System z Technical University – Vienna , Austria – May 2-664-Bit Virtual Storage Constraint Relief in DB2 Version 8  Most of thread storage stays below the 2GB bar – Implies you still need to manage threads – Experiences suggest some growth in thread-related storage  IRLM also is 64 bit – Allows many more concurrent locks – PC=NO is no longer supported • ECSA usage is no longer possible • PC instruction used instead to access IRLM address space  Expect to see significant growth in real memory usage – As subsystems are now able to grow • Larger buffer pools • Larger areas such as Prepared Statement Cache • Perhaps more threads – As long-term page fixing is available as an option • Pain if even a single page is not backed by real memory – As DB2 Version 8 requires more real memory anyhow © 2011 IBM Corporation

IBM System z Technical University – Vienna , Austria – May 2-664-Bit Virtual Storage Constraint Relief in DB2 Version 9  EDM Pool: – SKPT and SKCT sections moved above the bar – CT and PT sections split between above and below the bar – Hash tables and fixed pools moved above the bar – BUT what remains is not subject to LRU algorithm • So EDM Pool MUST be sized for the peak usage and then some  Dynamic Statement Cache: – Considerable movement of control blocks to above the bar  DBM1 Internal Monitor: – Reports with console messages when thresholds reached  DDF: – Uses 128GB Memory Object instead of ECSA • Always created at DB2 startup • All DB2 address spaces are registered to use it • Eliminates most data moves • Reduces total storage usage as only 1 copy • Reduces ECSA usage • Eases use of larger communications buffers  Utilities use Shared Memory Objects – Avoids the CPU to move rows between the Batch Utility and DBM1 Address Spaces © 2011 IBM Corporation

IBM System z Technical University – Vienna , Austria – May 2-6DB2 Long-Term Page Fixing  For each I/O the buffers must be page-fixed and –freed – Expensive in CPU terms  V8 allows LONG TERM page fixing by pool – PGFIX(YES) – Can save significant CPU • Most especially for high I/O rate buffer pools • At the other end of the scale – 100% hits – maybe LRU algorithm good  In V9 Group Buffer Pool castout buffers are ALWAYS long term page fixed – Separate area from buffer pools – If high castout I/O rate could be a significant saving  IMPORTANT: Long term page fixing MUST be considered in the light of system conditions: – For example, don’t page fix to the point of causing other memory users to page to death • Especially important for z/OS Release 8 and beyond © 2011 IBM Corporation

IBM System z Technical University – Vienna , Austria – May 2-6DB2 10 Vast majority of thread storage moved above The Bar –Expectation of 5 to 10 times as many threads supportable –Might allow coalescing of DB2 subsystems • But caution on the other reasons for maintaining the current number –Might allow other things • Such as RELEASE(DEALLOCATE) to cut CPU –Some of working storage (some of stack, xproc) remains below The Bar => Focus shifts to REAL memory Reworking of Dynamic SQL Prepared Statement Cache could alter the basis for its sizing –Two statements which are identical but with DIFFERENT literal values are now treated as the same • One copy in the cache • More hits per cached copy –Might allow a reduced cache • Equally, might make Prepared Statement Cache more worth doing –Still doesnt obviate the benefit of “keeping prepared statements beyond Commit” NOTE: MAXKEEPD could be bigger because area moved above The Bar © 2011 IBM Corporation

IBM System z Technical University – Vienna , Austria – May 2-6DB2 10 ... In-Memory Pagesets – Data read into buffer pool at startup – Useful for small lookup tables 1 MB Page Size – Requires long-term page-fixing the buffer pools © 2011 IBM Corporation

IBM System z Technical University – Vienna , Austria – May 2-6Conclusion  Virtual and Real memory and the paging subsystem are very much worth managing – For capacity planning – For resource optimisation – Because installations can still face constraints – Because running out of space can cause an outage • Whether real or virtual memory or paging space  Consider your stance on keeping memory free for e.g. dump capture  There’s lots of instrumentation to help you  Product capabilities are continuing to evolve – 2007 saw DB2 Version 9, IMS Version 10 and CICS TS 3.2 – 2008 has seen System z10 EC, z/OS Release 10, WAS V7 and MQ V7 – 2009 saw IMS Version 11 and CICS TS 4.1 and z/OS Release 11 – 2010 saw DB2 10 and z/OS Release 12  It’s important to look at the system, the subsystem & the application all together – And at the machine level wouldn’t be a bad idea, either – Applications drive subsystem memory demand • But supply comes from the system © 2011 IBM Corporation

Memory Matters in 2011

by Martin Packer

Accessibility

Upload Details

Usage Rights

Statistics