Time For DIME


Published on

Data In Memory Exploitation (DIME) presentation. Given UKCMG Annual Conference May 2013 and System z Technical University June 2013

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Time For DIME

  1. 1. Time For D.I.M.E?Martin Packer, IBMmartin_packer@uk.ibm.com
  2. 2. AbstractIn recent years memory has become cheaper, or certainly more plentiful. This enablesus to do new things, or old things faster and better.I believe it is indeed Time For DIME (Data In Memory Exploitation). But weve beenhere before - in the late 1980s. Much has changed but the basic concepts havent. Sothis presentation reminds us of "the way we were" but brings things right up to date. Itcovers why youd want to run a DIME project and how to go about it: It covers both theproject phases and technical aspects, preparing you to make a quick start on realisingthe benefits of DIME.While the main example presented here is DB2, the presentation also discussesCoupling Facility memory exploitation, as well as a number of other examples.
  3. 3. Agenda● What is DIME?● Benefits of DIME● Storage Hierarchy – Then And Now● DIME – Then And Now● Is Now The Time For DIME?● What Do YOU Mean By Full?● Breaking Down Memory Usage● Exploiting Data In Memory
  4. 4. What Is DIME?● DIME = Data In Memory Exploitation● Choices to use memory more aggressively● Examples:– Larger DB2 buffer pools– Larger Java heaps– Larger CICS Dynamic Storage Area limits– Moving temporary data sets to VIO in memory
  5. 5. Benefits of DIME● Keeping the right Data In Memory can:– Reduce response times– Support increased throughput– Speed up batch jobs● Secondarily it might:– Take some of the strain off disks and cache– Allow more parallelism– Save CPU● But it might easily cost you CPU
  6. 6. Storage Hierarchy – Then● 64 – 256 MB CentralStorage● Optional 128 – 512 MBExpanded Storage● MVPG Instruction– Hiperbatch– DB2 Hiperpools– VSAM LSR HiperspaceBuffer Pools● (ADMF was later)● VIO to Expanded Storage● “Paging” to ExpandedStorage● DIM projects abounded● 3990 Disk Cache● 3390 Disks● 3480 Tape
  7. 7. Storage Hierarchy - Now● 32 – 3040GB Memory● Optional Flash Express 1.4 –5.6TB– Dump Capture– Pageable Large Frames– Paging in general● Most of the previousmemory exploiters– VIO recast to use Central– DB2 V10 uses 64-Bit inmost places– Hiperbatch obsolete● IBM DS8870– Cache– Flash– Disk
  8. 8. DIME - Then and Now● THEN: “Marketing” Push to persuadecustomers to upgrade to latest and greatest– Proliferation of “Marketing Aids”● NOW: “Customer Pull to gain value– No Marketing Aids● But User F61 GTF trace still works
  9. 9. Is Now The Time For DIME?● The economics of exploiting memory have changed– Machines come with much more memory– Memory is much cheaper● Products are capable of supporting much more DIMExploitation– e.g. DB2 V10– e.g. CICS TS 5.1– e.g. 64-Bit JVM● New configuration possibilities– e.g System zEC12 Flash Express– 1MB page frames
  10. 10. When Might DIME Be A Bad Idea?● When there is no likely benefit● When you havent got the memory● When youre determined to save CPU● When CPU is already constrained● When you havent sufficient virtual storage● When solving the problem through a quickapplication change is better– Example: DB2 SQL Access Path fix
  11. 11. What Do YOU Mean By Full?● Not able to back all the dataspace memory for any dumpscenario?●Not many free pages at the peak? 1● Paging > 0?●Paging > 100 / sec? 2● Key workloads beginning to page?● Unable to create new address spaces?Notes:1Some programs can adjust usage based on available memory, so this is a tricky one2Alternatively, until page data set Contiguous Slot Allocation algorithm degrades
  12. 12. Breaking Down Memory Usage
  13. 13. Currently Active LPARs● Paging Activity Report gives System view– SMF 70 better than report– Breaks down into queues● e.g Private, CSA, Nucleus– Shows Free● Workload Activity Report gives Service ClassPeriod view– SMF 72-3– Accurate only for non-swappable workloads● SMF 30 gives Address Space view– Treat with suspicion– Virtual Storage numbers more solid● Size of other Active LPARs in Partition DataReport– SMF 70-1
  14. 14. Hypothecated – e.g. For Recovered LPARs● Inactive LPARs memory not reported inPartition Data Report– Or SMF 70-1– Existence of LPAR is● Use HMC– Or your spreadsheet :-(
  15. 15. Unhypothecated – Free For Use● Might need to deploy some into LPARs todo Data In Memory– Actually doing so requires planning● Total Purchased not in RMF● Obtainable from HMC● In Vital Product Data– IBMers have tooling for this● Subtract all LPARs– Active and Inactive
  16. 16. Exploiting Data In Memory
  17. 17. DB2● Remember this?
  18. 18. DB2 ...● That was Version 8● Since then– Version 9 provide much more Virtual StorageConstraint Relief (VSCR)– Version 10 much more still● It pays to examine DB2 Virtual Storage uponmigration– Use IFCID 225– In this case its a happy story
  19. 19. DB2 - Some Things To Consider● Accounting Trace (SMF 101) gives “DIM useful” time buckets● EDM Pool / Prepared Statement Cache– 64 Bit in Version 8– More liberal definition of a match in V10● More or larger threads– Best done on V10– RELEASE(DEALLOCATE) to cut CPU● Buffer pools– 64 Bit in Version 8– Long-Term Page Fixed– 1MB Pages– Flash Express– Bigger Group Buffer Pools
  20. 20. CICS - Some Things To Consider● Considerable VSCR in Versions 4 and 5– Both 24 → 31 and 31 → 64 Bit– But not as comprehensive as DB2● Fewer program compressions– But generally not a problem these days● Bigger VSAM LSR buffer pools– Installations typically not aggressive enough with these● Larger JVM heaps● Potential region consolidation– In turn might make above more effective● Use Statistics Trace (SMF 110) to analyse Virtual Storage first
  21. 21. Coupling Facility● Often see cache structures too small– Usually data elements– Occasionally directory entries● Rarely see lock structures too small– Would cause False Contentions– Knock-on effect on XCF traffic in IXCLOnnn group● Both diagnosable with Coupling Facility ActivityReport– Or SMF 74-4● Remember “white space”– Less of a concern with duplexing
  22. 22. Some Choreography● Timing is important● Example: DB2– The world looks very different DB2 V8 → V9 → V10● Processor family upgrades are a good opportunity– Be careful of “carry forward memory” timings● Some application ramp-ups need help– Have something “in your back pocket”● Every hardware / software upgrade should have anexploitation phase– Possibly nominal
  23. 23. Parting Thoughts
  24. 24. Parting Thoughts● When implementing a new application:– Build a list of potential DIME areas– Size memory accordingly● When tuning an existing application:– Likewise● When capacity planning for memory:– Consider a fresh stance on what “full” means● Day to day:– Be sure to measure memory usage properly