Life And Times Of An Address Space

1,757 views

Published on

Given at UKCMG Annual Conference May 2013 and System z Technical University June 2013

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,757
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Life And Times Of An Address Space

  1. 1. Life And Times Of An Address Space Martin Packer, IBM martin_packer@uk.ibm.com
  2. 2. Abstract A typical z/OS system has a wide variety of address spaces. So much so that managing their performance can be difficult. This presentation prepares you to handle this diversity, discussing what's common to all and what's different. Centred around SMF Type 30 records, it guides you in deciding when to rely on common instrumentation, and when to go to more specific data, such as CICS instrumentation or data set records.
  3. 3. Agenda ● Let's Treat An Address Space As A “Black Box” ● Different Kinds Of Address Space – Long Running Address Spaces – Batch Jobs ● When All Else Fails
  4. 4. Let's Treat An Address Space As A “Black Box”
  5. 5. Let's Treat An Address Space As A “Black Box” ● Typical installation: – Hundreds to thousands of address spaces – Spread across numerous LPARs – Informal analogy: DFSMS managing growing data ● Typical Performance Person (me?) :-) – Doesn't know everything that runs in a system – Has less z/OS knowledge than the average Sysprog ● Has even less middleware knowledge ● Apply rules that are relevant to e.g. all CICS regions ● Start by treating an address space as if it's a mystery – Add domain knowledge as necessary – Don't bother for 90% of them ● … and if a new one comes along ...
  6. 6. Common Attributes ● Matter of fact they're almost all common ● e.g. – CPU timings – EXCP counts – Memory and virtual storage – Unix System Services – Product Usage – WLM Information ● The clue is in the name: – “Common Address Space Work”
  7. 7. SMF 30 Record - Subtypes Subtype Synopsis Scope Usefulness 1 Start Low 2 Interval Whole interval, or from step start High 3 Step End Last interval High 4 Step End Whole Step High 5 Job End Whole Job High 6 Non-Full- Function Cumulative from the start Low
  8. 8. SMF 30 Record - Handling ● Handle with care – Some sections filled in only for certain subtypes – Not just the standard triplet ● Some “quadlets”: Count of sections in subsequent records ● Accounting section doesn't follow the rules – DDCONS “changes the rules” ● Sections: – Subsystem – Identification – I/O Activity – Completion – Processor – Accounting – Storage ─ Performance ─ Operator ─ EXCP ─ APPC/MVS (x2) ─ Unix ─ Usage ─ ARM ─ Multisystem Enclave
  9. 9. Who Am I? ● Address Space Name ● Program name – e.g DFHSIP, DSNYASCP – Unix program name ● STC vs TSO, vs JES... ● Mnemonic DD names ● Usage information – Includes e.g. CICS Version / Release ● Step counts ● WLM Workload, Service Class, Report Class – More with “Served By” in Type 72-3 ● Example: DDF and DIST – “Names are just character strings” ● Description fields even more so
  10. 10. Who Do I Talk To? ● Address Space – XCF – Usage – Important to understand which address space serves which in e.g. DB2 ● Transaction – DB2 ● Accounting Trace – MQ ● Accounting Trace – CICS ● Monitor Trace ● Note: 1 CICS region can only talk to 1 DB2 subsystem or MQ queue manager
  11. 11. Are You Being Served? ● Aside from WLM Service Class Topology ● Some address spaces host enclaves – “Traffic analysis” can reveal which served service classes they support ● Use Enclave SMF 30 Transaction Count and 72-3 Transaction Count
  12. 12. When Do I Run?
  13. 13. An aside on Reader Start Time ● An easy way to establish IPL time: IEEMB860 ● Likewise CICS region restarts: DFHSIP ● And DB2 subsystems: DSNYASCP – With subsystem encoded in job name ● Not to forget MQ subsystems: CSQYASCP ● And last but not least IMS: DVSMVRC0 and DFSRRC00 ● Establish restart patterns
  14. 14. What Data Do I Access? ● Intervals: – SMF 42-6 – SMF 30 EXCP sections – SMF 92 HFS / zFS ● OPEN / CLOSE: – 14, 15 Non-VSAM – 62, 64 VSAM – 92 HFS / zFS – 91 Pipes ● Sorting – SMF 16: SMF=FULL has data sets ● Data Managers – No correlation in SMF between user and data – Subsystem access as above
  15. 15. MEMLIMIT and IEFUSI ● Can establish MEMLIMIT for address space – And source of the limit – HVSHARE / HVCOMMON not in SMF ● Some address spaces have minimum requirements – e.g. CICS 4.2 = 4GB, CICS 5.1 = 6GB ● “Unlimited” is a potential exposure – Conversely limiting can affect eg DFSORT Large Memory Sorting ● DB2 Version 10 has 4TB – set by JCL
  16. 16. Different Kinds Of Address Space
  17. 17. Long-Running Address Spaces ● CICS and DB2 are exemplars – “Other long-running address spaces may be available” ● Treat on an interval basis – Generally plot by time of day – Can usefully plot with load ● Time not on x axis
  18. 18. Long-Running Address Spaces - CPU ● Comprehensive and accurate numbers ● Distinguish between non-enclave and enclave CPU – Example DB2 DIST address space ● Authorisation etc code under SRB ● Served application code under enclave TCB – Different service classes for the two ● Generally supply better service to non-enclave TCB / SRB ● OA39629: – SMF30_Highest_Task_CPU_Percent – SMF30_Highest_Task_CPU_Program – e.g. CICS QR TCB Constraint without Statistics Trace ● DFHKETCB? – e.g. Is this address space sensitive to engine speed?
  19. 19. CPU “Particle Zoo” Source: http://inspirehep.net/record/796674/files/sttfeyn.png ● TCB – Enclave ● Dependent ● Independent – OA39629 – in case you were asleep last time – zIIP vs non-zIIP – zIIP on GCP ● SRB
  20. 20. Long Running Address Spaces - Memory ● Difficult to obtain accurate real memory usage – If necessary use WLM Report Class ● At the single or “cloned” address space level ● Almost impossible to obtain Flash or Aux usage – But can see paging rates – Could use R724SLOT for Report Class ● Virtual storage a little different – Can see 24 / 31 bit and large memory object allocated ● Long runners often suballocate ● 64-bit design point is for user to suballocate – Proper analysis requires middleware-specific data – Use allocated as upper bound for real memory usage
  21. 21. CICS ● Suppose you've triaged your CICS regions and are only interested in a few ● Region Level: Statistics Trace – With a sensible interval – Lots of good information for general use: ● TCB ● Virtual storage ● File Control ● … ● Transaction Level: Monitor Trace – With sensible inclusion of fields – Time buckets ● Note CICS TS 5.1 and 5.2 provide more opportunities for consolidation – Requires some analysis and architectural understanding
  22. 22. DB2 ● Subsystem Level: Statistics Trace – With a sensible STATIME – An enormous amount of information ● Application Level: Accounting Trace – Response Time Buckets ● Treat as a grid if packages are involved – Package level is program level and can track down CPU pigs – SQL Statistics ● DB2 Catalog ● Performance Trace: If you're sufficiently desperate or focused
  23. 23. Batch Jobs ● Perhaps the most fun of all :-) ● List the step timings – Actually look for “sub steps” ● For example, Sort Input Phase ● See “Timestamp Analysis” below – Gives you an idea of what targets to go after – Use Timestamp Analysis to slot records into steps ● For those that don't have step numbers ● Understand how the data sets behave – “Life Of A Data Set” (LOADS Technique) ● Apply Creativity – Only a few expediters work “out of the box” – Generally the expensive ones :-)
  24. 24. When All Else Fails: Go talk to somebody … :-)
  25. 25. … But not before you've done your homework ● With the generic address space data – Handled appropriately ● And perhaps application-specific data
  26. 26. 1 Life And Times Of An Address Space Martin Packer, IBM martin_packer@uk.ibm.com
  27. 27. 2 Abstract A typical z/OS system has a wide variety of address spaces. So much so that managing their performance can be difficult. This presentation prepares you to handle this diversity, discussing what's common to all and what's different. Centred around SMF Type 30 records, it guides you in deciding when to rely on common instrumentation, and when to go to more specific data, such as CICS instrumentation or data set records.
  28. 28. 3 Agenda ● Let's Treat An Address Space As A “Black Box” ● Different Kinds Of Address Space – Long Running Address Spaces – Batch Jobs ● When All Else Fails
  29. 29. 4
  30. 30. 5 Let's Treat An Address Space As A “Black Box”
  31. 31. 6 Let's Treat An Address Space As A “Black Box” ● Typical installation: – Hundreds to thousands of address spaces – Spread across numerous LPARs – Informal analogy: DFSMS managing growing data ● Typical Performance Person (me?) :-) – Doesn't know everything that runs in a system – Has less z/OS knowledge than the average Sysprog ● Has even less middleware knowledge ● Apply rules that are relevant to e.g. all CICS regions ● Start by treating an address space as if it's a mystery – Add domain knowledge as necessary – Don't bother for 90% of them ● … and if a new one comes along ...
  32. 32. 7 Common Attributes ● Matter of fact they're almost all common ● e.g. – CPU timings – EXCP counts – Memory and virtual storage – Unix System Services – Product Usage – WLM Information ● The clue is in the name: – “Common Address Space Work”
  33. 33. 8 SMF 30 Record - Subtypes Subtype Synopsis Scope Usefulness 1 Start Low 2 Interval Whole interval, or from step start High 3 Step End Last interval High 4 Step End Whole Step High 5 Job End Whole Job High 6 Non-Full- Function Cumulative from the start Low
  34. 34. 9 SMF 30 Record - Handling ● Handle with care – Some sections filled in only for certain subtypes – Not just the standard triplet ● Some “quadlets”: Count of sections in subsequent records ● Accounting section doesn't follow the rules – DDCONS “changes the rules” ● Sections: – Subsystem – Identification – I/O Activity – Completion – Processor – Accounting – Storage ─ Performance ─ Operator ─ EXCP ─ APPC/MVS (x2) ─ Unix ─ Usage ─ ARM ─ Multisystem Enclave
  35. 35. 10 Who Am I? ● Address Space Name ● Program name – e.g DFHSIP, DSNYASCP – Unix program name ● STC vs TSO, vs JES... ● Mnemonic DD names ● Usage information – Includes e.g. CICS Version / Release ● Step counts ● WLM Workload, Service Class, Report Class – More with “Served By” in Type 72-3 ● Example: DDF and DIST – “Names are just character strings” ● Description fields even more so
  36. 36. 11 Who Do I Talk To? ● Address Space – XCF – Usage – Important to understand which address space serves which in e.g. DB2 ● Transaction – DB2 ● Accounting Trace – MQ ● Accounting Trace – CICS ● Monitor Trace ● Note: 1 CICS region can only talk to 1 DB2 subsystem or MQ queue manager
  37. 37. 12
  38. 38. 13
  39. 39. 14 Are You Being Served? ● Aside from WLM Service Class Topology ● Some address spaces host enclaves – “Traffic analysis” can reveal which served service classes they support ● Use Enclave SMF 30 Transaction Count and 72-3 Transaction Count
  40. 40. 15 When Do I Run?
  41. 41. 16 An aside on Reader Start Time ● An easy way to establish IPL time: IEEMB860 ● Likewise CICS region restarts: DFHSIP ● And DB2 subsystems: DSNYASCP – With subsystem encoded in job name ● Not to forget MQ subsystems: CSQYASCP ● And last but not least IMS: DVSMVRC0 and DFSRRC00 ● Establish restart patterns
  42. 42. 17 What Data Do I Access? ● Intervals: – SMF 42-6 – SMF 30 EXCP sections – SMF 92 HFS / zFS ● OPEN / CLOSE: – 14, 15 Non-VSAM – 62, 64 VSAM – 92 HFS / zFS – 91 Pipes ● Sorting – SMF 16: SMF=FULL has data sets ● Data Managers – No correlation in SMF between user and data – Subsystem access as above
  43. 43. 18 MEMLIMIT and IEFUSI ● Can establish MEMLIMIT for address space – And source of the limit – HVSHARE / HVCOMMON not in SMF ● Some address spaces have minimum requirements – e.g. CICS 4.2 = 4GB, CICS 5.1 = 6GB ● “Unlimited” is a potential exposure – Conversely limiting can affect eg DFSORT Large Memory Sorting ● DB2 Version 10 has 4TB – set by JCL
  44. 44. 19 Different Kinds Of Address Space
  45. 45. 20 Long-Running Address Spaces ● CICS and DB2 are exemplars – “Other long-running address spaces may be available” ● Treat on an interval basis – Generally plot by time of day – Can usefully plot with load ● Time not on x axis
  46. 46. 21 Long-Running Address Spaces - CPU ● Comprehensive and accurate numbers ● Distinguish between non-enclave and enclave CPU – Example DB2 DIST address space ● Authorisation etc code under SRB ● Served application code under enclave TCB – Different service classes for the two ● Generally supply better service to non-enclave TCB / SRB ● OA39629: – SMF30_Highest_Task_CPU_Percent – SMF30_Highest_Task_CPU_Program – e.g. CICS QR TCB Constraint without Statistics Trace ● DFHKETCB? – e.g. Is this address space sensitive to engine speed?
  47. 47. 22 CPU “Particle Zoo” Source: http://inspirehep.net/record/796674/files/sttfeyn.png ● TCB – Enclave ● Dependent ● Independent – OA39629 – in case you were asleep last time – zIIP vs non-zIIP – zIIP on GCP ● SRB
  48. 48. 23 Long Running Address Spaces - Memory ● Difficult to obtain accurate real memory usage – If necessary use WLM Report Class ● At the single or “cloned” address space level ● Almost impossible to obtain Flash or Aux usage – But can see paging rates – Could use R724SLOT for Report Class ● Virtual storage a little different – Can see 24 / 31 bit and large memory object allocated ● Long runners often suballocate ● 64-bit design point is for user to suballocate – Proper analysis requires middleware-specific data – Use allocated as upper bound for real memory usage
  49. 49. 24 CICS ● Suppose you've triaged your CICS regions and are only interested in a few ● Region Level: Statistics Trace – With a sensible interval – Lots of good information for general use: ● TCB ● Virtual storage ● File Control ● … ● Transaction Level: Monitor Trace – With sensible inclusion of fields – Time buckets ● Note CICS TS 5.1 and 5.2 provide more opportunities for consolidation – Requires some analysis and architectural understanding
  50. 50. 25 DB2 ● Subsystem Level: Statistics Trace – With a sensible STATIME – An enormous amount of information ● Application Level: Accounting Trace – Response Time Buckets ● Treat as a grid if packages are involved – Package level is program level and can track down CPU pigs – SQL Statistics ● DB2 Catalog ● Performance Trace: If you're sufficiently desperate or focused
  51. 51. 26 Batch Jobs ● Perhaps the most fun of all :-) ● List the step timings – Actually look for “sub steps” ● For example, Sort Input Phase ● See “Timestamp Analysis” below – Gives you an idea of what targets to go after – Use Timestamp Analysis to slot records into steps ● For those that don't have step numbers ● Understand how the data sets behave – “Life Of A Data Set” (LOADS Technique) ● Apply Creativity – Only a few expediters work “out of the box” – Generally the expensive ones :-)
  52. 52. 27 When All Else Fails: Go talk to somebody … :-)
  53. 53. 28 … But not before you've done your homework ● With the generic address space data – Handled appropriately ● And perhaps application-specific data

×