This document summarizes a presentation on Parallel Sysplex performance topics including Coupling Facility CPU instrumentation and structure-level performance analysis. It discusses enhancements in RMF reporting for CF CPU usage and XCF monitoring. Experiments were conducted to analyze CF CPU consumption and service times for different structures under increasing load. The document also covers matching CF CPU data between SMF records, structure duplexing performance impacts, and additional instrumentation available in newer releases.

1. IBM System z Technical University – Vienna , Austria – May 2-6
zZS27 Parallel Sysplex Performance Topics
Martin Packer
© 2011 IBM Corporation

2. Abstract
A while back RMF's reporting of Coupling Facility CPU was enhanced,
mainly to give more granularity.
Also RMF was enhanced to report on XCF better.
This presentation outlines the author's experience with this important
new instrumentation, both from the perspective of Capacity Planning
and from the perspective of how parallel sysplexes perform under
increasing load. It also covers other areas of Parallel Sysplex
performance.
IN CASE YOU WERE IN ANY DOUBT: “other areas” does not mean “ALL
other areas”. :-)
2 © 2011 IBM Corporation

3. Topics
 Structure-Level CPU
– Structure CPU Experiment
 CPU / LPAR Match Up Between 70-1 and 74-4
 Structure Duplexing Performance
 Conclusings and Musions
3 © 2011 IBM Corporation

4. Structure-Level CPU
© 2011 IBM Corporation 4

5. Structure-Level CPU Consumption
 CFLEVEL 15 and z/OS R.9
– Almost all customers are now this far advanced
 New SMF 74-4 Field: R744SETM
– “Structure Execution Time”
 Always 100% Capture Ratio
– Adds up to R744PBSY
 Multiple uses:
– Capacity planning for changing request rates
– Examine which structures are large consumers
– Compute CPU cost of a request
• And compare to service time
• Interesting number is “non-CPU” element of service time - as we shall see
– Understand whether CPU per request has degraded
– Estimating Structure Duplexing cost
 NOTE:
– Need to collect 74-4 data from all z/OS systems sharing to get total request rate
• Otherwise “CPU per request” calculation will overestimate
5 © 2011 IBM Corporation

6. Structure CPU Experiment
6 © 2011 IBM Corporation

7. Structure CPU Experiment
 Based on
– R744SETM Structure Execution Time
– Sync Request Rate
• Virtually no Async
– Sync Service Time
 One minute RMF intervals
– Sorted by request rate increasing
 Run was 1-way DB2 Datasharing
– Only really active structures ISGLOCK and LOCK1
 Red lines are CPU time per request
– Blue lines are Service time per request
 ISGLOCK “low volume”
– Shows amortization of some fixed cost effect
• Wondering also if some “practice effect” affects service times
– CF used IC links
 LOCK1 “high volume”
– More reliable for capacity planning
– CF used a mixture of ISC and ICB links
7 © 2011 IBM Corporation

8. ISGLOCK Requests
16
14
12
Microseconds
10
8
6
3us?
4
2
0
0 10 20 30 40 50 60 70
Requests / Second
CPU Time Service Time
8 © 2011 IBM Corporation

9. LOCK1 Requests
12
10
8
Microseconds
6
4 3.5us?
2
0
750 800 850 900
Requests / Second
CPU Time Service Time
9 © 2011 IBM Corporation

10. And From My Travels...
 Next chart isn't from the experiment just described
– A real customer system
 A Group Buffer Pool
 ISC-Connected
– Necessary for the customer's estate
 Clearly something goes wrong at about 1100 requests / second
– Especially in response time terms but also CPU
• (Coupling Facility not CPU constrained)
 Options include
– Managing the request rate to below 1100 / sec
– Working on the request mix
– Infrastructure reconfiguration
10 © 2011 IBM Corporation

11. 25us?
11 © 2011 IBM Corporation

12. CPU / LPAR Match Up Between
70-1 and 74-4
12 © 2011 IBM Corporation

13. Internal Coupling Facility - Basics
●
Managed out of Pool 5 in modern processor families
●
Pool numbers given in SMF 70 as index into table of labels
●
Recommendation: Manage in reporting as a separate pool
●
Follow special CF sizing guidelines
●
Especially for takeover situations
●
Always runs at full speed
●
So good technology match for coupled z/OS images on same footprint
●
Another good reason to use ICFs is IC links
●
Shared ICFs strongly discouraged for Production
●
Especially if the CF image has Dynamic Dispatch turned on
●
Should not run ANY coupling facility above 50% busy
●
Especially if we need to be able to recover structures onto it
13 © 2011 IBM Corporation

14. ICF CPU Instrumentation
SMF 74-4 view different from SMF 70-1 LPAR view of processor busy
•R744PBSY is CPU time processing requests
•R744PWAI is CPU time while CFCC is not processing requests but it is still using CF
cycles
•For Dynamic Dispatch PWAI is time when not processing CF requests but Logical
CP not yet taken back by PR/SM
•For dedicated or non-Dynamic Dispatch cases sum is constant
•For Dynamic Dispatch sum can vary.
Number of defined processors is number of CF Processor Data sections in 74-4
•Refined for CFLEVEL 15 by new fields for dedicated (R744FPDN) and shared
(R744FPSN) processors
•Also whether individual engine is dedicated (R744PTYP) and its weight (R744PWGT)
PBSY and PWAI Can be examined down to Coupling Facility engine level
SMF 74-4 has much more besides CF CPU instrumentation
14 © 2011 IBM Corporation

15. CF LPAR Identification In SMF 70-1 Is Complex
 Need to match LPARs in SMF 70-1 with coupling
facilities in SMF 74-4 to get proper CPU picture
 Since z/OS Release 8 74-4 has machine serial number
– Allows correlation in most cases
– But LPAR names and CF Names often don't match
– Often multiple CF's in same footprint with similar
configuration
– Sometimes there are multiple CF's with the same name
– My code – in extremis – uses the presence of IC links to
determine “colocality”
– I'm slowly learning :-) not all CF LPARs are in Pool 5
15 © 2011 IBM Corporation

16. Additional Instrumentation - OA21140
 Almost all customers have this support
 Introduced to support zHPF
– Has other SMF and reporting improvements
• HiperDispatch Vertical Polarisation indicators at ENGINE level
– Type 70
• Normalisation factor for zIIP – Type 70
 Adds CF LPAR Partition Number
– Allows matching with SMF 70-1
 RMF Level (SMFxxSRL) changed to X'55'
16 © 2011 IBM Corporation

17. Structure Duplexing Performance
 Additional Traffic
– For lock structures duplexing generates double the traffic
– Otherwise only the writes are duplicated
– Additional CPU cost
 Additional Physical Resources
– A second coupling facility
• Documented in 74-4
– Additional memory – but “white space” rules say “not really”
– Additional links – to second coupling facility and between it and the primary
• Documented in SMF 74-4
17 © 2011 IBM Corporation

18. Structure Duplexing Performance - Response Times
 For System-Managed Duplexed structures both requests must complete
– Response time is that of the slowest
• So all requests are essentially with “remote” response times
• High likelihood of requests becoming asynchronous
• For low contention rates applications might experience longer lock acquisition times
 For User-Managed Duplexed structures both requests must complete
– But only for writes
– So writes performed with “remote” response times
– With high a read-to-write ratio request response times might not be significantly extended
– Main example: DB2 Group Buffer Pools
 Response time elongation measured by RMF PR WT and PR CMP times
– Former suggests better link infrastructure
– Latter suggests a more capable peer coupling facility
18 © 2011 IBM Corporation

19. 19 © 2011 IBM Corporation

20. 20 © 2011 IBM Corporation

21. 21 © 2011 IBM Corporation

22. Conclusings and Musions
 I think we've come a long way with Coupling Facility CPU
– Capacity Planning is now down to the structure level
• But not to the structure-by-system level
– We can now tie up the Coupling Facility and LPAR views of CPU
• With a few “corner cases”
 I'd encourage you to revisit your Parallel Sysplex reporting
– Including for all the other aspects we didn't have time for
 Structure Duplexing needs particular care
– A very useful resilience feature that has performance
considerations
22 © 2011 IBM Corporation