The document discusses several myths about system performance and CPU utilization. It debunks the myths that CPU requirements are proportional to the number of users, and that the CPU must be upgraded when utilization is above 60%. It explains that response time is determined by I/O service time, I/O queueing delay, CPU service time, and CPU queueing delay. A CPU upgrade can sometimes hurt response times by increasing I/O contention for I/O-intensive users. The system does not truly "lock up" when CPU utilization is 100%; a high priority runaway process can cause the perception of a lock up by starving other processes.

1. Supervisory System Performance Myths Paul Guerin CR&C Automation Group

2. Performance Myths References Manuals/books: OpenVMS system manuals. OpenVMS system and network node management course notes. “ Optimizing Oracle Performance” (publisher: O’Reilly; authors: Millsap & Holt). Internet whitepapers: Plant Solutions: Press Releases/Papers page (author: Paul Guerin) http://www.plantsolutions.com.au/ “ Performance Management Myths and Facts” (author: Cary Millsap) http://www.hotsos.com/e-library/abstract.php?id=24 Intranet whitepapers: Automation Practitioners CoP submissions (authors: Paul Guerin) http://cop.bluescopesteel.net/cophome.asp?copid=119

3. Performance Myths Agenda Resource utilisation Myth: CPU requirements are proportional to the number of system users. Myth: The CPU must be upgraded when the utilisation is >60% and the system is slow. Prioritisation and response time Myth: A CPU upgrade can’t hurt response times. Myth: The system “locks-up” when the CPU is 100% utilised. Performance tuning and prioritisation reduces interactive response times CPU upgrade versus performance tuning and prioritisation Performance tuning and prioritisation strategy

4. Performance Myths Resource utilisation

5. Performance Myths Resource utilisation of an engineer An engineer is just a resource that provides a service. An engineer can be fully utilised: If a job is scheduled to take 3 days, and is completed in 3 days (utilisation=100%). Delivery performance is highest when the engineering resource is 100% utilised. An engineer can be under-utilised: Not smart to under utilise: e.g. 5 days for a 3 day job (utilisation=3/5=60%). No more work is performed when more capacity is added (i.e overtime). e.g. 7 days for a 3 day job (utilisation=3/7=43%). Increasing the capacity of an engineer without increasing the workload leads to wastage.

6. Performance Myths Resource utilisation of a Central Processing Unit (CPU) The CPU is just a resource that provides a service. The CPU can be fully utilised: If work is performed by the CPU without idle time then utilisation = 100%. Throughput is highest when the CPU resource is 100% utilised. The CPU can be under-utilised: Not smart to leave the CPU idle – idle time can not be recovered. No more work is performed when the CPU capacity is increased. Increasing the capacity of a CPU without increasing the workload leads to wastage.

7. Performance Myths Myth: CPU requirements are proportional to the number of system users. Debunk – The fact is only total workloads determine CPU requirements….. The CPU only services workloads on behalf of users. Many users can perform small workloads, and a few users can perform large workloads. Only need to purchase a CPU that is capable of servicing all current and anticipated workloads. Workloads can be prioritised so the most important workloads are serviced first.

8. Performance Myths Myth: The CPU must be upgraded when the utilisation is >60% and the system is slow. Debunk 1 – The fact is at 60% utilisation the CPU is clearly underutilised….. The CPU is underutilised especially for batch processing where the goal is maximum throughput (CPU utilisation = 100%). For batch processing, 60% utilisation is not too high - its too low!!!! The CPU is idle for 40% of the time because it keeps running out of work. While 40% of the CPU service time is wasted there is no absolutely no basis for a CPU upgrade. A CPU upgrade will waste even more service time – so why do it???

9. Performance Myths Debunk 2 – The fact is overall system performance is irrelevant anyway….. Interactive response time is generally regarded as being more important than batch throughput. In particular the time critical actions. Need to bias response times towards interactive jobs – not batch jobs. Reduce response times for workloads that are assigned a higher priority.

10. Performance Myths Prioritisation and response time

11. Performance Myths Examples of services that allocate resources according to the highest priority: Paramedic in an emergency ward: Terminal injuries are always given the highest priority. e.g. excessive blood loss. Non-terminal injuries are treated later. Community bush fire brigade volunteer: Callouts are the highest priority. i.e. fighting fires. Previous activities may be resumed after the emergency is over. Automation engineer: Callouts are the highest priority. i.e. plant has stopped. Other activities may be resumed after the problem has been rectified. Central Processing Unit ….. The highest priority jobs must receive the available resources first.

12. Performance Myths Response time of an engineer An engineer can be fully utilised but still attend callouts without delay: Callouts are always given first priority, and are attended to above all other jobs regardless of the current workload. After the callout is resolved, the other jobs are resumed. The other jobs will finish later than they otherwise would (no problem as the other jobs are of a lower priority than a callout). A callout is always given the highest priority and the response time has nothing to do with the engineers workload. Regardless of capacity utilisation the engineer performs the highest priority job at the time.

13. Performance Myths Response time of a CPU A common guideline when assessing response time: higher CPU utilisation = slower response time of a process

14. Performance Myths Response time of a CPU A common guideline when assessing response time: Unfortunately, the CPU utilisation does not indicate I/O related time, nor CPU queue delay. As for engineering utilisation – CPU utilisation does not indicate response time. Intuitive, as disk utilisation does not indicate disk response time. CPU utilisation only indicates the amount of CPU capacity used – nothing more. higher CPU utilisation = slower response time of a process

15. Performance Myths Response time is actually comprised of the following criteria: I/O service time: Total time to perform physical reads and writes to disk. I/O queueing delay: Total time waiting to perform physical reads and writes to disk. CPU service time: Total time to service a process. e.g. data sorting, logical I/O. CPU queueing delay: Total time waiting for CPU service. Physical I/O (i.e. disk) is always slower than logical I/O (i.e. memory). Therefore disk related time is a significant response time component. Note: Other components exist from the user perspective. e.g. network delay, delays in the client workstation. For simplicity, examination of response time will be from the database perspective, and row locks will be ignored. response time = I/O service time + I/O queueing delay + CPU service time + CPU queueing delay

16. Performance Myths Myth: A CPU upgrade can’t hurt response times. Debunk – The fact is a faster CPU can introduce more I/O contention. Example - different types of users utilise a system: CPU intensive batch users (Note: batch jobs are run without delay). e.g. update of output coil details, coil compliance test, messaging, printing. I/O intensive interactive users (Note: interactive jobs are separated by ‘think’ time). If the CPU is upgraded, then the results of the upgrade will be as follows: Batch users complete faster, and issue I/O at a faster rate. The greater I/O rate creates more contention on the I/O device. What does greater I/O contention mean for the response time of interactive users?

17. Performance Myths I/O service time: same I/O requirement, so no change in total time. I/O queueing delay: increased due to higher I/O contention. CPU service time: reduced due to higher clock speed. CPU queueing delay: reduced due to less service time. The I/O related component is the most expensive for I/O intensive interactive users, so overall performance for this group is worse than before. There is an assumption that the lower the CPU utilisation, the higher the performance – but in fact, I/O intensive processing can be worse off. response time = I/O service time + I/O queueing delay + CPU service time + CPU queueing delay

18. Performance Myths A CPU upgrade will increase capacity, but not reduce disk related time. When you add capacity, only add it to the bottleneck device of the user you are targeting.

19. Performance Myths Myth: The system “locks up” when the CPU is 100% utilised. Debunk – In fact a process starved of service time will give the perception of a system “lock up”….. A high priority process (e.g. a runaway) will receive service time at the expense of lower priority processes. Some processes may be given higher base priorities than others. These processes may enter a runaway state at some point in the future. A runaway process will starve all processes of a lower base priority than it. e.g. telnet session. A lower priority telnet session will be starved of service time and give the false impression of a system “lockup”. Despite the CPU utilisation being at 100%, the operating system continues to function as per standard scheduling policies – the system has not “locked up”.

20. Performance Myths Now we know a runaway process with a higher base priority will starve all other processes of a lower base priority – resulting in huge response times (and a callout to the engineer). However when a runaway process is at the lowest priority, what happens to the response times of all other processes?

21. Performance Myths Example: 100% CPU utilisation on DCL3 supervisory - Springhill Observed the effect of a low priority runaway process on general response times. Note: A runaway process causes the CPU to be 100% utilised. I/O service time: unchanged – runaway didn’t produce more I/O contention. I/O queueing delay: unchanged – runaway didn’t produce more I/O contention. CPU service time: unchanged – despite overall utilisation at 100%. Note: The runaway was only serviced in otherwise idle time which drove the CPU utilisation to 100%. CPU queueing delay: unchanged – despite a slightly longer execution queue. Note: Effective queueing delay was unchanged because the runaway was only serviced when the queue was otherwise empty. response time = I/O service time + I/O queueing delay + CPU service time + CPU queueing delay

22. Performance Myths Despite the CPU being at 100% utilisation there was no performance degradation of the application. e.g. screens and programs were unaffected. A runaway process only receives idle time when it is assigned a lower priority than all others. In fact, the runaway process can be the same or lower than the other processes without application performance degradation. A critical factor for satisfactory response times is not the CPU utilisation – it’s the relative priority between processes. Regardless of capacity utilisation the CPU performs the highest priority job at the time.

23. Performance Myths Performance tuning and prioritisation reduces interactive response times

24. Performance Myths How can process prioritisation reduce response time? The CPU executes the process with the highest execution priority at the time. execution priority = base priority + boost priority due to a wait event. Processes with a higher base priority tend to have a higher execution priority. Therefore processes with higher base priorities receive shorter CPU queueing delay at the expense of other lower priority processes that must wait longer. The CPU simply services the highest priorities first, and the lowest priorities last.

25. Performance Myths Response times are easily reduced for time critical user actions by raising process priorities for those processes. time critical user actions receive less CPU queueing delay, leading to a better response time. non-time critical user actions receive more CPU queueing delay, leading to a worse response time (no problem as the response times of non-time critical user actions are of no concern). I/O service time: no change. I/O queueing delay: no change. CPU service time: no change. CPU queueing delay: eliminated for processes with the highest execution priority. Response times are better for higher priority jobs than lower priority jobs. response time = I/O service time + I/O queueing delay + CPU service time + CPU queueing delay

26. Performance Myths How can performance tuning reduce response time? Performance tuning is all about reducing the workload to achieve the same result. e.g. index access instead of full table scans. e.g. instead of retrieving all rows to view the first row – just retrieve the first row from the beginning. “ working smarter – not harder”

27. Performance Myths I/O service time: reduced due to less physical I/O. I/O queueing delay: reduced due to less physical I/O. CPU service time: reduced due to less logical I/O and implicit sorting. CPU queueing delay: reduced for all other processes due to less logical I/O and implicit sorting. Performance tuning is most effective in reducing response times because it reduces ALL the components: Aim to reduce logical I/O, reduced physical I/O will follow. Aim to reduce implicit data sorting, reduced logical I/O will follow. Less service time results in reduced queueing delay. response time = I/O service time + I/O queueing delay + CPU service time + CPU queueing delay

28. Performance Myths Prioritising workloads alone reduces response time through the CPU queueing delay component. Performance tuning alone is highly effective in reducing response times via all components. However the best results occur when performance tuning and prioritisation is combined.

29. Performance Myths CPU upgrade versus Performance tuning and prioritisation

30. Performance Myths Interactive response time due to a CPU upgrade I/O service time: unchanged. I/O queueing delay: increased due to higher I/O contention. CPU service time: reduced due to higher clock speed. CPU queueing delay: reduced due to less service time. response time = I/O service time + I/O queueing delay + CPU service time + CPU queueing delay

31. Performance Myths Interactive response time due to a CPU upgrade I/O service time: unchanged. I/O queueing delay: increased due to higher I/O contention. CPU service time: reduced due to higher clock speed. CPU queueing delay: reduced due to less service time. Interactive response time due to performance tuning and prioritisation I/O service time: reduced due to less physical I/O. I/O queueing delay: reduced due to less physical I/O. CPU service time: reduced due to less logical I/O and implicit sorting. CPU queueing delay: eliminated for the highest priority process, and reduced for all other processes due to less logical I/O and implicit sorting. response time = I/O service time + I/O queueing delay + CPU service time + CPU queueing delay

32. Performance Myths Performance tuning reduces all I/O and CPU related time. Bonus : The code review also presents the opportunity to correct poor workmanship resulting in: Less code and code complexity. Improved code readability. A CPU upgrade can only reduce CPU related time – often the smallest response time component. Prioritisation eliminates CPU queueing delay for the highest priority process. Where process priorities are kept equal, a CPU upgrade can only reduce CPU queueing delay. Performance tuning alone reduces queueing delay anyway. Performance tuning and prioritisation maximises performance.

33. Performance Myths Compared with performance tuning and prioritisation a hardware upgrade results in: Spending lots of money for mediocre performance that hides poor workmanship. Is this where we want to be?

34. Performance Myths Example: ETL3 supervisory – Packaging Products I/O service time, I/O queueing delay: N/A CPU service time: 140-700 times improvement from the most inefficient SQL statements. Instantly eliminated 15% of the total system workload by removing redundant executions of the dynamic coil-cut stored procedure. Bonus : Also 33% reduction of each execution. 50% reduction from the coil tracking stored procedure. Bonus : untangled “spaghetti” code resulting in less complexity and improved readability. 50% reduction from the schedule download stored procedure. 80% reduction (>0.5 second) from the housekeeping stored procedure by deleting 39 redundant lines of code. Bonus : less code complexity and improved readability. CPU queueing delay: Halved the average length of the compute bound queue to levels not seen for 5 years!!! Also achieved without reassigning process priorities. Option : increase the priority of the coil tracking stored procedure to eliminate its queueing delay. response time = I/O service time + I/O queueing delay + CPU service time + CPU queueing delay

35. Performance Myths Performance tuning and prioritisation strategy

36. Performance Myths 1) Pre-performance tune (database): Eliminate redundant workloads. e.g. redundant executions. Archive obsolete data to lessen the I/O requirement. 2) Performance tune (database): Reduce the I/O and sorting workload (especially for time-critical user actions): Remove implicit data sorting. e.g. replace UNION with UNION ALL. Create indexes to reduce full table scans. Gather Cost Based Optimiser (CBO) statistics so the least cost execution plan will be chosen. Balance the workload: Reschedule resource intensive activities to a less critical time. May need to restrict batch processing if it is degrading interactive processing. 3) Prioritisation (operating system): Higher priorities for critical user actions. e.g. weld/stitch tracking, etc. Lower priorities for non-critical user actions. e.g. coil compliance, FTP transfers, system administration, etc.

37. Performance Myths Prioritisation - assignment of base execution priority General rules: Interactive processing is I/O intensive and never dominates CPU service time. Batch processing is typically CPU intensive and may dominate CPU service time. Batch processing is an excellent candidate for a lower execution priority. Base priority (typically interactive processing) Semi-time critical user actions Non-time critical user actions telnet priority Time critical user actions (typically batch processing)

38. Performance Myths Key Points: CPU only services the highest priority at the time: Assign the time critical user actions the highest priority, and we do not care if the CPU is 10% or 100% utilised. Operating system vendors allow process prioritisation to be assigned easily. Response times for interactive processes are I/O intensive – not CPU intensive: Reducing response times for interactive processes are all about reducing logical I/O so less physical I/O will result. All database vendors offer tools that assist in performance monitoring and tuning. Performance tuning and prioritisation is fast, easy, and cheap