“Where did my CPU go?”        monitoring & capacity planning        adventures on a consolidated                environmen...
whoamiKarl Arao    •   Senior Technical Consultant @ Enkitec    •   Performance and Capacity Planning Enthusiast6 years 11...
Agenda• HOWTO compare CPU speeds• Cores vs Threads• The different CPU events• CPU Monitoring/Capacity Planning on consolid...
12:27:15 SYS@DEMO1> show parameter cpu_countNAME                                 TYPE        VALUE------------------------...
Exadata V2 => 2s8c16tCPU0   CPU8    CPU1   CPU9    CPU2     CPU10       CPU3   CPU11  Core0          Core1             Cor...
PART1: compare CPU speeds          www.enkitec.com   6
Different methods:• Published benchmarks  – TPC-C  – SPECint_rate2006• Actual Benchmarking  – cputoolkit  – SLOB (lio test...
TPC-C• Transaction Processing Performance Council (TPC)   • Throughput => transactions per minute (tpmC)   • Price/Perform...
SPECint_rate2006  • Standard Performance Evaluation Corporation (SPEC)     • SPECint_rate2006         • Integer performanc...
2007 vs 2012   www.enkitec.com   10
Actual Benchmarking• cputoolkit and SLOB (lio test)• LIOs/sec          cputoolkit                                         ...
V2 and X2 CPU perf comparison                    3.6M LIOs/sec            2.1M LIOs/sec            www.enkitec.com        ...
V2 -> X2 migration                                   V2                                           X2chip efficiency factor...
Core0         Core1               Core2        Core3                        Socket0 PART2: Cores vs ThreadsCPU1   CPU5   C...
www.enkitec.com   15
~30% depends on the workload   cputoolkit                                       SLOB                17%                   ...
PART3: Different CPU events            CPU         CPU Wait      CPU Scheduler           www.enkitec.com    17
AAS CPU www.enkitec.com   18
www.enkitec.com   19
CPU Wait www.enkitec.com   20
www.enkitec.com   21
CPU Scheduler   www.enkitec.com   25
www.enkitec.com   26
Putting it all togetherProblem: A single SQL Stmt. overwhelming      Instances Caged   SQL Applied to lockCPU resources.  ...
BEFORE:procs -----r5258505849                     load average: 10.36, 17.76, 36.42AFTER:procs ------r1313                ...
PART4: CPU monitoring and    Capacity Planning          www.enkitec.com   29
OS Tools• The usual Operating System commands   – vmstat   – top   – mpstat –P ALL 1 5• Cool tools   – collectl –sC (http:...
Load Map www.enkitec.com   31
Performance Page – Historical View              www.enkitec.com        32
AWR Toolkit• DIY performance data warehouse                         run_awr             Extract AWR data points as csv fil...
www.enkitec.com   34
• Tableau auto creates a time dimension for the time  column “MM/DD/YY HH24:MI:SS” of AWR csv output              www.enki...
• Summary and Underlying data       1-2AM       2-3AM                     www.enkitec.com   36
CPU usage across half rack Exadata              www.enkitec.com        37
CPU usage per host      www.enkitec.com   38
CPU redistribution across nodes            www.enkitec.com       39
Wrap up!• HOWTO compare CPU speeds   o SPECint_rate2006, TPC-C, Actual benchmarking• Cores vs Threads   o Always have HT o...
Resources•   cputoolkit - http://karlarao.wordpress.com/scripts-resources/•   AWR Tableau and R toolkit Visualization Exam...
Fastest Growing Companies                       in Dallas42
Rmoug13 - where did my CPU go?
Rmoug13 - where did my CPU go?
Rmoug13 - where did my CPU go?
Upcoming SlideShare
Loading in...5
×

Rmoug13 - where did my CPU go?

181

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
181
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Rmoug13 - where did my CPU go?

  1. 1. “Where did my CPU go?” monitoring & capacity planning adventures on a consolidated environmentPresented by:Karl Arao 1
  2. 2. whoamiKarl Arao • Senior Technical Consultant @ Enkitec • Performance and Capacity Planning Enthusiast6 years 11 months 12 days DBA experience Oracle ACE, OCP-DBA, RHCE, OakTable Blog: karlarao.wordpress.com Wiki: karlarao.tiddlyspot.com Twitter: @karlarao www.enkitec.com 2
  3. 3. Agenda• HOWTO compare CPU speeds• Cores vs Threads• The different CPU events• CPU Monitoring/Capacity Planning on consolidated environments www.enkitec.com 3
  4. 4. 12:27:15 SYS@DEMO1> show parameter cpu_countNAME TYPE VALUE------------------------------------ ----------- --------cpu_count integer 16 www.enkitec.com 4
  5. 5. Exadata V2 => 2s8c16tCPU0 CPU8 CPU1 CPU9 CPU2 CPU10 CPU3 CPU11 Core0 Core1 Core2 Core3 Socket0CPU4 CPU12 CPU5 CPU13 CPU6 CPU14 CPU7 CPU15 Core0 Core1 Core2 Core3 Socket1 www.enkitec.com 5
  6. 6. PART1: compare CPU speeds www.enkitec.com 6
  7. 7. Different methods:• Published benchmarks – TPC-C – SPECint_rate2006• Actual Benchmarking – cputoolkit – SLOB (lio test) www.enkitec.com 7
  8. 8. TPC-C• Transaction Processing Performance Council (TPC) • Throughput => transactions per minute (tpmC) • Price/Performance => USD / tpmC• CPU performance => tpmC / core • 1609186.39 / 16 = 100574 www.enkitec.com 8
  9. 9. SPECint_rate2006 • Standard Performance Evaluation Corporation (SPEC) • SPECint_rate2006 • Integer performance • All CPUs are used • Used by OEM12c Consolidation Planner (SYSMAN.EMCT_SPEC_RATE_LIB) • CPU performance => SPECint_rate2006/core • 702/16 = 43.875$ cat spec.txt | grep -i sun | grep -i x3-2 | sort -rnk144.0625, 16, 2, 8, 2, 632, 705, Oracle Corporation, Sun Blade X3-2B (Intel Xeon E5-2690 2.9GHz)44.0625, 16, 2, 8, 2, 630, 705, Oracle Corporation, Sun Server X3-2L (Intel Xeon E5-2690 2.9GHz)43.875, 16, 2, 8, 2, 628, 702, Oracle Corporation, Sun Server X3-2 (Intel Xeon E5-2690 2.9GHz) www.enkitec.com 9
  10. 10. 2007 vs 2012 www.enkitec.com 10
  11. 11. Actual Benchmarking• cputoolkit and SLOB (lio test)• LIOs/sec cputoolkit SLOB ./runcputoolkit-auto <start CPU> <end CPU> <db name> ./runit.sh <writers> <readers> ./runcputoolkit-auto 1 2 dw while :; do ./runit.sh 0 2; done CPU1 CPU1 CPU2 CPU2 CPU3 CPU3 CPU4 Both at 25% CPU4 CPU utilization CPU5 CPU5 CPU6 CPU6 CPU7 CPU7 CPU8 CPU8 www.enkitec.com 11
  12. 12. V2 and X2 CPU perf comparison 3.6M LIOs/sec 2.1M LIOs/sec www.enkitec.com 12
  13. 13. V2 -> X2 migration V2 X2chip efficiency factor = (source LIOs/sec) / (destination LIOs/sec) = 2.1M / 3.6M = .5833X2 CPU requirement = source host CPUs * utilization * chip efficiency factor = 16 * .46 = 7.36 * .5833 = 4.29 CPUsX2 CPU Utilization = CPU requirement / CPU capacity = 4.29 / 24 = 17.8 % www.enkitec.com 13
  14. 14. Core0 Core1 Core2 Core3 Socket0 PART2: Cores vs ThreadsCPU1 CPU5 CPU2 CPU6 CPU3 CPU7 CPU4 CPU8 Core0 Core1 Core2 Core3 Socket0 www.enkitec.com 14
  15. 15. www.enkitec.com 15
  16. 16. ~30% depends on the workload cputoolkit SLOB 17% 21% Intel HT Technology Technical Users Guide http://goo.gl/3Ec5Z www.enkitec.com 16
  17. 17. PART3: Different CPU events CPU CPU Wait CPU Scheduler www.enkitec.com 17
  18. 18. AAS CPU www.enkitec.com 18
  19. 19. www.enkitec.com 19
  20. 20. CPU Wait www.enkitec.com 20
  21. 21. www.enkitec.com 21
  22. 22. CPU Scheduler www.enkitec.com 25
  23. 23. www.enkitec.com 26
  24. 24. Putting it all togetherProblem: A single SQL Stmt. overwhelming Instances Caged SQL Applied to lockCPU resources. at 12 CPU’s ea. in good plan. www.enkitec.com 27
  25. 25. BEFORE:procs -----r5258505849 load average: 10.36, 17.76, 36.42AFTER:procs ------r1313 After Before121214 www.enkitec.com 28
  26. 26. PART4: CPU monitoring and Capacity Planning www.enkitec.com 29
  27. 27. OS Tools• The usual Operating System commands – vmstat – top – mpstat –P ALL 1 5• Cool tools – collectl –sC (http://collectl.sourceforge.net) – turbostat.c – dcli (Exadata) • dcli -l oracle -g /home/oracle/dbs_group --vmstat 2 • dcli -l oracle -g /home/oracle/dbs_group uptime www.enkitec.com 30
  28. 28. Load Map www.enkitec.com 31
  29. 29. Performance Page – Historical View www.enkitec.com 32
  30. 30. AWR Toolkit• DIY performance data warehouse run_awr Extract AWR data points as csv files1 Customer site run_extract Package all the csv files FRESH_LOAD Create new client “dimension” tables2 DIY DW server CHECK_LOAD Check new data points DELTA_LOAD Load new data points awr_topevents_(ClientNameX) awr_topevents_(ClientNameY) awr_topevents_(ClientNameZ) awr_cpuwl_(ClientNameX) awr_cpuwl_(ClientNameY) awr_cpuwl_(ClientNameZ) awr_iowl_(ClientNameX) awr_iowl_(ClientNameY) awr_iowl_(ClientNameZ)3 Tableau Analytics www.enkitec.com 33
  31. 31. www.enkitec.com 34
  32. 32. • Tableau auto creates a time dimension for the time column “MM/DD/YY HH24:MI:SS” of AWR csv output www.enkitec.com 35
  33. 33. • Summary and Underlying data 1-2AM 2-3AM www.enkitec.com 36
  34. 34. CPU usage across half rack Exadata www.enkitec.com 37
  35. 35. CPU usage per host www.enkitec.com 38
  36. 36. CPU redistribution across nodes www.enkitec.com 39
  37. 37. Wrap up!• HOWTO compare CPU speeds o SPECint_rate2006, TPC-C, Actual benchmarking• Cores vs Threads o Always have HT on o ~30% performance benefit after core count• The different CPU events o 1 AAS CPU = 1 CPU core o Oracle CPU may not correlate with Host CPU if you have a lot of CPU activity outside of the database• CPU Monitoring/Capacity Planning on consolidated environments o AWR analytics www.enkitec.com 40
  38. 38. Resources• cputoolkit - http://karlarao.wordpress.com/scripts-resources/• AWR Tableau and R toolkit Visualization Examples - http://goo.gl/xZHHY• AAS investigation - http://goo.gl/5WaAg• Cores vs Threads - http://goo.gl/1MLFf• Turbostat.c - http://goo.gl/jDUKg• cpu_topology - http://goo.gl/EUDG7• CPU centric benchmark comparisons - http://goo.gl/nR9Yy• SLOB - http://goo.gl/yKa45• Kyle Hailey - http://dboptimizer.com/2011/07/21/oracle-cpu-time/• The mindmap of this presentation - http://goo.gl/XeY0e karl.arao@enkitec.com www.enkitec.com 41
  39. 39. Fastest Growing Companies in Dallas42

×