Exadata Consolidation Success Story                 Getting the kids to play nice with each other…Presented by:Karl Arao  ...
whoamiKarl Arao    •   Senior Technical Consultant @ Enkitec    •   Performance and Capacity Planning Enthusiast6 years 11...
Agenda• Architecture• Tools and Methodology• War Stories                  www.enkitec.com   3
General Architecture             Primary Site                     Standby SiteProduction                                  ...
General Architecture       www.enkitec.com   5
The StatsThree Half Rack Exadata clusters with High Cap. drivesCluster #1   36 Dev/Test DatabasesCluster #2   11 Productio...
Why Consolidate?Primary drivers for consolidation center around cost savings• Reduces Oracle software licensing• 3rd party...
A Simple Consolidation Example                7 Databases            www.enkitec.com      8
A Simple Consolidation Example     Let’s say we have the following databases to migrate on Exadata:                       ...
A Simple Consolidation Example     Let’s say we have the following databases to migrate on Exadata:                       ...
A Simple Consolidation Example                            25%           42%   33%      17%Cluster Level                   ...
A Simple Consolidation Example                             8%           83%   17%        8%Cluster Level                  ...
Tools And Methodology•   Gather Utilization Metrics (usage history)•   Create Provisioning Plan•   Implement Plan•   Audit...
www.enkitec.com   14
Provisioning Worksheet• Capacity Planning                           U =R  tilization              equirements   /C   apaci...
Capacity2 = quarter rack                       CPU_COUNT,                                                                 ...
CPU Core Comparison                                                                                  Sun Fire X4170 M2 X56...
The Perfect Storm     (Peoplesoft HR)                Month-end Processing                   + Weekly Time Entry           ...
CPU Allocation                               node                 node        node               nodeDB Uniq Name   DB Nam...
Load Map                    (our first stop…)Users Complaint: HR time entry and OBIEE reports painfully slow…             ...
Top Activity - HCMPRD       www.enkitec.com   21
Instance Activity – HCMPRD2                                             Node 2Problem: A single SQL stmt. overwhelming    ...
Memory Exhaustion              (OBIEE)  “1 Report = 1 SQL query, right?”             WRONG!            www.enkitec.com    ...
Node 1         Overlapping workloads of three databases         across 3 nodes.Node 2         BIPRD, HCMPRD, and MTAPRDNod...
Node Layout Revisited…        www.enkitec.com   25
Notice what happens to CPU waitsand the system load average whenthis report is run.           www.enkitec.com         26
PGA Memory Spikeswww.enkitec.com                       27
www.enkitec.com   28
www.enkitec.com   29
Storage Cell Saturation           (OBIEE)         www.enkitec.com   30
www.enkitec.com   31
I/O Intensive Workload         www.enkitec.com   32
Smart Scans as seen in Grid Control              www.enkitec.com         33
25 Sessions Doing Smart Scans…as seen in gv$sql                       www.enkitec.com     34
www.enkitec.com   35
Smart Scan in Action. The cells are scanning 1T but only returning 144G…***That’s on each of the highlighted row source be...
The databases on other nodes see the contention as “System I/O”Without I/O resource management even critical processes are...
Inter-database IORM Plan                         (only kicks in when needed)I/O requests from critical processes like CKPT...
IORM Plan Definition   (on each storage cell)         www.enkitec.com    39
Wrap up!Provisioning Methodology & Tools  – Utilization metrics and requirements  – Provisioning SpreadsheetSuccess Storie...
Q    &        A            www.enkitec.com   41
Contact Info…karl.arao@enkitec.com                   Fastest Growing Companies                                     in Dall...
Upcoming SlideShare
Loading in …5
×

A Consolidation Success Story

501 views
385 views

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
501
On SlideShare
0
From Embeds
0
Number of Embeds
49
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

A Consolidation Success Story

  1. 1. Exadata Consolidation Success Story Getting the kids to play nice with each other…Presented by:Karl Arao 1
  2. 2. whoamiKarl Arao • Senior Technical Consultant @ Enkitec • Performance and Capacity Planning Enthusiast6 years 11 months 12 days DBA experience Oracle ACE, OCP-DBA, RHCE, OakTable Blog: karlarao.wordpress.com Wiki: karlarao.tiddlyspot.com Twitter: @karlarao www.enkitec.com 2
  3. 3. Agenda• Architecture• Tools and Methodology• War Stories www.enkitec.com 3
  4. 4. General Architecture Primary Site Standby SiteProduction Future GrowthTest & Dev Disaster Recovery www.enkitec.com 4
  5. 5. General Architecture www.enkitec.com 5
  6. 6. The StatsThree Half Rack Exadata clusters with High Cap. drivesCluster #1 36 Dev/Test DatabasesCluster #2 11 Production DatabasesCluster #3 13 Dev/Test Databases 6 Standby DatabasesStill more databases to come… www.enkitec.com 6
  7. 7. Why Consolidate?Primary drivers for consolidation center around cost savings• Reduces Oracle software licensing• 3rd party products such as backup agents, ETL tools, etc…• More efficient use of system resources• Soft Costs – Floor space – Power & Cooling – Administration, Staffing Costs (training, etc.) www.enkitec.com 7
  8. 8. A Simple Consolidation Example 7 Databases www.enkitec.com 8
  9. 9. A Simple Consolidation Example Let’s say we have the following databases to migrate on Exadata: Cluster Level UtilizationFor example, the first row should read… Database ‘A’ requires 4 CPU’s and will run on nodes 1 and 2 (2 CPU’s each) www.enkitec.com 9
  10. 10. A Simple Consolidation Example Let’s say we have the following databases to migrate on Exadata: Per compute node UtilizationFor example, the first row should read… Database ‘A’ requires 4 CPU’s and will run on nodes 1 and 2 (2 CPU’s each) www.enkitec.com 10
  11. 11. A Simple Consolidation Example 25% 42% 33% 17%Cluster Level Per compute node UtilizationUtilization = 29.2% www.enkitec.com 11
  12. 12. A Simple Consolidation Example 8% 83% 17% 8%Cluster Level Per compute node UtilizationUtilization = 29.2% www.enkitec.com 12
  13. 13. Tools And Methodology• Gather Utilization Metrics (usage history)• Create Provisioning Plan• Implement Plan• Audit Your Implementation www.enkitec.com 13
  14. 14. www.enkitec.com 14
  15. 15. Provisioning Worksheet• Capacity Planning U =R tilization equirements /C apacity• Communication Tool **Supplement to existing Exadata installation tools: • Site planning checklist • Configuration Worksheet • Exadata Configurator sheet • CheckIP • OneCommand• Hand off www.enkitec.com 15
  16. 16. Capacity2 = quarter rack CPU_COUNT, Space will also depend on: • ASM redundancy4 = half rack threads, & cores • DATA/RECO allocation http://goo.gl/CunHN8 = full rack http://goo.gl/I3fjn 96 to 144GB Query Low (4x) (frequency of the SPECint_rate2006 Query High (6x) memory DIMMs http://goo.gl/doBI5 Archive Low (7x) drops to 800 MHz Archive High (12x) from 1333 MHz) www.enkitec.com 16
  17. 17. CPU Core Comparison Sun Fire X4170 M2 X5670@2.93GHz Source Destinationchip efficiency factor = source SPEC rating / Exadata SPEC rating = 16/26 how much of the multiplier for amount of CPU equivalent resources that will = .6154 source CPU cores are being used database be offloaded to the machine cores storage cellsEXA cores requirement = source host cores * utilization * chip efficiency factor * offload factor = 32 * .7 * .6154 * .5 = --------- 6.89 13.78 www.enkitec.com 17
  18. 18. The Perfect Storm (Peoplesoft HR) Month-end Processing + Weekly Time Entry + SQL Plan Change ------------------------------------ Uh-oh! www.enkitec.com 18
  19. 19. CPU Allocation node node node nodeDB Uniq Name DB Name 1 2 3 4 4 instance 5 instance 4 instance 3 instance 47% cpu used 75% cpu used 47% cpu used 18% cpu used 49% mem used 66% mem used 71% mem used 54% mem usedBIPRDDAL biprd P PDBFSPRD DBFSPRD P P P PHCMPRDDAL hcmprd P PMTAPRD11DAL mtaprd11 P PPAPRDDAL paprd P PRMPRDDAL rmprd P Pdbm dbm F F F FFsprddal fsprd P P = Preferred = Failover www.enkitec.com 19
  20. 20. Load Map (our first stop…)Users Complaint: HR time entry and OBIEE reports painfully slow… www.enkitec.com 20
  21. 21. Top Activity - HCMPRD www.enkitec.com 21
  22. 22. Instance Activity – HCMPRD2 Node 2Problem: A single SQL stmt. overwhelming HCMPRD Caged SQL Profile InstalledCPU resources. at 12 CPU’s to lock in good plan. www.enkitec.com 22
  23. 23. Memory Exhaustion (OBIEE) “1 Report = 1 SQL query, right?” WRONG! www.enkitec.com 23
  24. 24. Node 1 Overlapping workloads of three databases across 3 nodes.Node 2 BIPRD, HCMPRD, and MTAPRDNode 3Node 4 www.enkitec.com 24
  25. 25. Node Layout Revisited… www.enkitec.com 25
  26. 26. Notice what happens to CPU waitsand the system load average whenthis report is run. www.enkitec.com 26
  27. 27. PGA Memory Spikeswww.enkitec.com 27
  28. 28. www.enkitec.com 28
  29. 29. www.enkitec.com 29
  30. 30. Storage Cell Saturation (OBIEE) www.enkitec.com 30
  31. 31. www.enkitec.com 31
  32. 32. I/O Intensive Workload www.enkitec.com 32
  33. 33. Smart Scans as seen in Grid Control www.enkitec.com 33
  34. 34. 25 Sessions Doing Smart Scans…as seen in gv$sql www.enkitec.com 34
  35. 35. www.enkitec.com 35
  36. 36. Smart Scan in Action. The cells are scanning 1T but only returning 144G…***That’s on each of the highlighted row source below… www.enkitec.com 36
  37. 37. The databases on other nodes see the contention as “System I/O”Without I/O resource management even critical processes are affected (CKPT, LGWR, …) www.enkitec.com 37
  38. 38. Inter-database IORM Plan (only kicks in when needed)I/O requests from critical processes like CKPT, LGWR, LMON get priority automatically.Without IORM I/O requests from these important processes receive the same priorityas any other process.*Side Benefit (automatic when IORM is enabled) www.enkitec.com 38
  39. 39. IORM Plan Definition (on each storage cell) www.enkitec.com 39
  40. 40. Wrap up!Provisioning Methodology & Tools – Utilization metrics and requirements – Provisioning SpreadsheetSuccess Stories – CPU resource management – Tuning and provisioning adjustments – I/O resource management www.enkitec.com 40
  41. 41. Q & A www.enkitec.com 41
  42. 42. Contact Info…karl.arao@enkitec.com Fastest Growing Companies in Dallas 42

×