Exadata Consolidation Success Story
Getting the kids to play nice with each other…
1
Presented	
  by:	
  	
  
Karl	
  Arao...
whoami	
  
Karl	
  Arao	
  
•  Senior	
  Technical	
  Consultant	
  @	
  Enkitec	
  
•  Performance	
  and	
  Capacity	
  ...
www.enkitec.com	
   3	
  
100+
3
Agenda	
  
•  Architecture	
  
•  Tools	
  and	
  Methodology	
  
•  War	
  Stories	
  
www.enkitec.com	
   4	
  
General	
  Architecture	
  
www.enkitec.com	
   5	
  
Primary	
  Site	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	...
General	
  Architecture	
  
www.enkitec.com	
   6	
  
The	
  Stats	
  
Three	
  Half	
  Rack	
  Exadata	
  clusters	
  with	
  High	
  Cap.	
  drives	
  
Cluster	
  #1	
  
36	
...
Why	
  Consolidate?	
  
Primary	
  drivers	
  for	
  consolidaKon	
  center	
  around	
  cost	
  savings	
  
• Reduces	
  ...
www.enkitec.com	
   9	
  
7	
  Databases	
  
A	
  Simple	
  ConsolidaKon	
  Example	
  
www.enkitec.com	
   10	
  
For	
  example,	
  the	
  first	
  row	
  should	
  read…	
  
	
  	
  	
  	
  	
  Database	
  ‘A...
www.enkitec.com	
   11	
  
Let’s say we have the following databases to migrate on Exadata:	
  
Per	
  compute	
  node	
  ...
www.enkitec.com	
   12	
  
Cluster	
  Level	
  	
  
UKlizaKon	
  =	
  29.2%	
  
Per	
  compute	
  node	
  UKlizaKon	
  
	
...
www.enkitec.com	
   13	
  
Cluster	
  Level	
  	
  
UKlizaKon	
  =	
  29.2%	
  
Per	
  compute	
  node	
  UKlizaKon	
  
	
...
www.enkitec.com	
   14	
  
•  Gather	
  UKlizaKon	
  Metrics	
  (usage	
  history)	
  
•  Create	
  Provisioning	
  Plan	
...
www.enkitec.com	
   15	
  
Provisioning	
  Worksheet	
  
•  Capacity	
  Planning	
  
	
  
•  CommunicaKon	
  Tool	
  
	
  
•  Hand	
  off	
  
www.enki...
Capacity	
  
www.enkitec.com	
   17	
  
2	
  =	
  quarter	
  rack	
  
4	
  =	
  half	
  rack	
  
8	
  =	
  full	
  rack	
 ...
CPU	
  Core	
  Comparison	
  
www.enkitec.com	
   18	
  
Source	
  
chip	
  efficiency	
  factor	
  	
  =	
  source	
  SPEC	...
The	
  Perfect	
  Storm	
  
(Peopleso=	
  HR)	
  
	
  
www.enkitec.com	
   19	
  
Month-­‐end	
  Processing	
  
+	
  Weekl...
CPU	
  AllocaKon	
  
www.enkitec.com	
   20	
  
DB	
  Uniq	
  Name	
   DB	
  Name	
  
node	
  
1	
  
node	
  
2	
  
node	
...
www.enkitec.com	
   21	
  
Load	
  Map	
  
(our	
  first	
  stop…)	
  
Users	
  Complaint:	
  HR	
  Kme	
  entry	
  and	
  ...
www.enkitec.com	
   22	
  
Top	
  AcKvity	
  -­‐	
  HCMPRD	
  
www.enkitec.com	
   23	
  
Instance	
  AcKvity	
  –	
  HCMPRD2	
  
HCMPRD	
  Caged	
  
at	
  12	
  CPU’s	
  
SQL	
  Profile...
Memory	
  ExhausKon	
  
(OBIEE)	
  
	
  
	
  
“1	
  Report	
  =	
  1	
  SQL	
  query,	
  right?”	
  
	
  
WRONG!	
  
www.e...
www.enkitec.com	
   25	
  
Overlapping workloads of three databases
across 3 nodes.
BIPRD, HCMPRD, and MTAPRD
Node 1
Node ...
www.enkitec.com	
   26	
  
Node	
  Layout	
  Revisited…	
  
www.enkitec.com	
   27	
  
Notice what happens to CPU waits
and the system load average when
this report is run.
www.enkitec.com	
   28	
  
PGA Memory Spikes
www.enkitec.com	
   29	
  
www.enkitec.com	
   30	
  
Storage	
  Cell	
  SaturaKon	
  
(OBIEE)	
  
www.enkitec.com	
   31	
  
www.enkitec.com	
   32	
  
www.enkitec.com	
   33	
  
I/O	
  Intensive	
  Workload	
  
www.enkitec.com	
   34	
  
Smart	
  Scans	
  as	
  seen	
  in	
  Grid	
  Control	
  
www.enkitec.com	
   35	
  
25	
  Sessions	
  Doing	
  Smart	
  Scans	
  
…as	
  seen	
  in	
  gv$sql	
  
www.enkitec.com	
   36	
  
www.enkitec.com	
   37	
  
Smart	
  Scan	
  in	
  AcKon.	
  The	
  cells	
  are	
  scanning	
  1T	
  but	
  only	
  return...
www.enkitec.com	
   38	
  
The	
  databases	
  on	
  other	
  nodes	
  see	
  the	
  contenKon	
  as	
  “System	
  I/O”	
 ...
www.enkitec.com	
   39	
  
Inter-­‐database	
  IORM	
  Plan	
  
(only	
  kicks	
  in	
  when	
  needed)	
  
I/O	
  request...
www.enkitec.com	
   40	
  
IORM	
  Plan	
  DefiniKon	
  
(on	
  each	
  storage	
  cell)	
  
Wrap	
  up!	
  
Provisioning	
  Methodology	
  &	
  Tools	
  
– UKlizaKon	
  metrics	
  and	
  requirements	
  
– Provisio...
www.enkitec.com	
   42	
  
Q
&
A
43
Fastest Growing Companies
in Dallas
Contact	
  Info…	
  
karl.arao@enkitec.com
Upcoming SlideShare
Loading in...5
×

A Consolidation Success Story by Karl Arao

590
-1

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
590
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

A Consolidation Success Story by Karl Arao

  1. 1. Exadata Consolidation Success Story Getting the kids to play nice with each other… 1 Presented  by:     Karl  Arao  
  2. 2. whoami   Karl  Arao   •  Senior  Technical  Consultant  @  Enkitec   •  Performance  and  Capacity  Planning  Enthusiast       7  years  DBA  experience   Oracle  ACE,  OCP-­‐DBA,  RHCE,  OakTable   Blog:  karlarao.wordpress.com   Wiki:  karlarao.Kddlyspot.com   TwiLer:  @karlarao     www.enkitec.com   2  
  3. 3. www.enkitec.com   3   100+ 3
  4. 4. Agenda   •  Architecture   •  Tools  and  Methodology   •  War  Stories   www.enkitec.com   4  
  5. 5. General  Architecture   www.enkitec.com   5   Primary  Site                                                      Standby  Site   ProducKon   Test  &  Dev   Disaster  Recovery   Future  Growth  
  6. 6. General  Architecture   www.enkitec.com   6  
  7. 7. The  Stats   Three  Half  Rack  Exadata  clusters  with  High  Cap.  drives   Cluster  #1   36  Dev/Test  Databases     Cluster  #2   11  ProducKon  Databases   Cluster  #3   13  Dev/Test  Databases   6  Standby  Databases   SKll  more  databases  to  come…   www.enkitec.com   7  
  8. 8. Why  Consolidate?   Primary  drivers  for  consolidaKon  center  around  cost  savings   • Reduces  Oracle  soware  licensing   • 3rd  party  products  such  as  backup  agents,  ETL  tools,  etc…   • More  efficient  use  of  system  resources   • So  Costs   –  Floor  space   –  Power  &  Cooling   –  AdministraKon,  Staffing  Costs   (training,  etc.)   www.enkitec.com   8  
  9. 9. www.enkitec.com   9   7  Databases   A  Simple  ConsolidaKon  Example  
  10. 10. www.enkitec.com   10   For  example,  the  first  row  should  read…            Database  ‘A’  requires  4  CPU’s  and  will  run  on  nodes  1  and  2  (2  CPU’s  each)   Let’s say we have the following databases to migrate on Exadata:   Cluster  Level     UKlizaKon   A  Simple  ConsolidaKon  Example  
  11. 11. www.enkitec.com   11   Let’s say we have the following databases to migrate on Exadata:   Per  compute  node     UKlizaKon   For  example,  the  first  row  should  read…            Database  ‘A’  requires  4  CPU’s  and  will  run  on  nodes  1  and  2  (2  CPU’s  each)   A  Simple  ConsolidaKon  Example  
  12. 12. www.enkitec.com   12   Cluster  Level     UKlizaKon  =  29.2%   Per  compute  node  UKlizaKon      25%            42%            33%          17%   A  Simple  ConsolidaKon  Example  
  13. 13. www.enkitec.com   13   Cluster  Level     UKlizaKon  =  29.2%   Per  compute  node  UKlizaKon        8%              83%            17%              8%   A  Simple  ConsolidaKon  Example  
  14. 14. www.enkitec.com   14   •  Gather  UKlizaKon  Metrics  (usage  history)   •  Create  Provisioning  Plan   •  Implement  Plan   •  Audit  Your  ImplementaKon   Tools  And  Methodology  
  15. 15. www.enkitec.com   15  
  16. 16. Provisioning  Worksheet   •  Capacity  Planning     •  CommunicaKon  Tool     •  Hand  off   www.enkitec.com   16   **Supplement  to  exisKng  Exadata  installaKon  tools:   •  Site  planning  checklist   •  ConfiguraKon  Worksheet   •  Exadata  Configurator  sheet   •  CheckIP   •  OneCommand   UKlizaKon  =  Requirements  /  Capacity  
  17. 17. Capacity   www.enkitec.com   17   2  =  quarter  rack   4  =  half  rack   8  =  full  rack   SPECint_rate2006   hLp://goo.gl/doBI5   CPU_COUNT,   threads,  &  cores   hLp://goo.gl/CunHN   96  to  144GB   (frequency  of  the   memory  DIMMs   drops  to  800  MHz   from  1333  MHz)   Space  will  also  depend  on:   •  ASM  redundancy   •  DATA/RECO  allocaKon   hLp://goo.gl/I3pn   Query  Low  (4x)   Query  High  (6x)   Archive  Low  (7x)   Archive  High  (12x)  
  18. 18. CPU  Core  Comparison   www.enkitec.com   18   Source   chip  efficiency  factor    =  source  SPEC  raKng  /  Exadata  SPEC  raKng                                                                                =  16/26                                                                                =  .6154                                     EXA  cores  requirement      =  source  host  cores  *  uKlizaKon  *  chip  efficiency  factor                                                                                        =  32  *  .7  *  .6154                                                                                        =  13.78                                                                                                                                                                                                        *  offload  factor                                                                *  .5        -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐    6.89   Sun  Fire  X4170  M2  X5670@2.93GHz   DesKnaKon   how  much  of  the   source  CPU  cores   are  being  used   mulKplier  for   equivalent   database   machine  cores   amount  of  CPU   resources  that  will   be  offloaded  to  the   storage  cells  
  19. 19. The  Perfect  Storm   (Peopleso=  HR)     www.enkitec.com   19   Month-­‐end  Processing   +  Weekly  Time  Entry   +  SQL  Plan  Change   -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐   Uh-­‐oh!  
  20. 20. CPU  AllocaKon   www.enkitec.com   20   DB  Uniq  Name   DB  Name   node   1   node   2   node   3   node   4   4 instance 5 instance 4 instance 3 instance 47% cpu used 75% cpu used 47% cpu used 18% cpu used 49% mem used 66% mem used 71% mem used 54% mem used BIPRDDAL biprd P P DBFSPRD DBFSPRD P P P P HCMPRDDAL hcmprd P P MTAPRD11DAL mtaprd11 P P PAPRDDAL paprd P P RMPRDDAL rmprd P P dbm dbm F F F F Fsprddal fsprd P P = Preferred = Failover
  21. 21. www.enkitec.com   21   Load  Map   (our  first  stop…)   Users  Complaint:  HR  Kme  entry  and  OBIEE  reports  painfully  slow…  
  22. 22. www.enkitec.com   22   Top  AcKvity  -­‐  HCMPRD  
  23. 23. www.enkitec.com   23   Instance  AcKvity  –  HCMPRD2   HCMPRD  Caged   at  12  CPU’s   SQL  Profile  Installed   to  lock  in  good  plan.   Problem:  A  single  SQL  stmt.  overwhelming   CPU  resources.   Node 2
  24. 24. Memory  ExhausKon   (OBIEE)       “1  Report  =  1  SQL  query,  right?”     WRONG!   www.enkitec.com   24  
  25. 25. www.enkitec.com   25   Overlapping workloads of three databases across 3 nodes. BIPRD, HCMPRD, and MTAPRD Node 1 Node 2 Node 3 Node 4
  26. 26. www.enkitec.com   26   Node  Layout  Revisited…  
  27. 27. www.enkitec.com   27   Notice what happens to CPU waits and the system load average when this report is run.
  28. 28. www.enkitec.com   28   PGA Memory Spikes
  29. 29. www.enkitec.com   29  
  30. 30. www.enkitec.com   30  
  31. 31. Storage  Cell  SaturaKon   (OBIEE)   www.enkitec.com   31  
  32. 32. www.enkitec.com   32  
  33. 33. www.enkitec.com   33   I/O  Intensive  Workload  
  34. 34. www.enkitec.com   34   Smart  Scans  as  seen  in  Grid  Control  
  35. 35. www.enkitec.com   35   25  Sessions  Doing  Smart  Scans   …as  seen  in  gv$sql  
  36. 36. www.enkitec.com   36  
  37. 37. www.enkitec.com   37   Smart  Scan  in  AcKon.  The  cells  are  scanning  1T  but  only  returning  144G…   ***That’s  on  each  of  the  highlighted  row  source  below…  
  38. 38. www.enkitec.com   38   The  databases  on  other  nodes  see  the  contenKon  as  “System  I/O”   Without  I/O  resource  management  even  criKcal  processes  are  affected  (CKPT,  LGWR,  …)  
  39. 39. www.enkitec.com   39   Inter-­‐database  IORM  Plan   (only  kicks  in  when  needed)   I/O  requests  from  criKcal  processes  like  CKPT,  LGWR,  LMON  get  priority  automaKcally.   Without  IORM  I/O  requests  from  these  important  processes  receive  the  same  priority     as  any  other  process.     *Side Benefit (automatic when IORM is enabled)
  40. 40. www.enkitec.com   40   IORM  Plan  DefiniKon   (on  each  storage  cell)  
  41. 41. Wrap  up!   Provisioning  Methodology  &  Tools   – UKlizaKon  metrics  and  requirements   – Provisioning  Spreadsheet   Success  Stories   – CPU  resource  management   – Tuning  and  provisioning  adjustments   – I/O  resource  management   www.enkitec.com   41  
  42. 42. www.enkitec.com   42   Q & A
  43. 43. 43 Fastest Growing Companies in Dallas Contact  Info…   karl.arao@enkitec.com

×