Cloud Consolidation with Oracle (RAC) - How much is too much?

2,252 views
2,168 views

Published on

Cloud Consolidation with Oracle (RAC)
– How much is too much?

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,252
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
114
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Cloud Consolidation with Oracle (RAC) - How much is too much?

  1. 1. 5/14/20121 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Cloud Consolidation with Oracle (RAC) – How much is too much? Markus.Michalewicz@oracle.com Nitin.Vengurlekar@oracle.com Senior Principal Product Manager Technical Director, Oracle RAC2 Copyright © 2012, OracleRAC, Oraclereserved. Oracle and/or its affiliates. All rights America Development, Oracle America 1
  2. 2. 5/14/2012 Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.3 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Agenda • Database Cloud Architectures • General Considerations • Database Consolidation – Memory Management – CPU(_COUNT) Management • A word on Instance Caging – Database Resource Management – Summary Database Consolidation • Oracle RAC-specific Considerations for Consolidation – Per Server Limits – Real Time (RT) Processes – Cluster Limits • Summary and Q&A4 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 2
  3. 3. 5/14/2012 Database Cloud Architectures Common building blocks are shared server and storage pools Infrastructure Cloud Database Cloud Database CloudDW CRM ERP DW ERP CRM DW ERP CRM DB DB DB DB DB DB DB OS OS OS Hypervisor Hypervisor OS OS OS OS Server Database Schema5 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. General Considerations6 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 3
  4. 4. 5/14/2012 Oracle (RAC) Database Consolidation Registered vs. Running • Registered databases and instances could Registered: potentially start and run at the same time. • n DB instances are defined to run on a machine • Oracle’s Quality of Service Management (potentially) or scripts can be used to model policies to run certain databases only at certain times; e.g. geographic region over time vs. • Assume the cluster is PST based: Running: • EMEA based DBs run 10:00pm – 8am PST • Registered databases • APAC based DBs run 6:00pm – 3am PST and instances are (concurrently) running • USA based DBs run 8:00am – 6pm PST (active workload)7 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Database Consolidation General Considerations • Simplest of all consolidation cases: Database Cloud • One database Instance per Server DW ERP CRM • When consolidating more than one DB database on a server, consider the server capacity with any DB added. OS OS • More details: Best Practices for Database Consolidation in Private Clouds http://www.oracle.com/technetwork/database/focus- areas/database-cloud/database-cons-best-practices-1561461.pdf • Exadata Database Machine-specific: Schema http://www.oracle.com/technetwork/database/features/ availability/exadata-consolidation-522500.pdf8 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 4
  5. 5. 5/14/2012 General Considerations Consider Workload Characteristics during Capacity Planning Existing Workload Utilization Peak The smaller this Average gap, the better. Time New Workloads Workload A OR Workload B Utilization Utilization Time Time 9 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. General Considerations Consider Workload Characteristics during Capacity Planning Existing Workload Resulting Workload Utilization Peak The smaller this gap, the better. Average Peak Poor match: Utilization Time Gap increases (antagonistic) + Average Workload B Time Utilization Time10 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 5
  6. 6. 5/14/2012 General Considerations Consider Workload Characteristics during Capacity Planning Existing Workload Resulting Workload Utilization Peak The smaller this gap, the better. Average Utilization Time Good match: Peak Gap decreases Average + (complimentary) Workload A Time Utilization Time11 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. General Considerations Consider Workload Characteristics during Downtime Utilization Normal Operation Utilization Peak HR DW ERP CRM Average DBA Time DBA DBB OS OS OS OS12 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 6
  7. 7. 5/14/2012 General Considerations Consider Workload Characteristics during Downtime Utilization Normal Operation Utilization Peak HR DW ERP CRM Average DBA Time DBA DBB During Downtime Peak OS OS OS OS Utilization Average Time13 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. The Benefits of Standardization Easier deployment and better predictability • Standardization of software and hardware simplifies planning • Standardized hardware means a Nodes predictable behavior should demand increase and additional hardware needs to be added (horizontal scaling approach) New • Using “application profiling” (template Application A based deployment) based on current system(s) and performance baselines Large OLTP Medium allows for a predictable deployment of OLTP Small new applications on the same system OLTP using existing profiles. Then deploy14 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 7
  8. 8. 5/14/2012 Database Consolidation15 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Database Consolidation Starting block: One Database instance per server • Components to consider: • Memory • CPU DB • I/O • Processes OS • Network16 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 8
  9. 9. 5/14/2012 Database Consolidation One instance per server as the basis • An Oracle database by default assumes that there is only one database instance running on the server: • Instance parameters are based on this assumption • Consolidation changes that premise • Main resources used: DB • Memory • CPU • I/O • Resources “regulated by default”: • Memory OS • SGA / PGA Targets • CPU17 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Database Consolidation Recommendation 1: Manage Memory carefully (and dynamically) • Avoid memory starvation and swapping as it has negative impact on the system. • Do not oversubscribe memory resources • Define memory settings carefully – rule of thumb: • For the OS in general: • Shared Memory identifiers and segments DB • Use Hugepages, if possible – details: 80% • MOS notes 361323.1 and 401749.1 • For OLTP applications: • SUM (sga_target + pga_aggregated_target) <= 80% of physically available memory per DB server OS 20% • For DW / BI applications: • SUM (sga_target + 3* pga_aggregated_target) <= 80% of physically available memory per DB server18 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 9
  10. 10. 5/14/2012 Database Consolidation Recommendation 2: Use CPU_COUNT to “cage instances” • CPU usage should be regulated. • The OS scheduler schedules CPU as requested by each individual instance. • The OS scheduler does not know about the DBB priority of the various instances on the server. • Use CPU_COUNT or ideally Instance Caging DBA • Instance Caging is configured in just 2 steps: • 1. Set “cpu_count” parameter • Max. number of CPUs the instance can use at any time • 2. Set “resource_manager_plan” parameter OS • Enables CPU Resource Manager • E.g. out-of-box plan “DEFAULT_PLAN”19 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Database Consolidation Using CPU_COUNT as a “central knob” • CPU_COUNT regulates CPU usage and Data Structures dependent resources (to a certain degree): Concurrency Parallelism • Parallelism (PQ operations) fx(CPU_COUNT) • Processes Processes DBB • Load Calculation Memory Allocation Load Calculation • Processes and PQ DBA operations should be 80% considered explicitly. 16 OS 20%20 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 10
  11. 11. 5/14/2012 A Word on Instance Caging21 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Database Consolidation Instance Caging: Partitioning Approach • Provides maximum isolation 16 CPUs 32 28 • For performance-critical databases DBA DBB DBC DBD 24 • If one database instance is idle, 20 its CPU allocation is unused 16 Instance D: 4 CPUs 12 • The rule of thumb for the partitioning Instance C: 4 CPUs 8 approach is to set a general limit: Instance B: 4 CPUs • SUM (CPU_COUNT) < 75% x Total CPUs 16 OS 4 Instance A: 4 CPUs 022 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 11
  12. 12. 5/14/2012 Database Consolidation Instance Caging: Over-Provisioning Approach • Best used for non-critical databases 16 CPUs 32 that are typically well-behaved 28 DBA DBB DBC DBD • Contention for CPU if database 24 instances are sufficiently loaded 20 Instance D: 8 CPUs 16 • Typically not enough contention 12 to destabilize OS or DB instances Instance C: 6 CPUs 8 Instance B: 4 CPUs • Best approach if the goal 16 OS 4 Instance A: 4 CPUs is fully utilized CPUs 023 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Database Consolidation Instance Caging – under the covers • If cpu_count is set to 4 on a 16 CPU server Over-Provisioning • All foreground processes make progress Partitioning Approach Approach • But only 4 foregrounds are running at any time 32 32 28 28 • Most backgrounds are not managed 24 24 • Critical and use very little CPU 20 20 • MMON, Job Scheduler slaves are managed Instance D: 8 CPUs 16 16 Instance D: 4 CPUs • No CPU affinity 12 12 Instance C: 6 CPUs Instance C: 4 CPUs • Not meant for hard-partitioning or licensing 8 8 • All CPUs may be used Instance B: 4 CPUs Instance B: 4 CPUs 4 4 • CPU utilization averaged across all CPUs ≤ 25% Instance A: 4 CPUs Instance A: 4 CPUs 0 0 • More information: http://www.oracle.com/technetwork/database/focus-areas/performance/instance-caging-wp-166854.pdf24 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 12
  13. 13. 5/14/2012 Database Consolidation Over-Provisioning Approach – It’s still hardware that’s the limit • Best used for non-critical 140 120 databases that are typically 100 80 well-behaved – examples: CPU Util. 60 Average 40 • Complimentary workload 20 0 • Systems with little contention t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 t12 for CPU if database instances 32 16 CPUs are sufficiently loaded. 28 DBADBBDBCDBD 24 • Do not use, 20 Instance D: 8 CPUs 16 • If the load is significant and of longer duration, as system 12 Instance C: 6 CPUs stability can get impacted. 8 Instance B: 4 CPUs • For highly critical systems 16 OS 4 Instance A: 4 CPUs 025 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Database Consolidation How much over-provisioning is OK? • As CPU_COUNT does not consider the “quality of a CPU” an absolute maximum 32 is hard to determine / depends on the system. 16 CPUs 28 DBA DBB DBC DBD • The general rule of thumb is: 24 • SUM (CPU_COUNT) <= up to 2x Total CPUs 20 • Consider different system types: Instance D: 8 CPUs 16 Threaded Core based Engineered 12 “Total CPUs” Instance C: 6 CPUs # of threads # of cores # of threads are based on 8 Max. over- Instance B: 4 CPUs provisioning 1.5 (HT*) 2.0 3.0 (DBM) 16 OS 4 1.0 (hHT**) 2.0 (ODA) Instance A: 4 CPUs factor 0 (HT*): ratio 1:2; (hHT**): ratio 1:n, with n >226 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 13
  14. 14. 5/14/2012 Database Resource Management27 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Database Consolidation Recommendation 3: Use DB Resource Manager 3 steps to use Resource Manager: 1. Group sessions with similar performance objectives into Consumer Groups 2. Allocate resources to consumer groups using Resource Plans 3. Enable Resource Plan28 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 14
  15. 15. 5/14/2012 Summary Database Consolidation 29 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Database Consolidation Summary1. Define memory settings carefully: • For OLTP applications: 32 • SUM (sga_target + pga_aggregated_target) 28 <= 80% of physically available memory per DB server DBB • For DW / BI applications: 24 • SUM (sga_target + 3* pga_aggregated_target) 20 <= 80% of physically available memory per DB server 80% Instance D: 8 CPUs DBA 162. Use CPU_COUNT or ideally Instance Caging 12 Instance C: 6 CPUs • The general rule of thumb is: • SUM (CPU_COUNT) <= up to 2 x Total CPUs 8 OS 20% Instance B: 4 CPUs 4 • Consider different system types. Instance A: 4 CPUs 03. These are the most crucial per server limits. 30 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 15
  16. 16. 5/14/2012 Oracle RAC-specific Considerations for Consolidation31 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Oracle RAC based Consolidation Server limits are mostly reached before cluster limits apply • Most customers will experience “per server limits” before “cluster limits” apply. • Oracle RAC introduces a few more Per Server Limits DBA1 DBB1 DBC1 DBA2 DBB2 DBC2 processes (potential limits) to consider. • Oracle RAC DBs use LMS Real Time (RT) processes per instance. • LMS RT processes need to be considered in particular Clusterware Clusterware 16 OS 16 OS Cluster Limits32 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 16
  17. 17. 5/14/2012 Real Time (RT) Processes33 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Oracle RAC based Consolidation Considerations for Real Time (RT) Processes in general • A Real Time process can only run on one CPU (core) at a time. • The usage of the CPU is typically short. DBA1 DBB1 DBC1 DBA2 DBB2 DBC2 • The general rule of thumb is: • The aggregated number of RT processes per server should not exceed the number of cores per server • One Oracle RAC instance has typically at Clusterware Clusterware least one RT process (LMS) per default 16 OS 16 OS • An Oracle ASM instance has one RT process • Oracle Clusterware uses various RT processes34 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 17
  18. 18. 5/14/2012 Oracle RAC based Consolidation Background for LMS Real Time (RT) Process recommendation • The number of LMS RT processes per instance is determined by a function on CPU_COUNT. • In order to guarantee optimized DBA1 DBB1 DBC1 DBA2 DBB2 DBC2 performance and reliability, the general rule of thumb for RAC is: • The aggregated number of LMS RT processes per server should not exceed [cores per server]-1 • See MOS note: 558185.1 for details Clusterware Clusterware • This leaves one core free for additional RT processes to be assigned as needed, as 16 OS 16 OS LMS RT can stay on a core for a moment35 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Oracle RAC based Consolidation Automatic adjustment of LMS process priority in 11.2.0.3 • With 11.2.0.3 the number of LMS RT processes are monitored and adjusted according to the number of cores on DBA2 DBB2 DBC2 DBD2 DBA1 DBB1 DBC1 DBD1 the node periodically. Per Server Limits DBF1 DBF2 • The goal is to keep RT LMSs per server <= # cores per server • For details, see MOS note 1392248.1 – Auto-Adjustment of LMS Process Priority in Oracle RAC with 11.2.0.3 and later DBE1 DBE2 • This excludes any ASM instance running on the system as well as any pre-11.2.0.3 database instance 4 OS 4 OS • See also MOS note 1439551.1 – Oracle (RAC) Database Consolidation Guidelines for Environments using mixed Database Versions36 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 18
  19. 19. 5/14/2012 Oracle RAC based Consolidation Recommendation 4: Over-provision only based on CPU_COUNT • Do not over-provision the number 48 . of LMS RT processes on one server. . . . DBA2 DBB2 DBC2 DBD2 DBA1 DBB1 DBC1 DBD1 . Per Server Limits . • Limit the number of RT LMSs by: DBF1 DBF2 32 • Using CPU_COUNT 28 • Directly reducing the number of LMS 24 RT processes (gcs_server_processes) DBE1 DBE2 20 • Downgrading additional LMS RT Instance D: 8 CPUs 16 processes to time share (TS). 12 • In 11.2.0.3 and later, the RT to CPU OS OS 8 Instance C: 6 CPUs rule will be enforced by automatically Instance B: 4 CPUs downgrading subsequently started 4 LMS RT processes to TS. Instance A: 4 CPUs 037 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Cluster Limits38 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 19
  20. 20. 5/14/2012 Oracle RAC based Consolidation Cluster Limits – Starting DBs are currently the main concern • In most cases, per server limits will be Starting: reached before cluster limits are reached. • Registered databases and instances start • Cluster limits apply to: • Default is “starting • Registered resources (databases) at the same time”. DBC1 DBD1 DBC2 DBD2 DBA1 DBB1 DBA2 DBB2 • Starting databases in the cluster – reason: • Starting databases need to register with the cluster (Oracle Clusterware). • Currently and for example, 100 Clusterware Clusterware starting Oracle RAC databases on a 4 node cluster are supported. OS OS • The Number assumes each Oracle RAC database uses an instance on each node. Cluster Limits39 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Summary40 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 20
  21. 21. 5/14/2012 (Oracle RAC) Database Consolidation Summary • For a successful database consolidation, consider the following as rules of thumb: Existing Workload Peak DB Utilization 80% 1. General considerations for capacity planning Average 2. Manage Memory carefully (and dynamically) Time + OS 20% 3. Use CPU_COUNT 4. Use DB Resource Manager Data Structures 5. Over-provision only based on CPU_COUNT Concurrency Parallelism Processes Memory Allocation Load Calculation =41 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.42 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 21

×