Maximizing Performance on a  Cost Effective Oracle® 11g  SE RAC Database on  Dell™ EqualLogic™ PS4000XV  iSCSI Storage  A ...
THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAINTYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE C...
CONTENTSABSTRACT ............................................................................................................
CPU Performance Analysis ....................................................................................................
AbstractCustomers rely on Oracle databases for their mission critical workloads, but sometimes consideralternatives becaus...
RAC database on Dell EqualLogic PS4000XV iSCSI storage arrays, and the Dell PowerEdge11th generation R610 servers running ...
Feature Availability by EditionEach edition of the Oracle 11g database includes features to meet the varying businessenvir...
Enterprise-class FeaturesThe PS4000 Series storage arrays include advanced software features with no additionalsoftware li...
Figure 1: Oracle 11g SE on Dell EqualLogic PS4000XV iSCSI Storage ArchitectureOverviewSeamless ScalabilityOracle RAC offer...
Figure 2: "Pay-as-you-grow" Methodology to Scale up to Four Single Socket ServersHardware ConfigurationStorage Configurati...
switches are used, because if a switch fails in a single Ethernet switch environment all hosts willlose access to the SAN ...
RAID levels, or application types. In Figure 4, there is a pool with the name RAID-10 thatconsists of RAID 10 members.Befo...
Table 1 shows a sample volume configuration, with volumes for each of the three previouslydescribed shared storage areas.V...
the iSCSI initiator with the correct user name and password. This information must match anaccess control record for the v...
For more information on PS Series network configuration best practices and recommendations,refer to the Dell EqualLogic wh...
NIC port failure in a single NIC port environment, the host will lose access to the storage until thefailed NIC is physica...
Lastly, modify the /etc/multipath.conf template file with the appropriate information for yourexisting environment. It is ...
Configuring Shared Storage for the Database Using the ASM Library DriverTwo separate volumes are created for the data stor...
Performance Capabilities of Oracle 11g SE RACTest Tools and ConfigurationQuest Benchmark Factory TPC-CTo understand the pe...
Performance MonitoringThe testing utilized a number of monitoring tools to characterize cluster components. Byanalyzing an...
Real-Time DetailCounter              UserLoad   Sequence   Test Phase   Elapsed Seconds   Value   Time StampAVG_RESP_TIME ...
SwapFree: 8239712 kB…zzz ***Sun Jun 21 23:04:59 CDT 2009MemTotal: 49434036 kBMemFree: 34270564 kBBuffers:    354892 kBCach...
Tuning Oracle Memory SizeOn the test database baseline Benchmark Factory tests were conducted to optimize databasesettings...
Management. The size of the memory_target should be the total size of SGA and PGA area.The test team set the memory_target...
CPU utilization of a node within a 3-node RAC cluster with 24 GB RAM per node                         120                 ...
•     The overall CPU utilization still had room for more growth. For example, when Oracle                                ...
extracted data contained the average IOPS and disk latency data points in 30-minute timeintervals. SAN HQ allows you to ex...
Overall Performance AnalysisFigure 16 shows the final Benchmark Factory results for all the nodes with 24 GB of physicalme...
For the 48 GB cluster, Benchmark Factory ran again on one, two, three, and finally four nodeswithin the cluster and an inc...
CPU utilization of a node within a 2-node RAC cluster with 48 GB RAM per node                         120                 ...
CPU utilization of a node within a 4-node RAC cluster with 48 GB RAM per node                         120                 ...
Maximizing Performance on Oracle Database Using Dell EqualLogic Storage
Maximizing Performance on Oracle Database Using Dell EqualLogic Storage
Maximizing Performance on Oracle Database Using Dell EqualLogic Storage
Maximizing Performance on Oracle Database Using Dell EqualLogic Storage
Maximizing Performance on Oracle Database Using Dell EqualLogic Storage
Maximizing Performance on Oracle Database Using Dell EqualLogic Storage
Maximizing Performance on Oracle Database Using Dell EqualLogic Storage
Maximizing Performance on Oracle Database Using Dell EqualLogic Storage
Maximizing Performance on Oracle Database Using Dell EqualLogic Storage
Upcoming SlideShare
Loading in...5
×

Maximizing Performance on Oracle Database Using Dell EqualLogic Storage

3,192

Published on

This white paper is intended to help IT professionals design and configure Oracle® 11g Real Application Clusters (RAC) database solutions using Dell™ EqualLogic™ arrays and servers that apply best practices derived from laboratory and real-world experiences. It documents the Dell recommended approach for implementing a tested and validated solution for Oracle 11g RAC database on Dell EqualLogic PS4000XV Internet SCSI (iSCSI) storage arrays and Dell PowerEdge™ 11th-generation R610 servers running Red Hat® Enterprise Linux® 5 Update 3.

Published in: Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,192
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
81
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Transcript of "Maximizing Performance on Oracle Database Using Dell EqualLogic Storage"

  1. 1. Maximizing Performance on a Cost Effective Oracle® 11g SE RAC Database on Dell™ EqualLogic™ PS4000XV iSCSI Storage A Dell Technical White PaperDatabase SolutionsEngineeringBy Wendy Chen, Naveen Iyengar, and David MarDell Product GroupJuly 2009 Reference Configuration for Oracle 11g SE RAC on Dell EqualLogic PS4000XV Storage Arrays 1
  2. 2. THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAINTYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT ISPROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND.© Copyright 2009 Dell, Inc. All rights reserved. Reproduction in any manner whatsoever withoutthe express written permission of Dell Inc. is strictly forbidden. For more information, contact Dell.Dell, the Dell logo, PowerEdge, and EqualLogic are trademarks of Dell Inc; Intel and Xeon areregistered trademarks of Intel Corporation in the U.S. and other countries; Microsoft, SQL Server,Windows, and Windows Server are either trademarks or registered trademarks of MicrosoftCorporation in the United States and/or other countries; Oracle is a registered trademark ofOracle Corporation and/or its affiliates; Red Hat and Red Hat Enterprise Linux are registeredtrademarks of Red Hat, Inc. Other trademarks and trade names may be used in this document torefer to either the entities claiming the marks and names or their products. Dell disclaimsproprietary interest in the marks and names of others.Reference Configuration for Oracle 11g SE RAC on Dell EqualLogic PS4000XV Storage Arrays 2
  3. 3. CONTENTSABSTRACT ..................................................................................................................................... 5INTRODUCTION ............................................................................................................................. 5 DELL SOLUTIONS FOR ORACLE 11G DATABASE ............................................................................... 6ORACLE 11G DATABASE STANDARD EDITION OVERVIEW ................................................... 6 ORACLE 11G LICENSING OPTIONS .................................................................................................. 6 FEATURE AVAILABILITY BY EDITION................................................................................................. 7DELL EQUALLOGIC PS4000 STORAGE SYSTEMS OVERVIEW............................................... 7 EASE OF USE ................................................................................................................................ 7 HIGH PERFORMANCE ..................................................................................................................... 7 ENTERPRISE-CLASS FEATURES ...................................................................................................... 8 LOW COST OF OWNERSHIP ............................................................................................................ 8ORACLE 11G DATABASE SE RAC ARCHITECTURE OVERVIEW ............................................ 8 SEAMLESS SCALABILITY ................................................................................................................. 9HARDWARE CONFIGURATION .................................................................................................. 10 STORAGE CONFIGURATION .......................................................................................................... 10 Configuring Dell EqualLogic PS4000XV iSCSI Storage Connections .................................. 10 Configuring Volumes............................................................................................................. 12 Configuring Challenge Handshake Authentication Protocol (CHAP) ................................... 13 ISCSI SAN GIGABIT ETHERNET SWITCH CONFIGURATION ............................................................. 14 Configuring iSCSI SAN Network ........................................................................................... 14 SERVER CONFIGURATION............................................................................................................. 15 Configuring Fully Redundant Ethernet Interconnects ........................................................... 15 Configuring Multiple Ethernet Interfaces for iSCSI Storage Area Networks......................... 15SOFTWARE CONFIGURATION................................................................................................... 16 OPERATING SYSTEM CONFIGURATION .......................................................................................... 16 Configuring Host Access to iSCSI Volumes ......................................................................... 16 Configuring the Oracle Private Network NIC Teaming ......................................................... 17 Configuring Network Requirements on All Nodes ................................................................ 17 Configuring Host Equivalence .............................................................................................. 17 Configuring Shared Storage for Oracle Clusterware Using Block Device ............................ 17 Configuring Shared Storage for the Database Using the ASM Library Driver...................... 18 ORACLE 11G R1 CONFIGURATION ................................................................................................ 18 Installing Oracle Clusterware and Database Software ......................................................... 18 Installing Recommended Oracle Patches............................................................................. 18PERFORMANCE CAPABILITIES OF ORACLE 11G SE RAC ................................................... 19 TEST TOOLS AND CONFIGURATION ............................................................................................... 19 Quest Benchmark Factory TPC-C ........................................................................................ 19 Hardware and Software Configuration.................................................................................. 19 Performance Monitoring........................................................................................................ 20 TUNING ORACLE MEMORY SIZE.................................................................................................... 23 TESTING RAC SCALABILITY WITH 24 GB RAM PER NODE............................................................. 24 CPU Performance Analysis .................................................................................................. 24 Memory Utilization Analysis .................................................................................................. 26 Storage System Performance Analysis ................................................................................ 26 Examining both CPU and Memory Utilization ....................................................................... 27 Overall Performance Analysis .............................................................................................. 28 TESTING RAC SALABILITY WITH 48 GB RAM PER NODE ............................................................... 28Reference Configuration for Oracle 11g SE RAC on Dell EqualLogic PS4000XV Storage Arrays 3
  4. 4. CPU Performance Analysis .................................................................................................. 29 Memory Utilization Analysis .................................................................................................. 31 Examining both CPU and Memory Utilization ....................................................................... 32 Storage System Performance Analysis ................................................................................ 32 Overall Performance Analysis .............................................................................................. 33 PERFORMANCE CHARACTERISTICS SUMMARY ............................................................................... 34SOLUTION DELIVERABLES LIST FOR ORACLE 11G ON DELL EQUALLOGIC PS4000XVISCSI STORAGE........................................................................................................................... 36CONCLUSION............................................................................................................................... 38TABLES AND FIGURES .............................................................................................................. 39REFERENCES .............................................................................................................................. 39Reference Configuration for Oracle 11g SE RAC on Dell EqualLogic PS4000XV Storage Arrays 4
  5. 5. AbstractCustomers rely on Oracle databases for their mission critical workloads, but sometimes consideralternatives because of the cost of the enterprise edition database. With the significantly lowercost of the Oracle Standard Edition (SE) database, with Oracle Real Application Cluster (RAC)capabilities extending to 4-socket processors, Oracle SE RAC offers database customers a lowercost option that has enterprise level performance. Customers that are looking to implement anOracle RAC database can utilize the Oracle SE, Red Hat® Enterprise Linux® and commodityhardware in order to lower their total cost of ownership (TCO) of their enterprise data. This paperpresents a “pay-as-you-grow” methodology to build an Oracle SE RAC configuration that canscale up to 4 single-socket servers. It showcases the performance characteristics and scalabilityof incrementally adding RAC nodes.IntroductionOracle 11g Database Standard Edition is built from the same code base as the Oracle 11gEnterprise Edition, and is ideally suited to the needs of small to medium-sized business (SMB).Oracle 11g database enables business applications to take advantage of performance, reliability,security and scalability at a lower cost. Oracle 11g database includes the Oracle Real ApplicationClusters (RAC) for enterprise-class availability.Dell’s newest PowerEdge™ 11th generation servers feature energy-tuned technologies designedto reduce power consumption while increasing performance and capacity. The LifecycleController (LC) components simplify administrator tasks by performing a complete set ofprovisioning functions such as system deployment, system updates, hardware configuration anddiagnostics from a single intuitive interface called the Unified Server Configurator (USC) in a pre-OS environment. The new Dell Management Console (DMC) delivers a single view and acommon data source into the entire infrastructure. DMC helps to reduce or eliminate manualprocesses so that less time and money is spent on maintenance and more can be spent onstrategic uses of technology.The PowerEdge R610 1U rack server features Intel® 5500 series processors. This processorfeatures quad-core processing to maximize performance, and performance/watt, for data centerinfrastructures and highly dense deployments. Two other notable features that benefit multi-threaded demanding database workloads are: • Support for CPU turbo mode (on certain SKUs) that increases CPU frequency if operating below thermal, power, and current limits • Simultaneous multi-threading (hyper-threading) capability that increases application performance by delivering greater throughput and responsivenessThe PowerEdge R610 utilizes DDR3 memory that provides a high performance, high-speedmemory interface capable of low-latency response and high throughput.Dell EqualLogic PS4000 is the newest addition to the PS Series family of iSCSI storage arrays.Designed for deployment in SMB environments, the PS4000 offers an entry-point to the PSSeries providing easy administration, enterprise software, and virtualized architecture at anaffordable price point. The PS4000 arrays support both SAS and SATA disk drives and includethree product lines: the PS4000XV, PS4000X and PS4000E that are differentiated by the diskdrive type and speed supported. The PS4000XV offers the best performance by supporting 15 KRPM SAS disks. PS4000XV is an ideal choice to deploy highly reliable and sustainable Oracle11g RAC databases.This reference configuration white paper is intended to help IT professionals design and configureOracle 11g RAC database solutions using Dell EqualLogic arrays and servers that apply bestpractices derived from laboratory and real-world experiences. This white paper documents theDell recommended approach for implementing a tested and validated solution for Oracle 11gReference Configuration for Oracle 11g SE RAC on Dell EqualLogic PS4000XV Storage Arrays 5
  6. 6. RAC database on Dell EqualLogic PS4000XV iSCSI storage arrays, and the Dell PowerEdge11th generation R610 servers running Enterprise Linux 5 Update 3.Dell Solutions for Oracle 11g DatabaseDell solutions for Oracle 11g databases are designed to simplify operations, improve usability,and cost-effectively scale as your needs grow over time. In addition to providing server andstorage hardware, Dell solutions for Oracle 11g include: • Dell Configurations for Oracle – in-depth testing of Oracle 11g configurations for high- demand solutions; documentation and tools that help simplify deployment • Integrated Solution Management – standards-based management of Dell solutions for Oracle 11g that can lower operational costs through integrated hardware and software deployment, monitoring, and updates • Oracle Licensing – multiple licensing options that can simplify a customer purchase • Dell Enterprise Support and Infrastructure Services for Oracle – offerings for the planning, deployment, and maintenance of Dell solutions for Oracle 11gFor more information concerning Dell solutions for Oracle 11g database, please visitdell.com/oracle.Oracle 11g Database Standard Edition OverviewOracle 11g Licensing OptionsOracle 11g database is available in multiple editions, each tailored to meet the development anddeployment needs of different sizes of IT organizations. Here are the two most relevant to thiswhite paper:• Oracle Standard Edition (SE) is available for single or clustered servers supporting up a maximum of four CPU sockets. It includes selected, but not all, features that come with the Oracle Enterprise Edition (EE).• Oracle Enterprise Edition (EE) is available for single or clustered servers with no limit on the maximum number of CPU sockets. It contains all of the Oracle database components, and can be further enhanced with the purchase of options and packs.Oracle 11g SE includes Oracle Real Application Clusters (RAC) for enterprise-class availability.When used with Oracle RAC in a clustered server environment, Oracle 11g Standard Editionrequires the use of Oracle Clusterware. Automatic Storage Management (ASM) must be used tomanage all database-related files. No other cluster software can be installed on the system,including OCFS (Oracle Cluster File System), OCFS2, third-party clusterware, third-party clustervolume managers, or third-party cluster file systems.Oracle 11g SE can support clustered servers with up to four CPU sockets in the cluster. OracleDatabase 11g Standard Edition is available from Dell on versions of Windows, Linux and Solaris.Using the same proven concurrency techniques as Oracle 11g EE, Oracle 11g SE ensuresmaximum throughput for all workloads. With the implementation of Oracle RAC, workloads areautomatically balanced across the available servers in the cluster to ensure maximum hardwareutilization. RAC also protects the database from machine failures. When a node in the clusterfails, the database continues to run in a degraded mode on the surviving nodes of the cluster.Reference Configuration for Oracle 11g SE RAC on Dell EqualLogic PS4000XV Storage Arrays 6
  7. 7. Feature Availability by EditionEach edition of the Oracle 11g database includes features to meet the varying businessenvironment requirements. SE contains a different subset of database features that are allincluded in EE. Additionally, a number of advanced options and packs can be purchased with EEat an extra cost, to meet the most demanding requirements of mission-critical databases.A number of database features that come with the Oracle 11g EE are not available in SE. Someof these features are highlighted below: • Data guard • Rolling upgrades • Online index and table organization • Parallel backup and recovery • Table spaces point-in-time recovery • Flashback table / transaction / database • Parallel query / statistics gathering / index builds / data pump export and import • Transportable table spaces • Oracle connection manager • Infiniband supportA complete list of feature availability by edition can be found in the Oracle Database LicensingInformation 11g Release 1 (11.1) documentation. Before deploying your databases on OracleSE, make sure that you do not require one of non-supported features. If you do require one ofthese features, it is recommended that you deploy Oracle EE.Dell EqualLogic PS4000 Storage Systems OverviewThe Dell EqualLogic PS4000 Series of iSCSI SAN arrays are designed to bring enterprise-classfeatures, intelligence, automation, and reliability to SMB storage deployments. The PS4000Series addresses the storage needs of SMB with simplified administration, rapid deployment, andan affordable price while providing a full set of enterprise-class data management and dataprotection software features, self-optimized high performance, and pay-as-you-grow scaling.Ease of UseBy including SAN configuration features and capabilities that sense network connections,automatically build RAID sets, and conduct system health checks the PS4000 Series is designedto be installed and configured in less than one hour. The Dell EqualLogic SAN HeadQuarterssoftware tool enables consolidated performance and event monitoring across multiple EqualLogicSAN groups. SAN HeadQuarters helps administrators to better understand storage performanceand capacity usage, and quickly be informed of events and potential issues.High PerformanceAll Dell EqualLogic PS Series iSCSI storage arrays are built on a patented peer storagearchitecture, where all arrays in a storage pool are designed to work together to provide diskcapacity and evenly distribute the load. This architecture enables the arrays to continuousmonitor resource and performance, and automatically moves data pages to maximizeperformance.Reference Configuration for Oracle 11g SE RAC on Dell EqualLogic PS4000XV Storage Arrays 7
  8. 8. Enterprise-class FeaturesThe PS4000 Series storage arrays include advanced software features with no additionalsoftware licensing fee. These features enable easy management, data protection, and storagevirtualization. Examples of the software features include PS Group Manager, SAN HeadQuartersmulti-group monitoring, multi-path I/O, Auto-Snapshot Manager, and automatic load balancing.Low Cost of OwnershipThe Dell EqualLogic PS4000 Series storage system offers an entry-point to the PS Series at anaffordable price point. The PS4000 Series drives down the total cost of SAN storage with an all-inclusive package of highly-reliable hardware and comprehensive software features. ThePS4000 Series can expand to up to two PS4000 arrays in one storage group. To support higherperformance and capacity scalability requirements, PS4000 arrays can be upgraded to PS6000Series that is the high-end PS Series storage product.Oracle 11g Database SE RAC Architecture OverviewThe Dell reference configuration for Oracle 11g SE on Dell EqualLogic PS4000XV iSCSI storagearrays is intended to validate the following solution components: • Up to a four-node cluster comprised of Dell PowerEdge R610 servers • Dell EqualLogic PS4000XV iSCSI storage systems • Red Hat Enterprise Linux 5 Update 3 • Oracle 11g R1 (11.1.0.7) SE x86_64An architectural overview of the Dell solution for Oracle 11g SE on Dell EqualLogic PS4000XViSCSI storage arrays is shown in Figure 1. The sample architecture represents a two-node OracleRAC comprised of the following components: • Dell Optiplex™ desktop systems that will access data stored within the Oracle database • Client-server network made up of network controllers, cables, and switches • Dell PowerEdge R610 servers running Red Hat Enterprise Linux 5 Update 3 and Oracle 11g R1 RAC (11.1.0.7) • Redundant Dell gigabit Ethernet switches for the Oracle cluster interconnect network • Server-storage interconnect using redundant Dell PowerConnect™ 6248 gigabit Ethernet switches • Two Dell EqualLogic PS4000XV iSCSI storage arrays configured as RAID 10.Reference Configuration for Oracle 11g SE RAC on Dell EqualLogic PS4000XV Storage Arrays 8
  9. 9. Figure 1: Oracle 11g SE on Dell EqualLogic PS4000XV iSCSI Storage ArchitectureOverviewSeamless ScalabilityOracle RAC offers a number of benefits when compared to a traditional single instance database: • High availability – RAC is typically built to eliminate any single point of failure by implementing redundant components. In the event of a node or database instance failure, application connections can be automatically failed over to the surviving instances. • Scalability - Multiple nodes in a RAC allows the database to scale beyond the limit of a single node database. Additional servers can be added into an existing RAC cluster when the workload increases and demands more processing power. Adding an additional node to or removing existing nodes from, a RAC can be done without any database downtime. • Low cost of ownership - RAC systems are typically implemented on industry standard hardware that is less expensive than proprietary systems; this helps customers reduce the cost of purchase, maintenance, and support.With the ability to support up to four sockets worth of compute resources in a cluster, Oracle 11gSE allows a customer to start with one single socket machine at an attractive price point, andsimply scale up to four single socket machines as their business demands grow. This “pay-as-you-grow” methodology helps maximize hardware utilization and lowers the total cost ofownership.Scalability is generally measured by the throughput while increasing the workload; RACscalability is dependent on the scalability of the hardware, storage, operating systems, database,and application. The scalability that can be achieved by Oracle RAC varies according to thedesign and configurations of the hardware and software.Reference Configuration for Oracle 11g SE RAC on Dell EqualLogic PS4000XV Storage Arrays 9
  10. 10. Figure 2: "Pay-as-you-grow" Methodology to Scale up to Four Single Socket ServersHardware ConfigurationStorage ConfigurationConfiguring Dell EqualLogic PS4000XV iSCSI Storage ConnectionsThe Dell EqualLogic PS4000XV iSCSI storage array offers a highly-available hardware designthat includes redundancy features to ensure that there is no single point of failure. Its componentsare fully redundant and hot swappable, including the 16 SAS disks, dual active/standby controlmodules, dual fan trays, and dual power supplies. As illustrated in Figure 3, the control modulesare identified as control module 0 and control module 1. Each control module has two 1 gigabitEthernet interfaces for the SAN network, labeled as ethernet0 and ethernet1. Each controlmodule also has one 10/100 Ethernet management port to be used for management purposesonly. Each control module is equipped with a 2 GB battery-backed, write-back cache to ensurecache coherency between the two control modules. In the event of a control module failure, theother control module takes over automatically with no disruption to users.Figure 3: Recommended Dual Control Module Network ConfigurationHost servers can be attached to the PS4000XV through an IP storage area network (SAN)industry-standard gigabit Ethernet switch. Figure 3 shows the recommended networkconfiguration for a dual control module PS4000XV array. This configuration includes tworedundant Dell PowerConnect 6248 gigabit Ethernet switches to provide the highest networkavailability, and maximum network bandwidth. It is recommended that two gigabit EthernetReference Configuration for Oracle 11g SE RAC on Dell EqualLogic PS4000XV Storage Arrays 10
  11. 11. switches are used, because if a switch fails in a single Ethernet switch environment all hosts willlose access to the SAN until the switch is physically replaced and the configuration restored.From each of the control modules, it is recommended that one gigabit interface connects to oneEthernet switch, and the other gigabit interface connect to the other Ethernet switch for highavailability.Figure 4: Cabling Two-Node Oracle Database 11g Standard Edition Real ApplicationClusters DatabaseFigure 4 illustrates the cabling of a two-node PowerEdge cluster hosting the Oracle database andtwo PS4000XV storage arrays where the data resides. The blue colored cables are the iSCSIstorage area network, and the red colored cables are the Oracle RAC private interconnectnetwork. The black colored cables are the public network.The PS4000XV storage arrays provide the physical storage capacity for the Oracle 11g RACdatabase; host servers access the data through a single group IP address. A PS Series group isa storage area network comprised of one or more PS Series arrays. There can be up to twoPS4000 Series arrays in a PS Series group. Each array in a group is referred to as a member.All members in a group are managed and accessed as a single storage system using the groupIP address. A group or a member has a name assigned and each member must have a RAIDlevel specified when it is initialized: RAID 5, RAID 6, RAID 10, or RAID 50.As illustrated in Figure 4, the group named oracle-group includes two PS4000XV members:oracle-member01 and oracle-member02. A PS Series storage group can be segregated intomultiple tiers or pools; tiered storage provides administrators with greater control over how diskresources are allocated. At any one time, a member can be assigned to only one pool. It is easyto assign a member to a pool, and also to move a member between pools, with no impact to dataavailability. Pools can be organized according to different criteria, such as disk types or speeds,Reference Configuration for Oracle 11g SE RAC on Dell EqualLogic PS4000XV Storage Arrays 11
  12. 12. RAID levels, or application types. In Figure 4, there is a pool with the name RAID-10 thatconsists of RAID 10 members.Before data can be stored, the PS4000XV physical disks must be configured into usablecomponents, known as volumes. A volume represents a portion of the storage pool, with aspecific size, access controls, and other attributes. A volume can be spread across multiple disksand group members, and is seen on the network as an iSCSI target. Volumes are assigned to apool, and can be easily moved between pools, with no impact on data availability. In addition,automatic data placement and automatic load balancing occurs within a pool based on the overallworkload of the storage hardware resources within the pool. For details on volume configurationfor an Oracle RAC database, refer to the Configuring Volumes section.At the host level, in order to provide sufficient bandwidth to support two PS4000XV arrays it isrecommended that each Oracle 11g RAC node have multiple gigabit NIC ports with independentpaths to both gigabit Ethernet switches of the iSCSI SAN. By utilizing the Linux Device Mapper(DM) Multipathing on the cluster node, I/O can be balanced across NIC ports as well. As shownin Figure 4, each host server has separate connections to the redundant gigabit Ethernetswitches and to the dual control modules. In this configuration, there is pathway redundancy atthe host, at the two Ethernet switches, and at the dual RAID control modules. On each server thefour NIC ports used for iSCSI traffic, as well as the two NIC ports used for Oracle privateinterconnect network, are configured on separate NIC cards to protect against a single PCI busfailure.At the Ethernet switch interconnection level, the bandwidth of the inter-switch link is critical tosupport high-performing applications. In the Oracle configuration shown in Figure 4, the two DellPowerConnect 6248 gigabit Ethernet switches used for routing iSCSI traffic are stacked togetherwith 48 Gbps bandwidth.Configuring VolumesOracle ASM is a feature of Oracle 11g that provides vertical integration of the file system andvolume manager. ASM distributes the I/O load across all available resources to optimizeperformance, while removing the need for manual I/O tuning by spreading out the database filesto avoid “hotspots.” ASM helps DBAs manage a dynamic database environment by allowing themto grow the database size without having downtime to adjust the storage allocation.Oracle 11g RAC database storage can be divided into three shared storage areas:• Oracle Cluster Registry (OCR) and Cluster Synchronization Services (CSS) Voting Disk shared storage. The OCR stores the cluster configuration details, including the names and current status of the database, associated instances, services, and node applications such as the listener process. The CSS Voting Disk is used to determine which nodes are currently available within the cluster. Unlike traditional database files, these files cannot be placed on the disks managed by ASM because they must be accessible before the ASM instance starts. These files can be placed on block devices that are shared by all the RAC nodes.• Actual Oracle database that is stored in the physical files including the data files, online redo log files, control files, SPFILE for the database instances, and temp files for the temporary table spaces. The volumes(s) in this area are used to create the ASM disk group and managed by ASM instances.• Oracle Flash Recovery Area shared storage that is an optional storage location for all recovery-related files, as recommended by Oracle. If configured, the disk-based database backup files are all stored in the Flash Recovery Area. The Flash Recovery Area is also the default location for all archived redo log files, multiplexed copies of control files, and online redo log files. The size of the Flash Recovery Area will depend on what and how much is being stored.Reference Configuration for Oracle 11g SE RAC on Dell EqualLogic PS4000XV Storage Arrays 12
  13. 13. Table 1 shows a sample volume configuration, with volumes for each of the three previouslydescribed shared storage areas.Volume Minimum Size RAID Number of Used For OS Mapping PartitionsFirst Area 1024 MB 10 Three of Voting disk, Oracle Three block devices,Volume 300 MB Cluster Registry each for Voting Disk, each (OCR), and SPFILE OCR, and SPFILE for ASM instanceSecond Larger than the 10 One Data ASM disk groupArea size of your DATABASEDGVolume(s) databaseThird Area Depends on 10 One Flash Recovery ASM disk groupVolume(s) the size and Area FLASHBACKDG activity levels on the databaseTable 1: Volumes for the Oracle RAC ConfigurationRAID 10 is considered the optimal choice for Oracle 11g RAC LUN implementation because itoffers fault tolerance, excellent read performance, and outstanding write performance. ThePS4000XV array member where the data is allocated should be configured with RAID 10.In a PS Series storage group, access control records are used to control which hosts can accessa volume. Each PS Series volume has a single list of access control records. In each record,you can specify an IP address, iSCSI initiator name, or Challenge Handshake AuthenticationProtocol (CHAP) user name (or any combination of the three). A server must match all therequirements in one record in order to access the volume. The most secure way to controlaccess to your volumes is to use a combination of an IP address and a CHAP user name. Forexample, if a record includes both an IP address and a CHAP user name, a server must presentthe IP address and supply the CHAP user name and its associated password in order to matchthe record.Each storage volume will be presented to all the Oracle 11g RAC hosts and configured at the OSlevel. For details on the shared storage configuration at the OS level, refer to the “ConfiguringHost Access to the iSCSI Volumes,” “Configuring Shared Storage for the Oracle Clusterware,”and “Configuring Shared Storage for the Database Using the Automatic Storage Management(ASM)” sections.As previously discussed, the Flash Recovery Area is an optional disk-based recovery area thatcan be used to store files for backup and recovery operations. If users choose not to configurethe disk-based Flash Recovery Area, PS Series arrays provide alternative methods to performOracle database disk-based backups: snapshots, clones, and replications. Snapshots enablepoint in time copies of volume data that can be used for backups. Snapshots are the most space-efficient method of volume protection, and utilize reserve space to hold the delta between theoriginal base volume data and the snapshot data. Clones are a complete point-in-time copy of thebase volume. The replication feature allows volume data to be replicated to a secondary orremote PS Series storage group.Configuring Challenge Handshake Authentication Protocol (CHAP)As mentioned in the preceding section, the Challenge Handshake Authentication Protocol(CHAP) can be used to restrict host access to PS Series storage volumes. CHAP is an iSCSIauthentication method that authenticates access between the storage volumes (targets) and theiSCSI initiators on the host servers. CHAP is an optional feature, and is not required to useiSCSI. However, with CHAP authentication, volume access can be restricted to hosts that supplyReference Configuration for Oracle 11g SE RAC on Dell EqualLogic PS4000XV Storage Arrays 13
  14. 14. the iSCSI initiator with the correct user name and password. This information must match anaccess control record for the volume, in addition to an entry in a CHAP database, in order to gainaccess to the volume.iSCSI SAN Gigabit Ethernet Switch ConfigurationConfiguring iSCSI SAN NetworkFor the best performance, the iSCSI SAN network should be isolated from other networks bydedicating gigabit Ethernet switches for iSCSI traffic, or using VLANs to separate networks withina switch. As illustrated in Figure 1, the iSCSI network is physically isolated from the clientnetwork by using two dedicated Dell PowerConnect 6248 gigabit Ethernet switches.To achieve optimal performance, follow these SAN network guidelines for the PS Series iSCSIstorage network: • Use the Rapid Spanning Tree Protocol (RSTP) and enable the PortFast setting on the switch ports between switches. The Spanning Tree Protocol (STP) is a link management protocol that prevents loops in an Ethernet network by ensuring that only one active path exists between switches. Upon linkup, a switch performs a 30 to 50 second STP calculation to transition ports into forwarding or blocking state. STP can increase the time it takes to recover from a PS Series array control module failover or a network switch failure, so it is recommended to enable switch port settings that allow the immediate transition of the port into STP forwarding state upon linkup; this can reduce network interruptions that occur when devices restart. For example, the Dell PowerConnect 6248 gigabit Ethernet switch includes a feature called PortFast that immediately transitions a port into STP forwarding state upon linkup. It is also preferable to use Rapid Spanning Tree Protocol (RSTP) instead of STP. RSTP allows a switch port to bypass the STP listening and learning states, and quickly enter the STP forwarding state. • Enable flow control on switch ports and NIC ports. When the data transmissions from network senders exceed the throughput capacity of the network receivers, the receivers may drop packets forcing senders to retransmit the data after a delay. Although this will not result in any data loss, latency will increase due to the data packets re-transmissions resulting in I/O performance degradation. The flow control feature is designed to allow network receivers to slow down network senders to avoid data re-transmissions, and the delay time is much less than the overhead of re-transmitting packets. It is recommended to enable flow control on all switch ports and NIC ports that handle iSCSI traffic. • Disable unicast storm control on switch ports. Many switches have traffic storm control features that prevent ports from being disrupted by broadcast, multicast, or unicast traffic storms on physical interfaces. These features typically work by discarding network packages when the traffic on an interface reaches a certain percentage of the overall load. Because iSCSI traffic is unicast traffic and can typically use the entire link, it is recommended that the unicast storm control feature be disabled on switches that handle iSCSI traffic. However, the use of broadcast and multicast storm control is encouraged on all switches. • Enable jumbo frames on switches and NICs. Jumbo frames extend the standard Ethernet frame size to allow more data to be transferred with each Ethernet transaction. It is recommended to enable jumbo frames on switches that handle iSCSI traffic and on the NICs.Reference Configuration for Oracle 11g SE RAC on Dell EqualLogic PS4000XV Storage Arrays 14
  15. 15. For more information on PS Series network configuration best practices and recommendations,refer to the Dell EqualLogic white paper “PS Series Groups: Storage Array Network PerformanceGuidelines” available at http://www.equallogic.com/psp/PDF/tr-network-guidelines-TR1017.pdf.A high bandwidth, inter-switch link is critical to achieve optimal iSCSI SAN performance. Asillustrated in Figure 3, the two Dell PowerConnect 6248 gigabit Ethernet switches are stackedwith 48 Gbps inter-switch bandwidth.Server ConfigurationEach Oracle 11g RAC database cluster node should be architected for optimal availability. Thefollowing sections will detail how to set up the Oracle private interconnect network and iSCSI SANEthernet interfaces. These are the two channels used by the database nodes to communicatewith each other and to the storage system. Ensuring that these interfaces are fault tolerant willhelp increase the availability of the overall system.Configuring Fully Redundant Ethernet InterconnectsIn addition to the iSCSI storage area network Ethernet interfaces, each Oracle 11g RACdatabase server needs at least three additional network interface cards (NIC) ports: one port forthe external interface, and two ports for the private interconnect network. The servers in anOracle 11g RAC are bound together using cluster management software called OracleClusterware that enables the servers to work together as a single entity. Servers in the clustercommunicate and monitor cluster status using a dedicated private network, also known as thecluster interconnect or private interconnect. One of the servers in the RAC cluster is alwaysdesignated as the master node.In a non-redundant deployment, if the private interconnect or a network switch fails and servercommunication to the master node is lost, then the master node will initiate a recovery of thefailed database instance on a different server. This recovery is initiated to ensure that the criticaldata contained in the database will remain consistent and not become corrupted. The masternode will then proceed to recover all of the failed instances in the cluster before providing aservice from a single node that will result in a significant reduction in the level of service andavailable capacity. Therefore, Dell recommends that users implement a fully-redundantinterconnect network configuration, with redundant private NICs on each server and redundantprivate network switches.Figure 4 illustrates the CAT 5E/6 Ethernet cabling of a fully-redundant interconnect networkconfiguration of a two-node PowerEdge RAC cluster, with two private NICs on each server, andtwo private network switches, as shown in the red colored cables that represent the Oraclecluster private interconnect network. For this type of redundancy to operate successfully, itrequires an implementation of the Link Aggregation Group, where one or more links are providedbetween the switches themselves. These two private interconnect network connections workindependently from the public network connection.To implement a fully-redundant interconnect configuration requires the implementation of NICteaming software at the operating system level. This software operates at the network driverlevel to provide two physical network interfaces to operate underneath a single IP address. Fordetails on configuring NIC teaming, refer to “Configuring the Private NIC Teaming” in the followingsection.Configuring Multiple Ethernet Interfaces for iSCSI Storage Area NetworksFor high availability and optimal performance, it is recommended that multiple NIC ports are usedon each Oracle RAC host to communicate with the PS Series iSCSI storage arrays. A minimumof two NIC ports are required to provide redundant links to the storage arrays; in the event of aReference Configuration for Oracle 11g SE RAC on Dell EqualLogic PS4000XV Storage Arrays 15
  16. 16. NIC port failure in a single NIC port environment, the host will lose access to the storage until thefailed NIC is physically replaced. As illustrated in Figure 4, four NIC interfaces on each of thePowerEdge servers hosting the Oracle 11g RAC database are configured to route iSCSI traffic.Software ConfigurationOperating System ConfigurationConfiguring Host Access to iSCSI VolumesA number of volumes are created in the PS4000XV storage array to use as the Oracle storagespace. In order to access these volumes, the iSCSI initiator software needs to be installed andconfigured. The Open-iSCSI initiator enables the connection of a host running Red HatEnterprise Linux 5 to an external iSCSI storage array. During the installation of the open-iSCSIinitiator, ensure that the Initiator Service, the Software Initiator, and the Microsoft MPIOMultipathing Support for iSCSI installation options are all selected.On hosts running Red Hat Enterprise Linux 5, accessing these volumes requires the open-iSCSIinitiator rpm iscsi-initiator-utils to be installed. Once a connection has been establishedbetween the host and the external iSCSI storage, host initiators can be created. It is a bestpractice to have a host initiator for each physical NIC port to ensure high availability. Aftercreating the host initiators, bind the physical NIC port with the iSCSI host initiator using the NIC’shardware address. The example below shows the creation of a host initiator, and binding a NIC’shardware address to the iSCSI initiator.iscsiadm –m iface –I ieth# --op=newiscsiadm –m iface –I ieth# --op=update –n iface.hwaddress –v <MAC ADDRof eth#>Once your iSCSI interfaces have been created and attached to the appropriate NIC ports, targetscan be discovered. The following command discovers the targets at a given IP address.iscsiadm –-mode discovery –-type sendtargets –-portal <ip_address>Once all your Ethernet devices have been added, the configuration of a host accessing youriSCSI volumes is complete once you log the target sessions using the following command:iscsiadm –m node –-loginall=allFor hosts running RHEL 5, a device mapper allows them to have multiple connected paths to thesame SAN volume for increased redundancy, performance, and I/O rebalancing across all activepaths. To utilize the device mapper, the rpm package device-mapper-multipath must beinstalled. Once installed, verify that the iSCSI sessions are up and running using the iscsiadm–m session command.The installation of the rpm package will create a default /etc/multipath.conf file that willneed modification. Open the /etc/multipath.conf file and comment out the line devnode“*” from the file, as this will blacklist all block devices by default. Identify your local storage IDand exclude it from your /etc/multipath.conf configuration file under the blacklist section.The SCSI ID can be found using the following command:iscsi_id –gus /block/sd#where # is the letter of your local storage.Reference Configuration for Oracle 11g SE RAC on Dell EqualLogic PS4000XV Storage Arrays 16
  17. 17. Lastly, modify the /etc/multipath.conf template file with the appropriate information for yourexisting environment. It is important to remember that it is a best practice for enterpriseenvironments to be highly available, with multiple hosts that each have multiple paths to storageand volumes.Once your multipath configuration file is set, start the multipathd service by entering thefollowing command:service multipathd startConfiguring the Oracle Private Network NIC TeamingDell recommends that users install two physical private NICs on each of the Oracle 11g RACcluster servers to prevent private network communication failures. In addition to installing the twoNICs, NIC teaming software is required to bond the two private network interfaces together tooperate under a single IP address. The NIC teaming software provides failover functionality. If afailure occurs that impacts one of the NIC interfaces – such as a switch port failure, cabledisconnection, or NIC failures – network traffic is routed to the remaining operable NIC interface.The failover is transparent to the Oracle 11g RAC database, with no network communicationinterruption or changes to the private IP address.Configuring Network Requirements on All NodesIt is important that all nodes in an Oracle 11g RAC cluster have the same network interface namefor the public interface. For example, if “eth0” is configured as the public interface on the firstnode, then “eth0” should also be selected as the public interface on all of the other nodes. This isrequired for the correct operation of the Virtual IP (VIP) addresses configured during the OracleClusterware software installation.It is important to verify the network configuration using ping tests prior to the Oracle installation.The public IP address and the private IP address must be resolvable by pinging from each nodeto any other nodes of the cluster. Oracle provides the Cluster Verification Utility (CVU) to verifythe cluster configuration at any stage during the installation process, and also when the databaseis in production. The following command checks the general hardware and operating systemconfiguration and may be used to verify network connectivity between all cluster nodes:cluvfy stage –post hwos –n node_list [-verbose]Configuring Host EquivalenceDuring the installation of Oracle 11g RAC software, the Oracle Universal Installer (OUI) is initiatedon one of the RAC cluster nodes. OUI operates by copying files to, and running commands on,the other servers in the cluster. A secure shell (SSH) must be configured to allow OUI toaccomplish these tasks, and so that no prompts or warnings are received when connectingbetween hosts. Use the following CVU command to check the host equivalence configuration:cluvfy stage –post hwos –n node_list [-verbose]Configuring Shared Storage for Oracle Clusterware Using Block DeviceBefore installing Oracle 11g RAC Clusterware software, it is necessary, at a minimum, for sharedstorage to be available on all cluster nodes to be used by the OCR and the Clusterware CSSVoting Disk. As shown in Table 1, the sample volume configuration includes one volume for theOCR and the Voting Disk files. Starting with Oracle 11g, the OCR and CSS Voting disk files canbe placed on a shared block device.Reference Configuration for Oracle 11g SE RAC on Dell EqualLogic PS4000XV Storage Arrays 17
  18. 18. Configuring Shared Storage for the Database Using the ASM Library DriverTwo separate volumes are created for the data storage area and the Flash Recovery Area,respectively. It is recommended that these two volumes be configured as ASM disks to benefitfrom the ASM capabilities. For Oracle 11g R1 databases on RHEL 5 update 3, ASM requires theinstallation of a number of additional rpm packages including:• oracleasm-2.6.18-128.el5-2.0.5-1.el5.x86_64.rpm – the kernel module for the ASM library that is specific to kernel 2.6.18-128.el5• oracleasm-support-2.1.3-1.el5.x86_64.rpm – the utilities needed to administer ASMLib• oracleasmlib-2.0.4-1.el5.x86_64.rpm – the ASM librariesASM allows DBAs to define a storage pool called a disk group; the Oracle kernel manages the filenaming and placement of the database files in that storage pool. DBAs can change the storageallocation by, adding or removing disks with SQL commands such as create diskgroup, alterdiskgroup, and drop diskgroup. The disk groups can also be managed by the OracleEnterprise Manager (OEM) and the Oracle Database Configuration Assistant (DBCA). EachOracle 11g RAC node will contain an ASM instance that has access to the backend storage. TheASM instance, similar to database instance, communicates with other instances in the RACenvironment and also features failover technology. Use the following CVU command to check theaccessibility of the shared storage:cluvfy stage –post hwos –n node_list [-verbose]Oracle 11g R1 ConfigurationInstalling Oracle Clusterware and Database SoftwareThe preferred method to install Oracle Clusterware the database is to use OUI. OUI provides aninstallation wizard to install the Clusterware and database binaries on Red Hat Enterprise Linux.During the Clusterware and Oracle installation, the OUI will ask for general information such asinventory directory paths, multi-node information, and so on. The OUI RAC deployment feature isenhanced with the ability to push the required binaries to multiple RAC nodes from one masterserver.The general installation order is as follows: 1. Install Oracle 11g R1 (11.1.0.6) Clusterware software. 2. Install Oracle 11g R1 (11.1.0.6) database software. 3. Upgrade Oracle Clusterware software to 11.1.0.7 patch set. 4. Upgrade Oracle database software to 11.1.0.7 patch set. 5. Create the listener and cluster database.Installing Recommended Oracle PatchesOracle releases many patches with each database release to fix issues and enhance securityfeatures. It is critical to keep database patches up-to-date to ensure a healthy and secureenvironment. Check the Dell Oracle 11g deployment guide at www.dell.com/oracle for the Oracleand Dell recommended patch list.Reference Configuration for Oracle 11g SE RAC on Dell EqualLogic PS4000XV Storage Arrays 18
  19. 19. Performance Capabilities of Oracle 11g SE RACTest Tools and ConfigurationQuest Benchmark Factory TPC-CTo understand the performance capabilities of Oracle 11g SE RAC, the Dell Oracle Solutionsengineering team conducted a series of benchmark stress tests using the Benchmark Factory®for Databases from Quest software. While Benchmark Factory offers numerous industry-standard benchmarks, the team selected benchmarks similar to the TPC-C benchmark from theTransaction Processing Performance Council (TPC). This benchmark measures onlinetransaction processing (OLTP) workloads, combining read-only and update-intensive transactionsthat simulate the activities found in a complex OLTP enterprise environment. The BenchmarkFactory TPC-C tests conducted by the team simulated loads from 100 to 10,000 concurrent usersin an increment of 100. The test outputs include metrics such as transactions per second (TPS).The Solutions Engineering team set up the software to run on the cluster nodes as follows: • A single node Oracle 11.1.0.7 SE database: 100 to 10,000 users • Two-node Oracle 11.1.0.7 SE RAC database: 100 to 10,000 users • Three-node Oracle 11.1.0.7 SE RAC database: 100 to 10,000 users • Four-node Oracle 11.1.0.7 SE RAC database: 100 to 10,000 usersThe goal of these tests was to determine: • How many concurrent users each database may sustain • How the RAC cluster would scale as additional nodes are addedHardware and Software ConfigurationThe database used in the test was running Oracle 11g (11.1.0.7) Standard Edition RealApplication Cluster (RAC). The total database schema size was 160 GB, populated byBenchmark Factory. The Benchmark Factory test configuration is summarized in Table 2.Server Up to four Dell PowerEdge R610 servers with: • A single Intel® Xeon® X5560 quad-core 2.80 GHz CPU with 8M cache, 6.40 GT/s QPI and TURBO and HT mode enabled • 24 GB of RAM or 48 GB of RAM, with 4GB or 8GB DIMMs respectively • Four 1Gb Broadcom NetXtreme II NIC ports for iSCSI trafficExternal Storage Two DellTM EqualLogicTM PS4000XV iSCSI storage arrays, each with 16 15K RPM 146GB SAS hard drivesVolume Configuration Three 220 GB volumes for database files; One 150 GB volume for Flash Recovery area; One 2 GB volume for OCR, CSS, and SPFILEOS and Device Driver Red Hat Enterprise Linux 5 Update 3 • Open iSCSI initiator iscsi-initiator-utils-6.2.0.868-0.18.el5 • Device Mapper multipath driver device-mapper-multipath- 0.4.7-23.el5Storage Network Two stacked Dell PowerConnect 6248 gigabit Ethernet switches for iSCSI SANTest Software • Quest Benchmark Factory 5.7.1 • Oracle 64 bit 11.1.0.7 SE RACDatabase Configuration Up to four Oracle 11.1.0.7 SE RAC with: • 13 GB memory_target on each instanceTable 2: Hardware and Software Configurations for Benchmark Factory TestReference Configuration for Oracle 11g SE RAC on Dell EqualLogic PS4000XV Storage Arrays 19
  20. 20. Performance MonitoringThe testing utilized a number of monitoring tools to characterize cluster components. Byanalyzing and interpreting the statistics generated by these tools, we can identify where theperformance bottlenecks occur.Three system performance statistics were sampled during the performance stress runs: • CPU utilization • Physical memory utilization • Storage system throughputSARThe system activity report (sar) is a Linux command to collect, report, and save system activityinformation. It reports system performance information, with its default CPU utilization reportbeing the most commonly used utility to monitor CPU performance. If CPU utilization is near 100percent, the workload sampled may be CPU-bound.The sar command uses two user specified parameters – an interval time in seconds betweensampling periods, and a count parameter to set the number of sampling periods in total. The testteam executed the following command to collect database server CPU performance during eachbenchmark factory runs:nohup sar 1 30000 > ~/sar-4nodes-run195-node1.txt &In this example, CPU performance is collected every 1 second for 30000 seconds, approximately8 hours. The output is saved to a text file named sar-4nodes-run195-node1.txt. Figure 5is an excerpt of the CPU performance information displayed by the command:Linux 2.6.18-128.el5 (r610-n2.us.dell.com) 06/11/2009 CPU %user %nice %system %iowait %steal %idle02:31:43 PM all 0.38 0.00 0.25 4.25 0.00 95.1202:31:44 PM all 0.25 0.00 0.25 4.12 0.00 95.3802:31:45 PM all 0.12 0.00 0.50 0.25 0.00 99.1302:31:46 PM all 0.62 0.00 0.62 1.12 0.00 97.6202:31:47 PM all 0.87 0.00 1.12 1.99 0.00 96.0102:31:48 PM all 0.13 0.00 0.38 5.63 0.00 93.8702:31:49 PM all 0.62 0.00 0.37 1.87 0.00 97.13...02:33:29 PM all 0.38 0.00 0.12 0.25 0.00 99.2502:33:30 PM all 0.00 0.00 0.25 0.00 0.00 99.7502:33:31 PM all 0.25 0.00 0.62 0.25 0.00 98.8802:33:32 PM all 0.37 0.00 0.25 0.00 0.00 99.3802:33:33 PM all 0.50 0.00 0.50 1.13 0.00 97.8702:33:34 PM all 0.25 0.00 0.25 0.25 0.00 99.25Figure 5: Sample section of the SAR report showing CPU statistics during each intervalThe %idle column shows the CPU idle time percentage; the system CPU utilization can becalculated as 100 - %idle.The sar reports are time stamp based. In order to interpret the impact of the Benchmark Factorystress test on CPU performance, we developed a utility to correlate the sar report’s CPUperformance numbers with the Benchmark Factory user load by mapping the time stampsbetween the two tools.A sample of the Benchmark Factory’s “Real-Time Detail” report is shown in Figure 6. The originalreport did not list the time stamp column, so the Solutions Engineering team calculated it byadding the performance run start time stamp with the value in the “Elapsed Seconds” column.Reference Configuration for Oracle 11g SE RAC on Dell EqualLogic PS4000XV Storage Arrays 20
  21. 21. Real-Time DetailCounter UserLoad Sequence Test Phase Elapsed Seconds Value Time StampAVG_RESP_TIME 100 33 128 168 85 2:31:38 PMAVG_RESP_TIME 100 34 128 173 69 2:31:43 PMAVG_RESP_TIME 100 35 128 178 80 2:31:48 PMAVG_RESP_TIME 100 36 128 183 158 2:31:53 PM....AVG_RESP_TIME 200 54 32 279 39 2:33:29 PMAVG_RESP_TIME 200 55 32 284 1032 2:33:34 PMAVG_RESP_TIME 200 56 32 289 670 2:33:39 PMAVG_RESP_TIME 200 57 32 294 523 2:33:44 PM....Figure 6: Sample of the Benchmark Factory Real-Time Detail ReportIn order to generate a relationship between CPU utilization and the user load, the SolutionsEngineering team wrote a series of scripts that would look up the user load and thecorresponding CPU utilization at the same point in time; an example can been seen in thehighlighted results in Figure 5 and Figure 6. Once data points were captured for the CPUutilization percentage (100 - %idle) at different user loads, graphs were plotted to show thisrelationship.OS WatcherIn addition to CPU utilization, the engineering team examined the memory characteristics of eachdatabase cluster node. For simplicity, the team made the assumption that the characteristicsexhibited for one node within a cluster may be generalized to the other nodes within the cluster,due to the ability of Oracle RAC to evenly balance the workload within a cluster.The memory statistics were captured using OS Watcher (OSW). OSW is a collection of Linuxshell scripts used to collect and archive operating system metrics to aid in diagnosingperformance issues. OSW operates as a set of background processes on the server, andgathers OS data on a regular basis. The OSW utility is started by executing the startOSW.shshell script from the directory where OSW is installed. The script has two input parameters thatcontrol the frequency that data is collected and the number of hour’s worth of data to archive. Inthe following example command, OSW collects data every 5 seconds and archives the data fromthe past 4 days:nohup ./startOSW.sh 5 96 &One of the statistics that OSW collects is system memory usage. OSW saves the output of/proc/meminfo based on the user specified value. The sample /proc/meminfo output inFigure 7 shows a system with a total of 48 GB of RAM and has 34 GB of free physical memory.For each poll time that measured free memory, the engineering team developed a utility to crossreference the corresponding time stamp in the Benchmark Factory Real-Time Detail report (asshown in Figure 6), in order to plot memory usage versus user load. Although buffers and cacheswould be considered “free” memory, those data points were not charted within the graphs.zzz ***Sun Jun 21 23:00:04 CDT 2009MemTotal: 49434036 kBMemFree: 34270184 kBBuffers: 353068 kBCached: 12631124 kBSwapCached: 7732 kBActive: 10277800 kBInactive: 4483064 kBHighTotal: 0 kBHighFree: 0 kBLowTotal: 49434036 kBLowFree: 34270184 kBSwapTotal: 8388600 kBReference Configuration for Oracle 11g SE RAC on Dell EqualLogic PS4000XV Storage Arrays 21
  22. 22. SwapFree: 8239712 kB…zzz ***Sun Jun 21 23:04:59 CDT 2009MemTotal: 49434036 kBMemFree: 34270564 kBBuffers: 354892 kBCached: 12637136 kBSwapCached: 7732 kBActive: 10278772 kBInactive: 4483804 kBHighTotal: 0 kBHighFree: 0 kBLowTotal: 49434036 kBLowFree: 34270564 kBSwapTotal: 8388600 kBSwapFree: 8239712 kBFigure 7: Sample Section of OSW Memory ReportEqualLogic SAN HeadQuartersThe Dell EqualLogic SAN HeadQuarters (SAN HQ) is a centralized event and performancemonitoring tool for Dell EqualLogic storage systems. SAN HQ enables administrators to monitormultiple EqualLogic storage groups from a single graphical interface. It gathers and formatsperformance data and other vital group information, such as I/O performance and capacity, andso on. Figure 8 displays a screen shot of SAN HQ I/O performance view.Figure 8: Dell EqualLogic SAN HeadQuarters I/O Performance ViewTo better understand current or past SAN attribute characteristics, administrators can use asimple slide bar in SAN HQ to select and view a long or short time window. SAN HQ also canexport data sets into a .csv file for analysis and reporting.The test team used SAN HQ to collect storage I/O performance data in I/O per second (IOPS)and disk latency data points. This data was used to understand the EqualLogic PS4000XVperformance during the Benchmark Factory stress tests.Reference Configuration for Oracle 11g SE RAC on Dell EqualLogic PS4000XV Storage Arrays 22
  23. 23. Tuning Oracle Memory SizeOn the test database baseline Benchmark Factory tests were conducted to optimize databasesettings including the size of Oracle SGA memory and PGA memory. A larger SGA setting helpsreduce the amount of physical I/O. However, the relationship is not linear. The optimal size ofOracle SGA can get to the point where the marginal benefit begins to decline. For our testconfiguration, the Oracle AWR (Automatic Workload Repository) report obtained from baselinetesting shown in Figure 9 indicated that the optimal SGA size is in the range of 9 - 12 GB. Thereduction in physical reads is minimized when SGA is larger than 12 GB.SGA Target AdvisorySGA Target Size (M) SGA Size Factor Est DB Time (s) Est Physical Reads 3,168 0.38 1,327,870 37,779,644 4,224 0.50 895,776 20,459,187 5,280 0.63 728,220 13,742,069 6,336 0.75 641,710 10,272,675 7,392 0.88 598,910 8,558,231 8,448 1.00 569,142 7,364,453 9,504 1.13 549,509 6,576,457 10,560 1.25 534,597 5,979,936 11,616 1.38 524,011 5,555,743 12,672 1.50 519,685 5,381,942 13,728 1.63 516,046 5,236,863 14,784 1.75 516,112 5,236,863 15,840 1.88 516,117 5,236,863 16,896 2.00 516,119 5,236,863Figure 9: Baseline Tests AWR Report Section on SGA AdvisorySimilarly, the AWR report revealed that the optimal PGA setting should be set when theestimated PGA over-allocation count is 0. The AWR report obtained from baseline testing shownin Figure 10 indicated that the optimal PGA size is roughly 4 GB.PGA Memory Advisory • When using Auto Memory Mgmt, minimally choose a pga_aggregate_target value where Estd PGA Overalloc Count is 0PGA Target Size W/A MB Estd Extra W/A MB Estd PGA Estd PGA Estd Est (MB) Factr Processed Read/ Written to Disk Cache Hit % Overalloc Count Time 912 0.25 564.98 15.25 97.00 113 1,740 1,824 0.50 564.98 15.25 97.00 77 1,740 2,736 0.75 564.98 15.25 97.00 50 1,740 3,648 1.00 564.98 0.00 100.00 17 1,694 4,377 1.20 564.98 0.00 100.00 0 1,694 5,107 1.40 564.98 0.00 100.00 0 1,694 5,836 1.60 564.98 0.00 100.00 0 1,694 6,566 1.80 564.98 0.00 100.00 0 1,694 7,296 2.00 564.98 0.00 100.00 0 1,694Figure 10: Baseline Tests AWR report section on PGA AdvisoryBeginning with Oracle 11g, the database can manage SGA and PGA memory automatically. Anew database parameter named memory_target was introduced in Oracle 11g to designate thetotal memory size to be used by the instance. This capability is referred to as Automatic MemoryReference Configuration for Oracle 11g SE RAC on Dell EqualLogic PS4000XV Storage Arrays 23
  24. 24. Management. The size of the memory_target should be the total size of SGA and PGA area.The test team set the memory_target parameter to 13 GB, which is the sum of the optimal PGAand SGA size suggested by the AWR report.Testing RAC Scalability with 24 GB RAM Per NodeCPU Performance AnalysisIn the scalability test, the test team started with one node then scaled the Oracle RAC cluster upto four nodes. As a consequence, there can be up to four instances of SAR data for each clusterdepicting the CPU utilization for each of the four nodes. The graphs below depict only oneselected server’s SAR report that shows CPU utilization for one node; a single node is shown, butis representative of the others due to the ability of Oracle RAC to evenly distribute workloads inthe cluster.As a starting point, each database server has 24 GB of RAM installed; figures 11, 12, and 13show the CPU utilization versus the user load for each node with 24 GB of physical memory. CPU utilization of a node within a 2-node RAC cluster with 24 GB RAM per node 120 100 CPU utilization (%) 80 60 40 20 0 100 200 300 400 500 500 600 700 800 900 1000 1100 1100 1200 1300 1400 1500 1600 1700 1700 1800 1900 2000 2100 2200 2300 2300 2400 2500 User Load 2-node RAC, 24GB RAM 40 periods moving averageFigure 11: CPU Utilization vs. User Load - 2-node RAC with 24 GB Memory per NodeReference Configuration for Oracle 11g SE RAC on Dell EqualLogic PS4000XV Storage Arrays 24
  25. 25. CPU utilization of a node within a 3-node RAC cluster with 24 GB RAM per node 120 100 CPU utilization (%) 80 60 40 20 0 2300 2400 2400 2400 2500 2500 2500 2600 2600 2600 2700 2700 2700 2800 2800 2800 2800 2900 2900 2900 3000 3000 3000 3100 3100 3100 3200 3200 3200 User Load 3-node RAC, 24GB RAM 40 periods moving averageFigure 12: CPU Utilization vs. User Load - 3-node RAC with 24 GB Memory per Node CPU utilization of a node within a 4-node RAC cluster with 24 GB RAM per node 120 100 CPU utilization (%) 80 60 40 20 0 300 400 500 700 800 1000 1100 1300 1400 1500 1700 1800 2000 2100 2300 2400 2500 2700 2800 3000 3100 3300 3400 3500 3700 3800 4000 4100 4200 User Load 4-node RAC, 24GB RAM 40 periods moving averageFigure 13: CPU Utilization vs. User Load - 4-node RAC with 24 GB Memory per NodeIn each of the figures above, the black line indicates the moving average CPU utilizationpercentage. A few key general trends were observed: • As we add each node to the Oracle RAC cluster, not only do we scale out with the total number of users supported by the RAC cluster, but the moving average for CPU utilization per node also goes down. For example, with two nodes in the cluster, the moving average CPU utilization for a 2400 user load is approximately 50% (Figure 11), and as we add one more node to the cluster making it a 3 node cluster, the moving average CPU utilization for the same user load is down to 30% (Figure 12). We further observe this trend with the graph in Figure 13 that shows the moving average CPU utilization reduces further to approximately 20% for the same 2400 user load when we add another node to a 3 node RAC cluster.Reference Configuration for Oracle 11g SE RAC on Dell EqualLogic PS4000XV Storage Arrays 25
  26. 26. • The overall CPU utilization still had room for more growth. For example, when Oracle was not able to handle any more user connections due to other bottlenecks, for each of the RAC clusters the CPU utilization was around 60-65%.Thus the graphs indicate that even when scaled up to four single socket nodes with 24 GB ofmemory per node, the total number of users that the configuration can support was not limited bythe CPU.Memory Utilization AnalysisFigure 14 displays the memory utilization as it relates to increasing user loads. The red linerepresents the free memory available on one node as the user load increases in a 3-node RACcluster, while the blue line represents the same information in a 4-node RAC cluster. Each ofthese clusters has 24 GB of physical memory per node. In both cases, the graph was generatedusing data from a single node within an N node cluster. It may be assumed that these lines arerepresentative of the other cluster nodes because of the ability of Oracle RAC to evenly distributeworkloads. Memory utilization of RAC node with 24 GB RAM per node 100 Free memory (GB) on each RAC node 10 4 node RAC, 24 GB RAM 1 3 node RAC, 24 GB RAM ~ 125 MB free memory per node 0.1 1000 1200 1400 1600 1900 2100 2300 2600 2800 3000 3200 3500 3700 3900 4100 100 300 500 700 User LoadFigure 14: Memory Utilization vs. User Load – 3, 4-Node RAC with 24 GB RAM per NodeFor a three node cluster, the available physical memory is approximately 125 MB during runs with2300 – 3200 users; during this run the logs show swap usage.When the system is scaled to 4 nodes, represented by the blue line, at approximately 3900 usersthe system started to level out with 125 MB of available physical memory. Until the user loadreaches 4300 we see the use of swap, due to insufficient physical memory to handle theadditional transactions from Benchmark Factory. The operating system can accommodate theseincreasing transactions only by swapping out some existing pages in physical memory to thevirtual memory or swap disk; this is represented by the flattening lines in the figure above.Storage System Performance AnalysisSince two Dell EqualLogic PS4000 Series arrays can be allowed in a PS Series group, our testconfiguration was limited to two members. The Dell EqualLogic SAN HeadQuarters (SAN HQ)performance monitoring tool was used to capture the IOPS and disk latencies. IOPS data wasextracted from the compressed output generated by SAN HQ during the stress tests. TheseReference Configuration for Oracle 11g SE RAC on Dell EqualLogic PS4000XV Storage Arrays 26
  27. 27. extracted data contained the average IOPS and disk latency data points in 30-minute timeintervals. SAN HQ allows you to extract these data points from the monitoring tool into an Excelspread sheet that was used to plot Figure 15 below. All of the IOPS data points were well withinthe acceptable disk latency range of 20 ms or less. PS4000XV IOPS with 24 GB RAM in each RAC node 3500 3000 PS4000XV array IO Per Second 2500 2000 1500 member1 24GB RAM member2 24GB RAM 1000 500 0 1 2 3 4 Number of Oracle RAC nodesFigure 15: IOPS vs. # of RAC Nodes - PS4000XV with 24 GB RAM per NodeFigure 15 shows the member array IOPS performance during the scalability tests, ranging from asingle node to a 4-node Oracle RAC cluster. As shown in the figure, with an increasing number ofRAC nodes with 24 GB of physical memory, the two storage members were able to scale steadilyand reach a maximum of 1800 IOPS per member. This IOPS value is well within the maximumIOPS, and acceptable disk latency, that a single Dell EqualLogic PS4000XV array can deliver.The workload is equally balanced between the PS4000XV members, as the database volumedata blocks are evenly distributed between the two members.Examining both CPU and Memory UtilizationIf we examine CPU usage in Figure 12, when the 3-node cluster begins to swap (Figure 14), CPUutilization at a user load of 2400 was approximately 28%. As Benchmark Factory continued toincrease the user load, the swapping continues as the system is forced to swap to disk becauseof the lack of free physical memory. The stress test starts to fail once the user load reaches3200, and at this point we note that the CPU utilization was approximately 42%. Similar results(Figure 13 and 14) can be observed with the 4-node cluster, where we can see that the systemstarts to have low physical memory availability when the user load reaches 3800 with CPUutilization approximately around 40%. The test starts to fail once the user load reaches 4300,and at this point we note that CPU utilization is approximately 50%.During the swap time we can continue to load the database, but it’s likely in a yellow zonebecause as we keep increasing the user loads the database servers start to get unstable due to alow amount of free memory available.Reference Configuration for Oracle 11g SE RAC on Dell EqualLogic PS4000XV Storage Arrays 27
  28. 28. Overall Performance AnalysisFigure 16 shows the final Benchmark Factory results for all the nodes with 24 GB of physicalmemory per node. This result shows that while the RAC scaled as nodes were added, thescalability is less than linear. Benchmark Factory results for 1 to 4 RAC nodes with 24 GB RAM per node 250 200 Transaction Per Second 150 4 nodes 100 3 nodes 2 nodes 50 1 node 0 100 300 500 700 900 1100 1300 1500 1700 1900 2100 2300 2500 2700 2900 3100 3300 3500 3700 3900 4100 4300 User LoadFigure 16: TPS vs. User Load - 1 to 4 RAC Nodes with 24 GB RAM per NodeFrom all the above critical system’s performance metrics, there are few common trends: • As the systems near their maximum memory performance, they progressively utilize more swap space and the database starts to get unstable. • When comparing the CPU graph with the memory graph, at the maximum user load there is still ample CPU capability even once the test fails. • Given that the Oracle memory area (memory_target) is set to 13 GB to minimize physical I/O, even at the maximum 4-node cluster size the storage members were delivering IOPS well below their potential, suggesting that the storage disks were not the bottleneck for the overall performance.Therefore, we concluded that due to low available free physical memory, aggressive swapping,and low CPU utilization at failure time, the clusters are not balanced in respects to memory toCPU utilization ratios; this system’s performance was limited by memory. In order to confirm thisconclusion, the next step in our testing was to increase the memory from 24 GB to 48 GB pernode, in other words to increase the memory in a PowerEdge R610 server with 1 CPU socket tothe maximum.Testing RAC Salability with 48 GB RAM Per NodeFor the next testing phase, we scaled-up our clusters and upgraded each node in the cluster to48 GB of physical memory while keeping rest of the configuration the same. As of the authoringof this paper, 48 GB (8 GB RDIMMs * 6 Memory Banks) is the maximum memory capacity that aDell PowerEdge R610 can support per processor socket. As part of our scalability study, sinceeach cluster node has only one processor socket populated, all 48 GB of memory DIMMs mustbe placed in the banks adjacent to the populated processor due to the NUMA architecture of theserver system. Although the R610 server has additional free memory banks available next to theother processor socket, these banks cannot be used as the socket must be populated.Reference Configuration for Oracle 11g SE RAC on Dell EqualLogic PS4000XV Storage Arrays 28
  29. 29. For the 48 GB cluster, Benchmark Factory ran again on one, two, three, and finally four nodeswithin the cluster and an incremental user load of 100 on the database was applied until thedatabase failed.The same tool sets and scripts that were used for the 24 GB cluster were again used to correlateCPU utilization and memory utilization, with the corresponding user load for each of the nodeswithin the cluster. The following sections and graphs present the different performance metricscaptured, and the analysis for this configuration.CPU Performance AnalysisWith each node configured with 48 GB of physical memory, Figures 17-20 chart CPU utilizationon a point for point basis in relationship to the user load at a particular common time stamp withthe black line representing a 30 point moving average. Similar to the 24 GB cluster’s CPUperformance characteristics, observation and comparison of the 4 graphs demonstrate that for agiven user load CPU utilization continue to decrease as more nodes are added to the cluster. Forexample, at a 2300 user load the CPU utilization for the 1, 2, 3 and 4 node clusters areapproximately 60-80%, 20-40%, 20-35% and 10-25% respectively. CPU utilization of a node within a 1-node RAC cluster with 48 GB RAM per node 120 100 CPU utilization (%) 80 60 40 20 0 200 200 300 400 500 500 600 700 800 900 900 1000 1100 1200 1200 1300 1400 1500 1600 1600 1700 1800 1900 1900 2000 2100 2200 2300 2300 User Load 1-node RAC, 48GB RAM 30 periods moving averageFigure 17: CPU Utilization vs. User Load - 1-Node RAC with 48 GB RAM per NodeReference Configuration for Oracle 11g SE RAC on Dell EqualLogic PS4000XV Storage Arrays 29
  30. 30. CPU utilization of a node within a 2-node RAC cluster with 48 GB RAM per node 120 100 CPU utilization (%) 80 60 40 20 0 2300 2400 2500 2500 2600 2700 2800 2800 2900 3000 3000 3100 3200 3300 3300 3400 3500 3500 3600 3700 3700 3800 3900 4000 4000 4100 4200 4200 4300 User Load 2-node RAC, 48GB RAM 30 periods moving averageFigure 18: CPU Utilization vs. User Load - 2-Node RAC with 48 GB RAM per Node CPU utilization of a node within a 3-node RAC cluster with 48 GB RAM per node 120 100 CPU utilization (%) 80 60 40 20 0 300 500 700 800 1000 1200 1400 1600 1700 1900 2100 2300 2500 2600 2800 3000 3200 3400 3500 3700 3900 4100 4300 4400 4600 4800 5000 5100 5300 User Load 3-node RAC, 48GB RAM 30 periods moving averageFigure 19: CPU Utilization vs. User Load - 3-Node RAC with 48 GB RAM per NodeReference Configuration for Oracle 11g SE RAC on Dell EqualLogic PS4000XV Storage Arrays 30
  31. 31. CPU utilization of a node within a 4-node RAC cluster with 48 GB RAM per node 120 100 CPU utilization (%) 80 60 40 20 0 200 400 600 900 1100 1400 1600 1800 2100 2300 2600 2800 3000 3300 3500 3800 4000 4300 4500 4700 5000 5200 5500 6700 6900 7200 7400 7600 7800 User Load 4-node RAC, 48GB RAM 30 periods moving averageFigure 20: CPU Utilization vs. User Load - 4-Node RAC with 48 GB RAM per NodeComparing the 24 GB cluster’s CPU utilization graphs with the 48 GB cluster’s CPU utilizationgraphs, we can see that the CPU was utilized much more in the 48 GB cluster that resulted in asignificantly higher user load that could be sustained by the clusters. For example, the 3-nodecluster with 24 GB of RAM per node was able to support only up to 3200 users while the 3-nodecluster with 48 GB of RAM per node was able to support up to 6200 users. Similarly, the 4-node24 GB cluster was able to support up to 4300 users while the 4-node 48 GB cluster was able tosupport up to 8100 users. Given that the only thing that was changed between the twoconfigurations was doubling the physical memory, this suggests that the CPUs were underutilizedin the 24 GB configuration with memory being the issue.Memory Utilization AnalysisFigure 21 shows that eventually the free physical memory on the system will run out. The 48 GBclusters for both three and four nodes eventually leveled out to approximately 250 MB of freememory available, at which point some swap memory was utilized. In all cases, the databaseeventually failed as higher and higher user loads were applied.Reference Configuration for Oracle 11g SE RAC on Dell EqualLogic PS4000XV Storage Arrays 31

×