SUN REFERENCE ARCHITECTUREFOR ORACLE 11g GRIDWhite PaperOctober, 2008
Sun Microsystems, Inc.Table of ContentsExecutive Summary ....................................................................
3        Executive Summary                                             Sun Microsystems, Inc.Executive SummaryIn the endle...
4        Scope                                                        Sun Microsystems, Inc.ScopeThis document is intended...
5       IT Challenges and Grid Computing                               Sun Microsystems, Inc.Chapter 1IT Challenges and Gr...
6       IT Challenges and Grid Computing                               Sun Microsystems, Inc.Key BenefitsSun’s Reference A...
IT Challenges and Grid Computing                           Sun Microsystems, Inc.• Minimize costs. IT organizations are al...
8        Sun’s Reference Architecture for Oracle 11g Grid               Sun Microsystems, Inc.Chapter 2Sun’s Reference Arc...
9           Sun’s Reference Architecture for Oracle 11g Grid                                    Sun Microsystems, Inc.Refe...
10       Sun’s Reference Architecture for Oracle 11g Grid               Sun Microsystems, Inc.Sun SPARC™ Enterprise T5140 ...
11       Sun’s Reference Architecture for Oracle 11g Grid              Sun Microsystems, Inc.Solaris Cluster HA functional...
12      Sun’s Reference Architecture for Oracle 11g Grid             Sun Microsystems, Inc.InfiniBand TechnologyInfiniBand...
13      Sun’s Reference Architecture for Oracle 11g Grid             Sun Microsystems, Inc.Figure 2-2 illustrates the two ...
14      Sun’s Reference Architecture for Oracle 11g Grid               Sun Microsystems, Inc.Figure 2-3. The management ne...
15      Sun’s Reference Architecture for Oracle 11g Grid              Sun Microsystems, Inc.Application Logical Architectu...
16       Sun’s Reference Architecture for Oracle 11g Grid                 Sun Microsystems, Inc.System ComponentsTable 2-2...
1         Sun’s Reference Architecture for Oracle 11g Grid                              Sun Microsystems, Inc.Table 2-2. S...
18       Performance Characterization                                 Sun Microsystems, Inc.Chapter 3Performance Character...
19          Performance Characterization                                   Sun Microsystems, Inc. Test Summary • Interconn...
20       Performance Characterization                                Sun Microsystems, Inc.• Cache latency at cluster inte...
21      Performance Characterization                                 Sun Microsystems, Inc.The testing effort yielded the ...
22         Best Practices                                                Sun Microsystems, Inc.Chapter 4Best PracticesThis...
23         Best Practices                                                 Sun Microsystems, Inc.Network ndd tuning     Net...
24         Best Practices                                                    Sun Microsystems, Inc.QFS Mount options     S...
25         Best Practices                                               Sun Microsystems, Inc.DB ConfigurationsLMS Binding...
26       Summary: The Sun Advantage                                    Sun Microsystems, Inc.Summary: The Sun AdvantageIT ...
2         References                                                             Sun Microsystems, Inc.ReferencesMore info...
Sun Reference Architecture for Oracle 11g Grid                                                                            ...
Upcoming SlideShare
Loading in...5
×

Oracle11g arch

1,501

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,501
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
33
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Oracle11g arch"

  1. 1. SUN REFERENCE ARCHITECTUREFOR ORACLE 11g GRIDWhite PaperOctober, 2008
  2. 2. Sun Microsystems, Inc.Table of ContentsExecutive Summary ........................................................................................ 3Scope ............................................................................................................ 4IT Challenges and Grid Computing.................................................................... 5Sun’s Reference Architecture for Oracle 11g Grid ............................................... 8Reference Architecture Design ............................................................................. 8Reference Architecture Components..................................................................... 9 Sun SPARC™ Enterprise Servers ......................................................................... 9 Sun StorageTek™ 6140 Fiber Channel Array ...................................................... 10 Solaris™ 10 Operating System ......................................................................... 10 Solaris Cluster Advanced Edition for Oracle RAC............................................... 10 Sun xVM™ Ops Center ................................................................................... 11 Oracle Real Application Clusters 11g ............................................................... 11 Oracle Enterprise Manager 10g Grid Control ................................................... 11 InfiniBand Technology ................................................................................... 12 10GbE Technology ........................................................................................ 12Connecting the Network, Servers, Applications, and Storage .............................. 12Application Logical Architecture ........................................................................ 15 System Components...................................................................................... 16Performance Characterization........................................................................ 18Workload ......................................................................................................... 18Data Collection ................................................................................................ 18Test Matrix ....................................................................................................... 18Test Summary .................................................................................................. 19Best Practices............................................................................................... 22Storage ............................................................................................................ 22Network ........................................................................................................... 22Database Server ................................................................................................ 23DB Configurations ............................................................................................. 25Summary: The Sun Advantage ...................................................................... 26References ................................................................................................... 27
  3. 3. 3 Executive Summary Sun Microsystems, Inc.Executive SummaryIn the endless quest to do more with less, grid computing has emerged as one of themost effective tools in the IT arsenal. And one of the newest and most effective usesfor grid computing is enterprise database implementations.Enterprise grid and database software must be deployed on solid infrastructure—butthe endless choices can be daunting. Sun’s Reference Architecture for Oracle 11g Gridhelps ease this effort. Designed around proof-of-concept deployments and tested,validated configurations, Sun’s Reference Architecture for Oracle 11g Grid providesa high-performance, highly available and reliable database infrastructure in a cost-effective grid environment. Deployed on the latest hardware and software from Sunand incorporating software from Oracle, this reference architecture includes failovermechanisms and redundancy to eliminate single points of failure. By using theseadvanced products and technologies, customers can implement a robust, scalabledatabase platform that maintains or increases service levels.Sun’s Reference Architecture for Oracle 11g Grid is built on the Solaris™ OperatingSystem (OS) and takes full advantage of its indisputable performance advantages fordatabase services, its world-record benchmark results, and optimizations that yieldeven faster performance without requiring changes to existing applications. Runningon Sun’s new extremely fast Sun SPARC™ Enterprise servers and coupled with SolarisCluster™ software and Sun StorageTek™ QFS software, this reference architectureprovides an ideal platform for a powerful database grid solution.A key strength of this reference architecture, and one of its greatest benefits, is itsability to integrate complementary elements from both Sun and third-party vendors.Vendor interoperability fosters a more adaptable enterprise and enables computingresources and bandwidth to be re-provisioned to meet changing business demands.By combining low-cost servers with other architecture components, Sun’s ReferenceArchitecture for Oracle 11g Grid balances cost, performance, and availability.The outcome: a flexible, scalable, powerful solution that can help reduce coststhroughout the solution life cycle.
  4. 4. 4 Scope Sun Microsystems, Inc.ScopeThis document is intended for executives evaluating the use of a horizontallyscalable grid for database application deployment, as well as project managers,system engineers, integration specialists, and other technical staff looking todesign, implement, and tune database applications in a grid computing environ-ment. It provides an overview of database deployment challenges and options,and describes how Sun’s Reference Architecture for Oracle 11g Grid delivers on theserequirements. An architectural overview and descriptions of the hardware andsoftware components used in the reference architecture are also included. Notethat only the features and techniques directly relevant to the reference architectureare discussed here. For more complete information about Sun servers and storage,Oracle Real Application Clusters (RAC) 11g, InfiniBand switches, or other componentsof this architecture, please refer to the product documentation from Sun, Oracle, andInfiniBand switch vendor.
  5. 5. 5 IT Challenges and Grid Computing Sun Microsystems, Inc.Chapter 1IT Challenges and Grid ComputingPressure is mounting on IT executives to deploy reliable, high-performance databasesolutions within severely constrained capital and expense budgets. Expectations arehigh for horizontally scalable solutions that can maintain or increase applicationavailability, reliability, and performance while supporting service levels and simulta-neously reducing total cost of ownership (TCO).These requirements demand integrated solutions that are increasingly complex todesign, implement and maintain, and can require specialized technical and businessexpertise. Rapid changes in business cycles and increasing demand for computepower are prompting IT organizations to look for alternative solutions.Today, grid computing is shifting from a mere concept to a viable computing frame-work. IT organizations are now turning to the grid computing model for solutionsthat can adapt to changing business needs, deliver greater efficiency, minimizecosts, and provide investment protection and rapid return on investment (ROI).Organizations have high expectations for these new IT solutions, and the outcomeof implementations is critical. As a result, IT organizations should consider severalquestions when evaluating a grid computing solution:• Is low-cost grid architecture a viable option?• What are the cost factors in scaling a grid horizontally?• How do additional connections, software licensing fees, and implementation and management of more complex systems impact solution costs?• Is the grid solution capable of providing a reliable, horizontally scalable platform for running enterprise-grade, commercial applications at acceptable service levels?• What are the components of the grid architecture?• How are applications distributed across servers?• Where are the potential bottlenecks?• In a database cluster, how does the system recover from a component failure?• With respect to interconnect technology, how do Gigabit Ethernet, 10GbE and InfiniBand compare in terms of latency, bandwidth, and overhead?• How can Sun products add value to the grid solution?• How do Sun products complement Oracle RAC deployments?In an effort to help customers address these questions, and to assist in evaluatinga grid computing solution, Sun has invested tremendous resources in developingthe Reference Architecture for Oracle 11g Grid. Sun Reference Architectures are inte-grated solutions consisting of specific combinations of hardware and software prod-ucts that also incorporate third-party offerings from top tier software and networkingequipment vendors. These solutions include a set of recommended components thatare sized, tested, tuned, and documented to work together, delivering specific func-tionality and reducing the complexity, costs, and risks of deploying new technology.
  6. 6. 6 IT Challenges and Grid Computing Sun Microsystems, Inc.Key BenefitsSun’s Reference Architecture for Oracle 11g Grid provides a horizontally scalablesolution that delivers high performance and service levels at low cost. Using productsfrom Sun and Oracle, Sun’s Reference Architecture for Oracle 11g Grid also offers thefollowing benefits.• Utilize a proven solution portfolio. Following the reference architecture assures a valid blueprint for deploying Oracle RAC 11g in a grid computing environment with the rigor of a Sun tested, certified, and supported configuration. The reference architecture also formulates best practices for an Oracle RAC 11g infrastructure with other best-of-breed third-party products within a grid context.• Maximize availability. Sun’s Reference Architecture for Oracle 11g Grid maximizes database availability by leveraging Oracle RAC 11g and Solaris Cluster software, and by deploying redundant components to eliminate single points of failure. Using a clustered grid infrastructure, Sun’s Reference Architecture for Oracle 11g Grid employs built-in load balancing and failover mechanisms to minimize interruption of service levels due to individual component failure. Oracle RAC 11g enables databases to span across multiple physical servers, eliminating the server as a single point of failure and ensuring high availability for applications and users.• Maximize reliability. Implementing Solaris Cluster software enhances the reliability and availability of Oracle RAC 11g and helps improve the overall resiliency of the architecture. Solaris Cluster 3.2 Advanced Edition software offers the advantages of a well-proven and solid cluster framework that is tightly integrated with the Solaris 10 OS kernel for the quick detection and handling of failures. Combined with an effective I/O fencing mechanism and full redundancy and load-balanc- ing of interconnect traffic, the Solaris Cluster software helps ensure a secure and dependable Oracle environment on the Solaris 10 OS. • Maximize performance. Sun’s low-cost, high-performance Sun SPARC Enterprise Servers with UltraSPARC™ T2 and T2 Plus processors deliver the core computing power behind the reference architecture. These servers can be equipped up to 128 compute threads per server, which provide extreme versatility and set a new standard for server availability, price/performance, and reduced ownership cost.• Provide flexible scalability. System infrastructures must allow organizations to increase system capacity and throughput by scaling up or scaling out — deploying larger servers, adding a number of small servers, or both. The ability of Oracle RAC 11g to span the database over multiple physical servers provides the flexible scal- ing so vital to the architecture design.
  7. 7. IT Challenges and Grid Computing Sun Microsystems, Inc.• Minimize costs. IT organizations are always looking for opportunities to reduce overall implementation costs, including costs of acquisition, deployment, and maintenance. Sun’s Reference Architecture for Oracle 11g Grid addresses budget- ary concerns by utilizing low-cost, robust commodity hardware and software.• Simplify management. One of the challenges of grid computing is managing the additional complexity of a grid infrastructure. Comprehensive management frame- work solutions, including the Oracle Enterprise Manager 10g Grid Control and Sun xVM Ops Center Manager software, are incorporated in the reference architecture. These tools provide click-of-a-button, automated, self-management capabilities that can be leveraged to manage grid complexity, simplify support, and drive down operating costs.
  8. 8. 8 Sun’s Reference Architecture for Oracle 11g Grid Sun Microsystems, Inc.Chapter 2Sun’s Reference Architecture for Oracle 11g GridReference Architecture DesignThroughout the process of designing Sun’s Reference Architecture for Oracle 11gGrid, certain requirements have been paramount. These requirements include reli-ability, high availability, resiliency, cost-effectiveness, a high degree of scalability,and use of best-of-breed products. By adhering to these fundamental criteria, andinvesting extensive effort in development, integration, testing, and tuning the archi-tecture, Sun technologists have produced a database grid ideally suited for enter-prise applications. Figure 2-1 depicts the reference architecture implementation.Figure 2-1. Sun’s Reference Architecture for Oracle 11g Grid implementation
  9. 9. 9 Sun’s Reference Architecture for Oracle 11g Grid Sun Microsystems, Inc.Reference Architecture ComponentsSun’s Reference Architecture for Oracle 11g Grid combines quality products fromtop-tier vendors (Table 2-1). By utilizing fully redundant components, the referencearchitecture delivers a highly robust, available, and reliable cluster grid infrastruc-ture at a low acquisition cost.Table 2-1. Hardware and software components in Sun’s Reference Architecture for Oracle 11g Hardware Components Software Components Sun SPARC Enterprise Servers with Ultra- Solaris 10, Update5 (5/08) Operating System SPARC T2 UltraSPARC T2 Plus Processors (Database Tier) Sun Fire Server with UltraSPARC Processor Oracle 11g R1 Clusterware Database (Application Management Tier) Sun StorageTek 6140 Fibre Channel Array Solaris Cluster 3.2 U1 (2/08) Advanced (Configured with Dual RAID Controllers) Edition for Oracle RAC software StorageTek QFS 4.6.5 InfiniBand Server Switches Oracle Enterprise Manager 10g R4 Grid Control Storage Area Network (SAN) Switches Sun xVM 1.1 Ops Center 10-Gigabit Ethernet Switches Gigabit Ethernet Switches Terminal ConsoleSun SPARC™ Enterprise ServersSun SPARC Enterprise Servers 1 with CoolThreads technology deliver higher perfor-mance in less space and power than any competitive servers. They are based onUltraSPARC T2 Ultra SPARC T2 Plus processors, the industry’s first massively thread-ed Systems on a Chip (SoC), offering breakthrough performance and energy efficiency.CoolThreads™ TechnologyThird-generation CoolThreads technology, based on Sun’s Chip Multithreading orCMT technology, supports up to 128 threads in as little as one rack unit (RU), provid-ing increased computational density within constrained envelopes of power andcooling. Very high levels of integration help reduce latency, lower costs, and improvesecurity and reliability. Balanced system design provides support for a wide range ofapplication types from Web services to high performance computing (HPC).Sun SPARC™ Enterprise T5120 ServerThe Sun SPARC Enterprise T5120 server, based on the UltraSPARC T2 processor,is the world’s first 64-thread, general purpose server. For more information visithttp://www.sun.com/servers/coolthreads/t5120__________1 Note: Testing of the Sun Reference Architecture for Oracle 11g Grid focused on Sun servers equipped with UltraSPARC T2 UltraSPARC T2 Plus processors.
  10. 10. 10 Sun’s Reference Architecture for Oracle 11g Grid Sun Microsystems, Inc.Sun SPARC™ Enterprise T5140 ServerThe Sun SPARC Enterprise T5140 Server is the world’s first dual-socket, general-pur-pose server powered by the third-generation UltraSPARC T2 Plus “System on a Chip”processor. It comes with up to 128 compute threads per server. For more informationand technical specifications visit http://www.sun.com/servers/coolthreads/t5140.Sun StorageTek™ 6140 Fiber Channel ArrayThe Sun StorageTek 6140 Fiber Channel array delivers high performance, high avail-ability, and reliability all in one economical package. The array offers affordableenterprise-class features and functionality, such as dual hot-swap power and cooling,hot-swap redundant RAID controllers with mirrored cache, hot-swap disk drives,global and local hot sparing, dynamic LUN expansion, dynamic capacity expansion,non-disruptive firmware code loading, and remote status monitoring. It simplifiesmanagement and configuration and “pay-as-you-grow” scalability protects yourinvestment. Utilizing the Sun StorageTek 6140 Fiber Channel Array in the referencearchitecture offers reliability and impressive I/O performance in a small footprint.Solaris™ 10 Operating SystemSupported on over a thousand x86 and SPARC platforms, the Solaris 10 OS deliversthe performance, stability and security enterprise grid environments demand. Withmore applications available than for any other open operating system, one OS canspan your entire enterprise: the Web tier, the data warehouse, and the most de-manding technical compute applications.The latest version of the Solaris OS offers new features including virtualization,volume management support, and advanced real-time application debugging(DTrace) to meet today’s enterprises demand. The Solaris OS is capable of verticaland horizontal scaling and offers broad hardware platform, which are very importantcapabilities for enterprise grid solutions.Solaris Cluster Advanced Edition for Oracle RACSolaris Cluster Advanced Edition software for Oracle RAC provides enterprises withadditional high availability for Oracle RAC on the Solaris 10 OS. In addition to provid-ing I/O fencing to help guarantee data integrity, this product suite leverages twokey components of Solaris Cluster software to deliver high performance. The SunStorageTek QFS shared file system software helps eliminate administrative overheadand provides maximum data management and throughput for the most data-inten-sive applications. The Solaris Volume Manager software provides partitioning andautomatic device relocation for data mirroring and high availability. Together, thesecomponents result in a high-performance clustered file system for traditional filesystem management and administration. Furthermore, the Solaris Cluster AdvancedEdition for Oracle RAC is tightly integrated with Oracle CRS and the Solaris OS to helpminimize application downtime and maximize application performance.
  11. 11. 11 Sun’s Reference Architecture for Oracle 11g Grid Sun Microsystems, Inc.Solaris Cluster HA functionality supports active/active configurations. Solaris Clusteruses a single logical private interconnect (clprivnet0), where two or more interfacesare combined together for high throughput and stripe traffic across all the interfaces.For further details on the benefits of deploying Solaris Cluster in Oracle RAC environ-ments, seehttp://www.sun.com/software/whitepapers/solaris10/solaris_cluster.pdfSun xVM™ Ops CenterThis integrated management tool makes it possible to manage thousands of Win-dows, Linux, and Solaris x64, x86 and SPARC systems. Administrators can discover,provision, update, and monitor both virtual and physical systems—all from onegraphical interface.Sun xVM Ops Center is also designed to enable remote management of heteroge-neous environments. With its state-of-the-art, browser-based AJAX GUI, there is nocompromise between user experience and portability. Virtual machines and thephysical hardware they run on can be managed from anywhere through a standardbrowser. xVM Ops Center software is specifically designed to work in existing data-centers, even in companies that aren’t currently running Sun technology.Oracle Real Application Clusters 11gOracle Real Application Clusters (RAC) 11g is the database designed for grid imple-mentations. A proven technology that allows multiple, low-cost servers to performlike a single large server, Oracle RAC forms a key foundation for enterprise databasegrids. With Oracle RAC, even very high-end systems can be constructed from small,very low-cost clusters made from standard, commodity parts. Running Oracle RACon a cluster provides the highest level of database availability along with flexibilityin scaling. If a node in the cluster fails, the Oracle software continues running on theremaining nodes. If more processing power is necessary, new nodes can easily beadded to the cluster. As a result, organizations can deploy low cost solutions whilereaping the benefits of high availability and adaptability.Oracle Enterprise Manager 10g Grid ControlOracle Enterprise Manager 10g Grid Control software enables administrators to easilymanage business applications, end user services, and the entire grid infrastructure.It contains improved tools such as new service modeling, broader support for serviceprotocols, and comprehensive policies and templates, enabling management ofgrids as a single entity. Graphical service topologies and service dashboards providethe right information to high level decision makers and line of business managers formore effective planning, while automated provisioning and patching functionalitygive IT administrators the power to make efficient, error-free changes.
  12. 12. 12 Sun’s Reference Architecture for Oracle 11g Grid Sun Microsystems, Inc.InfiniBand TechnologyInfiniBand is a high-performance, switched fabric interconnect that fosters greaterscalability and improved manageability. It can help accelerate database server speedand provide an added boost to clustered databases.Reliable Datagram Sockets (RDS) over InfiniBand can provide a horizontally scalable,high-performance alternative to traditional vertical scaling for enterprises usingOracle Database 11g and Real Application Clusters (RAC).10GbE TechnologySun’s 10 Gigabit Ethernet Networking Technology is one of the industry’s first networkinterfaces specifically designed to accelerate multithreaded clustered applicationperformance by optimizing I/O throughput in environments that utilize parallelthreads. It enables systems to:• Improve utilization of compute resources• Minimize system I/O latency• Leverage existing wiring infrastructure• Take advantage of more network bandwidthSun SPARC Enterprise CMT-based servers come with network ports, which improvesthe Oracle RAC cache fusion traffic over interconnect.Connecting the Network, Servers, Applications, and StorageInterconnect latency and throughput are essential elements in the scalability ofa database grid cluster. Inter-node communication and cache fusion traffic canbenefit from low latency, high throughput, and low system overhead.Deploying InfiniBand further aids integration of the elements of Sun’s ReferenceArchitecture for Oracle 11g Grid since InfiniBand leverages the high performanceReliable Datagram Sockets (RDS) protocol, a very low-latency remote direct memoryaccess (RDMA) based protocol driver. This RDS protocol can be used to create aunified fabric for inter-process communication (IPC).10GbE networking technology is designed to help accelerate clustered applicationsthat require high throughput and low latency for performance.The Sun Reference Architecture for Oracle 11g Grid is validated with Infiniband and10GbE interconnect technologies. Tests have shown the benefit of using high band-width and law latency interconnect over traditional GbE technology. For more detailsplease refer to Chapter 3.
  13. 13. 13 Sun’s Reference Architecture for Oracle 11g Grid Sun Microsystems, Inc.Figure 2-2 illustrates the two key network connection areas: the private interconnectand public network. Using high-throughput (InfiniBand or 10GbE) technologies forthe private interconnects between the database servers accelerates cache fusiontraffic and Oracle RAC inter-node communication, and results in very low latency.The public network connection enables traffic between the application and databaseservers.Figure 2-2. The private interconnect and the public networkThe management network is reserved for telnet access, basic health and monitoringof components, connection to the Service Processor module, and monitoring andcommunication by the Oracle Enterprise Manager 10g Grid Control and xVM OpsCenter Manager software (Figure 2-3). The terminal console is connected to thebuilt-in management port of the servers to enable console access.
  14. 14. 14 Sun’s Reference Architecture for Oracle 11g Grid Sun Microsystems, Inc.Figure 2-3. The management networkAll four database nodes share a pair of Sun StorEdge 6140 FC arrays via two SANswitches (Figure 2-4). This configuration, together with multiplexed I/O (MPxIO),helps provide full redundancy in the event of failures.Figure 2-4. The database servers share the two storage arrays through the SAN switches.
  15. 15. 15 Sun’s Reference Architecture for Oracle 11g Grid Sun Microsystems, Inc.Application Logical ArchitectureFigure 2-5 represents the logical structure of Sun’s Reference Architecture for Oracle11g Grid. The management software stack meets the critical need for a grid manage-ment framework by leveraging software from the Oracle Enterprise Manager 10gGrid Control, the Sun xVM Ops Center, and the SunPlex Manager / Solaris ClusterManager software. In addition, other key components form the foundation of thelogical architecture, including the Solaris 10 OS, Solaris Cluster Advanced Edition forOracle RAC software, and Oracle RAC 11g. Oracle Clusterware provides the databasecluster framework, and is enhanced by the use of the Solaris Cluster software, arobust and proven cluster framework that is tightly integrated with Solaris OS kernel.Oracle datafile storage options can be raw devices through Oracle Automatic StorageManagement (ASM), the Solaris Cluster file system with Sun StorageTek QFS sharedfile system software, or both.Figure 2-5. The logical structure of Sun’s Reference Architecture for Oracle 11g Grid
  16. 16. 16 Sun’s Reference Architecture for Oracle 11g Grid Sun Microsystems, Inc.System ComponentsTable 2-2 details the reference architecture hardware and software componentsdeployed, tested, and tuned in the Authorized SunSM Solution Center by Sun staff.Hardware and software specifications are for models and releases available at thetime the reference architecture was developed.Note, however, that any Sun Fire server can be used for the application and Sun SPARCEnterprise servers for database. For updates on available products, please refer to indi-vidual vendors’ web sites.Sun and Oracle technologists at the Sun Competency Center for Oracle in MenloPark, California designed, integrated, tested, and tuned the reference architecturefor arbitrary workloads generated with the iGenOLTP. Developed at Sun, iGenOLTPis a benchmark suite that mimics an online customer order entry application.The reference architecture was validated and tested with different interconnecttechnologies available, such as Gigabit Ethernet, InfiniBand, and 10GbE. The Oraclecluster interconnect is one of the important factors for overall system performance.See the performance characterization section.
  17. 17. 1 Sun’s Reference Architecture for Oracle 11g Grid Sun Microsystems, Inc.Table 2-2. Specific reference architecture components Servers Database Servers 4 x Sun SPARC Enterprise Servers 5120 (1 x UltraSPARC T2 processor, 32 GB RAM, 2 x 146 GB Drives) 4 x Sun SPARC Enterprise Servers 5140 (2 x UltraSPARC T2 Plus processor, 32 GB RAM, 2 x 146 GB Drives) Management Servers 1 x Sun Fire (8 x UltraSPARC processor, 8 GB RAM , 2 x 146 GB disks) Application Servers 6 x Sun Fire (8 x UltraSPARC processor, 8 GB RAM , 2 x 146 GB disks) Storage Arrays 2 x Sun StorageTek 6140 Fiber Channel Array Interconnect Switches 2 x GbE switch – 12 port 2 x InfiniBand switch – 12 port 2 x 10GbE switch – 12 port Public Switches 2 x 24 port GbE switch Storage Management Switch 24 port GbE switch SAN Switches 2 x 16-port Sun SAN switch Network Adapter Cards (For Database 4 x dual-port GbE cards Servers) 4 x dual-port Fiber-Channel cards 4 x dual-port InfiniBand cards Terminal Console 16-port terminal console OS Solaris 10 OS Cluster Software Oracle ClusterWare Solaris Cluster Advanced Edition for Oracle RAC (SC3.2U1) DB Software Oracle 11g Database Management Software Oracle Enterprise Manager 10g R4 Grid Control xVM Ops Center 1.1 Software Volume Manager Software Oracle Automatic Storage Management Solaris Volume Manager and Sun StorageTek QFS Shared File System 4.6 Workloads iGenOLTP Oracle 11g Client Miscellaneous Oracle 10g Grid Control Agent
  18. 18. 18 Performance Characterization Sun Microsystems, Inc.Chapter 3Performance CharacterizationThis chapter addresses the method adopted to test the Reference Architecture forstability and performance, and explains some of the benchmarking results from thetesting. Sun performed a variety of tests, including functional tests to validate theReference Architecture, and performance tests to show Oracle RAC scalability withSun products horizontally. More detail is provided in the test matrix section below.WorkloadThe iGEN-OLTP workload benchmark was used in the reference architecture illus-trated in Chapter 2. An internally-developed benchmark toolkit, iGENOLTP is basedon real customer workloads. It simulates a Global Order System that includes fourcontinent-based schema all accessed via central views. The transactions containedvarious SQL statements: read-only selects, update and insert operations.Data CollectionIn the testing effort Sun gathered all the OS and Oracle performance statistics andused them as input to the tuning effort. In this way, we believe we have achievedthe maximum workload for the reference architecture. The iGEN tool collects thestatistics such as number of transactions, TPS response time, etc.Test MatrixThe Sun Reference Architecture for Oracle 11g Grid was validated and tested onvarious cluster interconnect technologies such as Gigabit Ethernet, InfiniBand withRDS, and 10Gigabit Ethernet. The reference architecture solution was also validatedand tested on various storage options for RAC. Tests were performed to compare thestorage option available for Oracle Cluster, including Oracle ASM and Sun Shared QFSfile system.The tests were performed to stress the servers at different workload (CPU utilization)levels by increasing the number of connections to the database. The purpose of thetesting was to show how servers perform at different workloads, such as 50% CPUutilization, 5% CPU utilization and 100% CPU utilization.The testing was conducted on four-node RAC clusters. We scaled the RAC one nodeat a time, from one node up to four nodes, to show how the server scales horizontally(scales out) in a Grid RAC architecture.The tests were performed on the CMT family of servers (Sun SPARC Enterprise servers5120 and Sun SPARC Enterprise 5220 servers).
  19. 19. 19 Performance Characterization Sun Microsystems, Inc. Test Summary • Interconnect Tests: Cluster interconnect is most important factor for RAC perfor- mance and scalability. Interconnects with high bandwidth, low interconnect latency and low CPU usage are always preferred solutions for RAC Grid Architec- ture. We tested the Sun Reference Architecture over different interconnects. Some results are shown in Figures 3-1 and 3-2. Figure 3-1 AWR Global Cache Stats: GbE-InfiniBand-10GbE Figure 3-2 Response time The interconnect tests showed that: • 10GbE InfiniBand with RDS improves overall response time 2. • Cache latency at cluster interconnect is ½ with 10GbE interconnect compared with GbE interconnect.__________2 The response time also depends on other components such as I/O and CPU.
  20. 20. 20 Performance Characterization Sun Microsystems, Inc.• Cache latency at cluster interconnect is 1/3 with InfiniBand RDS compared with GbE interconnect.• Server CPU utilization is 10 to 20 % lower with 10GbE InfiniBand compared with GbE Interconnect.ASM vs Shared QFS TestsRAC Grid implementation requires shared storage. Oracle ASM and Sun Shared QFSwere the options considered best for the Sun Reference Architecture implementation.The following charts (Figure 3-3 and Figure 3-4) show the performance difference forOracle ASM and Sun Shared QFS.Figure 3-3 ThroughputFigure 3-4 Server resources
  21. 21. 21 Performance Characterization Sun Microsystems, Inc.The testing effort yielded the following results:• Performance results for Shared QFS Performance and ASM are comparable.• Shared QFS extends the storage option for RAC with ease of management. OS Utility can be used to monitor or manage the filesystem.• Throughput response time are nearly the same for both ASM QFS.• There is not much system overhead using Shared QFS compared to ASM.RAC ScalingSun scaled the RAC one node at a time, from one node up to four nodes. At a highlevel, Figure 3-5 shows server scaling results achieved. The baseline was created onone node by looking at the overall CPU utilization, and the user count was increasedto archive higher CPU utilization (while scaling from one node to two nodes Sundoubled the user count, and so on). The scalability factor was calculated basedon TPS achieved. Scalability Factor = TPS Achieved / TPS baseline (single node).Figure 3-5 Sun Reference Architecture for Oracle 11g Grid scaling.The tests demonstrated that the Sun servers showed linear scaling in the RAC/gridarchitecture.
  22. 22. 22 Best Practices Sun Microsystems, Inc.Chapter 4Best PracticesThis chapter provides some of the best practices followed while designing variousaspects of the Sun Reference Architecture for Oracle 11g Grid.StorageQFS Storage The Shared QFS file system layout and file creation options are the keys to high database performance. QFS has two file allocation schemes available: an ms file system type, and an ma file system type. For the reference architecture Sun used the ma file system type, where QFS is installed on multiple partitions where metadata is written to mm devices, and data can be written to md devices. Both devices should be sharing the same spindle. For more details see http://docs.sun.com/source/819-935-10/index.html. While creating the file system with sammkfs command specify 1 MB DAU size which reduces the I/O and increases the application performance. Enabling the SAMAIO for RAC datafiles provides a performance boost by decreasing CPU utilization. You can use setfa with the -q option to enable it.NetworkVirtual local area network Virtual local area networks (VLANs) are logical groupings of network devices. Using VLANs to segment different types of traffic to specific subnets provides better throughput, manageability, application separation and security. This is optional, but for large and complex networking situations it helps to separate VLANs for applications, storage and RAC traffic.Driver patch It is always recommended to have latest updated drivers on all the network devices on the system. Please check sunsolve.sun.com for more details. Refer to SPARC Enterprise Servers Product Notes. For the reference architecture Sun used 10GbE Cards with XAUI interfaces for the cluster interconnect. The following patches are recommended: 1255-01,125891-01,12546-02,1212-11,138048-01.
  23. 23. 23 Best Practices Sun Microsystems, Inc.Network ndd tuning Network tuning is one of the important factors affecting the overall system performance. Learn more at: http://www.solarisinternals.com/wiki/index.php/Networks. The following are specific tuning recommendations related to the reference architecture. ndd -set /dev/tcp tcp_smallest_anon_port 2048 ndd -set /dev/tcp tcp_conn_req_max_q 81920 ndd -set /dev/tcp tcp_conn_req_max_q0 81920 ndd -set /dev/tcp tcp_xmit_hiwat 800000 ndd -set /dev/udp udp_max_buf 4194304 ndd -set /dev/udp udp_xmit_hiwat 262144 ndd -set /dev/udp udp_recv_hiwat 262144Database ServerSolaris Update Firmware The reference architecture is built on Solaris 10 Update 5. Before deploying the grid, verify the version shipped with the server and determine if it is the latest production version supported. Always check for the latest update or patches at the Sun website. Make sure you have the latest firmware on the system. For the CMT servers Sun installed the latest firmware available: .1.4.a.Memory The reference architecture testing was done with servers using 32GB of total RAM memory. Please refer to the database server documentation to determine the total memory required for your implementation. Sun recommends that you configure your servers with the maximum memory feasible to meet your scalability and performance needs. Intimate Shared Memory (ISM) is a Solaris facility that enables large pages (4 MB) and locking pages in memory. So the only thing needed from a database point of view is to define the maximum memory that Oracle application should use. This is defined by Oracle parameter memory_max_target.
  24. 24. 24 Best Practices Sun Microsystems, Inc.QFS Mount options Shared QFS is one of the important components of the reference architecture. Tuning the shared QFS file system mount options improves the overall database performance. QFS Mount options are specified in /etc/opt/SUNWsamfs/samfs.cmd file. Table 4-1 describes some important mount options.Table 4-1: Mount Options Option Value Description Stripe 1 This sets the stripe width for the file system to n disk allocation units (DAUs). Set the 1 for DAUs = 128 kilobytes. For our testing the DAU size was set to 1M. mh_write Enables simultaneous reads and writes to the same file for multiple hosts. Qwrite Enables simultaneous reads and writes to the same file from different threads. Forcedirectio Specifies direct I/O as the default I/O mode (meaning the data is transferred directly between user’s buffer and disk). Rdlease 600 The read release time option specifies the maximum num- ber of seconds that the file can be read before reacquiring the read release. Default is 30. Wrlease 600 The write release time option specifies the maximum num- ber of seconds that the file can be written before reacquir- ing the write release. Default is 30. Aplease 600 The append release time option specifies the maximum number of seconds that the file can be appended before reacquiring the append release. Default is 30.
  25. 25. 25 Best Practices Sun Microsystems, Inc.DB ConfigurationsLMS Binding Set the LMS processes to the optimum level using ADDM to maximize this value. For the workload Sun used, we found 8 LMS processes were optimal to drive the work and improve the cache fusion traffic. Sun used gc_server_processes to set the parameter. During the testing Sun found creating the process set and binding the LMS process to the process set improved the interconnect traffic. Use psrset -c to create the process set and psrset -b to bind the LMS process ids to the process set. Binded the process set at 12:25. Figure 4-1 illustrates the results.Figure 4-1: Average Active Sessions
  26. 26. 26 Summary: The Sun Advantage Sun Microsystems, Inc.Summary: The Sun AdvantageIT managers are turning to grid computing to help deliver the capabilities neededwhile meeting budget constraints. However, coordinating hardware, software, andnetworking equipment is a daunting task. In order to minimize the uncertainty ofimplementing a grid computing solution, Sun has taken on the effort of evaluat-ing best-of-breed hardware and software and creating an optimized grid computingsolution for Oracle databases. The result of this effort is Sun’s Reference Architecturefor Oracle 11g Grid, a portfolio of Sun and third-party products designed to reducethe risks, uncertainty, and costs associated with implementing an Oracle databasewithin a grid computing environment.Designed, tested, implemented, and tuned at the Sun Competency Center for Oraclein Menlo Park, California, the reference architecture consists of recommended,integrated hardware and software stacks for a proven grid database solution. Alongwith this architecture, Sun provides a best practices framework for obtaining optimalperformance, availability, and resiliency.The most compelling benefits of Sun’s Reference Architecture for Oracle 11g Gridcome not from the individual components, but from the ability to integrate comple-mentary elements—whether they come from Sun or another vendor. The combina-tion of SPARC Enterprise servers with InfiniBand technology or 10GbE technologyensures higher database throughput, while the adoption of Solaris Cluster AdvancedEdition for RAC enhances the reliability and availability of Oracle RAC 11g. The resultis a solution that can help deliver the highest service levels and satisfy the mostdemanding business requirements of a database grid.For more information on applying this technology solution to an Oracle databaseenvironment, contact a local Sun sales representative.
  27. 27. 2 References Sun Microsystems, Inc.ReferencesMore information about Sun and Oracle technologies can be found in the referencesbelow.Web Sites of Interest Description Web Site URL Hardware Sun servers http://sun.com/servers Sun CoolThread Servers http://sun.com/servers/CoolThreads Cluster Solaris Cluster http://sun.com/software/solaris/cluster Solaris Cluster and Oracle 11g http://sun.com/software/solaris/cluster/ gallery/index.xml Software Solaris Cluster Open Storage http://www.sun.com/software/cluster/osp Program Solaris OS http://sun.com/solaris Oracle Database 11g http://oracle.com/technology/products/ database/oracle11g Oracle Real Application Clusters http://oracle.com/technology/products/ database/clusterware Storage xVM Ops-Center http://sun.com/software/products/ xvmopscenter Sun Storage http://sun.com/storage Sun StorageTek 6140 Fiber http://sun.com/storagetek/disk_systems/ Channel Array midrange/6140 Solaris Cluster Open Storage http://www.sun.com/software/cluster/osp Program Joint Sun and Oracle Joint Grid Initiative http://sun.com/third-party/global/oracle/grid Initiatives
  28. 28. Sun Reference Architecture for Oracle 11g Grid Sun Microsystems, Inc. Sun Microsystems, Inc. 4150 Network Circle, Santa Clara, CA 95054 USA Phone 1-650-960-1300 or 1-800-555-9SUN (986) Web sun.com© 2008 Sun Microsystems, Inc. All rights reserved. Sun, Sun Microsystems, the Sun logo, CoolThreads, StorageTek, Solaris, Sun Fire, and xVM are trademarks or registered trademarks of Sun Microsystems, Inc. in theUnited States and other countries. Products bearing SPARC trademarks are based upon an architecture developed by Sun Microsystems, Inc. PNWP14530-0 Printed in USA 10/08

×