Oracle Services on RAC: FiveThings You Might Not Know                                                                     ...
Oracle Services on RAC: Five Things You Might Not Know continued from page 13Consumers” dashboard showing various metrics ...
management page that is available in two places: the database creation wizard   3. Service Configuration Is Stored in Four...
Oracle Services on RAC: Five Things You Might Not Know continued from page 15straightforward, yet often overlooked. (It’s ...
and there is an associated overhead (such as maintaining cache coherencyand global locks); hence, the performance profile ...
Upcoming SlideShare
Loading in …5

Oracle services on rac


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Oracle services on rac

  1. 1. Oracle Services on RAC: FiveThings You Might Not Know ARDENTSALES.LAB.ARDENTPERF.COM = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = rh4lab10)(PORT = 1521)) (CONNECT_DATA = (SID = ORCL)) ) ARDENTACCT.LAB.ARDENTPERF.COM = (DESCRIPTION = (ADDRESS_LIST= (FAILOVER = on) (LOAD_BALANCE = on) (ADDRESS= (PROTOCOL = TCP)(HOST = rh4lab12-vip)(PORT = 1521)) (ADDRESS= (PROTOCOL = TCP)(HOST = rh4lab13-vip)(PORT = 1521)) (ADDRESS= (PROTOCOL = TCP)(HOST = rh4lab14-vip)(PORT = 1521)) ) (CONNECT_DATA= (SERVICE_NAME = Jeremy Schneider )T he use of Services is fundamental to increasing The manuals give us the first clues that there is more to services than meets the eye. One example is the “Service Registration” illustration from chapter 5 availability and performance of Oracle Databases. This of the Oracle Database Net Services Administrator’s Guide. It illustrates is true for both RAC and non-RAC databases, although independence between listeners and databases. Databases can register with multiple listeners and listeners redirect connections to multipleservices are especially important in RAC. In this article, we databases. The service is the glue holding this together: The PMON processlearn five things about services that are not well understood for each database instance registers its service names with one or multiple listeners. This is illustrated in the figure below.and see how they can help.Every application wants to be a high-performance, zero-downtime applicationwhen it grows up. And like our teenagers, applications are growing up fasterthese days. Whether driven by industry buzzwords or real requirements, itseems like every DBA I know wants to dramatically increase availability andperformance on their systems. In Oracle Databases, the use of services isfundamental to meeting these challenges. This is true for both RAC and non-RAC databases, although services are especially important in RAC. Since their introduction in the late ’90s services have steadily become moreand more advanced. Oracle Database 10g saw the greatest growth in services-related functionality and today they touch almost every aspect of databasemanagement. Understanding services is a basic, core responsibility of anyonewho plans or maintains Oracle cluster database systems. Nonetheless, it issurprising how often services are overlooked or misunderstood! As we willsee in this article, there is a lot more to services than meets the eye.Before we begin our deep dive, let us state that since services apply to bothRAC and non-RAC environments, we will discuss both aspects in this article, Illustration from Oracle Database Net Services Administrator’salthough we will focus on RAC. When applicable to single-instance non-RAC Guide, Chapter 5environments, we will point that out specifically. However, there is a lot more to services than just connecting clients to databases. They are also an important aid for performance tuning, resource1. Services Are a Lot More Than Just a Connect Method management, and job control.What is the first thing you think of when you hear the word “services” with One of the most important performance tuning tasks is capturing the rightOracle? For most database administrators – even RAC DBAs – the first thing historical and current performance data – and services are an integral part ofthat comes to mind is the TNSNAMES file. Or perhaps LISTENER.ORA and the Oracle Database 10g statistics collection. The core dynamic views V$SESSIONoutput of the lsnrctl utility come to mind. Version 8 of Oracle was the first and V$SQL report service information. Wait event information is recorded inversion to introduce the SERVICE_NAME clause for the CONNECT_DATA V$SERVICE_EVENT and then rolled up into V$SERVICE_WAIT_CLASS. Thesesection of TNSNAMES – and Oracle immediately recommended using it rather service-level statistics are automatically collected by the database and storedthan SID. However, at the time it was not immediately clear what the in the AWR. Furthermore, alerts can be defined for specific warning andadvantages were—particularly in a single-instance environment. critical response time thresholds by service. The instrumentation even extends to database control and grid control where there is a “Top continued on page 14 3rd Qtr 2008 n Page 13
  2. 2. Oracle Services on RAC: Five Things You Might Not Know continued from page 13Consumers” dashboard showing various metrics of each service at a glance If you are not using RAC (or MTS/Shared Servers), then getting started withrelative to the others. A sample screenshot is shown below. services is as easy as setting a single-instance parameter―and it does not even require a database restart! All you have to do is assign a comma-separated list of service names to the initialization parameter SERVICE_NAMES and Oracle will automatically create each service on the fly. Each service name that does not have a domain gets the DB_DOMAIN, and the PMON process will immediately register each of these services with the listener. (It will blindly try port 1521 by default; if your listener is on a different port, then set the LOCAL_LISTENER parameter to inform PMON of the correct port.) Always remember to subsequently make your changes in the init file or spfile so that your changes persist. However, setting up services on RAC is a little different. On cluster databases you must never, ever, use the above described method. In other words, do not set the SERVICE_NAMES parameter directly! Although you cannot use SERVICE_NAMES here, there are three different tools you can use to set up services on RAC: • Command line and Inbuilt PL/SQL functions • DBCA – The “DataBase Creation Assistant” • EM Database Control. The command line is the lowest level interface; both DBCA and EM Database Control call these commands under the covers. Command line configuration in RAC has two steps: configuring the clusterware and configuring the database. The clusterware is first configured through the srvctl command. Next, the database is set up using the DBMS_SERVICE inbuilt PL/SQL package. A command line dialogue that creates a service named “reports” is shown below:Services aid in resource management. The building blocks of Oracle’sresource management system are consumer groups. These consumer groups [oracle@rh4lab15 ~]$ srvctl add service -d db2rac1 are mapped to services, modules, and actions graphically using the > -s reports -r db2rac11,db2rac12 -P basicEnterprise Manager (EM) Database Control or programmatically through theDBMS_RESOURCE_MANAGER package. The database engine is then capable SQL> begin 2 dbms_service.create_service(of assigning execution priorities based on services and can even dynamically 3 service_name=>’reports’,change the execution priority at runtime as subroutines move from one code 4 network_name=>’reports’,block to the next. 5 goal=>dbms_service.goal_none, 6 dtp=>FALSE,Services also aid and enhance job control. First, all of the performance tuning 7 aq_ha_notifications=>FALSE,and resource management benefits mentioned above also apply to jobs when 8 clb_goal=>dbms_service.clb_goal_long 9 );they run within a service. You can easily view performance statistics for 10 end;groups of jobs and control execution priority of jobs by assigning them to 11 /services. Second, job control is enhanced because job execution is tied to PL/SQL procedure successfully completed.the status of its service: In a multi-node RAC environment, the job will onlyexecute on cluster nodes where the service is enabled. If a service is [oracle@rh4lab15 ~]$ srvctl start service -d db2rac1 completely disabled, then jobs assigned to that service will be suspended > -s reportsuntil the service is enabled. Note that you must use DBMS_SCHEDULER to [oracle@rh4lab15 ~]$ lsnrctl servicesmake use of this functionality – the older, deprecated DBMS_JOB system is Service “” has 2 instance(s)not integrated with services. Note, also, that you can apply these principles in Instance “db2rac11”, status READY, has 2 handler(s) Instance “db2rac12”, status READY, has 1 handler(s)single-instance, non-RAC environments as well.As we noted, services are a lot more than just a connect method. They add The command line offers the most flexibility and control; however, someuseful functionality for tuning, resource management and job control to all DBAs prefer to use a “point-and-click” interface when possible. Both of theOracle databases – even non-RAC systems! other setup methods are “point-and-click.”2. There Are Four Different Ways to Set up Services In Oracle Database 10g, DBCA can also be used to set up services – DBCA isThere are several ways to set up services, and the procedure varies slightly much more than just a database “creation assistant” – it is also a tool fordepending on your environment. There are four possible ways you can reconfigurations such as adding new cluster nodes and reconfiguring ASM.achieve this. However, for some strange reason, this functionality has been removed in Oracle Database 11g. DBCA in Oracle Database 10g also has a servicePage 14 n 3rd Qtr 2008
  3. 3. management page that is available in two places: the database creation wizard 3. Service Configuration Is Stored in Four Different Placesand directly from the main menu. DBCA offers the advantage that it Perhaps one of the most surprising and little-known facts about services isautomatically maintains the server’s TNSNAMES.ORA file as well. This is just how spread out the configuration is. Configuration for services is storedespecially useful for PRECONNECT failover configurations that require a in no less than four places: the spfile (or init file), the data dictionary, theconfiguration that is not immediately obvious. However, DBCA only performs cluster registry and the filesystem. Naturally the first place that comes to mindhalf of the service configuration—it only calls srvctl but does not call the is the initialization file, because that is where the SERVICE_NAMES parameterDBMS_SERVICE package. In this case, the service will be automatically is stored. But there are, in fact, quite a few additional initializationcreated in the database with default values on the first client connect. A parameters that are also related to service configuration. These are listed inscreenshot of DBCA (shown below) is being used to create services. the following Table. SERVICE_NAMES Should never be set in RAC databases. For single-instance databases only, set this parameter to create services. LOCAL_LISTENER PMON needs this parameter to register services with the listener on the local machine. REMOTE_LISTENERS PMON needs this parameter to register its services with listeners on other servers. DB_DOMAIN This is automatically appended to service names that do not have a domain. DISPATCHERS Optionally, dispatchers can be explicitly configured for each service. STATISTICS_LEVEL Must be TYPICAL or ALL to enable gathering service-related statistics. As might be expected, the data dictionary contains the bulk of service-related configuration. Most importantly, it contains a master list of all configured services and their properties. This includes metadata, load balancing configuration, notification options, server-side failover configuration and the DTP (Distributed Transaction Processing) flag. The data dictionary alsoThe EM Database Control is Oracle’s recommended tool for maintaining contains consumer group mappings, alert thresholds and mapping to jobservices. It is just as simple as DBCA, but it allows you to edit a number of classes. The relevant data dictionary views are listed below in the table.additional service properties. You can specify service level thresholds, DBA_SERVICES Lists all services registered with the databaseconsumer groups, job classes, notification properties, load balancing goals (including offline services) and their properties.and distributed transaction processing options. Database Control calls both It does not include deleted services.the srvctl program and the DBMS_SERVICE package. However, it does not DBA_RSRC_MAPPINGS Lists mappings between services and consumermaintain the TNSNAMES.ORA file, so you will have to write the networking groups. These are used by the resource manager toentries yourself. prioritize jobs and client connections at run-time.The fourth and final method is shown below using a screenshot of EM DBA_THRESHOLDS Lists thresholds for Oracle database’s built-in alertingDatabase control creating this very same service. system. Includes response-time alerts for services. DBA_SCHEDULER_JOB_CLASSES Displays mappings between scheduler jobs and services. In Oracle cluster databases, services are managed by the clusterware. That is why the cluster registry (OCR) is absolutely critical even though it only exists on RAC systems. There are two types of configuration data stored in the OCR: resources and stringpairs. You should never manually edit these settings. They can be viewed with the crsstat and ocrdump utilities. The srvctl utility automatically creates a resource with the appropriate properties and a number of stringpairs for each service you create. Some of the properties include the start/stop scripts, dependencies, restart policies, preferred/ available status, enabled/disabled status and TAF policies. Finally, there is an important bit of configuration stored in the filesystem: the TNSNAMES.ORA file. On the database server, this file contains the connection information for LOCAL_LISTENER and REMOTE_LISTENERS. On the clients,OEM Database Control is generally the preferred method for service of course, it contains the connect descriptors for connecting to the databaseconfiguration. However, it is good to be aware of the different options, and with load balancing and client-side failover. Getting this file right is fairlywe have listed all of them in this section. continued on page 16 3rd Qtr 2008 n Page 15
  4. 4. Oracle Services on RAC: Five Things You Might Not Know continued from page 15straightforward, yet often overlooked. (It’s surprising how many peoplemistakenly use the server IP address instead of the VIP to connect!)4. There Are Multiple Layers of Load Balancing and Connection FailoverMost RAC database administrators know that services are used for loadbalancing and connection failover in RAC. However, many of them may notrealize just how many levels of load balancing and connection failover existfor services.There are three levels of load balancing in Oracle: client-side connectionload balancing, server-side connection load balancing, and run-time loadbalancing. Client-side connection load balancing is the simplest and moststraightforward; the client will randomly choose one of the TNSNAMESaddresses for each new connection, and the connections will be distributed There are also three levels of load balancing in the TNSNAMES and threeroughly the same across the listed connections. Server-side connection load different ways to configure run-time failover. In the TNSNAMES files, bothbalancing is enabled by default and is slightly more sophisticated than client- failover and load balancing can be configured in the DESCRIPTION_LISTside load balancing. The listener knows the current load on each database block, the DESCRIPTION block and the ADDRESS_LIST block. Personally, Iinstance and can re-route client connections to the most responsive instance have never yet needed to configure any more than just the ADDRESS_LIST.or the instance with the smallest number of sessions. Third, Oracle supportsrun-time load balancing within JDBC and ODP.NET connection pools. When a ARDENTACCT.LAB.ARDENTPERF.COM =thread requests a connection from the pool Oracle can automatically return a (DESCRIPTION_LIST =connection on the least loaded cluster instance. Which of these three (LOAD_BALANCE = ON)methods should be used for load balancing? At a minimum, I usually (FAILOVER = ON)recommend the first two—if not all three! Note that there is an inverse (DESCRIPTION = ... ) (DESCRIPTION =relationship between the quality of load balancing and the complexity of (LOAD_BALANCE = ON)setup. This relationship is shown here. (FAILOVER = ON) (ADDRESS_LIST = ... ) (ADDRESS_LIST = (LOAD_BALANCE = ON) (FAILOVER = ON) (ADDRESS= (PROTOCOL = TCP)(HOST = rh4lab12)(PORT = 1521)) (ADDRESS= (PROTOCOL = TCP)(HOST = rh4lab13)(PORT = 1521)) ) (CONNECT_DATA= (SERVICE_NAME = ) ) As for Transparent Application Failover (TAF), run-time failover can be configured with client-side TAF, server-side TAF and Fast Connection Failover (FCF). So, which of these methods should be used for failover? This is determined by your environment: FCF is only available for connectionThere are also three different levels of connection failover in RAC: the service pools and server-side TAF is only available in Oracle Database 10g andlevel, the connection level, and the runtime level. The broadest layer of newer databases.failover occurs at the service level on the database server. During normal Load balancing and failover are two of the most important features ofoperation a service runs only on its PREFERRED nodes, but if the clusterware services; they are fundamental to meeting the HA goals of cluster projects.detects the failure of an instance then it will automatically relocate that So it is very important that administrators are well acquainted with every levelservice to an AVAILABLE node. A second layer of failover occurs when the of load balancing and failover that services offer.client makes a connection to the database. Like client-side load balancing itis easily configured in the TNSNAMES file and causes the client to 5. Services Don’t Always Do What You Would Expectautomatically retry the connection with a different address when the firstaddress fails. Lastly, Oracle provides run-time failover: Under certain Lastly, it comes as no surprise that services do not always do what you wouldconditions, the client can detect a node failure and automatically move expect. We never know exactly how new technology will surprise us, so itexisting connections to a surviving node with no interruption to the seems worth reviewing a few specific points where services have behavedapplication. Experienced RAC administrators may notice that the third run- differently than what some database administrators expected.time failover option seems to come up a lot when talking about load First, a “preferred-available” configuration in a multi-node RAC environmentbalancing and failover. Again, there is an inverse relationship between the does not perform the same as a single-node configuration. If a service isclient-oriented and server-oriented connection failover in these three setups. configured to be “preferred” on one node and “available” on a second node,This is also shown in the next diagram. it does mean that the service only runs on the preferred node during normal operation. However, the cluster database is fully operational on both nodesPage 16 n 3rd Qtr 2008
  5. 5. and there is an associated overhead (such as maintaining cache coherencyand global locks); hence, the performance profile will not be the same as asingle-node database. If you are clustering for the sole reason of highavailability, then perhaps you should consider whether a cold-failover clusterwould meet your requirements. When configured properly, it is possible toachieve failover times of fewer than 120 seconds in such cold-failover clusterenvironments. Additionally, the license and support fees for failover clustersare often significantly lower than RAC, and that is an added bonus.The second unexpected behavior of services is that there is no automaticfailback of services on clusters. If a service fails over to an “available” node,then the service will not automatically move back to the preferred node whenyou restart it. The service must be manually switched back.The third unexpected behavior is related to the previous “feature”—servicessometimes will not automatically start when the server is started. In OracleDatabase 10g the clusterware always attempts to return services to their last-known state. If you shut down specific services using srvctl, then the servicewill not start after a reboot because the last known status of the service was“offline.” However, if a node shuts down unexpectedly, then the clusterwarewill restart the services on reboot.Finally, if you are on Oracle Database 10g or earlier, there is a bit ofunexpected behavior related to parallel execution on RAC. Parallel queryslaves inherit the service name for the purposes of reporting performancestatistics, but they will run on all nodes of the cluster, completely ignoringPREFERRED and AVAILABLE assignments for the service. The workaround isto use the INSTANCE_GROUPS and PARALLEL_INSTANCE_GROUPSparameters to limit parallel query execution to the nodes where your serviceis running. However, you will need to manually keep these parametersupdated as you move services around. This behavior is fixed in OracleDatabase 11g although the PARALLEL_INSTANCE_GROUPS parameter can stillbe used to override the default behavior and run the parallel query slaves ona different subset of cluster nodes.ConclusionAfter reviewing Oracle services, it is easy to see why they are so important tomodern databases—and especially to cluster database systems. They add acompletely new level of workload and performance management, and theyprovide the foundation for RAC’s most important availability features. Theextensive benefits of services far outweigh the shortcomings and smalllearning curve for using them. The bottom line is that understanding serviceshas become a basic, core responsibility of anyone who plans or maintainsOracle systems today, and that they should be configured for all databases. n n n About the Author Jeremy Schneider is an Oracle Database consultant who has worked with small and large organizations across the country. His interests include applying grid and cluster technology to large-scale problems and he spends a lot of time working with RAC databases. Schneider is also an enthusiastic supporter of user groups and open source projects, and maintains an Oracle technology blog at 3rd Qtr 2008 n Page 17