Whitepaper: Running Oracle e-Business Suite Database on Oracle Database Appliance


Published on

This is the whitepaper for my Collaborate 13 presentation with the same title. It describes how Pythian completed a migration project of eBS R12 database top ODA (Oracle Appliance Kit v2.2).

1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Whitepaper: Running Oracle e-Business Suite Database on Oracle Database Appliance

  1. 1. COLLABORATE 13OAUG ForumCopyright ©2013by Maris ElsinsPage 1Running E-Business Suite Database on Oracle Database ApplianceMaris Elsins, Yury VelikanovThe Pythian Group Inc.Oracle Database Appliance (ODA) is a pre-configured, simple setup, good performance engineeredsystem running an 11gR2 cluster. It is a great choice for small to medium sized DBs, and if you wish itcan be used for Oracle e-Business Suite too. This paper will show you how the standard configurationof ODA can be adjusted to comply with the specific requirements of e-Business Suite withoutsacrificing ODA’s flexibility and supportability. The paper will also share the authors’ experiencemigrating, running and maintaining R12.1.3 Oracle e-Business Suite (eBS) database tier on ODA.Pythian completed a database consolidation project last year. Four databases of versions were upgraded to and migrated to a single ODA. The project was especiallyinteresting because one of the databases was an e-Business Suite R12 database, and there was noinformation on My Oracle Support on how to cope with the specific e-Business Suite requirementsand how to preserve the supportability and ease of maintenance of ODAs technology stack. Thispaper is a case study of how Pythian configured the eBS database on ODA v2.2.Why choose ODA?ODA is the smallest of Oracles engineered systems designed for running Oracle Database, but it stillbrings a huge set of benefits – simple deployment as its pre-configured, reliable as its pre-tested,highly available as all the components are redundant, and affordable with its 50,000$ (60,000$ forODA X3-2) list price and pay-as-you-grow licensing model.ODA contains two database-server nodes, each configured with two Intel Xeon X5675 6-coreprocessors, giving total of 24 cores per ODA. Each node has 96 GB of memory and 500Gb ofmirrored storage for the operating system and software installations. The shared storage consists oftwenty 600Gb SAS drives that can be configured using high or normal redundancy, resulting in 4Tb or6Tb of space for storing the data and backups, and an additional four 73Gb SSD drives triple-mirroredfor online redo logs. The hardware specification of the ODA makes it attractive to many medium-sizeddatabases, including ones supporting e-Business Suite.A new Oracle Database Appliance X3-2 version was announced in March 2013. It provides the samebenefits, but the hardware is different. Each database-server node is configured with two Intel XeonE5-2690 8-core processors, so the total number of CPU cores is 32. Each node has 256 GB ofmemory and 600 GB of storage for operational system and software installations. The shared storageis built of twenty 900 GB 2.5” SAS-2 drives that provide 6Tb space if triple mirroring is configured and9Tb with double mirroring. There are four 200 GB SSD drives included for 260 Gb of space for redologs configured in ASM high redundancy disk group configuration.ChallengesAny engineered system will have limitations, and ODA is no exception. These limitations can bedivided into two categories - limitations by design and supportability limitations.Limitations by design are enforced by the provided hardware configuration. Depending on whichODA you choose (ODA or ODA X3-2), The computing capacity is limited to a single ODA and theconfiguration is non-expandable - it’s not possible to interconnect two ODAs to create a larger cluster.On the otherhand, even the old version of ODA, with its 96Gb of RAM and 24 CPU cores is more thanenough for most small and medium eBS installations. The limited disk space of 4Tb (6Tb from v2.4)can partially be addressed by storing backups on NFS-mounted storage over a 10Gbps interface andin case of X3-2 you have an additional option of adding a Storage Expansion to double the availablecapacity.Supportability limitations - There are strict rules on how to manage the ODA. Oracle only supportspatching and upgrading by using the oakcli (OracleApplianceKitClient) utility. The patches aredesigned for a consistent and uniformly patched system, so anything that is installed additionally
  2. 2. COLLABORATE 13OAUG ForumCopyright ©2013by Maris ElsinsPage 2using opatch can cause issues with patching in the future. There are certain situations when OracleSupport could allow installation of a one-off patch, but its also noted that these one-offs could berequired to be uninstalled before patching ODA in the future. The Oracle Global Inventory should bekept clean and unmodified. Additional products should not be installed on ODA, as oakcli relies on theinformation stored in the inventory and any unexpected entries can cause issues with patching.The first publicly available versions of ODA software were quite limited, Oracle releases new softwareversions every 3 to 6 months and they add new features. For example: v2.1 supported only a single11.2.0.2 oracle home, v2.2 gave us Oracle Database version, v2.3 introduced support formultiple oracle homes and allowed running multiple databases of different versions, v2.4 gave us achoice of normal redundancy for +DATA and +RECO diskgroups (the only option before that was highredundancy) and, finally, v2.5 brought Oracle Virtualization support to ODA. An organization needs tohave a roadmap on how to plan and upgrade ODA to keep it on a supportable version.There are also “supportability” challenges from the E-Business Suite side. The specificrequirements of E-Business Suite R12 are listed in the following My Oracle Support notes:"Interoperability Notes EBS 12.0 and 12.1 with Database 11gR2 [ID 1058763.1]" and "Using Oracle11g Release 2 Real Application Clusters with Oracle E-Business Suite Release 12 [ID 823587.1]"(similar notes exist for 11i - 823586.1 and 881505.1). These documents list a number of requirementsthat collide with the supportability limitations of ODA:• Install Oracle Database 11g Products from the 11g Examples CD• Apply additional RDBMS patches (4247037, 9858539, 12942119, 12960302,12985184, 13001379, 13004894, 13258936, 13366268)• Create a dedicated oracle listener for eBS database• Customized TNS_ADMIN and ORA_NLS10 settings• Specific initialization parametersAlthough the list is not very long, it’s clear the default Oracle Home configuration on ODA is notsuitable for running the eBS database. The next section of the paper describes how Pythiancustomized the configuration of ODA to be able to run the e-Business Suite database and to minimizethe negative effect it could leave on the future maintenance tasks.Preparation the ODA for eBS databaseThe E-Business Suite database configuration described in the following sections was implemented onODA (v1) and not ODA X3-2. But, as X3-2 introduced changes mostly to the Hardware configuration,the approach should be valid for the new ODA X3-2 too.A Dedicated Oracle Home was created for the e-Business Suite database by cloning the RDBMSOracle home available on ODA in /u01/app/oracle/product/ The new Oracle homewas created with path /u01/cst/{DB_NAME}/product/ – we created a new directory (cst) under/u01 to store all custom Oracle homes and we used DB_NAME in the path as we planned to createmultiple homes, and each of them could have its own patch requirements. The default Oracle homewas left intact to be able to run other databases in an unmodified configuration.Additionally it was decided to create a dedicated Oracle Inventory in /u01/cst/oraInventory for allcustomized installations to keep The default global inventory clean and avoid the possible negativeeffect on future patching of the ODA. The new Oracle home was registered in the new inventory. Aswe had two oracle inventories instead of one, we also introduced a need to maintain /etc/oraInst.locfile manually by adding the correct inventory location depending on the type of maintenance workbeing done – it should point to the default inventory (/u01/app/oraInventory) for normal operations andit should point to the custom inventory for the maintenance activities affecting the eBS databaseOracle home.By creating a configuration that was completely independent from the default ODA setup, we couldinstall the requirements of e-Business Suite database in the freshly created Oracle home withoutaffecting the seeded configuration. The Examples CD for patch set was downloaded fromMy Oracle Support (patch 10404530) and installed into the new Oracle home along with all therequired one-off patches.
  3. 3. COLLABORATE 13OAUG ForumCopyright ©2013by Maris ElsinsPage 3Please note the default grid infrastructure Oracle home was used by all the databases running onODA, since eBS didn’t have any specific grid infrastructure patching requirements.From a certification point of view, ODA is an Oracle Linux 5.6+ server cluster running Oracle gridinfrastructure and database versions Both OS and database versions are certified forrunning Oracle e-Business Suite R12.1. It’s also important to note that despite the fact a new Oraclehome was installed manually (a customization of the Oracle Appliance Kit product) all components –oracle grid infrastructure, oracle database, e-Business Suite, and even the oracle home we cloned –are still set up in a fully certified and supported configuration.An important preparation step for going live with ODA is implementation of a monitoringframework for timely notification about issues. The best way for monitoring the ODA hardware isby configuring the Auto Service Request (ASR) support feature that is included in the ODA softwarebundle. ASR automatically creates Service Requests, provides diagnostic information to OracleSupport, and notifies administrators about hardware faults. ODA also supports installation of 3rdpartyagents to manage, monitor, backup, replicate, authenticate, or otherwise act on the database, theserver, or the environment (note ID 1415773.1), giving the opportunity to handle the ODA the sameway as any other server in your IT infrastructure.It is worth mentioning that one day before our go live activities, one of the HDDs failed, but as therewas triple-redundancy it had a little impact on the project. After a short rebalancing period, ODArecovered the protection level with one less HDD. A few weeks later a new HDD was addedseamlessly to the running production system.Database migration and configurationAs part of the migration project to ODA we also had to upgrade the database from to11.2.0.3. Since the old HW was not very powerful we wanted to utilize the performance of the ODAto speed up the upgrade process. This requirement revealed another problem – how to migrate10gR2 database to 11gR2 on ODA? The following options were considered:• Upgrade the DB on the old hardware and then move to ODA – the option was rejected as itrequired too much downtime.• Install 10gR2 Oracle home and create a physical standby on ODA, then switchover and dothe upgrade on ODA – this approach would allow us to meet the downtime requirements butthe problem was the ASM diskgroups on ODA were shipped with database compatibilitysetting of, and therefore the 10gR2 RDBMS software couldn’t use them. Aworkaround of recreating the ASM diskgroups with compatibility setting of 10.2 wasconsidered, but was found too intrusive and too risky as it may break ODA supportability.The chosen upgrade method was based on the fact that RMAN is able to restore backups taken onan older version of the database into a newer release (MOS ID 369644.1). The backup of the was restored using software into ASM disk groups with database compatibilitysetting of After completing the restore, the archived log apply process was scripted to keepthe target database synchronized with the source. Remaining redo was applied and “alter databaseopen resetlogs upgrade” was executed to put the database into the upgrade mode at the switchover,after which we could continue the normal upgrade process on the ODA. The pre-upgrade proceduresstill had to be performed on the source (10gR2) database.Oracle e-Business Suite has requirements for specific initialization parameters listed in My OracleSupport note “Database Initialization Parameters for Oracle E-Business Suite Release 12 [ID396009.1]”, but in addition to these it was found that a vanilla database created by “oakcli createdatabase” command had the following non-default initialization parameters:• _disable_interface_checking=TRUE• _ENABLE_NUMA_SUPPORT=FALSE• _FILE_SIZE_INCREASE_INCREMENT=2143289344• _gc_policy_time=0• _gc_undo_affinity=FALSE• _KGL_CLUSTER_LOCK_READ_MOSTLY=TRUE• _kill_diagnostics_timeout=140• _lm_rcvr_hang_allow_time=140
  4. 4. COLLABORATE 13OAUG ForumCopyright ©2013by Maris ElsinsPage 4• db_block_checking=FULL• db_block_checksum=FULL• db_lost_write_protect=TYPICAL• filesystemio_options=setall• parallel_adaptive_multi_user=FALSE• parallel_execution_message_size=16384• parallel_min_servers=0• parallel_threads_per_cpu=2• use_large_pages=ONLY• compatible= the parameters had obviously been fine tuned for ODA we set them in our eBS database too.Clusterware resources for the eBS database and the listener were created to ease the operationaltasks of the database. A dedicated listener was created for the e-Business Suite database usingport pool 1; it’s important to choose other port pool then 0 to avoid port conflicts with the defaultlistener running on port 1521. The following commands were used to create the listener:• srvctl add listener -l LISTENER_EBSDB -o $ORACLE_HOME -p 1522• srvctl setenv listener -l LISTENER_EBSDB -TTNS_ADMIN=$ORACLE_HOME/network/adminThe important bit for creating the database resource for an eBS database is to set the ORA_NLS10environment parameter as required in note 1058763.1:• srvctl add database -d EBSDB -o /u01/cst/EBSDB/product/ -p+DATA/EBSDB/PARAMETERFILE/spfile.704.792145311 -a "DATA,RECO,REDO" -n EBSDB• srvctl setenv database -d EBSDB -t"TNS_ADMIN=$ORACLE_HOME/network/admin,ORA_NLS10=$ORACLE_HOME/nls/data/9idata"• srvctl add instance -d EBSDB -i EBSDB1 -n oda01a-net1• srvctl add instance -d EBSDB -i EBSDB2 -n oda01a-net1It’s also critical to note the TNS_ADMIN variable for cluster resources points to the default location in$ORACLE_HOME/network/admin, but, as AutoConfig creates node specific directories, they have tobe made visible from the default path by adding the following IFILE settings on both database servers:• echo "IFILE=${ORACLE_HOME}/network/admin/${CONTEXT_NAME}/tnsnames.ora" >$ORACLE_HOME/network/admin/tnsnames.ora• echo "IFILE=${ORACLE_HOME}/network/admin/${CONTEXT_NAME}/listener.ora" >$ORACLE_HOME/network/admin/listener.ora• echo "IFILE=${ORACLE_HOME}/network/admin/${CONTEXT_NAME}/sqlnet.ora" >$ORACLE_HOME/network/admin/sqlnet.oraImpact on Maintenance of ODA and eBS databaseThe implemented configuration of ODA v2.2 with the eBS database running in a separate oraclehome has been very stable for almost half a year – the pre-configured hardware and Gridconfiguration seems to be working. The first installation of an ODA Patch Bundle is still in front of us.As a dedicated Oracle home and inventory have been created the impact on ODA’s patchingshould be minimal. There are, however, a few configuration points that are visible to oakcli utility andcould affect the way it works:• /etc/oraInst.loc – the inventory_loc setting should point to the default or the custom oracleinventory depending on the type of the maintenance being performed.• /etc/oratab – the file lists all the databases running on the server, including the eBS databaserunning from the customized oracle home. One should avoid running “oakcli update -patch2.x.x.x.x” to update all components as it upgrades all databases on the ODA by default,instead oakcli options --infra, --gi and --database should be used to update the softwarefollowed by “--database <db_name>“ for each database running from the default oracle home.The custom Oracle home will have to be patched manually or it will need to be re-cloned fromthe upgraded default oracle home of ODA.
  5. 5. COLLABORATE 13OAUG ForumCopyright ©2013by Maris ElsinsPage 5• Clusterware resources – the cluster resources are globally visible, so if the customizedresources cause trouble they should be temporarily removed.• Data on ASM diskgroups – it’s unlikely any of ODA Patches would be built in a way that couldharm data on the ASM diskgroups, but precautions should be taken by taking a backup of thedatabase and storing it outside the ODA.On the other side, the ODA platform brings some restrictions for maintaining the eBS database aswell:• Patching – as a completely separate Oracle home is used there are no restrictions to applyone-off patches, however Patchsets and Patch Set Updates (PSUs) can’t be installed ifthe grid Infrastructure is not upgraded because of the dependencies between RDBMS andgrid Oracle homes. The grid home should be upgraded only by applying ODA bundle patches.• Security Patch Updates – based on My Oracle Support note “Patch Set Updates for OracleProducts [ID 854428.1]”, the security patches are developed only for the base release, andonce a PSU has been installed, the recommended way to get future security content is toapply subsequent PSUs. The Oracle Home that was cloned for running e-Business Suitedatabase was (with a PSU applied), therefore the only way to get the securityfixes is to install the latest ODA Patch bundles.Future PerspectiveThe future possibilities of running e-Business Suite databases on ODA depend on what features getincluded in the coming ODA Patch Bundles. Already we see the trend of transforming the OracleDatabase Appliance into an “Oracle Database and Application Appliance” – the Patch Bundle v2.5brings virtualization to ODA. Oracle e-Business Suite is already certified with Oracle VM 3.1.1 (seenote ID 465915.1), giving us a great opportunity to place the whole e-Business Suite implementationon a single ODA box.For the implementation described in this article (running on ODA bundle patch v2.2), an upgrade to anODA software version supporting multiple Oracle homes should be considered. It would allow runningthe e-Business Suite database from an ODA-supported Oracle home by installing additional one-offpatches (they would probably need to be uninstalled before ODA patching), giving us a cleaner andmore easily understandable configuration with a single Oracle inventory.ConclusionsODA is great HW for running midsized databases, but official support for the e-Business Suitedatabase tier on ODA is not yet available. Configuring eBS on ODA requires balancing betweensupportability of the ODA’s configuration and implementation of all its specific requirements. Oursolution was to create a dedicated Oracle home and inventory for e-Business Suite database to allowinstallation of additional one off patches and products from the Examples CD. The impact the chosenconfiguration leaves on the further maintenance of ODA and e-Business suite is low and does notreduce the benefits ODA brings to the operation of e-Business Suite. It is also expected Oracle willcontinue to develop the capabilities of ODA and one day perhaps they will provide complete supportfor e-Business Suite too.