SlideShare a Scribd company logo
1 of 22
Download to read offline
Engineered Systems




     Oracle database appliance v1 - rac in a box – some strings attached
                                                                                           Fuad Arshad, Walgreens Company



ABSTRACT
The Oracle Database Appliance is an engineered system that is targeted towards small and medium businesses, but also has
all of the bells and whistles required for enterprise deployments. This paper will introduce the Oracle Database Appliance
(ODA), and review the deployment of the ODA from a non sysadmin perspective. This paper includes the deployment and
patching processes and the best practices built into the Oracle Database Appliance. The paper will review the DBA's role in
managing the ODA

TARGET AUDIENCE
This paper is intended mostly for Database Architects and people that deploy Oracle database technologies. This paper will
review various techniques for deploying the Oracle Database Appliance (ODA) successfully as well as how to perform day to
day activities. The various components all tied together provide an effective use case for companies that want to invest in
Engineered Systems, and provide an entry point to understand the management and maintenance considerations for deploying
an engineered system.

EXECUTIVE SUMMARY
Attendees and readers of this paper will learn the following after reading this paper.
Oracle Database Appliance - Overview
Oracle Database Appliance - Predeployment
Oracle Database Appliance - Deployment
Oracle Database Appliance - Patching
Oracle Database Appliance - One Button Database Appliance



INTRODUCTION
The Oracle Database Appliance (ODA) is an entry level engineered system that provides easy access to preconfigured Real
Application Cluster (RAC) database hardware and software. The ODA is accessible to customers at an entry level cost with
software licensing available starting from 2 cores to 24 cores. The ODA is intended to provide companies with an easy to
deploy model that includes all of the hardware and software required to run a complete system into one 4u rack mountable
unit.

INSIDE THE ORACLE DATABASE APPLIANCE
The Oracle Database Appliance (ODA) is a 4u rack mountable unit which consists of two 2u Sun Fire X4370 M2 units
known as Server units (SN). Oracle has provided MOS Note 1385831.1 which is kept fairly up-to-date with the ODA
components and software versions. Below is a snapshot as of Dec 14, 2012 and this paper only focuses on V1 functionality.



                                        Sun X4370M2 System Server Components
                                             (2 Server Nodes per X4370M2)
      CPU                                         2x 6-core Intel Xeon X5675 3.07GHz
                                                              1                                             108
Engineered Systems



   Memory                                 96 GB RAM (12 x DDR3-1333 8GB DIMMs)
                                                 2x 10GbE (SFP+) PCIe card
   Network                                           6x 1GbE PCIe card
                            2x 1GbE (Intel 82571) on-board Integrated redundant cluster interconnect
                                             2x 500GB SATA - for operating system
Internal Storage
                                                    1x 4GB USB internal
    RAID
                                                  2x LSI SAS9211-8i SAS HBA
   Controller
                    20x 600GB - 3.5" SAS 15k RPM HDD (Seagate Cheetah) - for RDBMS DATA (any slot
                                               except the top row of disks)
Shared Storage
                        4x 73GB - 3.5" SAS2 SSDs - for RDBMS REDO (slot in the top row of 4 disks)
                        SSD from STEC (ZeusIOPS - multi-level cell (MLC) version with SAS interface)
   Operating
                              Oracle Enterprise Linux 5.5 (on ODA 2.1) 5.8 (on ODA 2.2) x86_64
    System
 The two ODA Server Nodes are packaged in a 4u chassis that includes 2 server nodes as well as direct attached
disk in the front. Below are front and back images of an ODA. looks like from the front and back




The bottom server is Node 0 and the top is Node 1.These will be referred to as oak1 and oak2 in the factory
default configuration. It is very important to keep this picture handy to work with the data center installation
team to ensure they understand how each port is used to cable the ODA to meet your requirements. Cabling
depends on the data center security zone specs that your infrastructure defines. There might be sensitive and non-
sensitive zones as well as various cabling zones. These requirements need to be considered as part of the
installation process. Attached in the Appendix is a sample spreadsheet that outlines the cabling specs needed to
complete an ODA Installation.




                                                       2                                          108
Engineered Systems




Callout    Label        Ethernet        Bond                                Description
  1                                              Power connectors
                                               Eth 4 and Eth 5 are configured as bond1. Eth 6 and Eth 7 are
                      Eth 7, 6, 5, 4    bond1, configured as bond 2. These ports are used for custom
  2        PCIe 1
                      (left to right)   bond2 configurations or for separate backup, disaster recovery, and
                                               network management.
                                                 Two 10-GbE ports. In 10-GbE systems, these are connected to the
  3        PCIe 0     Eth 8, Eth 9      xbond0
                                                 public network.
  4        SerMgt                                Serial connector to Oracle ILOM and system console.
          Net 0,Net                              Two 1-GbE connectors. In 1-GbE systems, these are connected to
  5                    Eth2, Eth3       bond0
              1                                  the public network.
  6       NetMgt                                 Ethernet connection for Oracle ILOM
          USB and
  7                                              Used for connecting to system console.
           Video
  8                                              Server Node 0
  9                                              Server Node 1

                                                          3                                        108
Engineered Systems




As you can see from above, it, is very important to understand the various components. The complete details are available in
the MOS note referenced above, as well as the ODA Install, Configuration and User’s Guide. This paper will only touch the
basic pieces that are important to create the fundamental understanding about the Oracle Database Appliance. Additional
references are available at the end of this paper.

PREDEPLOYMENT OF THE ORACLE DATABASE APPLIANCE
The Oracle Database Appliance requires a lot of pre-deployment tasks to be completed to prepare for installing the ODA.
These pre-deployment steps are standard tasks for deploying a server in a data-center and we are going to walk thru them
from a Database Administrator perspective. The Oracle Database Appliance Owners guide covers most of these details.
In order to get the environment ready, the site needs to meet the environmental & electrical guidelines that are explained in
the Owner’s manual. The ODA weighs about 96 lbs., and utmost care must be taken while handling the hardware. Four-post
racks are compatible. Two-post racks are not compatible. Horizontal and vertical pitch must conform to ANSI/EIA 310-D-
1992 or IEC 60927 standards. An Oracle Database Appliance requires 4 rack units (4RU) of rack space. A rack mounting kit
is shipped as part of the hardware.
Once the hardware is racked and stacked in the data-center floor, the appliance needs to cabled. If 10g network connections
will be used, SFA Adapters will need to be included in the ODA order for attaching the 10g network cables to the appliance.
There is a minimum requirement of 2 power connections and 4 network connections to the public network in order to access
the machine. There are other networks available for use for various purposes (connecting to a NAS, ILOM or Backup
Network). Once all cabling is confirmed, it is very important to create a pre-deployment checklist to ensure that all
appropriate information about the system is captured.An example template can be seen below.



     Needed information                                  Your Information

     System Name: i.e. sales                             TSODA1
     Database Deployment Type: RAC, RAC One or           RAC
     Standalone
     Internal or External Backup via NFS device          External
     Root Passwords: default for nodes is welcome1       Leave the default for now
     Domain Name:                                        Example.com.com
     DNS Server(s): min 1                                Dns.example.com
     NTP Server(s): optional but recommended             Ntp.example.com
     Node 1 Name: i.e. sales1                            Tsoda11
     Node 1 Public IP Address
     Node 1 VIP Name: i.e. sales1-vip                    Tsoda11-vip
     Node 1 Public Virtual IP Address
     Node 2 Name: i.e. sales2                            Tsoda12
     Node 2 Public IP Address
     Node 2 VIP Name: i.e. sales2-vip                    Tsoda12-vip
     Node 2 Public Virtual IP Address
     SCAN Name: sales-scan                               Tsoda1-scan
     SCAN IP Addresses: min 2
     Netmask:
     Gateway IP:
     ILOM Names: ie. sales1-ilom                         Tsoda11-ilom
     ILOM IP Address:
     ILOM Names: i.e. sales2-ilom                        Tsoda12-ilom

                                                            4                                                108
Engineered Systems


     ILOM IP Address:
     ILOM Passwords: default is changeme                      Leave the default for now
     ILOM Gateway:
     ILOM Netmask:
     Database Name:                                           Tsodadb
     Database class: very small, small, medium, large, very   Small
     large
     (optional) Proxy Server Name: for Auto Service
     Request
     Metalink Login Information: User ID/Pwd


Since each Oracle Database Appliance comes with the same configuration, it is very easy to provide a configuration
spreadsheet that includes various configuration specs and networking information to record the right information from the
network and data center teams . An example spreadsheet that can be used to streamline the process is attached below.


  Microsoft Excel
    Worksheet

It is very important to develop a standard process for deploying the ODA. This includes defining roles and responsibilities for
deploying and managing the ODA. It is very important to understand these organizational responsibilities prior to
deployment. A lot of activities that are required to deploy the ODA require system access and organizational structures
typically have role separation. The ODA deployment activity will need the assistance of all teams within an organization from
infrastructure to system administration if the role separation methodology is used. Conversely the Database Administrator
deploying the ODA needs to understand various additional processes in order to successfully deploy the appliance.
In the Scenario, if the Database team is responsible for deploying the Oracle Database Appliance, the following is a brief
example list of activities that will need to be performed.
      1. Determine hardware placement zones.
      2. Submit a request to get network ip’s
      3. Submit DNS and Firewall requests
      4. Work with datacenter team to determine rack space.
      5. Ensure cables and adapters are available
      6. Read thru the ODA manual
      7. Provide datacenter team cabling specs
      8. Get the ODA connected to a KVM or access via Serial cable to ensure the hosts are powered on and ILOM is
          accessible.
      9. Walk thru deployment activities.


DEPLOYMENT OF THE ORACLE DATABASE APPLIANCE
In order to understand how to deploy an Oracle Database Appliance (ODA) in a datacenter, one needs to understand the
appliance itself. As illustrated above, The ODA consists of 2 x86 servers that have direct attached SAS as well as SSD
storage. The environment is fully redundant from a hardware perspective. Storage redundancy is provided by Oracle ASM and
can be triple mirrored or double mirrored (version 2.4 and above) for the SAS disk. The SSD is designed and recommended
to be only used for the on-line redo logs and is triple mirrored regardless of ODA version. Oracle ASM also ensures that all
the extents are mapped appropriately among the available disk to ensure that data extents are balanced. Oracle RAC is used to
ensure any hardware failures on one node will not impact the running of the system as a whole.

                                                                 5                                            108
Engineered Systems


As we look at the appliance we see 1 host node and 1 Service Processor node per hardware entity. The service processor also
includes an Integrated Lights Out Management (ILOM) console which is used to manage each appliance. Deploying the
ODA starts with getting the ILOM ready and accessible to the network.
DEPLOYMENT OF THE ILOM
The ILOM, or the Integrated Lights Out Manager, is the hardware administration and management brains and is also referred
to as the service processor. The ILOM is used for various purposes in the lifecycle of a host including, but not limited to,
performing a bare metal restore of an ODA. An Oracle Database Appliance contains 2 ILOM’s, or one for each of the two
ODA servers. Configuring the ILOM is step 1 in the process of getting an ODA up and running. There are multiple ways to
get into the ILOM. If you have a KVM device you can connect the KVM to the ODA serial management port to do the
ILOM configuration remotely. The configuration steps for setting up the ILOM directly via the serial port are detailed below.
ACCESSING THE ILOM VIA SERIAL PORT
The ODA ILOM can be accessed via a Serial management (SerMgt) port labeled #4 in the picture above. MOS Note ID
1395445.1 explains the process of connecting to the ODA to configure the ILOM port.
The process requires direct physical access to the serial port , which means, this process will need to be run at the place where
the Oracle Database Appliance is installed. Each ODA ships with an RJ45-serial convertor like the one pictured below.




Using an RJ-45 cable, you can connect the cable to the serial port on your laptop or Datacenter terminal cart and get into the
ILOM using putty. In the case that you do not have access to a serial port, Amazon sells a usb to serial converter cable at
http://amzn.com/B00425S1H8 . The cable looks like below and requires prolific drivers installed on your laptop in order to
use the cable. This cable will be used in conjunction to the rj45-serial converter to connect to the ODA ILOM.




Once the connection cable is connected to the laptop and ODA serial port, check the device manager to see which com port
the USB device is connected to and specify the com port in your Putty connection. This doc will only talk about windows but
the MOS note above includes the Linux configuration as well. Once the port is determined, you can open up putty and
connect to the serial connection.




                                                              6                                                 108
Engineered Systems




Once the connection is made you will see a login prompt. Once the ODA is cabled, in and power cords are attached, the
ILOM is automatically powered on. You will need to use the root account and the default root password to connect to the
ILOM.




                                  .
An initial configuration of the ILOM should have no IP, and this can be validated by issuing the following command:
                           # show /SP/network
While the OAK deployment can IP the ILOM, it is always faster to configure the ILOM IP’s using the following method.
This will allow for a faster deployment by using the ILOM to deploy the ODA. You will need the following information to be
able to IP the ILOM for both ODA servers.
    1. Public IP Address
    2. Netmask
    3. Gateway Address
The following command will assign a static IP address to the ILOM
# set /SP/network pendingipdiscovery=static pendingipaddress=1.2.2.65 pendingipnetmask=255.255.255.0
pendingipgateway=1.2.2.254 commitpending=true
After the IP is established you can also use this session to power on each of the database nodes remotely.
# start /SYS


                                                           7                                              108
Engineered Systems



Once this is executed you should be able to access the ILOM via ssh or an internet browser, as well as connect to the
database node remotely. The following screen shows browser access to an ILOM.




This is the ILOM console and this confirms the successful configuration of the ILOM. Once the ILOM is successfully
configured you are now ready to configure the Oracle Database appliance database nodes. The ILOM provides a lot of
functionality including fault management and remote access. The functions of the ILOM are beyond the scope of this
document but below you can see a picture of some of the ILOM’s capabilities. We will be using the ILOM in the next section
to remote control into the Database Node to configure the ODA.




DEPLOYMENT OF THE DATABASE NODE
The setup of the ILOM is the first step in deploying an ODA and configuring the two database nodes. As part of the
deployment process you will need to create the deployment configuration files or enter the configuration information during
the deployment process. To validate your configuration, it is often wise to use the offline configurator that is available via the
Oracle Technology Network (OTN) website http://bit.ly/VpakT8. The Offline Configurator allows an administrator to
enter, and validate, the complete set of configuration information prior to starting the deployment. The offline configurator
which is meant currently for Bare Metal installations allows for complete network validation prior to deployment.




                                                               8                                                 108
Engineered Systems


The first step in the deployment is to access each of the database nodes through the ILOM, using SSH as follows:




The ILOM can also be used to access the database node by launching the remote console through a standard web browser.
The host first needs to be powered on either remotely using the ILOM or through the power button on the front panel on
the Oracle Database Appliance boxes.




Once you launch the remote console. you will be taken to the host.




After logging into the host, you will need to IP each of the two database hosts by running the following command using the
root account:
        # /opt/oracle/oak/bin/oakcli configure firstnet
This will allow for public access to the appliance and allow for completion of the deployment. Discussion about patching
will be in a separate section but it is important to ensure that the version of OAK (Oracle Application Kit) you are
deploying is not very far behind.
                                                            9                                              108
Engineered Systems



        #/opt/oracle/oak/bin/oakcli show version
If you feel that the software needs patching, please do so prior to the deployment as there are some factors like disk
configuration that will change based on the version of the software . These include, but are not limited to, disk
redundancy (double vs. triple mirroring) and the introduction of virtualization to the Oracle Database Appliance. The
discussion about redeploying an ODA is a 3 part blog post series that can be found at http://bit.ly/VpcWQH . Once you
have determined and patched your ODA to the level you want to start out with, the deployment process can start. In the
earlier versions of the OAK software, the only option seemed to be using the GUI interface to deploy the software. In
later versions (2.3 and above), the copy command has been introduced which allows for a scripted approach to
deployment. For a safer GUI deployment it is recommended to use the vnc utility in order to avoid any issues.
As root you can execute
# vncserver :1  will allow for vncserver to run on port 5901.
You will be prompted to setup a password. After which you will gain access to the server via vnc. Below is what a vnc
screen will look like on a vnc connect . You can use publically available free vnc viewers like Tight VNC available at




If you have created a deployment configuration file you can copy it to the first node of the host. You can deploy an ODA
by entering the configuration information manually, but it is a cumbersome process. The screens below will show a
typical deployment using a configuration file prebuilt via the offline configurator




For complete control we will select custom and load the file.




                                                            10                                            108
Engineered Systems




You will provide generic network info in the next page or the information will be prefilled form a config file that is previously
created.




                                                             11                                                 108
Engineered Systems


The Public network information including which interface to use is the next set of information. You can either use the 10G
network (xbond0) or the 1G network (bond0) depending on your network configuration. It is very important that you
consult your network team regarding your configuration and ensure cables are plugged into the right ports.




If you have any other networks to configure like a Backup network/NAS network etc . you will get the oppurtunity to
configure it in the next screen.




The next screen will walk you through the database configuration. Typically you should select a very small database to get a
dummy database in place and create the actual databases that are more customized at a later time.




                                                           12                                                108
Engineered Systems


The Next screen walks you thru the ASR setup process. It is essential to get an inderstanding ASR (Automatic Service
Request) functionality and the limitations of setting up the ASR utility via the deployment process. If you have direct access to
the internet from the ODA, the ASR deployment should be fairly simple. If you are accessing the internet via a proxy it is
important to note that the ODA ASR registration only supports port 80 for the proxy setup and does not support using
another port using the deployment GUI. Your ASR deployment might fail if you your proxy uses a different port, due to this
at which point you would have to manually setup the ASR.. Later versions of the ODA software (2.5) fixes the proxy port
issue.




The final decision is to the setup the ACFS file system - or as Oracle calls it: CloudFS. ACFS is a shared file system that can
be used for storing a variety of files. Currently ACFS is a restricted license option. Please consult your licensing docs to get
more information.




In conclusion, you will receive a summary of the choices made and you can being the install. The install can take anywhere
from 45 mins to an hour after which you will have a fully functional RAC cluster. This concludes the deployment of the
Oracle Database Appliance




                                                             13                                                 108
Engineered Systems




PATCHING THE ORACLE DATABASE APPLIANCE
The Oracle Database appliance (ODA) is unique in the sense that Oracle provides an all-inclusive patching strategy. An
ODA patch includes the following components
Infrastructure
    1. Integrated Lights Out Management (ILOM)/ BIOS
    2. Oracle Appliance Kit (OAK)
    3. Hardware Management Pack HMP
    4. Automatic Service Request (ASR)
    5. Intelligent Platform Management Interface IPMI
    6. Operating SSystem
    7. Storage Firmware


Grid Infrastructure


The GI patch allows for patching and/or upgrading the Clusterware and ASM software which are now collectively known as
the grid infrastructure.


Database


The integrated patching allows for patching and/or upgrading the database software. Oracle currently supports 11.2.0.2.x as
well as 11.2.0.3.x on the Database appliance.
Oracle =can also introduce new features in each Oracle Appliance Kit patch. Below is the history of some of the features that
were introduced during each patchset.
HISTORY     OF OAK/ODA PATCHES
ODA 2.1.0.1/2.1.0.2
      1. Base Release
      2. 11.2.0.2.4
ODA 2.1.0.3/2.1.0.3.1
      1. 11.2.0.2.5
      2. Added ASR ILOM configuration
      3. ILOM/BIOS and Firmware fixes
      4. 2.1.0.3.1 patches a critical disk flaw
ODA 2.2.0
      1. Introduced 11.2.0.3.2 and provided the means to upgrade from 11.2.0.2
      2. Upgraded the kernel to UEK 5.8
      3. Odachk introduced perform ODA health checks

                                                           14                                               108
Engineered Systems


ODA 2.3.0
      1.    Prerequisite of ODA 2.2
      2.    Multiple Database Homes now supported
      3.    11.2.0.2.7 and 11.2.0.3.3 supported in a multi home environment
      4.    Oakcli command set enhanced to create databases and database homes


ODA 2.4.0
      1. Normal redundancy option for new deployment of DATA and RECO(existing deployments require
          redeployment)
      2. Trace File analyzer
      3. Grid Infrastructure and Database patching is rolling upgradable.
ODA 2.5.0
      1. Virtualization as an option.
      2. External ASR option
      3. ASR Proxy port change


PATCHING
Patching the ODA is performed using the Oracle Appliance Kit (OAK) interface. Each version comes with a
comprehensive readme that describes patching behavior. For the purposes of this document we will talk about applying a
standard patch. As just described, an ODA patch will patch the OS as well as database and clusterware components.
Depending on the criticality of the application, as of ODA 2.4 OS patching still requires downtime. However the GI and
Database patching are both rolling upgradable. It depends on your patching strategy and you might want to take a smaller
downtime to patch your OS and follow it up with the rolling patching for GI and Database components.
Since the ODA patch includes a lot of components it is important to understand each component and the impact it will have
on your system. The patch readme as well as MOS Note 888888.1 does a good job of explaining all the changes in the patch.
It is important that the person applying the patch assess the impact and apply accordingly.
Before starting a patching session it is always important to check the version of the patch deployed.

                                   #/opt/oracle/oak/bin/oakcli show version


The version will help determine which patchsets are necessary. If you are on a pre ODA 2.2 patch set, it is important to first
patch up to 2.2 before moving further. This is due to significant OS changes that were made in ODA patch 2.2.
Once the patch determination has been made it is important to understand your network components to understand how the
patching works. As an example, ILOM and BIOS patching will take significantly longer if your ILOM is on a different
network than the database node public network. Typically this is due to network zoning/segmentation.
If you have such a case it is always better to patch the ILOM outside of the ODA patch for speed. To do so you can follow
the procedures outlined below. Please note below we are assuming a system with ODA 2.3 installed patching to 2.4. The first
3 steps can be done ahead of time and require no downtime is required




                                                           15                                                108
Engineered Systems




     # ./oakcli unpack -package <absolute_path_to_patch_file p14752755_24000_Linux-x86-64.zip> (On
     Both Nodes)
     # cd /opt/oracle/oak/pkgrepos/thirdpartypkgs/Firmware/Ilom/SUN/541-4116-03/3.0.16.22.a-es-
     r75629/Base (On Both Nodes)
     # ipmiflash -v write ILOM-3_0_16_22_a_r75629-Sun_Fire_X4370_M2-rom.pkg force script config delaybios
     warning=0 (On Both Nodes)
     # ipmitool chassis power cycle (Will bring down the node and does require downtime. Needs to be run on
The infrastructure patch a delay)
     each node but with will recognize that the ILOM/BIOS has already been patched and will ignore them in the
patching cycle.nodes come up. You can patch the infra and it will be significantly faster
     Once the
     # /opt/oracle/oak/bin/oakcli update -patch 2.4.0.0.0 –infra
The downtime related tasks end here if the purpose is patching only the server and not upgrade the grid and database
software. The next step is patching the Grid Infrastructure and the patch is rolling upgradable which means a full
downtime is not required.


                            # cd /opt/oracle/oak/bin
                            # ./oakcli update -patch 2.4.0.0.0 –gi
This will patch your grid infrastructure components to 11.2.0.3.4.
The final piece in the patching puzzle is the database and that is rolling upgradable as well. As part of the command you will
be asked which Oracle homes to patch.
                          # cd /opt/oracle/oak/bin
                          # ./oakcli update -patch 2.4.0.0.0 –database

This concludes the patching exercise on the ODA. As documented patching and maintaining and ODA is much simpler than
managing and patching a similar set of components separately
.

ONE BUTTON ORACLE DATABASE APPLIANCE
Oracle has used the concept of one button for a lot of activities surrounding the Oracle Database Appliance. The Oracle
Appliance Kit (OAK) is the management brains of the Oracle Database Appliance. The Oracle Appliance Kit Daemon
(oakd) maintains the link between hardware and software in the ODA.
ONE BUTTON DEPLOY
We’ve already discussed deployment but this is one of the key features of the Oracle Database Appliance (ODA). One
button deployment allows for rapidly deploying or re-deploying the ODA with one command.

                          # cd /opt/oracle/oak/bin
                          # ./oakcli deploy




                                                            16                                                 108
Engineered Systems



ONE BUTTON VALIDATE
Oracle Database appliance comes with one button validation which should be performed to ensure that the system is
performing optimally prior to going live. It is very important to understand the various options that the validation tool offers
from the documentation.

                           # cd /opt/oracle/oak/bin
                           # ./oakcli validate -h

The table of options below tells you about the various validation commands you can run. It is essential to
execute basic validation on a regular basis to ensure the system is stable.




Oracle Database Appliance Validation Tool Options
Option                                                          Purpose
         Run all system checks, including Disk Calibration. Oracle recommends you use this command to validate
-a       system readiness before deployment. Do not run oakcli validate with this option on a loaded system,
         because the Disk Calibration system check can cause performance issues.
-c       Run a comma-delimited list of checks. Use this command to run a specific check or list of checks.
         Run the default system checks. The default system checks are SystemComponents, OSDiskStorage,
-d
         SharedStorage, and NetworkComponents
         Create an output file. Provide the name of the file after the -f flag. The output for the validation commands
-f       is printed to this file. If you do not specify that output is sent to a file, then it is sent to the Page display
         (stdout).
-h       Print help information.


-v       Provide verbose output.
-V       Print the validation tool version.

An example of a system problem can be shown here.

RESULT: Powersupply PS0 V_IN=Disabled I_IN=Disabled V_OUT=0.88 Volts I_OUT=2 Amps
IN_POWER=Disabled OUT_POWER=Disabled – as you can see I_IN and V_IN is disabled which mean there is an
issue
RESULT: Powersupply PS1 V_IN=206 Volts I_IN=3.75 Amps V_OUT=12.08 Volts I_OUT=4.40 Amps
IN_POWER=720 Watts OUT_POWER=670 Watts



A system without a power supply issue would display something like this:
RESULT: Powersupply PS0 V_IN=204 Volts I_IN=1.88 Amps V_OUT=12.08 Volts I_OUT=27.20 Amps
IN_POWER=380 Watts OUT_POWER=330 Watts --
RESULT: Powersupply PS1 V_IN=202 Volts I_IN=1.88 Amps V_OUT=12.08 Volts I_OUT=27.20 Amps108
                                                     17
IN_POWER=350 Watts OUT_POWER=340 Watts
SUCCESS: Both the powersupply are ok and functioning
Engineered Systems




As you can see, even though the system is functioning due to the redundant nature of the system components. If there is an
issue, then running an oakcli validate can identify the source of the system or hardware problem. These types of issues can be
proactively determined by using ASR (Automatic Service Request).


Validate also provides DiskCalibration data to ensure that the disk is functioning as expected and no disk failures.
RESULT: Random read throughput across all 20 SASDisk = 4037 IOPS
RESULT: Random read throughput across all 4 SSDDisk = 16368 IOPS
RESULT: Random read throughput across all 20 SASDisk = 1844 MBPS
RESULT: Random read throughput across all 4 SSDDisk = 941 MBPS
An issue with disk access or calibration can also be seen below. In this example, a disk was manually dropped from the
diskgroup to simulate this test.

 ERROR: Error when running orion for HDD_E0_S01_705927132_MBPS : ORION: ORacle IO Numbers -- Version 11.2.0.3.0
 Calibration will take approximately 1 minutes.
 Using a large value for -cache_size may take longer.
 Error completing IO (storax_aiowait)
 ORA-27061: waiting for async I/Os failed
 Linux-x86_64 Error: 5: Input/output error
 Additional information: -1


A Network Check will look at the network components to validate the network speed and Ethernet ports that are currently
active. Oracle Database Appliance has a one 10G interface per node as well several 1G interfaces. In the example only 1G
interfaces are active.


INFO: Doing oak network checks
RESULT: Detected active link for interface eth0 with link speed 1000Mb/s
RESULT: Detected active link for interface eth1 with link speed 1000Mb/s
RESULT: Detected active link for interface eth2 with link speed 1000Mb/s
RESULT: Detected active link for interface eth3 with link speed 1000Mb/s
RESULT: Detected active link for interface eth4 with link speed 1000Mb/s
WARNING: No Link detected for interface eth5
WARNING: No Link detected for interface eth6
WARNING: No Link detected for interface eth7
WARNING: No Link detected for interface eth8
WARNING: No Link detected for interface eth
99

The oakcli validate one button deployment provides a plethora of diagnostic information that provide insight into the
operation of the Oracle Database Appliance. It should be used regularly, but with caution, to ensure the machine is
functioning as intended.




                                                                           18                                       108
Engineered Systems



ONE BUTTON STORAGE DIAGNOSTICS
Disks can fail, so it is important to ensure that storage diagnostics are key as part of the one button deployment of the ODA.
Storage diagnostics are a key component in the oakcli command set. From the ODA documentation, below are some of the
commands that are available for storage diagnostic and disk location.


                             Option                                                       Purpose
                                                              Locates the disk associated with the ASM disk that you
                                                              indicate by its ASM disk name, where:

                                                              -h outputs the oakcli locate options

oakcli locate[-h] [disk diskname on|off]                      disk diskname locates the ASM disk whose name you
                                                              designate

                                                              on turns on the LED of the specified disk

                                                              off turns off the LED of the specified disk
oakcli show controller
                                                              Displays the configuration for active cores as defined by
oakcli show core_config_key
                                                              the registered core key
oakcli show disk                                              Displays disks
oakcli show diskgroup                                         Displays disk groups
oakcli show expander                                          Displays expanders
oakcli show –h                                                Show help command
oakcli show storage                                           Displays storage device status
Provides a detailed output patch versions for software
and firmware on the appliance

It is very important to understand and execute some of the above commands to ensure the stability and understand the state
of the disk.


    #oakcli show disk
     NAME          PATH      TYPE     STATE     STATE_DETAILS
     pd_00       /dev/sdam   HDD      ONLINE     Good
     pd_01       /dev/sdaw   HDD      ONLINE    Good
     pd_02       /dev/sdaa   HDD      ONLINE    Good
     pd_03       /dev/sdak   HDD      ONLINE    Good



Based on the output above, if there is a bad disk, you can work with Oracle Field Services or your internal hardware resources
to replace the disk. The oakcli locate disk helps in ensuring the right bad disk is located and replaced. Also, the show storage
option provides a complete picture of the storage which includes the controllers, expanders as well as firmware version of the
disks.

                                                            19                                                 108
Engineered Systems



ONE BUTTON ODA HEALTHCHECKS
This piece is new since ODA 2.3.x and is not part of the OAK framework. However it is important to note and use as
needed. Oracle Database Appliance Checks, or ODAchk as the tool is known, is the Oracle Database Appliance version of
the popular RACchk and EXAchk tools.
Most of the OAK built in tools look at hardware and software, but the ODAchk provides more in-depth looks the system
from the database perspective and not the appliance perspective. The tool ensures things like RAC, Grid, ASM as well as
common database configuration parameters are set up right. ODAchk should be run on a regular basis and is recommended
before and after any system change as well as any system patching. It is surprising how sometimes on a running system how
many best practices are not being followed.

   $/opt/oracle/oak/odachk/odachk – Please run as oracle software owner.
FAIL => Database parameter DB_LOST_WRITE_PROTECT is NOT set to recommended value on tst12 instance
FAIL => Database parameter GLOBAL_NAMES is NOT set to recommended value on tst12 instance
FAIL => One or more errors or warnings for OS Disk Storage checks
WARNING => One or file generated network and bonding interface checks
There is an html more warnings for with detail information as well              as best practices to follow.


ONE BUTTON MANAGEMENT DIAGNOSTIC COLLECTION
The OAK framework comes with its own management and diagnostic collection framework including a cleanup script to re-
initialize the Oracle Database Appliance and establish “pay as you go” Oracle licensing core counts.
Oakcli manage diagcollect – Collect and package diagnostic files for Oracle Support

        # cd /opt/oracle/oak/bin
        # ./oakcli manage diagcollect – this will collect diagnostic data from CRS and various DB and OAK components
        to submit to oracle


Oakcli manage cleanrepo – Remove old versions of ODA patch and version files

      #cd/opt/oracle/oak/bin
      # oakcli manage cleanrepo --ver 2.4.0.0.0
      Can not delete the current or latest version repository.
      #oakcli manage cleanrepo --ver 2.3.0.0.0 – to cleanup previous version files
       # oakcli update –patch 2.4.0.0.0 –clean – in 2.4 has been enhanced to cleanup temporary files after
          patching



CleanupDeploy.pl
If there is ever a need to reimage the box or uninstall all software you can use the CleanupDeploy.pl to do so

     1. Log in as root.
     2. Change directory to /opt/oracle/oak/onecmd.
     3. Run the command cleanupDeploy.pl.

     For example:
     # ./cleanupDeploy.pl
                                                                           20                                    108
Engineered Systems




Configuring the Oracle licensing core counts
If you are using the pay as go Oracle Licensing scheme you can use oakcli to apply the core cap and install the key you have
generated on MOS (My Oracle Support).
    # oakcli apply –h
    Usage:oakcli apply core_config_key <key_file>

    # oakcli show core_config_key
    Optional core_config_key is not applied on this machine yet !

   # oakcli show core_config_key
   Host's serialnumber = ABC123
   Configured Cores = 20
Trace File Analysis


In Oracle Database Appliance 2.4 a new option for TFA was installed. The Trace File Analyzer is a new framework for the
collection and analysis of trace files that help the proactive collection of errors in real-time mode and automatically collects
the right log files as the problem is happening. More information on TFA can be found in MOS note 1513912.1.


ONE BUTTON PATCHING
This paper already discusses patching but in the one button section it is important to mention the capabilities of the one
button infrastructure to be able to patch all software or individual components using the one button patching capabilities built
into the Oracle Appliance Kit.

    # cd /opt/oracle/oak/bin
    # ./oakcli update -patch version




While the patching section of this paper discusses more detail, the command above is the but in essence of single button
patching, The command above can patch an appliance.


ONE BUTTON DATABASE DEPLOYMENT
Starting with ODA version 2.3 and above, newer releases of the ODA 2.3 and above , the option is now provided to create
multiple database homes as well as create new databases. There are options to upgrade database as well delete databases and
homes as well.


To create a new database home
    # oakcli create dbhome -version 11.2.0.3.3 – will create a new dbhome with 11.2.0.3.3



                                                              21                                                 108
Engineered Systems


To create a new database

    # oakcli create database -db testdb -version 11.2.0.3.3 – will create a database and a new database home
    #oakcli create database -db testdb –oh OraDb11203_home2 – will create a new database in an existing dbhome.

To delete a database or database home
 # oakcli delete database –db olddb
 # oakcli delete dbhome –oh dbhome_name
To. upgrade a database from one version to another (11.2.0.2 to 11.2.0.3)

   #oakcli upgrade database {-db <db_names> | -from <source_home>} -to <destination_home>.

SUMMARY
In summary, the Oracle Database Appliance packs a lot of bang for the buck. As the basic model in Oracle engineered
Appliance product line, it is the stepping stone to understand and evaluate if an appliance model is a viable option for a
company. An Oracle Database Appliance provides ease of setup, patching as well as diagnostics. As an administrator that has
previously only had a specific focus, there are a lot of things that an administrator has to be aware of, including the network,
hardware and operating system. Oracle has made it easy for people to manage the appliance as one unit by making various
aspects easier to manage and maintain using the Oracle Appliance Kit (OAK) framework. There are various monitoring
aspects built into the box as well as one button activities to ensure that management of the appliance is simple.


REFERENCES

Oracle Database Appliance - 2.X Supported Versions & Known Issues [ID 888888.1]
Information Center: Oracle Database Appliance [ID 1417713.2]
ODA (Oracle Database Appliance): New Deploy Step-by-Step Short Guide (version 2.3) [ID 1486534.1]
ODA (Oracle Database Appliance): oakcli command reference [ID 1417879.1]
ODA (Oracle Database Appliance): How To Configure xbond,bond1,bond2 post deploy [ID 1422563.1]
ODA (Oracle Database Appliance): The Steps to replace failing disks [ID 1496114.1]
Gathering data for the Oracle Database Appliance server [ID 1358795.1]
ODA (Oracle Database Appliance): Test Plan Outline [ID 1474273.1]
Oracle Database Appliance Setup Poster
TFA Collector- The Preferred Tool for Automatic or ADHOC Diagnostic Gathering Across All Cluster Nodes [ID 1513912.1]




                                                                22                                                108

More Related Content

What's hot

Running E-Business Suite Database on Oracle Database Appliance
Running E-Business Suite Database on Oracle Database ApplianceRunning E-Business Suite Database on Oracle Database Appliance
Running E-Business Suite Database on Oracle Database ApplianceMaris Elsins
 
My First 100 days with an Exadata (PPT)
My First 100 days with an Exadata (PPT)My First 100 days with an Exadata (PPT)
My First 100 days with an Exadata (PPT)Gustavo Rene Antunez
 
Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP
Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISPOptimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP
Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISPSecure-24
 
Exadata x3 workshop
Exadata x3 workshopExadata x3 workshop
Exadata x3 workshopFran Navarro
 
Customer overview oracle solaris cluster, enterprise edition
Customer overview oracle solaris cluster, enterprise editionCustomer overview oracle solaris cluster, enterprise edition
Customer overview oracle solaris cluster, enterprise editionsolarisyougood
 
Exadata Patching Demystified
Exadata Patching DemystifiedExadata Patching Demystified
Exadata Patching DemystifiedEnkitec
 
Sharing experience implementing Direct NFS
Sharing experience implementing Direct NFSSharing experience implementing Direct NFS
Sharing experience implementing Direct NFSYury Velikanov
 
Sun Oracle Exadata Technical Overview V1
Sun Oracle Exadata Technical Overview V1Sun Oracle Exadata Technical Overview V1
Sun Oracle Exadata Technical Overview V1jenkin
 
Oracle Flex ASM - What’s New and Best Practices by Jim Williams
Oracle Flex ASM - What’s New and Best Practices by Jim WilliamsOracle Flex ASM - What’s New and Best Practices by Jim Williams
Oracle Flex ASM - What’s New and Best Practices by Jim WilliamsMarkus Michalewicz
 
Understanding Oracle RAC 11g Release 2 Internals
Understanding Oracle RAC 11g Release 2 InternalsUnderstanding Oracle RAC 11g Release 2 Internals
Understanding Oracle RAC 11g Release 2 InternalsMarkus Michalewicz
 
Oracle 11G SCAN: Concepts and Implementation Experience Sharing
Oracle 11G SCAN: Concepts and Implementation Experience SharingOracle 11G SCAN: Concepts and Implementation Experience Sharing
Oracle 11G SCAN: Concepts and Implementation Experience SharingYury Velikanov
 
Výhody a benefity nasazení Oracle Database Appliance
Výhody a benefity nasazení Oracle Database ApplianceVýhody a benefity nasazení Oracle Database Appliance
Výhody a benefity nasazení Oracle Database ApplianceMarketingArrowECS_CZ
 
Why Oracle on IBM POWER7 is Better Than Oracle Exadata - The Advantages of IB...
Why Oracle on IBM POWER7 is Better Than Oracle Exadata - The Advantages of IB...Why Oracle on IBM POWER7 is Better Than Oracle Exadata - The Advantages of IB...
Why Oracle on IBM POWER7 is Better Than Oracle Exadata - The Advantages of IB...miguelnoronha
 
Exadata is still oracle
Exadata is still oracleExadata is still oracle
Exadata is still oracleugur candan
 
Enterprise Deployments: The Real World of Best Practices
Enterprise Deployments: The Real World of Best PracticesEnterprise Deployments: The Real World of Best Practices
Enterprise Deployments: The Real World of Best PracticesSimon Haslam
 

What's hot (20)

Running E-Business Suite Database on Oracle Database Appliance
Running E-Business Suite Database on Oracle Database ApplianceRunning E-Business Suite Database on Oracle Database Appliance
Running E-Business Suite Database on Oracle Database Appliance
 
My First 100 days with an Exadata (PPT)
My First 100 days with an Exadata (PPT)My First 100 days with an Exadata (PPT)
My First 100 days with an Exadata (PPT)
 
Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP
Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISPOptimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP
Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP
 
Exadata x3 workshop
Exadata x3 workshopExadata x3 workshop
Exadata x3 workshop
 
Exadata Backup
Exadata BackupExadata Backup
Exadata Backup
 
Customer overview oracle solaris cluster, enterprise edition
Customer overview oracle solaris cluster, enterprise editionCustomer overview oracle solaris cluster, enterprise edition
Customer overview oracle solaris cluster, enterprise edition
 
Smooth as Silk Exadata Patching
Smooth as Silk Exadata PatchingSmooth as Silk Exadata Patching
Smooth as Silk Exadata Patching
 
Exadata Patching Demystified
Exadata Patching DemystifiedExadata Patching Demystified
Exadata Patching Demystified
 
Sharing experience implementing Direct NFS
Sharing experience implementing Direct NFSSharing experience implementing Direct NFS
Sharing experience implementing Direct NFS
 
Sun Oracle Exadata Technical Overview V1
Sun Oracle Exadata Technical Overview V1Sun Oracle Exadata Technical Overview V1
Sun Oracle Exadata Technical Overview V1
 
Oow Ppt 2
Oow Ppt 2Oow Ppt 2
Oow Ppt 2
 
Sparc solaris servers
Sparc solaris serversSparc solaris servers
Sparc solaris servers
 
Oracle Flex ASM - What’s New and Best Practices by Jim Williams
Oracle Flex ASM - What’s New and Best Practices by Jim WilliamsOracle Flex ASM - What’s New and Best Practices by Jim Williams
Oracle Flex ASM - What’s New and Best Practices by Jim Williams
 
Understanding Oracle RAC 11g Release 2 Internals
Understanding Oracle RAC 11g Release 2 InternalsUnderstanding Oracle RAC 11g Release 2 Internals
Understanding Oracle RAC 11g Release 2 Internals
 
Oracle 11G SCAN: Concepts and Implementation Experience Sharing
Oracle 11G SCAN: Concepts and Implementation Experience SharingOracle 11G SCAN: Concepts and Implementation Experience Sharing
Oracle 11G SCAN: Concepts and Implementation Experience Sharing
 
Výhody a benefity nasazení Oracle Database Appliance
Výhody a benefity nasazení Oracle Database ApplianceVýhody a benefity nasazení Oracle Database Appliance
Výhody a benefity nasazení Oracle Database Appliance
 
Why Oracle on IBM POWER7 is Better Than Oracle Exadata - The Advantages of IB...
Why Oracle on IBM POWER7 is Better Than Oracle Exadata - The Advantages of IB...Why Oracle on IBM POWER7 is Better Than Oracle Exadata - The Advantages of IB...
Why Oracle on IBM POWER7 is Better Than Oracle Exadata - The Advantages of IB...
 
Exadata is still oracle
Exadata is still oracleExadata is still oracle
Exadata is still oracle
 
Enterprise Deployments: The Real World of Best Practices
Enterprise Deployments: The Real World of Best PracticesEnterprise Deployments: The Real World of Best Practices
Enterprise Deployments: The Real World of Best Practices
 
Oracle Exadata X2-8: A Critical Review
Oracle Exadata X2-8: A Critical ReviewOracle Exadata X2-8: A Critical Review
Oracle Exadata X2-8: A Critical Review
 

Similar to Oracle Database Appliance - RAC in a box Some strings attached

Presentation sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overviewPresentation   sparc m6 m5-32 server technical overview
Presentation sparc m6 m5-32 server technical overviewsolarisyougood
 
Sparc m6 32 in-memory infrastructure for the entire enterprise
Sparc m6 32 in-memory infrastructure for the entire enterpriseSparc m6 32 in-memory infrastructure for the entire enterprise
Sparc m6 32 in-memory infrastructure for the entire enterprisesolarisyougood
 
Sparc t4 1 system technical overview
Sparc t4 1 system technical overviewSparc t4 1 system technical overview
Sparc t4 1 system technical overviewsolarisyougood
 
My First 100 days with an Exadata (WP)
My First 100 days with an Exadata  (WP)My First 100 days with an Exadata  (WP)
My First 100 days with an Exadata (WP)Gustavo Rene Antunez
 
SDN, OpenFlow, NFV, and Virtual Network
SDN, OpenFlow, NFV, and Virtual NetworkSDN, OpenFlow, NFV, and Virtual Network
SDN, OpenFlow, NFV, and Virtual NetworkTim4PreStartup
 
DAOS - Scale-Out Software-Defined Storage for HPC/Big Data/AI Convergence
DAOS - Scale-Out Software-Defined Storage for HPC/Big Data/AI ConvergenceDAOS - Scale-Out Software-Defined Storage for HPC/Big Data/AI Convergence
DAOS - Scale-Out Software-Defined Storage for HPC/Big Data/AI Convergenceinside-BigData.com
 
Nové vlastnosti Oracle Database Appliance
Nové vlastnosti Oracle Database ApplianceNové vlastnosti Oracle Database Appliance
Nové vlastnosti Oracle Database ApplianceMarketingArrowECS_CZ
 
Sparc t4 4 system technical overview
Sparc t4 4 system technical overviewSparc t4 4 system technical overview
Sparc t4 4 system technical overviewsolarisyougood
 
Shak larry-jeder-perf-and-tuning-summit14-part1-final
Shak larry-jeder-perf-and-tuning-summit14-part1-finalShak larry-jeder-perf-and-tuning-summit14-part1-final
Shak larry-jeder-perf-and-tuning-summit14-part1-finalTommy Lee
 
Application Verification Calibration Circuit For Workstations
Application Verification Calibration Circuit For WorkstationsApplication Verification Calibration Circuit For Workstations
Application Verification Calibration Circuit For WorkstationsPeggy Johnson
 
Using SAS GRID v 9 with Isilon F810
Using SAS GRID v 9 with Isilon F810Using SAS GRID v 9 with Isilon F810
Using SAS GRID v 9 with Isilon F810Boni Bruno
 
Squash Those IoT Security Bugs with a Hardened System Profile
Squash Those IoT Security Bugs with a Hardened System ProfileSquash Those IoT Security Bugs with a Hardened System Profile
Squash Those IoT Security Bugs with a Hardened System ProfileSteve Arnold
 
En3702 server specifications
En3702 server specificationsEn3702 server specifications
En3702 server specificationsHeo Gòm
 
Syllabus for interview
Syllabus for  interviewSyllabus for  interview
Syllabus for interviewSwapnil Kapate
 
Virtualization with Lenovo X6 Blade Servers: white paper
Virtualization with Lenovo X6 Blade Servers: white paperVirtualization with Lenovo X6 Blade Servers: white paper
Virtualization with Lenovo X6 Blade Servers: white paperLenovo Data Center
 
ODSA Use Case - SmartNIC
ODSA Use Case - SmartNICODSA Use Case - SmartNIC
ODSA Use Case - SmartNICODSA Workgroup
 
Network Processing on an SPE Core in Cell Broadband EngineTM
Network Processing on an SPE Core in Cell Broadband EngineTMNetwork Processing on an SPE Core in Cell Broadband EngineTM
Network Processing on an SPE Core in Cell Broadband EngineTMSlide_N
 
Hpe pro liant dl180 generation9 (gen9)
Hpe pro liant dl180 generation9 (gen9) Hpe pro liant dl180 generation9 (gen9)
Hpe pro liant dl180 generation9 (gen9) Sourav Dash
 

Similar to Oracle Database Appliance - RAC in a box Some strings attached (20)

Presentation sparc m6 m5-32 server technical overview
Presentation   sparc m6 m5-32 server technical overviewPresentation   sparc m6 m5-32 server technical overview
Presentation sparc m6 m5-32 server technical overview
 
Sparc m6 32 in-memory infrastructure for the entire enterprise
Sparc m6 32 in-memory infrastructure for the entire enterpriseSparc m6 32 in-memory infrastructure for the entire enterprise
Sparc m6 32 in-memory infrastructure for the entire enterprise
 
Sparc t4 1 system technical overview
Sparc t4 1 system technical overviewSparc t4 1 system technical overview
Sparc t4 1 system technical overview
 
My First 100 days with an Exadata (WP)
My First 100 days with an Exadata  (WP)My First 100 days with an Exadata  (WP)
My First 100 days with an Exadata (WP)
 
SDN, OpenFlow, NFV, and Virtual Network
SDN, OpenFlow, NFV, and Virtual NetworkSDN, OpenFlow, NFV, and Virtual Network
SDN, OpenFlow, NFV, and Virtual Network
 
DAOS - Scale-Out Software-Defined Storage for HPC/Big Data/AI Convergence
DAOS - Scale-Out Software-Defined Storage for HPC/Big Data/AI ConvergenceDAOS - Scale-Out Software-Defined Storage for HPC/Big Data/AI Convergence
DAOS - Scale-Out Software-Defined Storage for HPC/Big Data/AI Convergence
 
Nové vlastnosti Oracle Database Appliance
Nové vlastnosti Oracle Database ApplianceNové vlastnosti Oracle Database Appliance
Nové vlastnosti Oracle Database Appliance
 
Sparc t4 4 system technical overview
Sparc t4 4 system technical overviewSparc t4 4 system technical overview
Sparc t4 4 system technical overview
 
Shak larry-jeder-perf-and-tuning-summit14-part1-final
Shak larry-jeder-perf-and-tuning-summit14-part1-finalShak larry-jeder-perf-and-tuning-summit14-part1-final
Shak larry-jeder-perf-and-tuning-summit14-part1-final
 
Application Verification Calibration Circuit For Workstations
Application Verification Calibration Circuit For WorkstationsApplication Verification Calibration Circuit For Workstations
Application Verification Calibration Circuit For Workstations
 
Using SAS GRID v 9 with Isilon F810
Using SAS GRID v 9 with Isilon F810Using SAS GRID v 9 with Isilon F810
Using SAS GRID v 9 with Isilon F810
 
Squash Those IoT Security Bugs with a Hardened System Profile
Squash Those IoT Security Bugs with a Hardened System ProfileSquash Those IoT Security Bugs with a Hardened System Profile
Squash Those IoT Security Bugs with a Hardened System Profile
 
Clustering
ClusteringClustering
Clustering
 
En3702 server specifications
En3702 server specificationsEn3702 server specifications
En3702 server specifications
 
Syllabus for interview
Syllabus for  interviewSyllabus for  interview
Syllabus for interview
 
Virtualization with Lenovo X6 Blade Servers: white paper
Virtualization with Lenovo X6 Blade Servers: white paperVirtualization with Lenovo X6 Blade Servers: white paper
Virtualization with Lenovo X6 Blade Servers: white paper
 
ODSA Use Case - SmartNIC
ODSA Use Case - SmartNICODSA Use Case - SmartNIC
ODSA Use Case - SmartNIC
 
Network Processing on an SPE Core in Cell Broadband EngineTM
Network Processing on an SPE Core in Cell Broadband EngineTMNetwork Processing on an SPE Core in Cell Broadband EngineTM
Network Processing on an SPE Core in Cell Broadband EngineTM
 
Session 307 ravi pendekanti engineered systems
Session 307  ravi pendekanti engineered systemsSession 307  ravi pendekanti engineered systems
Session 307 ravi pendekanti engineered systems
 
Hpe pro liant dl180 generation9 (gen9)
Hpe pro liant dl180 generation9 (gen9) Hpe pro liant dl180 generation9 (gen9)
Hpe pro liant dl180 generation9 (gen9)
 

Recently uploaded

20140402 - Smart house demo kit
20140402 - Smart house demo kit20140402 - Smart house demo kit
20140402 - Smart house demo kitJamie (Taka) Wang
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
 
UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2DianaGray10
 
March Patch Tuesday
March Patch TuesdayMarch Patch Tuesday
March Patch TuesdayIvanti
 
Where developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is goingWhere developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is goingFrancesco Corti
 
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptxEmil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptxNeo4j
 
Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.IPLOOK Networks
 
From the origin to the future of Open Source model and business
From the origin to the future of  Open Source model and businessFrom the origin to the future of  Open Source model and business
From the origin to the future of Open Source model and businessFrancesco Corti
 
Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...DianaGray10
 
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024Alkin Tezuysal
 
Oracle Database 23c Security New Features.pptx
Oracle Database 23c Security New Features.pptxOracle Database 23c Security New Features.pptx
Oracle Database 23c Security New Features.pptxSatishbabu Gunukula
 
Automation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsAutomation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsDianaGray10
 
The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)codyslingerland1
 
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedInOutage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedInThousandEyes
 
How to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptxHow to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptxKaustubhBhavsar6
 
EMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? WebinarEMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? WebinarThousandEyes
 
2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdf2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdfThe Good Food Institute
 
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES LiveKeep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES LiveIES VE
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightSafe Software
 
UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4DianaGray10
 

Recently uploaded (20)

20140402 - Smart house demo kit
20140402 - Smart house demo kit20140402 - Smart house demo kit
20140402 - Smart house demo kit
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdf
 
UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2
 
March Patch Tuesday
March Patch TuesdayMarch Patch Tuesday
March Patch Tuesday
 
Where developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is goingWhere developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is going
 
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptxEmil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
 
Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.
 
From the origin to the future of Open Source model and business
From the origin to the future of  Open Source model and businessFrom the origin to the future of  Open Source model and business
From the origin to the future of Open Source model and business
 
Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...
 
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
 
Oracle Database 23c Security New Features.pptx
Oracle Database 23c Security New Features.pptxOracle Database 23c Security New Features.pptx
Oracle Database 23c Security New Features.pptx
 
Automation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsAutomation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projects
 
The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)
 
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedInOutage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
 
How to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptxHow to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptx
 
EMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? WebinarEMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? Webinar
 
2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdf2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdf
 
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES LiveKeep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4
 

Oracle Database Appliance - RAC in a box Some strings attached

  • 1. Engineered Systems Oracle database appliance v1 - rac in a box – some strings attached Fuad Arshad, Walgreens Company ABSTRACT The Oracle Database Appliance is an engineered system that is targeted towards small and medium businesses, but also has all of the bells and whistles required for enterprise deployments. This paper will introduce the Oracle Database Appliance (ODA), and review the deployment of the ODA from a non sysadmin perspective. This paper includes the deployment and patching processes and the best practices built into the Oracle Database Appliance. The paper will review the DBA's role in managing the ODA TARGET AUDIENCE This paper is intended mostly for Database Architects and people that deploy Oracle database technologies. This paper will review various techniques for deploying the Oracle Database Appliance (ODA) successfully as well as how to perform day to day activities. The various components all tied together provide an effective use case for companies that want to invest in Engineered Systems, and provide an entry point to understand the management and maintenance considerations for deploying an engineered system. EXECUTIVE SUMMARY Attendees and readers of this paper will learn the following after reading this paper. Oracle Database Appliance - Overview Oracle Database Appliance - Predeployment Oracle Database Appliance - Deployment Oracle Database Appliance - Patching Oracle Database Appliance - One Button Database Appliance INTRODUCTION The Oracle Database Appliance (ODA) is an entry level engineered system that provides easy access to preconfigured Real Application Cluster (RAC) database hardware and software. The ODA is accessible to customers at an entry level cost with software licensing available starting from 2 cores to 24 cores. The ODA is intended to provide companies with an easy to deploy model that includes all of the hardware and software required to run a complete system into one 4u rack mountable unit. INSIDE THE ORACLE DATABASE APPLIANCE The Oracle Database Appliance (ODA) is a 4u rack mountable unit which consists of two 2u Sun Fire X4370 M2 units known as Server units (SN). Oracle has provided MOS Note 1385831.1 which is kept fairly up-to-date with the ODA components and software versions. Below is a snapshot as of Dec 14, 2012 and this paper only focuses on V1 functionality. Sun X4370M2 System Server Components (2 Server Nodes per X4370M2) CPU 2x 6-core Intel Xeon X5675 3.07GHz 1 108
  • 2. Engineered Systems Memory 96 GB RAM (12 x DDR3-1333 8GB DIMMs) 2x 10GbE (SFP+) PCIe card Network 6x 1GbE PCIe card 2x 1GbE (Intel 82571) on-board Integrated redundant cluster interconnect 2x 500GB SATA - for operating system Internal Storage 1x 4GB USB internal RAID 2x LSI SAS9211-8i SAS HBA Controller 20x 600GB - 3.5" SAS 15k RPM HDD (Seagate Cheetah) - for RDBMS DATA (any slot except the top row of disks) Shared Storage 4x 73GB - 3.5" SAS2 SSDs - for RDBMS REDO (slot in the top row of 4 disks) SSD from STEC (ZeusIOPS - multi-level cell (MLC) version with SAS interface) Operating Oracle Enterprise Linux 5.5 (on ODA 2.1) 5.8 (on ODA 2.2) x86_64 System The two ODA Server Nodes are packaged in a 4u chassis that includes 2 server nodes as well as direct attached disk in the front. Below are front and back images of an ODA. looks like from the front and back The bottom server is Node 0 and the top is Node 1.These will be referred to as oak1 and oak2 in the factory default configuration. It is very important to keep this picture handy to work with the data center installation team to ensure they understand how each port is used to cable the ODA to meet your requirements. Cabling depends on the data center security zone specs that your infrastructure defines. There might be sensitive and non- sensitive zones as well as various cabling zones. These requirements need to be considered as part of the installation process. Attached in the Appendix is a sample spreadsheet that outlines the cabling specs needed to complete an ODA Installation. 2 108
  • 3. Engineered Systems Callout Label Ethernet Bond Description 1 Power connectors Eth 4 and Eth 5 are configured as bond1. Eth 6 and Eth 7 are Eth 7, 6, 5, 4 bond1, configured as bond 2. These ports are used for custom 2 PCIe 1 (left to right) bond2 configurations or for separate backup, disaster recovery, and network management. Two 10-GbE ports. In 10-GbE systems, these are connected to the 3 PCIe 0 Eth 8, Eth 9 xbond0 public network. 4 SerMgt Serial connector to Oracle ILOM and system console. Net 0,Net Two 1-GbE connectors. In 1-GbE systems, these are connected to 5 Eth2, Eth3 bond0 1 the public network. 6 NetMgt Ethernet connection for Oracle ILOM USB and 7 Used for connecting to system console. Video 8 Server Node 0 9 Server Node 1 3 108
  • 4. Engineered Systems As you can see from above, it, is very important to understand the various components. The complete details are available in the MOS note referenced above, as well as the ODA Install, Configuration and User’s Guide. This paper will only touch the basic pieces that are important to create the fundamental understanding about the Oracle Database Appliance. Additional references are available at the end of this paper. PREDEPLOYMENT OF THE ORACLE DATABASE APPLIANCE The Oracle Database Appliance requires a lot of pre-deployment tasks to be completed to prepare for installing the ODA. These pre-deployment steps are standard tasks for deploying a server in a data-center and we are going to walk thru them from a Database Administrator perspective. The Oracle Database Appliance Owners guide covers most of these details. In order to get the environment ready, the site needs to meet the environmental & electrical guidelines that are explained in the Owner’s manual. The ODA weighs about 96 lbs., and utmost care must be taken while handling the hardware. Four-post racks are compatible. Two-post racks are not compatible. Horizontal and vertical pitch must conform to ANSI/EIA 310-D- 1992 or IEC 60927 standards. An Oracle Database Appliance requires 4 rack units (4RU) of rack space. A rack mounting kit is shipped as part of the hardware. Once the hardware is racked and stacked in the data-center floor, the appliance needs to cabled. If 10g network connections will be used, SFA Adapters will need to be included in the ODA order for attaching the 10g network cables to the appliance. There is a minimum requirement of 2 power connections and 4 network connections to the public network in order to access the machine. There are other networks available for use for various purposes (connecting to a NAS, ILOM or Backup Network). Once all cabling is confirmed, it is very important to create a pre-deployment checklist to ensure that all appropriate information about the system is captured.An example template can be seen below. Needed information Your Information System Name: i.e. sales TSODA1 Database Deployment Type: RAC, RAC One or RAC Standalone Internal or External Backup via NFS device External Root Passwords: default for nodes is welcome1 Leave the default for now Domain Name: Example.com.com DNS Server(s): min 1 Dns.example.com NTP Server(s): optional but recommended Ntp.example.com Node 1 Name: i.e. sales1 Tsoda11 Node 1 Public IP Address Node 1 VIP Name: i.e. sales1-vip Tsoda11-vip Node 1 Public Virtual IP Address Node 2 Name: i.e. sales2 Tsoda12 Node 2 Public IP Address Node 2 VIP Name: i.e. sales2-vip Tsoda12-vip Node 2 Public Virtual IP Address SCAN Name: sales-scan Tsoda1-scan SCAN IP Addresses: min 2 Netmask: Gateway IP: ILOM Names: ie. sales1-ilom Tsoda11-ilom ILOM IP Address: ILOM Names: i.e. sales2-ilom Tsoda12-ilom 4 108
  • 5. Engineered Systems ILOM IP Address: ILOM Passwords: default is changeme Leave the default for now ILOM Gateway: ILOM Netmask: Database Name: Tsodadb Database class: very small, small, medium, large, very Small large (optional) Proxy Server Name: for Auto Service Request Metalink Login Information: User ID/Pwd Since each Oracle Database Appliance comes with the same configuration, it is very easy to provide a configuration spreadsheet that includes various configuration specs and networking information to record the right information from the network and data center teams . An example spreadsheet that can be used to streamline the process is attached below. Microsoft Excel Worksheet It is very important to develop a standard process for deploying the ODA. This includes defining roles and responsibilities for deploying and managing the ODA. It is very important to understand these organizational responsibilities prior to deployment. A lot of activities that are required to deploy the ODA require system access and organizational structures typically have role separation. The ODA deployment activity will need the assistance of all teams within an organization from infrastructure to system administration if the role separation methodology is used. Conversely the Database Administrator deploying the ODA needs to understand various additional processes in order to successfully deploy the appliance. In the Scenario, if the Database team is responsible for deploying the Oracle Database Appliance, the following is a brief example list of activities that will need to be performed. 1. Determine hardware placement zones. 2. Submit a request to get network ip’s 3. Submit DNS and Firewall requests 4. Work with datacenter team to determine rack space. 5. Ensure cables and adapters are available 6. Read thru the ODA manual 7. Provide datacenter team cabling specs 8. Get the ODA connected to a KVM or access via Serial cable to ensure the hosts are powered on and ILOM is accessible. 9. Walk thru deployment activities. DEPLOYMENT OF THE ORACLE DATABASE APPLIANCE In order to understand how to deploy an Oracle Database Appliance (ODA) in a datacenter, one needs to understand the appliance itself. As illustrated above, The ODA consists of 2 x86 servers that have direct attached SAS as well as SSD storage. The environment is fully redundant from a hardware perspective. Storage redundancy is provided by Oracle ASM and can be triple mirrored or double mirrored (version 2.4 and above) for the SAS disk. The SSD is designed and recommended to be only used for the on-line redo logs and is triple mirrored regardless of ODA version. Oracle ASM also ensures that all the extents are mapped appropriately among the available disk to ensure that data extents are balanced. Oracle RAC is used to ensure any hardware failures on one node will not impact the running of the system as a whole. 5 108
  • 6. Engineered Systems As we look at the appliance we see 1 host node and 1 Service Processor node per hardware entity. The service processor also includes an Integrated Lights Out Management (ILOM) console which is used to manage each appliance. Deploying the ODA starts with getting the ILOM ready and accessible to the network. DEPLOYMENT OF THE ILOM The ILOM, or the Integrated Lights Out Manager, is the hardware administration and management brains and is also referred to as the service processor. The ILOM is used for various purposes in the lifecycle of a host including, but not limited to, performing a bare metal restore of an ODA. An Oracle Database Appliance contains 2 ILOM’s, or one for each of the two ODA servers. Configuring the ILOM is step 1 in the process of getting an ODA up and running. There are multiple ways to get into the ILOM. If you have a KVM device you can connect the KVM to the ODA serial management port to do the ILOM configuration remotely. The configuration steps for setting up the ILOM directly via the serial port are detailed below. ACCESSING THE ILOM VIA SERIAL PORT The ODA ILOM can be accessed via a Serial management (SerMgt) port labeled #4 in the picture above. MOS Note ID 1395445.1 explains the process of connecting to the ODA to configure the ILOM port. The process requires direct physical access to the serial port , which means, this process will need to be run at the place where the Oracle Database Appliance is installed. Each ODA ships with an RJ45-serial convertor like the one pictured below. Using an RJ-45 cable, you can connect the cable to the serial port on your laptop or Datacenter terminal cart and get into the ILOM using putty. In the case that you do not have access to a serial port, Amazon sells a usb to serial converter cable at http://amzn.com/B00425S1H8 . The cable looks like below and requires prolific drivers installed on your laptop in order to use the cable. This cable will be used in conjunction to the rj45-serial converter to connect to the ODA ILOM. Once the connection cable is connected to the laptop and ODA serial port, check the device manager to see which com port the USB device is connected to and specify the com port in your Putty connection. This doc will only talk about windows but the MOS note above includes the Linux configuration as well. Once the port is determined, you can open up putty and connect to the serial connection. 6 108
  • 7. Engineered Systems Once the connection is made you will see a login prompt. Once the ODA is cabled, in and power cords are attached, the ILOM is automatically powered on. You will need to use the root account and the default root password to connect to the ILOM. . An initial configuration of the ILOM should have no IP, and this can be validated by issuing the following command: # show /SP/network While the OAK deployment can IP the ILOM, it is always faster to configure the ILOM IP’s using the following method. This will allow for a faster deployment by using the ILOM to deploy the ODA. You will need the following information to be able to IP the ILOM for both ODA servers. 1. Public IP Address 2. Netmask 3. Gateway Address The following command will assign a static IP address to the ILOM # set /SP/network pendingipdiscovery=static pendingipaddress=1.2.2.65 pendingipnetmask=255.255.255.0 pendingipgateway=1.2.2.254 commitpending=true After the IP is established you can also use this session to power on each of the database nodes remotely. # start /SYS 7 108
  • 8. Engineered Systems Once this is executed you should be able to access the ILOM via ssh or an internet browser, as well as connect to the database node remotely. The following screen shows browser access to an ILOM. This is the ILOM console and this confirms the successful configuration of the ILOM. Once the ILOM is successfully configured you are now ready to configure the Oracle Database appliance database nodes. The ILOM provides a lot of functionality including fault management and remote access. The functions of the ILOM are beyond the scope of this document but below you can see a picture of some of the ILOM’s capabilities. We will be using the ILOM in the next section to remote control into the Database Node to configure the ODA. DEPLOYMENT OF THE DATABASE NODE The setup of the ILOM is the first step in deploying an ODA and configuring the two database nodes. As part of the deployment process you will need to create the deployment configuration files or enter the configuration information during the deployment process. To validate your configuration, it is often wise to use the offline configurator that is available via the Oracle Technology Network (OTN) website http://bit.ly/VpakT8. The Offline Configurator allows an administrator to enter, and validate, the complete set of configuration information prior to starting the deployment. The offline configurator which is meant currently for Bare Metal installations allows for complete network validation prior to deployment. 8 108
  • 9. Engineered Systems The first step in the deployment is to access each of the database nodes through the ILOM, using SSH as follows: The ILOM can also be used to access the database node by launching the remote console through a standard web browser. The host first needs to be powered on either remotely using the ILOM or through the power button on the front panel on the Oracle Database Appliance boxes. Once you launch the remote console. you will be taken to the host. After logging into the host, you will need to IP each of the two database hosts by running the following command using the root account: # /opt/oracle/oak/bin/oakcli configure firstnet This will allow for public access to the appliance and allow for completion of the deployment. Discussion about patching will be in a separate section but it is important to ensure that the version of OAK (Oracle Application Kit) you are deploying is not very far behind. 9 108
  • 10. Engineered Systems #/opt/oracle/oak/bin/oakcli show version If you feel that the software needs patching, please do so prior to the deployment as there are some factors like disk configuration that will change based on the version of the software . These include, but are not limited to, disk redundancy (double vs. triple mirroring) and the introduction of virtualization to the Oracle Database Appliance. The discussion about redeploying an ODA is a 3 part blog post series that can be found at http://bit.ly/VpcWQH . Once you have determined and patched your ODA to the level you want to start out with, the deployment process can start. In the earlier versions of the OAK software, the only option seemed to be using the GUI interface to deploy the software. In later versions (2.3 and above), the copy command has been introduced which allows for a scripted approach to deployment. For a safer GUI deployment it is recommended to use the vnc utility in order to avoid any issues. As root you can execute # vncserver :1  will allow for vncserver to run on port 5901. You will be prompted to setup a password. After which you will gain access to the server via vnc. Below is what a vnc screen will look like on a vnc connect . You can use publically available free vnc viewers like Tight VNC available at If you have created a deployment configuration file you can copy it to the first node of the host. You can deploy an ODA by entering the configuration information manually, but it is a cumbersome process. The screens below will show a typical deployment using a configuration file prebuilt via the offline configurator For complete control we will select custom and load the file. 10 108
  • 11. Engineered Systems You will provide generic network info in the next page or the information will be prefilled form a config file that is previously created. 11 108
  • 12. Engineered Systems The Public network information including which interface to use is the next set of information. You can either use the 10G network (xbond0) or the 1G network (bond0) depending on your network configuration. It is very important that you consult your network team regarding your configuration and ensure cables are plugged into the right ports. If you have any other networks to configure like a Backup network/NAS network etc . you will get the oppurtunity to configure it in the next screen. The next screen will walk you through the database configuration. Typically you should select a very small database to get a dummy database in place and create the actual databases that are more customized at a later time. 12 108
  • 13. Engineered Systems The Next screen walks you thru the ASR setup process. It is essential to get an inderstanding ASR (Automatic Service Request) functionality and the limitations of setting up the ASR utility via the deployment process. If you have direct access to the internet from the ODA, the ASR deployment should be fairly simple. If you are accessing the internet via a proxy it is important to note that the ODA ASR registration only supports port 80 for the proxy setup and does not support using another port using the deployment GUI. Your ASR deployment might fail if you your proxy uses a different port, due to this at which point you would have to manually setup the ASR.. Later versions of the ODA software (2.5) fixes the proxy port issue. The final decision is to the setup the ACFS file system - or as Oracle calls it: CloudFS. ACFS is a shared file system that can be used for storing a variety of files. Currently ACFS is a restricted license option. Please consult your licensing docs to get more information. In conclusion, you will receive a summary of the choices made and you can being the install. The install can take anywhere from 45 mins to an hour after which you will have a fully functional RAC cluster. This concludes the deployment of the Oracle Database Appliance 13 108
  • 14. Engineered Systems PATCHING THE ORACLE DATABASE APPLIANCE The Oracle Database appliance (ODA) is unique in the sense that Oracle provides an all-inclusive patching strategy. An ODA patch includes the following components Infrastructure 1. Integrated Lights Out Management (ILOM)/ BIOS 2. Oracle Appliance Kit (OAK) 3. Hardware Management Pack HMP 4. Automatic Service Request (ASR) 5. Intelligent Platform Management Interface IPMI 6. Operating SSystem 7. Storage Firmware Grid Infrastructure The GI patch allows for patching and/or upgrading the Clusterware and ASM software which are now collectively known as the grid infrastructure. Database The integrated patching allows for patching and/or upgrading the database software. Oracle currently supports 11.2.0.2.x as well as 11.2.0.3.x on the Database appliance. Oracle =can also introduce new features in each Oracle Appliance Kit patch. Below is the history of some of the features that were introduced during each patchset. HISTORY OF OAK/ODA PATCHES ODA 2.1.0.1/2.1.0.2 1. Base Release 2. 11.2.0.2.4 ODA 2.1.0.3/2.1.0.3.1 1. 11.2.0.2.5 2. Added ASR ILOM configuration 3. ILOM/BIOS and Firmware fixes 4. 2.1.0.3.1 patches a critical disk flaw ODA 2.2.0 1. Introduced 11.2.0.3.2 and provided the means to upgrade from 11.2.0.2 2. Upgraded the kernel to UEK 5.8 3. Odachk introduced perform ODA health checks 14 108
  • 15. Engineered Systems ODA 2.3.0 1. Prerequisite of ODA 2.2 2. Multiple Database Homes now supported 3. 11.2.0.2.7 and 11.2.0.3.3 supported in a multi home environment 4. Oakcli command set enhanced to create databases and database homes ODA 2.4.0 1. Normal redundancy option for new deployment of DATA and RECO(existing deployments require redeployment) 2. Trace File analyzer 3. Grid Infrastructure and Database patching is rolling upgradable. ODA 2.5.0 1. Virtualization as an option. 2. External ASR option 3. ASR Proxy port change PATCHING Patching the ODA is performed using the Oracle Appliance Kit (OAK) interface. Each version comes with a comprehensive readme that describes patching behavior. For the purposes of this document we will talk about applying a standard patch. As just described, an ODA patch will patch the OS as well as database and clusterware components. Depending on the criticality of the application, as of ODA 2.4 OS patching still requires downtime. However the GI and Database patching are both rolling upgradable. It depends on your patching strategy and you might want to take a smaller downtime to patch your OS and follow it up with the rolling patching for GI and Database components. Since the ODA patch includes a lot of components it is important to understand each component and the impact it will have on your system. The patch readme as well as MOS Note 888888.1 does a good job of explaining all the changes in the patch. It is important that the person applying the patch assess the impact and apply accordingly. Before starting a patching session it is always important to check the version of the patch deployed. #/opt/oracle/oak/bin/oakcli show version The version will help determine which patchsets are necessary. If you are on a pre ODA 2.2 patch set, it is important to first patch up to 2.2 before moving further. This is due to significant OS changes that were made in ODA patch 2.2. Once the patch determination has been made it is important to understand your network components to understand how the patching works. As an example, ILOM and BIOS patching will take significantly longer if your ILOM is on a different network than the database node public network. Typically this is due to network zoning/segmentation. If you have such a case it is always better to patch the ILOM outside of the ODA patch for speed. To do so you can follow the procedures outlined below. Please note below we are assuming a system with ODA 2.3 installed patching to 2.4. The first 3 steps can be done ahead of time and require no downtime is required 15 108
  • 16. Engineered Systems # ./oakcli unpack -package <absolute_path_to_patch_file p14752755_24000_Linux-x86-64.zip> (On Both Nodes) # cd /opt/oracle/oak/pkgrepos/thirdpartypkgs/Firmware/Ilom/SUN/541-4116-03/3.0.16.22.a-es- r75629/Base (On Both Nodes) # ipmiflash -v write ILOM-3_0_16_22_a_r75629-Sun_Fire_X4370_M2-rom.pkg force script config delaybios warning=0 (On Both Nodes) # ipmitool chassis power cycle (Will bring down the node and does require downtime. Needs to be run on The infrastructure patch a delay) each node but with will recognize that the ILOM/BIOS has already been patched and will ignore them in the patching cycle.nodes come up. You can patch the infra and it will be significantly faster Once the # /opt/oracle/oak/bin/oakcli update -patch 2.4.0.0.0 –infra The downtime related tasks end here if the purpose is patching only the server and not upgrade the grid and database software. The next step is patching the Grid Infrastructure and the patch is rolling upgradable which means a full downtime is not required. # cd /opt/oracle/oak/bin # ./oakcli update -patch 2.4.0.0.0 –gi This will patch your grid infrastructure components to 11.2.0.3.4. The final piece in the patching puzzle is the database and that is rolling upgradable as well. As part of the command you will be asked which Oracle homes to patch. # cd /opt/oracle/oak/bin # ./oakcli update -patch 2.4.0.0.0 –database This concludes the patching exercise on the ODA. As documented patching and maintaining and ODA is much simpler than managing and patching a similar set of components separately . ONE BUTTON ORACLE DATABASE APPLIANCE Oracle has used the concept of one button for a lot of activities surrounding the Oracle Database Appliance. The Oracle Appliance Kit (OAK) is the management brains of the Oracle Database Appliance. The Oracle Appliance Kit Daemon (oakd) maintains the link between hardware and software in the ODA. ONE BUTTON DEPLOY We’ve already discussed deployment but this is one of the key features of the Oracle Database Appliance (ODA). One button deployment allows for rapidly deploying or re-deploying the ODA with one command. # cd /opt/oracle/oak/bin # ./oakcli deploy 16 108
  • 17. Engineered Systems ONE BUTTON VALIDATE Oracle Database appliance comes with one button validation which should be performed to ensure that the system is performing optimally prior to going live. It is very important to understand the various options that the validation tool offers from the documentation. # cd /opt/oracle/oak/bin # ./oakcli validate -h The table of options below tells you about the various validation commands you can run. It is essential to execute basic validation on a regular basis to ensure the system is stable. Oracle Database Appliance Validation Tool Options Option Purpose Run all system checks, including Disk Calibration. Oracle recommends you use this command to validate -a system readiness before deployment. Do not run oakcli validate with this option on a loaded system, because the Disk Calibration system check can cause performance issues. -c Run a comma-delimited list of checks. Use this command to run a specific check or list of checks. Run the default system checks. The default system checks are SystemComponents, OSDiskStorage, -d SharedStorage, and NetworkComponents Create an output file. Provide the name of the file after the -f flag. The output for the validation commands -f is printed to this file. If you do not specify that output is sent to a file, then it is sent to the Page display (stdout). -h Print help information. -v Provide verbose output. -V Print the validation tool version. An example of a system problem can be shown here. RESULT: Powersupply PS0 V_IN=Disabled I_IN=Disabled V_OUT=0.88 Volts I_OUT=2 Amps IN_POWER=Disabled OUT_POWER=Disabled – as you can see I_IN and V_IN is disabled which mean there is an issue RESULT: Powersupply PS1 V_IN=206 Volts I_IN=3.75 Amps V_OUT=12.08 Volts I_OUT=4.40 Amps IN_POWER=720 Watts OUT_POWER=670 Watts A system without a power supply issue would display something like this: RESULT: Powersupply PS0 V_IN=204 Volts I_IN=1.88 Amps V_OUT=12.08 Volts I_OUT=27.20 Amps IN_POWER=380 Watts OUT_POWER=330 Watts -- RESULT: Powersupply PS1 V_IN=202 Volts I_IN=1.88 Amps V_OUT=12.08 Volts I_OUT=27.20 Amps108 17 IN_POWER=350 Watts OUT_POWER=340 Watts SUCCESS: Both the powersupply are ok and functioning
  • 18. Engineered Systems As you can see, even though the system is functioning due to the redundant nature of the system components. If there is an issue, then running an oakcli validate can identify the source of the system or hardware problem. These types of issues can be proactively determined by using ASR (Automatic Service Request). Validate also provides DiskCalibration data to ensure that the disk is functioning as expected and no disk failures. RESULT: Random read throughput across all 20 SASDisk = 4037 IOPS RESULT: Random read throughput across all 4 SSDDisk = 16368 IOPS RESULT: Random read throughput across all 20 SASDisk = 1844 MBPS RESULT: Random read throughput across all 4 SSDDisk = 941 MBPS An issue with disk access or calibration can also be seen below. In this example, a disk was manually dropped from the diskgroup to simulate this test. ERROR: Error when running orion for HDD_E0_S01_705927132_MBPS : ORION: ORacle IO Numbers -- Version 11.2.0.3.0 Calibration will take approximately 1 minutes. Using a large value for -cache_size may take longer. Error completing IO (storax_aiowait) ORA-27061: waiting for async I/Os failed Linux-x86_64 Error: 5: Input/output error Additional information: -1 A Network Check will look at the network components to validate the network speed and Ethernet ports that are currently active. Oracle Database Appliance has a one 10G interface per node as well several 1G interfaces. In the example only 1G interfaces are active. INFO: Doing oak network checks RESULT: Detected active link for interface eth0 with link speed 1000Mb/s RESULT: Detected active link for interface eth1 with link speed 1000Mb/s RESULT: Detected active link for interface eth2 with link speed 1000Mb/s RESULT: Detected active link for interface eth3 with link speed 1000Mb/s RESULT: Detected active link for interface eth4 with link speed 1000Mb/s WARNING: No Link detected for interface eth5 WARNING: No Link detected for interface eth6 WARNING: No Link detected for interface eth7 WARNING: No Link detected for interface eth8 WARNING: No Link detected for interface eth 99 The oakcli validate one button deployment provides a plethora of diagnostic information that provide insight into the operation of the Oracle Database Appliance. It should be used regularly, but with caution, to ensure the machine is functioning as intended. 18 108
  • 19. Engineered Systems ONE BUTTON STORAGE DIAGNOSTICS Disks can fail, so it is important to ensure that storage diagnostics are key as part of the one button deployment of the ODA. Storage diagnostics are a key component in the oakcli command set. From the ODA documentation, below are some of the commands that are available for storage diagnostic and disk location. Option Purpose Locates the disk associated with the ASM disk that you indicate by its ASM disk name, where: -h outputs the oakcli locate options oakcli locate[-h] [disk diskname on|off] disk diskname locates the ASM disk whose name you designate on turns on the LED of the specified disk off turns off the LED of the specified disk oakcli show controller Displays the configuration for active cores as defined by oakcli show core_config_key the registered core key oakcli show disk Displays disks oakcli show diskgroup Displays disk groups oakcli show expander Displays expanders oakcli show –h Show help command oakcli show storage Displays storage device status Provides a detailed output patch versions for software and firmware on the appliance It is very important to understand and execute some of the above commands to ensure the stability and understand the state of the disk. #oakcli show disk NAME PATH TYPE STATE STATE_DETAILS pd_00 /dev/sdam HDD ONLINE Good pd_01 /dev/sdaw HDD ONLINE Good pd_02 /dev/sdaa HDD ONLINE Good pd_03 /dev/sdak HDD ONLINE Good Based on the output above, if there is a bad disk, you can work with Oracle Field Services or your internal hardware resources to replace the disk. The oakcli locate disk helps in ensuring the right bad disk is located and replaced. Also, the show storage option provides a complete picture of the storage which includes the controllers, expanders as well as firmware version of the disks. 19 108
  • 20. Engineered Systems ONE BUTTON ODA HEALTHCHECKS This piece is new since ODA 2.3.x and is not part of the OAK framework. However it is important to note and use as needed. Oracle Database Appliance Checks, or ODAchk as the tool is known, is the Oracle Database Appliance version of the popular RACchk and EXAchk tools. Most of the OAK built in tools look at hardware and software, but the ODAchk provides more in-depth looks the system from the database perspective and not the appliance perspective. The tool ensures things like RAC, Grid, ASM as well as common database configuration parameters are set up right. ODAchk should be run on a regular basis and is recommended before and after any system change as well as any system patching. It is surprising how sometimes on a running system how many best practices are not being followed. $/opt/oracle/oak/odachk/odachk – Please run as oracle software owner. FAIL => Database parameter DB_LOST_WRITE_PROTECT is NOT set to recommended value on tst12 instance FAIL => Database parameter GLOBAL_NAMES is NOT set to recommended value on tst12 instance FAIL => One or more errors or warnings for OS Disk Storage checks WARNING => One or file generated network and bonding interface checks There is an html more warnings for with detail information as well as best practices to follow. ONE BUTTON MANAGEMENT DIAGNOSTIC COLLECTION The OAK framework comes with its own management and diagnostic collection framework including a cleanup script to re- initialize the Oracle Database Appliance and establish “pay as you go” Oracle licensing core counts. Oakcli manage diagcollect – Collect and package diagnostic files for Oracle Support # cd /opt/oracle/oak/bin # ./oakcli manage diagcollect – this will collect diagnostic data from CRS and various DB and OAK components to submit to oracle Oakcli manage cleanrepo – Remove old versions of ODA patch and version files #cd/opt/oracle/oak/bin # oakcli manage cleanrepo --ver 2.4.0.0.0 Can not delete the current or latest version repository. #oakcli manage cleanrepo --ver 2.3.0.0.0 – to cleanup previous version files # oakcli update –patch 2.4.0.0.0 –clean – in 2.4 has been enhanced to cleanup temporary files after patching CleanupDeploy.pl If there is ever a need to reimage the box or uninstall all software you can use the CleanupDeploy.pl to do so 1. Log in as root. 2. Change directory to /opt/oracle/oak/onecmd. 3. Run the command cleanupDeploy.pl. For example: # ./cleanupDeploy.pl 20 108
  • 21. Engineered Systems Configuring the Oracle licensing core counts If you are using the pay as go Oracle Licensing scheme you can use oakcli to apply the core cap and install the key you have generated on MOS (My Oracle Support). # oakcli apply –h Usage:oakcli apply core_config_key <key_file> # oakcli show core_config_key Optional core_config_key is not applied on this machine yet ! # oakcli show core_config_key Host's serialnumber = ABC123 Configured Cores = 20 Trace File Analysis In Oracle Database Appliance 2.4 a new option for TFA was installed. The Trace File Analyzer is a new framework for the collection and analysis of trace files that help the proactive collection of errors in real-time mode and automatically collects the right log files as the problem is happening. More information on TFA can be found in MOS note 1513912.1. ONE BUTTON PATCHING This paper already discusses patching but in the one button section it is important to mention the capabilities of the one button infrastructure to be able to patch all software or individual components using the one button patching capabilities built into the Oracle Appliance Kit. # cd /opt/oracle/oak/bin # ./oakcli update -patch version While the patching section of this paper discusses more detail, the command above is the but in essence of single button patching, The command above can patch an appliance. ONE BUTTON DATABASE DEPLOYMENT Starting with ODA version 2.3 and above, newer releases of the ODA 2.3 and above , the option is now provided to create multiple database homes as well as create new databases. There are options to upgrade database as well delete databases and homes as well. To create a new database home # oakcli create dbhome -version 11.2.0.3.3 – will create a new dbhome with 11.2.0.3.3 21 108
  • 22. Engineered Systems To create a new database # oakcli create database -db testdb -version 11.2.0.3.3 – will create a database and a new database home #oakcli create database -db testdb –oh OraDb11203_home2 – will create a new database in an existing dbhome. To delete a database or database home # oakcli delete database –db olddb # oakcli delete dbhome –oh dbhome_name To. upgrade a database from one version to another (11.2.0.2 to 11.2.0.3) #oakcli upgrade database {-db <db_names> | -from <source_home>} -to <destination_home>. SUMMARY In summary, the Oracle Database Appliance packs a lot of bang for the buck. As the basic model in Oracle engineered Appliance product line, it is the stepping stone to understand and evaluate if an appliance model is a viable option for a company. An Oracle Database Appliance provides ease of setup, patching as well as diagnostics. As an administrator that has previously only had a specific focus, there are a lot of things that an administrator has to be aware of, including the network, hardware and operating system. Oracle has made it easy for people to manage the appliance as one unit by making various aspects easier to manage and maintain using the Oracle Appliance Kit (OAK) framework. There are various monitoring aspects built into the box as well as one button activities to ensure that management of the appliance is simple. REFERENCES Oracle Database Appliance - 2.X Supported Versions & Known Issues [ID 888888.1] Information Center: Oracle Database Appliance [ID 1417713.2] ODA (Oracle Database Appliance): New Deploy Step-by-Step Short Guide (version 2.3) [ID 1486534.1] ODA (Oracle Database Appliance): oakcli command reference [ID 1417879.1] ODA (Oracle Database Appliance): How To Configure xbond,bond1,bond2 post deploy [ID 1422563.1] ODA (Oracle Database Appliance): The Steps to replace failing disks [ID 1496114.1] Gathering data for the Oracle Database Appliance server [ID 1358795.1] ODA (Oracle Database Appliance): Test Plan Outline [ID 1474273.1] Oracle Database Appliance Setup Poster TFA Collector- The Preferred Tool for Automatic or ADHOC Diagnostic Gathering Across All Cluster Nodes [ID 1513912.1] 22 108