Engineered Systems     Oracle database appliance v1 - rac in a box – some strings attached                                ...
Engineered Systems   Memory                                 96 GB RAM (12 x DDR3-1333 8GB DIMMs)                          ...
Engineered SystemsCallout    Label        Ethernet        Bond                                Description  1              ...
Engineered SystemsAs you can see from above, it, is very important to understand the various components. The complete deta...
Engineered Systems     ILOM IP Address:     ILOM Passwords: default is changeme                      Leave the default for...
Engineered SystemsAs we look at the appliance we see 1 host node and 1 Service Processor node per hardware entity. The ser...
Engineered SystemsOnce the connection is made you will see a login prompt. Once the ODA is cabled, in and power cords are ...
Engineered SystemsOnce this is executed you should be able to access the ILOM via ssh or an internet browser, as well as c...
Engineered SystemsThe first step in the deployment is to access each of the database nodes through the ILOM, using SSH as ...
Engineered Systems        #/opt/oracle/oak/bin/oakcli show versionIf you feel that the software needs patching, please do ...
Engineered SystemsYou will provide generic network info in the next page or the information will be prefilled form a confi...
Engineered SystemsThe Public network information including which interface to use is the next set of information. You can ...
Engineered SystemsThe Next screen walks you thru the ASR setup process. It is essential to get an inderstanding ASR (Autom...
Engineered SystemsPATCHING THE ORACLE DATABASE APPLIANCEThe Oracle Database appliance (ODA) is unique in the sense that Or...
Engineered SystemsODA 2.3.0      1.    Prerequisite of ODA 2.2      2.    Multiple Database Homes now supported      3.   ...
Engineered Systems     # ./oakcli unpack -package <absolute_path_to_patch_file p14752755_24000_Linux-x86-64.zip> (On     B...
Engineered SystemsONE BUTTON VALIDATEOracle Database appliance comes with one button validation which should be performed ...
Engineered SystemsAs you can see, even though the system is functioning due to the redundant nature of the system componen...
Engineered SystemsONE BUTTON STORAGE DIAGNOSTICSDisks can fail, so it is important to ensure that storage diagnostics are ...
Engineered SystemsONE BUTTON ODA HEALTHCHECKSThis piece is new since ODA 2.3.x and is not part of the OAK framework. Howev...
Engineered SystemsConfiguring the Oracle licensing core countsIf you are using the pay as go Oracle Licensing scheme you c...
Engineered SystemsTo create a new database    # oakcli create database -db testdb -version 11.2.0.3.3 – will create a data...
Upcoming SlideShare
Loading in …5
×

Oracle Database Appliance - RAC in a box Some strings attached

5,858 views

Published on

Published in: Technology, News & Politics
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
5,858
On SlideShare
0
From Embeds
0
Number of Embeds
1,485
Actions
Shares
0
Downloads
108
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Oracle Database Appliance - RAC in a box Some strings attached

  1. 1. Engineered Systems Oracle database appliance v1 - rac in a box – some strings attached Fuad Arshad, Walgreens CompanyABSTRACTThe Oracle Database Appliance is an engineered system that is targeted towards small and medium businesses, but also hasall of the bells and whistles required for enterprise deployments. This paper will introduce the Oracle Database Appliance(ODA), and review the deployment of the ODA from a non sysadmin perspective. This paper includes the deployment andpatching processes and the best practices built into the Oracle Database Appliance. The paper will review the DBAs role inmanaging the ODATARGET AUDIENCEThis paper is intended mostly for Database Architects and people that deploy Oracle database technologies. This paper willreview various techniques for deploying the Oracle Database Appliance (ODA) successfully as well as how to perform day today activities. The various components all tied together provide an effective use case for companies that want to invest inEngineered Systems, and provide an entry point to understand the management and maintenance considerations for deployingan engineered system.EXECUTIVE SUMMARYAttendees and readers of this paper will learn the following after reading this paper.Oracle Database Appliance - OverviewOracle Database Appliance - PredeploymentOracle Database Appliance - DeploymentOracle Database Appliance - PatchingOracle Database Appliance - One Button Database ApplianceINTRODUCTIONThe Oracle Database Appliance (ODA) is an entry level engineered system that provides easy access to preconfigured RealApplication Cluster (RAC) database hardware and software. The ODA is accessible to customers at an entry level cost withsoftware licensing available starting from 2 cores to 24 cores. The ODA is intended to provide companies with an easy todeploy model that includes all of the hardware and software required to run a complete system into one 4u rack mountableunit.INSIDE THE ORACLE DATABASE APPLIANCEThe Oracle Database Appliance (ODA) is a 4u rack mountable unit which consists of two 2u Sun Fire X4370 M2 unitsknown as Server units (SN). Oracle has provided MOS Note 1385831.1 which is kept fairly up-to-date with the ODAcomponents and software versions. Below is a snapshot as of Dec 14, 2012 and this paper only focuses on V1 functionality. Sun X4370M2 System Server Components (2 Server Nodes per X4370M2) CPU 2x 6-core Intel Xeon X5675 3.07GHz 1 108
  2. 2. Engineered Systems Memory 96 GB RAM (12 x DDR3-1333 8GB DIMMs) 2x 10GbE (SFP+) PCIe card Network 6x 1GbE PCIe card 2x 1GbE (Intel 82571) on-board Integrated redundant cluster interconnect 2x 500GB SATA - for operating systemInternal Storage 1x 4GB USB internal RAID 2x LSI SAS9211-8i SAS HBA Controller 20x 600GB - 3.5" SAS 15k RPM HDD (Seagate Cheetah) - for RDBMS DATA (any slot except the top row of disks)Shared Storage 4x 73GB - 3.5" SAS2 SSDs - for RDBMS REDO (slot in the top row of 4 disks) SSD from STEC (ZeusIOPS - multi-level cell (MLC) version with SAS interface) Operating Oracle Enterprise Linux 5.5 (on ODA 2.1) 5.8 (on ODA 2.2) x86_64 System The two ODA Server Nodes are packaged in a 4u chassis that includes 2 server nodes as well as direct attacheddisk in the front. Below are front and back images of an ODA. looks like from the front and backThe bottom server is Node 0 and the top is Node 1.These will be referred to as oak1 and oak2 in the factorydefault configuration. It is very important to keep this picture handy to work with the data center installationteam to ensure they understand how each port is used to cable the ODA to meet your requirements. Cablingdepends on the data center security zone specs that your infrastructure defines. There might be sensitive and non-sensitive zones as well as various cabling zones. These requirements need to be considered as part of theinstallation process. Attached in the Appendix is a sample spreadsheet that outlines the cabling specs needed tocomplete an ODA Installation. 2 108
  3. 3. Engineered SystemsCallout Label Ethernet Bond Description 1 Power connectors Eth 4 and Eth 5 are configured as bond1. Eth 6 and Eth 7 are Eth 7, 6, 5, 4 bond1, configured as bond 2. These ports are used for custom 2 PCIe 1 (left to right) bond2 configurations or for separate backup, disaster recovery, and network management. Two 10-GbE ports. In 10-GbE systems, these are connected to the 3 PCIe 0 Eth 8, Eth 9 xbond0 public network. 4 SerMgt Serial connector to Oracle ILOM and system console. Net 0,Net Two 1-GbE connectors. In 1-GbE systems, these are connected to 5 Eth2, Eth3 bond0 1 the public network. 6 NetMgt Ethernet connection for Oracle ILOM USB and 7 Used for connecting to system console. Video 8 Server Node 0 9 Server Node 1 3 108
  4. 4. Engineered SystemsAs you can see from above, it, is very important to understand the various components. The complete details are available inthe MOS note referenced above, as well as the ODA Install, Configuration and User’s Guide. This paper will only touch thebasic pieces that are important to create the fundamental understanding about the Oracle Database Appliance. Additionalreferences are available at the end of this paper.PREDEPLOYMENT OF THE ORACLE DATABASE APPLIANCEThe Oracle Database Appliance requires a lot of pre-deployment tasks to be completed to prepare for installing the ODA.These pre-deployment steps are standard tasks for deploying a server in a data-center and we are going to walk thru themfrom a Database Administrator perspective. The Oracle Database Appliance Owners guide covers most of these details.In order to get the environment ready, the site needs to meet the environmental & electrical guidelines that are explained inthe Owner’s manual. The ODA weighs about 96 lbs., and utmost care must be taken while handling the hardware. Four-postracks are compatible. Two-post racks are not compatible. Horizontal and vertical pitch must conform to ANSI/EIA 310-D-1992 or IEC 60927 standards. An Oracle Database Appliance requires 4 rack units (4RU) of rack space. A rack mounting kitis shipped as part of the hardware.Once the hardware is racked and stacked in the data-center floor, the appliance needs to cabled. If 10g network connectionswill be used, SFA Adapters will need to be included in the ODA order for attaching the 10g network cables to the appliance.There is a minimum requirement of 2 power connections and 4 network connections to the public network in order to accessthe machine. There are other networks available for use for various purposes (connecting to a NAS, ILOM or BackupNetwork). Once all cabling is confirmed, it is very important to create a pre-deployment checklist to ensure that allappropriate information about the system is captured.An example template can be seen below. Needed information Your Information System Name: i.e. sales TSODA1 Database Deployment Type: RAC, RAC One or RAC Standalone Internal or External Backup via NFS device External Root Passwords: default for nodes is welcome1 Leave the default for now Domain Name: Example.com.com DNS Server(s): min 1 Dns.example.com NTP Server(s): optional but recommended Ntp.example.com Node 1 Name: i.e. sales1 Tsoda11 Node 1 Public IP Address Node 1 VIP Name: i.e. sales1-vip Tsoda11-vip Node 1 Public Virtual IP Address Node 2 Name: i.e. sales2 Tsoda12 Node 2 Public IP Address Node 2 VIP Name: i.e. sales2-vip Tsoda12-vip Node 2 Public Virtual IP Address SCAN Name: sales-scan Tsoda1-scan SCAN IP Addresses: min 2 Netmask: Gateway IP: ILOM Names: ie. sales1-ilom Tsoda11-ilom ILOM IP Address: ILOM Names: i.e. sales2-ilom Tsoda12-ilom 4 108
  5. 5. Engineered Systems ILOM IP Address: ILOM Passwords: default is changeme Leave the default for now ILOM Gateway: ILOM Netmask: Database Name: Tsodadb Database class: very small, small, medium, large, very Small large (optional) Proxy Server Name: for Auto Service Request Metalink Login Information: User ID/PwdSince each Oracle Database Appliance comes with the same configuration, it is very easy to provide a configurationspreadsheet that includes various configuration specs and networking information to record the right information from thenetwork and data center teams . An example spreadsheet that can be used to streamline the process is attached below. Microsoft Excel WorksheetIt is very important to develop a standard process for deploying the ODA. This includes defining roles and responsibilities fordeploying and managing the ODA. It is very important to understand these organizational responsibilities prior todeployment. A lot of activities that are required to deploy the ODA require system access and organizational structurestypically have role separation. The ODA deployment activity will need the assistance of all teams within an organization frominfrastructure to system administration if the role separation methodology is used. Conversely the Database Administratordeploying the ODA needs to understand various additional processes in order to successfully deploy the appliance.In the Scenario, if the Database team is responsible for deploying the Oracle Database Appliance, the following is a briefexample list of activities that will need to be performed. 1. Determine hardware placement zones. 2. Submit a request to get network ip’s 3. Submit DNS and Firewall requests 4. Work with datacenter team to determine rack space. 5. Ensure cables and adapters are available 6. Read thru the ODA manual 7. Provide datacenter team cabling specs 8. Get the ODA connected to a KVM or access via Serial cable to ensure the hosts are powered on and ILOM is accessible. 9. Walk thru deployment activities.DEPLOYMENT OF THE ORACLE DATABASE APPLIANCEIn order to understand how to deploy an Oracle Database Appliance (ODA) in a datacenter, one needs to understand theappliance itself. As illustrated above, The ODA consists of 2 x86 servers that have direct attached SAS as well as SSDstorage. The environment is fully redundant from a hardware perspective. Storage redundancy is provided by Oracle ASM andcan be triple mirrored or double mirrored (version 2.4 and above) for the SAS disk. The SSD is designed and recommendedto be only used for the on-line redo logs and is triple mirrored regardless of ODA version. Oracle ASM also ensures that allthe extents are mapped appropriately among the available disk to ensure that data extents are balanced. Oracle RAC is used toensure any hardware failures on one node will not impact the running of the system as a whole. 5 108
  6. 6. Engineered SystemsAs we look at the appliance we see 1 host node and 1 Service Processor node per hardware entity. The service processor alsoincludes an Integrated Lights Out Management (ILOM) console which is used to manage each appliance. Deploying theODA starts with getting the ILOM ready and accessible to the network.DEPLOYMENT OF THE ILOMThe ILOM, or the Integrated Lights Out Manager, is the hardware administration and management brains and is also referredto as the service processor. The ILOM is used for various purposes in the lifecycle of a host including, but not limited to,performing a bare metal restore of an ODA. An Oracle Database Appliance contains 2 ILOM’s, or one for each of the twoODA servers. Configuring the ILOM is step 1 in the process of getting an ODA up and running. There are multiple ways toget into the ILOM. If you have a KVM device you can connect the KVM to the ODA serial management port to do theILOM configuration remotely. The configuration steps for setting up the ILOM directly via the serial port are detailed below.ACCESSING THE ILOM VIA SERIAL PORTThe ODA ILOM can be accessed via a Serial management (SerMgt) port labeled #4 in the picture above. MOS Note ID1395445.1 explains the process of connecting to the ODA to configure the ILOM port.The process requires direct physical access to the serial port , which means, this process will need to be run at the place wherethe Oracle Database Appliance is installed. Each ODA ships with an RJ45-serial convertor like the one pictured below.Using an RJ-45 cable, you can connect the cable to the serial port on your laptop or Datacenter terminal cart and get into theILOM using putty. In the case that you do not have access to a serial port, Amazon sells a usb to serial converter cable athttp://amzn.com/B00425S1H8 . The cable looks like below and requires prolific drivers installed on your laptop in order touse the cable. This cable will be used in conjunction to the rj45-serial converter to connect to the ODA ILOM.Once the connection cable is connected to the laptop and ODA serial port, check the device manager to see which com portthe USB device is connected to and specify the com port in your Putty connection. This doc will only talk about windows butthe MOS note above includes the Linux configuration as well. Once the port is determined, you can open up putty andconnect to the serial connection. 6 108
  7. 7. Engineered SystemsOnce the connection is made you will see a login prompt. Once the ODA is cabled, in and power cords are attached, theILOM is automatically powered on. You will need to use the root account and the default root password to connect to theILOM. .An initial configuration of the ILOM should have no IP, and this can be validated by issuing the following command: # show /SP/networkWhile the OAK deployment can IP the ILOM, it is always faster to configure the ILOM IP’s using the following method.This will allow for a faster deployment by using the ILOM to deploy the ODA. You will need the following information to beable to IP the ILOM for both ODA servers. 1. Public IP Address 2. Netmask 3. Gateway AddressThe following command will assign a static IP address to the ILOM# set /SP/network pendingipdiscovery=static pendingipaddress=1.2.2.65 pendingipnetmask=255.255.255.0pendingipgateway=1.2.2.254 commitpending=trueAfter the IP is established you can also use this session to power on each of the database nodes remotely.# start /SYS 7 108
  8. 8. Engineered SystemsOnce this is executed you should be able to access the ILOM via ssh or an internet browser, as well as connect to thedatabase node remotely. The following screen shows browser access to an ILOM.This is the ILOM console and this confirms the successful configuration of the ILOM. Once the ILOM is successfullyconfigured you are now ready to configure the Oracle Database appliance database nodes. The ILOM provides a lot offunctionality including fault management and remote access. The functions of the ILOM are beyond the scope of thisdocument but below you can see a picture of some of the ILOM’s capabilities. We will be using the ILOM in the next sectionto remote control into the Database Node to configure the ODA.DEPLOYMENT OF THE DATABASE NODEThe setup of the ILOM is the first step in deploying an ODA and configuring the two database nodes. As part of thedeployment process you will need to create the deployment configuration files or enter the configuration information duringthe deployment process. To validate your configuration, it is often wise to use the offline configurator that is available via theOracle Technology Network (OTN) website http://bit.ly/VpakT8. The Offline Configurator allows an administrator toenter, and validate, the complete set of configuration information prior to starting the deployment. The offline configuratorwhich is meant currently for Bare Metal installations allows for complete network validation prior to deployment. 8 108
  9. 9. Engineered SystemsThe first step in the deployment is to access each of the database nodes through the ILOM, using SSH as follows:The ILOM can also be used to access the database node by launching the remote console through a standard web browser.The host first needs to be powered on either remotely using the ILOM or through the power button on the front panel onthe Oracle Database Appliance boxes.Once you launch the remote console. you will be taken to the host.After logging into the host, you will need to IP each of the two database hosts by running the following command using theroot account: # /opt/oracle/oak/bin/oakcli configure firstnetThis will allow for public access to the appliance and allow for completion of the deployment. Discussion about patchingwill be in a separate section but it is important to ensure that the version of OAK (Oracle Application Kit) you aredeploying is not very far behind. 9 108
  10. 10. Engineered Systems #/opt/oracle/oak/bin/oakcli show versionIf you feel that the software needs patching, please do so prior to the deployment as there are some factors like diskconfiguration that will change based on the version of the software . These include, but are not limited to, diskredundancy (double vs. triple mirroring) and the introduction of virtualization to the Oracle Database Appliance. Thediscussion about redeploying an ODA is a 3 part blog post series that can be found at http://bit.ly/VpcWQH . Once youhave determined and patched your ODA to the level you want to start out with, the deployment process can start. In theearlier versions of the OAK software, the only option seemed to be using the GUI interface to deploy the software. Inlater versions (2.3 and above), the copy command has been introduced which allows for a scripted approach todeployment. For a safer GUI deployment it is recommended to use the vnc utility in order to avoid any issues.As root you can execute# vncserver :1  will allow for vncserver to run on port 5901.You will be prompted to setup a password. After which you will gain access to the server via vnc. Below is what a vncscreen will look like on a vnc connect . You can use publically available free vnc viewers like Tight VNC available atIf you have created a deployment configuration file you can copy it to the first node of the host. You can deploy an ODAby entering the configuration information manually, but it is a cumbersome process. The screens below will show atypical deployment using a configuration file prebuilt via the offline configuratorFor complete control we will select custom and load the file. 10 108
  11. 11. Engineered SystemsYou will provide generic network info in the next page or the information will be prefilled form a config file that is previouslycreated. 11 108
  12. 12. Engineered SystemsThe Public network information including which interface to use is the next set of information. You can either use the 10Gnetwork (xbond0) or the 1G network (bond0) depending on your network configuration. It is very important that youconsult your network team regarding your configuration and ensure cables are plugged into the right ports.If you have any other networks to configure like a Backup network/NAS network etc . you will get the oppurtunity toconfigure it in the next screen.The next screen will walk you through the database configuration. Typically you should select a very small database to get adummy database in place and create the actual databases that are more customized at a later time. 12 108
  13. 13. Engineered SystemsThe Next screen walks you thru the ASR setup process. It is essential to get an inderstanding ASR (Automatic ServiceRequest) functionality and the limitations of setting up the ASR utility via the deployment process. If you have direct access tothe internet from the ODA, the ASR deployment should be fairly simple. If you are accessing the internet via a proxy it isimportant to note that the ODA ASR registration only supports port 80 for the proxy setup and does not support usinganother port using the deployment GUI. Your ASR deployment might fail if you your proxy uses a different port, due to thisat which point you would have to manually setup the ASR.. Later versions of the ODA software (2.5) fixes the proxy portissue.The final decision is to the setup the ACFS file system - or as Oracle calls it: CloudFS. ACFS is a shared file system that canbe used for storing a variety of files. Currently ACFS is a restricted license option. Please consult your licensing docs to getmore information.In conclusion, you will receive a summary of the choices made and you can being the install. The install can take anywherefrom 45 mins to an hour after which you will have a fully functional RAC cluster. This concludes the deployment of theOracle Database Appliance 13 108
  14. 14. Engineered SystemsPATCHING THE ORACLE DATABASE APPLIANCEThe Oracle Database appliance (ODA) is unique in the sense that Oracle provides an all-inclusive patching strategy. AnODA patch includes the following componentsInfrastructure 1. Integrated Lights Out Management (ILOM)/ BIOS 2. Oracle Appliance Kit (OAK) 3. Hardware Management Pack HMP 4. Automatic Service Request (ASR) 5. Intelligent Platform Management Interface IPMI 6. Operating SSystem 7. Storage FirmwareGrid InfrastructureThe GI patch allows for patching and/or upgrading the Clusterware and ASM software which are now collectively known asthe grid infrastructure.DatabaseThe integrated patching allows for patching and/or upgrading the database software. Oracle currently supports 11.2.0.2.x aswell as 11.2.0.3.x on the Database appliance.Oracle =can also introduce new features in each Oracle Appliance Kit patch. Below is the history of some of the features thatwere introduced during each patchset.HISTORY OF OAK/ODA PATCHESODA 2.1.0.1/2.1.0.2 1. Base Release 2. 11.2.0.2.4ODA 2.1.0.3/2.1.0.3.1 1. 11.2.0.2.5 2. Added ASR ILOM configuration 3. ILOM/BIOS and Firmware fixes 4. 2.1.0.3.1 patches a critical disk flawODA 2.2.0 1. Introduced 11.2.0.3.2 and provided the means to upgrade from 11.2.0.2 2. Upgraded the kernel to UEK 5.8 3. Odachk introduced perform ODA health checks 14 108
  15. 15. Engineered SystemsODA 2.3.0 1. Prerequisite of ODA 2.2 2. Multiple Database Homes now supported 3. 11.2.0.2.7 and 11.2.0.3.3 supported in a multi home environment 4. Oakcli command set enhanced to create databases and database homesODA 2.4.0 1. Normal redundancy option for new deployment of DATA and RECO(existing deployments require redeployment) 2. Trace File analyzer 3. Grid Infrastructure and Database patching is rolling upgradable.ODA 2.5.0 1. Virtualization as an option. 2. External ASR option 3. ASR Proxy port changePATCHINGPatching the ODA is performed using the Oracle Appliance Kit (OAK) interface. Each version comes with acomprehensive readme that describes patching behavior. For the purposes of this document we will talk about applying astandard patch. As just described, an ODA patch will patch the OS as well as database and clusterware components.Depending on the criticality of the application, as of ODA 2.4 OS patching still requires downtime. However the GI andDatabase patching are both rolling upgradable. It depends on your patching strategy and you might want to take a smallerdowntime to patch your OS and follow it up with the rolling patching for GI and Database components.Since the ODA patch includes a lot of components it is important to understand each component and the impact it will haveon your system. The patch readme as well as MOS Note 888888.1 does a good job of explaining all the changes in the patch.It is important that the person applying the patch assess the impact and apply accordingly.Before starting a patching session it is always important to check the version of the patch deployed. #/opt/oracle/oak/bin/oakcli show versionThe version will help determine which patchsets are necessary. If you are on a pre ODA 2.2 patch set, it is important to firstpatch up to 2.2 before moving further. This is due to significant OS changes that were made in ODA patch 2.2.Once the patch determination has been made it is important to understand your network components to understand how thepatching works. As an example, ILOM and BIOS patching will take significantly longer if your ILOM is on a differentnetwork than the database node public network. Typically this is due to network zoning/segmentation.If you have such a case it is always better to patch the ILOM outside of the ODA patch for speed. To do so you can followthe procedures outlined below. Please note below we are assuming a system with ODA 2.3 installed patching to 2.4. The first3 steps can be done ahead of time and require no downtime is required 15 108
  16. 16. Engineered Systems # ./oakcli unpack -package <absolute_path_to_patch_file p14752755_24000_Linux-x86-64.zip> (On Both Nodes) # cd /opt/oracle/oak/pkgrepos/thirdpartypkgs/Firmware/Ilom/SUN/541-4116-03/3.0.16.22.a-es- r75629/Base (On Both Nodes) # ipmiflash -v write ILOM-3_0_16_22_a_r75629-Sun_Fire_X4370_M2-rom.pkg force script config delaybios warning=0 (On Both Nodes) # ipmitool chassis power cycle (Will bring down the node and does require downtime. Needs to be run onThe infrastructure patch a delay) each node but with will recognize that the ILOM/BIOS has already been patched and will ignore them in thepatching cycle.nodes come up. You can patch the infra and it will be significantly faster Once the # /opt/oracle/oak/bin/oakcli update -patch 2.4.0.0.0 –infraThe downtime related tasks end here if the purpose is patching only the server and not upgrade the grid and databasesoftware. The next step is patching the Grid Infrastructure and the patch is rolling upgradable which means a fulldowntime is not required. # cd /opt/oracle/oak/bin # ./oakcli update -patch 2.4.0.0.0 –giThis will patch your grid infrastructure components to 11.2.0.3.4.The final piece in the patching puzzle is the database and that is rolling upgradable as well. As part of the command you willbe asked which Oracle homes to patch. # cd /opt/oracle/oak/bin # ./oakcli update -patch 2.4.0.0.0 –databaseThis concludes the patching exercise on the ODA. As documented patching and maintaining and ODA is much simpler thanmanaging and patching a similar set of components separately.ONE BUTTON ORACLE DATABASE APPLIANCEOracle has used the concept of one button for a lot of activities surrounding the Oracle Database Appliance. The OracleAppliance Kit (OAK) is the management brains of the Oracle Database Appliance. The Oracle Appliance Kit Daemon(oakd) maintains the link between hardware and software in the ODA.ONE BUTTON DEPLOYWe’ve already discussed deployment but this is one of the key features of the Oracle Database Appliance (ODA). Onebutton deployment allows for rapidly deploying or re-deploying the ODA with one command. # cd /opt/oracle/oak/bin # ./oakcli deploy 16 108
  17. 17. Engineered SystemsONE BUTTON VALIDATEOracle Database appliance comes with one button validation which should be performed to ensure that the system isperforming optimally prior to going live. It is very important to understand the various options that the validation tool offersfrom the documentation. # cd /opt/oracle/oak/bin # ./oakcli validate -hThe table of options below tells you about the various validation commands you can run. It is essential toexecute basic validation on a regular basis to ensure the system is stable.Oracle Database Appliance Validation Tool OptionsOption Purpose Run all system checks, including Disk Calibration. Oracle recommends you use this command to validate-a system readiness before deployment. Do not run oakcli validate with this option on a loaded system, because the Disk Calibration system check can cause performance issues.-c Run a comma-delimited list of checks. Use this command to run a specific check or list of checks. Run the default system checks. The default system checks are SystemComponents, OSDiskStorage,-d SharedStorage, and NetworkComponents Create an output file. Provide the name of the file after the -f flag. The output for the validation commands-f is printed to this file. If you do not specify that output is sent to a file, then it is sent to the Page display (stdout).-h Print help information.-v Provide verbose output.-V Print the validation tool version.An example of a system problem can be shown here.RESULT: Powersupply PS0 V_IN=Disabled I_IN=Disabled V_OUT=0.88 Volts I_OUT=2 AmpsIN_POWER=Disabled OUT_POWER=Disabled – as you can see I_IN and V_IN is disabled which mean there is anissueRESULT: Powersupply PS1 V_IN=206 Volts I_IN=3.75 Amps V_OUT=12.08 Volts I_OUT=4.40 AmpsIN_POWER=720 Watts OUT_POWER=670 WattsA system without a power supply issue would display something like this:RESULT: Powersupply PS0 V_IN=204 Volts I_IN=1.88 Amps V_OUT=12.08 Volts I_OUT=27.20 AmpsIN_POWER=380 Watts OUT_POWER=330 Watts --RESULT: Powersupply PS1 V_IN=202 Volts I_IN=1.88 Amps V_OUT=12.08 Volts I_OUT=27.20 Amps108 17IN_POWER=350 Watts OUT_POWER=340 WattsSUCCESS: Both the powersupply are ok and functioning
  18. 18. Engineered SystemsAs you can see, even though the system is functioning due to the redundant nature of the system components. If there is anissue, then running an oakcli validate can identify the source of the system or hardware problem. These types of issues can beproactively determined by using ASR (Automatic Service Request).Validate also provides DiskCalibration data to ensure that the disk is functioning as expected and no disk failures.RESULT: Random read throughput across all 20 SASDisk = 4037 IOPSRESULT: Random read throughput across all 4 SSDDisk = 16368 IOPSRESULT: Random read throughput across all 20 SASDisk = 1844 MBPSRESULT: Random read throughput across all 4 SSDDisk = 941 MBPSAn issue with disk access or calibration can also be seen below. In this example, a disk was manually dropped from thediskgroup to simulate this test. ERROR: Error when running orion for HDD_E0_S01_705927132_MBPS : ORION: ORacle IO Numbers -- Version 11.2.0.3.0 Calibration will take approximately 1 minutes. Using a large value for -cache_size may take longer. Error completing IO (storax_aiowait) ORA-27061: waiting for async I/Os failed Linux-x86_64 Error: 5: Input/output error Additional information: -1A Network Check will look at the network components to validate the network speed and Ethernet ports that are currentlyactive. Oracle Database Appliance has a one 10G interface per node as well several 1G interfaces. In the example only 1Ginterfaces are active.INFO: Doing oak network checksRESULT: Detected active link for interface eth0 with link speed 1000Mb/sRESULT: Detected active link for interface eth1 with link speed 1000Mb/sRESULT: Detected active link for interface eth2 with link speed 1000Mb/sRESULT: Detected active link for interface eth3 with link speed 1000Mb/sRESULT: Detected active link for interface eth4 with link speed 1000Mb/sWARNING: No Link detected for interface eth5WARNING: No Link detected for interface eth6WARNING: No Link detected for interface eth7WARNING: No Link detected for interface eth8WARNING: No Link detected for interface eth99The oakcli validate one button deployment provides a plethora of diagnostic information that provide insight into theoperation of the Oracle Database Appliance. It should be used regularly, but with caution, to ensure the machine isfunctioning as intended. 18 108
  19. 19. Engineered SystemsONE BUTTON STORAGE DIAGNOSTICSDisks can fail, so it is important to ensure that storage diagnostics are key as part of the one button deployment of the ODA.Storage diagnostics are a key component in the oakcli command set. From the ODA documentation, below are some of thecommands that are available for storage diagnostic and disk location. Option Purpose Locates the disk associated with the ASM disk that you indicate by its ASM disk name, where: -h outputs the oakcli locate optionsoakcli locate[-h] [disk diskname on|off] disk diskname locates the ASM disk whose name you designate on turns on the LED of the specified disk off turns off the LED of the specified diskoakcli show controller Displays the configuration for active cores as defined byoakcli show core_config_key the registered core keyoakcli show disk Displays disksoakcli show diskgroup Displays disk groupsoakcli show expander Displays expandersoakcli show –h Show help commandoakcli show storage Displays storage device statusProvides a detailed output patch versions for softwareand firmware on the applianceIt is very important to understand and execute some of the above commands to ensure the stability and understand the stateof the disk. #oakcli show disk NAME PATH TYPE STATE STATE_DETAILS pd_00 /dev/sdam HDD ONLINE Good pd_01 /dev/sdaw HDD ONLINE Good pd_02 /dev/sdaa HDD ONLINE Good pd_03 /dev/sdak HDD ONLINE GoodBased on the output above, if there is a bad disk, you can work with Oracle Field Services or your internal hardware resourcesto replace the disk. The oakcli locate disk helps in ensuring the right bad disk is located and replaced. Also, the show storageoption provides a complete picture of the storage which includes the controllers, expanders as well as firmware version of thedisks. 19 108
  20. 20. Engineered SystemsONE BUTTON ODA HEALTHCHECKSThis piece is new since ODA 2.3.x and is not part of the OAK framework. However it is important to note and use asneeded. Oracle Database Appliance Checks, or ODAchk as the tool is known, is the Oracle Database Appliance version ofthe popular RACchk and EXAchk tools.Most of the OAK built in tools look at hardware and software, but the ODAchk provides more in-depth looks the systemfrom the database perspective and not the appliance perspective. The tool ensures things like RAC, Grid, ASM as well ascommon database configuration parameters are set up right. ODAchk should be run on a regular basis and is recommendedbefore and after any system change as well as any system patching. It is surprising how sometimes on a running system howmany best practices are not being followed. $/opt/oracle/oak/odachk/odachk – Please run as oracle software owner.FAIL => Database parameter DB_LOST_WRITE_PROTECT is NOT set to recommended value on tst12 instanceFAIL => Database parameter GLOBAL_NAMES is NOT set to recommended value on tst12 instanceFAIL => One or more errors or warnings for OS Disk Storage checksWARNING => One or file generated network and bonding interface checksThere is an html more warnings for with detail information as well as best practices to follow.ONE BUTTON MANAGEMENT DIAGNOSTIC COLLECTIONThe OAK framework comes with its own management and diagnostic collection framework including a cleanup script to re-initialize the Oracle Database Appliance and establish “pay as you go” Oracle licensing core counts.Oakcli manage diagcollect – Collect and package diagnostic files for Oracle Support # cd /opt/oracle/oak/bin # ./oakcli manage diagcollect – this will collect diagnostic data from CRS and various DB and OAK components to submit to oracleOakcli manage cleanrepo – Remove old versions of ODA patch and version files #cd/opt/oracle/oak/bin # oakcli manage cleanrepo --ver 2.4.0.0.0 Can not delete the current or latest version repository. #oakcli manage cleanrepo --ver 2.3.0.0.0 – to cleanup previous version files # oakcli update –patch 2.4.0.0.0 –clean – in 2.4 has been enhanced to cleanup temporary files after patchingCleanupDeploy.plIf there is ever a need to reimage the box or uninstall all software you can use the CleanupDeploy.pl to do so 1. Log in as root. 2. Change directory to /opt/oracle/oak/onecmd. 3. Run the command cleanupDeploy.pl. For example: # ./cleanupDeploy.pl 20 108
  21. 21. Engineered SystemsConfiguring the Oracle licensing core countsIf you are using the pay as go Oracle Licensing scheme you can use oakcli to apply the core cap and install the key you havegenerated on MOS (My Oracle Support). # oakcli apply –h Usage:oakcli apply core_config_key <key_file> # oakcli show core_config_key Optional core_config_key is not applied on this machine yet ! # oakcli show core_config_key Hosts serialnumber = ABC123 Configured Cores = 20Trace File AnalysisIn Oracle Database Appliance 2.4 a new option for TFA was installed. The Trace File Analyzer is a new framework for thecollection and analysis of trace files that help the proactive collection of errors in real-time mode and automatically collectsthe right log files as the problem is happening. More information on TFA can be found in MOS note 1513912.1.ONE BUTTON PATCHINGThis paper already discusses patching but in the one button section it is important to mention the capabilities of the onebutton infrastructure to be able to patch all software or individual components using the one button patching capabilities builtinto the Oracle Appliance Kit. # cd /opt/oracle/oak/bin # ./oakcli update -patch versionWhile the patching section of this paper discusses more detail, the command above is the but in essence of single buttonpatching, The command above can patch an appliance.ONE BUTTON DATABASE DEPLOYMENTStarting with ODA version 2.3 and above, newer releases of the ODA 2.3 and above , the option is now provided to createmultiple database homes as well as create new databases. There are options to upgrade database as well delete databases andhomes as well.To create a new database home # oakcli create dbhome -version 11.2.0.3.3 – will create a new dbhome with 11.2.0.3.3 21 108
  22. 22. Engineered SystemsTo create a new database # oakcli create database -db testdb -version 11.2.0.3.3 – will create a database and a new database home #oakcli create database -db testdb –oh OraDb11203_home2 – will create a new database in an existing dbhome.To delete a database or database home # oakcli delete database –db olddb # oakcli delete dbhome –oh dbhome_nameTo. upgrade a database from one version to another (11.2.0.2 to 11.2.0.3) #oakcli upgrade database {-db <db_names> | -from <source_home>} -to <destination_home>.SUMMARYIn summary, the Oracle Database Appliance packs a lot of bang for the buck. As the basic model in Oracle engineeredAppliance product line, it is the stepping stone to understand and evaluate if an appliance model is a viable option for acompany. An Oracle Database Appliance provides ease of setup, patching as well as diagnostics. As an administrator that haspreviously only had a specific focus, there are a lot of things that an administrator has to be aware of, including the network,hardware and operating system. Oracle has made it easy for people to manage the appliance as one unit by making variousaspects easier to manage and maintain using the Oracle Appliance Kit (OAK) framework. There are various monitoringaspects built into the box as well as one button activities to ensure that management of the appliance is simple.REFERENCESOracle Database Appliance - 2.X Supported Versions & Known Issues [ID 888888.1]Information Center: Oracle Database Appliance [ID 1417713.2]ODA (Oracle Database Appliance): New Deploy Step-by-Step Short Guide (version 2.3) [ID 1486534.1]ODA (Oracle Database Appliance): oakcli command reference [ID 1417879.1]ODA (Oracle Database Appliance): How To Configure xbond,bond1,bond2 post deploy [ID 1422563.1]ODA (Oracle Database Appliance): The Steps to replace failing disks [ID 1496114.1]Gathering data for the Oracle Database Appliance server [ID 1358795.1]ODA (Oracle Database Appliance): Test Plan Outline [ID 1474273.1]Oracle Database Appliance Setup PosterTFA Collector- The Preferred Tool for Automatic or ADHOC Diagnostic Gathering Across All Cluster Nodes [ID 1513912.1] 22 108

×