Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Smooth as Silk Exadata Patching


Published on

The more components a system has, the more challenging its maintenance becomes. Oracle Exadata marries storage with computation through a fast, reliable network, and patching all of these seems daunting. Many companies seem to struggle with it, with some even avoiding it altogether by keeping it “pending.” This session presents tested, applied, and working tips to make the Oracle Exadata patching experience smooth as silk, like vacationing in Hawaii.

Published in: Technology
  • Be the first to comment

Smooth as Silk Exadata Patching

  1. 1. Smooth-as-Silk Oracle Exadata Patching Fahd Mirza
  2. 2. © 2012 – Pythian About Fahd Mirza • Oracle DBA since 2001 • Oracle ACE • Oracle Blogger • Blog at • Oracle DBA Evangelist • Conference Speaker • Interviewed by Oracle Magazine Sep 11 • Currently Working at Pythian • Managing Exadata for 2+ years • @fahdmirza (Twitter)
  3. 3. © 2012 – Pythian Why Pythian • Recognized Leader: • Global industry leader in data infrastructure managed services and consulting with expertise in Oracle, Oracle Applications, Microsoft SQL Server, MySQL, big data and systems administration • Work with over 200 multinational companies such as, Fox Sports, Nordion and Western Union to help manage their complex IT deployments • Expertise: • One of the world’s largest concentrations of dedicated, full-time DBA expertise. Employ 8 Oracle ACEs/ACE Directors • Hold 7 Specializations under Oracle Platinum Partner program, including Oracle Exadata, Oracle GoldenGate & Oracle RAC • Global Reach & Scalability: • 24/7/365 global remote support for DBA and consulting, systems administration, special projects or emergency response
  4. 4. © 2012 – Pythian • Overview • Types of Exadata Patching • Patching Scenarios • Best Practices • RecommendedTools
  5. 5. © 2012 – Pythian Pythia, Pythian, Patching
  6. 6. © 2012 – Pythian Golden Rule of Exadata Patching Don’t Delay Don’t RushAND
  7. 7. © 2012 – Pythian Patch It • The more components a system has, the more challenging its maintenance becomes. • But is it daunting really? • It's lengthy but with careful planning, preparation and focussed execution, it's a breeze.
  8. 8. © 2012 – Pythian Exadata Architecture DB Nodes idb / RDS Storage Nodes Infiniband11gR2 / ASM CELLSRV
  9. 9. © 2012 – Pythian Exadata Software • At Database Server • Oracle 11gR2 • At Storage Server • CELLSRV • Management Server • Restart Server • OSWatcher
  10. 10. © 2012 – Pythian Types of Exadata Patches • There are different types of patches that are required to maintain Exadata Database Machine: • Exadata Storage Server • Database server • RDBMS and GI • Operating system and firmware • InfiniBand switch • Additional components
  11. 11. © 2012 – Pythian Exadata Storage Server Patches • An Exadata Storage Server patch contains updates to • Firmware • Operating system • Exadata Storage Server software. • Only use single downloadable patch from MOS for Storage Server update. • Don’t manually update firmware or software on storage servers. • Storage Server patch may contain updates to firmware and OS to apply to DB servers. Known as a database server minimal pack.
  12. 12. © 2012 – Pythian Installation of Exadata Storage Server Patches • Storage Server patches are installed using patchmgr, which is a script supplied with the patch. • The patches may be installed in one of two ways: • Rolling: Storage server patching performed in a rolling manner is applied one storage server at a time while databases remain operational until all storage servers are patched. • Non-rolling: Storage server patching performed in a non-rolling manner is applied to all storage servers simultaneously with databases offline. • Prerequisites and instructions for installing the patch are provided in a README supplied with the patch.
  13. 13. © 2012 – Pythian Compatibility of Exadata Storage Server Patches • Exadata Storage Server patches are supplied independent of Oracle Database Server patches • i.e. patches applied to RDBMS or Grid Infrastructure homes using OPatch. • Exadata Storage Server patch may require a specific Oracle Database Server patch level, or database server firmware or operating system version.
  14. 14. © 2012 – Pythian Database Server Patches • Database servers have two types of patches • Oracle Database Software Patches • Firmware / Operating System Patches
  15. 15. © 2012 – Pythian DB Server : DB Software Patches • Oracle Database Server software patches contain updates to Oracle Database and Oracle Clusterware software. • They are supplied as one or more downloadable patches from MOS. • Most updates are delivered in bundle patches created specifically for Exadata • For Oracle Database (DB_BP) and Oracle Clusterware (GI_BP).
  16. 16. © 2012 – Pythian DB Server : DB Software Patches • Bundle patches contain no Exadata-specific code and may be installed on non-Exadata systems. • However, that priority for requests to merge one-off fixes into Exadata bundle patches is given to Exadata customers. • Bundle patches are cumulative. • Bundle patches contain a recently released Patch Set Update (PSU), which, in turn, contain a recently released Critical Patch Update (CPU). • Patches in addition to bundle patches may be recommended or required.
  17. 17. © 2012 – Pythian DB Server : DB Software Patches Installation • Opatch utility is used to apply the DB server patches in RDBMS and Grid Homes. • Prerequisites and instructions for installing a patch are provided in a README supplied with the patch. • There may be supplementary documentation for a patch that is referenced in a MOS note. • Oracle Database Server software patches may be installed in any order unless otherwise indicated in a patch README
  18. 18. © 2012 – Pythian DB Server : DB Software Patches Compatibility • Oracle Database Server software may require a minimum Exadata Storage Server software version. • Patches listed together as the current set of recommended patches in Note 888828.1 have been tested as a group by MOS. • If OPatch reports a conflict with one or more patches in this list, you should not force apply your patch. • Contact Oracle Support for assistance to resolve the patch conflict.
  19. 19. © 2012 – Pythian DB Server: Firmware / OS Patches • Minimal Pack for pre / yum for and above • Obtain updates for firmware from the hardware support site that is specific to the database server. • Firmware downloads for Sun Fire DB can be obtained from OTN. • Don’t update Storage Servers with firmware downloaded from OTN. • Use only Exadata Storage Server patches. • The database server manufacturer and model used can be obtained with the commands: • 'dmidecode --string system-manufacturer' and 'dmidecode --string system-product- name'.
  20. 20. © 2012 – Pythian DB Server: Firmware / OS Patches • Oracle Linux (OL, earlier Oracle Enterprise Linux OEL) is the required Linux distribution. • Obtain updates for OL from the Unbreakable Linux Network (ULN). • Do not automatically apply kernel updates because it may break compatibility with OFED software. • Be careful when installing packages that reconfigure operating system parameters, such as the oracle-validated RPM. • Parameter changes may remove Exadata-specific best practice configuration provided when the system was deployed.
  21. 21. © 2012 – Pythian DB Server: Firmware / OS Patches Database Server Minimal Pack • Storage Server patch may contain updates to firmware and operating system to apply to database servers. • This is known as a database server minimal pack. • Minimal pack is designed to only patch database servers that • are running the original deployed operating system, • or have been updated only with a previous minimal pack. • Minimal pack update to a specific version does not provide a database server that is equivalent to a newly shipped database server at the same version.
  22. 22. © 2012 – Pythian DB Server: Firmware / OS Patches Installation • Any firmware or OS update may be installed, uninstalled, or reconfigured provided the following guidelines are satisfied: • Requirements to run Oracle Database 11g, Release 2 (11.2) remain intact. • Storage Servers and database servers should interoperate with same InfiniBand host channel adapter (HCA) firmware version. • Obtain installed InfiniBand HCA firmware version with command "/usr/sbin/ibstat". • Exadata Storage Servers and database servers should interoperate with same InfiniBand (OFED) software versions. • OFED software version is obtained with command "rpm --query ofa-$(uname -r)".
  23. 23. © 2012 – Pythian DB Server: QDPE • New strategy for patch releases for Database and GI on Exadata • Customers will be less stressed because • Number and Frequency of patches will decrease • Quality of patching will improve • Quarterly Bundle Patches will be released for GI and RDBMS • They are being called as Quarterly Database Patches for Exadata • They are cumulative and include latest PSU which include CPU. • Interim bundle patches will still be available to address critical issues • - Oct 2012 QDPE is scheduled for Oct 2012
  24. 24. © 2012 – Pythian InfiniBand Switch Patching • An InfiniBand switch patch contains updates to the software and/or firmware for InfiniBand switches. • Updates to InfiniBand switch software and/or firmware must occur only with the patch downloaded from MOS. • Prerequisites and instructions for installing a patch are provided in a README supplied with the patch. • InfiniBand switch software version has no dependency on Exadata Storage Server software version, unless indicated patch README.
  25. 25. © 2012 – Pythian InfiniBand Switch Patching • IB Switch patches are installed and rolled back via ILOM • Infiniband patches are not cumulative. • You have to upgrade version by version in order. • For migrate from version 1.0.1-1 to 1.3.3-2, first upgrade 1.0.1-1 to 1.1.3-2 and then 1.1.3-2 to 1.3.3-2. • First patch Spine switch, reboot it, and then proceed it to patching the leaf switches.
  26. 26. © 2012 – Pythian Additional Exadata Component Patches • Additional components refer to the Ethernet switch, KVM, and PDU. • Firmware and/or software may be maintained at your discretion unless otherwise specific in Note 888828.1.
  27. 27. © 2012 – Pythian Quarterly Full Stack Download Patch(QFSDP) • Oracle is making things easier for downloading Exadata patching • QFSDP includes all the patches for: • Exadata Storage Server • RDBMS Home and Grid Infrastructure Home • Infiniband Switches • OPLAN and OPATCH • PDU • EM Agent and EM Plug-in and EM OMS • Individual downloads for the above components will also be available.
  28. 28. © 2012 – Pythian Experience of Upgrading Exadata to • This is about Upgrade from version BP10 to BP2. • MOS note [ID 1373255.1] is your friend • Most of steps can be done rolling mode so don't require full downtime. • Prepare the list of all one-off patches which you need to apply and see if they have been fixed in
  29. 29. © 2012 – Pythian Experience of Upgrading Exadata to • As always, fulfillment of pre-requisites for is paramount. • Make sure InfiniBand switch software release version is 1.3.3-2 or later • Use "/opt/oracle.SupportTools/ -I" command. • Make sure exadata Storage Server software is release or later. • Use "imageinfo" command. • Check version of your box. • Use "dmidecode -s system-product-name" command.
  30. 30. © 2012 – Pythian Experience of Upgrading Exadata to • Check your BP level and install the patch for bug 12539000 if you have BP11 and lower. • The bug leads to an oracle fatal error during the rolling upgrade. • It can be done in the rolling mode and takes about 1 – 1.5 hours to complete patching for 4 database nodes. • Make sure you have satisfied all pre-reqs in MOS note ID 1373255.1
  31. 31. © 2012 – Pythian Experience of Upgrading Exadata to • Then comes installation of the new GI software. • Oracle recommends “out of place” upgrade for 11gR2. • Did it using silent installation for GI with “UPGRADE” option. • It can be done during normal business hours without any impact to the environment. • It took just above one hour for us to complete. • We had the GI at that stage and could continue with database software.
  32. 32. © 2012 – Pythian Experience of Upgrading Exadata to • We upgraded the database software and databases to • Installed the new database software to a new directory and recompile it with RDS option. • Used silent install with “INSTALL_DB_SWONLY” and recompiled the libs to use RDS after it. • Verify if the database using RDS for IB by running command $ORACLE_HOME/bin/skgxpinfo (for version starting from • The installation was simple and took about 1 hour. • Then applied latest bundle patch to GI and RDBMS software and it took about an hour.
  33. 33. © 2012 – Pythian Experience of Upgrading Exadata to • Next we upgraded the database after verifying all the parameters and pre- upgrade checks. • The MOS note [ID 1373255.1] is very helpful. • It should not take more than one hour to perform upgrade and postupgrade steps on a database. • Make sure to gather statistics for the fixed objects after the upgrade. • Adjust database compatible parameter,ASM and disk group compatibility accordingly.
  34. 34. © 2012 – Pythian Step by Step Patching Scenario • Half Rack ofV2 Exadata - 4 database nodes, 7 cell nodes • Exadata Storage Cell Patch (Upgrade to • Minimal Pack on the database Node
  35. 35. © 2012 – Pythian Exadata Storage Cell Patch (Upgrade to - Step by Step • Run as root. • This is only done on one database node only • This will call dcli under the cover to patch all storage Cell nodes
  36. 36. © 2012 – Pythian Exadata Storage Cell Patch (Upgrade to - Step by Step • Ensure the various cell network configurations are consistent • The following command shouldn't return any data • dcli -g /root/cell_group -l root '/opt/oracle.cellos/ipconf –verify’ • Shut down the Oracle components on all the database hosts. • dcli -g /root/dbs_group -l root "/opt/oracle.oswatcher/osw/" • dcli -g /root/dbs_group -l root "/u01/app/ stop crs -f" • Verify no cluster processes, SMON,TNS are running
  37. 37. © 2012 – Pythian Exadata Storage Cell Patch (Upgrade to - Step by Step • Shut down all cell services on all cells that will be patched. • dcli -g /root/cell_group -l root "cellcli -e alter cell shutdown services all“ • Start Patching using the patchmgr Utility • ./patchmgr -cells /root/cell_group -patch • Make sure screen utility is being used • Monitor patch activity using less -rf patchmgr.stdout from another terminal session • This could run for 4-5 hours
  38. 38. © 2012 – Pythian Exadata Storage Cell Patch (Upgrade to - Step by Step • Once patching is done, a few verification steps • Gather image status and history dcli -g /root/cell_group -l root "imageinfo" dcli -g /root/cell_group -l root "imagehistory" • The imageinfo command shows the current (Active), and previous (Inactive) system info on the storage cell.
  39. 39. © 2012 – Pythian Exadata Storage Cell Patch (Upgrade to - Step by Step • Check the /var/log/cellos/validations.log and /var/log/cellos/vldrun*.log files for any failures. • In output of imageinfo, the image status is marked as failure when there is a failure reported in one or more places in these validations. • If a specific validation failed, then the log will indicate where to look for the logs for that validation. • Examine the specific log to determine the problem, and check against know issues. • If the issue is not listed as a known issue in the My Oracle Support note for the patch, then contact MOS
  40. 40. © 2012 – Pythian Exadata Storage Cell Patch (Upgrade to - Step by Step • Check following logs with the commands for any failure: • dcli -g /root/cell_group -l root "tail -50 /var/log/cellos/validations.log" • dcli -g /root/cell_group -l root "tail -50 /var/log/cellos/vldrun.each_boot.log" • dcli -g /root/cell_group -l root "tail -50 /var/log/cellos/vldrun.first_upgrade_boot.log" • dcli -g /root/cell_group -l root "tail -50 /var/log/cellos/vldrun.upgrade_reimage_boot.log"
  41. 41. © 2012 – Pythian Exadata Storage Cell Patch (Upgrade to - Step by Step • To rollback the patch: • Check the prerequisites: • ./patchmgr -cells /root/cell_group -rollback_check_prereq • Perform the rollback: • ./patchmgr -cells /root/cell_group -rollback
  42. 42. © 2012 – Pythian Exadata Minimal Patch on Database Nodes • The storage cell patching is tightly knitted together with the database patching. • After the storage cell patching, you have to apply the minimal patch to the database nodes. • Not doing so could result in bugs, errors, uncertain behaviors like disabling of storage indexes • Apply database minimal patch on each database node as root • Go to patch install directory and run it as ./ –force • This will run it in background and database nodes will reboot • Check imageinfo and after patching for any errors
  43. 43. © 2012 – Pythian Patching Exadata with OEM • Oracle Enterprise Manager currently does not support patching of the Exadata Database Machine • Compute Node and the Storage Cells. • No support available for Oracle Databases running on Oracle Data Guard Configuration. • It can apply Bundle Patches on DB and GI homes
  44. 44. © 2012 – Pythian Patching Exadata with OEM • Quarterly Exadata Bundle Patches Released by Oracle • Bundle Patches Include Fixes to both GI and DB Homes • Manual Patching is Lengthy and Cumbersome • On Quarter Rack it requires 100+ steps • OPatch auto does not provide automation of out-of-place patching.
  45. 45. © 2012 – Pythian Patching Exadata with OEM • Oracle Enterprise Manager provides • Complete automation to patch the cluster in zero-downtime • Both in In-place and Out of Place modes. • Oracle recommends that customers apply these patches out of place.
  46. 46. © 2012 – Pythian Patching Exadata with OEM • Out of Place Patching • OEM 12c introduces Out of Place patching method. • Clones of the existing oracle homes are created, and the patches are applied to the cloned home instead of the original home. • Once the cloned home is patched, user can selectively switch the database instances to run from the cloned home.
  47. 47. © 2012 – Pythian Patching Exadata with OEM • Advantages of Out of Place Patching by OEM • Minimal downtime • Flexibility • Recoverability • Easy and Intuitive Approach
  48. 48. © 2012 – Pythian Patching Guidelines • Review the patch README file for known issues, patch installation and deinstallation instructions, and special notes. • Validate patch installation on theTEST system by applying it. • Evaluate technologies and practices that may reduce downtime and risk, such as transient logical standby patching, Data Guard Standby-First Patch Apply, and rolling patch apply. • Define, document, and test fallback procedures required if a patch must be rolled back.
  49. 49. © 2012 – Pythian Patching Guidelines • If applying the patch to address a specific issue, verify that the patch provides the functionality desired. • Verify there is no performance, availability, or operational regression while running workload that is representative to that which runs on the PRODUCTION system. • AWR and SQL Performance Analyzer may be used to assess performance improvement or regression resulting from the patch. • Before and after patch application, run exachk
  50. 50. © 2012 – Pythian Patch Helping Features • Cell Rolling Apply • Exadata Storage Server software patches may be applied in a rolling fashion, one storage server at a time. • The supplied patching utility manages rolling patch application to all storage servers.
  51. 51. © 2012 – Pythian Patch Helping Features • Bundle Patch Merge • In patches for Oracle Database and Oracle Clusterware were delivered separately. • A bundle patch for combines both sets of updates which reduces patching time by reducing the number of patches to install.
  52. 52. © 2012 – Pythian Patch Helping Features • OPatch Auto Installable: • Opatch Auto command replaces manual steps to apply BP • Opatch Auto: • Stops the Clusterware and Database • Applies the Patch to Grid home and database home • Starts the Clusterware and the database • Opatch Auto is repeated on all database nodes
  53. 53. © 2012 – Pythian OPLAN • No need to manually identify environment specific patching commands • No need to spend huge time in understanding target configuration and generating commands • OPLAN automatically collects configuration information for target • Step by step patching instructions can be generated • Instructions cover both patch application and patch rollback steps. • Instructions cover patch options for In place / Out-of-Place methods. • The OPLAN utility works directly on the target, similar to OPatch.
  54. 54. © 2012 – Pythian OPLAN Installation • Oplan available from MOS, look for patch 11846294 • Download patch file to Oracle Home in the target • Take backup of $ORACLE_HOME/Opatch, just in case. • Remove $ORACLE_HOME/OPatch/oplan if already present by running: • rm -r $ORACLE_HOME/OPatch/oplan • As Oracle Home owner, unzip the oplan patch in the Oracle Home under 'OPatch' folder: • unzip p11846294_<version_platform>.zip -d $ORACLE_HOME/Opatch • Oplan will be installed under $ORACLE_HOME/OPatch/oplan folder.
  55. 55. © 2012 – Pythian OPLAN Configuration • Download actual patch you want to apply to local directory on target • Set the ORACLE_HOME variable to theTargets Oracle Home • As Oracle Home owner, run the following command: • $ORACLE_HOME/oplan/oplan generateApplySteps <bundle patch location> • OPLAN generates instructions in HTML and Text format. • $ORACLE_HOME/cfgtoollogs/oplan/<TimeStamp>/InstallInstructions.html • $ORACLE_HOME/cfgtoollogs/oplan/<TimeStamp>/InstallInstructions.txt
  56. 56. © 2012 – Pythian OPLAN Configuration • Verify the configuration by Oplan matches your configuration. • Oplan may offer options across In Place / Out of Place patching • Follow the patching instructions and apply the patch to the target. • In order to rollback the patch use: • $ORACLE_HOME/oplan/oplan generateRollbackSteps <bundle patch location> • The OPlan logs are located in $ORACLE_HOME/cfgtoollog/Oplan.
  57. 57. © 2012 – Pythian Exachk • Exachk is Exadata assessment tool • Exachk audits various important configuration settings for Exadata: • Database Servers, • Storage Servers • Infiniband Switches. • Recommended to run routinely to check for any errors or warnings in the system at every level. • Provides detailed recommendations
  58. 58. © 2012 – Pythian Exachk • The tool audits configuration settings within the following categories • OS kernel parameters • OS packages • Many other OS configuration settings important to RAC. • CRS/Grid Infrastructure • RDBMS • ASM • Database parameters • Many other configuration settings.
  59. 59. © 2012 – Pythian How to run Exachk • Log in to the system as the oracle • Stage kit on the node on which the tool will be executed • Unzip kit, leaving script / driver files in same directory • Verify permissions on exachk • Run the tool • $ ./exachk • Follow the prompts. Read and understand all the messages • Upon completion, review the report findings.
  60. 60. © 2012 – Pythian Screen Utility • Keeps it Alive all theTime • No Worries About Network Disconnections • No Worries About Loosing Remote Sessions • Creates aVirtualTerminal • 40VirtualTerminals Can be Created • Type screen in shell and it gets Created • Leave that session /reconnect / Switch among them
  61. 61. © 2012 – Pythian Patch Release Frequency • Subject to Change By Oracle anytime • Exadata Storage Server Patches • Quarterly • Database Patch for Exadata • Quarterly • Infiniband Patches • Between 6 months to anYear
  62. 62. © 2012 – Pythian Word of Caution • Exadata is still New • Evolving Rapidly • Many components, many changes, many bugs, many patches • Consult My Oracle Support for Latest Patches and Releases available before applying
  63. 63. © 2012 – Pythian Conclusion • The complexity of the Exadata patching can be a bit daunting at first • Careful planning and execution can make it a breeze • Open a Service Request with MOS before patching and get them review your patching procedure • Make sure to run Exachk before and after patching • Patch it but don’t rush it and don’t delay it too much.
  64. 64. © 2012 – Pythian Thank you and Q&A @pythian 1-877-PYTHIAN To contact us… To follow us… @pythianjobs