Effective Oracle Home Management
with the New Release Model
Ludovico Caldara
Computing Engineer
IT Department, CERN
ludovico.caldara@cern.ch
@ludodba
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
2
About Ludovico Caldara
■ 18+ Years DBA (Not Only Oracle)
■ ITOUG co-founder
■ OCP (11g, 12c, MySQL) & OCE
■ Italian living in Switzerland
■ http://www.ludovicocaldara.net
■ @ludodba
■ ludovicocaldara
05.12.2018 3
Effective Oracle Home Management
#UKOUG_Tech18
CERN
• CERN - European Laboratory for Particle Physics
• Founded in 1954 by 12 Countries for fundamental
physics research in the post-war Europe
• Today 22 member states and world-wide collaborations
• About ~1100 MCHF yearly budget
• 2’300 CERN personnel
• 10’000 users from 110 countries
405.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
Fundamental Research
• What is 95% of the Universe made of?
• Why do particles have mass?
• Why is there no antimatter
left in the Universe?
• What was the Universe like,
just after the "Big Bang"?
505.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
Effective Oracle Home Management
#UKOUG_Tech18
6
The Large Hadron Collider (LHC)
Largest machine in the world
27km, 6000+ superconducting magnets
Emptiest place in the solar system
High vacuum inside the magnets
Hottest spot in the galaxy
During Lead ion collisions create temperatures 100 000x hotter than the heart of the sun
Fastest racetrack on Earth
Protons circulate 11245 times/s (99.9999991% the speed of light)
Quick overview of the
New Release Model
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
7
What does it change?
Jan 18 Apr Jul Oct Jan 19 Apr Jul Oct
RU 18.1.0 18.2.0 18.3.0 18.4.0 18.5.0 18.6.0 18.7.0 18.8.0
RUR1 18.2.1 18.3.1 18.4.1 18.5.1 18.6.1 18.7.1
RUR2 18.2.2 18.3.2 18.4.2 18.5.2 18.6.2
RU 19.1.0 19.2.0 19.3.0 19.4.0
RUR1 19.2.1 19.3.1
RUR2 19.2.2
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
8
2
What does it change?
Jan 18 Apr Jul Oct Jan 19 Apr Jul Oct
RU 18.1.0 18.2.0 18.3.0 18.4.0 18.5.0 18.6.0 18.7.0 18.8.0
RUR1 18.2.1 18.3.1 18.4.1 18.5.1 18.6.1 18.7.1
RUR2 18.2.2 18.3.2 18.4.2 18.5.2 18.6.2
RU 19.1.0 19.2.0 19.3.0 19.4.0
RUR1 19.2.1 19.3.1
RUR2 19.2.2
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
9
2
What does it change?
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
10
Short
Development
cycle
Frequent
Upgrades
And Updates
“Patch soon, patch often” vs reality
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
11
Frequent patching?
• Time consuming
• Complex
• Risky
• Unnecessary (?)
• Error-prone
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
12
How many bugs could you encounter?
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
13
How many bugs could you encounter?
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
14
18.3.0 18.3.1 18.4.0
How many bugs could you encounter?
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
15
18.3.0 18.3.1 18.4.0
634
How many bugs could you encounter?
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
16
18.3.0 18.3.1 18.4.0
634 1089
How many bugs could you encounter?
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
17
18.3.0 18.3.1 18.4.0
634 1089 1257
Which patches should I apply?
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
18
Brave vs Fearful
12.1 >= 12.2
Effective Oracle Home Management
#UKOUG_Tech18
19
How
do I
feel?
PSU
Bundle
Patch
BraveFearful How
do I
feel?
Release
Update
Revision
Release
Update
BraveFearful
• 05.12.2018
Brave vs Fearful
12.1 >= 12.2
Effective Oracle Home Management
#UKOUG_Tech18
20
How
do I
feel?
PSU
Bundle
Patch
BraveFearful How
do I
feel?
Release
Update
Revision
Release
Update
BraveFearful
Ludovico(*) Ludovico(*)
(*) yes, it’s Comic Sans
https://mikedietrichde.com/2017/10/24/differences-psu-bp-ru-rur/
• 05.12.2018
Common patching patterns
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
21
Common worst practices in maintaining homes
• Installing software by hand
• Different patch levels for Oracle Homes with the same path
• Not tracking the installed patches
• Oracle Home paths hard-coded in the operational scripts
• Not minding about naming conventions
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
22
Worst approach (big bang)
1. Stop everything
2. In-place binaries patching
3. Database patching, all at once
4. Start everything
Effective Oracle Home Management
#UKOUG_Tech18
23
UP
UP
UP
UP
DOWN
DOWN
DOWN
DOWN
UP
UP
UP
UP
UP DOWN UP
STOP ALL
START ALLPATCH BINARIES
PATCH DBs
• 05.12.2018
Worst approach (big bang)
1. Stop everything
2. In-place binaries patching
3. Database patching, all at once
4. Start everything
Effective Oracle Home Management
#UKOUG_Tech18
2405.12.2018
UP
UP
UP
UP
DOWN
DOWN
DOWN
DOWN
UP
UP
UP
UP
UP DOWN UP
STOP ALL
START ALLPATCH BINARIES
PATCH DBs
No easy
rollback
Long
downtime
Complex
planning
Different approach (still bad)
1. Install new binaries manually
2. Patch the new binaries
3. Stop, change OH, patch
databases one by one
4. Decommission old binaries
Effective Oracle Home Management
#UKOUG_Tech18
2505.12.2018
UP
UP
UP
UP
DOWN
DOWN
DOWN
DOWN
UP
UP
UP
UP
PATCH BINARIES STOP, MOVE, PATCH
ONE BY ONE
Different approach (still bad)
1. Install new binaries manually
2. Patch the new binaries
3. Stop, change OH, patch
databases one by one
4. Decommission old binaries
Effective Oracle Home Management
#UKOUG_Tech18
2605.12.2018
UP
UP
UP
UP
DOWN
DOWN
DOWN
DOWN
UP
UP
UP
UP
PATCH BINARIES STOP, MOVE, PATCH
ONE BY ONE
Complex to
do manually
Repetitive task
when many servers
Automate the installation!
• Automated creation
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
27
Soft
ware Patch Patch
Patched
Software
• runInstaller -silent
• opatch apply
Automate the installation!
• Automated creation
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
28
Soft
ware Patch Patch
Patched
Software
• runInstaller -silent
• opatch apply
https://github.com/MarisElsins/getMOSPatch
Automate the installation!
• Automated creation
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
29
Soft
ware Patch Patch
Patched
Software
• runInstaller -silent
• opatch apply
https://github.com/MarisElsins/getMOSPatch
https://github.com/ilmarkerm/ansible-oracle-home-mgmt
Image repo
Automate the installation!
• Automated creation • Software cloning
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
30
Soft
ware Patch Patch
Patched
Software
• runInstaller -silent
• opatch apply
18.3.1
18.4.0
Server
18.3.1
Image repo
Automate the installation!
• Automated creation • Software cloning
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
31
Soft
ware Patch Patch
Patched
Software
• runInstaller -silent
• opatch apply
18.3.1
18.4.0
Server
18.3.1
RUNDECK
ANSIBLE
RHP
OTHER
Less paths
Effective Oracle Home Management
#UKOUG_Tech18
3205.12.2018
18.3.0 18.4.0 18.4.0
+ one-off X
18.5.0
+ one-off X
18.5.0
+ one-off X
+ one-off Y
18.3.0 18.4.0 18.4.0
+ one-off X
18.5.0
+ one-off X
18.5.0
+ one-off X
+ one-off Y
18.4.1 18.4.1
+ one-off Y
GOOD:
BAD:
Workarounds vs one-offs
• Sometimes through underscore parameters
• Mitigate the number of one-offs
• Less merge requests
• Document it! (why, when, who)
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
33
1
2
3
alter system set "_px_groupby_pushdown"=off
comment='Ludo, 03.05.16: W/A for bug 18499088'
scope=both sid='*‘;
Rapid Home provisioning
• It might be the perfect product…
• Requires Lifecycle Management Pack
• The technology stack is complex
(Clusterware, ASM, ACFS, NFS, GNS…)
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
34
Rapid Home provisioning
• It might be the perfect product…
• Requires Lifecycle Management Pack
• The technology stack is complex
(Clusterware, ASM, ACFS, NFS, GNS…)
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
35
Managing Gold Images for Easy Patching &
Upgrading with Oracle RHP
Ian Cookson - Oracle
Room 1A, today 15:20
Custom approach
• Simple script
• Golden images
• Solid naming conventions
• I blogged about it!
http://www.ludovicocaldara.net/dba/oh-mgmt-[1..7]/
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
36
Implementation at CERN
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
37
Naming Conventions
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
38
Central Inventory server 1
<HOME NAME= "OraDB12Home1" LOC="/u01/app/oracle/product/12.2.0.1" TYPE="O" IDX="2">
<HOME NAME= "OraDB12Home2" LOC="/u01/app/oracle/product/12.2.0.1_2" TYPE="O" IDX=“3">
Central Inventory server 2
<HOME NAME= "OraDB12Home1" LOC="/u01/app/oracle/product/12.1.0.2" TYPE="O" IDX=“1">
<HOME NAME= "OraDB12Home2" LOC="/u01/app/oracle/product/12.2.0.1" TYPE="O" IDX="2">
Naming Conventions
• Central Inventory contains OH name
• OH name matches OH location
• No dots (.) allowed (inventory limitation)
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
39
Central Inventory
<HOME NAME=“18_3_0"
LOC="/u01/app/oracle/product/18_3_0"
TYPE="O" IDX="2">
• 1 to 1 mapping: easy to code
• 18c read-only OH uses the name
as subdir in `orabasehome`:
consistent naming!
Golden images
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
40
Golden Images Store
18_3_0.zip
18_4_0.zip
18_3_1.zip
SERVER1
18_3_0.zip
18_3_1.zip
SERVER2
18_3_1.zip
SERVER3
18_4_0.zip
Golden Image Metadata
Name Path
------- -----------------
18_3_0 /store/18_3_0.zip
18_3_1 /store/18_3_1.zip
18_4_0 /store/18_4_0.zip
Image store
Typical workflow
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
42
SERVER1
18_3_0
SERVER2…
First Install
Image store
Typical workflow
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
43
18_3_0.zip
SERVER1
18_3_0
SERVER2…
$ ohctl -c -n 18_3_0
Image store
Typical workflow
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
44
18_3_0.zip
SERVER1
18_3_0
SERVER2…
$ ohctl -i 18_3_018_3_0
Image store
Typical workflow
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
45
18_3_0.zip
SERVER1
18_3_0
SERVER2…
$ ohctl -i 18_3_0 -n 18_4_0
18_3_0
18_4_0
Image store
Typical workflow
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
46
18_3_0.zip
SERVER1
18_3_0
SERVER2…
Manual patching
18_3_0
18_4_0
Image store
Typical workflow
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
47
18_3_0.zip
SERVER1
18_3_0
SERVER2…
$ ohctl -c
18_3_0
18_4_0
18_4_0.zip
Image store
Typical workflow
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
48
18_3_0.zip
SERVER1
18_3_0
SERVER2…
$ ohctl -i 18_4_0
18_3_0
18_4_0
18_4_0.zip
18_4_0
What it actually does
Creation workflow
• Copy the Home
• Clean it up
• Create the zip
• Store the zip in the repo
• Store metadata
Effective Oracle Home Management
#UKOUG_Tech18
4905.12.2018
What it actually does
Creation workflow
• Copy the Home
• Clean it up
• Create the zip
• Store the zip in the repo
• Store metadata
Installation workflow
• Security checks
• Create new path
• Unzip from the store
• Attach home
• Relink (opt. with RAC)
• Run setasmgid
• Other stuff (e.g. TNS)
Effective Oracle Home Management
#UKOUG_Tech18
5005.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
5105.12.2018
DEMO
Further development at CERN
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
52
Read-only Oracle Home
$ roohctl –enable
Enabling Read-Only Oracle home.
Update orabasetab file to enable Read-Only Oracle home.
Orabasetab file has been updated successfully.
Create bootstrap directories for Read-Only Oracle home.
Bootstrap directories have been created successfully.
Bootstrap files have been processed successfully.
Read-Only Oracle home has been enabled successfully.
Check the log file /u01/app/oracle/cfgtoollogs/roohctl/roohctl-
180822AM102911.log.
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
53
Read-only Oracle Home
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
54
18_3_0
18_4_0
/u01/app/oracle/product/
/u01/app/oracle/product/
$ cat $ORACLE_HOME/install/orabasetab
#orabasetab file is used to track Oracle Home associated with Oracle Base
/u01/app/oracle/product/18_3_0:/u01/app/oracle:18_3_0:N:
ORACLE_HOME
ORACLE_HOME
Read only
Read-only Oracle Home
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
55
18_3_0
18_4_0
/u01/app/oracle/product/
/u01/app/oracle/product/
/u01/app/oracle/homes/
$ cat $ORACLE_HOME/install/orabasetab
#orabasetab file is used to track Oracle Home associated with Oracle Base
/u01/app/oracle/product/18_3_0:/u01/app/oracle:18_3_0:N:
ORACLE_HOME
/u01/app/oracle/homes/
ORACLE_HOME
orabasehome
orabasehome
18_3_0
18_4_0
Read only
TNS, init, etc.
AGILE INFRASTRUCTURE
SERVER
Insertion in the Agile Infrastructure
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
56
Image Store Scripts
REST client
Oracle Home
Grid Infrastructure Installation
• Puppet + Rundeck for automatic GI install
• Patched Grid Infrastructure versions beside the first one
https://www.markusdba.net/?p=294
• Out-of-place patching with 18c independent local-mode automaton
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
57
$ rhpctl move gihome –sourcehome old_path -destinationhome new_path
Independent Local-mode Automaton (18c)
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
58
# $GI_HOME/srvctl enable volume -volume GHCHKPT -diskgroup mgmt
# $GI_HOME/srvctl enable filesystem -volume GHCHKPT -diskgroup mgmt
# $GI_HOME/srvctl start filesystem -volume GHCHKPT -diskgroup mgmt
$ $GI_HOME/srvctl start rhpserver
Local Resources
---------------------------------------------------------------
ora.MGMT.GHCHKPT.advm
ora.mgmt.ghchkpt.acfs
ora.helper
Cluster Resources
---------------------------------------------------------------
ora.rhpserver
$ opatch lspatches
27908644;UPDATE 18.3 DATABASE CLIENT JDK IN ORACLE HOME TO JDK8U171
27923415;OJVM RELEASE UPDATE: 18.3.0.0.180717 (27923415)
28256701;TOMCAT RELEASE UPDATE 18.3.0.0.0 (28256701)
28090564;DBWLM RELEASE UPDATE 18.3.0.0.0 (28090564)
28090557;ACFS RELEASE UPDATE 18.3.0.0.0 (28090557)
28090553;OCW RELEASE UPDATE 18.3.0.0.0 (28090553)
28090523;Database Release Update : 18.3.0.0.180717 (28090523)
OPatch succeeded.
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
59
Independent Local-mode Automaton (18c)
• Damn-easy move of GI HOME
• Automatic resource relocation
• Enable Grid Infrastructure Golden Images
• Patching (apply and rollback) is done automatically
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
60
Back to the demo…
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
61
Questions?
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
62
Thank you!
Ludovico Caldara
Computing Engineer
IT Department, CERN
ludovico.caldara@cern.ch
@ludodba
05.12.2018
Effective Oracle Home Management
#UKOUG_Tech18
63

Effective Oracle Home Management - UKOUG_Tech18

  • 2.
    Effective Oracle HomeManagement with the New Release Model Ludovico Caldara Computing Engineer IT Department, CERN ludovico.caldara@cern.ch @ludodba 05.12.2018 Effective Oracle Home Management #UKOUG_Tech18 2
  • 3.
    About Ludovico Caldara ■18+ Years DBA (Not Only Oracle) ■ ITOUG co-founder ■ OCP (11g, 12c, MySQL) & OCE ■ Italian living in Switzerland ■ http://www.ludovicocaldara.net ■ @ludodba ■ ludovicocaldara 05.12.2018 3 Effective Oracle Home Management #UKOUG_Tech18
  • 4.
    CERN • CERN -European Laboratory for Particle Physics • Founded in 1954 by 12 Countries for fundamental physics research in the post-war Europe • Today 22 member states and world-wide collaborations • About ~1100 MCHF yearly budget • 2’300 CERN personnel • 10’000 users from 110 countries 405.12.2018 Effective Oracle Home Management #UKOUG_Tech18
  • 5.
    Fundamental Research • Whatis 95% of the Universe made of? • Why do particles have mass? • Why is there no antimatter left in the Universe? • What was the Universe like, just after the "Big Bang"? 505.12.2018 Effective Oracle Home Management #UKOUG_Tech18
  • 6.
    Effective Oracle HomeManagement #UKOUG_Tech18 6 The Large Hadron Collider (LHC) Largest machine in the world 27km, 6000+ superconducting magnets Emptiest place in the solar system High vacuum inside the magnets Hottest spot in the galaxy During Lead ion collisions create temperatures 100 000x hotter than the heart of the sun Fastest racetrack on Earth Protons circulate 11245 times/s (99.9999991% the speed of light)
  • 7.
    Quick overview ofthe New Release Model 05.12.2018 Effective Oracle Home Management #UKOUG_Tech18 7
  • 8.
    What does itchange? Jan 18 Apr Jul Oct Jan 19 Apr Jul Oct RU 18.1.0 18.2.0 18.3.0 18.4.0 18.5.0 18.6.0 18.7.0 18.8.0 RUR1 18.2.1 18.3.1 18.4.1 18.5.1 18.6.1 18.7.1 RUR2 18.2.2 18.3.2 18.4.2 18.5.2 18.6.2 RU 19.1.0 19.2.0 19.3.0 19.4.0 RUR1 19.2.1 19.3.1 RUR2 19.2.2 05.12.2018 Effective Oracle Home Management #UKOUG_Tech18 8 2
  • 9.
    What does itchange? Jan 18 Apr Jul Oct Jan 19 Apr Jul Oct RU 18.1.0 18.2.0 18.3.0 18.4.0 18.5.0 18.6.0 18.7.0 18.8.0 RUR1 18.2.1 18.3.1 18.4.1 18.5.1 18.6.1 18.7.1 RUR2 18.2.2 18.3.2 18.4.2 18.5.2 18.6.2 RU 19.1.0 19.2.0 19.3.0 19.4.0 RUR1 19.2.1 19.3.1 RUR2 19.2.2 05.12.2018 Effective Oracle Home Management #UKOUG_Tech18 9 2
  • 10.
    What does itchange? 05.12.2018 Effective Oracle Home Management #UKOUG_Tech18 10 Short Development cycle Frequent Upgrades And Updates
  • 11.
    “Patch soon, patchoften” vs reality 05.12.2018 Effective Oracle Home Management #UKOUG_Tech18 11
  • 12.
    Frequent patching? • Timeconsuming • Complex • Risky • Unnecessary (?) • Error-prone 05.12.2018 Effective Oracle Home Management #UKOUG_Tech18 12
  • 13.
    How many bugscould you encounter? 05.12.2018 Effective Oracle Home Management #UKOUG_Tech18 13
  • 14.
    How many bugscould you encounter? 05.12.2018 Effective Oracle Home Management #UKOUG_Tech18 14 18.3.0 18.3.1 18.4.0
  • 15.
    How many bugscould you encounter? 05.12.2018 Effective Oracle Home Management #UKOUG_Tech18 15 18.3.0 18.3.1 18.4.0 634
  • 16.
    How many bugscould you encounter? 05.12.2018 Effective Oracle Home Management #UKOUG_Tech18 16 18.3.0 18.3.1 18.4.0 634 1089
  • 17.
    How many bugscould you encounter? 05.12.2018 Effective Oracle Home Management #UKOUG_Tech18 17 18.3.0 18.3.1 18.4.0 634 1089 1257
  • 18.
    Which patches shouldI apply? 05.12.2018 Effective Oracle Home Management #UKOUG_Tech18 18
  • 19.
    Brave vs Fearful 12.1>= 12.2 Effective Oracle Home Management #UKOUG_Tech18 19 How do I feel? PSU Bundle Patch BraveFearful How do I feel? Release Update Revision Release Update BraveFearful • 05.12.2018
  • 20.
    Brave vs Fearful 12.1>= 12.2 Effective Oracle Home Management #UKOUG_Tech18 20 How do I feel? PSU Bundle Patch BraveFearful How do I feel? Release Update Revision Release Update BraveFearful Ludovico(*) Ludovico(*) (*) yes, it’s Comic Sans https://mikedietrichde.com/2017/10/24/differences-psu-bp-ru-rur/ • 05.12.2018
  • 21.
    Common patching patterns 05.12.2018 EffectiveOracle Home Management #UKOUG_Tech18 21
  • 22.
    Common worst practicesin maintaining homes • Installing software by hand • Different patch levels for Oracle Homes with the same path • Not tracking the installed patches • Oracle Home paths hard-coded in the operational scripts • Not minding about naming conventions 05.12.2018 Effective Oracle Home Management #UKOUG_Tech18 22
  • 23.
    Worst approach (bigbang) 1. Stop everything 2. In-place binaries patching 3. Database patching, all at once 4. Start everything Effective Oracle Home Management #UKOUG_Tech18 23 UP UP UP UP DOWN DOWN DOWN DOWN UP UP UP UP UP DOWN UP STOP ALL START ALLPATCH BINARIES PATCH DBs • 05.12.2018
  • 24.
    Worst approach (bigbang) 1. Stop everything 2. In-place binaries patching 3. Database patching, all at once 4. Start everything Effective Oracle Home Management #UKOUG_Tech18 2405.12.2018 UP UP UP UP DOWN DOWN DOWN DOWN UP UP UP UP UP DOWN UP STOP ALL START ALLPATCH BINARIES PATCH DBs No easy rollback Long downtime Complex planning
  • 25.
    Different approach (stillbad) 1. Install new binaries manually 2. Patch the new binaries 3. Stop, change OH, patch databases one by one 4. Decommission old binaries Effective Oracle Home Management #UKOUG_Tech18 2505.12.2018 UP UP UP UP DOWN DOWN DOWN DOWN UP UP UP UP PATCH BINARIES STOP, MOVE, PATCH ONE BY ONE
  • 26.
    Different approach (stillbad) 1. Install new binaries manually 2. Patch the new binaries 3. Stop, change OH, patch databases one by one 4. Decommission old binaries Effective Oracle Home Management #UKOUG_Tech18 2605.12.2018 UP UP UP UP DOWN DOWN DOWN DOWN UP UP UP UP PATCH BINARIES STOP, MOVE, PATCH ONE BY ONE Complex to do manually Repetitive task when many servers
  • 27.
    Automate the installation! •Automated creation 05.12.2018 Effective Oracle Home Management #UKOUG_Tech18 27 Soft ware Patch Patch Patched Software • runInstaller -silent • opatch apply
  • 28.
    Automate the installation! •Automated creation 05.12.2018 Effective Oracle Home Management #UKOUG_Tech18 28 Soft ware Patch Patch Patched Software • runInstaller -silent • opatch apply https://github.com/MarisElsins/getMOSPatch
  • 29.
    Automate the installation! •Automated creation 05.12.2018 Effective Oracle Home Management #UKOUG_Tech18 29 Soft ware Patch Patch Patched Software • runInstaller -silent • opatch apply https://github.com/MarisElsins/getMOSPatch https://github.com/ilmarkerm/ansible-oracle-home-mgmt
  • 30.
    Image repo Automate theinstallation! • Automated creation • Software cloning 05.12.2018 Effective Oracle Home Management #UKOUG_Tech18 30 Soft ware Patch Patch Patched Software • runInstaller -silent • opatch apply 18.3.1 18.4.0 Server 18.3.1
  • 31.
    Image repo Automate theinstallation! • Automated creation • Software cloning 05.12.2018 Effective Oracle Home Management #UKOUG_Tech18 31 Soft ware Patch Patch Patched Software • runInstaller -silent • opatch apply 18.3.1 18.4.0 Server 18.3.1 RUNDECK ANSIBLE RHP OTHER
  • 32.
    Less paths Effective OracleHome Management #UKOUG_Tech18 3205.12.2018 18.3.0 18.4.0 18.4.0 + one-off X 18.5.0 + one-off X 18.5.0 + one-off X + one-off Y 18.3.0 18.4.0 18.4.0 + one-off X 18.5.0 + one-off X 18.5.0 + one-off X + one-off Y 18.4.1 18.4.1 + one-off Y GOOD: BAD:
  • 33.
    Workarounds vs one-offs •Sometimes through underscore parameters • Mitigate the number of one-offs • Less merge requests • Document it! (why, when, who) 05.12.2018 Effective Oracle Home Management #UKOUG_Tech18 33 1 2 3 alter system set "_px_groupby_pushdown"=off comment='Ludo, 03.05.16: W/A for bug 18499088' scope=both sid='*‘;
  • 34.
    Rapid Home provisioning •It might be the perfect product… • Requires Lifecycle Management Pack • The technology stack is complex (Clusterware, ASM, ACFS, NFS, GNS…) 05.12.2018 Effective Oracle Home Management #UKOUG_Tech18 34
  • 35.
    Rapid Home provisioning •It might be the perfect product… • Requires Lifecycle Management Pack • The technology stack is complex (Clusterware, ASM, ACFS, NFS, GNS…) 05.12.2018 Effective Oracle Home Management #UKOUG_Tech18 35 Managing Gold Images for Easy Patching & Upgrading with Oracle RHP Ian Cookson - Oracle Room 1A, today 15:20
  • 36.
    Custom approach • Simplescript • Golden images • Solid naming conventions • I blogged about it! http://www.ludovicocaldara.net/dba/oh-mgmt-[1..7]/ 05.12.2018 Effective Oracle Home Management #UKOUG_Tech18 36
  • 37.
    Implementation at CERN 05.12.2018 EffectiveOracle Home Management #UKOUG_Tech18 37
  • 38.
    Naming Conventions 05.12.2018 Effective OracleHome Management #UKOUG_Tech18 38 Central Inventory server 1 <HOME NAME= "OraDB12Home1" LOC="/u01/app/oracle/product/12.2.0.1" TYPE="O" IDX="2"> <HOME NAME= "OraDB12Home2" LOC="/u01/app/oracle/product/12.2.0.1_2" TYPE="O" IDX=“3"> Central Inventory server 2 <HOME NAME= "OraDB12Home1" LOC="/u01/app/oracle/product/12.1.0.2" TYPE="O" IDX=“1"> <HOME NAME= "OraDB12Home2" LOC="/u01/app/oracle/product/12.2.0.1" TYPE="O" IDX="2">
  • 39.
    Naming Conventions • CentralInventory contains OH name • OH name matches OH location • No dots (.) allowed (inventory limitation) 05.12.2018 Effective Oracle Home Management #UKOUG_Tech18 39 Central Inventory <HOME NAME=“18_3_0" LOC="/u01/app/oracle/product/18_3_0" TYPE="O" IDX="2"> • 1 to 1 mapping: easy to code • 18c read-only OH uses the name as subdir in `orabasehome`: consistent naming!
  • 40.
    Golden images 05.12.2018 Effective OracleHome Management #UKOUG_Tech18 40 Golden Images Store 18_3_0.zip 18_4_0.zip 18_3_1.zip SERVER1 18_3_0.zip 18_3_1.zip SERVER2 18_3_1.zip SERVER3 18_4_0.zip Golden Image Metadata Name Path ------- ----------------- 18_3_0 /store/18_3_0.zip 18_3_1 /store/18_3_1.zip 18_4_0 /store/18_4_0.zip
  • 41.
    Image store Typical workflow 05.12.2018 EffectiveOracle Home Management #UKOUG_Tech18 42 SERVER1 18_3_0 SERVER2… First Install
  • 42.
    Image store Typical workflow 05.12.2018 EffectiveOracle Home Management #UKOUG_Tech18 43 18_3_0.zip SERVER1 18_3_0 SERVER2… $ ohctl -c -n 18_3_0
  • 43.
    Image store Typical workflow 05.12.2018 EffectiveOracle Home Management #UKOUG_Tech18 44 18_3_0.zip SERVER1 18_3_0 SERVER2… $ ohctl -i 18_3_018_3_0
  • 44.
    Image store Typical workflow 05.12.2018 EffectiveOracle Home Management #UKOUG_Tech18 45 18_3_0.zip SERVER1 18_3_0 SERVER2… $ ohctl -i 18_3_0 -n 18_4_0 18_3_0 18_4_0
  • 45.
    Image store Typical workflow 05.12.2018 EffectiveOracle Home Management #UKOUG_Tech18 46 18_3_0.zip SERVER1 18_3_0 SERVER2… Manual patching 18_3_0 18_4_0
  • 46.
    Image store Typical workflow 05.12.2018 EffectiveOracle Home Management #UKOUG_Tech18 47 18_3_0.zip SERVER1 18_3_0 SERVER2… $ ohctl -c 18_3_0 18_4_0 18_4_0.zip
  • 47.
    Image store Typical workflow 05.12.2018 EffectiveOracle Home Management #UKOUG_Tech18 48 18_3_0.zip SERVER1 18_3_0 SERVER2… $ ohctl -i 18_4_0 18_3_0 18_4_0 18_4_0.zip 18_4_0
  • 48.
    What it actuallydoes Creation workflow • Copy the Home • Clean it up • Create the zip • Store the zip in the repo • Store metadata Effective Oracle Home Management #UKOUG_Tech18 4905.12.2018
  • 49.
    What it actuallydoes Creation workflow • Copy the Home • Clean it up • Create the zip • Store the zip in the repo • Store metadata Installation workflow • Security checks • Create new path • Unzip from the store • Attach home • Relink (opt. with RAC) • Run setasmgid • Other stuff (e.g. TNS) Effective Oracle Home Management #UKOUG_Tech18 5005.12.2018
  • 50.
    Effective Oracle HomeManagement #UKOUG_Tech18 5105.12.2018 DEMO
  • 51.
    Further development atCERN 05.12.2018 Effective Oracle Home Management #UKOUG_Tech18 52
  • 52.
    Read-only Oracle Home $roohctl –enable Enabling Read-Only Oracle home. Update orabasetab file to enable Read-Only Oracle home. Orabasetab file has been updated successfully. Create bootstrap directories for Read-Only Oracle home. Bootstrap directories have been created successfully. Bootstrap files have been processed successfully. Read-Only Oracle home has been enabled successfully. Check the log file /u01/app/oracle/cfgtoollogs/roohctl/roohctl- 180822AM102911.log. 05.12.2018 Effective Oracle Home Management #UKOUG_Tech18 53
  • 53.
    Read-only Oracle Home 05.12.2018 EffectiveOracle Home Management #UKOUG_Tech18 54 18_3_0 18_4_0 /u01/app/oracle/product/ /u01/app/oracle/product/ $ cat $ORACLE_HOME/install/orabasetab #orabasetab file is used to track Oracle Home associated with Oracle Base /u01/app/oracle/product/18_3_0:/u01/app/oracle:18_3_0:N: ORACLE_HOME ORACLE_HOME Read only
  • 54.
    Read-only Oracle Home 05.12.2018 EffectiveOracle Home Management #UKOUG_Tech18 55 18_3_0 18_4_0 /u01/app/oracle/product/ /u01/app/oracle/product/ /u01/app/oracle/homes/ $ cat $ORACLE_HOME/install/orabasetab #orabasetab file is used to track Oracle Home associated with Oracle Base /u01/app/oracle/product/18_3_0:/u01/app/oracle:18_3_0:N: ORACLE_HOME /u01/app/oracle/homes/ ORACLE_HOME orabasehome orabasehome 18_3_0 18_4_0 Read only TNS, init, etc.
  • 55.
    AGILE INFRASTRUCTURE SERVER Insertion inthe Agile Infrastructure 05.12.2018 Effective Oracle Home Management #UKOUG_Tech18 56 Image Store Scripts REST client Oracle Home
  • 56.
    Grid Infrastructure Installation •Puppet + Rundeck for automatic GI install • Patched Grid Infrastructure versions beside the first one https://www.markusdba.net/?p=294 • Out-of-place patching with 18c independent local-mode automaton 05.12.2018 Effective Oracle Home Management #UKOUG_Tech18 57 $ rhpctl move gihome –sourcehome old_path -destinationhome new_path
  • 57.
    Independent Local-mode Automaton(18c) 05.12.2018 Effective Oracle Home Management #UKOUG_Tech18 58 # $GI_HOME/srvctl enable volume -volume GHCHKPT -diskgroup mgmt # $GI_HOME/srvctl enable filesystem -volume GHCHKPT -diskgroup mgmt # $GI_HOME/srvctl start filesystem -volume GHCHKPT -diskgroup mgmt $ $GI_HOME/srvctl start rhpserver Local Resources --------------------------------------------------------------- ora.MGMT.GHCHKPT.advm ora.mgmt.ghchkpt.acfs ora.helper Cluster Resources --------------------------------------------------------------- ora.rhpserver
  • 58.
    $ opatch lspatches 27908644;UPDATE18.3 DATABASE CLIENT JDK IN ORACLE HOME TO JDK8U171 27923415;OJVM RELEASE UPDATE: 18.3.0.0.180717 (27923415) 28256701;TOMCAT RELEASE UPDATE 18.3.0.0.0 (28256701) 28090564;DBWLM RELEASE UPDATE 18.3.0.0.0 (28090564) 28090557;ACFS RELEASE UPDATE 18.3.0.0.0 (28090557) 28090553;OCW RELEASE UPDATE 18.3.0.0.0 (28090553) 28090523;Database Release Update : 18.3.0.0.180717 (28090523) OPatch succeeded. 05.12.2018 Effective Oracle Home Management #UKOUG_Tech18 59
  • 59.
    Independent Local-mode Automaton(18c) • Damn-easy move of GI HOME • Automatic resource relocation • Enable Grid Infrastructure Golden Images • Patching (apply and rollback) is done automatically 05.12.2018 Effective Oracle Home Management #UKOUG_Tech18 60
  • 60.
    Back to thedemo… 05.12.2018 Effective Oracle Home Management #UKOUG_Tech18 61
  • 61.
  • 62.
    Thank you! Ludovico Caldara ComputingEngineer IT Department, CERN ludovico.caldara@cern.ch @ludodba 05.12.2018 Effective Oracle Home Management #UKOUG_Tech18 63