What can and will go wrong when patching EBS 12.2.X and how to fix it!

1. Session ID:
Prepared by:
ADOP Patching Gotchas!
What can and will go wrong when patching EBS
12.2.X and how to fix it!
10175
Oti Ometie – Oracle Applications DBA
@OhOmetie

2. April 2-6, 2017 in Las Vegas, NV USA #C17LV
Agenda
• Introduction
• Overview of ADOP
• Patching Multiple Nodes
• Patching a single node in a multi-node configuration
• Troubleshoot and recover from a failure.
• Perform cleanup operations after a failed patching cycle
• Dropping Old Editions
• Impact of DR
• Extra stuff
• Summary
• Q&A
2

3. April 2-6, 2017 in Las Vegas, NV USA #C17LV
About me
• 17+ years of experience with Oracle products
• 15 years experience as an Oracle DBA
• Oracle Certified Expert, OCP 9i,10G,11G, EBS R12
• Worked as a DBA on 3 continents in a variety of
industries (Banking, Consulting, Construction,
Manufacturing)
• Martial artist and crazy about Obstacle Course Races –
Tough Mudder, Mud hero, Spartan…
3

4. April 2-6, 2017 in Las Vegas, NV USA #C17LV
About New Flyer Industries
• The largest transit bus and motor coach
manufacturer and parts distributor in North America
• Fabrication, manufacturing, distribution and service
centers across Canada and the United States
• Employs approximately 5,000 team members
• Multi Billion dollar revenue
• Completed a 9 month upgrade from 11i to 12.2.4 in
September 2015
4

5. Overview of ADOP

6. April 2-6, 2017 in Las Vegas, NV USA #C17LV
Overview of ADOP
• AD Online Patching is the ONLY way to apply patches in
EBS from Version 12.2
• Patching is carried out while users are online and the
application is in use.
• ADOP phases: prepare, apply, finalize, cutover, cleanup.
• Small downtime required for when the flip of the run and
patch file systems to occur
6

7. April 2-6, 2017 in Las Vegas, NV USA #C17LV
ADOP Patching Cycle
7

8. April 2-6, 2017 in Las Vegas, NV USA #C17LV
Simple Commands
adop phase=prepare
adop phase=apply patches=123456
adop phase=finalize
adop phase=cutover
adop phase=cleanup
8

9. April 2-6, 2017 in Las Vegas, NV USA #C17LV
More commands
adop phase=prepare
adop phase=apply workers=24 patches=123456, 98765
adop phase=finalize
adop phase=cutover cm_wait=2
adop phase=cleanup
9

10. April 2-6, 2017 in Las Vegas, NV USA #C17LV
Still more …
adop phase=prepare
adop phase=apply workers=24 patches=123456, 98765
adop phase=finalize,cutover cm_wait=1
adop phase=cleanup
10

11. Patching Multiple Nodes

12. April 2-6, 2017 in Las Vegas, NV USA #C17LV
Patching Multiple Nodes
• Ensure that the patch(es) is/are available on the
$PATCH_TOP of all nodes.
• Run the commands you would run if you were patching a
single node. The commands will be run sequentially on
all nodes
• Patches are applied on the primary node before being
applied on other nodes.
12

13. April 2-6, 2017 in Las Vegas, NV USA #C17LV
Patching Multiple Nodes
• Patching is non interactive i.e. you need to specify the
patch(es)
Which of the following is right?
• adop phase=apply patches=123456
• adop phase=apply
13

14. Patching a single node in a
multi-node configuration

15. April 2-6, 2017 in Las Vegas, NV USA #C17LV
Patching a single node in a multi-node
configuration
• Run adop phase=prepare on the primary node
• Apply the patch to the node using:
• PRIMARY NODE- adop phase=apply patch=1234 allnodes=no
action=db
• SECONDARY NODES- adop phase=apply patch=1234
allnodes=no action=nodb
• Continue as usual – finalize, cutover
15

16. April 2-6, 2017 in Las Vegas, NV USA #C17LV
Patching a single node in a multi-node
configuration
• Commands I’ve tried out and work for different
situations
adop phase=prepare allnodes=no action=db
adop phase=cutover allnodes=no action=db cm_wait=1
16

17. April 2-6, 2017 in Las Vegas, NV USA #C17LV
• You cannot run a prepare phase on a non-admin node.
adop phase=prepare allnodes=no action=nodb
The above command will fail with the error below:
Patching a single node in a multi-node
configuration
17

18. Troubleshoot and recover from
a failure

19. April 2-6, 2017 in Las Vegas, NV USA #C17LV
Troubleshooting the error
adop –status –detail
adop phase=apply patches=xyz analytics=yes
19

20. April 2-6, 2017 in Las Vegas, NV USA #C17LV
Recover from a failure
• Examples
adop phase=apply patches=123456 abandon=no restart=yes
flags=autoskip
adop phase=apply patches=21473943 allnodes=no action=nodb
abandon=no restart=yes flags=autoskip
20

21. April 2-6, 2017 in Las Vegas, NV USA #C17LV
If all else fails…
• ABORT
• An abort is always my last resort, it means pulling the
plug on all the patches that have been applied.
• Command:
• adop phase=abort
21

22. April 2-6, 2017 in Las Vegas, NV USA #C17LV
Aborting a patch cycle
22

23. April 2-6, 2017 in Las Vegas, NV USA #C17LV
Aborting a patch cycle
• The abort command terminates the patching cycle.
• All actions being performed are discarded.
• After an abort is performed a cleanup is required and will
be performed during the next adop prepare phase if not
initiated by the DBA.
• Command:
• adop phase=abort, cleanup
• adop phase=abort,cleanup cleanup_mode=full
• adop phase=fs_clone
23

24. Perform cleanup operations
after a failed patching cycle

25. April 2-6, 2017 in Las Vegas, NV USA #C17LV
Perform cleanup operations after a failed
patching cycle
• A cleanup is required after every failed patching session.
• Failure to explicitly run the cleanup command means
that your next patching cycle will be delayed while the
prepare phase kicks off a cleanup
• Commands:
• adop phase=cleanup
• adop phase=cleanup cleanup_mode=full
25

26. Dropping Old Editions

27. April 2-6, 2017 in Las Vegas, NV USA #C17LV
Dropping Old editions
• After each online patching cycle completes, the database
stores the previous edition.
• After a certain point this begins to impact performance
• Oracle recommends that when the number of editions
reaches 25 it’s time to purge/drop old editions
27

28. April 2-6, 2017 in Las Vegas, NV USA #C17LV
Steps to Drop Old Editions
• Commands:
adop phase=prepare
adop phase=actualize_all
adop phase=finalize finalize_mode=full
adop phase=cutover
adop phase=cleanup cleanup_mode=full
28

29. Impact of DR – Dataguard
Standby Databases

30. April 2-6, 2017 in Las Vegas, NV USA #C17LV
Impact of DR – Dataguard Standby Databases
[WARNING] adop has detected a configured disaster recovery site.
[WARNING] Follow the instructions in the section "Oracle E-Business Suite
[WARNING] Maintenance with Standby Database" of Business Continuity for
[WARNING] Oracle E-Business Suite Release 12.2 depending on the database version
used.
Do you want to continue with the prepare phase [Y/N] ?
Note: The error above has been fixed see Doc ID 2131833.1
30

31. April 2-6, 2017 in Las Vegas, NV USA #C17LV
Impact of DR – Dataguard Standby Databases
• If the standby database is available while a patch is being
applied, the changes on the primary will be automatically
propagated to the standby.
31

32. April 2-6, 2017 in Las Vegas, NV USA #C17LV
Impact of DR – Dataguard Standby Databases
• Doc ID 1491855.1: Section 7 goes into details
• If the standby is running during a patch application, changes
will be automatically replicated on the standby.
• Oracle recommends that before cutover:
1. Stop recovery on the standby database
2. Stopping the shipping of redo on the Primary Database
SQL>alter database recover managed standby database cancel;
SQL>alter system set log_archive_dest_state_<n>=defer;
32

33. Extra stuff

34. April 2-6, 2017 in Las Vegas, NV USA #C17LV
Patch Wizard
34

35. April 2-6, 2017 in Las Vegas, NV USA #C17LV
Abandoning a node
• When a node in a multi-node architecture is inaccessible
during an online patching cycle, it will be abandoned.
• If an adop patch fails on a secondary node. if you choose
to proceed with the cutover, node will be marked as
abandoned.
Do you want adop to continue with other completed nodes [y/n]
• Select “n”, resolve the issue and run something similar
to:
adop phase=apply patches=123456 abandon=no restart=yes
flags=autoskip allnodes=no action=nodb
35

36. April 2-6, 2017 in Las Vegas, NV USA #C17LV
Examples
• If you need help and are unable to access online
resources use examples.
adop -examples
36

37. April 2-6, 2017 in Las Vegas, NV USA #C17LV
Summary
• AD Online Patching is the ONLY way to apply patches in
EBS from Version 12.2
• Patching is carried out while users are online and the
application is in use.
• When patching multiple nodes ensure that the patch is
copied to the $PATCH_TOP of all nodes.
• Patching multiple nodes is non-interactive
• You indicate you want to patch a single node using the
“allnodes=no” argument and specify if it’s the primary or
secondary node using action=db/nodb
37

38. April 2-6, 2017 in Las Vegas, NV USA #C17LV
Summary cont.
• You can troubleshoot by using the analytics=yes
argument and –status –details argument
• Use the autoskip argument to recover from pesky errors
and continue with the session on the node with the error
using the allnodes=no argument.
• If all else fails – ABORT
• Running a cleanup before your next patch cycle saves
time
• Drop old editions to improve performance – 25 is a good
number
• Shutdown/defer Dataguard standby before cutover
38

39. April 2-6, 2017 in Las Vegas, NV USA #C17LV
References
• Oracle E-Business Suite Maintenance Guide Release 12.2
– Patching procedures
• Business Continuity for Oracle E-Business Suite Release
12.2 Using Oracle 11g (11gR2) Physical Standby Database
(Doc ID 1491855.1)
• Steven Chan’s blog:
https://blogs.oracle.com/stevenChan/entry/op_faq2
39

41. Please Complete Your
Session Evaluation
Evaluate this session in your COLLABORATE app.
Pull up this session and tap "Session Evaluation"
to complete the survey.
Session ID:
Oti Ometie
LinkedIn: linkedin.com/in/otiometie
Email: ometie@gmail.com
10175