How to Upgrade Hundreds or Thousands of Databases

How to Upgrade Hundreds or
Thousands of Databases
In a Reasonable Amount of Time
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Roy Swonger
Senior Director & Product Manager
Database Upgrade & Utilities
Oracle Corporation
Mike Dietrich
Master Product Manager
Database Upgrade
Oracle Corporation

Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.
How to Upgrade Hundreds or Thousands of Databases 2

$> whoami
3
20 years w/Oracle
Previously with DEC Rdb
11+ years managing
Data Pump
Database Upgrade
SQL*Loader
Transportable Tablespaces
How to Upgrade Hundreds or Thousands of Databases
Roy Swonger

6 years
RDBMS Core & Mission Critical Support
5.5 years
Technology Presales for DataGuard, Upgrades
>7.5 years
ST Upgrade Development Team
50%
Reference
Projects
50%
Workshops
Worldwide
+ x%
Development
Work
$> whoami
4How to Upgrade Hundreds or Thousands of Databases
Mike Dietrich
Master Product Manager
Database Upgrades & Migrations
MikeDietrichDE
http://blogs.oracle.com/UPGRADE

Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
Download Slides from the Database Upgrade Blog
 http://blogs.oracle.com/UPGRADE

Supported Versions
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
Oracle 10.1
(GA: Jan 2004)
Oracle 10.2
(GA: Jul 2005)
Oracle 11.1
(GA: Aug 2007)
Oracle 11.2
(GA: Sep 2009)
Oracle 12.1
(GA: Jun 2013)
Oracle 12.2
(GA: xxx 2016)
today
Premier Support Waived Extended Support Paid Extended Support Sustaining Support
Jan 2009 Jan 2012
Jul 2010 Jul 2013
Aug 2012 Aug 2015
Jan 2015
Jun 2018 Jun 2021
Dec 2020May 2017

We fully understand that things often are
complicated and business critical

And things can become really complicated ...
 Oracle 8i to 11.2
 26 TB
 Oracle EBS
 5-6 hours max downtime
Good luck!!!

Or more work ...
 Triple hops
↘ Oracle 9.2.0.1
↘ Oracle 9.2.0.4
↘ Oracle 10.2.0.5
↘ Oracle 12.1.0.2

But even if it does not look complicated at first sight ...
0
50
100
150
200
250
300
350
400
10.2.0.3.0 10.2.0.4.0 10.2.0.5.0 11.2.0.1.0 11.2.0.2.0 11.2.0.3.0 11.2.0.4.0
2 21 9 10
113
292
382
Databases
54%
(447 out of 829 databases)
have no bug fixing support anymore

Another example ...
0%4%
5%
2%
3%
17%
35%
32%
2%
Databases
9.2.0.7.0
10.2.0.3.0
10.2.0.4.0
10.2.0.5.0
11.2.0.1.0
11.2.0.2.0
11.2.0.3.0
11.2.0.4.0
12.1.0.2.0
828

Don't maintain a zoo!
 135 Oracle databases
– 7 different Oracle releases
– 13 different Oracle patch levels
 6 different OS vendors
– 15 different OS versions
2% of all databases
under bug fixing support

Recommendation
 Don't sit it out
– It will just get more complicated and risky
 Beware of application dependencies
 Establish constant database inventory monitoring

Where should you start?

Project Approach?
Planning
Data
Gathering
Testing
Upgrade/
Migration
Go-Live
Quarterly
Patching

You need a plan
Week T-10 Migration Initiation, & Plan
Baseline & Migration
Migration & Unit test initiation
Unit Test Complete
Application & Performance Test Start
Application & Performance Tests
Application & Performance Tests
Application Testing Complete and Application Freeze
Week T- 9
Week T-8
Week T-7
Week T- 6
Week T-5
Week T-4
Week T-3
Week T-2
Week T-1
Cutover Plan & Tests
Migrate Go/No go
Migration Weekend

You need to know Which / Where / Who / When
 Database inventory
 OS inventory
 Depending applications, certification
 Responsibilities
 Desired time frame

Build a tracking app – Use APEX

Build Workstreams and Database Groups
Estate
Analysis
Workstream 1
Workstream 2
Workstream 3
Upgrade/Migration Group 1
Post Migration
Monitoring,
Troubleshooting,
and Tuning Team
Production
Production with
additional monitoring

General Test Guidelines
Upgrade/Migration Process
Post-Upgrade Functionality
Post-Upgrade Performance
ProdLoad

Testing Effort
 Relation between test completeness and costs
Costslow high
Completeness0%100%

Use the Right Tools: AWR Diff Reports
AWR Diff
Report
Migrate
R2
10.2.0.4 12.1.0.2
Snapshot Interval Snapshot Interval
IBM AIX SOLARIS
Compare

Use the Right Tools: Database Replay
Database Replay
Migrate
R2
10.2.0.4 12.1.0.2
Capture Workload Replay Workload
Compare – then restore
and run again with changes
IBM AIX SOLARIS
Preprocess Workload

Use the Right Tools: SQL Performance Analyzer
SQL Performance Analyzer
STS
Capture SQLs
into a STS
Load STS
STS
Evaluate, compare and tune –
over and over again!
Migrate
R2
10.2.0.4 12.1.0.2
IBM AIX SOLARIS

Use the Right Tools: SPA Resulting Reports
 Example: Report showing regressed statements and plans

Now RELAX ... Stay Calm ...
And open an SR with Oracle Support in case of real trouble
Picture taken from: https://aycw.wordpress.com/2015/06/10/young-peoples-attitudes-toward-work/

And don't forget:
The next patching day is right in front of you!
Oracle PSUs get delivered on a quarterly basis:
20-OCT-2015 / 19-Jan-2016 / 19-Apr-2016 / 19-Jul-2016

Use Oracle EM Lifecycle Management Pack

With a little help from ... Real Application Testing
Upgrade over 300 databases at CSX

Real World Checkpoint
 CSX
– One of the nation‘s leading transportation suppliers
 Encompasses 21,000 miles of tracks
in 23 states, from Florida to Ontario
 HQ in Jacksonville, Florida
Customer
Project
Constraints
Preparation
Upgrade
Success?
Remarks

 Upgrade and migration of 304 databases to 11.2.0.2 over a
24 month period (2011-2012)
– 1/3 production, 2/3 test and development systems
– Source versions: Oracle 10.2.0.4 and a few Oracle 8i/9i
– Single Instance on IBM AIX  RAC on Linux
 Plan to use Real Application Testing
– Guarantee quality and mitigate the risk of upgrading
 Define standards
– Install 11.2.0.2 and patches and apply latest PSU
Customer
Project
Constraints
Preparation
Upgrade
Success?
Remarks

 Limited downtime on some business critical systems
 Testing cycles vary from 1-6 months depending on
complexity and criticality
 Tight on available testing resources
– Time
– People
– Systems
 New to Real Application Testing
Customer
Project
Constraints
Preparation
Upgrade
Success?
Remarks

 Close alliance with Oracle's Upgrade Development Group and
Oracle's RAT Pack
– CSX agreed to become an official
Oracle Database 11.2 Reference Customer
 Monthly status meetings
 Onsite training
 Live upgrade and RAT demo
– Cloning databases methods for use with RAT
Customer
Project
Constraints
Preparation
Upgrade
Success?
Remarks

 Upgrade and migration methods used:
 Policy:
– Install Oracle SW with latest patches and PSU
– Combine HW refresh and OS upgrades
 Internal application to track owner, status, success
80%
20% Data Pump for OS Migration
DBUA for Upgrades
Customer
Project
Constraints
Preparation
Upgrade
Success?
Remarks

 Yes –project took 2 years
– 88% of all databases upgraded within 18 months
– Some systems took longer due to business and/or resource
constraints
 Performance comparison mechanisms are very helpful
 Regressions found and fixed before upgrade
 RAT helped a lot to predict workload performance and
ensure careful testing
Customer
Project
Constraints
Preparation
Upgrade
Success?
Remarks

“The Real Application Testing tool provided a comprehensive
and flexible solution for assessing the impact of the Oracle
11g database upgrade into CSX systems. At CSX we were
able to capture real production workloads, replay it in the
11g environment, identify poor performing queries and,
fine tune these queries in a test environment before the
production implementation.“
Maritza Gonzalez, Technical Director
CSX Corporation
Customer
Project
Constraints
Preparation
Upgrade
Success?
Remarks

 More Information?
https://blogs.oracle.com/UPGRADE/entry/csx_corporation_upgrades_databases_2x
Customer
Project
Constraints
Preparation
Upgrade
Success?
Remarks

A small team and an excellent plan
Upgrade 300 databases at Mobiliar Insurance, Switzerland

 Swiss Mobiliar
– Switzerland's most personal insurer
– Founded 1826 in Bern, oldest Swiss insurance
– Legal form:
 Cooperative association (mutual company)
– Over 1.7 million persons and firms insured
– Switzerland's no.1 insurer for:
 Businesses, term life insurances, households
– 80 general agencies at 160 locations
– Over 4,400 employees and 325 trainees
– Awarded Most Trusted Insurance Brand in
Switzerland for 13 consecutive years
39
Customer
Project
Constraints
Preparation
Upgrade
Success?
Remarks

 Upgrade 289 databases
– Oracle Database 11.2.0.3 to Oracle Database 12.1.0.2
– 80 production databases
 Oracle Cloud Control
 AWR Warehouse (home-made)
 Shell scripts
 Oracle Restart with ASM
 Oracle In Memory
 OID
40
Customer
Project
Constraints
Preparation
Upgrade
Success?
Remarks

 Project timeline: 24 months
– Goal: 18 months
41
Customer
Project
Constraints
Preparation
Upgrade
Success?
Remarks
0
5
10
15
20
25
30
35
40
45
Jan-15 Feb-15 Mrz-15 Apr-15 Mai-15 Jun-15 Jul-15 Aug-15 Sep-15 Okt-15 Nov-15 Dez-15
Migrations per month

 Small DBA Team
– Only 3.8 FTE + 1 newbie
 2 major software releases of Mobiliar Applications per year
resulting in a full month of code freeze
 Bug fixing support for Oracle 11.2.0.3 ended Aug 27, 2015
 Highly complex applications
– Queries with more than 1000 bind variables and 250 outer joins
 One core team member broke his leg in
September 2015 and did not return to work yet
42
Customer
Project
Constraints
Preparation
Upgrade
Success?
Remarks

 All queries, execution time and execution paths are tracked on a self
made AWR Performance DWH
 Possibility to compare performance before and after the upgrade on
query level
 Performance tests performed by application owner
 Regression tests done during the testing phase of the Mobi Software
Release
– PSU RDBMS 12.1.0.2.4, PSU GRID 12.1.0.2.2
– Two Oracle Homes on the same server
– Upgrade with catctl.pl embedded into home-built shell script
43
Customer
Project
Constraints
Preparation
Upgrade
Success?
Remarks

 Project Monitoring with Excel Sheet
44
Customer
Project
Constraints
Preparation
Upgrade
Success?
Remarks
0
50
100
150
200
250
300
350
DBs 12c 0 7 10 18 26 62 105 126 136 152 157 170
DBs 11g 289 282 279 271 263 227 184 163 153 137 132 119
Oracle Database Swiss Mobiliar
DBs 11g DBs 12c

 Automated upgrade via home-built scripts:
45
Customer
Project
Constraints
Preparation
Upgrade
Success?
Remarks

 Live? And alive?
– Yes! First system went live on Feb 4, 2015
– Current status (Oct 12, 2015):
 152 out of 289 databases live on Oracle Database 12.1.0.2
 53% upgraded already
 Project perfectly on schedule
46
Customer
Project
Constraints
Preparation
Upgrade
Success?
Remarks
137
152
No. Databases
Oracle. 11.2.0.3 Oracle 12.1.0.2

 No major issues after go live
– Issues caught during testing, e.g.:
– Bug 20776435: PARSE OF INMEMORY QUERY IS SPINNING ON CPU
 Query with 250 Left Joins
 Workaround:
alter session set "_optimizer_inmemory_table_expansion"=false;
– Bug 19677469: NO QUERY REWRITE
 Workaround:
CREATE MATERIALIZED VIEW …. SELECT /*+ MV_MERGE */ …
or:
alter system set "_fix_control"='10145667:OFF';
47
Customer
Project
Constraints
Preparation
Upgrade
Success?
Remarks

 Oracle In-Memory boosts performance
– Read on: http://tinyurl.com/Mobi12c
48
Customer
Project
Constraints
Preparation
Upgrade
Success?
Remarks

"The entire upgrade project of our 300 Oracle databases at
Die Mobiliar is running very well.
When we catch issues, we fix them before going live.
And features such as Oracle In-Memory add a lot of
performance boost to some of our applications.
Especially the Oracle Upgrade Reference Program was a
great help and added a lot of value to our project's progress.
We are very happy and satisfied with the upgrade process
and the reliability and performance of Oracle Database 12c".
Paolo Kreth, Group Manager Databases, Die Mobiliar
49
Customer
Project
Constraints
Preparation
Upgrade
Success?
Remarks

Famous Last Words
Conclusion

Conclusion
 If you have a complicated environment, you are not alone!
 But, you can do it:
– Plan your effort
– Use the right tools
– Track and
report progress

Download Slides from the Database Upgrade Blog
 http://blogs.oracle.com/UPGRADE

Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential – Internal 53
Classroom Training
Learning Subscription
Live Virtual Class
Training On Demand
Keep Learning with Oracle University
education.oracle.com
Cloud
Technology
Applications
Industries

How to Upgrade Hundreds or Thousands of Databases

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to How to Upgrade Hundreds or Thousands of Databases

Similar to How to Upgrade Hundreds or Thousands of Databases (20)

More from Guatemala User Group

More from Guatemala User Group (12)

Recently uploaded

Recently uploaded (20)

How to Upgrade Hundreds or Thousands of Databases