My presentation on a case study of 10g to 11g upgrade at LVOUG meetup #4 in 2012. Includes preserving execution plans by exporting them from 10g and importing as SQL Plan Baselines in 11gR2

1. Case Study
10gR2 to 11gR2
Maris Elsins
Oracle Applications DBA
28.06.2012

2. © 2012 Pythian
Few words about me
• 11y  Oracle
• 3y – PL/SQL Developer
• 8y – Oracle [Apps] DBA
• Certificates:
• 10g OCM, 9i/10g/11g OCP,
• 11i Apps DBA OCP, 11i System Administrator OCE
• Employed by Pythian since July 2011
• Speaker at conferences: 5* , 4* , 3*
• How to find me?
• Blog – http://www.pythian.com/news/author/elsins/
• Earlier blog posts – http://appsdbalife.wordpress.com/
• LinkedIn – http://lv.linkedin.com/in/mariselsins
• Twitter – @MarisElsins
• Email – elsins@pythian.com
2

3. © 2012 Pythian
AGENDA
• Starting Point
• The Goal
• Closer look at key things which ensurred the successful
upgrade
• Minimizing the downtime
• Performance management
• Repeatability of the upgrade process
3

4. © 2012 Pythian
Starting Point
• The business
• Gas Detection as a Service
• 24x7 web based application displaying alerts, trends, measurements
taken by gas detection devices
• 70 000 gas detection devices
• 3 500 worksites
• Data uploaded to the DB using docking stations
• 10.2.0.3 EE + Tuning Pack + Diagnostics Pack
• RHEL AS 4 (Update 7)
• ~800Gb
• HW doesn’t matter
4

5. © 2012 Pythian
The Goal
• Migration to new HW + Upgrade
• 11.2.0.3 EE + Tuning Pack + Diagnostics Pack
• OEL 5.8 + UEK
• VMWare
• 32G RAM
• (8 cores / 16 threads with HT) Xeon X7560@2.27GHz
• QUIZ! How many cores does a Xeon X7560 have?
• The Challenge
• We have 4 hours of downtime window
• Old HW -> New HW = ~27MB/s (800Gb = ~8h)
• upgrade itself takes ~1h
• Collecting fresh statistics requires ~3h
• Poor testing, how to guarantee at least the same performance?
• Fallback strategy and required time (why is this important?)
5

6. © 2012 Pythian
BRAINSTORM
How do we fit into the downtime window?
6

7. © 2012 Pythian
Chosen solution - Time
• «Manual» Standby
• Before the downtime
• Clone 10.2.0.3 software
• Preinstall 11.2.0.3 software
• R/O NFS mount source backup locations
• Use RMAN to duplicate the database (make sure it’s not open after
duplication)
• Establish custom archived log apply process
• During the downtime
• Complete the recovery and open the database (~20minutes)
• Perform the upgrade (~1 hour)
• Perform other tasks... (~30 minutes)
7

8. © 2012 Pythian
RMAN duplicate the database
• Clone 10.2.0.3 software
• Prepare Auxiliary instance parameter file and tnsnames.ora
• SQLNet connectivity to source DB
• Define [db|log]_file_name_convert parameters
• Use RMAN to duplicate the DB
• Make sure the archived logs are NOT available while duplicate is
running
8
connect target sys/***@PRD
connect auxiliary /
run
{
allocate auxiliary channel a1 type disk;
allocate auxiliary channel a2 type disk;
allocate auxiliary channel a3 type disk;
allocate auxiliary channel a4 type disk;
allocate auxiliary channel a5 type disk;
allocate auxiliary channel a6 type disk;
set until sequence 45458;
duplicate target database to "PRD";
}

9. © 2012 Pythian
Establish custom archived log apply process
9
export ORACLE_HOME=/u01/PRD/oracle/10.2.0
export PATH=$ORACLE_HOME/bin:$PATH
export ORACLE_SID=PRD
sqlplus -S "/ as sysdba" << EOF
spoo recstart.lst
alter database recover until cancel using backup controlfile;
spoo off
EOF
# ORA-00279: change 5975222350297 generated at 05/08/2012 19:30:06 needed for thread 1
# ORA-00289: suggestion : /u02/PRD/archive/PRD_45035_1_694087161.arc
# ORA-00280: change 5975222350297 for thread 1 is in sequence #45035
RECFILE=`grep suggestion recstart.lst | awk -F " : " '{print $2}' | xargs -n 1 basename`
FULL_RECFILE=`find /u01/oradata/PRD/archive -type f -name $RECFILE -mmin +5`
if [ "A${FULL_RECFILE}A" != "AA" ]
then
CMD="ALTER DATABASE RECOVER LOGFILE '${FULL_RECFILE}';"
find /u01/oradata/PRD/archive -name "PRD_*_1_694087161.arc" -newer $FULL_RECFILE -mmin +5 |
sort | sed "s,^,ALTER DATABASE RECOVER LOGFILE '," | sed "s,$,';,g" > dorecover_archlogs.sql
sqlplus -S "/ as sysdba" << EOF
set echo on
alter database recover until cancel using backup controlfile;
$CMD
@dorecover_archlogs.sql
EOF
fi

10. © 2012 Pythian
Complete the recovery and open the database
10
• Apply the remaining archived logs manually
• Transfer and apply the required redo logs manually
• v$log.archived='NO' order by SEQUENCE#
• open resetlogs
• alter tablespace temp add tempfile ...

11. © 2012 Pythian
BRAINSTORM
11
How do we ensure the performance
does not degrade after upgrade?

12. © 2012 Pythian
Chosen solution – Performance management
• «SQL Plan Management» to avoid degradation of Execution
plans
• Capture all execution plans into SQL Tuning Set (STS) in production
before QA testing (Tuning Pack needed)
• Duplicate PROD to QA DB
• Upgrade QA to 11.2.0.3
• Import execution plans from STS into SPM (create SQL baselines) in QA
• 2 iterations of testing – Evolve the baselines before the 2nd iteration of
testing.
• Upgrade PROD to 11.2.0.3
• Import all plans from QA SPM into PROD
• Statistics
• Collect statistics in QA
• Import statistics from QA into PROD
12

13. © 2012 Pythian
Capture all execution plans into SQL Tuning Set
13
SQL> BEGIN
DBMS_SQLTUNE.CREATE_SQLSET(
sqlset_name => 'ALL_PLANS_FOR_11G',
description => 'Plans to be imported in SQL Management Base after 11g Upgrade');
END;
/
PL/SQL procedure successfully completed.
SQL> BEGIN
DBMS_SQLTUNE.CAPTURE_CURSOR_CACHE_SQLSET (
sqlset_name => 'ALL_PLANS_FOR_11G',
time_limit => 86400, -- 24 hours
repeat_interval => 3600, -- collect plans every hour
capture_option => 'MERGE',
capture_mode => DBMS_SQLTUNE.MODE_REPLACE_OLD_STATS,
basic_filter => 'parsing_schema_name = ''PRD'' and executions>5',
sqlset_owner => 'SYS'
);
END;
/
PL/SQL procedure successfully completed.
Useless if bind variables are not used.

14. © 2012 Pythian
Import execution plans from STS into SPM
14
SQL> create table MEL_SQLSET_sqlid_bckup as select sql_id from WRH$_SQLTEXT
where sql_id in (select sql_id from WRI$_SQLSET_STATEMENTS) and dbid=475466233;
SQL> update WRH$_SQLTEXT set dbid=1675823245 where sql_id in (select sql_id from
MEL_SQLSET_sqlid_bckup) and dbid=475466233;
39013 rows updated.
SQL> commit;
Commit complete.
SQL> declare
2 n number:=0;
3 begin
4 n:=DBMS_SPM.LOAD_PLANS_FROM_SQLSET (
5 sqlset_name=>'ALL_PLANS_FOR_11G',
6 sqlset_owner=>'SYS', basic_filter=>NULL,
7 fixed=>'NO', enabled=>'YES', commit_rows=>1000);
8 dbms_output.put_line('PLANS LOADED: '||n);
9 end;
10 /
PLANS LOADED: 37859
PL/SQL procedure successfully completed.
SQL> update WRH$_SQLTEXT set dbid=475466233 where sql_id in (select sql_id from
MEL_SQLSET_sqlid_bckup) and dbid=1675823245;
39013 rows updated.
SQL> drop table MEL_SQLSET_sqlid_bckup;
Table dropped.
Statements in STS are bound to DBID
I changed the DBID as part of cloning
Some hacking was needed to make
this work

15. © 2012 Pythian
Evolve the baselines before the 2nd testing
15
SQL> select count(*) from dba_sql_plan_baselines where accepted='NO' and last_verified is null;
COUNT(*)
----------
42
SQL> var rep clob
SQL> set timing on
SQL> set pages 50000 lines 240
SQL> exec :rep := DBMS_SPM.evolve_sql_plan_baseline();
PL/SQL procedure successfully completed.
Elapsed: 00:00:12.15
SQL> set long 999999999
SQL> print :rep
...
-------------------------------------------------------------------------------
Report Summary
-------------------------------------------------------------------------------
Number of plans verified: 81
Number of plans accepted: 13

16. © 2012 Pythian
Import all plans from QA SPM into PROD
16
# in QA
SQL> exec DBMS_SPM.CREATE_STGTAB_BASELINE(table_name=>'SMB_UPG_11GR2',table_owner=>'SYSTEM');
SQL> var n number
SQL> exec :n :=DBMS_SPM.PACK_STGTAB_BASELINE(table_name=>'SMB_UPG_11GR2',table_owner=>'SYSTEM');
PL/SQL procedure successfully completed.
SQL> print n
638
SQL> ! exp file=/tmp/SMB_UPG_11GR2.dmp tables=SYSTEM.SMB_UPG_11GR2
# in PROD
SQL> ! imp file=/tmp/SMB_UPG_11GR2.dmp fromuser=SYSTEM touser=SYSTEM tables=SMB_UPG_11GR2
SQL> var n number
SQL> exec :n
:=DBMS_SPM.UNPACK_STGTAB_BASELINE(table_name=>'SMB_UPG_11GR2',table_owner=>'SYSTEM');
PL/SQL procedure successfully completed.
SQL> print n
638
SQL> select count(*) from DBA_SQL_PLAN_BASELINES;
638
SQL> drop table system.SMB_UPG_11GR2;

17. © 2012 Pythian
Import statistics into PROD after upgrade
17
# in QA
SQL> exec DBMS_STATS.CREATE_STAT_TABLE ('SYSTEM','UPG_STATS_11GR2','USERS');
SQL> exec DBMS_STATS.EXPORT_SYSTEM_STATS(stattab=>'UPG_STATS_11GR2',statown=>'SYSTEM');
SQL> exec DBMS_STATS.EXPORT_DICTIONARY_STATS(stattab=>'UPG_STATS_11GR2',statown=>'SYSTEM');
SQL> exec DBMS_STATS.EXPORT_FIXED_OBJECTS_STATS(stattab=>'UPG_STATS_11GR2',statown=>'SYSTEM');
SQL> exec DBMS_STATS.EXPORT_DATABASE_STATS(stattab=>'UPG_STATS_11GR2',statown=>'SYSTEM');
SQL> ! exp file=/tmp/11GR2_UPG_STATS.dmp tables=SYSTEM.UPG_STATS_11GR2
SQL> exec DBMS_STATS.DROP_STAT_TABLE(ownname=>'SYSTEM', stattab=>'UPG_STATS_11GR2');
# in PROD
$ imp file=/tmp/11GR2_UPG_STATS.dmp fromuser=SYSTEM touser=SYSTEM tables=UPG_STATS_11GR2
$ rm /tmp/11GR2_UPG_STATS.dmp
exec DBMS_STATS.IMPORT_SYSTEM_STATS(stattab=>'UPG_STATS_11GR2',statown=>'SYSTEM');
exec DBMS_STATS.IMPORT_DICTIONARY_STATS(stattab=>'UPG_STATS_11GR2',statown=>'SYSTEM');
exec DBMS_STATS.IMPORT_FIXED_OBJECTS_STATS(stattab=>'UPG_STATS_11GR2',statown=>'SYSTEM');
exec DBMS_STATS.IMPORT_DATABASE_STATS(stattab=>'UPG_STATS_11GR2',statown=>'SYSTEM');
exec DBMS_STATS.DROP_STAT_TABLE(ownname=>'SYSTEM', stattab=>'UPG_STATS_11GR2');

18. © 2012 Pythian
18
Repeatability of the upgrade process

19. © 2012 Pythian
Repeatability of the upgrade process
• Why repeatability is important
• Stress
• Limited time
• No space for errors
• Doing something different may impact the result.
• Take notes while you’re testing
• Commands executed
• Outputs seen
• Describe navigation through GUIs / take screenshots
• Problems and solutions
• Produce an «upgrade guide» to fallow during the PROD
upgrade
19

20. © 2012 Pythian
?
Tweet about the event: @MarisElsins and @lvoug
Ask more questions: elsins@pythian.com, @MarisElsins, LVOUG mailing list
Follow Pythian on twitter @pythian and LinkedIn http://linkd.in/pythian
20