Oracle Database 19c Upgrade Virtual Classroom Series By Mike and Roy: Webinar 3

1. Upgrade to Oracle Database 19c
Performance Stability, Tips and Tricks and Underscores
Mike Dietrich
Master Product Manager
Database Upgrade and Migration
Roy Swonger
Vice President
Database Upgrade, Utilities and Patching
Virtual Classroom Series

2. Vice President
Database Upgrade,
Utilities & Patching
Roy F. Swonger
@RoyFSwonger

3. Master Product Manager
Database Upgrade
and Migrations
Mike Dietrich
https://MikeDietrichDE.com
@MikeDietrichDE
mikedietrich

4. Slides | https://MikeDietrichDE.com

5. Content Overview
Migration to
Multitenant
4
Ensure
Performance Stability
3
Upgrade to Oracle
Database 19c
2Release and
Patching Strategy
1

6. Your key to successful Database Upgrades
1 2 3 4
Download and install
Oracle 19c
Download and install
newest RU
Download and use
newest AutoUpgrade
eDelivery MOS: 2521164.1 MOS: 2485457.1
SPM, STA, RAT for Performance
Stability

7. Upgrade 19c | Speed it up
• Check when dictionary stats have been gathered the last time
• Refresh stats a day before the upgrade
column OPERATION format a40
set linesize 200
select to_char(max(END_TIME),'DD-MON-YY hh24:mi') LATEST, OPERATION
from DBA_OPTSTAT_OPERATIONS
where OPERATION in ('gather_dictionary_stats', 'gather_fixed_objects_stats')
group by operation;

8. Upgrade 19c | Make it run faster …
• Stats not gathered on several clustered indexes (bug 25286819)
• But wait … autoupgrade.jar does this for you already!
exec dbms_stats.gather_schema_stats('SYS');
exec dbms_stats.gather_index_stats('SYS','I_OBJ#');
exec dbms_stats.gather_index_stats('SYS','I_FILE#_BLOCK#');
exec dbms_stats.gather_index_stats('SYS','I_TS#');
exec dbms_stats.gather_index_stats('SYS','I_USER#');
exec dbms_stats.gather_index_stats('SYS','I_TOID_VERSION#');
exec dbms_stats.gather_index_stats('SYS','I_MLOG#');
exec dbms_stats.gather_index_stats('SYS','I_RG#');

9. Upgrade 19c | Effect of having good stats
• The larger the dictionary, the bigger the effect
• This example has been done with one of the tiny Hands-On Lab databases
Duration Reduction
No dictionary and fixed objects stats 15 min 55 sec
Gathered dictionary and fixed objects stats 14 min 10 sec 11 %
Gathered schema and cluster index stats 13 min 41 sec 3.4 % to previous
Total downtime saved 2 min 14 sec 14 % overall

10. Effect of Having Good Statistics | EBS
• Upgrade duration for Oracle E-Business Suite
Duration Reduction
No dictionary and fixed objects stats 10 hrs 56 min 52 sec
Gathered dictionary and fixed objects stats 52 min 42 sec 93 %
Gathered schema and cluster index stats 52 min 25 sec 0.5 % to previous
Total downtime saved 10 hrs 4 min 14 sec 93.5 % overall

11. Effect of Having Good Statistics
• Stale/no stats: 9:59:23.87 • Good stats: 0:00:02.33
ID OPERATION OPTIONS OBJECT_NAME
----- -------------------- -------------------- -----------
0 UPDATE STATEMENT
1 UPDATE DEPENDENCY$
2 FILTER
3 TABLE ACCESS FULL DEPENDENCY$
4 INDEX FULL SCAN I_OBJ2
5 INDEX FULL SCAN I_OBJ2
6 TABLE ACCESS BY INDEX ROWID BATCHED OBJ$
7 INDEX RANGE SCAN I_OBJ1
8 TABLE ACCESS BY INDEX ROWID BATCH OBJ$
9 INDEX RANGE SCAN I_OBJ1
ID OPERATION OPTIONS OBJECT_NAME
----- -------------------- -------------------- --------------
0 UPDATE STATEMENT
1 UPDATE DEPENDENCY$
2 FILTER
3 TABLE ACCESS FULL DEPENDENCY$
4 INDEX RANGE SCAN I_OBJ1
5 INDEX RANGE SCAN I_OBJ1
6 TABLE ACCESS BY INDEX ROWID BATCHED OBJ$
7 INDEX RANGE SCAN I_OBJ1
8 TABLE ACCESS BY INDEX ROWID BATCH OBJ$
9 INDEX RANGE SCAN I_OBJ1

12. AutoUpgrade
The recommended way
to upgrade databases
PhotobyJoshuaSortinoonUnsplash

13. Download | Get the latest autoupgrade.jar
MOS Note: 2485457.1

14. Upgrade | AutoUpgrade
upg>
Job Console
11.2.0.4
12.1.0.2
12.2.0.1
18c
autoupgrade.jar
MOS Note: 2485457.1
2MB
config.cfg
autoupgrade.jar
12.2.0.1
18.5.0
19c
19c

15. Performance Best Practices
A prescription to ensure
performance stability

16. Parameter Recommendations
General guidelines:
• The fewer parameters you have in your spfile, the better
• Remove outdated parameters, underscores, events ...
• Unless directed by applications (e.g. E-Business Suite, Siebel, SAP etc.) to set them
• For EBS: Interoperability notes

17. Things to do right after upgrade
Configure statistics history retention period
• Check space usage:
• Check retention
• Default: 31 days
• Adjust setting
• Example: 10 days
SQL> select SPACE_USAGE_KBYTES/1024 MB from
V$SYSAUX_OCCUPANTS where
OCCUPANT_NAME='SM/OPTSTAT';
SQL> select
DBMS_STATS.GET_STATS_HISTORY_RETENTION from DUAL;
SQL> exec
DBMS_STATS.ALTER_STATS_HISTORY_RETENTION(10);

18. Help me - I have an upgrade problem …
In >95%of all cases
PhotobynikkomacaspaconUnsplash
“ ”
an "upgrade problem" in reality is a performance issue
after upgrade, or not a database upgrade topic.
There is exactly one way to mitigate the risk:
TESTING!

19. The entire upgrade project of our 300 Oracle
databases at La Mobilier is running very well.
When we catch issues, we fix them before
going live.
Paolo Kreth
Group Manager Databases
La Mobiliere
“
”

20. Testing | Typical Mistakes
• Only 10% of real data used
• Data sets artificially created
• Tests done on a laptop
• No testing tools used
• Data from 6 months ago
• No statistics refreshed
• Testing?? Waste of time!
• Real experts fix it after go-live …
PhotobyElijahO'DonnellonUnsplash

21. Best Practice | COMPATIBLE vs Optimizer
• COMPATIBLE and OPTIMIZER_FEATURES_ENABLE
• Fully independent from each other
• Set COMPATIBLE to the default of the release, e.g. 19.0.0
• Change OPTIMIZER_FEATURES_ENABLE only when you have to
• Avoid it if possible
• This is not a Swiss Army knife!
• You will turn off a lot of great features

22. Testing | Completeness versus Costs
Costs
Completeness
low high
0%100%
80% - 90%
Costslow high
Completeness
0%100%

23. Testing Tools | Workflow
Collect execution
plans before upgrade
Compare AWR
Snapshots
Verify them with
SQL Performance
Analyzer
Regressed plans?
SQL Plan
Management
SQL Tuning Advisor
Verify functionality
and performance with
Database Replay

24. Testing Tools | Collect Plans on Production
STS

Capturing from cursor cache and AWR
into an STS does not require an extra
license for EE anymore:
Find it under: "SQL Tuning Sets" at:
https://docs.oracle.com/database/121/DBLIC/editions.htm#DBLIC109

25. Testing Tools | Comparison
Collect execution
plans before upgrade
Compare AWR
Snapshots
Verify them with
SQL Performance
Analyzer
Regressed plans?
SQL Plan
Management
SQL Tuning Advisor
Verify functionality
and performance with
Database Replay

26. Testing Tools | AWR Diff Reports
Upgrade / Migrate
11.2.0.4 19.7.0
Snapshot Interval
Compare
AWR Diff Report: awrddrpt.sql Snapshot Interval

27. Testing Tools | SQL Performance Analyzer
Collect execution
plans before upgrade
Compare AWR
Snapshots
Verify them with
SQL Performance
Analyzer
Regressed plans?
SQL Plan
Management
SQL Tuning Advisor
Verify functionality
and performance with
Database Replay

28. Testing Tools | SQL Performance Analyzer
SQL Performance Analyzer
STS
Capture SQLs
into a STS
Load STS
STS
Evaluate, compare and tune –
over and over again!
11.2.0.4 19.7.0
Upgrade / Migrate

29. Testing Tools | SPA - Regressed Report

30. Testing Tools | SQL Plan Management
Collect execution
plans before upgrade
Compare AWR
Snapshots
Verify them with
SQL Performance
Analyzer
Regressed plans?
SQL Plan
Management
Verify functionality
and performance with
Database Replay

31. Testing Tools | Situation after change
• Challenging to "freeze" execution plans
Parse Execute
Plan acceptable
Plan possibly
not acceptable
Conditions change:
New statistics, parameter change, upgrade


GB
HJ
...HJ
GB
NL
...NL
SQL
SQL
ExecuteParse

32. SQL Plan Management | Mechanism
• Phase 1 – Baseline Capture
• Set OPTIMIZER_CAPTURE_SQL_PLAN_BASELINES=TRUE
• If OPTIMIZER_CAPTURE_SQL_PLAN_BASELINES=TRUE then all found plans will be put into the baseline
Parse Execute
SQL
GB
HJ
...HJ
SQL Plan acceptable

2X
GB
HJ
...HJ
Plan Baseline
SQL MANAGEMENT BASE
Residing in SYSAUX TS.
Will occupy max. 10% of SYSAUX.
Weekly job will delete plans
not used since 53 weeks [default].
Before Oracle 12c a massive number of hints get stored –
since Oracle 12c entire plans will be kept
SQL Profiles
Initial plan
will be accepted
if repeated and
will be added to the
SQL Plan Baseline
Plan History

33. SQL Plan Management | Mechanism
• Phase 2 – Selection
• Same statement parsed again but a different plan will be created
GB
HJ
...HJ
Plan Baseline
Plan History
Parse Execute
SQL
GB
NL
...NL
Plan acceptable

• OPTIMIZER_USE_SQL_PLAN_BASELINES=TRUE
• OPTIMIZER_CAPTURE_SQL_PLAN_BASELINES=FALSE
• Both switched to their defaults
• Annotation:
If OCSPB=TRUE then this plan would have been added to
the baseline
New plan will be added
to the Plan History but
it won‘t be used unless
it has been verified
GB
NL
...NL

34. SQL Plan Management | Mechanism
• Phase 3 – Evolution
• Since Oracle 12.1.0.2:
GB
HJ
...HJ
Plan Baseline
Plan History
GB
NL
...NL
GB
NL
HJHJ
GB
HJ
...HJ
Plan Baseline
Plan History
GB
NL
...NL
GB
NL
HJHJ
Equal or better plans
can be added to the
SQL Plan Baseline
Optimizer
Worse plans
will be kept
in the Plan
History and
will get purged
New SYS_AUTO_SPM_EVOLVE_TASK
job as part of the Automatic SQL Tuning Task
Report: DBMS_SPM.REPORT_AUTO_EVOLVE_TASK
Manual: DBMS_SPM.CREATE_EVOLVE_TASK

35. SQL> select PARAMETER_NAME, PARAMETER_VALUE
from DBA_SQL_MANAGEMENT_CONFIG;
SQL Plan Management | Settings
• Configure SQL Plan Management (SPM)
• Check settings:
• Change retention:
• Default: 53 weeks
• Change space consumption:
• Default: 10% of SYSAUX
• Plans will be stored in a LOB
• Sources to load plans from:
SQL> exec DBMS_SPM.LOAD_PLANS_FROM_AWR (begin_snap=>29, end_snap=>69);
SQL> exec
DBMS_SPM.CONFIGURE('plan_retention_weeks',5);
SQL> exec
DBMS_SPM.CONFIGURE('space_budget_percent',5);

36. SQL Plan Management | Plan Stability
GB
HJ
...HJ
Plan Baseline
Plan History
STS

STSExport/Import for Migrations
Upgrade
Plans will be takenfrom STS and
addedto the SQL Plan Baseline

DBMS_SPM.LOAD_PLANS_FROM_SQLSET

37. Testing Tools | SQL Tuning Advisor
Collect execution
plans before upgrade
Compare AWR
Snapshots
Verify them with
SQL Performance
Analyzer
Regressed plans?
SQL Plan
Management
SQL Tuning Advisor
Verify functionality
and performance with
Database Replay

38. SQL Tuning Advisor | SQL Profiles
• SQL Profiles provide more information to the optimizer
• Part of the Tuning Pack
• FORCE_MATCH=TRUE
• Don't change the application code
• Persistent, transportable (MOS Note:751068.1), can be tested without risk
• Example:

39. PhotobyTjSedisaonUnsplash
SQL Tuning Advisor: Example
1. Problem to be solved
2. Select candidate statement to tune
3. Get SQL statement details
4. Create and run SQL Tuning task
5. View report results

40. SQL Tuning Example
1. Problem to be solved

41. SQL Tuning Example
2. Select candidate statement to tune
• Generate an AWR Report

42. SQL Tuning Example
3. Get SQL Statement Details

43. SQL Tuning Example
4. Create and run SQL Tuning task
VARIABLE stmt_task VARCHAR2(64);
EXEC :stmt_task := DBMS_SQLTUNE.CREATE_TUNING_TASK(sql_id =>
'f4k19gvr3nu38');
EXEC DBMS_SQLTUNE.EXECUTE_TUNING_TASK(:stmt_task);
SET LONG 10000
SET LONGCHUNKSIZE 10000
SET LINESIZE 100
SPOOL sql_tune_f4k19gvr3nu38.txt
SELECT DBMS_SQLTUNE.REPORT_TUNING_TASK( :stmt_task )FROM DUAL;
SPOOL OFF;

44. SQL Tuning Example
5. View report results

45. SQL Tuning Example
5. View report results

46. SQL Tuning Example
5. View report results

47. SQL Profile - evaluation
• SQL Profiles should be evaluated before making them available to every user:
• Now evaluate the statement's profile in a limited user context
• If verification went fine, make it accessible to everybody
exec
:p_name:=dbms_sqltune.accept_sql_profile
task_name=>‘TASK_21944',name=>'XT_PROFILE',
category=>'TEST_ENV', FORCE_MATCH=>TRUE)
alter session set SQLTUNE_CATEGORY='TEST_ENV';
exec
dbms_sqltune.alter_sql_profile
(name=>'XT_PROFILE',
attribute_name=>'CATEGORY',value=>'DEFAULT')

48. SQL Tuning Example
5. View report results

49. SQL Tuning Example
5. View report results

50. SQL Tuning Example
6. Act on findings
• Follow 5 statistics recommendations to gather stats on 5 tables
• Result: 20x improvement!

51. Testing Tools | Database Replay
Collect execution
plans before upgrade
Compare AWR
Snapshots
Verify them with
SQL Performance
Analyzer
Regressed plans?
SQL Plan
Management
SQL Tuning Advisor
Verify functionality
and performance with
Database Replay

52. Testing Tools | Database Replay
Database Replay
Capture
Workload
Replay Workload
Compare – then restore
and run again with changes
11.2.0.4 19.7.0
Migrate
Preprocess
Workload

53. Slides | https://MikeDietrichDE.com

54. Time for a break

55. The "hidden" gems
Fix it when you upgrade
PhotobyandréspilborghsonUnsplash

56. Behavior Change 19c | Automatic SPM
• SQL Plan Management: "Automatic" and enabled on Exadata

57. Behavior Change 19c | Automatic SPM
• SQL Plan Management: "Automatic" and enabled on Exadata
• If you'd like to revert to 12.1 or 12.2/18c behavior:
• https://mikedietrichde.com/2019/06/03/automatic-sql-plan-management-in-oracle-database-19c/

58. Statistics Advisor
Since Oracle 12.2.0.1, the Statistics Advisor is active
• Some customers reported high growth of entries
• Active on Standard Edition (SE2) as well
• https://mikedietrichde.com/2017/08/22/oracle-optimizer-statistics-advisor-in-oracle-database-12-2-0-1/
• If you desire to turn it off:
• BEGIN
dbms_advisor.set_task_parameter ('AUTO_STATS_ADVISOR_TASK','_AUTO_MMON_INTERVAL',9999999999);
dbms_advisor.set_task_parameter ('AUTO_STATS_ADVISOR_TASK','_AUTO_STATS_INTERVAL',9999999999);
END;
/

59. Automatic SQL Tuning Set
In Oracle 19.7.0, a Automatic SQL Tuning Set gets populated
• Some customers reported high growth and consumption in SYSAUX
• See: https://mikedietrichde.com/2020/05/28/do-you-love-unexpected-surprises-sys_auto_sts-in-oracle-19-7-0/
• If you want to disable it:
• BEGIN
DBMS_AUTO_TASK_ADMIN.DISABLE(client_name=>'Auto STS Capture Task', operation=>NULL, window_name=>NULL);
END;
/
• Task is not enabled by default from Oracle 19.8.0 on

60. Oracle 19c | _cursor_obsolete_threshold
• MOS Note: 2431353.1
High Version Counts for SQL statements (>1024) post upgrade To 12.2 and above causing
database slow performance
• Since Oracle 12.2.0.1, the old default of 1024 has been raised to 8192
• This may lead to various drastic performance issues
• Recommendation:
• Set it to the old default to avoid mutex concurrency and other issues:
_cursor_obsolete_threshold=1024

61. Oracle 19c | deferred_segment_creation
• MOS Note: 1216282.1
All sorts of issues and bugs - from corruption to mutex contention to customer scripts not
operating correctly anymore
• Recommendation:
Set it to
deferred_segment_creation=false
unless you have a valid reason

62. Oracle 19c | Adaptive Features
• Recommendation: Set optimizer_adaptive_statistics explicitly in your SPFILE
• See:
• https://mikedietrichde.com/2018/01/18/additional-info-adaptive-features-fixes-oracle-12-1-0-2/
• MOS Note: 2187449.1 - Recommendations for Adaptive Features in Oracle Database 12.1
Oracle 12.1 optimizer_adaptive_features = TRUE
Oracle 12.2
Adaptive
Statistics
optimizer_adaptive_statistics
Default: FALSE
Adaptive
Plans
optimizer_adaptive_plans
Default: TRUE

63. Oracle 19c | _sql_plan_directive_mgmt_control
• MOS Note: 2209560.1 - How To Disable SQL Plan Directive (SPD)
• In order to fully disable SQL Plan Directives, you need to set:
• _sql_plan_directive_mgmt_control=0
• Otherwise the database collects SPDs in the background, but won't use it
• Having optimizer_adaptive_statistics=false - which is the default - disables only the usage
of SPDs but not their creation

64. "Hidden" Features | Extended VARCHAR2
• Feature Introduced in Oracle 12.1.0.2
• Allow 32kB support for (N)VARCHAR2
• Allows length >4000 byte
ALTER SYSTEM set
MAX_STRING_SIZE=EXTENDED
scope=SPFILE;
CREATE TABLE Applicants
(id NUMBER GENERATED AS IDENTITY,
first_name varchar2(30),
last_name varchar2(30),
application date,
CV varchar2(32767)
);
SHUTDOWN IMMEDIATE
STARTUP UPGRADE
@?/rdbms/admin/utl32k.sql

65. "Hidden" Features | Extended VARCHAR2
• Fine print:
• Oracle Forms and Reports don't support Extended VARCHAR2
• MOS Note: 2221740.1
ORA-24920 In Oracle Forms 12.2.x when using MAX_STRING_SIZE=EXTENDED
• Above 3964 byte, data will be stored in an inline SecureFile LOB
• See: http://www.ludovicocaldara.net/dba/extended-data-types-storage/
• Be aware of performance impact
• See: https://blog.dbi-services.com/12c-extended-datatypes-better-than-clob/

66. Cool Features | Incremental Statistics
• Introduced in Oracle 11g
• DWH, Partitioning
• Synopses on disk can consume hundreds of GB
• Since Oracle Database 12.2.0.1:
New algorithm for synopses can drastically reduce space consumption

67. Cool Features | Incremental Statistics
Sales Table
May 22nd 2019
May 23rd 2019
May 18th 2019
May 19th 2019
May 20th 2019
May 21st 2019
May 24rd 2019
Global
Statistic
 Global stats generated by aggregating the
original partition synopsis with the new one
A new partition is added
to the table & data is loaded

 Global stats generated by aggregating partition
level statistics and synopsis
 Retrieve synopsis for each of the other partitions
from SYSAUX
Gather partition statistics for new partition
Partition level stats are gathered & synopsis created
S1
S2
S3
S4
S5
S6
S7
SYSAUX Tablespace

68. Cool Features | Inc Stats Enhancement
• Enable that "changed" partitions won't be eligible for new stats generation
• Adjust the stale percentage if needed
SQL> exec
DBMS_STATS.SET_DATABASE_PREFS('INCREMENTAL_STALENESS','USE_STALE_PERCENT');
SQL> exec DBMS_STATS.SET_DATABASE_PREFS('STALE_PERCENT','12');
NEW IN
12.1

69. Cool Features | Inc Stats Enhancement
• Significantly smaller synopses with new algorithm
• Preference name: APPROXIMATE_NDV_ALGORITHM
• 'REPEAT OR HYPERLOGLOG' [Default]
• If synopses of old format exists, continue with the "old" algorithm
• If no or new synopses exist, use the "new" algorithm
• 'ADAPTIVE SAMPLING'
• Force to use pre-12.2 adaptive sampling algorithm – delete existing 12.2 synopses and regather - "old behavior"
• 'HYPERLOGLOG'
• Force to use the new algorithm
• Lab example:
• 8TB table with 84 partitions
• → Size in Oracle 12.1.0.2: 167 GB
• → Size in Oracle 12.2.0.1: 7 GB
NEW IN
12.2

70. Cool Features | Inc Stats Enhancement
• Preference: HYPERLOGLOG
• Replace "old" with "new"
- Recommended approach
• Preference: REPEAT OR HYPERLOGLOG
• Coexistence of "old" and "new"
- Issues seen at customers with stats gathering
SALESSYSAUX Tablespace
P2
P1 has a synopsis
in pre-12.2 format
New synopsis for P1 in
compact 12.2 format
Partitioned
Table
P1
exec dbms_stats.set_table_prefs(owner, table,
'approximate_ndv_algorithm', 'hyperloglog');
exec dbms_stats.set_table_prefs(owner, table,
'incremental_staleness', 'NULL')
SALESSYSAUX Tablespace
P2
P1 has a synopsis
in pre-12.2 format
Partitioned
Table
P1
exec dbms_stats.set_table_prefs(owner, table,
'approximate_ndv_algorithm', 'hyperloglog');
exec dbms_stats.set_table_prefs(owner, table,
'incremental_staleness', 'allow_mixed_format')
Synopsis forP2 in
compact12.2 format
NEW IN
12.2

71. Copyright © 2020 Oracle and/or its affiliates.
Now RELAX ...
Stay Calm ...
And open an SR with
Oracle Support in case
of real trouble

72. DEMO

73. Further Information
Finally …

74. Slides | https://MikeDietrichDE.com

75. HOL | https://MikeDietrichDE.com

76. Thank you!

77. Upcoming Webinars
Webinar 4:
Thursday, July 2 at 11:00h CEST
Migration to Oracle Multitenant
https://tinyurl.com/upgrade19c