MAA Best Practices for Oracle Database 19c

Maximum Availability Architecture –
Best Practices for Oracle Database 19c
Markus Michalewicz
Senior Director of Database HA & Scalability Product Management
@OracleRACpm
http://www.linkedin.com/in/markusmichalewicz
http://www.slideshare.net/MarkusMichalewicz
Copyright © 2019 Oracle and/or its affiliates.
Lawrence To
Senior Director of MAA Development

The following is intended to outline our general product direction. It is intended for information purposes
only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code,
or functionality, and should not be relied upon in making purchasing decisions. The development,
release, timing, and pricing of any features or functionality described for Oracle’s products may change
and remains at the sole discretion of Oracle Corporation.
Statements in this presentation relating to Oracle’s future plans, expectations, beliefs, intentions and
prospects are “forward-looking statements” and are subject to material risks and uncertainties. A detailed
discussion of these factors and other risks that affect our business is contained in Oracle’s Securities and
Exchange Commission (SEC) filings, including our most recent reports on Form 10-K and Form 10-Q
under the heading “Risk Factors.” These filings are available on the SEC’s website or on Oracle’s website
at http://www.oracle.com/investor. All information in this presentation is current as of September 2019
and Oracle undertakes no duty to update any statement in light of new information or future events.
Safe Harbor

Program Agenda
• Motivation & Benefits
• Improvements in Bronze
• Improvements in Silver
• Improvements in Gold
• Improvements in Platinum
• Cloud MAA

Average cost of
downtime per hour
Average cost of
unplanned data center
outage or disaster
Average amount of
downtime per year
Percentage of
companies that have
experienced an
unplanned data center
outage in the last 24
months
Impact of Database Downtime
91%
$10M$350K
Source: Gartner, Data Center Knowledge, IT Process Institute, Forrester Research
87 hours

Oracle Maximum Availability Architecture (MAA)
Reference
Architectures
Deployment Choices
HA Features,
Configurations &
Operational
Practices
Customer Insights &
Expert Recommendations
Production Site Replicated Site
Platinum
Gold
Silver
Bronze
Replication
Data Protection
Flashback RMAN + ZDLRA
Continuous Availability
Application
Continuity
Global Data
Services
Generic
Systems
Engineered
Systems
DBCS
ExaCS/ExaCC
Autonomous DB
Active Replication
Active Data Guard GoldenGate
24
Scale Out
RAC ShardingASM

Provide the best HA, Disaster Recovery
and data protection solutions for the
Oracle databases x
Continue to enhance validated
Maximum Availability Architecture
(MAA) solutions
Your success is truly our success!!!

MAA Solutions: On-Premises to Cloud
On-Premises
On-Premises Exadata and Recovery
Appliance
DBCS/ExaCS/ExaCC
Autonomous Database
MAA Reference Architectures and
Best Practices
MAA integrated Engineered Systems
(config practices, exachk, lowest
brownouts, HA QoS, data protection)
Adding MAA Config and Life Cycle
Operations, Shifting Admin Ownership
to Oracle with MAA SLAs

Oracle MAA Reference Architectures
Align Oracle Capabilities with Customer Service Level Requirements
Business Critical
Mission Critical
Dev, Test, Prod
Extreme Critical
Single Instance with Restart
Online Maintenance
Validated Backup/Restore
Silver +
Physical Replication
Comprehensive Data Protection
Gold +
Logical Active/Active Replication
Advanced HA Options
GOLD
BRONZE
SILVER
PLATINUM
Bronze +
Database HA
Active/Active Clustering
Application Continuity

Best Practices for the Database 19c
Improvements in Bronze

Outage Matrix
Unplanned Outage RTO / RPO*
Recoverable node or instance failure Minutes to hour
Disasters: corruptions and site failures Hours to days. RPO since last
backup or near zero with ZDLRA
Planned Maintenance
Software/hardware updates Minutes to hour
Major database upgrade Minutes to hour
Single
Instance
Database
Primary Availability Domain Secondary Availability Domain
Local Backup Replicated
Backups
Dev, Test, Prod - Single Instance
Database with Backups
• Single Instance with Clusterware
Restart
• Advanced backup/restore with
RMAN
• Optional ZDLRA with
incremental forever and near
zero RPO
• Storage redundancy and
validation with ASM
• Multitenant Database/Resource
Management with PDB features
• Online Maintenance
• Inherent corruption protection
• Flashback technologies
BRONZE
* RPO=0 unless explicitly specified

MAA Score Card
MAA architectural
readiness and
configuration practices
Database and Exadata Health Checks
Assessment Report
Health Score, Summary,
Findings
Findings &
Recommendations
How to Solve the
problem?
Automated Orachk/Exachk Healthcheck (Doc ID 107954.1) updated frequently

Recovery Appliance Recommended
Cloud
Storage
Remote
Replica
Tape
End-to-End Oracle Recovery Validation
Near Zero Data Loss for DR
Day 1 Full
a
Day 2 Changes
Day N Changes
Virtual
Full Backup
EM Real-Time
Protection Status
& Space Monitoring
Day 1 StateDay 2 StateDay N State
Databases
Transactional
Block Changes
No More Full Backups,
Incremental Forever
Oracle DB 12c-19c
on Any Platform

RA SF normally replicates to RA Austin
When Primary Appliance (RA SF) is not available,
backups and redo are redirected to Replica Appliance (RA Austin)
• Virtual fulls are created as normal – full recoverability supported
• Size Replica per Recovery Window Goal (RWG) requirement:
1x full backup + N RWG days of incremental and redo/arch log backups
Bare minimum: 1x full backup + 1 day redo/arch logs backups.
When Primary is back online, Replica backups are transferred
• Backups are ingested and processed into virtual fulls
• Normal backups to upstream can be restarted immediately
• Virtual fulls for new backups are created after all transferred backups
have completed processing
X
High Availability for Backup & Recovery
RA SF
Replication
RA SF RA Austin
RA Austin
Replication
RA SF RA Austin
Backups to
Replica
Appliance
Replica Appliance
Backups transferred to Primary
Configuring High Availability ZDLRA Client for Backup and Recovery (Doc ID 2432144.1)

Improvements in Silver

Prod/Departmental
SILVER
Bronze +
• Real Application Clustering (RAC)
• Application Continuity
Unplanned Outage RTO/RPO*
Recoverable node or instance failure Seconds
Disasters: corruptions and site failures Hours to days. RPO since last
backup or near zero with ZDLRA
Planned Maintenance
Software/hardware updates Zero
Major database upgrade Minutes to hour
Outage Matrix
RAC Database
Primary Availability Domain Secondary Availability Domain
Local Backup Replicated
Backups

Oracle RAC Provides Active-Active HA
• RAC provides scalability by actively
using resources across all nodes
• Zero-downtime planned maintenance
for typical software patches (DB, GI, OS)
• Node and instance failures are
automatically & transparently handled
• Supports all applications that are
supported on single instance databases
Only
with
Oracle
BB
Private Network
B
Private Network
Failover

Transparent Application Continuity
• Failures in the database stack & network lead to
• Interruptions or timeouts in user sessions
• Unknown state of transactions
• Application Continuity masks errors from applications by
recovering session state and replaying in-flight requests
• Replay performed on a surviving RAC instance or Data Guard standby
• Eliminates the need to create custom exception code
• Application Continuity extends Oracle’s HA capabilities
from bottom-to-top – from infrastructure to applications
Preserving and Replaying Database Requests Across Outages
1
2
3
4
5
6
UCP, JDBC, ODP.Net,
OCI Session Pool,
Tuxedo, WebLogic
✓

Checklist for Achieving Zero Application Downtime
1. Use Oracle Clusterware Service (never use default service)
2. Use Recommended Connection String
3. Configure FAN for Connection Pool
4. Drain your service for maintenance
5. Use (Transparent) Application Continuity
Application Checklist for Continuous Service for MAA Solutions
Using RHPhelper to Minimize Downtime During Planned Maintenance on Exadata (Doc ID 2385790.1)
Fleet Patching and Provisioning incorporates MAA practices documentation

Improvements in Gold

Outage Matrix
Disasters: corruptions and site failures Seconds. RPO zero or seconds
Planned Maintenance
Software/hardware updates Zero
Major database upgrade Seconds
Primary Region Secondary Region
Local
backup
Remote
Standby
Primary
Local
Standby
Local
backup
AD2 AD1
Mission Critical
Silver +
• Active Data Guard
• Comprehensive Data Protection
MAA Architecture:
• At least one standby required
across AD or region
• Primary in one data center(or AD)
replicated to a Standby in another
data center
• Local backups on both primary and
standby
GOLD

Data Guard: Real-time Data Protection & Availability
Primary Data Center DR Data Center
Data Guard Broker
(Enterprise Manager Cloud Control or DGMGRL)

Active Data Guard Provides
Active-Active Disaster Recovery (DR)
Zero data loss at any distance
Automatic Block Repair
DML Redirection
• Synchronous zero data loss replication
• Database rolling upgrade to reduce downtime
for planned maintenance
• Automatic failover for High Availability
Primary
Open Read-Write
Standby
Open Read-Mostly
Multi-instance Redo
Apply for RAC
(In Memory supported)

DML Redirect / Updates on Standby
• Updates on Standby are automatically performed from
an Active Data Guard standby to the primary database
without compromising ACID
• New documented parameter ADG_REDIRECT_DML controls DML Redirection
• New alter session ADG_REDIRECT_DML allows for per-session override
• New ADG_REDIRECT_PLSQL commands
• Supported with Oracle Database 19c
• Targeted for “Read-Mostly,
Occasional Updates” applications

(Active) Data Guard Features 19c
19c Data Guard Hidden Gems
• New Parameters for tuning automatic outage resolution with Data Guard
• Flashback Database enhancements with Data Guard
• Buffer Cache on Active Data Guard preserved after role transition for RAC
• Improved Data Guard Multi-Instance Redo Apply (with in-Memory)

Multi-Instance Redo Apply Performance
• Utilizes all RAC nodes on the Standby database to parallelize recovery
• OLTP workloads on Exadata show great scalability
Lower Latency Active Data Guard Standby Databases
190 380 740
1480700
1400
2752
5000
0
1000
2000
3000
4000
5000
6000
7000
1 Instance 2 Instances 4 Instances 8 Instances
Batch
OLTP
Standby
Apply
Rate
MB/sec

Data Guard Best Practices at a Glance
Creation
• 12.1.0.2 or higher: use
“RMAN restore from
service method”
• Creating a Physical
Standby using RMAN
Duplicate (RAC or Non-
RAC) (Doc ID 1617946.1)
• Assessing and Tuning
Network Performance for
Data Guard and RMAN
(Doc ID 2064368.1)
Network Performance
Use oratcptest tool to:
• Assess network
bandwidth prior to
deployment
• Tune ASYNC transport
• send_buffer_size
• recv_buffer_size
• OS socket limits
• Determine network
roundtrip latency with
SYNC transport
Async Transport
• Push ASYNC performance
to provide near zero data
loss protection.
• Best Practices for
Asynchronous Redo
Transport - Data Guard and
Active Data Guard:
• https://www.oracl
e.com/technetwo
rk/database/avail
ability/async-
2587521.pdf
Sync Transport
• Size online log file properly
• Avoid frequent log switches
• Configure single member
standby redo log on fast storage
• Best Practices for Synchronous
Redo Transport - Data Guard and
Active Data Guard
• https://www.oracle
.com/technetwork
/database/availabil
ity/sync-
2437177.pdf
ß

Why Pluggable Databases for High Availability?
• Pluggable Databases are an inherent database feature
• Enable common lifecycle operations in an online fashion
• Integrated with other Oracle High Availability (HA) features

De-support of Non-Container Database Architecture
• The Oracle Database non-CDB architecture will be
de-supported from Oracle Database 20c onwards
• To ease the migration to this architecture,
from Oracle Database 19c onwards, the multitenant architecture
supports up to 3 user-created Pluggable Databases of any type
• The Multitenant Option is required for 4 or more user-created PDBs

• “The Question”:
• How to migrate / upgrade to 20c supporting PDBs only?
• Answers:
1. https://mikedietrichde.com/2019/08/13/database-migration-from-non-cdb-to-pdb-
the-minimal-downtime-challenge/
2. Using Transient Logical Rolling Upgrade for Database Migration (Doc ID 2350945.1)

PDB Failover: Normal Operation
PDB1 PDB2
CDB 1
Read-Write
CDB 1
Standby
Read- OnlyData Guard
CDB 2
Read-Write
PDB4
PDB2 PDB3PDB1 PDB3

Unplug/plug PDB2 from CDB1 standby to CDB2 and failover application connections
PDB Failover after PDB 2 Outage
PDB1
Data Guard
PDB4
PDB2PDB1 PDB2 PDB3 PDB3
PDB2
CDB 1
Read-Write
CDB 1
Standby
Read- Only
CDB 2
Read-Write

Multitenant “Gold” MAA
Unplanned Outages Key Features for Solution RTO RPO
Recoverable node or instance failure Real Application Cluster (RAC)
Application Continuity (AC/TAC)
Secs Zero
Disasters: corruptions and site failures Active Data Guard Fast-Start
Failover
Secs Zero or Secs
PDB unrecoverable failure or “sick” PDB
(NEW)
PDB Failover (unplug/plug)
Another target CDB on the same
cluster required (MOS 2088201.1)
Secs Zero or Secs
Planned Maintenance Solution RTO
Software and hardware updates RAC, AC or TAC Zero
Major database upgrade Active Data Guard DBMS_ROLLING Secs
Migration to remote CDB (NEW) PDB Relocate Mins
Migration plus upgrade (NEW) PDB Relocate + Upgrade Mins
Updated MAA Best Practices Papers: Best Practices For Database Consolidation On Oracle Exadata Database Machine

Refreshable PDB Switchover
Per-PDB replica with only two CDBs to manage!
Server1
CDB1
CDB2
Server2
1. create pluggable database Red;
4. create pluggable database Brown;
6. create pluggable database Grey
from Grey@CDB2_Link
refresh mode every 2 minutes;
2. create pluggable database Red
from Red@CDB1_Link
refresh mode every 2 minutes;
3. create pluggable database Gold;
5. create pluggable database Grey;

Planned switchover
Server1
CDB1
CDB2
Server2
1. alter pluggable database Grey
refresh mode every 2 minutes
from Grey@dblink switchover;

Unplanned switchover
Server1
CDB1
CDB2
Server2 1. alter pluggable database Grey
refresh;
refresh mode none;
open read write;
Does not interoperate with Data
Guard Fast-Start Failover, auto-
block repair, DB rolling upgrade
so NOT part of Gold MAA

Database Rolling Upgrade
Database Rolling Upgrade with DBMS_ROLLING
• Pre-checks and early problem detection
• Fault tolerant, resumable and rollback capabilities
• Three Role Transition Steps: Start, Switchover, Finish
• Potential Maintenance Window: Hours
• Potential Database and Application Downtime: Seconds
Automated Database Upgrades using Oracle Active Data Guard and DBMS_ROLLING

Improvements in Platinum

Gold +
• GoldenGate Active/Active
Replication
• Optional: Edition-Based Redefinition
MAA Architecture:
• Each GoldenGate “primary” replica
protected by RAC and Active Data
Guard
• Primary in one data center (or AD)
replicated to another Primary in
remote data center (or AD)
• Oracle GG & Edition-Based
Redefinition for zero downtime
application upgrade
• Local backups on both sites
• Achieve zero downtime through
custom failover to GG replica
Extreme Critical
PLATINUM Primary Region Secondary Region
Local
backup
Local
backup
AD2 AD1
GG
Replication
AD1 AD2
Standby StandbyPrimary Primary
Outage Matrix
** application failover is custom
Disasters: corruptions and site failures Zero**
Planned Maintenance
Software and hardware updates Zero
Major database upgrade,
application upgrade, migration
Zero**

• “The Question”:
• How to migrate / upgrade to 20c supporting PDBs only?
• Answer:
1. GoldenGate can provide Zero Downtime Migration.

Oracle Sharding for a New Generation of Apps
Using standard MAA techniques to improve availability
Linear Scalability
Add shards online to increase
database size and throughput.
Online split and rebalance.
Extreme Availability
Shared-nothing hardware
architecture. Fault of one shard
has no impact on others.
Geographic Distribution
User defined data placement for
performance, availability, DR or to
meet regulatory requirements.
…
…
…
…
• Oracle RAC and Data Guard meet most application needs preserving application transparency.
• Some large scale applications want to shard data across independent databases and are
willing to modify the application to do so and for getting the benefits listed above.

Best Practices for the Oracle Database 19c
Cloud MAA

MAA Evolution from On-Premises to Autonomous
On-Premises
On-Premises
Exadata
Exadata
Cloud
Autonomous
Database
• Architecture
• Database Management (Tooling)
• Configuration, Tuning
• Lifecycle Operations (Tooling)
• Application Performance
• Choosing the SLA policy
• Application performance
• Infrastructure
Management
• Architecture
• Database Management
• Lifecycle operations
• Application Performance• Infrastructure
Management
• Architecture
• Database Management
• Lifecycle Operations
• Application Performance
• Blueprints
• Feedback to
products & features
• Blueprints
• Exadata is the best
integrated MAA DB
platform
• Oracle owns and
manages the best
integrated MAA
DB platform
• Cloud automation
for provisioning
and life cycle
operations
• Oracle owns and
manages Infrastructure
• Policy driven
deployments
• MAA Integrated cloud
• Fully automated Self-
Driving, Self-Securing,
Self-Repairing Database
Customer
Oracle

MAA Deployment Automation in the Cloud
• Simple UI / CLI / REST interfaces configured with MAA in mind
• Databases are provisioned with MAA parameter configurations
• MAA made easy in the Cloud
• Oracle Cloud Infrastructure (or)
• Cloud at Customer
Primary
Region#1
Standby
Region#2
GOLD(DR)
AD#1AD#2
PLATINUM(HA)
GG replication
Primary
FSFO
FSFO
Standby
BRONZE
Single
Instance
DB Backup
Service RAC
SILVER(HA)
DB Backup
Service

Cloud Configuration Best Practices
- Exadata Cloud deployment has built-in Exadata and MAA best practices
- Target: 100% MAA Healthscore at deployment time
ExaCS and ExaCC are Deployed with Exadata and MAA Best Practices
Oracle Exadata Database Machine EXAchk or HealthCheck (Doc ID 1070954.1)

Cloud Life Cycle & MAA
Life Cycle Operations MAA Practices
Create Cloud Databases Uses Exadata MAA templates or MAA templates if done via UI or Cloud APIs
Do NOT use custom scripts or DBCA
Migration to Cloud Zero Downtime Migration (ZDM) uses MAA practices with Data Guard
Logical migration can use GG hub which will be available in ZDM in the future
Infrastructure Software
Updates
Zero Database Downtime
Zero Application Downtime requires Continuous Availability - Application Checklist
for Continuous Service for MAA Solutions
DB/GI Software Updates Zero Database Downtime
Zero Application Downtime requires Continuous Availability - Application Checklist
for Continuous Service for MAA Solutions
MAA collaborating with cloud to add more prereqs and Data Guard support
Fleet Patch and Provisioning in the Cloud

Cloud Life Cycle & MAA
Life Cycle Operations MAA Practices
Exadata OS Updates How to update the Exadata System Software (DomU) to 19c from 18c on the
Exadata Cloud Service in OCI (Doc ID 2521053.1)
How to update the Exadata System Software (DomU) on the Exadata Cloud Service
in OCI (19.x to 19.x) (Doc ID 2566035.1)
Backup/Restore Use Automatic Backup/Restore with MAA practices
Oracle Cloud Infrastructure Exadata Backup & Restore Best Practices using Cloud
Object Storage
Health Checks Oracle Exadata Database Machine exachk or HealthCheck (Doc ID 1070954.1)
Real Time Monitoring and
Alerting
Enterprise Manager
Oracle Enterprise Manager for Exadata Cloud,
Exadata Health and Resource Utilization Monitoring - Exadata Database Machine
KPIs and
Exadata Health and Resource Utilization Monitoring - Adaptive Thresholds

Autonomous Database Cloud
• Exadata in a single AD with nightly backup replicated across other ADs
• Protects from the common sources of downtime such as hardware failures, software crashes,
and quarterly software updates
• Service Uptime SLA per Month: 99.95% < 22 minutes of downtime*
• Suitable for test, development and non-mission critical production databases
High Availability Policy
* SLA excludes AD or Region
failures, data corruptions and
certain planned maintenance
tasks like major upgradesDB Backup Service
Region #1
Database
Backups
Primary Database

Autonomous Database Cloud
• Exadata with Active Data Guard and Backup
• Protection from hardware failures, crashes, corruptions, patches, upgrades, disasters
• Service Uptime SLA per Month: 99.995NRX% (NRX = No Ridiculous Exclusions)
• 99.995% Uptime = at most 2m 12s of downtime per month
• Goal is for application impact from any one event to be well under 30 seconds
• Suitable for Mission Critical production databases
Extreme Availability Policy
Primary Database
Region #1, AD #1 Region #1, AD #2
Backup
Standby Database
Active
Data
Guard

Thank you!
Markus Michalewicz (Markus.Michalewicz@oracle.com)
Senior Director of Database Product Management
@OracleRACpm
www.linkedin.com/in/markusmichalewicz
www.slideshare.net/MarkusMichalewicz

MAA Best Practices for Oracle Database 19c

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to MAA Best Practices for Oracle Database 19c

Similar to MAA Best Practices for Oracle Database 19c (20)

More from Markus Michalewicz

More from Markus Michalewicz (17)

Recently uploaded

Recently uploaded (20)

MAA Best Practices for Oracle Database 19c