Keynote - first presented in course of the DOAG HA Day in Hamburg in September 2018. This deck discusses the Maximum Availability Architecture approach against the background of "Four More Often Than Not Disregarded HA Fundamentals" and also shares HA thoughts that eventually will lead to “Continuous Availability” rather than (just) “High Availability”.

1. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
From HA to MAA
A Holistic Historical Discussion
Markus Michalewicz – Senior Director of Product Management, Database HA & Scalability
September 18, 2018
Markus.Michalewicz@oracle.com
@OracleRACpm
http://www.linkedin.com/in/markusmichalewicz
http://www.slideshare.net/MarkusMichalewicz

2. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
How It All Began – At Least For Me
Edition One, John Wiley & Sons Inc,
New York 2000

3. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
High Availability – A Definition
High Availability (HA) [Computer System] The ability of a system to perform its function
continuously (without interruption) for a significantly longer period of time than the
reliabilities of its individual components would suggest.
High availability is most often achieved through failure tolerance. High availability is not
an easily quantifiable term. Both the bounds of a system that is called highly available
and the degree to which its availability is extraordinary must be clearly understood on a
case-by-case basis.
Failure Tolerance [Computer System]: The ability of a system to continue to perform its
function (possibly at a reduced performance level) when one or more of its components
has failed. (https://www.snia.org/education/dictionary/f#failure_tolerance)
https://www.snia.org/education/dictionary
This definition does not
consider planned
maintenance per se…

4. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
• Redundancy [General]: The inclusion of extra components of a
given type in a system (beyond those required by the system to
carry out its function) for the purpose of enabling continued
operation in the event of a component failure.
(https://www.snia.org/education/dictionary/r )
• Redundancy helps with
• Failure Tolerance [Computer System]: The ability of a system to
continue to perform its function (possibly at a reduced performance
level) when one or more of its components has failed.
(https://www.snia.org/education/dictionary/f#failure_tolerance)
The Main High Availability Feature – Redundancy

5. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Oracle Maximum Availability Architecture
Best Practices Blueprints for High Availability
Oracle Maximum Availability Architecture
(MAA) is Oracle's best practices blueprint
• based on proven Oracle high
availability technologies,
• expert recommendations
• and customer experiences.
The goal of MAA is to achieve optimal
high availability for Oracle customers at
the lowest cost and complexity.

6. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Four More Often Than Not Disregarded HA Fundamentals
“A fool with a
tool is still a fool”
Fewest Layer
Principle
Automatic
Failover often
requires an
arbiter / referee
Standardize
and keep it simple!

7. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
“A fool with a tool is still a fool”
• Backups are considered the minimum for RPO (Recovery Point Objective),
but may not be best when it comes to RTO (Recovery Time Objective)
– Have you checked how your Restore and Recovery performs in terms of RTO?
• Data Mirroring is considered better than backups in terms of RTO
but may not be best when it comes to RPO
– Mirrored data is a mirror – it mirrors the good and the bad…
• Concluding, don’t use backups or mirroring when replication is the better choice.
Or “Use the right tool for the job”

8. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
MAA Reference Architectures
Meet Downtime (RTO) and Data Loss (RPO) SLAs
Downtime & Data Loss
BRONZE Single Instance + Backup Dev, Test, Prod
SILVER HA Clustering + Backup Prod/Departmental
GOLD HA Clustering + Disaster Recovery + Backup Mission Critical
PLATINUM Zero Data Loss & Zero Downtime Extreme Critical
MAA Reference
Architectures Topology Suitable Databases
Addresses SLAs for Data Loss and Downtime during Planned & Unplanned Outages

9. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Fewest Layer Principle
• Each layer while perhaps helping to achieve HA has its own
potential for failure. Hence, the more layers, the more likely
a system can fail. Examples include, but are not limited to:
– Multipathing, (C)VM, OS (or vice versa), clusters, replication (if used)
• The problem gets worse in virtualized and cloud environments
in which “software defined”-hardware is used, especially when
allowing for dynamic re-configuration of these components, as
those also allow for human errors during (re-)configuration.
• Testing the system including all layers
as well as a good operations manual is mandatory therefore.
HA is best achieved using as little layers as possible

10. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Oracle MAA
Common Platform – On Premises, Cloud, and Hybrid Cloud
On Premises On Cloud
Oracle
Database
Designed to Address the Complete Range of Business Requirements in all Environments

11. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Fewest Layer Principle
• Hang detection and resolution is the trickiest of all HA problems.
• Multiple inherent solutions exist to monitor a system;
some of those solutions intend to detect and resolve hangs.
– Inherent hang detection is always timeout based and subject to hangs.
• Best to use an external system for monitoring and hang detection...
– Ensure that the monitoring system works across layers.
• E.g. detects OS, database and application hangs independently if possible.
Exception: Add a layer (on an auxiliary system) for Monitoring and Hang detection

12. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Automatic Failover Often Requires an Arbiter / Referee
• Automatic Failover between the independent, redundant components of
an HA solution is the most effective way to recover from failures.
– It eliminates reaction delay by performing pre-defined actions.
• Automatic Failover between two (or generally, an even number of) data-
managing components or sites requires an arbiter in some way, as
independent components cannot independently agree which one has the
right to continue (assuming there is such a choice).
– Clusters use voting or tie-breaker disks or quorums typically.
– EMC VPLEX Metro uses quorums and a witness disk (configurable).
– Oracle Data Guard uses the Observer for automatic fast failover.
• The arbiter should be hosted in a third location and not be part of the
systems or system components between which the failover occurs.
• Systems with an uneven number of sites are inherently protected.
Reason: two (or an even number of) independent sites cannot independently agree

13. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Even Faster, Remove Delay for Human Response
• Automatic:
– Failure detection
– Database failover
– Client failover using FAN
– Automatic reinstatement
of failed primary as a new standby database SYNC / ASYNC
Data
Guard
Observer
Primary Standby
Site A Site B
Primary
Site B
SYNC / ASYNC
Standby
Site B
Data Guard Fast-Start Failover

14. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
The Problem with Choices…
Venus
Oracle GI
Oracle RAC
Virtualization
Operating System
Earth
Oracle GI
Oracle RAC
Virtualization
Operating System
Storage Layer
Storage Access Layer
Virtualization Layer
Operating System Layer
Software Layer
Server Hardware Layer
Network Layer

15. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
The Problem with Choices … A Solution
Venus
Oracle GI
Oracle RAC
Virtualization
Operating System
Earth
Oracle GI
Oracle RAC
Virtualization
Operating System
Storage Layer
Storage Access Layer
Virtualization Layer
Operating System Layer
Software Layer
Server Hardware Layer
Network Layer

16. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Using a mainstream solution usually leads to smoother day-to-day operations
For High Availability Solutions Uniqueness Does Not Pay Off
Supported Configurations
Recommended Configuration

17. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Edition-based Redefinition,
Online Redefinition, Data Guard, GoldenGate
– Minimal downtime maintenance, upgrades, migrations
Active Data Guard
– Data Protection, DR
– Query Offload
GoldenGate
– Active-active replication
– Heterogeneous
Active Standby Site
RMAN, Oracle Secure Backup,
Zero Data Loss Recovery Appliance
– Backup to disk, tape or cloud
Enterprise Manager Cloud Control
– Site Guard, Coordinated Site Failover
RAC / RAC One
– Scalability
– Server HA
ASM
– Local storage protection
Production Site
Flashback
– Human error
correction
Oracle Maximum Availability Architecture (MAA)
Application
Continuity
– Application HA
Global Data Services
– Service Failover / Load Balancing
Sharding
– Horizontal Partitioning,
Scalability, Shared Nothing
architecture

18. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Consider your applications
as part of your HA planning.
Include planned maintenance
into your HA planning.
Start thinking
“Continuous Availability”
rather than “High Availability”.
Three Last Thoughts

19. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Consider Your Applications
• The higher (in a three-tier architecture) outages can be masked,
the more successful the masking from a user perspective usually.
• When making your application / clients aware of redundancy,
the “problem of the first one” can be solved much easier.
– Unless the entry point into a system (point of first connection)
transparently provides HA, the application / client needs to be
aware that an alternative can be used if the first attempt fails.
• Concluding, instead of solving any HA problem on the database
and lower levels, spend some effort talking to the apps-layer.
Best to ensure HA on application level

20. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Application Continuity Minimizes Application Downtime
• Application Continuity (AC) shields end users and
applications from interruptions due to instance failures.
– Sessions are automatically reconnected to surviving instances.
– Queries are replayed and appear to resume from the point of failure.
– Inflight transactions are replayed and it is ensured that they
are either committed exactly once or rolled back completely.
• In most cases applications will only experience a higher
response time without being made aware of the failure.
– Avoids cascading outages due to application servers and middle tier
components restarting as a mislead reaction to database failures.
• AC is available with Oracle RAC and Active Data Guard:
– Failover to another RAC instance or a Data Guard standby database.
Inst
1
Inst
2
write

21. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Include Planned Maintenance Into Your HA Planning
• Planned maintenance such as OS and database patching as well as
upgrading or migrating database environments and changing part
of the database schema have become more frequent and hence are
more often than not the main contributor to system downtime.
• Online Operations and Rolling Patching or ideally Online Patching
are the best ways to reduce planned maintenance downtime.
• Draining should be used to manage user experience during any type
of patching, ideally online patching, but is not required in this case.
Planned maintenance is one of the main contributors to system downtime

22. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Online Operations
• DBMS_REDEFINITION allows you to reorganize and redefine tables online
– Add/drop/rename columns, switch physical storage structures, reorg table, transform data … online
• DBMS_REDEFINITION enhanced in 12.2 for even the largest, busiest databases:
– Full restartability (resume at point of failure)
– Entire redefinition process runs without acquiring Exclusive DDL lock
– Optionally track changes to enable fast rollback to table as it was before the redef
– Maintain dependent MVs during redefinition
– V$online_redef for progress monitoring
– Support for Binary XML storage changes, BFILE, invisible columns
Online Redefinition Improvements
Source
Table
Update
Tracking
Transform
Copy
Table
Transform
Updates
Result
Table
Continuous
Queries &
Updates
Store
Updates

23. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
RAC Rolling Patching
• Oracle RAC supports rolling patches
• Rolling patching describes a patch application by
which only one or a group of instances are taken
down for patching
• Rolling patching can be used for
– OS and security patches
– Grid Infrastructure patches
– Certain database patches
• Draining and Application Continuity should be
used to manage user experience during patching
Inst
1
Inst
3
Inst
2

24. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Online Patching Improvements
• OJVM patching will be RAC rolling
– Non-Java service available all time (at least one instance)
– Java service available all time, except ~10 sec brown-out
• No error reported during brown-out
• Zero Downtime GI Patching
– Patch Oracle Grid Infrastructure without interrupting database operations.
– Patches applied out-of-place and in a rolling fashion with one node being patched
at a time while the database instance(s) on that node remain up and running.
– Supported on clusters with two or more nodes.

25. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Continuous Availability
• Removes downtime from user perspective
• In-flight work is preserved
• Maintenance is hidden
• Predictable performance during outages
• Errors only if unrecoverable
• Designed for multiple concurrent failures
• Builds on top of HA
High Availability
• Minimizes downtime
• In-flight work is lost
• Rolling maintenance at DB
• Predictable runtime performance
• Errors may be visible
• Design for single failure
• Basic HA building blocks
From High Availability to Continuous Availability

26. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
TUNABLES
• Hardware &
adjustable misscount
• SCSI Timeout
• Flex ASM
• FAST_START_MTTR_TARGET
• Service Isolation 18C
• PING_TIMEOUT
• FSFO Observer
Recover in a Timely Manner to Remove Downtime
Node Panic – non-
master
Node Panic – master
Public Network
Instance Join
Dynamic Remastering
Private Interconnect

27. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
For Oracle Customers wanting to
improve their system availability
to reduce costs caused by downtime.
For non-Oracle Customers to get an
idea what failure scenarios need to
be covered and how Oracle can help.
For Application Developers to
understand which failure scenarios
should be tackled by the application.
Conclusion – Oracle MAA is for Everybody!

28. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
… Oracle’s expertise built in ; a.k.a MAA
The Oracle Cloud Benefits from …
http://oracle.com/goto/maa

29. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |