Oracle OpenWorld 2017 presentation on Oracle RAC 12c Rel. 2 & Cluster Architecture Internals. Presented by Anil Nair together with Dave Hickson, Database Architect, British Telecom (BT). This presentation focuses on new Cluster Architectures introduced with Oracle RAC 12c Rel. 2 and how internal enhancements in Oracle RAC can help to facilitate them.

2. Copyright
©
2017,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Oracle
RAC
12c
Release
2
and
Cluster
Architecture
Internals
Anil
Nair,
Ian
Cookson
-­‐
Oracle
RAC
Product
Management
-­‐
Oracle
Dave
Hickson
–
Database
Architect
–
BT
October
3,
2017

3. Copyright
©
2017,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Safe
Harbor
Statement
The
following
is
intended
to
outline
our
general
product
direcRon.
It
is
intended
for
informaRon
purposes
only,
and
may
not
be
incorporated
into
any
contract.
It
is
not
a
commitment
to
deliver
any
material,
code,
or
funcRonality,
and
should
not
be
relied
upon
in
making
purchasing
decisions.
The
development,
release,
and
Rming
of
any
features
or
funcRonality
described
for
Oracle’s
products
remains
at
the
sole
discreRon
of
Oracle.
3

4. Copyright
©
2017,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Program
Agenda
Cluster
Domain
Architecture
Member
Cluster
use
cases
BT
–
Cluster
Domains
in
AcRon
1
2
4
3

5. Copyright
©
2017,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Program
Agenda
Cluster
Domain
Architecture
Member
Cluster
use
cases
BT
–
Cluster
Domains
in
AcRon
1
2
5
3

6. Copyright
©
2017,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Oracle
RAC
12c
Rel.
2
Cluster
Domain
• Simplified
Management
– Fleet
Management
for
installaRon,
update,
patching
and
maintenance
• Reduced
Local
Overhead
– Member
Clusters
benefit
from
the
consolidaRon
of
common
services
on
the
Domain
Services
Cluster
• Improved
IO
Performance
– URlizing
consolidated
shared
storage
6
Centralized
Management
for
Cluster
Estates
“too
big
to
manage”
otherwise

7. Copyright
©
2017,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
7
Cluster
Domain
Applica@on
Member
Cluster
Uses
ASM
Database
Member
Cluster
Uses
local
ASM
Database
Member
Cluster
Uses
ASM
Service
Database
Member
Cluster
Uses
IO
Service
Domain
Services
Cluster
(DSC)
Shared
ASM
IO
Service
ACFS
Service
ASM
Service
TFA
Service
Management
Service
RHP
Service

8. Copyright
©
2017,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
8
The
DSC
–
The
Heart
of
the
Cluster
Domain
Domain
Services
Cluster
(DSC)
IO
Service
ASM
Service
TFA
Service
Management
Service
Shared
ASM
ACFS
Service
RHP
Service
• The
DSC
hosts
services
that
are
consumed
by
Member
Clusters,
including:
– Management
Service
for
centralized
and
simplified
management
– Trace
File
Analyzer
(TFA)
for
centralized
diagnosRcs
– Rapid
Home
Provisioning
(RHP)
for
so`ware
fleet
management
– Storage
Services
(ACFS,
ASM
direct
or
indirect
over
IO
Service)

9. Copyright
©
2017,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Domain
Services
Cluster
(DSC)
IO
Service
ASM
Service
Shared
ASM
ACFS
Service
RHP
Service
9
The
DSC
Management
Service
TFA
Service
Management
Service
Applied
Machine
Learning
for
Database
Diagnos@cs
• Efficient
diagnosis
using
Machine
Learning
• AutomaRcally
performs
correcRve
acRons
to
prevent
possible
issues
• Provides
simple
alerts
&
recommendaRons
for
issues
that
require
manual
intervenRon
Subject
MaOer
Expert
ASH
ML
Knowledge
Extrac@on
Model
Genera@on
Human
Supervision
Applica@on
Op@mized
Models
Feedback

10. Copyright
©
2017,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
10
Rapid
Home
Provisioning
Service
Domain
Services
Cluster
(DSC)
IO
Service
ASM
Service
Shared
ASM
ACFS
Service
RHP
Service
TFA
Service
Management
Service
• Provision
new
pools
onto
base
machines
• DB
and
GI:
provision,
scale,
patch,
upgrade
• Custom
workflow
framework
• NoRficaRon
model
• Audit
capabiliRes
RHP
Service
Member
Cluster
Member
Cluster
Member
Cluster
Fleet
Management
installa@on,
update,
patching
and
maintenance

11. Copyright
©
2017,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
11
Rapid
Home
Provisioning
Service
Domain
Services
Cluster
(DSC)
IO
Service
ASM
Service
Shared
ASM
ACFS
Service
RHP
Service
TFA
Service
Management
Service
• Provision
new
pools
onto
base
machines
• DB
and
GI:
provision,
scale,
patch,
upgrade
• Custom
workflow
framework
• NoRficaRon
model
• Audit
capabiliRes
RHP
Service
Member
Cluster
Member
Cluster
Member
Cluster
Fleet
Management
in
the
Oracle
Cloud
and
On-­‐Premise

12. Copyright
©
2017,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Zero
Impact
Patching
• Zero
Impact
Patching
enables
patching
of
the
Oracle
Grid
Infrastructure
without
interrupRng
database
operaRons.
• Patches
are
applied
out-­‐of-­‐place
and
in
a
rolling
fashion
with
one
node
being
patched
at
a
Rme
while
the
database
instance(s)
on
that
node
remain
up
and
running.
• Zero
Impact
Patching
supports
Oracle
Real
ApplicaRon
Clusters
(RAC)
databases
on
clusters
with
two
or
more
nodes.
12
Never
take
down
a
database
instance
to
patch
Grid
Infrastructure

13. Copyright
©
2017,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
13
Domain
Services
Cluster
(DSC)
Availability
• Services
provided
by
the
DSC
are
unaffected
by
nodes
joining/
leaving
the
DSC
cluster
• DSC
can
be
patched
and
upgraded
independently
without
affecRng
the
services
provided
by
DSC
• Use
Member
Clusters
for
user
databases
(not
the
DSC)

14. Copyright
©
2017,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
14
Cluster
Domain
Applica@on
Member
Cluster
Uses
ASM
Database
Member
Cluster
Uses
local
ASM
Database
Member
Cluster
Uses
ASM
Service
Database
Member
Cluster
Uses
IO
Service
Domain
Services
Cluster
(DSC)
Shared
ASM
IO
Service
ACFS
Service
ASM
Service
TFA
Service
Management
Service
RHP
Service

15. Copyright
©
2017,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
• A
Member
Cluster
is
a
Standalone
Cluster
uRlizing
shared
services
on
the
Domain
Services
Cluster
• It
automaRcally
benefits
from
the
management,
TFA
&
RHP
service.
– ASM
services
are
opRonal
– and
can
be
uRlized
as
needed
15
Member
Clusters
=
Standalone
Cluster
+
Benefits
Database
Member
Cluster
Uses
local
ASM
Database
Member
Cluster
Uses
ASM
Service
of
DSC
Database
Member
Cluster
Uses
ASM
IO
Service

16. Copyright
©
2017,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
16
Same
Tools
and
Commands
for
all
Types
of
Deployments
[GRID]>
crsctl
get
cluster
name
CRS-­‐6724:
Current
cluster
name
is
'SolarCluster'
[GRID]>
crsctl
get
cluster
class
CRS-­‐41008:
Cluster
class
is
'Standalone
Cluster'
[GRID]>
crsctl
get
cluster
type
CRS-­‐6539:
The
cluster
type
is
'flex'.
[GRID]>
crsctl
get
cluster
name
CRS-­‐6724:
Current
cluster
name
is
'SalesCluster'
[GRID]>
crsctl
get
cluster
class
CRS-­‐41008:
Cluster
class
is
‘Database
Member
Cluster'
[GRID]>
crsctl
get
cluster
type
CRS-­‐6539:
The
cluster
type
is
'flex'.

17. Copyright
©
2017,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
• Easy
conversion
from
Standalone
to
Member
Clusters
• Member
Cluster
GI
version
can
be
higher
or
equal
to
the
GI
version
on
the
Domain
Services
Cluster
17
Convert
a
Standalone
Cluster
to
Member
Cluster
Convert
Database
Member
Cluster
Uses
local
ASM

18. Copyright
©
2017,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Program
Agenda
Cluster
Domain
Architecture
Member
Cluster
use
cases
BT
–
Cluster
Domains
in
AcRon
1
2
18
3

19. Copyright
©
2017,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
19
Cluster
Domain
Applica@on
Member
Cluster
Uses
ASM
Database
Member
Cluster
Uses
local
ASM
Database
Member
Cluster
Uses
ASM
Service
Database
Member
Cluster
Uses
IO
Service
Domain
Services
Cluster
(DSC)
Shared
ASM
IO
Service
ACFS
Service
ASM
Service
TFA
Service
Management
Service
RHP
Service

20. Copyright
©
2017,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Database
Member
Cluster
with
Local
ASM
• For
databases
requiring
– Full
IsolaRon
and
performance
stability
– That
can
benefit
from
the
centralized
Management
Service
on
the
DSC
• ParRcularly
suitable
for
unpredictable
workloads,
or
highly
variable
workloads
• Examples
include
– Business
Intelligence
and
AnalyRcs
systems
– Batch
processing
systems
– Response-­‐criRcal,
user-­‐facing
systems
20
Standalone
isola@on
with
reduced
local
overhead
Database
Member
Cluster
Uses
local
ASM

21. Copyright
©
2017,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
21
Cluster
Domain
Applica@on
Member
Cluster
Uses
ASM
Database
Member
Cluster
Uses
ASM
Service
Database
Member
Cluster
Uses
IO
Service
Domain
Services
Cluster
(DSC)
Shared
ASM
IO
Service
ACFS
Service
ASM
Service
TFA
Service
Management
Service
RHP
Service
Database
Member
Cluster
Uses
local
ASM

22. Copyright
©
2017,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Database
Member
Cluster
Uses
ASM
Service
of
DSC
Database
Member
Cluster
Using
ASM
Service
• For
databases
requiring
– IsolaRon
and
performance
stability
– That
can
benefit
from
the
centralized
Management
Service
on
the
DSC
– And
the
centralized
ASM
Storage
Management
Service
on
the
DSC
• Best
suited
for
workloads
for
which
IO
stability
is
important,
but
benefit
from
the
centralized
ASM
Services
on
the
DSC
• Examples
include
– OLTP
systems
– ReporRng
systems
22
Standalone
isola@on
benefihng
from
consolidated
shared
storage
ASM
Service

23. Copyright
©
2017,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
23
Cluster
Domain
Applica@on
Member
Cluster
Uses
ASM
Database
Member
Cluster
Uses
local
ASM
Database
Member
Cluster
Uses
ASM
Service
Database
Member
Cluster
Uses
IO
Service
Domain
Services
Cluster
(DSC)
Shared
ASM
IO
Service
ACFS
Service
ASM
Service
TFA
Service
Management
Service
RHP
Service

24. Copyright
©
2017,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Database
Member
Cluster
Uses
ASM
IO
Service
Database
Member
Cluster
Using
the
IO
Service
• For
databases
– That
need
to
be
deployed
quickly
– or
cloned
or
duplicated
frequently
• Ideal
for
databases
that
can
allow
for
IO
path
sharing
with
other
Member
Clusters,
while
maintaining
control
of
local
cluster
processing,
workload
and
resources
• Examples
include
– Test,
integraRon,
development
systems
• For
which
maintaining
addiRonal
hardware
(e.g.
SAN
networks
and
storage)
is
inefficient
24
Consolida@on
at
its
best
u@lizing
full
resource
sharing
IO
Service
ASM
Service

25. Copyright
©
2017,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
25
All
Deployments
Benefit
from
Oracle
RAC
Features
PDB
Isola@on
Hang
Manager
Choose
the
“right
path”

26. Copyright
©
2017,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
1010101000010011100101001
1011100101001110010100101
0010010100001111010100101
26
• Using
Oracle
MulRtenant,
PDBs
can
be
opened
as
singletons
(in
one
database
instance
only),
in
a
subset
of
instances
or
all
in
instances
at
once.
• If
certain
PDBs
are
only
opened
on
some
instances,
Pluggable
Database
Isola/on
– improves
performance
by
• Reducing
DLM
operaRons
for
PDBs
not
open
in
all
instances.
• OpRmizing
block
operaRons
based
on
in-­‐memory
block
separaRon.
– improves
availability
by
• Ensuring
that
instance
failures
of
instances
only
hosRng
singleton
PDBs
will
not
impact
other
instances
of
the
same
RAC-­‐based
CDB.
Pluggable
Database
and
Service
IsolaRon
NEW
IN
12.2
Inter
Cluster
Messages

27. Copyright
©
2017,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Database
Member
Cluster
Uses
ASM
Service
27
Pluggable
Database
and
Service
IsolaRon
Provides
higher
IsolaRon
in
Member
Cluster
Consolidated
Environments

28. Copyright
©
2017,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Hang
Manager
• Hang
Manager
“manages”
database
process(es)
that
are
not
progressing
– Manage
=
detec@on
and
resolu@on
– Considers
cross-­‐layer
hangs
• I.E:
Hangs
caused
by
a
blocked
ASM
resource.
• In
addiRon,
deadlocks
are
now
resolved
by
Hang
Manager
• User
defined
control
via
PL/SQL
• Early
Warning
exposed
via
(V$
view)
28
Resolving
cross-­‐layer
hangs
in
the
Cluster
Domain
Database
Member
Cluster
Uses
ASM
IO
Service
IO
Service
ASM
Service

29. Copyright
©
2017,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
29
Resolving
cross-­‐layer
hangs
in
the
Cluster
Domain
Provides
higher
Availability
in
Member
Cluster
Consolidated
Environments
Database
Member
Cluster
Uses
ASM
Service
W
W
W
B

30. Copyright
©
2017,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Oracle
RAC
Always
Chooses
the
“Right
Path”
30
• RAC
determines
the
opRmal
path
to
serve
blocks
–
network
or
disk
• SSDs
and
NVMe
storage
technology
conRnue
to
drive
down
latency
• e.g.
flash
storage
may
provide
beser
access
Rmes
to
data
than
the
private
network
under
high
load
• RAC
takes
those
staRsRcs
into
account
Query
Block
Block
Network
congesRon

31. Copyright
©
2017,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Database
Member
Cluster
Uses
ASM
Service
31
The
“Right
Path”
Ensures
Performance
on
Member
Clusters
EffecRvely
reducing
the
impact
of
compeRng
workloads
overwhelming
the
I/O
on
the
DSC
B
u
s
y

32. Copyright
©
2017,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Program
Agenda
Cluster
Domain
Architecture
Member
Cluster
use
cases
BT
–
Cluster
Domains
in
AcRon
1
2
32
3

33. BT:
Cluster
Domains
in
AcRon
Dave
Hickson
–
Database
Architect
BT
October
3,
2017

34. ©
BriRsh
TelecommunicaRons
plc
About
BT
• The
UK’s
largest
broadband
provider
• The
UK’s
largest
last-­‐mile
network
provider
• The
UK’s
largest
wide
area
network
provider
• In
EE,
the
UK’s
best
largest
and
best
mobile
network
provider
• A
global
footprint
operaRng
in
180
countries
• BT
Sport
delivering
Premiership
and
UEFA
football
• All
underpinned
by
Technology,
Service
and
OperaRons

35. ©
BriRsh
TelecommunicaRons
plc
Context
–
ExisRng
Database
and
RAC
Services
in
BT
• Large
scale,
on-­‐premise
Enterprise
Cloud
• Thousands
of
databases,
hundreds
of
RACs
• Increasing
rate
of
growth
• Lots
of
automaRon
but
…
• We
need
smarter
ways
to
– Deliver
RAC
clusters
more
quickly
– Administrate
more
efficiently
– Enable
customer
self-­‐service
• What
we
need
is
a
more
Cloud-­‐oriented
RAC
architecture

36. ©
BriRsh
TelecommunicaRons
plc
Cluster
Domains
–
What
we’ve
been
doing
• Test
environment
on
BT
Enterprise
Cloud
– Four
node
Domain
Services
Cluster
– 8
Member
Clusters
– OEM
13.2
–
VMware-­‐based
infrastructure
• Key
Features
we’re
interested
in
(in
no
par@cular
order!)
– IO
Server
– Rapid
Home
Provisioning
– Autonomous
Health
Framework
– ApplicaRon
Containers

37. ©
BriRsh
TelecommunicaRons
plc
Cluster
Domains
–
Why
is
this
architecture
asracRve
to
us?
• I/O
Server
– Replace
hundreds
of
independent
pools
of
storage
with
centralised
pools
– Increase
storage
on
member
clusters
without
infrastructure
changes
• Rapid
Home
Provisioning
– Centralised
management
of
Oracle
so`ware
for
patching
and
upgrading
• Autonomous
Health
Framework
– Replace
many
independent
management
repositories
with
one
Goal
for
BT
is
simple:
Reduce
Overhead
of
Many
RAC
Clusters

38. ©
BriRsh
TelecommunicaRons
plc
Cluster
Domains
–
our
overall
impression
• Architecturally
this
is
the
right
direc@on
– Database
servers
run
databases
without
having
to
administer
infrastructure
– “Infrastructure”
tasks
such
as
storage,
performance,
so`ware
managed
centrally
• Applica@on
Containers
on
Member
clusters
enable
customer
self-­‐service
of
new
databases
without
overheads
of
infrastructure
management
on
each
cluster
• Organisa@onal
implica@ons:
– Infrastructure
Team
to
manage
Cluster
Domains
– Database
Team
to
manage
Member
clusters

39. Copyright
©
2017,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Oracle
RAC
Cluster
Domain
Architecture
• Benefit
from
the
Centralized
Manageability
services
• Benefit
from
a
centralized
pool
of
consolidated
shared
storage
• Benefit
from
a
centralized
console
to
provision,
patch,
upgrade
39
The
new
standard
in
consolida@on

40. Copyright
©
2017,
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
40