Maximum Availability Architecture with Fusion Middleware 12c and Oracle Database 12c : Retail Case study

1. Maximum Availability Architecture
with Fusion Middleware 12c and
Oracle Database 12c:
A Retail Case Study
Nikitas Xenakis, The Co-operative Group
Simon Haslam, eProseed

2. About Us
Nikitas Xenakis
Solutions Specialist / Architect, The Co-op
• 16+ years as Lead Oracle DBA (v7-12cR2)
• Design & implementation of HA solutions (inc.
CRS/GI, RAC, Data Guard, EM, GG, WLS)
• Member of Oracle Data Integration Solutions
(DIS) Customer Advisory Board
• 12cR2 Beta tester (User Group, On-Premises)

3. About Us
Simon Haslam
Technical Director, eProseed UK
• Platform / Infrastructure Architect
• Using Oracle products since ~1995 (Oracle7)
• Formerly UKOUG App Server & Middleware SIG Chair

5. Introduction
The Dalton in Furness Co-op Store
http://www.coop.co.uk/corporate/aboutus/ourhistory/
5
• Eight ‘Rochdale rules’

6. “Being Co-op”
• Do what matters most
• Be yourself always
• Show your care
• Succeed together
6

7. Technology Drivers
Business Context
• Technology as an enabler for business growth & to streamline operations
• Acquisitions have increased diversity of technology landscape
Drivers
• Simplification and standardisation of Database & Middleware platforms
• Increase availability, scalability, agility – downtime is costly
• Strategic alignment of technology to business strategy, cloud-enabling capability
• Simplified & centralized Operations to improve SLAs
• Reduce OpEx following 2007 financial crisis
7

8. IT Context
Co-op
Stores
Distribution
Centres
Independent
Co-op Stores
Co-op
Data Centre
B
Support
Centre
Co-op
Data Centre
A
Cloud
2800 1500
121
70,000 colleagues
4,000 users
8
6.1 million members

9. Original Platform
Oracle
Database
Inst
Oracle
Database
Inst
SAN
Remote Mirroring
WAS
IHS IHS IHS IHS App Tier
Data Tier
Cold Failover Clusters with
HP Service Guard
DC1 DC2
WAS WAS WAS WAS WASiAS iAS
OHS OHS
iAS iAS
WLS WLS WLS WLS
OHS OHS
Not shown:
RAC
(for WMS+WFC)

10. Target (To-Be) Architecture

11. What is Maximum Availability
Architecture (MAA)?
• A dedicated team within Oracle
• Blueprints & tested configurations
• You may be “following MAA guidelines”
• Typically Active + Standby
• Started as Database team but
• also covers FMW & Engineered Systems
• focused around DR but some other RAS considerations too
11

12. To-Be Architecture – General Approach
RAC
Database
Inst Inst
RAC
Database
Inst Inst
Data
Guard*
WLS WLS WLS WLS WLS WLS
OHS OHS OHS OHS
OHS OHS OHS OHS
DMZ
App Tier
Data Tier
* ADG/GG in special cases
Autonomous
Domains
DC1 DC2
12

13. To-Be architecture – Web Tier
13
OHS OHS
OHS OHS
DMZ
Load
Bal
Load
Bal • Mirror env at other DC
• Minimal change in this tier
• Live & DR URLs accessible
Load
Bal
Load
Bal

14. To-Be architecture – App Tier
14
MS MS MS
MS MS MS
AS
• Mirror env at other DC
• 3 node clusters (n+1)
• Deploy to both Live & DR
domains simultaneously
• Live & DR connect to Live DB
• Active GridLink to connect to
RAC databases

15. To-Be architecture – Data Tier
15
• Data Guard to standby at
other DC (2-node RAC)
• 2 node clusters
• Active Data Guard
• GoldenGate
RAC
Database
Inst Inst

16. Example using Active Data Guard
16
2 node
RAC
Standby
2 node RAC
ADG
WebLogic
via
Active
GridLink
12 depots:
24h / 365d operation
ADG
PL/SQL extract
DG
Strict
SLA
Active
Active
Physical
Standby
2 node RAC
2 node
RAC
Standby
2 node RAC
2 node
RAC

17. More about our use of GoldenGate
17
Wednesday 8:50am

18. 18

19. Oracle Products in Use
Database
• EE (12.1.0.2) including NON-CDB, CDB with single PDB
• GI/RAC12c
• DG and ADG
• GoldenGate12cR2 with RAC12c (NON-CDB), DBFS, XAG
FMW
• WebLogic (12.1.3)
• Oracle HTTP Server (12.1.3)
• Forms & Reports 10.1.2.0.2
EM
• 12.1.0.4 inc WL EE pack, DB Diagnostics & Tuning packs, GG pack
19

20. Disaster Recovery
1862
1873
1844

21. Disaster Recovery Approach
Twin sites, ~symmetric, equal capacity
• Both sites run Live, DR & Test (fully licensed)
• Services spread: Active/Passive + Passive/Active
• Nothing stretched!
http://www.veriton.co.uk/roller/fmw/entry/deploying_soa_across_active_active
• Consciously no auto failover
• Exec level authorisation needed for whole DC
21

22. Design Patterns
22
Database Middleware
1a Single instance
DB
SAN replication WLS 1 or 3 nodes
1 domains/site
1b RAC
(1 PDB in CDB)
Data Guard WLS 3 node
2 RAC
(1 PDB in CDB)
Active Data Guard WLS 3 node
3 RAC
(non-CDB)
Data Guard
{& GoldenGate}
(non-Oracle)
Preferred

23. Traditional
23
Instance
WLS WLS WLS
OHS OHS
WLS WLS WLS
OHS OHS
Datafiles Datafiles
SAN Replication
Resync to switch back

24. New Way: MAA
24
Primary Standby
WLS WLS WLS
OHS OHS
WLS WLS WLS
OHS OHS

25. DR Notes
Data centres are as autonomous as possible
 No use of stretched VLANs
 No state in the middle tier (wherever possible)
 Dual domains – 2 copies of application
 Standby databases – 2 copies of data
25

26. DR Testing
DR switchover tests
 For Gold & Silver tiers: regular switchover to alternate DC 1
or 2 times per year
 Symmetrical: DR is sized as per production so no reduction
in performance when running on alternate site
 Typically tested on a “groups of applications” basis
 Data Guard switchover and DNS update for application
 Runs in alternate DC for a minimum of 7 days
26

27. Enterprise Manager
Cloud Control
1966

28. EMCC – Current Platform
28

29. EMCC - Dashboard
29

30. EMCC – Aggregate Service
30

31. EMCC – Middleware Dashboard
31

32. EMCC – 12c(CDB/PDB)
32

33. EM Target Architecture33

34. Lessons Learnt
&
Next Steps
34
1998

35. Findings & Lessons Learnt 1
• DMZ & green zone layers of OHS: WLProxyPassThrough &
WLProxySSL
• 4 standalone OHS (2 at each site)
o independent so not monitored through domain
o OHS as a target in EM CC instead (need WL EE Pack licence)
• No easy way to create standalone silently & UI doesn’t work
with VIP (12.1.3) – needed to make post creation changes
to nodemanager.properties & start script
• Co-located domain OHS domain can use WLST createOHSInstance()
• Pen-test: some changes needed (debatable but generated
by standard pen-test tools probably)

36. Findings & Lessons Learnt 2
• Performance issues on WLS 12.1.3 BP2 found in load
testing. No definitive outcome from SR – rolled out on base
release
• Not able to use FMWC (e.g. no DMS Spy which might have
been useful) due to unwanted DB dependency in 12c
o Now fixed with ‘Restricted JRF’ in 12.2
o We ended up installing FMW Infrastructure on a dev end to debug
• 12.1.0.2 Data Guard password configuration issue with DG
Broker

37. Benefits Realised
• Substantial cost savings over 3 years for:
• Platform support/updates
• OS/Infra support
• Hardware maintenance
• Simplified/consolidated
• Single development platform
• Optimisations/integration between Database & FMW
• Single pane of glass management
37

38. Next steps on the journey
• Single Instance upgrades  12c(R2?) RAC &
GridLink
• EM12c  13cR2 and MAA topology
• Upgrade Forms  12cR2 and Weblogic12c (R2)
• Simplify DB/MW to a single Linux platform
38

39. Future Plans
Exploiting more EM CC features
Cloudy services (IaaS, JCS, DBCS)
Hybrid Cloud – On-Premises with Public Cloud
39

40. For Further Information
40
Oracle MAA Pages
http://www.oracle.com/technetwork/database/features/availability/maa-096107.html

41. Thank You!
Q&A
https://en.wikipedia.org/wiki/One_Angel_Square
2013
Co-op HQ
One of the most sustainable large buildings in the world

42. Our Other Sessions
• Experiences at CIMA with SOACS – Mon 17:55
• 3 Ways to Connect to Oracle Cloud – Tues 11:35
• All about OVM Templates – Wed 08:50
• MAA with GoldenGate 12c – Wed 08:50
42

43. Contact us
@Nikitas_Xenakis
43
@simon_haslam