Oracle RAC Presentation at Oracle Open World

Oracle on Linux at BA From Evaluation To Implementation 15 th November 2007

Imtiaz Mazhary Technical Architect Paul Marden Project Manager

little teaser … SQL> create tablespace IMTEST_TS datafile '+DGTEST' size 10m; create tablespace IMTEST_TS datafile '+DGTEST' size 10m * ERROR at line 1: ORA-01119: error in creating database file '+DGTEST' ORA-17502: ksfdcre:4 Failed to create file +DGTEST ORA-15001: diskgroup "DGTEST" does not exist or is not mounted SQL> create diskgroup DGTEST 2 normal redundancy 3 failgroup DGTEST_BH 4 disk '/dev/raw/raw1001' name DGTEST_BH_DISK1001 5 failgroup DGTEST_CB 6 disk '/dev/raw/raw2001' name DGTEST_CB_DISK2001; Diskgroup created.

little teaser …

SQL> create tablespace IMTEST_TS datafile '_DGTEST' size 10m;

Tablespace created.

SQL> !ls -lart $ORACLE_HOME/dbs | grep DGTEST -rw-rw---- 1 oracle oinstall 1056768 Sep 4 12:32 _DGTEST Bug 6391521 RAC DATABASE DATAFILE CREATION ALLOWS SPECIFICATION OF NON-SHARED STORAGE

Oracle on Linux Evaluation

Proposition

Cost reduction

Improved resilience

Simplification

People

Infrastructure Design & Delivery

DBA Teams

Linux Infrastructure

Oracle RAC Pack, IBM, Red Hat, Scaleabilities

Oracle on Linux Evaluation

Platform Design Stage

Design Patterns

Technology Stack

Areas of Testing

List of Issues

Deliverables

BA, Oracle and IBM Validated Configuration

Design Patterns – Database Tier

SAN Storage

DB01 Service X Basic Pattern Application X

Design Patterns – Database Tier SAN Storage Application X DB10R1 Service X DB10R2 Service X RAC Lite Pattern

Design Patterns – Database Tier SAN Storage DB10R1 Service X DB10R2 RAC Lite Pattern for Consolidation Application Y Application X Service Y Service X Service Y Service Y Service X

Design Patterns – Database Tier Application X DB20R1 Service X DB20R4 DB20R2 Service X DB20R3 Service X Service X Full RAC Pattern SAN Storage

Technology Stack IBM SDD v’s RHEL Device Mapper 3 rd Voting Disk on NFS

Areas of Testing

Operating Environment

RHEL 4.5 – CFS, LVM2, Device Mapper (and IBM SDD)

SAN Attachment – QLogic vs RHEL QLogic driver

RHEL 4.5 Cluster Suite

BMC Patrol, EMC Legato Networker, IBM MQ Series, Control-M . . . .

Oracle DB 10g R2 (142 tests)

Serviceability – build, patching & configuration

Clusterware resilience – interconnect, OCR & voting disks

ASM configuration

ASM resilience & recovery

Stretched cluster performance

Client connectivity & failover

Backup & recovery, using EMC Legato Networker

System Monitoring, using BMC Patrol

EM Grid Control

List of key issues – all fixed in 10.2.0.3.0 ` Similar to the above 5621599.992 OCR synchronisation behaviour OCR storage too small when replacing the failed OCR disk 5698198.992 Failure to replace OCR device Bug – back port issue 5682851.993 DBVerify Segmentation Fault Internal RAID array firmware issue? 1768384 (RH) "read only filesystem" error Feature - to prevent OCR data loss 5593646.992 Loss of one OCR causes CW errors Bug - CSS error when processing stale member data requests 5594045.992 ASM & CW crashes/reboots on loss of one storage array Bug - Block corruptions when dropping & adding failure groups 5556890.993 ASM block corruption rootaddnode.sh using incorrect VIP 5449529.993 Problems adding additional node Bug – RHEL libaio driver 5473171.993 885818 (RH) Kernel panic caused by aio_put_req Use of NFS for 3 rd voting disk 5527179.993 CLSFMT error on CW install

Evaluation Project Deliverables

Tested reusable patterns for use across the airline

Supplier verification of our infrastructure - Oracle Validated Configuration undertaken by:

IBM

Oracle

Red Hat

OVC link: http://www.oracle.com/technology/tech/linux/validated-configurations/index.html

Database Consolidation

Proposition

reduce the size of Oracle DB estate

cost reduction / simplification

first RAC Lite and Basic Pattern – learning exercise

Process

LEAN - Runners / Repeaters

pre-provisioning

People - internal, RAC Pack

IT - criteria, examples, experiences

FTD Batch (lxlhrl214) OAW Batch (lxlhrl213) Database Service - RAC Lite Pattern BH CB Batch/Feed Service (Linux HA) Node X ConMan Batch Node Y VIP X RACDBR1 VIP Y DGDATA_BH DGFRA_BH Public Network RAC Network Storage Network Database Consolidation - architecture DGDATA_CH DGFRA_CH ConMan Batch RACDBserviceFAST RACDBserviceB RACDBserviceC RACDBserviceD EXT3 RACDBR2 RACDBserviceC RACDBserviceD RACDBserviceFAST RACDBserviceB FAST Batch FAST Batch VIP X Database Service Basic Pattern DB1 DBservice_ConMan DBservice2 DBservice3 … . DB2 DBservice1 DBservice2 … . RACDBserviceFAST RACDBserviceB

Database Consolidation – Progress so far

Current Production Environment:

30+ applications consolidated

3 on RAC Lite

Capacity provisioned for 20 more

Experiences

Very few technical issues with infrastructure

Mainly classic problems associated with migration

Database Consolidation - Criteria Any 4GB or less 2GB or less Memory (SGA + User) Any Greater than 3,600 per hour Less than 3,600 per hour Business transactions Any Less than 50 10 or less Concurrent clients (cpu constraint) Any Less than 50% Less than 25% Overall server capacity (cpu + memory constraint) No Yes or No Yes 3 rd party supplier contract does preclude database consolidation OLTP or Warehouse / DSS OLTP OLTP Transaction type IBM xSeries (standard BA standard model) or other Intel platform IBM xSeries (BA standard model) IBM xSeries (BA standard model) Hardware RHEL4 64 bit or other RH release RHEL4 64 bit RHEL4 64 bit Operating platform Full RAC pattern or other Basic or RAC Lite Pattern Basic or RAC Lite pattern Pattern type Any Greater than 60GB 60GB or less Data size (storage constraint) Oracle 10g RAC Oracle 10g DB or Oracle 10g RAC Oracle 10g DB or Oracle 10g RAC Database Stranger Repeater Runner

DW to Full RAC Pattern

Proposition

first Full RAC implementation – learning exercise

important but not key business component

Process - LEAN -- Strangers

People - internal, NIIT, RAC Pack, Scaleabilities

IT

Where are we coming from?

What does the cluster look like?

Gotchas / lessons learnt?

Legacy platform

Old SUN ES6500, 16*400MHz CPUs, 20GB RAM

Solaris 8, Oracle 9.2

1.6 TB Database …. Operational Activity Warehouse & FTD

300+ batch feed jobs

Oracle Forms 6!

MQ Series licence costs for 16 CPUs

Severe performance problems

No HA

FTD Batch (lxlhrl214) OAW Batch (lxlhrl213) FICO DCS StaffAdmin Baggage … .. … .. WA21R Database - Full RAC Pattern VIP1 BH CB Batch/Feed Service (Linux HA) Node X OAW Batch (lxlhrl213) FTD Batch (A) Node Y FTD Batch (lxlhrl214) OAW Batch (A) VIP2 DGDATA_CB DGFRA_CB DGDATA_BH DGFRA_BH Public Network RAC Network Storage Network Target Platform WA21R1 WAPRD_BATCH WAPRD_APP FDPRD_APP WA21R2 FDPRD_APP WAPRD_APP WAPRD_BATCH VIP3 WA21R3 WAPRD_APP FDPRD_APP FDPRD_BATCH VIP4 WA21R4 FDPRD_BATCH FDPRD_APP WAPRD_APP

DW to Full RAC Pattern