Zero Data Loss Recovery Appliance - Deep Dive

BASLE BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA
HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH
ZDLRA – Deep Dive
Daniele Massimi – Principal Consultant
BE-IMS

Agenda
ZDLRA - in Action2 21.06.2017
1. ZDLRA – Functionality under the hood
2. Installation and Configuration from bottom up
3. Have a look inside – (just particularities)
4. From protected Databases to backups
5. Backup and Recovery Capabilities
6. Recovery Appliance Views and Utilities
7. Conclusion

Who am I
Daniele Massimi, Trivadis AG (CH, Bern)
Principal Consultant daniele.massimi@trivadis.com
Seit 2012 bei Trivadis IMS (Infrastructure Service Management)
Oracle DBA seit 2000 (Exadata > 6 Jahre)
Infrastruktur-Architekturen, Operations & Administration, Upgrades and Migration,
High Availability, Backup & Recovery, Virtualizierung
Engineered Systems (Exadata, ODA, ZDLRA, PCA)
Trainer (Oracle Architectur and Internal, 12c New Features, Exadata)
Oracle Certifications
Oracle Certified Master (11g)
Oracle Certified Professional (8i – 12c)
Oracle Certified Expert (RAC)
Oracle Implementation Specialist (Exadata, OVM)

Our company.
© Trivadis – The Company4 21.06.2017
Trivadis is a market leader in IT consulting, system integration, solution engineering
and the provision of IT services focusing on and
technologies
in Switzerland, Germany, Austria and Denmark. We offer our services in the following
strategic business fields:
Trivadis Services takes over the interactive operation of your IT systems.
O P E R A T I O N

ZDLRA
Functionality under the Hood

Common problems with Oracle backups
RPO depends on archive backup frequency (many minutes or hours)
Backup window and nightly batches compete for resources
High network bandwidth usage for full backups
Low-speed backups and restores
Backup and restore success ratio penalized by shared infrastructure
Critical databases require Data Guard for transaction safety
– Additional storage space, license and computation required
Expiration of backup set on ML: need to crosscheck and delete expired
Lack of backup validation
…

Recovery Appliance Architecture
Source: Oracle

Minimal Backup Overhead
8 21.06.2017
Traditional Backup
– Read/Write for Disk/Tape Backup
– Deduplication
– Compression
– Validation
– Deletion
– Merging
All on database host
 Degrade performance
Recovery Appliance
– Offloading operations
– Delta Push
– Delta Store
Source: Oracle

Delta Push
Delta Push is a highly optimized form of source-
side optimization
– Through RMAN block change tracking
– No reading of unchanged data
Two operations on the protected Database
– Incremental "forever" backup
– Real time redo transport
One-time full backup as prerequirement
Afterwards Incremental forever backup
Validate incoming Backups against corrupted
physical blocks
Compress the Backups using special block-level
Algorithm
Backuped Database
Changed Data
 Less data, less I/O, less network traffic
Source: Oracle

Eliminate data loss – Real Time redo transport
Real Time redo transport
– Data Guard like but asynchronous
– Changes out of Logbuffer transfered
– Validate and writes to a stage area
Redolog switch on database
– RA converts the staged redo data to a
compressed archived redolog backup
– Writes this archlog backup to catalog
– Ready for future restores
Explicit Archivelog backup no more necessary
 Reduces RPO to (near) zero
Source: Oracle

Delta Store – the “brains” of the Recovery Appliance
Backup
Totality of all Database Backup in a
Storage Location
validates the incoming blocks
compresses, indexes and stores
deduplicate, less storage usage
High number of virtual full backups
Higher recover window
Delta Store
Day N
Incr
Day 1 Virtual Full
Day N Virtual Full
Day 1
Incr
Day 0
Full
Source: Oracle

Delta Store – efficient restore/recover
Restore
Creates a Virtual Full Database
Backups
No need of transfer and apply
incremental backups during restore
operations
Roll forward with restored redologs
Uses Exadata performance and
scalability
Delta Store
Day 0
Full
Day 1
Incr
Day N
Incr
Day ‘N’ Full Backup
Source: Oracle

Delta Pools
Chunks inside the Delta Store
Backups will be indexed and stored inside the Delta Pools
Set of datafiles blocks from which RA constucts Virtual Full Backups
Each backuped datafile will have his own delta Pool
Automated delta pool space management
– Protection Policy will be inherited to database retention policy
– Delete automatically expired or obseleted Backups
– Periodically reorganising for performance improving
– Delta pool optimization when old blocks are deleted and new incremental Backup
arriving
Source: Oracle

Delta Push – how it works (1/2)
Clients address the HTTP Servlet on ZDLRA
RMAN ships incremental Backup to Staging Area
Data will be validated and Metadata will be stored on RAC Database
Data Blocks of Backupsets will be indexed and stored on Delta Store
Source: Oracle

Delta Push – how it works (2/2)
Optionally Real Time Redo Transport could be activated
ZDLRA will be using Data Guard Technology (RFS on RA)
Validated Redo Blocks will be stored on Delta Store
Archive Logs will be generated whenever a Log Switch happen on Prod DB
Optionally you can replicate or copy to a remote ZDLRA or Tape
Source: Oracle

Restore – how it works
Clients address the HTTP Servlet on ZDLRA
ZDLRA re-assemble and validate "virtual Backupsets"
Data Blocks will be shipped back to Client
Source: Oracle

Policy-Based Database Protection as a Service
ZDLRA Workshop - Introduction17 21.06.2017
Standardized
protection policies
– Recovery window
– Retention
– Replication
Gold Policy – Customer Critical
Disk: 45 days
Tape: 90 days
Silver Policy – Internal Critical
Disk: 30 days
Tape: 45 days
Bronze Policy - Test/Dev
Disk: 15 days
Tape: 30 days

Protection Policies Attributes
ZDLRA Workshop - Protection Policies18 21.06.2017
Storage Location  Recovery Appliance storage location for storing backups
Recovery Window Disk  Tape recovery window goal for the protected database
Recovery Window Tape  Tape recovery window goal for the protected database
Max Retention Policy  maximum length of time that the Recovery Appliance retains
backups for databases that use this retention policy
Guaranteed Copy  guaranteed copy setting, which determines whether backups
protected by this policy must be copied to tape or replicated before being considered
for deletion
Polling Policy  optional backup polling policy that determines whether Recovery
Appliance polls a storage location for backups or deletion
Unprotected Window maximum acceptable difference between the current time
and the latest time that the database can be restored

Recovery Window – Index Efficiency (1/2)
Only new version of Data Blocks will be backed up (inc level 1)
Recovery windows controls deletion policy of the blocks
Main scope is to maintain the restorability within the recovery windows
Old and no more needed blocks will be deleted
If enough space is available the Recovery Windows can be overachieved
Source: Oracle

Recovery Window – Index Efficiency (2/2)
Once the old blocks was deleted, we will have a defragmentation within the
storage
This situation will provide random I/O instead of sequential I/O  Performance
degradation
Intermittently the RA will automatically reorganize the blocks
Implicitly all touched blocks will be phisical and logical checked on his integrity
Source: Oracle

Storage Locations
Main Storage Location are the Container within the ASM Diskgroup
It is possible to have more than one Storage Location in ASM
Backup Polling Locations are Storage Location outside from ZDLRA (e.g. NFS Mount)
• Backups are placed directly to this location without interacting with ZDLRA
• With Polling Policies you can define how often and where
is the polling directory
• Once all requirement meets  e.g. backup related to a
protected database a copy to the RA will be created
• After copying process the protected database will be delete
the backup Automatically from polling directory Source: Oracle

Installation and Configuration
from bottom up

Installation Steps
Create the ZDLRA configuration using
the latest OEDA
Start the Re-Imaging and Setup from the first
Computing Node (like an Exadata Setup)
[root@zdldbadm01 linux-x64]# ./install.sh -cf ./WorkDir/zdl-zdl.xml –r 1-16

Installation Steps
Recovery Appliance specific Installation Steps
[root@zdldbadm01 install]# ./ra_install.pl --help
ra_install.pl - Recovery Appliance Installer
You must be logged in as root to run this command.
All steps should be run successfully for install.
Usage: ./ra_install.pl --help | --step=STEP_NUMBER {--install|--uninstall} [--
stdout]
Options:
--help: displays this help message
--step=number: Which step to run
--stdout: Print all messages to stdout rather then the log file
--install: Installs the step [Default]
--uninstall: Uninstalls the step
Step Numbers:
Step 1 - Validation and Configuration Prep
Step 2 - OS Setup
Step 3 - Oracle User Setup
Step 4 - DBFS Setup
Step 5 - Tape Backup configuration
Step 6 - ZDLRA DB Backup Setup
Step 7 - Enable ZDLRA Services

Configuration Steps
Deployment of the Agents on the compute Nodes
Recovery Appliance Discovery
Installation of the Backup Module on any Clients
[oracle@odax30 zdlra]$ java -jar ra_install.jar -dbUser ra_orcl02 -dbPass
manager -host zdlingest-scan -port 1521 -serviceName zdlra -walletDir
/u01/app/oracle/admin/orcl02/xdb_wallet -libDir $ORACLE_HOME/lib
Recovery Appliance Install Tool, build 2015-11-03
Recovery Appliance credentials are valid.
Recovery Appliance wallet created in directory
/u01/app/oracle/admin/orcl02/xdb_wallet.
Recovery Appliance initialization file
/u01/app/oracle/product/12.1.0.2/dbhome_1/dbs/raorcl02.ora created.
Downloading Recovery Appliance Software Library from file ra_linux64.zip.
Downloaded 27189305 bytes in 5 seconds. Transfer rate was 5437861 bytes/second.
Download complete.
[oracle@odax30 zdlra]$

Have a look inside
(just particularities)

On the computing Nodes
Metadata Database
– RAC Database (also used for Recovery Catalog (VPC))
– Non Container Database
– DBFS Mount-Points for Backup and Replication purpose
– DBFS added as Cluster Resources
– HTTP Servlet inside the database will be started for communication between the
clients and the ZDLRA (dbms_ra.startup_recovery_appliance)
oracle@zdldbadm01:~/ [+ASM1] df -h | grep dbfs
dbfs-@repdbfs.local:/
957M 120K 956M 1% /dbfs_repdbfs
dbfs-@obdbfs.local:/ 300G 23G 278G 8% /dbfs_obdbfs
SQL> exec dbms_ra.startup_recovery_appliance;
PL/SQL procedure successfully completed.

– Three new FS for internally purposes are provided
[oracle@zdldbadm01 ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VGExaDb-LVDbSys1
30G 15G 14G 54% /
tmpfs 253G 53M 253G 1% /dev/shm
/dev/sda1 504M 72M 407M 15% /boot
/dev/mapper/VGExaDb-LVDbOra1
99G 59G 36G 63% /u01
/dev/mapper/VGExaDb-LVRADump
296G 191M 281G 1% /radump  Dump destination for internal Stuff
/dev/mapper/VGExaDb-LVRABackup
99G 351M 94G 1% /rabackup
/dev/mapper/VGExaDb-LVOBIndex
296G 191M 281G 1% /obindex
dbfs-@repdbfs.local:/
957M 120K 956M 1% /dbfs_repdbfs
dbfs-@obdbfs.local:/ 300G 4.6M 300G 1% /dbfs_obdbfs

Backup of Metadata Database
– Done using Export of RASYS Schema every 2 Hours
– Backup Location on DBFS FS: /dbfs_obdbfs/OSB/ra_export
– Scheduling via "anacron"
[root@zdldbadm01 ~]# cat /etc/anacrontab
SHELL=/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
# the maximal random delay added to the base delay of the jobs
#RANDOM_DELAY=45
# the jobs will be started during the following hours only
START_HOURS_RANGE=3-22
#period in days delay in minutes job-identifier command
1 65 cron.daily nice run-parts /etc/cron.daily
7 70 cron.weekly nice run-parts /etc/cron.weekly
30 75 cron.monthly nice run-parts /etc/cron.monthly
[root@zdldbadm01 cron.hourly]# ls -l /etc/cron.hourly/
total 4
-rwx------ 1 root root 409 Nov 10 2015 0anacron
lrwxrwxrwx 1 root root 46 Aug 25 16:20 ra_export.pl -> /opt/oracle.RecoveryAppliance/bin/ra_export.pl

ASM Configuration
– Two Diskgroups  DELTA (for Backups), CATALOG (Metadatas)
– New special sub-Directory CONTAINER for storing Container Files which
contains Delta Pools and Delta Stores
– Each Container File as a size of 2 TB
ASMCMD [+delta/zdlra] > ls -l
Type Redund Striped Time Sys Name
Y ARCHIVELOG/
Y CONTAINER/
Y DATAFILE/
Y TEMPFILE/
ASMCMD [+delta/zdlra/CONTAINER] > ls -s
Block_Size Blocks Bytes Space Name
512 4294959104 2199019061248 4398059094016 con.365.920823681
512 4294959104 2199019061248 4398059094016 con.366.920823687
512 4294959104 2199019061248 4398059094016 con.367.920823699
...

On the cell Nodes
Fortunatelly nothing special ☺
Same appearance as in Exadata Storage Cell
During the Installation the "Write Back" Option will be enabled automatically for the
Flashcache
Little difference on the Grid Disks compared with the Exadata, only 10 Grid Disks
instead of 12 for the CATALOG ASM Disk Group

From protected Databases
to backups

Protected Databases
Once the Environment meets all the requirements we can add the databases to the
ZDLRA
The requirement are:
• Network connectivity
• Backup Module is installed for every Oracle Home
• Definition of Protection Policies are created (Retention Time)
 Block Change Tracking not mandatory, but recommended !!!
Two Methods are possible:
• Enterprise Manager
• Manually  DBMS_RA PL/SQL Package

Prerequisite for Adding Protected Database
(EM and manually)
Create an VPD User on the Metadata Database
Grant the Create Session privilege
SQL> create user ra_orcl03 identified by <pwd>;
SQL> grant create session to ra_orcl03;

Adding Protected Database with EM
Add Protected Database on Recovery Appliance Page
– Choose the Protection Policy

Adding Protected Database with EM
Configuring Backup Settings
– Register database
– Add RMAN configuration (configure...)
– Enabling Redo Transport
(add parameter to spfile and restart DB)

Scheduling Protected Database with EM
Protected Database is ready for backing up
– We need a Schedule

Configure the repeating interval for
the Incremental-Forever Backups
Configure the Archive Log Backups and
deletion rule

Also useful as Script Generator  here the Script is not modifiable !!!

Output of an Backup Schedule on EM13c

Adding Protected Database manually
Add Database for the protected database
Grant access to Virtual Private Catalog
Configure the RMAN Channel for ZDLRA
begin
dbms_ra.add_db (
db_unique_name => 'orcl03',
protection_policy_name => 'bronze',
reserved_space => '250G');
end;
begin
dbms_ra.grant_db_access (
db_unique_name => 'orcl03',
username => 'ra_orcl03');
end;
CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' FORMAT '%d_%U' PARMS
"SBT_LIBRARY=/u01/app/oracle/product/12.1.0.2/dbhome_1/lib/libra.so,
ENV=(RA_WALLET='location=file:/u01/app/oracle/product/12.1.0.2/dbhome_1/dbs/ra_wallet
credential_alias=zdlingest-scan:1521/zdlra:dedicated')";

Adding Protected Database manually
Enable Real-Time Redo Transport
Restart the Database
ALTER SYSTEM SET REDO_TRANSPORT_USER=ra_orcl03 SCOPE=BOTH;
[oracle@odax30 ~]$ srvctl stop database -db orcl03
[oracle@odax30 ~]$ srvctl start database -db orcl03

Execute Backup manually
Backups works like in traditional way
Only "forever" incremental level 1 backups should be schedule
Level 0 Backup are of course always possible
It works also with old fashion syntax  soft link from libra.so to libobk.so is
needed !!!
Scheduling is also free selectable, EM is recommended !
[oracle@odax30 ~]$ rman target / catalog ra_orcl02/manager@zdlingest-scan:1521/zdlra
...
connected to target database: ORCL02 (DBID=3592015831)
connected to recovery catalog database
RMAN> backup incremental level 1 database;
run {
allocate channel c1 type sbt_tape;
backup incremental level 1 database plus archivelog delete input;
}
...
ORA-27211: Failed to load Media Management Library

Copy to Tape

Purpose of Copy to Tape
Tape Infrastructure provides a second or third copy of your backups
Tape Infrastructure could be located on different Data Center  DR capability !
Tape backups are "cheaper" for long-term Storage
High efficiency is given in combination with ZDLRA
Oracle Secure Backup just pre-installed on ZDLRA

Setting up Copy to Tape
Configuring Media Management Library
• Library Name
• Number of Drives
• Number of Restore Drives
• Media Manager Location (libobk.so)

Setting up Copy to Tape Template
Template helps to define specific attributes
• Backup types
• Database or Protection Policies
• Recovery Windows will be inherited
• Number of copies
• Runtime Window
• Schedule

Executing Copy to Tape
With the scheduling will be iniatiate the copy to tape process
backupset of defined database or protection policy will be copied to tape
Long Term Backup should be done with copy_backup procedure
Restores of backupset from tape will be retrieved transparently !!!

Recovery Capabilities

General statement about Recoveries of protected
Databases
Recoveries of protected databases can be done with EM or manually
With EM several Recovery-Scenarios are possible
All Recovery scenarios with EM are full automated
(e.g. Creation of auxiliary Databases, etc.)
EM generates RMAN Scripts for each Scenario
with the possibility of adaption
Possibility of selection between full and point-in-time
recoveries
Duplicate Database (for standby) need some preparation
(e.g. FS-Structure (admin tree))

Protected Database Recovery with EM
From database Page under Availability  Backup & Recovery  Perform Recovery

Choose the needed Recovery Scenario
... You can choose an other restore
location if needed...

A schedule will be created
... And you can adapt the script if needed and submit the job...

The Job can be monitored by clicking the "View Job" button...

Protected Database Recovery manually
Restore can be done also manually
– of course db must be mounted before...
– RMAN Client should be configured otherwise parameter are needed (e.g. ENV=...)
Or a point-in-time Recovery
run {
restore database;
recover database;
alter database open;
}
run {
set until time '08.09.2016 08:45:00';
restore database;
recover database;
alter database open resetlogs;
}

Real Time Redo Transport – what happens really ?
We tested what happens really when the protected database crash…
▪ We did a backup of a protected database with enabled real time redo transport
▪ then we shutdown the database inconsistently (abort)… before we checked the actual SCN
▪ the ZDLRA figure out the crash of the protected database and catalog the redo stream from the
staging area as archivelog with the latest SCN he know… that will be latest consistent SCN for
restore and recover the database
RMAN> backup incremental level 1 database plus archivelog delete input;
SQL> select current_scn from v$database;
CURRENT_SCN
-----------
28933241
SQL> shutdown abort
RMAN> list backup of archivelog all;
.
.
1 26 28933197 18-APR-17 28933202 18-APR-17
1 27 28933202 18-APR-17 28933232 18-APR-17
1 28 28933232 18-APR-17 28933246 18-APR-17

Recovery Appliance Views and
Utilities

Some RA Views
It is also possible to read the Recovery Appliance Information over Views from Recover
Appliace Metadata Database
View Name Description
RA_DATABASE Information about Protected Databases
RA_DB_ACCESS Describes User Accounts that have access to which protected database
RA_PROTECTION_POLICY Protection Policies defined on the Recovery Appliance
RA_DATABASE_STORAGE_USAGE Storage usage for each protected database
RA_RESTORE_RANGE Describe Restore Range for each protected database
RA_INCIDENT_LOG Describe Recovery Appliance Incidents
RA_REPLICATION_SERVER Replication Server Information (if used)

rastat.pl Utility
rastat Utility help to evaluate the performance of Recovery Appliance
– NETBACKUP/NETRESTORE
• Measure the network performance
– ASMREAD/ASMWRITE
• Measure the ASM Disk I/O Performance
– CONTAINERREAD/CONTAINERWRITE/CONTAINERALLOC
• Measure the Container Disk I/O Performance
[oracle@zdldbadm01 ~]$ perl /opt/oracle.RecoveryAppliance/client/rastat.pl --test=CONTAINERWRITE --
filesize=2048M --rasys=rasys/manager@zdlingest-scan:1521/zdlra --diskgroup=/:DELTA
RECOVERY APPLIANCE WRITE IO TEST TO CONTAINER FILES
Disk Group: /:DELTA
2048 MB, .75s write IO time, .51s CPU time, 2724.80 MB/s, 68.51% CPU usage
PL/SQL procedure successfully completed.

dbms_ra Package
dbms_ra provides administration function from inside the RA Database
Ca. 50 Procedure for different types of administration
– Start/Stop of Recovery Appliance
– Add/Delete Protected Databases
– Grants Access to specific User to enable to backup and restore
– Add/Update Protection Policies
– Manage Tape Configuration (if available)
– Add Repliaction Server (if available)
But, the utilization of Enterprise Manager is recommended !!!

Conclusion

Conclusion
ZDLRA worked in our Tests as
expected
Very good implementation of well
known Backup and Recovery
Technology on Engineered System
RPO is (near) zero with Real Time
Redo Transport
Even RTO ist much better then
existing systems
Very simple Management with EM
 Risk reduced, cost reduced,
qualitiy improved
 ZDLRA will find it’s
customer !

Questions and Answers...
Daniele Massimi – Principal Consultant
Tel. +41 58 459 50 92
daniele.massimi@trivadis.com
21.06.2017 ZDLRA - in Action63

Zero Data Loss Recovery Appliance - Deep Dive

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Zero Data Loss Recovery Appliance - Deep Dive

Similar to Zero Data Loss Recovery Appliance - Deep Dive (20)

Recently uploaded

Recently uploaded (20)

Zero Data Loss Recovery Appliance - Deep Dive