Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Dueling Duplications
Physical vs. Virtual Data Cloning

Steve Karam – oraclealchemist.com
Kyle Hailey – kylehailey.com
1
Your Presenters
• Kyle Hailey
– Oracle Ace, Oaktable Member
– A principle designer of Oracle EM performance pages
– Twitte...
Introduction
In this corner…we have RMAN!
•
•
•
•
•
•

Undisputed(ish) Oracle Database Heavyweight Backup Champion
Raised on the mean s...
And in this corner, the contender: Delphix!
•
•
•
•
•
•

The Agile powerhouse out of Menlo Park, CA
Light on its feet, and...
Why We Clone
There are plenty of good reasons to clone a database.
•Creating Dev, QA, Reporting, UAT, and other environmen...
The problem is…
Reports
Production

First copy

QA and UAT

•
•
•
•

CERN -  European Organization for Nuclear Research
14...
What We’ve Seen

1.
2.
3.
4.
5.

Inefficient QA: Higher costs of QA
QA Delays : Greater re-work of code
Sharing DB Environ...
1. Inefficient QA: Long Build times
Build

Build Time
Build Time

QA
Test

96% of QA time was building
environment
$.04/$1...
2. QA Delays:

bugs found late require more code re-

work
Build QA Env
Build QA Env

Sprint 1
X

Sprint 2

Q
A

Build QA ...
3. Full Copy Shared : Bottlenecks

Old Unrepresentative Data

Frustration Waiting
4. Subsets : cause bugs
4. Subsets : cause
bugs
The Production ‘Wall’
Classic problem is that queries
that run fast on subsets hit the
wall in pro...
5. Slow Environment Builds:
Slow Environment Builds: 3-6 Months to Deliver
Data
Developer
Asks for
Get Access
DB
Manager
D...
Why We Can’t Clone
And there are many reasons that we can’t make the clones we need.
•We don’t have the disk space for ano...
5. Slow Environment Builds: culture of no
DBA

Developer
Never enough environments
bottlenecks
What We’ve Seen

1. Inefficient QA: Higher costs
2. QA Delays : Increased re-work
3. Sharing DB : Bottlenecks
4. Subset DB...
The problem is…
The problem is that RMAN is still a
backup and recovery tool. It’s capable
of duplicating databases, it ha...
Round 1 RMAN
RMAN is…
The official description: Oracle Recovery Manager (RMAN) provides a
comprehensive foundation for efficiently back...
Backup and Recovery in RMAN
Backup and Recovery in RMAN is dead simple, particularly if the environment
is configured ahea...
So what does a backup look like?
RMAN backups are, of course, collections of files. What those collections look
like is de...
Surely it does more than that!
No doubt! RMAN is capable of many things, all centered around its ability as a backup and r...
And 12c action
•

Table and Pluggable Database Recovery
– Recover at the table level instead of the tablespace or datafile...
But can it clone?
Yes Virginia, RMAN can clone. Since RMAN’s primary functionality is to backup
databases and then restore...
So what’s the problem?
•

Backups take space.
– They can be compressed but slow, BZIP2, built in
– ZLIB fast but have to p...
•

Start RMAN backup
Round 2 Delphix
99% of blocks are identical

Clone 1

Clone 2

Clone 3
Thin Clone
Clone 1

Clone 2

Clone 3
Where we Are
Production
Instance

Database

File system
File system

QA

UAT

Instance

Instance

Instance

Database

Data...
Want be here
Production
Instance

Database

Development

QA

UAT

Instance

Instance

Instance

Database

Database

Databa...
Database Virtualization
Three Physical Copies

Three Virtual Copies

Data
Virtualization
Appliance
Install Delphix on x86 hardware

Intel hardware
Allocate Any Storage to Delphix

Allocate storage of any type
One time backup of source database
Production

Production

Instance

Database
File system

Beta or in Development
DxFS (Delphix) Compress Data
Production
Instance
Instance

Database

File system

Data is
compressed
typically 1/3
size
•

Start Delphix linking
Incremental forever change collection
Production
Instance
Instance

Database

File system

Changes
Time Window

• Collecte...
Source Full Copy

Source backup
from SCN 1
Snapshot 2
Snapshot 1
Snapshot 2
Snapshot 1
Backup from SCN
Snapshot 3
Snapshot 2
Snapshot 1
Snapshot 3
Snapshot 2

Drop
Snapshot 1
Cloning
Production
Instance

Inst

Instance
Instance

Database
Database

File system

Time
Window
Typical Architecture
Production
Instance

Database

File system
File system

Development

QA

UAT

Instance

Instance

Ins...
With Delphix
Production
Instance

Database

File system

Development

QA

UAT

Instance

Instance

Instance

Database

Dat...
Three Core Parts
Production
Instance

1
Source Sync’ing
Original copy
Pull in changes
Purging > time
window

Storage
File ...
Overview of Database Virtualization
1.
2.
3.
4.
5.

Fast, Fresh, Full
Free
Branching
Federated
Self Serve
Fast, Fresh, Full
Source

Development VDB

Instance

Instance

Time Window
Free

Source

Instance

Instance

Instance

Instance
Branching
Source
Instance

Dev
Instance

QA
branched from Dev
Instance
Federated
Instance

Source1
Instance

Instance

Source2
Instance

Source1
Source2
VDB Source 1
VDB Source 2
Battle Royale
Prepararation
•

Source and Target
– ORACLE_HOME /home/oracle/oracle1123/product
– Backup directory /home/oracle/rman

•

...
RMAN

Source
Source

Target
Target

/home/oracle/rman

/home/oracle/rman

/home/oracle/oradata
1. RMAN backup set

/home/o...
Delphix

Source
Source

Delphix
Delphix

/home/oracle/oradata
Initial Link

Target
Target
/home/oracle/oradata
Create VBD
Delphix
Source
Source

Delphix
Delphix

/home/oracle/oradata

Target
Target
/home/oracle/oradata

Initial Link

Target 2
T...
Delphix
Source
Source

Delphix
Delphix

/home/oracle/oradata

Target
Target
/home/oracle/oradata

Initial Link

Target 2
T...
Upcoming SlideShare
Loading in …5
×

Dueling duplications RMAN vs Delphix

2,684 views

Published on

Published in: Technology
  • Be the first to comment

Dueling duplications RMAN vs Delphix

  1. 1. Dueling Duplications Physical vs. Virtual Data Cloning Steve Karam – oraclealchemist.com Kyle Hailey – kylehailey.com 1
  2. 2. Your Presenters • Kyle Hailey – Oracle Ace, Oaktable Member – A principle designer of Oracle EM performance pages – Twitter: @kylehhailey – Blog: kylehailey.com • Steve Karam – Joined Delphix in December – Oracle Certified Master, ACE, and other acronyms – Just a little social • Blog: http://www.oraclealchemist.com/ • Twitter: @OracleAlchemist • Facebook: OracleAlchemist
  3. 3. Introduction
  4. 4. In this corner…we have RMAN! • • • • • • Undisputed(ish) Oracle Database Heavyweight Backup Champion Raised on the mean streets of Oracle 8 Became really useful in Oracle 9i Became really really useful in Oracle 10g Just keeps getting better and better Used as a tool for: – – – – Backup and Recovery Cloning Oracle Databases Cross Platform Migrations Taking up lots of disk space
  5. 5. And in this corner, the contender: Delphix! • • • • • • The Agile powerhouse out of Menlo Park, CA Light on its feet, and built for speed Compatible with 9.2.0.8 to 12c Provisions virtual databases (VDBs) off any Oracle or SQL Server Ready to disrupt your business practices—but in a good way Used as a tool for: – – – – – Cloning Oracle and SQL Server fast Reducing the storage Removing data movement bottlenecks Making sure everyone gets a copy of the database Making people jealous on Twitter
  6. 6. Why We Clone There are plenty of good reasons to clone a database. •Creating Dev, QA, Reporting, UAT, and other environments •Migrating data from one server to another •ETL or other warehouse workflow requirements •Data versioning •Data archiving
  7. 7. The problem is… Reports Production First copy QA and UAT • • • • CERN -  European Organization for Nuclear Research 145 TB database 75 TB growth each year Dozens of developers want copies. Developers
  8. 8. What We’ve Seen 1. 2. 3. 4. 5. Inefficient QA: Higher costs of QA QA Delays : Greater re-work of code Sharing DB Environments : Bottlenecks Using DB Subsets: More bugs in Prod Slow Environment Builds: Delays “if you can't measure it you can’t manage it”
  9. 9. 1. Inefficient QA: Long Build times Build Build Time Build Time QA Test 96% of QA time was building environment $.04/$1.00 actual testing vs. setup
  10. 10. 2. QA Delays: bugs found late require more code re- work Build QA Env Build QA Env Sprint 1 X Sprint 2 Q A Build QA Env Build QA Env Q A Sprint 3 Bug Code Cost To Correct Delay in Fixing the bug Software Engineering Economics – Barry Boehm (1981)
  11. 11. 3. Full Copy Shared : Bottlenecks Old Unrepresentative Data Frustration Waiting
  12. 12. 4. Subsets : cause bugs
  13. 13. 4. Subsets : cause bugs The Production ‘Wall’ Classic problem is that queries that run fast on subsets hit the wall in production. Developers are unable to test against all data ucti Prod n o
  14. 14. 5. Slow Environment Builds: Slow Environment Builds: 3-6 Months to Deliver Data Developer Asks for Get Access DB Manager DBA System Admin Storage Admin approves Request system Setup DB Request storage Setup machine Allocate storage (take snapshot)
  15. 15. Why We Can’t Clone And there are many reasons that we can’t make the clones we need. •We don’t have the disk space for another copy (or three) •We don’t have enough environments •The source database is too huge •There’s just not enough time
  16. 16. 5. Slow Environment Builds: culture of no DBA Developer
  17. 17. Never enough environments
  18. 18. bottlenecks
  19. 19. What We’ve Seen 1. Inefficient QA: Higher costs 2. QA Delays : Increased re-work 3. Sharing DB : Bottlenecks 4. Subset DB : Bugs 5. Slow Environment Builds: Delays
  20. 20. The problem is… The problem is that RMAN is still a backup and recovery tool. It’s capable of duplicating databases, it has great features for duplicating databases, but it’s not made to duplicate databases. It’s not made for agility. In short…
  21. 21. Round 1 RMAN
  22. 22. RMAN is… The official description: Oracle Recovery Manager (RMAN) provides a comprehensive foundation for efficiently backing up and recovering the Oracle database. So how does it do that? •Backups to disk or tape (or 3rd party media management layers) •Integrated catalog , retention policies and cleanup •Backup-time corruption detection •Full and incremental backup capabilities •Backup set and Copy backup types
  23. 23. Backup and Recovery in RMAN Backup and Recovery in RMAN is dead simple, particularly if the environment is configured ahead of time with proper paths and requirements. Backup the whole database: backup database; Restore and recover the whole database: restore database; recover database;
  24. 24. So what does a backup look like? RMAN backups are, of course, collections of files. What those collections look like is determined by the type of backup you’ve performed. • Backup as Backupset • Backs up non-empty blocks into one or more backupset files • Multiple datafiles per backupset file • Datafiles are reconstructed at restore time • Backup as Copy • Backs up complete files to complete copies, byte-for-byte, old school • Fast recovery • Takes up more space • Useful for incrementally updated backups, recover into full backup
  25. 25. Surely it does more than that! No doubt! RMAN is capable of many things, all centered around its ability as a backup and recovery tool. •Cross platform conversions – Convert a database from Windows to Linux, Linux to Windows, or even to platforms with a different endian format like SPARC Solaris •Incrementally updated backups – A single full backup copy that is recovered into over time by way of incremental backups •Proactive health checks – Database validation to spot signs of corruption •Compression – Compression during backup using BZIP2 or ZLIB •Cleanup policies – Using backup retention policies, it is easy to report on and delete unnecessary backup files
  26. 26. And 12c action • Table and Pluggable Database Recovery – Recover at the table level instead of the tablespace or datafile level, or recover pluggable databases with the Multitenant Database option • Simplified Cross Platform backup and recovery – Now you can use BACKUP TO PLATFORM and RESTORE FROM PLATFORM instead of doing separate conversions. Solaris: “Backup to Linux” • Cross Platform incremental backups – Make it possible to do cross platform conversions with minimal downtime • Multi-Section Image Copy and Incremental Backups – Divide individual data file backups into smaller sections to backup in parallel
  27. 27. But can it clone? Yes Virginia, RMAN can clone. Since RMAN’s primary functionality is to backup databases and then restore them, it makes sense that you could backup from one server and restore on another. Even better, RMAN has allowed duplicating with a relatively simple DUPLICATE command since Oracle 8i. This feature has of course grown better over time. As of Oracle 12c you can: •Duplicate from Backup ( only option in pre-12c) •Duplicate from an active database, no backup required •Use compression and multi-section backups for DUPLICATE FROM ACTIVE DATABASE commands 64% of DBAs use RMAN to clone
  28. 28. So what’s the problem? • Backups take space. – They can be compressed but slow, BZIP2, built in – ZLIB fast but have to pay advanced compression package • Clones take space – The cloned databases take up lots of space as well. They will always at least double your space usage. • Takes time – Even if space isn’t a concern, the process takes itme – While several features have sped up cloning with RMAN, you still need to physically copy a database from one system to another.
  29. 29. • Start RMAN backup
  30. 30. Round 2 Delphix
  31. 31. 99% of blocks are identical Clone 1 Clone 2 Clone 3
  32. 32. Thin Clone Clone 1 Clone 2 Clone 3
  33. 33. Where we Are Production Instance Database File system File system QA UAT Instance Instance Instance Database Database Database File system File system File system File system File system File system File system Development
  34. 34. Want be here Production Instance Database Development QA UAT Instance Instance Instance Database Database Database Snapshots
  35. 35. Database Virtualization
  36. 36. Three Physical Copies Three Virtual Copies Data Virtualization Appliance
  37. 37. Install Delphix on x86 hardware Intel hardware
  38. 38. Allocate Any Storage to Delphix Allocate storage of any type
  39. 39. One time backup of source database Production Production Instance Database File system Beta or in Development
  40. 40. DxFS (Delphix) Compress Data Production Instance Instance Database File system Data is compressed typically 1/3 size
  41. 41. • Start Delphix linking
  42. 42. Incremental forever change collection Production Instance Instance Database File system Changes Time Window • Collected incrementally forever • Old data purged
  43. 43. Source Full Copy Source backup from SCN 1
  44. 44. Snapshot 2 Snapshot 1
  45. 45. Snapshot 2 Snapshot 1 Backup from SCN
  46. 46. Snapshot 3 Snapshot 2 Snapshot 1
  47. 47. Snapshot 3 Snapshot 2 Drop Snapshot 1
  48. 48. Cloning Production Instance Inst Instance Instance Database Database File system Time Window
  49. 49. Typical Architecture Production Instance Database File system File system Development QA UAT Instance Instance Instance Database Database Database File system File system File system File system File system File system File system
  50. 50. With Delphix Production Instance Database File system Development QA UAT Instance Instance Instance Database Database Database
  51. 51. Three Core Parts Production Instance 1 Source Sync’ing Original copy Pull in changes Purging > time window Storage File System 2 Storage Snapshot Compress Share Cache Storage Agnostic Development Instance 3 Self Service Mount, recover, rename Roles & Security Rollback & Refresh Branch & Tag
  52. 52. Overview of Database Virtualization 1. 2. 3. 4. 5. Fast, Fresh, Full Free Branching Federated Self Serve
  53. 53. Fast, Fresh, Full Source Development VDB Instance Instance Time Window
  54. 54. Free Source Instance Instance Instance Instance
  55. 55. Branching Source Instance Dev Instance QA branched from Dev Instance
  56. 56. Federated Instance Source1 Instance Instance Source2 Instance Source1 Source2 VDB Source 1 VDB Source 2
  57. 57. Battle Royale
  58. 58. Prepararation • Source and Target – ORACLE_HOME /home/oracle/oracle1123/product – Backup directory /home/oracle/rman • Target Preparation – directories • • • • mkdir -p /home/oracle/oracle1123/fast_recovery_area/dup mkdir -p /home/oracle/oracle1123/admin/dup mkdir /home/oracle/oradata/dup mkdir -p /home/oracle/oradata/O1123/changetracking/ – init.ora /home/oracle/oracle1123/product/dbs/initdup.ora • added – *.db_file_name_convert='o1123','dup’ – *.log_file_name_convert='o1123','dup’ – Create a TNSNAMES entry for 'source’
  59. 59. RMAN Source Source Target Target /home/oracle/rman /home/oracle/rman /home/oracle/oradata 1. RMAN backup set /home/oracle/oradata 2. scp 3. RMAN duplicate
  60. 60. Delphix Source Source Delphix Delphix /home/oracle/oradata Initial Link Target Target /home/oracle/oradata Create VBD
  61. 61. Delphix Source Source Delphix Delphix /home/oracle/oradata Target Target /home/oracle/oradata Initial Link Target 2 Target 2 Create VBD /home/oracle/oradata
  62. 62. Delphix Source Source Delphix Delphix /home/oracle/oradata Target Target /home/oracle/oradata Initial Link Target 2 Target 2 /home/oracle/oradata Create VBD /home/oracle/oradata

×