What is Delphix
kylehailey.com/delphix

Use Cases
What is
Delphix

Competition
What is Delphix
Three Physical Copies

Three Virtual Copies

Delphix
Install Delphix on Intel commodity hardware

Intel hardware
Allocate Any Storage to Delphix

Allocate Storage
Any type

Pure Storage + Delphix
Better Performance for
1/10 the cost
One time backup of source database
Production

Supports

Instance
Instance
Instance

Database

File system

Application St...
DxFS (Delphix) Compress Data
Production
Instance
Instance
Instance

Database

File system

Data is
compressed
typically 1/...
Incremental forever change collection
Production
Instance
Instance
Instance

Database

Changes
Time Window

File system

•...
Change collection works with snapshots

Snapshot 0:
Source Full Copy

Snapshot 1:
Changed blocks at
time T1
Snapshots immediately look like full copies as they are
mapped
onto the previous copy
Snapshot 0

Snapshot 1
Snapshot 0

Snapshot 1

Snapshot 2: Changed
blocks at time T2
Snapshot 0

Snapshot 2
Snapshot 1
Purging old snapshot releases any blocks that have
been since replaced, but keeps all unchanged blocks
Snapshot 2
Snapshot...
Cloning : almost no storage at creation since it is all
shared. Upon usage changed blocks will add to storage
used
Product...
Typical Architecture

Production

Dev, QA, UAT

Reporting

Instance
Instance
Instance
Instance
Instance
Instance

Instance...
With Delphix
Production
Instance
Instance

Database

File system

Dev & QA
Instance
Instance
Instance
Instance
Instance
In...
Delphix Use Cases

1.
2.
3.
4.
5.

Fast, Fresh, Full
Free
Branching
Federated
Self Serve
Fast, Fresh, Full
Source

Development VDB

Instance

Instance

Time Window
Free

Instance

Source
Instance

Instance

Instance

gif by Steve Karam
Branching
Source
Instance

branching

Dev
Instance

Source

QA
branched from Dev
Instance
Federated Cloning
Federated
Source1

Instance

Source1
Instance

Source2
Instance

Instance
“I looked like a hero”
Tony Young, CIO Informatica
Self Service
Use Cases
1.

Development Acceleration

2.

Quality

3.

BI
DevOps
DevOps With Delphix

1.
2.
3.
4.
5.

Efficient QA: Low cost, high utilization
Quick QA : Fast Bug Fix
Every Dev gets DB: P...
Without Delphix
1. Inefficient QA: Long Build times

Build

QA Test
Build Time

96% of QA time was building environment
$....
: With Delphix
1. Efficient QA: Lower cost

Build

QA Test
Build Time
B
u
i
l
d

T
i
m
e

QA Test

1% of QA time was build...
Rapid QA via Branching
Without Delphix
2. QA Delays: bugs found late require more code re-work
Build QA Env

Sprint 3

Sprint 2

Sprint 1

X

Bui...
: With Delphix
2. QA Immediate: bugs found fast and fixed
Build QA Env

Sprint 2

Sprint 1

X

Q
A

Build QA Env

Q
A

Spr...
Without Delphix
3. Full Copy Shared : Bottlenecks

Old Unrepresentative Data

Frustration Waiting
: With Delphix
3. Private Copies: Parallelize

gif by Steve Karam
Without Delphix

4. Subsets : cause bugs
Without Delphix
4. Subsets : cause
bugs
The Production ‘Wall’

Classic problem is that queries that
run fast on subsets hi...
: With Delphix
4. Full Size DB : Eliminate bugs
Without Delphix
5. Slow Environment Builds:

Slow Environment Builds: 3-6 Months to Deliver
Data
Developer
Asks for DB
Get...
Without Delphix
5. Slow Environment Builds: culture of no
DBA

Developer
Without Delphix
Never enough environments
: With Delphix
5. Self Service: Fast, Efficient. Culture of Yes!
Quality

1. Forensics
2. Testing
3. Recovery
1. Forensics: Investigate Production Bugs
Development
Instance

Instance

Time Window

Anomaly on Prod
Possible code bug
A...
2. Testing : Rewind for patch and QA testing
Prod

Development

Instance

Instance

Time Window

Time Window
2. Testing: A/B

Instance

Test A with Index 1

Instance

Instance

Time Window

• Keep tests for compare
• Production vs ...
3. Recovery: Surgical recover of Production
Source

Development

Instance

Instance

Spin VDB up
Before drop
Time Window
P...
3. Recovery Surgical or Full Recovery on VDB
Dev1 VDB
Source

Instance

Instance

Dev2 VDB Branched
Source
Time Window
Dev...
3. Recovery: Virtual to Physical
Source

VDB

Instance

Instance

Spin VDB up
Before drop
Time Window

Corruption
50 days of backup in size of
Original

3. Recovery
Business Intelligence
ETL and Refresh Windows

1pm
noon

10pm

8am
ETL and DW refreshes taking longer

1pm
noon

10pm

2011
2012
2013
2014
2015

8am
ETL and Refresh
Windows
Globalization limits ETL windows to
weekends

6am

8am

10pm

10pm

1pm
noon

8am

10pm

2011
2012...
ETL and DW Refreshes
Prod

DW & BI

Instance

Instance

Data Guard – requires full refresh if used
Active Data Guard – rea...
Fast Refreshes
• Collect only Changes
• Refresh in minutes

Prod
Instance

BI

DW

Instance

Instance

ETL
24x7
Temporal Data
Oracle 12c
80MB buffer cache ?
200GB
Cache
with

Latency

Tnxs / min

5000

300
ms

1

5

10 20 30 60 100 200

Users

1

5

10 20 30 60 100 200
Latency

Tnxs / min

8000

600
ms

1

5

10 20 30 60 100 200

Users

1

5

10 20 30 60 100 200
Five 200GB database copies are
cached with :

$1,000,000
1TB cache on SAN

$6,000
200GB shared cache on Delphix
Data Center Migration : clone migration

5x Source Data Copy

< 1x Source Data Copy
Data Center Migration : clone migration + source

S

S

5x Source Data Copy
< 2 x Source Data Copy
Data Center Migration : clone migration + source

S

C

C

C

5x Source Data Copy

C

S

V

V

V

< 1 x Source Data Copy

...
Consolidation
Without Delphix
Active

Active

With Delphix
Idle
Active

Idle
Active
DBMS License Utilization

•

Most DBMS time spent waiting on environment setup, data loads, backups,
etc.
•
Coupling of da...
Dev
v2.6

v2.6

QA
v2.6

v2.6

UAT
v2.6

v2.6

v2.6

v2.7

v2.6

v2.7

v2.7

v2.6

v2.6

v2.6

v2.8

v2.8

v2.6

v2.8
Production

Dev
v2.6

v2.6

QA
v2.6

v2.6

UAT
v2.6

v2.6

v2.6

v2.7

v2.6

v2.7

v2.7

v2.6

v2.6

v2.6

v2.8

v2.8

v2....
2.6

branching

Dev
Prod

checkout

bookmark
2.6
QA
Dev
Prod
2.6
UAT
QA
Dev
Prod
2.6
UAT
QA
Dev
Prod
Dev
QA
UAT

2.7
2.6

2.8

UAT
QA

UAT
QA

Dev

Dev

Prod
Dev
QA

UAT

2.7
2.6

2.8

UAT
QA

UAT
QA

Dev

Dev

Prod
Dev
QA

UAT

2.7
Data Control = Source Control for the Database
2.6

2.8

UAT
QA

UAT

QA

Dev

Dev

Production Time Flow
Dev
QA
UAT

2.7

Data Control = Source Control for the Database
What is Delphix
Upcoming SlideShare
Loading in...5
×

What is Delphix

1,760

Published on

0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,760
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
73
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • Interview Delphix blew me away. As a DBA I had to spend 50% of my time making copiesAfter joining Delphix, I banged on Delphix for 2 years. It worksBlog entryReinforce ideas we’ve already seen from a different perspectiveWork for a company called DelphixWe write software that enables Oracle and SQL Server customers toCopy their databases in 2 minutes with almost no storage overheadWe accomplish that by taking one initial copy and sharing the duplicate blocks Across all the clones
  • In the physical database world, 3 clones take up 3x the storage.In the virtual world 3 clones take up 1/3 the storage thanks to block sharing and compressionDelphix orchestrates access to the shared compressed blocks on the storage that Delphix manages.Delphix manages changes block access only making changed blocks visible to the database that made that block change.
  • Software installs an any x86 hardware uses any storage supports Oracle 9.2-12c, standard edition, enterprise edition, single instance and RAC on AIX, Sparc, HPUX, LINUX support SQL Server
  • EMC, Netapp, Fujitsu, Or newer flash storage likeViolin, Pure Storage, Fusion IO etc
  • Delphix does a one time only copy of the source database onto Delphix
  • Physically independent but logically correlatedCloning multiple source databases at the same time can be a daunting task
  • One example with our customers is InformaticaWho had a project to integrate 6 databases into one central databaseThe time of the project was estimated at 12 monthsWith much of that coming from trying to orchestratingGetting copies of the 6 databases at the same point in timeLike herding cats
  • Informatical had a 12 month project to integrate 6 databases.After installing Delphix they did it in 6 months.I delivered this earlyI generated more revenueI freed up money and put it into innovationwon an award with Ventana Research for this project
  • Developer each get a copyFast, fresh, full, frequentSelf serviceQA branch from DevelopmentFederated cloning easyForensicsA/B testingRecovery : Logical and physical Development Provision and RefreshFullFreshFrequent (Many) Source control for code, data control for the database Data version per release version Federated cloning QA fork copies off to QA QA fork copies back to Dev Instant replay – set up and run destructive tests performance A/B Upgrade patching Recovery Backup 50 days in size of 1 copy, continuous data protection (use recent slide ob backup schedules full, incr,inrc,inrc, full) Restore logical recovery on prod logical recovery on Dev Debugging debug on clone instead of prod debug on data at the time of a problem Validate physical integrity (test for physical corruption)
  • We talked to Presbyterian HealthcareAnd they told us that they spend 96% of their QA cycle time building the QA environmentAnd only 4% actually running the QA suiteThis happens for every QA suitemeaningFor every dollar spent on QA there was only 4 cents of actual QA value Meaning 96% cost is spent infrastructure time and overhead
  • Presbyterian when from 10 hour builds to 10 minute buildsTotal Investment in Test Environment: $2M/year10 QA engineersDBA, storage team dedicated to support testingApp, Oracle server, storage, backupsRestore load competes with backup jobsRequirements: fast data refresh, rollbackData delivery takes 480 out of 500 minute test cycle (4% value)$.04/$1.00 actual testing vs. setup
  • Because of the time required to set up QA environmentsThe actual QA tests suites lag behind the end of a sprint or code freezeMeaning that the amount of time that goes by after the introduction of a bug in code and before the bug is found increasesAnd the more time that goes by after the introduction of a bug into the codeThe more dependent is written on top of the bug Increasing the amount of code rework required after the bug is finally foundIn his seminal book that some of you may be familiar with, “Software Engineering Economics”, author Barry Boehm Introduce the computer world to the idea that the longer one delays fixing a bug in the application design lifescyleThe more expensive it is to to fix that bug and these cost rise exponentially the laterThe bug is address in the cycle
  • Not sure if you’ve run into this but I have personally experience the followingWhen I was talking to one group at Ebay, in that development group they Shared a single copy of the production database between the developers on that team.What this sharing of a single copy of production meant, is that whenever a Developer wanted to modified that database, they had to submit their changes to codeReview and that code review took 1 to 2 weeks.I don’t know about you, but that kind of delay would stifle my motivationAnd I have direct experience with the kind of disgruntlement it can cause.When I was last a DBA, all schema changes went through me.It took me about half a day to process schema changes. That delay was too much so it was unilaterally decided byThey developers to go to an EAV schema. Or entity attribute value schemaWhich mean that developers could add new fields without consulting me and without stepping on each others feat.It also mean that SQL code as unreadable and performance was atrocious.Besides creating developer frustration, sharing a database also makes refreshing the data difficult as it takes a while to refresh the full copyAnd it takes even longer to coordinate a time when everyone stops using the copy to make the refreshAll this means is that the copy rarely gets refreshed and the data gets old and unreliable
  • For example Stubhub went from 5 copies of production in development to 120Giving each developer their own copy
  • To circumvent the problems of sharing a single copy of productionMany shops we talk to create subsets.One company we talked to , RBS spends 50% of time copying databases have to subset because not enough storagesubsetting process constantly needs fixing modificationNow What happens when developers use subsets -- ****** -----
  • Stubhub (ebay) estimates that 20% of there production bugs arise from testing onSubsets instead of full database copies.
  • Stubhub estimated a 20% reduction in bugs that made it to production
  • Due to the constraints of building clone copy database environments one ends up in the “culture of no”Where developers stop asking for a copy of a production database because the answer is “no”If the developers need to debug an anomaly seen on production or if they need to write a custom module which requires a copy of production they know not to even ask and just give up.
  • State of Colorado has a 100 projects support 3KLA tencor can only support 2 projects of a dozen
  • Take for example a 9TB source databaseIt’s compressed to 3TB on Delphix.Then we still have 6TB on Delphix before we reach the size of the original database, but that 6TB is actuall 18TB of data at a 1/3 compressionWhich means we can have the original database and 18TB of changes in the size of the original databaseWe see on average about 50 days worth of back up in the size of the original databaseMultiple scripted dumps or RMAN backups are used to move data today. With application awareness, we only request change blocks—dramatically reducing production loads by as much as 80%. We also eliminate the need for DBAs to manage custom scripts, which are expensive to maintain and support over time.
  • Once Last Thinghttp://www.dadbm.com/wp-content/uploads/2013/01/12c_pluggable_database_vs_separate_database.png
  • 250 pdb x 200 GB = 50 TBEMC sells 1GB$1000Dell sells 32GB $1,000.terabyte of RAM on a Dell costs around $32,000terabyte of RAM on a VMAX 40k costs around $1,000,000.
  • http://www.emc.com/collateral/emcwsca/master-price-list.pdf    These prices obtain on pages 897/898:Storage engine for VMAX 40k with 256 GB RAM is around $393,000Storage engine for VMAX 40k with  48 GB RAM is around $200,000So, the cost of RAM here is 193,000 / 208 = $927 a gigabyte.   That seems like a good deal for EMC, as Dell sells 32 GB RAM DIMMs for just over $1,000.    So, a terabyte of RAM on a Dell costs around $32,000, and a terabyte of RAM on a VMAX 40k costs around $1,000,000.2) Most DBs have a buffer cache that is less than 0.5% (not 5%, 0.5%) of the datafile size.
  • Source Control for the database data
  • Source Control for the database data
  • Transcript of "What is Delphix"

    1. 1. What is Delphix
    2. 2. kylehailey.com/delphix Use Cases What is Delphix Competition
    3. 3. What is Delphix
    4. 4. Three Physical Copies Three Virtual Copies Delphix
    5. 5. Install Delphix on Intel commodity hardware Intel hardware
    6. 6. Allocate Any Storage to Delphix Allocate Storage Any type Pure Storage + Delphix Better Performance for 1/10 the cost
    7. 7. One time backup of source database Production Supports Instance Instance Instance Database File system Application Stack Virtualization Upcoming
    8. 8. DxFS (Delphix) Compress Data Production Instance Instance Instance Database File system Data is compressed typically 1/3 size
    9. 9. Incremental forever change collection Production Instance Instance Instance Database Changes Time Window File system • Collected incrementally forever • Old data purged
    10. 10. Change collection works with snapshots Snapshot 0: Source Full Copy Snapshot 1: Changed blocks at time T1
    11. 11. Snapshots immediately look like full copies as they are mapped onto the previous copy Snapshot 0 Snapshot 1
    12. 12. Snapshot 0 Snapshot 1 Snapshot 2: Changed blocks at time T2
    13. 13. Snapshot 0 Snapshot 2 Snapshot 1
    14. 14. Purging old snapshot releases any blocks that have been since replaced, but keeps all unchanged blocks Snapshot 2 Snapshot 1 Drop Snapshot 0
    15. 15. Cloning : almost no storage at creation since it is all shared. Upon usage changed blocks will add to storage used Production Instance Instance Instance Instance Instance Database Database Time Window File system
    16. 16. Typical Architecture Production Dev, QA, UAT Reporting Instance Instance Instance Instance Instance Instance Instance Instance Database Database Database Database Database File system File system File system File system File system File system File system File system File system File system File system Backup Instance Instance Instance File system File system
    17. 17. With Delphix Production Instance Instance Database File system Dev & QA Instance Instance Instance Instance Instance Instance Database Database Database Reporting Backup Instance Instance Database Database
    18. 18. Delphix Use Cases 1. 2. 3. 4. 5. Fast, Fresh, Full Free Branching Federated Self Serve
    19. 19. Fast, Fresh, Full Source Development VDB Instance Instance Time Window
    20. 20. Free Instance Source Instance Instance Instance gif by Steve Karam
    21. 21. Branching Source Instance branching Dev Instance Source QA branched from Dev Instance
    22. 22. Federated Cloning
    23. 23. Federated Source1 Instance Source1 Instance Source2 Instance Instance
    24. 24. “I looked like a hero” Tony Young, CIO Informatica
    25. 25. Self Service
    26. 26. Use Cases 1. Development Acceleration 2. Quality 3. BI
    27. 27. DevOps
    28. 28. DevOps With Delphix 1. 2. 3. 4. 5. Efficient QA: Low cost, high utilization Quick QA : Fast Bug Fix Every Dev gets DB: Parallelized Dev Full DB : Less Bugs Fast Builds: Culture of Yes
    29. 29. Without Delphix 1. Inefficient QA: Long Build times Build QA Test Build Time 96% of QA time was building environment $.04/$1.00 actual testing vs. setup
    30. 30. : With Delphix 1. Efficient QA: Lower cost Build QA Test Build Time B u i l d T i m e QA Test 1% of QA time was building environment $.99/$1.00 actual testing vs. setup
    31. 31. Rapid QA via Branching
    32. 32. Without Delphix 2. QA Delays: bugs found late require more code re-work Build QA Env Sprint 3 Sprint 2 Sprint 1 X Build QA Env QA Bug Code 70 60 50 40 30 20 10 0 Cost To Correct 1 2 3 4 5 6 7 Delay in Fixing the bug Software Engineering Economics – Barry Boehm (1981) QA
    33. 33. : With Delphix 2. QA Immediate: bugs found fast and fixed Build QA Env Sprint 2 Sprint 1 X Q A Build QA Env Q A Sprint 3 Bug Code QA QA Sprint 2 Sprint 1 X Bug Code Sprint 3
    34. 34. Without Delphix 3. Full Copy Shared : Bottlenecks Old Unrepresentative Data Frustration Waiting
    35. 35. : With Delphix 3. Private Copies: Parallelize gif by Steve Karam
    36. 36. Without Delphix 4. Subsets : cause bugs
    37. 37. Without Delphix 4. Subsets : cause bugs The Production ‘Wall’ Classic problem is that queries that run fast on subsets hit the wall in production. Developers are unable to test against all data
    38. 38. : With Delphix 4. Full Size DB : Eliminate bugs
    39. 39. Without Delphix 5. Slow Environment Builds: Slow Environment Builds: 3-6 Months to Deliver Data Developer Asks for DB Get Access Manager DBA System Admin Storage Admin approves Request system Setup DB Request storage Setup machine Allocate storage (take snapshot)
    40. 40. Without Delphix 5. Slow Environment Builds: culture of no DBA Developer
    41. 41. Without Delphix Never enough environments
    42. 42. : With Delphix 5. Self Service: Fast, Efficient. Culture of Yes!
    43. 43. Quality 1. Forensics 2. Testing 3. Recovery
    44. 44. 1. Forensics: Investigate Production Bugs Development Instance Instance Time Window Anomaly on Prod Possible code bug At noon yesterday Spin up VDB of Prod as it was during anomaly
    45. 45. 2. Testing : Rewind for patch and QA testing Prod Development Instance Instance Time Window Time Window
    46. 46. 2. Testing: A/B Instance Test A with Index 1 Instance Instance Time Window • Keep tests for compare • Production vs Virtual – invisible index on Prod – Creating index on virtual • Flashback vs Virtual Test B with Index 2
    47. 47. 3. Recovery: Surgical recover of Production Source Development Instance Instance Spin VDB up Before drop Time Window Problem on Prod Dropped Table Accidently
    48. 48. 3. Recovery Surgical or Full Recovery on VDB Dev1 VDB Source Instance Instance Dev2 VDB Branched Source Time Window Dev1 VDB Time Window Instance
    49. 49. 3. Recovery: Virtual to Physical Source VDB Instance Instance Spin VDB up Before drop Time Window Corruption
    50. 50. 50 days of backup in size of Original 3. Recovery
    51. 51. Business Intelligence
    52. 52. ETL and Refresh Windows 1pm noon 10pm 8am
    53. 53. ETL and DW refreshes taking longer 1pm noon 10pm 2011 2012 2013 2014 2015 8am
    54. 54. ETL and Refresh Windows Globalization limits ETL windows to weekends 6am 8am 10pm 10pm 1pm noon 8am 10pm 2011 2012 2013 2014 2015 noon 9pm 8am
    55. 55. ETL and DW Refreshes Prod DW & BI Instance Instance Data Guard – requires full refresh if used Active Data Guard – read only, most reports don’t work
    56. 56. Fast Refreshes • Collect only Changes • Refresh in minutes Prod Instance BI DW Instance Instance ETL 24x7
    57. 57. Temporal Data
    58. 58. Oracle 12c
    59. 59. 80MB buffer cache ?
    60. 60. 200GB Cache
    61. 61. with Latency Tnxs / min 5000 300 ms 1 5 10 20 30 60 100 200 Users 1 5 10 20 30 60 100 200
    62. 62. Latency Tnxs / min 8000 600 ms 1 5 10 20 30 60 100 200 Users 1 5 10 20 30 60 100 200
    63. 63. Five 200GB database copies are cached with : $1,000,000 1TB cache on SAN $6,000 200GB shared cache on Delphix
    64. 64. Data Center Migration : clone migration 5x Source Data Copy < 1x Source Data Copy
    65. 65. Data Center Migration : clone migration + source S S 5x Source Data Copy < 2 x Source Data Copy
    66. 66. Data Center Migration : clone migration + source S C C C 5x Source Data Copy C S V V V < 1 x Source Data Copy V
    67. 67. Consolidation Without Delphix Active Active With Delphix Idle Active Idle Active
    68. 68. DBMS License Utilization • Most DBMS time spent waiting on environment setup, data loads, backups, etc. • Coupling of data and DB servers forces over-provisioning, limits utilization • Delphix can attach / detach data to server tier on-demand to drive consolidation • Potential 2X reduction in DB server footprint cuts HW and DBMS license © 2014 DELPHIX. ALL RIGHTS RESERVED. PRIVATE & CONFIDENTIAL. costs 69
    69. 69. Dev v2.6 v2.6 QA v2.6 v2.6 UAT v2.6 v2.6 v2.6 v2.7 v2.6 v2.7 v2.7 v2.6 v2.6 v2.6 v2.8 v2.8 v2.6 v2.8
    70. 70. Production Dev v2.6 v2.6 QA v2.6 v2.6 UAT v2.6 v2.6 v2.6 v2.7 v2.6 v2.7 v2.7 v2.6 v2.6 v2.6 v2.8 v2.8 v2.6 v2.8 Source Control for the database data
    71. 71. 2.6 branching Dev Prod checkout bookmark
    72. 72. 2.6 QA Dev Prod
    73. 73. 2.6 UAT QA Dev Prod
    74. 74. 2.6 UAT QA Dev Prod Dev QA UAT 2.7
    75. 75. 2.6 2.8 UAT QA UAT QA Dev Dev Prod Dev QA UAT 2.7
    76. 76. 2.6 2.8 UAT QA UAT QA Dev Dev Prod Dev QA UAT 2.7 Data Control = Source Control for the Database
    77. 77. 2.6 2.8 UAT QA UAT QA Dev Dev Production Time Flow Dev QA UAT 2.7 Data Control = Source Control for the Database
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×