Agile Data :
Virtual Data Revolution
Kyle@delphix.com
kylehailey.com
slideshare.com/khailey
The Goal - Theory of Constraints (TOC)
Improvement
not made at the constraint
is an illusion
Factory floor : straight forward
Factory floor : straight forward
constraint
Factory floor : optimize at the constraint
constraint
Tuning here
Stock piling
Factory floor : optimize at the constraint
constraint
Tuning here
Stock piling
Tuning here
Starvation
Factory floor : straight forward
constraint
Factory Floor Optimization
Drum, Rope, Buffer (DBR)
45 75
305080
constraint
25
For IT does the Theory of Constraints work ?
The Phoenix Project
• Constraints Identify
• Metrics Define
• Priorities Set
• Goals Clarify
• Iterations Fast
• CI contin...
The Phoenix Project
What is the constraint in IT ?
“One of the most powerful things that IT can do is get
environments to ...
In this presentation :
• Data Constraint
• Solution
• Use Cases
In this presentation :
• Data Constraint
I. strains IT
II. price is huge
III. companies unaware
• Solution
• Use Cases
Data is the constraint
60% Projects Over Schedule
85% delayed waiting for data
Data is the Constraint
CIO Magazine Survey:...
In this presentation :
• Data Constraint
I. strains IT
II. price is huge
III. companies unaware
• Solution
• Use Cases
I. Data Constraint strains IT
If you can’t satisfy the business demands then your process is broken.
I. Data Constraint : moving data is hard
– Storage & Systems
– Personnel
– Time
Typical Architecture
Production
Instance
File system
Database
Typical Architecture
Production
Instance
Backup
File system
Database
File system
Database
Typical Architecture
Production
Instance
Reporting Backup
File system
Database
Instance
File system
Database
File system
D...
Typical Architecture
Production
Instance
File system
Database
Instance
File system
Database
File system
Database
File syst...
Typical Architecture
Production
Instance
File system
Database
Instance
File system
Database
File system
Database
File syst...
In this presentation :
• Data Constraint
I. strains IT
II. price is huge
III. companies unaware
• Solution
• Use Cases
II. Data Constraint price is huge
I. Data constraint: Data floods company
infrastructure
92% of the cost of business
, in financial services business ,
is “...
II. Data constraint price is Huge
• Four Areas data tax hits
1. IT Capital resources $
2. IT Operations personnel $
3. App...
II. Data constraint price is Huge
• Four Areas data tax hits
1. IT Capital resources
2. IT Operations personnel
3. Applica...
II. Data constraint price is huge : 1. IT Capital
• Hardware
–Servers
–Storage
–Network
–Data center floor space, power,
c...
II. Data constraint price is Huge
• Four Areas data tax hits
1. IT Capital resources
2. IT Operations personnel
3. Applica...
II. Data constraint price is huge : 2. IT Operations
• People
– DBAs
– SYS Admin
– Storage Admin
– Backup Admin
– Network ...
II. Data constraint price is Huge
• Four Areas data tax hits
1. IT Capital resources
2. IT Operations personnel
3. Applica...
II. Data constraint price is Huge : 3. App Dev
• Inefficient QA: Higher costs of QA
• QA Delays : Greater re-work of code
...
II. Data Tax is Huge : 3. App Dev
Slow Environment Builds
Never enough environments
Part II. Data constraint price is Huge
• Four Areas data tax hits
1. IT Capital resources
2. IT Operations personnel
3. Ap...
II. Data constraint price is Huge : 4. Business
Ability to capture revenue
• Business Intelligence
– Old data = less intel...
II. Data constraint price is Huge : 4. Business
II. Data constraint price is Huge : 4. Business
0 5 10 15 20 25 30
Storage
IT Ops
Dev
Revenue
Billion $
II. Data constraint price is Huge
• Four Areas data tax hits
1. IT Capital resources $
2. IT Operations personnel $
3. App...
In this presentation :
• Data Constraint
I. strains IT
II. price is huge
III. companies unaware
• Solution
• Use Cases
Part III. Data Constraint companies unaware
III. Data Constraint companies unaware
DBA Developer
III. Data Constraint companies unaware
#1 Biggest Enemy :
IT departments believe
– best processes
– greatest technology
– ...
There are always new and better ways to do things
III. Data Constraint companies unaware
Why do I need an iPhone ?
Don’t we already do that ?
SQL scripts
Alter database beg...
III. Data Constraint companies unaware
• Ask Questions
– me: we provision environments in minutes for
almost not extra sto...
III. Data Constraint companies unaware
How to enlighten? Ask for metrics
– How long does it take a developer to get a DB c...
In this presentation :
• Data Constraint
I. strains IT
II. price is huge
III. companies unaware
• Solution
• Use Cases
Clone 1 Clone 3Clone 2
99% of blocks are identical
Solution
Clone 1 Clone 2 Clone 3
Thin Clone
Technology Core : file system snapshots
• Vmware Linked Clones
– Not supported for Oracle
• EMC
– 16 snapshots
– Write per...
Engine not equal car
Three Core Parts
Production
File System Instance
DevelopmentStorage
21 3
Copy
Sync
Snapshots
Time Flow
Purge
Clone
(snapsh...
Database Virtualization
Three Physical Copies
Three Virtual Copies
Data
Virtualization
Appliance
Install Delphix on x86 hardware
Intel hardware
Allocate Any Storage to Delphix
Allocate Storage
Any type
Pure Storage + Delphix
Better Performance for
1/10 the cost
One time backup of source database
Database
Production
File systemFile system
Upcoming
Supports
InstanceInstanceInstance
A...
DxFS (Delphix) Compress Data
Database
Production
Data is
compressed
typically 1/3
size
File system
InstanceInstanceInstance
Incremental forever change collection
Database
Production
File system
Changes
• Collected incrementally forever
• Old data...
Virtual DB
62 / 30
Jonathan Lewis
© 2013
Snapshot 1 – full backup once only at link time
a b c d e f g h i
We start with a...
Virtual DB
63 / 30
Jonathan Lewis
© 2013
Snapshot 2 (from SCN)
b' c'
a b c d e f g h i
The "backup from SCN" is analogous ...
Virtual DB
64 / 30
Jonathan Lewis
© 2013
a b c d e f g h i
Apply Snapshot 2
b' c'
The Delphix appliance unpacks the rman b...
Virtual DB
65 / 30
Jonathan Lewis
© 2013
Drop Snapshot 1
b' c'a d e f g h i
The call to rman leaves us with a new level 0 ...
Virtual DB
66 / 30
Jonathan Lewis
© 2013
Creating a vDB
b' c'a d e f g h i
The first step in creating a vDB is to take a s...
Virtual DB
67 / 30
Jonathan Lewis
© 2013
Creating a vDB
b' c'a d e f g h i
The first step in creating a vDB is to take a s...
Before Delphix
Production Dev, QA, UAT
Instance
Reporting Backup
File system
Database
Instance
File system
Database
File s...
With Delphix
Production
Instance
Database
Dev & QA
Instance
Database
Reporting
Instance
Database
Backup
Instance Instance ...
In this presentation :
• Problem in the Industry
• Solution
• Use Cases
Use Cases
1. Development
2. QA
3. Recovery
4. Business Intelligence
5. Modernization
Use Cases
1. Development
2. QA
3. Recovery
4. Business Intelligence
5. Modernization
Development
• Parallelized Environments
• Full size environments
• Self Service
Development
Development without Agile Data: bottlenecks
Frustration Waiting
Old Unrepresentative Data
Development without Agile Data: subsets of DB
Development without Agile Data: bugs
Development with Agile Data: Parallelize
Environments
gif by Steve Karam
Development with Agile Data: Full size copies
Development without Agile Data: slow env build
times
Developer Asks for DB Get Access
Manager approves
DBA Request
system
...
Development without Agile Data: slow env build
times
Why are hand offs so expensive?
1hour
1 day
9 days
http://martinfowler.com/bliki/NoDBA.html
Development without Agile Data: slow env build
times
Development with Agile Data: Self Service
Use Cases
1. Development
2. QA
3. Recovery
4. Business Intelligence
5. Modernization
QA
• Fast
• Parallel
• Rollback
• A/B testing
QA without Agile Data : Long Build times
Build Time
96% of QA time was building environment
$.04/$1.00 actual testing vs. ...
QA without Agile Data : slow QA
Build QA Env QA Build QA Env QA
Sprint 1 Sprint 2 Sprint 3
Bug CodeX
0
10
20
30
40
50
60
7...
QA with Agile Data : Fast environments with
Branching
Instance
Instance
Instance
Source Dev
QA
branched from Dev
Source
de...
QA with Agile Data: Fast environments with
Branching
B
u
i
l
d
T
i
m
e
QA Test
1% of QA time was building environment
$.99...
QA with Agile Data: bugs found fast
Sprint 1 Sprint 2 Sprint 3
Bug CodeX
QA QA
Build QA
Env
Q
A
Build QA
Env
Q
A
Sprint 1 ...
QA with Agile Data: Parallel environments
Instance
Instance
Instance
Instance
Source
QA with Agile Data: Rewind for patch and QA
testing
Instance Instance
Development
Time Window
Prod
QA with Agile Data: A/B testing
Instance
Instance
Instance
Index 1
Index 2
Use Cases
1. Development
2. QA
3. Quality
4. Business Intelligence
5. Modernization
Quality
• Prod & Dev Backups
• Surgical recovery
• Recovery of Production
• Recovery of Development
• Bug Forensics
Quality : 50 days of backup in size of
production
Quality : Surgical recovery
Instance Instance
Development
Time Window
Before dropDrop
Source
Quality: recovery of development
Instance
Instance
Dev1 VDB
Time Window
Time Window
Dev1 VDB
Instance
Source
Source
Dev2 V...
Quality : recovery of production
Instance Instance
VDBSource
Time Window
Corruption
Quality : Forensics - Investigate Production Bugs
Instance
Time Window
Instance
Development
Bug
Yesterday
Yesterday
Use Cases
1. Development
2. QA
3. Quality
4. Business Intelligence
5. Modernization
Business Intelligence
• 24x7 Batches
• Low Bandwidth
• Temporal Data
• Confidence Testing
Business Intelligence: ETL and Refresh Windows
1pm 10pm 8am
noon
Business Intelligence: ETL and DW refreshes
taking longer
1pm 10pm 8am
noon
2011
2012
2013
2014
2015
Business Intelligence ETL and Refresh
Windows
2011
2012
2013
2014
2015
1pm 10pm 8am
noon
10pm 8am noon 9pm
6am 8am 10pm
Business Intelligence: ETL and DW Refreshes
Instance
Prod
Instance
DW & BI
Data Guard – requires full refresh if used
Acti...
Business Intelligence: Fast Refreshes
• Collect only Changes
• Refresh in minutes
Instance Instance
Prod
Instance
BI and D...
Business Intelligence: Temporal Data
Business Intelligence
a) 24x7 Batches & Refreshes
a) Temporal queries
b) Confidence testing
Use Cases
1. Development
2. QA
3. Quality
4. Business Intelligence
5. Modernization
Modernization
1. Federated
2. Consolidation
3. Migration
4. Auditing
Modernization: Federated
Instance
Instance
Instance
Instance
Source1
Source2
Source1
Modernization: Federated
“I looked like a hero”
Tony Young, CIO Informatica
Modernization: Federated
Data movement required for 1 source DB and 4
clones
5x Source Data Copy < 1 x Source Data Copy
S SC C C C V V V V
Without ...
Modernization: Consolidation
Without Delphix With Delphix
Dev
QA
UAT
Dev
QA
UAT
2.6
2.7
Dev
QA
UAT
2.8
Data Control = Source Control for the Database
Production Time Flow
Moderniza...
Use Cases
1. Development
• Parallelized Environments
• Full size environments
• Self Service
2. QA
• Fast
• Parallel
• Rol...
Use Case Summary
1. Development
2. QA
3. Quality
4. Business Intelligence
5. Modernization
How expensive is the Data Constraint?
Before and after Delphix w/ Fortune 500 :
Dev throughput increase by 2x
How expensive is the Data Constraint?
• 10 x Faster Financial Close
– 21 days down to 2
• 9x Faster BI refreshes
– 3 weeks...
Agile Data Quotes
• “Allowed us to shrink our project schedule from 12
months to 6 months.”
– BA Scott, NYL VP App Dev
• "...
Summary
• Problem: Data is the constraint
• Solution: Agile data is small & fast
• Results: Deliver projects
– Half the Ti...
Oracle 12c
80MB buffer cache ?
200GB
Cache
5000
Tnxs/minLatency
300
ms
1 5 10 20 30 60 100 200
with
1 5 10 20 30 60 100 200
Users
8000
Tnxs/minLatency
600
ms
1 5 10 20 30 60 100 200
Users
1 5 10 20 30 60 100 200
$1,000,000
1TB cache on SAN
$6,000
200GB shared cache on Delphix
Five 200GB database copies are
cached with :
Use Cases
1. Development
• Parallelized Environments
• Full size environments
• Self Service
2. QA
• Fast
• Parallel
• Rol...
•END
Source Full Copy Source backup
from SCN 1
Snapshot 1
Snapshot 2
Snapshot 1
Snapshot 2
Backup from SCN
Snapshot 1
Snapshot 2
Snapshot 3
Drop Snapshot
Snapshot 1
Snapshot 2
Snapshot 3
Snapshot 2
Snapshot 3
Drop
Snapshot 1
Agile Data: revolutionizing data and database cloning
Agile Data: revolutionizing data and database cloning
Agile Data: revolutionizing data and database cloning
Agile Data: revolutionizing data and database cloning
Agile Data: revolutionizing data and database cloning
Agile Data: revolutionizing data and database cloning
Upcoming SlideShare
Loading in...5
×

Agile Data: revolutionizing data and database cloning

334

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
334
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
25
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Agile Data: revolutionizing data and database cloning

  1. 1. Agile Data : Virtual Data Revolution Kyle@delphix.com kylehailey.com slideshare.com/khailey
  2. 2. The Goal - Theory of Constraints (TOC) Improvement not made at the constraint is an illusion
  3. 3. Factory floor : straight forward
  4. 4. Factory floor : straight forward constraint
  5. 5. Factory floor : optimize at the constraint constraint Tuning here Stock piling
  6. 6. Factory floor : optimize at the constraint constraint Tuning here Stock piling Tuning here Starvation
  7. 7. Factory floor : straight forward constraint
  8. 8. Factory Floor Optimization
  9. 9. Drum, Rope, Buffer (DBR) 45 75 305080 constraint 25
  10. 10. For IT does the Theory of Constraints work ?
  11. 11. The Phoenix Project • Constraints Identify • Metrics Define • Priorities Set • Goals Clarify • Iterations Fast • CI continuous integration • Cloud • Agile • Kanban
  12. 12. The Phoenix Project What is the constraint in IT ? “One of the most powerful things that IT can do is get environments to development and QA when they need it” 2x throughput increase with agile data
  13. 13. In this presentation : • Data Constraint • Solution • Use Cases
  14. 14. In this presentation : • Data Constraint I. strains IT II. price is huge III. companies unaware • Solution • Use Cases
  15. 15. Data is the constraint 60% Projects Over Schedule 85% delayed waiting for data Data is the Constraint CIO Magazine Survey: Current situation: only getting worse … Data Doomsday
  16. 16. In this presentation : • Data Constraint I. strains IT II. price is huge III. companies unaware • Solution • Use Cases
  17. 17. I. Data Constraint strains IT If you can’t satisfy the business demands then your process is broken.
  18. 18. I. Data Constraint : moving data is hard – Storage & Systems – Personnel – Time
  19. 19. Typical Architecture Production Instance File system Database
  20. 20. Typical Architecture Production Instance Backup File system Database File system Database
  21. 21. Typical Architecture Production Instance Reporting Backup File system Database Instance File system Database File system Database
  22. 22. Typical Architecture Production Instance File system Database Instance File system Database File system Database File system Database Instance Instance Instance File system Database File system Database Dev, QA, UAT Reporting Backup Triple Tax
  23. 23. Typical Architecture Production Instance File system Database Instance File system Database File system Database File system Database Instance Instance Instance File system Database File system Database
  24. 24. In this presentation : • Data Constraint I. strains IT II. price is huge III. companies unaware • Solution • Use Cases
  25. 25. II. Data Constraint price is huge
  26. 26. I. Data constraint: Data floods company infrastructure 92% of the cost of business , in financial services business , is “data” www.wsta.org/resources/industry-articles Most companies have 2-9% IT spending http://uclue.com/?xq=1133 Data management is the largest Part of IT expense Gartner: Data Doomsday
  27. 27. II. Data constraint price is Huge • Four Areas data tax hits 1. IT Capital resources $ 2. IT Operations personnel $ 3. Application Development $$$ 4. Business $$$$$$$
  28. 28. II. Data constraint price is Huge • Four Areas data tax hits 1. IT Capital resources 2. IT Operations personnel 3. Application Development 4. Business
  29. 29. II. Data constraint price is huge : 1. IT Capital • Hardware –Servers –Storage –Network –Data center floor space, power, cooling
  30. 30. II. Data constraint price is Huge • Four Areas data tax hits 1. IT Capital resources 2. IT Operations personnel 3. Application Development 4. Business
  31. 31. II. Data constraint price is huge : 2. IT Operations • People – DBAs – SYS Admin – Storage Admin – Backup Admin – Network Admin • Hours : 1000s just for DBAs • $100s Millions for data center modernizations
  32. 32. II. Data constraint price is Huge • Four Areas data tax hits 1. IT Capital resources 2. IT Operations personnel 3. Application Development 4. Business
  33. 33. II. Data constraint price is Huge : 3. App Dev • Inefficient QA: Higher costs of QA • QA Delays : Greater re-work of code • Sharing DB Environments : Bottlenecks • Using DB Subsets: More bugs in Prod • Slow Environment Builds: Delays “if you can't measure it you can’t manage it”
  34. 34. II. Data Tax is Huge : 3. App Dev Slow Environment Builds Never enough environments
  35. 35. Part II. Data constraint price is Huge • Four Areas data tax hits 1. IT Capital resources 2. IT Operations personnel 3. Application Development 4. Business
  36. 36. II. Data constraint price is Huge : 4. Business Ability to capture revenue • Business Intelligence – Old data = less intelligence • Business Applications – Delays cause => Lost Revenue
  37. 37. II. Data constraint price is Huge : 4. Business
  38. 38. II. Data constraint price is Huge : 4. Business 0 5 10 15 20 25 30 Storage IT Ops Dev Revenue Billion $
  39. 39. II. Data constraint price is Huge • Four Areas data tax hits 1. IT Capital resources $ 2. IT Operations personnel $ 3. Application Development $$$ 4. Business $$$$$$$ Review
  40. 40. In this presentation : • Data Constraint I. strains IT II. price is huge III. companies unaware • Solution • Use Cases
  41. 41. Part III. Data Constraint companies unaware
  42. 42. III. Data Constraint companies unaware DBA Developer
  43. 43. III. Data Constraint companies unaware #1 Biggest Enemy : IT departments believe – best processes – greatest technology – Just the way it is
  44. 44. There are always new and better ways to do things
  45. 45. III. Data Constraint companies unaware Why do I need an iPhone ? Don’t we already do that ? SQL scripts Alter database begin backup Back up datafiles Redo Archive Alter database end backup RMAN
  46. 46. III. Data Constraint companies unaware • Ask Questions – me: we provision environments in minutes for almost not extra storage. – Customer: We already do that – me: How long does it take a developer to get an environment after they ask ? – Customer: 2-3 weeks – me: we do it in 2-3 minutes
  47. 47. III. Data Constraint companies unaware How to enlighten? Ask for metrics – How long does it take a developer to get a DB copy? • QA? – How old is data ? • BI and DW : ETL batch windows • QA and Dev : how often refreshed – How much storage used for copies? • How much DBA time?
  48. 48. In this presentation : • Data Constraint I. strains IT II. price is huge III. companies unaware • Solution • Use Cases
  49. 49. Clone 1 Clone 3Clone 2 99% of blocks are identical
  50. 50. Solution
  51. 51. Clone 1 Clone 2 Clone 3 Thin Clone
  52. 52. Technology Core : file system snapshots • Vmware Linked Clones – Not supported for Oracle • EMC – 16 snapshots – Write performance impact • Netapp – 255 snapshots • ZFS – Unlimited snapshots
  53. 53. Engine not equal car
  54. 54. Three Core Parts Production File System Instance DevelopmentStorage 21 3 Copy Sync Snapshots Time Flow Purge Clone (snapshot) Compress Share Cache Storage Mount, recover, rename Self Service, Roles & Security Rollback & Refresh Branch & Tag Instance
  55. 55. Database Virtualization
  56. 56. Three Physical Copies Three Virtual Copies Data Virtualization Appliance
  57. 57. Install Delphix on x86 hardware Intel hardware
  58. 58. Allocate Any Storage to Delphix Allocate Storage Any type Pure Storage + Delphix Better Performance for 1/10 the cost
  59. 59. One time backup of source database Database Production File systemFile system Upcoming Supports InstanceInstanceInstance Application Stack Data
  60. 60. DxFS (Delphix) Compress Data Database Production Data is compressed typically 1/3 size File system InstanceInstanceInstance
  61. 61. Incremental forever change collection Database Production File system Changes • Collected incrementally forever • Old data purged File system Time Window Production InstanceInstanceInstance
  62. 62. Virtual DB 62 / 30 Jonathan Lewis © 2013 Snapshot 1 – full backup once only at link time a b c d e f g h i We start with a full backup - analogous to a level 0 rman backup. Includes the archived redo log files needed for recovery. Run in archivelog mode.
  63. 63. Virtual DB 63 / 30 Jonathan Lewis © 2013 Snapshot 2 (from SCN) b' c' a b c d e f g h i The "backup from SCN" is analogous to a level 1 incremental backup (which includes the relevant archived redo logs). Sensible to enable BCT. Delphix executes standard rman scripts
  64. 64. Virtual DB 64 / 30 Jonathan Lewis © 2013 a b c d e f g h i Apply Snapshot 2 b' c' The Delphix appliance unpacks the rman backup and "overwrites" the initial backup with the changed blocks - but DxFS makes new copies of the blocks
  65. 65. Virtual DB 65 / 30 Jonathan Lewis © 2013 Drop Snapshot 1 b' c'a d e f g h i The call to rman leaves us with a new level 0 backup, waiting for recovery. But we can pick the snapshot root block. We have EVERY level 0 backup
  66. 66. Virtual DB 66 / 30 Jonathan Lewis © 2013 Creating a vDB b' c'a d e f g h i The first step in creating a vDB is to take a snapshot of the filesystem as at the backup you want (then roll it forward) My vDB (filesystem) Your vDB (filesystem) b' c'a d e f g h i
  67. 67. Virtual DB 67 / 30 Jonathan Lewis © 2013 Creating a vDB b' c'a d e f g h i The first step in creating a vDB is to take a snapshot of the filesystem as at the backup you want (then roll it forward) My vDB (filesystem) Your vDB (filesystem) i’b' c'a d e f g h ib' c'a d e f g h i
  68. 68. Before Delphix Production Dev, QA, UAT Instance Reporting Backup File system Database Instance File system Database File system Database File system Database Instance Instance Instance File system Database File system Database “triple data tax”
  69. 69. With Delphix Production Instance Database Dev & QA Instance Database Reporting Instance Database Backup Instance Instance Instance Database InstanceInstance Database InstanceInstance File system Database
  70. 70. In this presentation : • Problem in the Industry • Solution • Use Cases
  71. 71. Use Cases 1. Development 2. QA 3. Recovery 4. Business Intelligence 5. Modernization
  72. 72. Use Cases 1. Development 2. QA 3. Recovery 4. Business Intelligence 5. Modernization
  73. 73. Development • Parallelized Environments • Full size environments • Self Service Development
  74. 74. Development without Agile Data: bottlenecks Frustration Waiting Old Unrepresentative Data
  75. 75. Development without Agile Data: subsets of DB
  76. 76. Development without Agile Data: bugs
  77. 77. Development with Agile Data: Parallelize Environments gif by Steve Karam
  78. 78. Development with Agile Data: Full size copies
  79. 79. Development without Agile Data: slow env build times Developer Asks for DB Get Access Manager approves DBA Request system Setup DB System Admin Request storage Setup machine Storage Admin Allocate storage (take snapshot) 3-6 Months to Deliver Data
  80. 80. Development without Agile Data: slow env build times Why are hand offs so expensive? 1hour 1 day 9 days
  81. 81. http://martinfowler.com/bliki/NoDBA.html Development without Agile Data: slow env build times
  82. 82. Development with Agile Data: Self Service
  83. 83. Use Cases 1. Development 2. QA 3. Recovery 4. Business Intelligence 5. Modernization
  84. 84. QA • Fast • Parallel • Rollback • A/B testing
  85. 85. QA without Agile Data : Long Build times Build Time 96% of QA time was building environment $.04/$1.00 actual testing vs. setup Build Build Time QA Test QA Test Build
  86. 86. QA without Agile Data : slow QA Build QA Env QA Build QA Env QA Sprint 1 Sprint 2 Sprint 3 Bug CodeX 0 10 20 30 40 50 60 70 1 2 3 4 5 6 7 Delay in Fixing the bug Cost To Correct Software Engineering Economics – Barry Boehm (1981)
  87. 87. QA with Agile Data : Fast environments with Branching Instance Instance Instance Source Dev QA branched from Dev Source dev QA
  88. 88. QA with Agile Data: Fast environments with Branching B u i l d T i m e QA Test 1% of QA time was building environment $.99/$1.00 actual testing vs. setup Build Time QA Test Build
  89. 89. QA with Agile Data: bugs found fast Sprint 1 Sprint 2 Sprint 3 Bug CodeX QA QA Build QA Env Q A Build QA Env Q A Sprint 1 Sprint 2 Sprint 3 Bug Cod e X
  90. 90. QA with Agile Data: Parallel environments Instance Instance Instance Instance Source
  91. 91. QA with Agile Data: Rewind for patch and QA testing Instance Instance Development Time Window Prod
  92. 92. QA with Agile Data: A/B testing Instance Instance Instance Index 1 Index 2
  93. 93. Use Cases 1. Development 2. QA 3. Quality 4. Business Intelligence 5. Modernization
  94. 94. Quality • Prod & Dev Backups • Surgical recovery • Recovery of Production • Recovery of Development • Bug Forensics
  95. 95. Quality : 50 days of backup in size of production
  96. 96. Quality : Surgical recovery Instance Instance Development Time Window Before dropDrop Source
  97. 97. Quality: recovery of development Instance Instance Dev1 VDB Time Window Time Window Dev1 VDB Instance Source Source Dev2 VDB Branched
  98. 98. Quality : recovery of production Instance Instance VDBSource Time Window Corruption
  99. 99. Quality : Forensics - Investigate Production Bugs Instance Time Window Instance Development Bug Yesterday Yesterday
  100. 100. Use Cases 1. Development 2. QA 3. Quality 4. Business Intelligence 5. Modernization
  101. 101. Business Intelligence • 24x7 Batches • Low Bandwidth • Temporal Data • Confidence Testing
  102. 102. Business Intelligence: ETL and Refresh Windows 1pm 10pm 8am noon
  103. 103. Business Intelligence: ETL and DW refreshes taking longer 1pm 10pm 8am noon 2011 2012 2013 2014 2015
  104. 104. Business Intelligence ETL and Refresh Windows 2011 2012 2013 2014 2015 1pm 10pm 8am noon 10pm 8am noon 9pm 6am 8am 10pm
  105. 105. Business Intelligence: ETL and DW Refreshes Instance Prod Instance DW & BI Data Guard – requires full refresh if used Active Data Guard – read only, most reports don’t work
  106. 106. Business Intelligence: Fast Refreshes • Collect only Changes • Refresh in minutes Instance Instance Prod Instance BI and DW ETL 24x7
  107. 107. Business Intelligence: Temporal Data
  108. 108. Business Intelligence a) 24x7 Batches & Refreshes a) Temporal queries b) Confidence testing
  109. 109. Use Cases 1. Development 2. QA 3. Quality 4. Business Intelligence 5. Modernization
  110. 110. Modernization 1. Federated 2. Consolidation 3. Migration 4. Auditing
  111. 111. Modernization: Federated Instance Instance Instance Instance Source1 Source2 Source1
  112. 112. Modernization: Federated
  113. 113. “I looked like a hero” Tony Young, CIO Informatica Modernization: Federated
  114. 114. Data movement required for 1 source DB and 4 clones 5x Source Data Copy < 1 x Source Data Copy S SC C C C V V V V Without Delphix (c = clone) With Delphix (v = virtual DB)
  115. 115. Modernization: Consolidation Without Delphix With Delphix
  116. 116. Dev QA UAT Dev QA UAT 2.6 2.7 Dev QA UAT 2.8 Data Control = Source Control for the Database Production Time Flow Modernization: Auditing & Version Control CIO Insurance 600 Applications CIO Investment Banking 180 Applications CIO South America 65 Applications
  117. 117. Use Cases 1. Development • Parallelized Environments • Full size environments • Self Service 2. QA • Fast • Parallel • Rollback • A/B testing 3. Recovery • Prod & Dev Backups • Surgical recovery • Recovery of Production • Recovery of Development • Bug Forensics 4. Business Intelligence • 24x7 Batches • Low Bandwidth • Temporal Data • Confidence Testing 5. Modernization • Federated • Consolidation • Migration • Auditing
  118. 118. Use Case Summary 1. Development 2. QA 3. Quality 4. Business Intelligence 5. Modernization
  119. 119. How expensive is the Data Constraint? Before and after Delphix w/ Fortune 500 : Dev throughput increase by 2x
  120. 120. How expensive is the Data Constraint? • 10 x Faster Financial Close – 21 days down to 2 • 9x Faster BI refreshes – 3 weeks per refresh to 3x a week • 2x faster Projects • 20 % less bugs
  121. 121. Agile Data Quotes • “Allowed us to shrink our project schedule from 12 months to 6 months.” – BA Scott, NYL VP App Dev • "It used to take 50-some-odd days to develop an insurance product, … Now we can get a product to the customer in about 23 days.” – Presbyterian Health • “Can't imagine working without it” – Ramesh Shrinivasan CA Department of General Services
  122. 122. Summary • Problem: Data is the constraint • Solution: Agile data is small & fast • Results: Deliver projects – Half the Time – Higher Quality – Increase Revenue Kyle@delphix.com kylehailey.com slideshare.net/khailey
  123. 123. Oracle 12c
  124. 124. 80MB buffer cache ?
  125. 125. 200GB Cache
  126. 126. 5000 Tnxs/minLatency 300 ms 1 5 10 20 30 60 100 200 with 1 5 10 20 30 60 100 200 Users
  127. 127. 8000 Tnxs/minLatency 600 ms 1 5 10 20 30 60 100 200 Users 1 5 10 20 30 60 100 200
  128. 128. $1,000,000 1TB cache on SAN $6,000 200GB shared cache on Delphix Five 200GB database copies are cached with :
  129. 129. Use Cases 1. Development • Parallelized Environments • Full size environments • Self Service 2. QA • Fast • Parallel • Rollback • A/B testing 3. Recovery • Prod & Dev Backups • Surgical recovery • Recovery of Production • Recovery of Development • Bug Forensics 4. Business Intelligence • 24x7 Batches • Low Bandwidth • Temporal Data • Confidence Testing 5. Modernization • Federated • Consolidation • Migration • Auditing
  130. 130. •END
  131. 131. Source Full Copy Source backup from SCN 1
  132. 132. Snapshot 1 Snapshot 2
  133. 133. Snapshot 1 Snapshot 2 Backup from SCN
  134. 134. Snapshot 1 Snapshot 2 Snapshot 3
  135. 135. Drop Snapshot Snapshot 1 Snapshot 2 Snapshot 3 Snapshot 2 Snapshot 3 Drop Snapshot 1
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×