This document discusses how data virtualization can help address the challenges posed by data being a constraint in IT. It describes how data flooding infrastructure strains resources and costs companies huge amounts of money. Typical database architectures result in a "triple data tax". The document proposes data virtualization as a solution, allowing for unlimited clones of database environments using thin cloning. This enables faster development and testing, quicker disaster recovery, and real-time business intelligence. Case studies show benefits like doubling development throughput, 9x faster analytics refreshes, and cutting project timelines in half.

1. Data Virtualization Platform
revolutionizing database cloning
How can the DBA make the biggest
impact on the company
6/18/2014 1
http://kylehailey.com
kyle@delphix.com

2. The Goal
Theory of Constraints
Improvement
not made
at the constraint
is an illusion
factory floor optimization

3. Factory floor :
straight forward

4. Factory floor :
straight forward
constraint

5. Factory floor :
optimize at the constraint
constraint
Tuning here
Stock piling

6. Factory floor :
optimize at the constraint
constraint
Tuning here
Stock piling
Tuning here
Starvation

7. Factory floor :
straight forward
constraint

8. Factory Floor
Optimization

9. Theory of Constraints
work for IT ?
• Goals Clarify
• Metrics Define
• Constraints Identify
• Priorities Set
• Iterations Fast
• CI
• Cloud
• Agile
• Kanban
“IT is the factory floor of this century”

10. The Phoenix Project
What is the
constraint
in IT ?
“One of the most powerful things that organizations
can do is to enable development and testing to get
environment they need when they need it“

11. What is the constraint in IT
If you can’t satisfy
the business demands
then your process is broken.

12. Data is the constraint
60% Projects Over Schedule
85% delayed waiting for data
Data is the Constraint
CIO Magazine Survey:
only getting worse

13. • Data Constraint
I. strains IT
II. price is huge
III. companies unaware
• Solution
• Use Cases
In this presentation :

14. • Data Constraint
I. strains IT
II. price is huge
III. companies unaware
• Solution
• Use Cases
In this presentation :

15. – Storage & Systems
– Personnel
– Time
I. Data Constraint :
moving data is hard

16. Typical Architecture
Production
Instance
File system
Database

17. Typical Architecture
Production
Instance
Backup
File system
Database
File system
Database

18. Typical Architecture
Production
Instance
Reporting Backup
File system
Database
Instance
File system
Database
File system
Database

19. Typical Architecture
Production
Instance
File system
Database
Instance
File system
Database
File system
Database
File system
Database
Instance
Instance
Instance
File system
Database
File system
Database
Dev, QA, UAT Reporting Backup
Triple Tax

20. Typical Architecture
Production
Instance
File system
Database
Instance
File system
Database
File system
Database
File system
Database
Instance
Instance
Instance
File system
Database
File system
Database

21. I. Data constraint:
Data floods infrastructure
92% of the cost of business,
in financial services business , is “data”
www.wsta.org/resources/industry-articles
Most companies have
2-9% IT spending , ½ on “data”
http://uclue.com/?xq=1133
Gartner: Data Doomsday

22. • Data Constraint
I. strains IT
II. price is huge
III. companies unaware
• Solution
• Use Cases
In this presentation :

23. • Four Areas data tax hits
1. IT Capital resources $
2. IT Operations personnel $
3. Application Development $$$
4. Business $$$$$$$
II. Data constraint:
price is Huge

24. • Four Areas data tax hits
1. IT Capital resources $
2. IT Operations personnel $
3. Application Development $$$
4. Business $$$$$$$
II. Data constraint:
price is Huge

25. • Hardware
–Servers
–Storage
–Network
–Data center floor space, power, cooling
II. Data constraint price is huge :
IT Capital

26. • Four Areas data tax hits
1. IT Capital resources $
2. IT Operations personnel $
3. Application Development $$$
4. Business $$$$$$$
II. Data constraint:
price is Huge

27. • People
– DBAs
– SYS Admin
– Storage Admin
– Backup Admin
– Network Admin
• Hours : 1000s just for DBAs
• $100s Millions for data center modernizations
II. Data constraint price is huge:
IT Operations

28. • Four Areas data tax hits
1. IT Capital resources $
2. IT Operations personnel $
3. Application Development $$$
4. Business $$$$$$$
II. Data constraint:
price is Huge
“One of the most powerful things that IT can do is get
environments to development and QA when they need it”
- Gene Kim author of The Phoenix Project

29. • Inefficient QA: Higher costs of QA
• QA Delays : Greater re-work of code
• Sharing DB Environments : Bottlenecks
• Using DB Subsets: More bugs in Prod
• Slow Environment Builds: Delays
II. Data constraint price is Huge :
Application Development

30. • Four Areas data tax hits
1. IT Capital resources $
2. IT Operations personnel $
3. Application Development $$$
4. Business $$$$$$$
Part II. Data constraint:
price is Huge

31. Ability to capture revenue
• Business Applications
– Delays cause lost revenue
• Business Intelligence
– Old data = less intelligence
II. Data constraint price is Huge :
Business

32. • Data Constraint
I. strains IT
II. price is huge
III. companies unaware
• Solution
• Use Cases
In this presentation :

33. Part III. Data Constraint
companies unaware

34. III. Data Constraint
companies unaware
DeveloperDBA

35. Metrics
–Time
–Old Data
–Storage
–Analysts
–Audits
III. Data Constraint
companies unaware

36. • Data Constraint
I. strains IT
II. price is huge
III. companies unaware
• Solution
• Use Cases
In this presentation :

37. Clone 1 Clone 3Clone 2
99% of blocks are
identical

38. Solution

39. Clone 1 Clone 2 Clone 3
Thin Clone

40. • Vmware Data Director
– Linked clones not supported for Oracle
• EMC
– 16 snapshots on Symmetrix
– Write performance impact
– No snapshots of snapshots
• Netapp
– 255 snapshots
• ZFS
– Compression
– Unlimited snapshots
– Snapshots of Snapshots
• DxFS
– Compression
– Unlimited snapshots
– Snapshots of Snapshots
– Storage agnostic
– Shared cache in memory
Technology Core :
file system snapshots
Also check out new SSD storage such as:
Pure Storage, EMC XtremIO

41. Fuel not equal car
Challenges
1. Technical
2. Bureaucracy

42. 1. Technical Challenge
Database
Luns
Production Filer
Target A
Target B
Target C
snapshot
clones
InstanceInstance
InstanceInstance
InstanceInstance
InstanceInstance
Instance
Source

43. Database
LUNs
snapshot
clonesProduction Filer
Development Filer
1. Technical Challenge
Instance
Target A
Target B
Target C
InstanceInstance
InstanceInstance
InstanceInstance
Instance

44. 1. Technical Challenge
Copy
Time Flow
Purge
Production
File System Instance
DevelopmentStorage
21 3
Clone (snapshot)
Compress
Share Cache
Provision
Mount, recover, rename
Self Service, Roles & Security
Instance

45. 2. Bureaucracy Challenge
Developer Asks for DB Get Access
Manager approves
DBA Request
system
Setup DB
System
Admin
Request
storage
Setup
machine
Storage
Admin
Allocate
storage
(take snapshot)

46. Data Virtualization
How to get a Data Virtualization?
– EMC + SRDF + scripting
– Netapp + SMO + scripting
– Oracle EM 12c DBaaS + data guard + Netapp /ZFS + scripting
– Delphix
2 31
Production DevelopmentStorage
21 3
2 31
23 1
2 31

47. Data Supply Chain
6/18/2014 47
Data Supply Chain
• Security
• Masking
• Chain of custody
• Self Service
• Roles
• Restrictions
• Developer
• Data Versioning
• Refresh, Rollback
• Audit:
• Live Archive
Snap Shots
Thin Cloning
Data Virtualization
Data Supply Chain

48. Install Delphix on x86
hardware
Intel hardware
Application Stack Data

49. Allocate Any Storage to
Delphix
Allocate Storage
Any type
Pure Storage + Delphix
Better Performance for
1/10 the cost

50. One time backup of
source database
Database
Production
File systemFile system
InstanceInstanceInstance

51. DxFS (Delphix) Compress
Data
Database
Production
Data is
compressed
typically 1/3
size
File system
InstanceInstanceInstance

52. Incremental forever
change collection
Database
Production
File system
Changes
• Collected incrementally forever
• Old data purged
File system
Time Window
Production
InstanceInstanceInstance

53. Virtual DB
53 / 30Jonathan Lewis
© 2013
Snapshot 1 – full backup
once only at link time
a b c d e f g h i
We start with a full backup - analogous to a level 0 rman backup. Includes
the archived redo log files needed for recovery. Run in archivelog mode.

54. Virtual DB
54 / 30Jonathan Lewis
© 2013
Snapshot 2 (from SCN)
b' c'
a b c d e f g h i
The "backup from SCN" is analogous to a level 1 incremental backup (which
includes the relevant archived redo logs). Sensible to enable BCT.
Delphix executes
standard rman scripts

55. Virtual DB
55 / 30Jonathan Lewis
© 2013
a b c d e f g h i
Apply Snapshot 2
b' c'
The Delphix appliance unpacks the rman backup and "overwrites" the initial
backup with the changed blocks - but DxFS makes new copies of the blocks

56. Virtual DB
56 / 30Jonathan Lewis
© 2013
Drop Snapshot 1
b' c'a d e f g h i
The call to rman leaves us with a new level 0 backup, waiting for recovery.
But we can pick the snapshot root block. We have EVERY level 0 backup

57. Virtual DB
57 / 30Jonathan Lewis
© 2013
Creating a vDB
b' c'a d e f g h i
The first step in creating a vDB is to take a snapshot of the filesystem as at
the backup you want (then roll it forward)
My vDB
(filesystem)
Your vDB
(filesystem)
b' c'a d e f g h i

58. Virtual DB
58 / 30Jonathan Lewis
© 2013
Creating a vDB
b' c'a d e f g h i
The first step in creating a vDB is to take a snapshot of the filesystem as at
the backup you want (then roll it forward)
My vDB
(filesystem)
Your vDB
(filesystem)
i’b' c'a d e f g h ib' c'a d e f g h i

60. Database Virtualization

61. Three Physical Copies
Three Virtual Copies
Data
Virtualization
Appliance

62. Before Virtual Data
Production Dev, QA, UAT
Instance
Reporting Backup
File system
Database
Instance
File system
Database
File system
Database
File system
Database
Instance
Instance
Instance
File system
Database
File system
Database
“triple data
tax”

63. With Virtual Data
Production
Instance
Database
Dev & QA
Instance
Database
Reporting
Instance
Database
Backup
Instance Instance Instance
Database
InstanceInstance
Database
InstanceInstance
File system
Database
Data
Virtualization
Appliance

64. • Problem in the Industry
• Solution
• Use Cases
In this presentation :

65. 1. Development and QA
2. Recovery
3. Business
Use Cases

66. 1. Development and QA
2. Recovery
3. Business
Use Cases

67. Development : bottlenecks
Frustration Waiting
Old Unrepresentative Data

68. Development : subsets

69. Development : bugs

70. Development : slow
Developer Asks for DB Get Access
Manager approves
DBA Request
system
Setup DB
System
Admin
Request
storage
Setup
machine
Storage
Admin
Allocate
storage
(take snapshot)
Weeks to Months to Deliver Data

71. Development without Virtual Data: slow env build
times
Why are hand offs so expensive?
1hour
1 day
9 days

72. Development :
Lack of resources
Never enough environments

73. http://martinfowler.com/bliki/NoDBA.html
Development without Virtual Data: slow env build
times

74. Development: Virtual Data
• Unlimited
• Full size
• Self Service
Development:
Virtual Data
Development

75. Virtual Data:
Easy
Instance
Instance
Instance
Instance
Source
Data
Virtualization
Appliance
DVA
Parallel
Environments
• QA
• Dev

76. Development Virtual Data:
Parallelize
gif by Steve Karam

77. Development Virtual
Data: Full size

78. Development Virtual
Data: Self Service

79. Dev2
Dev1Merge to dev1
Trunk
DB
VC
Fork
Fork
Fork
Fork
DBmaestro

80. QA : Virtual Data
• Fast
• Parallel
• Rollback
• A/B testing
QA
Virtual Data

81. QA : Long Build times
96% of QA time was building environment
$.04/$1.00 actual testing vs. setup
QA Build QA
QA Build QA
QA before virtual : resource expensive

82. QA : slow
Bug CodeX
0
10
20
30
40
50
60
70
1 2 3 4 5 6 7
Delay in Fixing the bug
Cost
To
Correct
Software Engineering Economics – Barry Boehm (1981)
Sprint 1 Sprint 2
QA Build QA QA Build QA
Sprint 1 Sprint 2
QA before virtual : slow

83. QA Virtual Data : Fast
83
Dev
QA
Instance
Prod
DVA
Time Flow
1% of QA time was building environment
$.99/$1.00 actual testing vs. setup
QA
Build QA
QA
Build QA
QA Virtual Data : Fast

84. QA Virtual Data : Fast
Dev
QA
Instance
Prod
DVA
Time Flow
Bugs found quickly QA
Build QA
Sprint 1 Sprint 2
QA
Build QA
X
QA Virtual Data : Fast with Branching

85. QA with Virtual Data:
Rewind
Instance
Instance
Development
Prod

86. QA with Virtual Data: A/B
Instance
Instance
Instance
Index 1
Index 2

87. Data Version Control
6/18/2014 87
Dev
QA
2.1
Dev
QA
2.2
2.1 2.2
Instance
Prod
DVA

88. 1. Development and QA
2. Recovery
3. Business
Use Cases

89. • Backups
• Recovery
• Forensics
Recovery

90. Recovery: Backups

91. Recovery: scenarios
Instance
Instance
Recover VDB
Drop
Source
DVA

92. Recovery: Forensics
Instance
Instance
DevelopmentDVA
Source

93. Recovery: Development
Instance
Instance
Development
DVA
Source
Instance
Development

94. 1. Development and QA
2. Recovery
3. Business Intelligence
Use Cases

95. Business Intelligence

96. Business Intelligence:
ETL and Refresh Windows
1pm 10pm 8am
noon

97. Business Intelligence:
batch taking too long
1pm 10pm 8am
noon
2011
2012
2013
2014
2015

98. 2011
2012
2013
2014
2015
1pm 10pm 8am
noon
10pm 8am noon 9pm
6am 8am 10pm
Business Intelligence:
batch taking too long

99. Business Intelligence:
ETL and DW Refreshes
Instance
Prod
Instance
DW & BI
Before Virtual: limited, slow ETL and DW refreshes

100. • Collect only Changes
• Refresh in minutes
Virtual Data:
Fast Refreshes
Instance Instance
Prod BI and DW
ETL
24x7
DVA Instance
Virtual Data: Fast Refreshes

101. Temporal Data

102. Confidence testing

103. 1. Federated
2. Migration
3. Auditing
Modernization

104. Modernization:
Federated
Instance
Instance
Instance
Instance
Source1
Source2
DVA

105. Modernization:
Federated

106. “I looked like a hero”
Tony Young, CIO Informatica
Modernization:
Federated

107. Modernization:
Migration

108. Audit
6/18/2014 108
Instance
Prod
DVA
Live Archive

109. Consolidation

110. 1. Development & QA
2. Recovery
3. Business
Use Case Summary

111. How expensive is the
Data Constraint?
DVA at Fortune 500 :
Dev throughput increase by 2x

112. • 10 x Faster Financial Close
• 9x Faster BI refreshes
• 8x Faster surgical recovery
• 3x Project tracks
• 2x Faster Projects
How expensive is the
Data Constraint?

113. • Projects “12 months to 6 months.”
– New York Life
• Insurance product “about 50 days ... to about 23 days”
– Presbyterian Health
• “Can't imagine working without it”
– State of California
Virtual Data Quotes

115. • Problem: Data is the constraint
• Solution: Virtual Data
• Results:
– Half the time for projects
– Higher quality
– Increase revenue
Summary

116. Thank you!
• Kyle Hailey| Oracle ACE and Technical
Evangelist, Delphix
– Kyle@delphix.com
– kylehailey.com
– slideshare.net/khailey

117. Oracle 12c

118. 80MB buffer cache ?

119. 200GB
Cache

120. 5000
Tnxs/minLatency
300
ms
1 5 10 20 30 60 100 200
with
1 5 10 20 30 60 100 200
Users

121. 8000
Tnxs/minLatency
600
ms
1 5 10 20 30 60 100 200
Users
1 5 10 20 30 60 100 200

122. $1,000,000
1TB cache on SAN
$6,000
200GB shared cache on Delphix
Five 200GB database copies are
cached with :

123. 6/18/2014 123

124. 6/18/2014 124

125. Business Intelligence
a) 24x7 Batches
b) Temporal queries
c) Confidence testing

126. Thin Cloning

129. Snap
Manager
SnapManager
Repository
Protection
Manager
Snap Drive
Snap
Manager
Snap Mirror
Flex Clone
RMAN
Repository
Production
Development
DBA
Storage
Admin
1
tr-3761.pdf
Netapp

130. NetApp Filer - DevelopmentNetApp Filer - Production
Database
Luns
Snap
mirror
Snapshot Manager
for Oracle
Flexclone
Repository
Database
Snap
Drive
Protection
Manage
Production
Development
1 Netapp
Target A
Target B
Target C
InstanceInstance
InstanceInstance
InstanceInstance
Instance

131. Where we want to be
Database
File system
Production
Instance
Database
Development
Instance
Database
QA
Instance
Database
UAT
Instance
Snapshots
Instance Instance Instance Instance

132. EM 12c: Snap Clone
Production Development
Flexclone Flexclone
Netapp
Snap Manager for Oracle

133. II. Data constraint price is Huge : 4. Business

134. II. Data constraint price is
Huge : 4. Business
0 5 10 15 20 25 30
Storage
IT Ops
Dev
Revenue
Billion $

135. III. Data Constraint
companies unaware
#1 Biggest Enemy :
IT departments believe
– best processes
– greatest technology
– Just the way it is

136. There are always new
and better ways to do
things

137. III. Data Constraint
companies unaware
Why do I need an iPhone ?
Don’t we already do that ?
SQL scripts
Alter database begin backup
Back up datafiles
Redo
Archive
Alter database end backup
RMAN