This document discusses data virtualization and its use in DevOps. It begins by explaining that data virtualization, also known as copy data management, is becoming more common. It then discusses how data virtualization enables DevOps practices like continuous integration by allowing fast provisioning of full database environments. The document outlines some of the typical challenges with traditional database architectures, including long setup times, lack of parallel environments, and high storage costs due to many full database copies. It presents data virtualization as a solution, allowing instant provisioning of thin clones from a production database. Finally, it provides examples of how data virtualization can help with development/QA, production support, and business intelligence use cases.

1. Data Virtualization:
Revolutionizing data cloning
a.k.a. copy data management
1
kylehailey.com kyle@delphix.com @datavirt

2. Data virtualization
• Fast becoming the new norm
• Used by Over 100 of Fortune 500
• Enables DevOps

3. DevOps movement
• Goals Clarify
• Metrics Define
• Constraints Identify
• Priorities Set
• Iterations Fast

4. DevOps :
• Goals Clarify
• Metrics Define
• Constraints Identify
• Priorities Set
• Iterations Fast
• Continuous Integration
• Cloud
• Agile
• Kanban
• Kata
“IT is the factory floor of this century”

5. The Goal : Theory of Constraints
Improvement
not made
at the constraint
is an illusion
factory floor optimization

6. Factory floor

7. Factory floor
constraint
Not a relay race

8. Tune before constraint
constraint
Tuning here
Stock piling

9. Tune after constraint
constraint
Tuning here
Starvation

10. Factory floor : straight forward
constraint
Goal: find constraint
optimize it

11. The Phoenix Project
What is the
constraint
in IT ?

12. Put your energy into the constraint
Top 5 constraints in IT
1. Dev environments setup
2. QA setup
3. Code Architecture
4. Development
5. Product management
- Gene Kim
“One of the most powerful
things that organizations
can do is to enable development
and testing to get
environment they need when
they need it“

13. Data is the constraint
60% Projects Over Schedule
85% delayed waiting for data
Data is the Constraint
CIO Magazine Survey:
only getting worse
Gartner: Data Doomsday, by 2017 1/3rd IT in crisis

14. • Data Constraint
• Solution
• Use Cases
In this presentation :

15. Typical Architecture
Production
Instance
File system
Database

16. Typical Architecture
Production
Instance
Backup
File system
Database
File system
Database

17. Typical Architecture
Production
Instance
Reporting Backup
File system
Database
Instance
File system
Database
File system
Database

18. Typical Architecture
Production
Instance
File system
Database
Instance
File system
Database
File system
Database
File system
Database
Instance
Instance
Instance
File system
Database
File system
Database
Dev, QA, UAT Reporting Backup
Triple Tax

19. Typical Architecture
Production
Instance
File system
Database
Instance
File system
Database
File system
Database
File system
Database
Instance
Instance
Instance
File system
Database
File system
Database

20. Typical Architecture
Production
Instance
File system
Database
Instance
File system
Database
File system
Database
File system
Database
Instance
Instance
Instance
File system
Database
File system
Database

21. Copies
21
• Oracle customers : 8-12 copies per db
• Fortune 2K: 1000s multi-TB db
• Downstream storage staggering
- 3 petabytes at just one client

22. • Hardware
– storage, systems, network,
– rack space, power cooling
• People
– 1000s hours per year just for DBAs
– DBAs
– SYS Admin
– Storage Admin
– Backup Admin
– Network Admin
• $10s Millions for data center modernizations
Copies require People & Time

23. companies unaware

24. companies unaware
Developer or AnalystBoss, Storage Admin, DBA

25. Metrics
–Time
–Old Data
–Storage
Other
–Analysts
–Audits
–Data Center Modernization
companies unaware
"we say no, no, no until we can't say no anymore"
response when IT asked for copies of prod DB

26. 1. Waiting to check in code
2. Production Bugs
3. Expensive Slow QA
Biggest problem in Application Development

27. Development : bottlenecks
Frustration Waiting

28. Development : Bugs
Old Unrepresentative Data

29. Development : subsets
False Negatives
False Positives
Bugs in Production

30. Production Wall
30

31. QA : Long setup times
BugX
0
10
20
30
40
50
60
70
1 2 3 4 5 6 7
Delay in Fixing the bug
Cost
To
Correct
Software Engineering Economics
– Barry Boehm (1981)

32. QA : destructive tests refresh time
32
20 MIN TEST 20 MIN TEST 20 MIN TEST 20 MIN TEST 20 MIN TEST 20 MIN TEST 20 MIN TEST
8 Hrs8 Hrs8 Hrs8 Hrs8 Hrs8 Hrs8 Hrs 8 Hrs

33. • Data Constraint
• Solution
• Use Cases
In this presentation :

34. Development UATQA
99% of blocks are identical

35. Solution

36. Development QA UAT
Thin Clone

37. • EMC Symmetrix
– 16 snapshots
– Write performance impact
– No snapshots of snapshots
• Netapp & EMC VNX
– 255 snapshots
• ZFS
– Compression
– Unlimited snapshots
– Snapshots of Snapshots
• DxFS
– Compression
– Unlimited snapshots
– Snapshots of Snapshots
– Shared cache in memory
Technology Core : file system snapshots
Also check out new SSD storage such as: Pure Storage, EMC XtremIO

38. Snapshot 1 – full backup once only at link time
Jonathan Lewis © 2013
Virtual DB
38 / 30
a b c d e f g h i
We start with a full backup - analogous to a level 0 rman backup. Includes
the archived redo log files needed for recovery. Run in archivelog mode.

39. Snapshot 2 (from SCN)
Jonathan Lewis © 2013
b' c'
a b c d e f g h i
The "backup from SCN" is analogous to a level 1
incremental backup (which includes the relevant
archived redo logs). Sensible to enable BCT.
Delphix executes
standard rman scripts

40. Apply Snapshot 2
Jonathan Lewis © 2013
a b c d e f g h ib' c'
The Delphix appliance unpacks the rman backup and "overwrites" the
initial backup with the changed blocks - but DxFS makes new copies of
the blocks

41. Drop Snapshot 1
Jonathan Lewis © 2013
b' c'a d e f g h i
The call to rman leaves us with a new level 0 backup, waiting for recovery.
But we can pick the snapshot root block. We have EVERY level 0 backup

42. Creating a vDB
Jonathan Lewis © 2013
b' c'a d e f g h i
The first step in creating a vDB is to take a snapshot of the filesystem as at
the backup you want (then roll it forward)
My vDB
(filesystem)
Your vDB
(filesystem)
b' c'a d e f g h i

43. Creating a vDB
Jonathan Lewis
© 2013
b' c'a d e f g h i
The first step in creating a vDB is to take a snapshot of the filesystem as at
the backup you want (then roll it forward)
My vDB
(filesystem)
Your vDB
(filesystem)
i’b' c'a d e f g h ib' c'a d e f g h i

44. Fuel not equal car
Challenges
1. Technical
2. Bureaucracy

45. Bureaucracy
Developer Asks for DB Get Access
Manager approves
DBA Request
system
Setup DB
System
Admin
Request
storage
Setup
machine
Storage
Admin
Allocate
storage
(take snapshot)

46. Why are hand offs so expensive?
1hour
1 day
9 days
Bureaucracy

47. Technical Challenge
Database
Luns
Production Filer
Target A
Target B
Target C
snapshot
clones
InstanceInstance
InstanceInstance
InstanceInstance
InstanceInstance
Instance
Source

48. Database
LUNs
snapshot
clonesProduction Filer
Development Filer
Technical Challenge
Instance
Target A
Target B
Target C
InstanceInstance
InstanceInstance
InstanceInstance
Instance

49. Technical Challenge
Copy
Time Flow
Purge
Production
File System Instance
TargetStorage
Clone (snapshot)
Compress
Share Cache
Provision
Mount, recover, rename
Self Service, Roles & Security
Instance
21 3

50. How to get a Data Virtualization?
Source
sync
Target
Deploy
Storage
snapshots
21 3
Source Sync Storage Snapshots Deploy automation
ZFS Yes (unlimited)
EMC SRDF Yes (16 or 255)
Netapp SMO Yes (255)
Oracle EM 12c Data Guard Netapp, ZFS Yes (oracle only, no
branching)
Actifio Yes Yes Yes (no branching)
Delphix Yes Yes yes

51. Actifio
Production
InstanceInstanceInstance
Actifio
InstanceInstance Instance
TargetActifio
Instance
Target

52. Oracle Snap Clone
ZFSSA
or
NetApp
Instance
TargetEM 12c
Instance
Target
Production
InstanceInstanceInstance

53. Oracle Snap Clone
Production
InstanceInstanceInstance
Data Guard
InstanceInstanceInstance
ZFSSA
or
NetApp
Instance
TargetEM 12c
Instance
Target

54. Oracle Snap Clone
Production
InstanceInstanceInstance EM 12c
Solaris
ZFS
Instance
TargetData Guard
Instance
Instance
Target
Any
storage

55. Incremental forever collect changes
Production
InstanceInstanceInstance
Time Flow
Changes
Instance
NFS
Target
Instance
Target

56. Database Virtualization

57. Three Physical Copies
Three Virtual Copies
Data
Virtualization
Appliance

58. Before Virtual Data
Production Dev, QA, UAT
Instance
Reporting Backup
File system
Database
Instance
File system
Database
File system
Database
File system
Database
Instance
Instance
Instance
File system
Database
File system
Database
“triple data
tax”

59. With Virtual Data
Production
Instance
Dev & QA
Instance
Reporting
Instance
Backup
Instance Instance Instance
InstanceInstance
Instance
File system
Database
Data
Virtualization
Appliance
Instance

60. • Problem in the Industry
• Solution
• Use Cases

61. 1. Development and QA
2. Production Support
3. Business
Use Cases

62. 1. Development & QA
2. Production Support
3. Business
Use Cases

63. Development: Virtual Data
Development
* Fast * Free * Full size * Self service

64. Virtual Data: Easy
Instance
Instance
Instance
Instance
Source
DVA

65. Development Virtual Data: Parallelize
gif by Steve Karam

66. Development Virtual Data: Full size

67. Development Virtual Data: Self Service

68. QA : Virtual Data
• Fast
• Parallel
• Rollback
• A/B testing

69. Dev
QA
Instance
Prod
DVA
• Eliminate build time
• Find bugs Fast
• Run Parallel QA
QA Virtual Data : Parallel
Production Time Flow

70. QA Virtual Data : Fast Refresh
70
20 MIN TEST 20 MIN TEST 20 MIN TEST 20 MIN TEST 20 MIN TEST 20 MIN TEST 20 MIN TEST
• Fast
• Full
• Fresh
• Efficient
8 Hrs8 Hrs8 Hrs8 Hrs8 Hrs8 Hrs8 Hrs 8 Hrs
20 MIN
TEST

71. QA with Virtual Data: Rewind
DVAInstance
QA
Prod
Production Time Flow

72. QA with Virtual Data: A/B
DVAInstance
Instance
Instance
Index 1
Index 2
Production Time Flow

73. Data Version Control
1/30/2015 73
Dev
QA
2.1
Dev
QA
2.2
2.1 2.2
Instance
Prod
DVA Production Time Flow

74. 1. Development and QA
2. Production Support
3. Business
Use Cases

75. • Backups
• Recovery
• Forensics
• Migration
• Consolidation
Production Support

76. 9TB database 1TB change day
30 day backups storage requirements
76
0
10
20
30
40
50
60
70
week1
week2
week3
week4
original
Oracle
Delphix

77. Recovery
Instance
Instance
Recover VDB
Drop
Source
DVA Production Time Flow

78. Forensics
Instance
Development
DVA
Source
Production Time Flow

79. Development (the new production)
Instance
Development
DVA
Source
Development
Prod & VDB Time Flow

80. Migration

81. 1. Development and QA
2. Production Support
3. Business Intelligence
Use Cases

82. Business Intelligence
• ETL
• Temporal
• Confidence Testing
• Federated Databases
• Audits

83. Business Intelligence: ETL and Refresh Windows
1pm 10pm 8am
noon

84. Business Intelligence: batch taking too long
1pm 10pm 8am
noon
2011
2012
2013
2014
2015

85. 2011
2012
2013
2014
2015
1pm 10pm 8am
noon
10pm 8am noon 9pm
6am 8am 10pm

86. Business Intelligence: ETL and DW Refreshes
Instance
Prod
Instance
DW & BI

87. • Collect only Changes
• Refresh in minutes
Instance
Prod
BI and DW
ETL
24x7
DVA
Virtual Data: Fast Refreshes
Production Time Flow

88. Temporal Data

89. Confidence testing

90. Modernization: Federated
Instance
Instance
Source1
Source2
DVA
Production Time Flow 1
Production Time Flow 2

91. Modernization: Federated

92. “I looked like a hero”
Tony Young, CIO Informatica
Modernization: Federated

93. Production Time Flow
Audit
1/30/2015 93
Instance
Prod
DVA
Live Archive

94. 1. Development & QA
2. Production Support
3. Business
Use Case Summary

95. How expensive is the Data Constraint?
DVA at Fortune 500 :
Dev throughput increase by 2x

96. Faster
• Financial Close
• BI refreshes
• Surgical recovery
• Projects
How expensive is the Data Constraint?

97. • Projects “12 months to 6 months.”
– New York Life
• Insurance product “about 50 days ... to about 23 days”
– Presbyterian Health
• “Can't imagine working without it”
– State of California
Virtual Data Quotes

98. • Problem: Data is the constraint
• Solution: Virtualize Data
• Results:
• Half the time for projects
• Higher quality
• Increase revenue
Summary

99. Thank you!
• Kyle Hailey| Oracle ACE and Technical
Evangelist, Delphix
– Kyle@delphix.com
– kylehailey.com
– slideshare.net/khailey
– @datavirt