Kscope 14 Presentation : Virtual Data Platform

729 views

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
729
On SlideShare
0
From Embeds
0
Number of Embeds
113
Actions
Shares
0
Downloads
26
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Kscope 14 Presentation : Virtual Data Platform

  1. 1. Virtual Data Platform: or Revolutionizing Database Cloning How can the DBA make the biggest impact on the company 6/26/2014 1 http://kylehailey.com kyle@delphix.com
  2. 2. The Goal Theory of Constraints Improvement not made at the constraint is an illusion factory floor optimization
  3. 3. Factory floor : straight forward
  4. 4. Factory floor : straight forward constraint Not a relay race
  5. 5. Factory floor : Tune before constraint constraint Tuning here Stock piling
  6. 6. Factory floor : Tune after constraint constraint Tuning here Starvation
  7. 7. Factory floor : straight forward constraint Goal: find the constraint and optimize it
  8. 8. Factory Floor Optimization Goal: find the constraint and optimize it
  9. 9. Theory of Constraints work for IT ? • Goals Clarify • Metrics Define • Constraints Identify • Priorities Set • Iterations Fast • CI • Cloud • Agile • Kanban “IT is the factory floor of this century”
  10. 10. The Phoenix Project What is the constraint in IT ? “One of the most powerful things that organizations can do is to enable development and testing to get environment they need when they need it“
  11. 11. 11 1. Dev environments 2. QA setup 3. Code Architecture 4. Development 5. Product management What is the constraint in IT?
  12. 12. What is the constraint in IT If you can’t satisfy the business demands then your process is broken.
  13. 13. Data is the constraint 60% Projects Over Schedule 85% delayed waiting for data Data is the Constraint CIO Magazine Survey: only getting worse
  14. 14. • Data Constraint I. strains IT II. price is huge III. companies unaware • Solution • Use Cases In this presentation :
  15. 15. • Data Constraint I. strains IT II. price is huge III. companies unaware • Solution • Use Cases In this presentation :
  16. 16. – Storage & Systems – Personnel – Time I. Data Constraint : moving data is hard
  17. 17. Typical Architecture Production Instance File system Database
  18. 18. Typical Architecture Production Instance Backup File system Database File system Database
  19. 19. Typical Architecture Production Instance Reporting Backup File system Database Instance File system Database File system Database
  20. 20. Typical Architecture Production Instance File system Database Instance File system Database File system Database File system Database Instance Instance Instance File system Database File system Database Dev, QA, UAT Reporting Backup Triple Tax
  21. 21. Typical Architecture Production Instance File system Database Instance File system Database File system Database File system Database Instance Instance Instance File system Database File system Database
  22. 22. I. Data constraint: Data floods infrastructure 92% of the cost of business, in financial services business , is “data” www.wsta.org/resources/industry-articles Most companies have 2-9% IT spending , ½ on “data” http://uclue.com/?xq=1133 Gartner: Data Doomsday
  23. 23. • Data Constraint I. strains IT II. price is huge III. companies unaware • Solution • Use Cases In this presentation :
  24. 24. • Four Areas data tax hits 1. IT Capital resources $ 2. IT Operations personnel $ 3. Application Development $$$ 4. Business $$$$$$$ II. Data constraint: price is Huge
  25. 25. • Four Areas data tax hits 1. IT Capital resources $ 2. IT Operations personnel $ 3. Application Development $$$ 4. Business $$$$$$$ II. Data constraint: price is Huge
  26. 26. • Hardware –Servers –Storage –Network –Data center floor space, power, cooling II. Data constraint price is huge : IT Capital
  27. 27. II. Data constraint price is huge : IT Capital Never enough environments
  28. 28. • Four Areas data tax hits 1. IT Capital resources $ 2. IT Operations personnel $ 3. Application Development $$$ 4. Business $$$$$$$ II. Data constraint: price is Huge
  29. 29. • People – DBAs – SYS Admin – Storage Admin – Backup Admin – Network Admin • Hours : 1000s just for DBAs • $100s Millions for data center modernizations II. Data constraint price is huge: IT Operations
  30. 30. II. Data constraint price is huge: IT Operations Developer Asks for DB Get Access Manager approves DBA Request system Setup DB System Admin Request storage Setup machine Storage Admin Allocate storage (take snapshot)
  31. 31. Why are hand offs so expensive? 1hour 1 day 9 days II. Data constraint price is huge: IT Operations
  32. 32. • Four Areas data tax hits 1. IT Capital resources $ 2. IT Operations personnel $ 3. Application Development $$$ 4. Business $$$$$$$ II. Data constraint: price is Huge “One of the most powerful things that IT can do is get environments to development and QA when they need it” - Gene Kim author of The Phoenix Project
  33. 33. • Inefficient QA: Higher costs of QA • QA Delays : Greater re-work of code • Sharing DB Environments : Bottlenecks • Using DB Subsets: More bugs in Prod • Slow Environment Builds: Delays II. Data constraint price is Huge : Application Development
  34. 34. • Four Areas data tax hits 1. IT Capital resources $ 2. IT Operations personnel $ 3. Application Development $$$ 4. Business $$$$$$$ Part II. Data constraint: price is Huge
  35. 35. Ability to capture revenue • Business Applications – Delays cause lost revenue • Business Intelligence – Old data = less intelligence II. Data constraint price is Huge : Business
  36. 36. • Data Constraint I. strains IT II. price is huge III. companies unaware • Solution • Use Cases In this presentation :
  37. 37. Part III. Data Constraint companies unaware
  38. 38. III. Data Constraint companies unaware Developer or AnalystBoss, Storage Admin, DBA
  39. 39. Metrics –Time –Old Data –Storage –Analysts –Audits III. Data Constraint companies unaware
  40. 40. • Data Constraint I. strains IT II. price is huge III. companies unaware • Solution • Use Cases In this presentation :
  41. 41. Clone 1 Clone 3Clone 2 99% of blocks are identical
  42. 42. Solution
  43. 43. Clone 1 Clone 2 Clone 3 Thin Clone
  44. 44. • EMC – 16 snapshots on Symmetrix – Write performance impact – No snapshots of snapshots • Netapp – 255 snapshots • ZFS – Compression – Unlimited snapshots – Snapshots of Snapshots • DxFS – “” – Storage agnostic – Shared cache in memory Technology Core : file system snapshots Also check out new SSD storage such as: Pure Storage, EMC XtremIO
  45. 45. Fuel not equal car Challenges 1. Technical 2. Bureaucracy
  46. 46. II. Data constraint price is huge: IT Operations Developer Asks for DB Get Access Manager approves DBA Request system Setup DB System Admin Request storage Setup machine Storage Admin Allocate storage (take snapshot)
  47. 47. 1. Technical Challenge Database Luns Production Filer Target A Target B Target C snapshot clones InstanceInstance InstanceInstance InstanceInstance InstanceInstance Instance Source
  48. 48. Database LUNs snapshot clonesProduction Filer Development Filer 1. Technical Challenge Instance Target A Target B Target C InstanceInstance InstanceInstance InstanceInstance Instance
  49. 49. 1. Technical Challenge Copy Time Flow Purge Production File System Instance DevelopmentStorage 21 3 Clone (snapshot) Compress Share Cache Provision Mount, recover, rename Self Service, Roles & Security Instance
  50. 50. Data Virtualization How to get a Data Virtualization? – EMC + SRDF + scripting – Netapp + SMO + scripting – Oracle EM 12c DBaaS + data guard + Netapp /ZFS + scripting – Delphix 2 31 Production DevelopmentStorage 21 3 2 31 23 1 2 31
  51. 51. Final Goal 6/26/2014 51 Data Supply Chain • Security • Masking • Chain of custody • Self Service • Roles • Restrictions • Developer • Data Versioning • Refresh, Rollback • Audit: • Live Archive Snap Shots Thin Cloning Data Virtualization Data Supply Chain
  52. 52. Install Delphix on x86 hardware Intel hardware Application Stack Data
  53. 53. Allocate Any Storage to Delphix Allocate Storage Any type Pure Storage + Delphix Better Performance for 1/10 the cost
  54. 54. One time backup of source database Database Production File systemFile system InstanceInstanceInstance
  55. 55. DxFS (Delphix) Compress Data Database Production Data is compressed typically 1/3 size File system InstanceInstanceInstance
  56. 56. Incremental forever change collection Database Production File system Changes • Collected incrementally forever • Old data purged File system Time Window Production InstanceInstanceInstance
  57. 57. Virtual DB 57 / 30Jonathan Lewis © 2013 Snapshot 1 – full backup once only at link time a b c d e f g h i We start with a full backup - analogous to a level 0 rman backup. Includes the archived redo log files needed for recovery. Run in archivelog mode.
  58. 58. Virtual DB 58 / 30Jonathan Lewis © 2013 Snapshot 2 (from SCN) b' c' a b c d e f g h i The "backup from SCN" is analogous to a level 1 incremental backup (which includes the relevant archived redo logs). Sensible to enable BCT. Delphix executes standard rman scripts
  59. 59. Virtual DB 59 / 30Jonathan Lewis © 2013 a b c d e f g h i Apply Snapshot 2 b' c' The Delphix appliance unpacks the rman backup and "overwrites" the initial backup with the changed blocks - but DxFS makes new copies of the blocks
  60. 60. Virtual DB 60 / 30Jonathan Lewis © 2013 Drop Snapshot 1 b' c'a d e f g h i The call to rman leaves us with a new level 0 backup, waiting for recovery. But we can pick the snapshot root block. We have EVERY level 0 backup
  61. 61. Virtual DB 61 / 30Jonathan Lewis © 2013 Creating a vDB b' c'a d e f g h i The first step in creating a vDB is to take a snapshot of the filesystem as at the backup you want (then roll it forward) My vDB (filesystem) Your vDB (filesystem) b' c'a d e f g h i
  62. 62. Virtual DB 62 / 30Jonathan Lewis © 2013 Creating a vDB b' c'a d e f g h i The first step in creating a vDB is to take a snapshot of the filesystem as at the backup you want (then roll it forward) My vDB (filesystem) Your vDB (filesystem) i’b' c'a d e f g h ib' c'a d e f g h i
  63. 63. Database Virtualization
  64. 64. Three Physical Copies Three Virtual Copies Data Virtualization Appliance
  65. 65. Before Virtual Data Production Dev, QA, UAT Instance Reporting Backup File system Database Instance File system Database File system Database File system Database Instance Instance Instance File system Database File system Database “triple data tax”
  66. 66. With Virtual Data Production Instance Database Dev & QA Instance Database Reporting Instance Database Backup Instance Instance Instance Database InstanceInstance Database InstanceInstance File system Database Data Virtualization Appliance
  67. 67. • Problem in the Industry • Solution • Use Cases In this presentation :
  68. 68. 1. Development and QA 2. Recovery 3. Business Use Cases
  69. 69. 1. Development and QA 2. Recovery 3. Business Use Cases
  70. 70. Development : bottlenecks Frustration Waiting Old Unrepresentative Data
  71. 71. Development : subsets
  72. 72. Development : bugs
  73. 73. http://martinfowler.com/bliki/NoDBA.html Development without Virtual Data: slow env build times
  74. 74. Development: Virtual Data • Unlimited • Full size • Self Service Development: Virtual Data Development
  75. 75. Virtual Data: Easy Instance Instance Instance Instance Source Data Virtualization Appliance DVA Parallel Environments • QA • Dev
  76. 76. Development Virtual Data: Parallelize gif by Steve Karam
  77. 77. Development Virtual Data: Full size
  78. 78. Development Virtual Data: Self Service
  79. 79. Dev2 Dev1Merge to dev1 Trunk DB VC Fork Fork Fork Fork DBmaestro
  80. 80. QA : Virtual Data • Fast • Parallel • Rollback • A/B testing QA Virtual Data
  81. 81. QA : Long Build times 96% of QA time was building environment $.04/$1.00 actual testing vs. setup QA Build QA QA Build QA QA before virtual : resource expensive & slow BugX 0 10 20 30 40 50 60 70 1 2 3 4 5 6 7 Delay in Fixing the bug Cost To Correct Software Engineering Economics – Barry Boehm (1981)
  82. 82. QA Virtual Data : Fast Dev QA Instance Prod DVA Time Flow • Low Resource • Find bugs Fast QA Virtual Data : Fast
  83. 83. QA with Virtual Data: Rewind Instance Instance Development Prod
  84. 84. QA with Virtual Data: A/B Instance Instance Instance Index 1 Index 2
  85. 85. Data Version Control 6/26/2014 86 Dev QA 2.1 Dev QA 2.2 2.1 2.2 Instance Prod DVA
  86. 86. 1. Development and QA 2. Recovery 3. Business Use Cases
  87. 87. • Backups • Recovery • Forensics Recovery
  88. 88. Recovery: Backups
  89. 89. Recovery: scenarios Instance Instance Recover VDB Drop Source DVA
  90. 90. Recovery: Forensics Instance Instance DevelopmentDVA Source
  91. 91. Recovery: Development Instance Instance Development DVA Source Instance Development
  92. 92. 1. Development and QA 2. Recovery 3. Business Intelligence Use Cases
  93. 93. Business Intelligence
  94. 94. Business Intelligence: ETL and Refresh Windows 1pm 10pm 8am noon
  95. 95. Business Intelligence: batch taking too long 1pm 10pm 8am noon 2011 2012 2013 2014 2015
  96. 96. 2011 2012 2013 2014 2015 1pm 10pm 8am noon 10pm 8am noon 9pm 6am 8am 10pm Business Intelligence: batch taking too long
  97. 97. Business Intelligence: ETL and DW Refreshes Instance Prod Instance DW & BI Before Virtual: limited, slow ETL and DW refreshes
  98. 98. • Collect only Changes • Refresh in minutes Virtual Data: Fast Refreshes Instance Instance Prod BI and DW ETL 24x7 DVA Instance Virtual Data: Fast Refreshes
  99. 99. Temporal Data
  100. 100. Confidence testing
  101. 101. 1. Federated 2. Migration 3. Auditing Modernization
  102. 102. Modernization: Federated Instance Instance Instance Instance Source1 Source2 DVA
  103. 103. Modernization: Federated
  104. 104. “I looked like a hero” Tony Young, CIO Informatica Modernization: Federated
  105. 105. Modernization: Migration
  106. 106. Audit 6/26/2014 107 Instance Prod DVA Live Archive
  107. 107. Consolidation
  108. 108. 1. Development & QA 2. Recovery 3. Business Use Case Summary
  109. 109. How expensive is the Data Constraint? DVA at Fortune 500 : Dev throughput increase by 2x
  110. 110. • 10 x Faster Financial Close • 9x Faster BI refreshes • 8x Faster surgical recovery • 3x Project tracks • 2x Faster Projects How expensive is the Data Constraint?
  111. 111. • Projects “12 months to 6 months.” – New York Life • Insurance product “about 50 days ... to about 23 days” – Presbyterian Health • “Can't imagine working without it” – State of California Virtual Data Quotes
  112. 112. • Problem: Data is the constraint • Solution: Virtual Data • Results: – Half the time for projects – Higher quality – Increase revenue Summary
  113. 113. Thank you! • Kyle Hailey| Oracle ACE and Technical Evangelist, Delphix – Kyle@delphix.com – kylehailey.com – slideshare.net/khailey
  114. 114. Oracle 12c
  115. 115. 80MB buffer cache ?
  116. 116. 200GB Cache
  117. 117. 5000 Tnxs/minLatency 300 ms 1 5 10 20 30 60 100 200 with 1 5 10 20 30 60 100 200 Users
  118. 118. 8000 Tnxs/minLatency 600 ms 1 5 10 20 30 60 100 200 Users 1 5 10 20 30 60 100 200
  119. 119. $1,000,000 1TB cache on SAN $6,000 200GB shared cache on Delphix Five 200GB database copies are cached with :
  120. 120. 6/26/2014 122
  121. 121. 6/26/2014 123
  122. 122. Business Intelligence a) 24x7 Batches b) Temporal queries c) Confidence testing
  123. 123. Thin Cloning
  124. 124. Snap Manager SnapManager Repository Protection Manager Snap Drive Snap Manager Snap Mirror Flex Clone RMAN Repository Production Development DBA Storage Admin 1 tr-3761.pdf Netapp
  125. 125. NetApp Filer - DevelopmentNetApp Filer - Production Database Luns Snap mirror Snapshot Manager for Oracle Flexclone Repository Database Snap Drive Protection Manage Production Development 1 Netapp Target A Target B Target C InstanceInstance InstanceInstance InstanceInstance Instance
  126. 126. Where we want to be Database File system Production Instance Database Development Instance Database QA Instance Database UAT Instance Snapshots Instance Instance Instance Instance
  127. 127. EM 12c: Snap Clone Production Development Flexclone Flexclone Netapp Snap Manager for Oracle
  128. 128. II. Data constraint price is Huge : 4. Business
  129. 129. II. Data constraint price is Huge : 4. Business 0 5 10 15 20 25 30 Storage IT Ops Dev Revenue Billion $
  130. 130. III. Data Constraint companies unaware #1 Biggest Enemy : IT departments believe – best processes – greatest technology – Just the way it is
  131. 131. There are always new and better ways to do things
  132. 132. III. Data Constraint companies unaware Why do I need an iPhone ? Don’t we already do that ? SQL scripts Alter database begin backup Back up datafiles Redo Archive Alter database end backup RMAN

×