Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

DevOps, Databases and The Phoenix Project UGF4042 from OOW14

966 views

Published on

What's the biggest constraint in IT, how can you fix it and what are the use cases

Published in: Software
  • Be the first to comment

DevOps, Databases and The Phoenix Project UGF4042 from OOW14

  1. 1. Virtual Data Platform: or Revolutionizing Database Cloning How can the DBA make the biggest impact on the company 1 http://kylehailey.com kyle@delphix.com
  2. 2. The Goal : Theory of Constraints Improvement not made at the constraint is an illusion factory floor optimization
  3. 3. Factory floor
  4. 4. Factory floor constraint Not a relay race
  5. 5. Tune before constraint constraint Tuning here Stock piling
  6. 6. Tune after constraint constraint Tuning here Starvation
  7. 7. Factory floor : straight forward constraint Goal: find constraint optimize it
  8. 8. Theory of Constraints work for IT ? • Goals Clarify • Metrics Define • Constraints Identify • Priorities Set • Iterations Fast • CI • Cloud • Agile • Kanban • Kata “IT is the factory floor of this century”
  9. 9. The Phoenix Project What is the constraint in IT ?
  10. 10. What are the top 5 constraints in IT? 1. Dev environments setup 2. QA setup 3. Code Architecture 4. Development 5. Product management “One of the most powerful things that organizations can do is to enable development and testing to get environment they need when they need it“ - Gene Kim
  11. 11. Data is the constraint CIO Magazine Survey: 60% Projects Over Schedule 85% delayed waiting for data Data is the Constraint only getting worse Gartner: Data Doomsday, by 2017 1/3rd IT in crisis
  12. 12. In this presentation : • Data Constraint • Solution • Use Cases
  13. 13. • Data Constraint I. strains IT II. price is huge III. companies unaware • Solution • Use Cases
  14. 14. moving data is hard – Storage & Systems – Personnel – Time
  15. 15. Typical Architecture Production Instance Database File system
  16. 16. Typical Architecture Production Instance Backup Database File system Database File system
  17. 17. Typical Architecture Production Instance Reporting Backup Database File system Instance Database File system Database File system
  18. 18. Typical Architecture Production Instance Database File system Triple Tax Dev, QA, UAT Reporting Backup Instance Instance Instance Instance Database Database File system Database File system File system Database File system Database File system
  19. 19. Typical Architecture Production Instance Database File system Instance Instance Instance Instance Database Database File system Database File system File system Database File system Database File system
  20. 20. Data floods infrastructure 92% of the cost of business, in financial services business , is “data” www.wsta.org/resources/industry-articles Most companies have 2-9% IT spending , ½ on “data” http://uclue.com/?xq=1133
  21. 21. In this presentation : • Data Constraint I. strains IT II. price is huge III. companies unaware • Solution • Use Cases
  22. 22. price is Huge Four Areas data tax hits – IT Capital resources $ – IT Operations personnel $ – Application Development $$$ – Business $$$$$$$
  23. 23. $Hardware –Servers –Storage –Network –Data center floor space, power, cooling
  24. 24. $ Never enough environments
  25. 25. $ IT Operations • People – DBAs – SYS Admin – Storage Admin – Backup Admin – Network Admin • Hours : 1000s just for DBAs • $100s Millions for data center modernizations
  26. 26. $ Application Development • Inefficient QA: Higher costs of QA • QA Delays : Greater re-work of code • Sharing DB Environments : Bottlenecks • Using DB Subsets: More bugs in Prod • Slow Environment Builds: Delays
  27. 27. $ Business Ability to capture revenue • Business Applications – Delays cause lost revenue • Business Intelligence – Old data = less intelligence
  28. 28. In this presentation : • Data Constraint I. strains IT II. price is huge III. companies unaware • Solution • Use Cases
  29. 29. companies unaware
  30. 30. companies unaware Boss, Storage Admin, DBA Developer or Analyst
  31. 31. companies unaware Metrics –Time –Old Data –Storage –Analysts –Audits
  32. 32. In this presentation : • Data Constraint I. strains IT II. price is huge III. companies unaware • Solution • Use Cases
  33. 33. 99% of blocks are identical Clone 1 Clone 2 Clone 3
  34. 34. Solution
  35. 35. Thin Clone Clone 1 Clone 2 Clone 3
  36. 36. Technology Core : file system snapshots • EMC – 16 snapshots on Symmetrix – Write performance impact – No snapshots of snapshots • Netapp – 255 snapshots • ZFS – Compression – Unlimited snapshots – Snapshots of Snapshots • DxFS – “” – Storage agnostic – Shared cache in memory Also check out new SSD storage such as: Pure Storage, EMC XtremIO
  37. 37. Fuel not equal car Challenges 1. Technical 2. Bureaucracy
  38. 38. Bureaucracy Developer Asks for DB Get Access Manager approves DBA Request system Setup DB System Admin Request storage Setup machine Storage Admin Allocate storage (take snapshot)
  39. 39. 1hour 9 days 1 day Why are hand offs so expensive? Bureaucracy
  40. 40. Technical Challenge Production Filer Database Luns Target A Target B Target C snapshot clones InsIntsatannccee InInssttaanncece InInssttaannccee InInssttaannccee Instance Source
  41. 41. Development Filer Production Filer clones Database LUNs snapshot Technical Challenge Instance Target A InInssttaannccee Target B InInssttaanncece Target C InInssttaannccee Instance
  42. 42. Technical Challenge Production Copy Time Flow Purge Storage Development File System Instance 1 2 3 Clone (snapshot) Compress Share Cache Provision Mount, recover, rename Self Service, Roles & Security Instance
  43. 43. How to get a Data Virtualization? 2 1 – EMC + SRDF – Netapp 2 + SMO 1 – Oracle EM 12c + data guard + Netapp /ZFS – Delphix 3 1 2 Source sync Deploy automation Storage snapshots 1 2 3
  44. 44. Goal : virtualize, govern, deliver • Security • Masking • Chain of custody • Self Service • Roles • Restrictions • Developer • Data Versioning • Refresh, Rollback • Audit: Data Supply Chain Data Virtualization Thin Cloning • Live Archive Snap Shots 10/3/2014 44
  45. 45. Intel hardware Unstructured Data Install Delphix on x86 hardware
  46. 46. Allocate Any Storage to Delphix Allocate Storage Any type Pure Storage + Delphix Better Performance for 1/10 the cost
  47. 47. One time backup of source database Production InsIIntnsasttanannccceee Database File system
  48. 48. DxFS (Delphix) Compress Data Production InsIIntnsasttanannccceee Database Data is compressed typically 1/3 size File system
  49. 49. Incremental forever change collection Production Database File system Changes Time Window • Collected incrementally forever • Old data purged InsIIntnsasttanannccceee
  50. 50. Snapshot 1 – full backup once only at link time Jonathan Lewis © 2013 Virtual DB 50 / 30 a b c d e f g h i We start with a full backup - analogous to a level 0 rman backup. Includes the archived redo log files needed for recovery. Run in archivelog mode.
  51. 51. Snapshot 2 (from SCN) a b c d e f g h i b' c' The "backup from SCN" is analogous to a level 1 incremental backup (which includes the relevant archived redo logs). Sensible to enable BCT. Jonathan Lewis © 2013 Delphix executes standard rman scripts
  52. 52. Apply Snapshot 2 a b b' c c' d e f g h i The Delphix appliance unpacks the rman backup and "overwrites" the initial backup with the changed blocks - but DxFS makes new copies of the blocks Jonathan Lewis © 2013
  53. 53. Drop Snapshot 1 a b' c' d e f g h i The call to rman leaves us with a new level 0 backup, waiting for recovery. But we can pick the snapshot root block. We have EVERY level 0 backup Jonathan Lewis © 2013
  54. 54. Creating a vDB a b' c' d e f g h i The first step in creating a vDB is to take a snapshot of the filesystem as at the backup you want (then roll it forward) Jonathan Lewis © 2013 My vDB (filesystem) Your vDB (filesystem)
  55. 55. Creating a vDB a b' c' d e f g h i The first step in creating a vDB is to take a snapshot of the filesystem as at the backup you want (then roll it forward) Jonathan Lewis © 2013 My vDB (filesystem) Your vDB (filesystem) b'' c'' ff ii i’
  56. 56. Database Virtualization
  57. 57. Three Physical Copies Three Virtual Copies Data Virtualization Appliance
  58. 58. Before Virtual Data Production Dev, QA, UAT Instance Reporting Backup Database File system Instance Instance Instance Instance Database Database File system Database File system File system Database File system Database File system “triple data tax”
  59. 59. With Virtual Data Production Instance Dev & QA Instance InInssttaannccee InInssttaannccee Database Reporting Instance Database Backup Database Instance Instance Instance Database Database Database File system Data Virtualization Appliance
  60. 60. In this presentation : • Problem in the Industry • Solution • Use Cases
  61. 61. 1. Development and QA 2. Production Support 3. Business Use Cases
  62. 62. 1. Development and QA 2. Production Support 3. Business Use Cases
  63. 63. Development : bottlenecks Frustration Waiting Old Unrepresentative Data
  64. 64. Development : subsets False Negatives False Positives Bugs in Production
  65. 65. Development : bugs
  66. 66. Development : slow env build times http://martinfowler.com/bliki/NoDBA.html
  67. 67. Development: Virtual Data • Unlimited • Full size • Self Service Development
  68. 68. Virtual Data: Easy Instance Instance Instance Instance Source DVA
  69. 69. Development Virtual Data: Parallelize gif by Steve Karam
  70. 70. Development Virtual Data: Full size
  71. 71. Development Virtual Data: Self Service
  72. 72. QA : Virtual Data • Fast • Parallel • Rollback • A/B testing
  73. 73. QA : Long Build times QA Build QA 96% of QA time was building environment $.04/$1.00 actual testing vs. setup QA Build QA X Bug 70 60 50 40 30 20 10 0 1 2 3 4 5 6 7 Delay in Fixing the bug Cost To Correct Software Engineering Economics – Barry Boehm (1981)
  74. 74. Dev QA QA Virtual Data : Fast Prod Instance DVA Time Flow • Low Resource • Find bugs Fast
  75. 75. QA with Virtual Data: Rewind Instance Development Instance Prod
  76. 76. QA with Virtual Data: A/B Instance Instance Instance Index 1 Index 2
  77. 77. Data Version Control Dev QA 2.1 Dev QA 2.2 2.1 2.2 Prod Instance DVA 10/3/2014 77
  78. 78. 1. Development and QA 2. Production Support 3. Business Use Cases
  79. 79. • Backups • Recovery • Forensics • Migration • Consolidation Recovery
  80. 80. Backups
  81. 81. Recovery Source Instance Recover VDB Instance Drop DVA
  82. 82. Forensics Instance DVA Development Instance Source
  83. 83. Development (the new production) Instance Development Instance DVA Source Development Instance
  84. 84. Migration
  85. 85. Consolidation
  86. 86. 1. Development and QA 2. Production Support 3. Business Intelligence Use Cases
  87. 87. Business Intelligence • ETL • Temporal • Confidence Testing • Federated Databases • Audits
  88. 88. Business Intelligence: ETL and Refresh Windows 1pm 10pm 8am noon
  89. 89. Business Intelligence: batch taking too long 1pm 10pm 8am noon 2011 2012 2013 2014 2015
  90. 90. 6am 8am 10pm 10pm 8am noon 9pm 1pm 10pm 8am noon 2011 2012 2013 2014 2015
  91. 91. Business Intelligence: ETL and DW Refreshes Prod Instance DW & BI Instance
  92. 92. • Collect only Changes • Refresh in minutes Prod BI and DW DVA Instance Instance Instance ETL 24x7 Virtual Data: Fast Refreshes
  93. 93. Temporal Data
  94. 94. Confidence testing
  95. 95. Modernization: Federated Source1 Instance Instance Instance Source2 Instance DVA
  96. 96. Modernization: Federated
  97. 97. Modernization: Federated “I looked like a hero” Tony Young, CIO Informatica
  98. 98. Audit Prod Instance DVA Live Archive 10/3/2014 98
  99. 99. Use Case Summary 1. Development & QA 2. Production Support 3. Business
  100. 100. How expensive is the Data Constraint? DVA at Fortune 500 : Dev throughput increase by 2x
  101. 101. How expensive is the Data Constraint? • Faster Financial Close • Faster BI refreshes • Faster surgical recovery • More Project tracks • Faster Projects
  102. 102. Virtual Data Quotes • Projects “12 months to 6 months.” – New York Life • Insurance product “about 50 days ... to about 23 days” – Presbyterian Health • “Can't imagine working without it” – State of California
  103. 103. Summary • Problem: Data is the constraint • Solution: Virtualize Data • Results: • Half the time for projects • Higher quality • Increase revenue
  104. 104. Oaktable World & hands on labs 105 We are here Oaktable World Moscone South
  105. 105. Thank you! • Kyle Hailey| Oracle ACE and Technical Evangelist, Delphix – Kyle@delphix.com – kylehailey.com – slideshare.net/khailey
  106. 106. Oracle 12c
  107. 107. 80MB buffer cache ?
  108. 108. 200GB Cache
  109. 109. 5000 Latency Tnxs / min 300 ms 1 5 10 20 30 60 100 200 with 1 5 10 20 30 60 100 200 Users
  110. 110. 8000 Latency Tnxs / min 600 ms 1 5 10 20 30 60 100 200 Users 1 5 10 20 30 60 100 200
  111. 111. Five 200GB database copies are cached with : $1,000,000 1TB cache on SAN $6,000 200GB shared cache on Delphix
  112. 112. 10/3/2014 113
  113. 113. 10/3/2014 114
  114. 114. Business Intelligence a) 24x7 Batches b) Temporal queries c) Confidence testing
  115. 115. Thin Cloning
  116. 116. SnapManager Repository Snap Manager Protection Manager Snap Manager Snap Drive Flex Clone Snap Mirror RMAN Repository Production Development DBA Storage Admin 1 tr-3761.pdf Netapp
  117. 117. 1 Netapp NetApp Filer - Production NetApp Filer - Development Database Luns Snap mirror Snapshot Manager for Oracle Flexclone Repository Database Snap Drive Protection Manage Production Development Target A InInssttaannccee Target B InInssttaanncece Target C InInssttaannccee Instance
  118. 118. Where we want to be Production Instance Instance Instance Instance Instance Database File system Development Instance Database QA Instance Database UAT Instance Database Snapshots
  119. 119. EM 12c: Snap Clone Production Development Flexclone Flexclone Netapp Snap Manager for Oracle
  120. 120. II. Data constraint price is Huge : 4. Business
  121. 121. II. Data constraint price is Huge : 4. Business 0 5 10 15 20 25 30 Revenue Dev IT Ops Storage Billion $
  122. 122. III. Data Constraint companies unaware #1 Biggest Enemy : IT departments believe – best processes – greatest technology – Just the way it is
  123. 123. Are you Innovative?
  124. 124. III. Data Constraint companies unaware Don’t we already do that ? Why do I need an iPhone ? SQL scripts Alter database begin backup Back up datafiles Redo Archive Alter database end backup RMAN
  125. 125. Merge to dev1 Dev1 Dev2 Trunk DB VC Fork Fork Fork Fork DBmaestro
  126. 126. Modernization 1. Federated 2. Migration 3. Auditing
  127. 127. What is the constraint in IT If you can’t satisfy the business demands then your process is broken.

×