UKOUG, Instant Cloning: Boosting Application Development

527 views

Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

UKOUG, Instant Cloning: Boosting Application Development

  1. 1. Instant Cloning: Boosting Application Development Kyle Hailey kyle@delphix.com http://kylehailey.com
  2. 2. Problem in IT CIO Magazine Survey: 60% Projects Over Schedule 85% delayed waiting for data Data is the problem Get the right data To the right people At the right time
  3. 3. The Phoenix Project • • • • • IT bottlenecks Setting Priorities Company Goals Defining Metrics Fast Iterations IT version of “The Goal” by E. Goldratt One of the most powerful things that IT can do is get environments to development and QA when they need it
  4. 4. Data Flood Sandbox QA Development Productio n UAT Business Intelligence Analytics Reporting Backup Data Guard Tape Forensics
  5. 5. Part I : Problem: Copying Databases Part II : Solutions: Thin cloning to Databases Virtualization Part III : Project Acceleration with Virtualization
  6. 6. Database Cloning Challenge If you can’t satisfy the business demands then your process is broken.
  7. 7. Problem Reports Production First copy QA and UAT • CERN - European Organization for Nuclear Research • 145 TB database • 75 TB growth each year • Dozens of developers want copies. Developers
  8. 8. Tradeoff: Speed, Quality, Cost
  9. 9. What We’ve Seen 1. 2. 3. 4. 5. Inefficient QA: Higher costs of QA QA Delays : Greater re-work of code Sharing DB Environments : Bottlenecks Using DB Subsets: More bugs in Prod Slow Environment Builds: Delays “if you can't measure it you can’t manage it”
  10. 10. 1. Inefficient QA: Long Build times Build QA Test Build Time 96% of QA time was building environment $.04/$1.00 actual testing vs. setup
  11. 11. 2. QA Delays: bugs found late require more code re-work Build QA Env Sprint 3 Sprint 2 Sprint 1 X Build QA Env QA Bug Code 70 60 50 40 30 20 10 0 Cost To Correct 1 2 3 4 5 6 7 Delay in Fixing the bug Software Engineering Economics – Barry Boehm (1981) QA
  12. 12. 3. Full Copy Shared : Bottlenecks Old Unrepresentative Data Frustration Waiting
  13. 13. http://martinfowler.com/bliki/NoDBA.html
  14. 14. 4. Subsets : cause bugs
  15. 15. 4. Subsets : cause bugs The Production ‘Wall’ Classic problem is that queries that run fast on subsets hit the wall in production. Developers are unable to test against all data
  16. 16. 5. Slow Environment Builds: 3-6 Months to Deliver Data Developers Management Submit Request Approve Request $$ (2 Weeks) Approve Request $$ (2 Weeks) Approve Request $$ (1 Week) (2 Days) DBA System Admin (3 Days) (3 Days) Disk Capacity? Storage Admin (3 Days) Begin Work …….1-2 Weeks of Approvals, Delays, and Provisioning…… Request Additional Storage? File System Configured? Coordinate Replication w/ Infrastructure Configure LUNS & Build File System ReParameterize & Configure DB Mount Recovery DB to Specific PIT (3 Days) Provision Capacity 16
  17. 17. 5. Slow Environment Builds: culture of no DBA Developer
  18. 18. Never enough environments
  19. 19. bottlenecks
  20. 20. What We’ve Seen 1. 2. 3. 4. 5. Inefficient QA: Higher costs QA Delays : Increased re-work Sharing DB : Bottlenecks Subset DB : Bugs Slow Environment Builds: Delays
  21. 21. 99% of blocks are identical Clone 1 Clone 2 Clone 3
  22. 22. Thin Clone Clone 1 Clone 2 Clone 3
  23. 23. 2. Thin Cloning I. EMC II. Netapp III. ZFS 1. 2. 3. 4. 5. Put database in hot backup Take Snapshot (limits EMC 16, Netapp ,VNX 255, ZFS unlimted) Take database out of hot backup Clone Snapshot (Netapp , ZFS, EMC VNX) Export Clone, Mount on target host Target A Production Filer clones Source snapshot Instance Instance Target B Instance Instance Database Luns Instance Instance Target C Instance Instance Instance
  24. 24. Problem: How do you get data off Production? Target A Instance Instance Development Filer Production Filer clones snapshot Instance Target B Database LUNs Instance Instance Target C Instance Instance Instance
  25. 25. Three Core Parts Storage Production Instance File System 2 1 Copy & sync Netapp Snapshot & Clone Development Instance 3 Mount, recover, rename EM snap clone
  26. 26. Three Core Parts Production Instance 1 Copy & sync Storage File System Development Instance
  27. 27. Netapp 1 SnapManager Repository DBA Snap Manager Protection Manager tr-3761.pdf RMAN Repository Snap Manager Flex Clone Production Snap Drive Snap Mirror Storage Admin Development
  28. 28. Netapp 1 Target A NetApp Filer - Production Production NetApp Filer - Development Snap Protection mirror Manage Snap Drive Flexclone Instance Instance Target B Database Luns Instance Instance Snapshot Manager for Oracle Repository Database Development Target C Instance Instance Instance
  29. 29. Three Core Parts3 Production Instance Storage File System Development Instance 3 Mount, recover, rename
  30. 30. 3 Oracle EM 12c Snap Clone EM 12c Test Master Source Instance ? instance Profile • • • • • • • • Register Netapp or ZFS with Storage Credentials Install agents on a LINUX machine to manage the Netapp or ZFS storage. Register test master database Enable Snap Clone for the test master database Set up a zone – set max CPU and Memory and the roles that can see these zones Set up a pool – a pool is a set of machines where databases can be provisioned Set up a profile – a source database that can be used for thin cloning Set up a service template – init.ora values Linux Clone instance Agents Pool Template Zone ZFS or NetApp
  31. 31. Where we are Production Instance Instance Database File system File system QA UAT Instance Instance Instance Instance Instance Instance Database Database Database File system File system File system File system File system File system File system Development
  32. 32. Where we want to be Production Development Instance Instance Database Instance Instance Database QA UAT Instance Instance Instance Instance Database File system Snapshots Database
  33. 33. EM 12c: Snap Clone Production Flexclone Development Flexclone Netapp Snap Manager for Oracle
  34. 34. Thin Cloning
  35. 35. 3. Database Virtualization
  36. 36. Three Physical Copies Three Virtual Copies Data Virtualization Appliance
  37. 37. Install Delphix on x86 hardware Intel hardware
  38. 38. Allocate Any Storage to Delphix Allocate Storage Any type Pure Storage + Delphix Better Performance for 1/10 the cost
  39. 39. One time backup of source database Production Supports Instance Instance Instance Database File system Upcoming
  40. 40. DxFS (Delphix) Compress Data Production Instance Instance Instance Database File system Data is compressed typically 1/3 size
  41. 41. Incremental forever change collection Production Instance Instance Instance Database Changes Time Window File system • Collected incrementally forever • Old data purged
  42. 42. Source Full Copy Source backup from SCN 1
  43. 43. Snapshot 1 Snapshot 2
  44. 44. Snapshot 1 Snapshot 2 Backup from SCN
  45. 45. Snapshot 1 Snapshot 3 Snapshot 2
  46. 46. Snapshot 3 Snapshot 2 Drop Snapshot 1
  47. 47. Cloning Production Instance Instance Instance Instance Instance Database Database Time Window File system
  48. 48. Typical Architecture Production Instance Instance Database File system File system QA UAT Instance Instance Instance Instance Instance Instance Database Database Database File system File system File system File system File system File system File system Development
  49. 49. With Delphix Production Instance Instance Database File system Development QA UAT Instance Instance Instance Instance Instance Instance Database Database Database
  50. 50. Three Core Parts Production Instance 1 Copy Sync Snapshots Time Flow Purge Storage File System 2 Clone (snapshot) Compress Share Cache Storage Agnostic Development Instance 3 Mount, recover, rename Self Service, Roles & Security Rollback & Refresh Branch & Tag
  51. 51. Overview of Database Virtualization 1. 2. 3. 4. 5. Fast, Fresh, Full Free Branching Federated Self Serve
  52. 52. Fast, Fresh, Full Source Development VDB Instance Instance Time Window
  53. 53. Free Instance Source Instance Instance Instance gif by Steve Karam
  54. 54. Branching Source Instance Dev Instance Source QA branched from Dev Instance
  55. 55. Federated Cloning
  56. 56. Federated Source1 Instance Source1 Instance Source2 Instance Instance
  57. 57. “I looked like a hero” Tony Young, CIO Informatica
  58. 58. Self Service
  59. 59. Use Cases 1. Development Acceleration 2. Quality 3. BI
  60. 60. DevOps
  61. 61. DevOps With Delphix 1. 2. 3. 4. 5. Efficient QA: Low cost, high utilization Quick QA : Fast Bug Fix Every Dev gets DB: Parallelized Dev Full DB : Less Bugs Fast Builds: Culture of Yes
  62. 62. 1. Efficient QA: Lower cost Build QA Test Build Time B u i l d T i m e QA Test 1% of QA time was building environment $.99/$1.00 actual testing vs. setup
  63. 63. Rapid QA via Branching
  64. 64. 2. QA Immediate: bugs found fast and fixed Build QA Env Sprint 2 Sprint 1 X Q A Build QA Env Q A Sprint 3 Bug Code QA QA Sprint 2 Sprint 1 X Bug Code Sprint 3
  65. 65. 3. Private Copies: Parallelize gif by Steve Karam
  66. 66. 4. Full Size DB : Eliminate bugs
  67. 67. 5. Self Service: Fast, Efficient. Culture of Yes! Developers Management Submit Request Approve Request $$ (2 Weeks) Approve Request $$ (2 Weeks) Approve Request $$ (1 Week) (2 Days) DBA System Admin (3 Days) (3 Days) Disk Capacity? Storage Admin (3 Days) Begin Work …….1-2 Weeks of Approvals, Delays, and Provisioning…… Request Additional Storage? File System Configured? Provision Capacity Coordinate Replication w/ Infrastructure Configure LUNS & Build File System ReParameterize & Configure DB (3 Days) Mount Recovery DB to Specific PIT
  68. 68. Quality 1. Forensics 2. Testing 3. Recovery
  69. 69. 1. Forensics: Investigate Production Bugs Development Instance Instance Time Window Anomaly on Prod Possible code bug At noon yesterday Spin up VDB of Prod as it was during anomaly
  70. 70. 2. Testing : Rewind for patch and QA testing Prod Development Instance Instance Time Window Time Window
  71. 71. 2. Testing: A/B Instance Test A with Index 1 Instance Instance Time Window • Keep tests for compare • Production vs Virtual – invisible index on Prod – Creating index on virtual • Flashback vs Virtual Test B with Index 2
  72. 72. 3. Recovery: Surgical recover of Production Source Development Instance Instance Spin VDB up Before drop Time Window Problem on Prod Dropped Table Accidently
  73. 73. 3. Recovery Surgical or Full Recovery on VDB Dev1 VDB Source Instance Instance Dev2 VDB Branched Source Time Window Dev1 VDB Time Window Instance
  74. 74. 3. Recovery: Virtual to Physical Source VDB Instance Instance Spin VDB up Before drop Time Window Corruption
  75. 75. 3. Recovery
  76. 76. Business Intelligence
  77. 77. ETL and Refresh Windows 1pm noon 10pm 8am
  78. 78. ETL and DW refreshes taking longer 1pm noon 10pm 2011 2012 2013 2014 2015 8am
  79. 79. ETL and Refresh Windows 6am 8am 10pm 10pm 1pm noon 8am 10pm 2011 2012 2013 2014 2015 noon 9pm 8am
  80. 80. ETL and DW Refreshes Prod DW & BI Instance Instance Data Guard – requires full refresh if used Active Data Guard – read only, most reports don’t work
  81. 81. Fast Refreshes • Collect only Changes • Refresh in minutes Prod Instance BI DW Instance Instance ETL 24x7
  82. 82. Temporal Data
  83. 83. BI a) Fast refreshes a) Temporal queries b) Confidence testing
  84. 84. Review: Use Cases 1. Development Acceleration a) b) c) 2. Quality a) b) c) 3. Full, Fresh, Fast , Self Serve QA Branching Federated Forensics Testing : A/B, upgrade, patch Recovery: logical, physical BI a) b) c) Fast refresh Temporal Data Confidence testing
  85. 85. Oracle 12c
  86. 86. 80MB buffer cache ?
  87. 87. 200GB Cache
  88. 88. with Latency Tnxs / min 5000 300 ms 1 5 10 20 30 60 100 200 Users 1 5 10 20 30 60 100 200
  89. 89. Latency Tnxs / min 8000 600 ms 1 5 10 20 30 60 100 200 Users 1 5 10 20 30 60 100 200
  90. 90. $1,000,000 $6,000
  91. 91. About Delphix • Founded in 2008, launched in 2010 • CEO Jedidiah Yueh (founder of Avamar: >$1B revenue)) • Based in Silicon Valley, Global Operations Fortune #1 Walmart #1 social: Facebook #1 US Bank: Wells Fargo #1 Networking : Cisco #1 cable provider: Comcast #1 auction site: Ebay #1 insurance: New York Life #1 chip manufacture: Intel
  92. 92. Dev v2.6 v2.6 QA v2.6 v2.6 UAT v2.6 v2.6 v2.6 v2.7 v2.6 v2.7 v2.7 v2.6 v2.6 v2.6 v2.8 v2.8 v2.6 v2.8
  93. 93. Production Dev v2.6 v2.6 QA v2.6 v2.6 UAT v2.6 v2.6 v2.6 v2.7 v2.6 v2.7 v2.7 v2.6 v2.6 v2.6 v2.8 v2.8 v2.6 v2.8 Source Control for the database data
  94. 94. 2.6 Dev Prod
  95. 95. 2.6 QA Dev Prod
  96. 96. 2.6 UAT QA Dev Prod
  97. 97. 2.6 UAT QA Dev Prod Dev QA UAT 2.7
  98. 98. 2.6 2.8 UAT QA UAT QA Dev Dev Prod Dev QA UAT 2.7
  99. 99. 2.6 2.8 UAT QA UAT QA Dev Dev Prod Dev QA UAT 2.7 Data Control = Source Control for the Database
  100. 100. 2.6 2.8 UAT QA UAT QA Dev Dev Production Time Flow Dev QA UAT 2.7 Data Control = Source Control for the Database
  101. 101. Database Virtualization
  102. 102. Business Develop IT Storage $27,000M $850M $75M $40M
  103. 103. Good, Cheap, Fast : choose two Good Cheap Fast
  104. 104. FS vs. ZFS • • • FS per Volume FS limited bandwidth Storage stranded FS FS FS Volume Volume Volume • • • Many FS in a pool Grow automatically All bandwidth ZFS ZFS Storage Pool ZFS
  105. 105. Increased Revenue For Company # of database copies
  106. 106. Three Core Parts 1 2 Production 3 Development Instance Instance Time Window Source Syncing Storage (DxFS)Self Service
  107. 107. Database going Global
  108. 108. Globalization Reduces Windows 10pm 1pm noon 8am 10pm 2011 2012 2013 2014 2015 noon 9pm 8am
  109. 109. 6am 8am 10pm
  110. 110. 2. Thin Cloning I. Clonedb Oracle II. EMC • Copy on first write (COFW) III. Netapp • write anywhere file system (WAFL) • & EMC VNX redirect on write (ROW) IV. ZFS
  111. 111. I. clonedb dNFS sparse file RMAN backup
  112. 112. I. clonedb dNFS sparse file RMAN backup
  113. 113. CloneDB 1. dNFS 11.2.0.2+ – cd $ORACLE_HOME/rdbms/lib – make -f ins_rdbms.mk dnfs_on 2. Clonedb.pl initSOURCE.ora output.sql – MASTER_COPY_DIR="/rman_backup” – CLONE_FILE_CREATE_DEST="/nfs_mount” – CLONEDB_NAME="clone" 3. sqlplus / as sysdba @output.sql – startup nomount PFILE=initclone.ora – Create control file backup location – dbms_dnfs.clonedb_renamefile ('/backup/file.dbf' , '/clone/file.dbf'); – alter database open resetlogs; Hall Tim www.oracle-base.com/articles/11g/clonedb-11gr2.php
  114. 114. Thin Cloning I. Clonedb Oracle II. EMC • Copy on first write (COFW) III. Netapp • write anywhere file system (WAFL) • & EMC VNX redirect on write (ROW) IV. ZFS
  115. 115. II. EMC Copy on Write Active File System Snapshot A B C D
  116. 116. II. EMC Copy on Write Active File System Snapshot A B C D Write penalty (read and two writes) Limit 16 snapshots No Branching (snapshots of snapshots) D
  117. 117. I. Clonedb Oracle II. EMC • Copy on first write (COFW) III. Netapp • write anywhere file system (WAFL) • & EMC VNX redirect on write (ROW) IV. ZFS
  118. 118. III. Netapp (WAFL) and EMC (ROW) root Data Blocks • 255 snapshots • Branching possible
  119. 119. I. Clonedb Oracle II. EMC • Copy on first write (COFW) III. Netapp • write anywhere file system (WAFL) • EMC redirect on write (ROW) IV. ZFS
  120. 120. IV. ZFS Allocate on Write Snapshot root Live root Zil Intent Log Unlimited Instantaneous Snapshots Unlimited Instantaneous Clones Branching easy and unlimited
  121. 121. 3 Oracle EM 12c Snap Clone Create a “Test Master” database On ZFSSA or Netapp Test Master Source Instance ? instance ZFSSA or Netapp
  122. 122. 3 Oracle EM 12c Snap Clone Install EM 12c Install Snap Clone option Test Master Source Instance EM 12c ? • Register Netapp or ZFS with Storage Credentials instance ZFS or NetApp
  123. 123. 3 Oracle EM 12c Snap Clone EM 12c Test Master Source Instance ? instance • Register Netapp or ZFS with Storage Credentials • Install agents on a LINUX machine to manage the Netapp or ZFS storage. Linux Agents ZFS or NetApp
  124. 124. 3 Oracle EM 12c Snap Clone EM 12c Test Master Source Instance • • • • ? instance Register Netapp or ZFS with Storage Credentials Install agents on a LINUX machine to manage the Netapp or ZFS storage Register test master database . Linux Agents ZFS or NetApp
  125. 125. 3 Oracle EM 12c Snap Clone Enable Snap Clone EM 12c Test Master Source Instance • • • • ? instance Register Netapp or ZFS with Storage Credentials Install agents on a LINUX machine to manage the Netapp or ZFS storage Register test master database Enable Snap Clone for the test master database • . Linux Agents ZFS or NetApp
  126. 126. 3 Oracle EM 12c Snap Clone EM 12c Test Master Source Instance • • • • • ? instance Register Netapp or ZFS with Storage Credentials Install agents on a LINUX machine to manage the Netapp or ZFS storage. Register test master database Enable Snap Clone for the test master database Set up a zone – set max CPU and Memory and the roles that can see these zones Linux CPU limits Memory limits Agents Zone ZFS or NetApp
  127. 127. 3 Oracle EM 12c Snap Clone EM 12c Test Master Source Instance ? instance Linux Clone instance Agents Pool • • • • • • Register Netapp or ZFS with Storage Credentials Install agents on a LINUX machine to manage the Netapp or ZFS storage. Register test master database Enable Snap Clone for the test master database Set up a zone – set max CPU and Memory and the roles that can see these zones Set up a pool – a pool is a set of machines where databases can be provisioned Zone ZFS or NetApp
  128. 128. 3 Oracle EM 12c Snap Clone EM 12c Test Master Source Instance ? instance Profile • • • • • • • Register Netapp or ZFS with Storage Credentials Install agents on a LINUX machine to manage the Netapp or ZFS storage. Register test master database Enable Snap Clone for the test master database Set up a zone – set max CPU and Memory and the roles that can see these zones Set up a pool – a pool is a set of machines where databases can be provisioned Set up a profile – a source database that can be used for thin cloning Linux Clone instance Agents Pool Zone ZFS or NetApp
  129. 129. 3 Oracle EM 12c Snap Clone EM 12c Test Master Source Instance ? instance Profile • • • • • • • • Register Netapp or ZFS with Storage Credentials Install agents on a LINUX machine to manage the Netapp or ZFS storage. Register test master database Enable Snap Clone for the test master database Set up a zone – set max CPU and Memory and the roles that can see these zones Set up a pool – a pool is a set of machines where databases can be provisioned Set up a profile – a source database that can be used for thin cloning Set up a service template – init.ora values Linux Clone instance Agents Pool Template Zone ZFS or NetApp
  130. 130. SMU ZFS Storage Appliance Choose your virtualization Layer: • Delphix and Oracle SMU
  131. 131. Oracle 12c SMU Oracle Snap Management Utility for ZFS Appliance • Requires ZFS Appliance • Supports Linux , Solaris 10+, Windows 2008+ • GUI – snapshot source databases – provision virtual databases
  132. 132. perhaps the single largest storage consolidation opportunity history“ over 10 times
  133. 133. Original Snapshot Snapshot Day 2

×