Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Oow 2013 cloning_kyle_final copy

2,382 views

Published on

Published in: Technology, Spiritual
  • Be the first to comment

  • Be the first to like this

Oow 2013 cloning_kyle_final copy

  1. 1. Agile Data Platform: Revolutionizing Database Cloning Kyle Hailey http://kylehailey.com
  2. 2. • Mind-meld with the illustrious members of the OakTable • 32 talks over 2 days, right next door to Oracle OpenWorld More info. and registration: http://oaktableworld.com OakTable World: Sept 23 & 24, San Francisco NOW
  3. 3. Problem in IT Get the right data To the right people At the right time
  4. 4. Part I : Cloning Technology VirtualThin ClonePhysical Part II : Agile Data Acceleration
  5. 5. Database Cloning Challenge If you can’t satisfy the business demands then your process is broken.
  6. 6. Problem Developers QA and UAT Reports First copy Production • CERN - European Organization for Nuclear Research • 145 TB database • 75 TB growth each year • Dozens of developers want copies.
  7. 7. Tradeoff: Speed, Quality, Cost
  8. 8. What We’ve Seen 1. Inefficient QA: Higher costs of QA 2. QA Delays : Greater re-work of code 3. Sharing DB Environments : Bottlenecks 4. Using DB Subsets: More bugs in Prod 5. Slow Environment Builds: Delays “if you can't measure it you can’t manage it”
  9. 9. 1. Inefficient QA: Long Build times Build Time QA Test 96% of QA time was building environment $.04/$1.00 actual testing vs. setup Build
  10. 10. 2. QA Delays: bugs found late require more code re-work Build QA Env QA Build QA Env QA Sprint 1 Sprint 2 Sprint 3 Bug CodeX 0 10 20 30 40 50 60 70 1 2 3 4 5 6 7 Delay in Fixing the bug Cost To Correct Software Engineering Economics – Barry Boehm (1981)
  11. 11. 3. Full Copy Shared : Bottlenecks Frustration Waiting Old Unrepresentative Data
  12. 12. 4. Subsets : cause bugs
  13. 13. 4. Subsets : cause bugs Classic problem is that queries that run fast on subsets hit the wall in production. Developers are unable to test against all data The Production ‘Wall’
  14. 14. 5. Slow Environment Builds: 3-6 Months to Deliver Data Management DBA System Admin Storage Admin Developers Submit Request Disk Capacity? Approve Request $$ (2 Weeks) Approve Request $$ (1 Week) Request Additional Storage? Provision Capacity File System Configured? Configure LUNS & Build File System Coordinate Replication w/ Infrastructure Re- Parameterize & Configure DB Mount Recovery DB to Specific PIT Begin Work Approve Request $$ (2 Weeks) (3 Days) (3 Days) (2 Days) (3 Days) (3 Days) …….1-2 Weeks of Approvals, Delays, and Provisioning…… 15
  15. 15. 5. Slow Environment Builds: culture of no DBA Developer
  16. 16. Never enough environments
  17. 17. bottlenecks
  18. 18. What We’ve Seen 1. Inefficient QA: Higher costs 2. QA Delays : Increased re-work 3. Sharing DB : Bottlenecks 4. Subset DB : Bugs 5. Slow Environment Builds: Delays
  19. 19. Clone 1 Clone 3Clone 2 99% of blocks are identical
  20. 20. Clone 1 Clone 2 Clone 3 Thin Clone
  21. 21. I. Clonedb Oracle II. EMC • Copy on first write (COFW) III. Netapp • write anywhere file system (WAFL) • & EMC VNX redirect on write (ROW) IV. ZFS 2. Thin Cloning
  22. 22. RMAN backup dNFS sparse file I. clonedb
  23. 23. RMAN backup dNFS sparse file I. clonedb
  24. 24. CloneDB 1. dNFS 11.2.0.2+ – cd $ORACLE_HOME/rdbms/lib – make -f ins_rdbms.mk dnfs_on 2. Clonedb.pl initSOURCE.ora output.sql – MASTER_COPY_DIR="/rman_backup” – CLONE_FILE_CREATE_DEST="/nfs_mount” – CLONEDB_NAME="clone" 3. sqlplus / as sysdba @output.sql – startup nomount PFILE=initclone.ora – Create control file backup location – dbms_dnfs.clonedb_renamefile ('/backup/file.dbf' , '/clone/file.dbf'); – alter database open resetlogs;Tim Hall www.oracle-base.com/articles/11g/clonedb-11gr2.php
  25. 25. I. Clonedb Oracle II. EMC • Copy on first write (COFW) III. Netapp • write anywhere file system (WAFL) • & EMC VNX redirect on write (ROW) IV. ZFS 2. Thin Cloning
  26. 26. D Active File System Snapshot CBA II. EMC Copy on Write
  27. 27. D Active File System Snapshot DCBA Write penalty (read and two writes) Limit 16 snapshots No Branching (snapshots of snapshots) II. EMC Copy on Write
  28. 28. I. Clonedb Oracle II. EMC • Copy on first write (COFW) III. Netapp • write anywhere file system (WAFL) • & EMC VNX redirect on write (ROW) IV. ZFS 2. Thin Cloning
  29. 29. Data Blocks root III. Netapp and EMC VNX • 255 snapshots • Branching possible
  30. 30. I. Clonedb Oracle II. EMC • Copy on first write (COFW) III. Netapp • write anywhere file system (WAFL) • & EMC VNX redirect on write (ROW) IV. ZFS 2. Thin Cloning
  31. 31. Snapshot root Live root Zil Intent Log IV. ZFS Allocate on Write Unlimited Instantaneous Snapshots Unlimited Instantaneous Clones Branching easy and unlimited
  32. 32. FS vs. ZFS • FS per Volume • FS limited bandwidth • Storage stranded • Many FS in a pool • Grow automatically • All bandwidth Storage PoolVolume FS Volume FS Volume FS ZFS ZFS ZFS
  33. 33. I. Clonedb Oracle II. EMC III. Netapp IV. ZFS 2. Thin Cloning Database Luns Production Filer Target A Target B Target C snapshot clones 1. Put database in hot backup 2. Take Snapshot 3. Clone Snapshot (ZFS & Netapp) 4. Export Clone 5. Mount on target host InstanceInstance InstanceInstance InstanceInstance InstanceInstance Instance Source
  34. 34. Database LUNs snapshot clonesProduction Filer Development Filer Problem: How do you get data off Production? Instance Target A Target B Target C InstanceInstance InstanceInstance InstanceInstance Instance
  35. 35. Three Core Parts Production File System Instance DevelopmentStorage 21 3 Copy Sync Snapshots Purge Time Flow Clone (snapshot) Compress Share Cache Storage Agnostic Mount, recover, rename Self Service, Roles & Security Rollback & Refresh Branch & Tag Instance
  36. 36. Three Core Parts Production File System Instance DevelopmentStorage 1 Copy Sync Snapshots Purge Time Flow Instance
  37. 37. Snap Manager SnapManager Repository Protection Manager Snap Drive Snap Manager Snap Mirror Flex Clone RMAN Repository Production Development DBA Storage Admin 1 tr-3761.pdf Netapp
  38. 38. NetApp Filer - DevelopmentNetApp Filer - Production Database Luns Snap mirror Snapshot Manager for Oracle Flexclone Repository Database Snap Drive Protection Manage Production Development 1Netapp Target A Target B Target C InstanceInstance InstanceInstance InstanceInstance Instance
  39. 39. Three Core Parts Production File System Instance DevelopmentStorage Instance 3 Mount, recover, rename Self service, refresh & rollback Branch & tag Roles & security 3
  40. 40. Oracle EM 12c Snap Clone3 EM 12c Agentsinstance • Register Netapp or ZFS with Storage Credentials • Install agents on a LINUX machine to manage the Netapp or ZFS storage. • Register test master database • Enable Snap Clone for the test master database • Set up a zone – set max CPU and Memory and the roles that can see these zones • Set up a pool – a pool is a set of machines where databases can be provisioned • Set up a profile – a source database that can be used for thin cloning • Set up a service template – init.ora values Test Master Instance Source ? instance Clone Linux ZFS or NetApp PoolProfile Zone Template
  41. 41. Where we Are Production Development QA UAT Instance Instance Instance InstanceInstance Instance Instance Instance Database File systemFile system Database File systemFile system Database File system Database File systemFile systemFile systemFile system
  42. 42. Database File system Production Instance Database Development Instance Database QA Instance Database UAT Instance Snapshots Instance Instance Instance Instance Want be here
  43. 43. EM 12c: Snap Clone Production Development Flexclone Flexclone Netapp Snap Manager for Oracle
  44. 44. Thin Cloning
  45. 45. 3. Database Virtualization
  46. 46. Three Physical Copies Three Virtual Copies Data Virtualization Applaince
  47. 47. Choose your virtualization Layer: • Delphix and Oracle SMU SMU ZFS Storage Appliance
  48. 48. Oracle 12c SMU Oracle Snap Management Utility for ZFS Appliance • Requires ZFS Appliance • Supports Linux , Solaris 10+, Windows 2008+ • GUI – snapshot source databases – provision virtual databases
  49. 49. Install Delphix on x86 hardware Intel hardware
  50. 50. Allocate Any Storage to Delphix Allocate Storage Any type
  51. 51. One time backup of source database Database Production Instance File systemFile system Beta or in Development Production
  52. 52. DxFS (Delphix) Compress Data Database Production Instance Data is compressed typically 1/3 size File system
  53. 53. Incremental forever change collection Database Production Instance File system Changes • Collected incrementally forever • Old data purged File system Time Window
  54. 54. Typical Architecture Production Development QA UAT Instance Instance Instance InstanceInstance Instance Instance Instance Database File systemFile system Database File systemFile system Database File system Database File systemFile systemFile systemFile system
  55. 55. With Delphix Database Production Instance Database Development Instance Database QA Instance Database UAT InstanceInstance Instance Instance Instance File system
  56. 56. Three Core Parts Production Instance Time Window Instance Self Service Development 21 3 Source Syncing Storage (DxFS)
  57. 57. Fast, Fresh, Full Instance Time Window Instance Development VDB Source
  58. 58. Free Instance Time Window Instance Instance Instance gif by Steve Karam Source Source
  59. 59. Self Service
  60. 60. Branching Instance Instance Instance Time Window Time Window Dev1 VDB Source Source Dev VDB QA VDB (branched from Dev) End of Sprint Or a Code Freeze
  61. 61. Federated Cloning
  62. 62. Federated Instance Time Window Instance Instance Instance Time Window Source1 Source2 Source1 Source2
  63. 63. “I looked like a hero” Tony Young, CIO Informatica
  64. 64. DevOps
  65. 65. DevOps With Delphix 1. Efficient QA: Low cost, high utilization 2. Quick QA : Fast Bug Fix 3. Every Dev gets DB: Parallelized Dev 4. Full DB : Less Bugs 5. Fast Builds: Fast Dev, Culture of Yes
  66. 66. 1. Efficient QA: Lower cost B u i l d T i m e QA Test 1% of QA time was building environment $.99/$1.00 actual testing vs. setup Build Time QA Test Build
  67. 67. Rapid QA via Branching
  68. 68. 2. QA Immediate: bugs found fast and fixed Sprint 1 Sprint 2 Sprint 3 Bug CodeX QA QA Build QA Env Q A Build QA Env Q A Sprint 1 Sprint 2 Sprint 3 Bug CodeX
  69. 69. 3. Private Copies: Parallelize
  70. 70. 4. Full Size DB : Eliminate bugs
  71. 71. 5. Self Service: Fast, Efficient. Culture of Yes! Management DBA System Admin Storage Admin Developers Submit Request Disk Capacity? Approve Request $$ (2 Weeks) Approve Request $$ (1 Week) Request Additional Storage? Provision Capacity File System Configured? Configure LUNS & Build File System Coordinate Replication w/ Infrastructure Re- Parameterize & Configure DB Mount Recovery DB to Specific PIT Begin Work Approve Request $$ (2 Weeks) (3 Days) (3 Days) (2 Days) (3 Days) (3 Days) …….1-2 Weeks of Approvals, Delays, and Provisioning……
  72. 72. Quality • Forensics • A/B testing • Recovery
  73. 73. Investigate Production Bugs Instance Time Window Instance Development Anomaly on Prod Possible code bug At noon yesterday Spin up VDB of Prod as it was during anomaly
  74. 74. Rewind for patch and QA testing Instance Time Window Instance Development Time Window Prod
  75. 75. A/B testing Instance Time Window Instance Instance • Keep tests for compare • Production vs Virtual – invisible index on Prod – Creating index on virtual • Flashback vs Virtual Test A with Index 1 Test B with Index 2
  76. 76. Surgical recover of Production Instance Instance Development Time Window Spin VDB up Before drop Problem on Prod Dropped Table Accidently Source
  77. 77. Time Window Surgical or Full Recovery on VDB Instance Instance Dev1 VDB Time Window Dev1 VDB InstanceSource Source Dev2 VDB Branched
  78. 78. Virtual to Physical Instance Instance VDBSource Time Window Spin VDB up Before drop Corruption
  79. 79. Recovery
  80. 80. Business Intelligence
  81. 81. ETL and Refresh Windows 1pm 10pm 8am noon
  82. 82. ETL and DW refreshes taking longer 1pm 10pm 8am noon 2011 2012 2013 2014 2015
  83. 83. Database going Global
  84. 84. Globalization Reduces Windows 2011 2012 2013 2014 2015 1pm 10pm 8am noon 10pm 8am noon 9pm
  85. 85. 6am 8am 10pm
  86. 86. ETL and Refresh Windows 2011 2012 2013 2014 2015 1pm 10pm 8am noon 10pm 8am noon 9pm 6am 8am 10pm
  87. 87. ETL and DW Refreshes Instance Prod Instance DW & BI Data Guard – requires full refresh if used Active Data Guard – read only, most reports don’t work
  88. 88. Fast Refreshes • Collect only Changes • Refresh in minutes Instance Instance BI / DW Prod
  89. 89. Temporal Data
  90. 90. BI a) Fast refreshes a) Temporal queries b) Confidence testing
  91. 91. Review: Use Cases 1. Development Acceleration a) Full, Fresh, Fast , Self Serve b) QA Branching c) Federated 2. Quality a) Forensics b) Testing : A/B, upgrade, patch c) Recovery: logical, physical 3. BI a) Fast refresh b) Temporal Data c) Confidence testing
  92. 92. over 10 times perhaps the single largest storage consolidation opportunity history“
  93. 93. Oracle 12c
  94. 94. 80MB buffer cache ?
  95. 95. 200GB Cache
  96. 96. 5000 Tnxs/minLatency 300 ms 1 5 10 20 30 60 100 200 with 1 5 10 20 30 60 100 200 Users
  97. 97. 8000 Tnxs/minLatency 600 ms 1 5 10 20 30 60 100 200 Users 1 5 10 20 30 60 100 200
  98. 98. Database Virtualization
  99. 99. Memory Prices • EMC sells $1000/GB • X86 memory $30/1GB • TB RAM on a x86 costs around $32,000 • TB RAM on a VMAX 40K costs around $1,000,000
  100. 100. $1,000,000 $6,000
  101. 101. ERP Project Failures 2011 • NYC CityTime : delays $63 M => $760 M • Montclair Uni: delays sues PeopleSoft • Idaho : delays ERP cost millions Standish : IT Project Failure Rate 1994 1996 1998 2000 2002 2004 2009 31% 40% 28% 23% 15% 18% 24% ★http://www.galorath.com/wp/software-project-failure-costs-billions-better-estimation-planning-can- help.php*http://www.pcworld.com/article/246647/10_biggest_erp_software_failures_of_2011.html
  102. 102. Dev v2.6 v2.6v2.6 QA UAT v2.6 v2.6 v2.6v2.6 v2.7 v2.6 v2.6v2.6 v2.8 v2.6 v2.6 v2.6 v2.6 v2.6 v2.7 v2.6 v2.7 v2.6 v2.8 v2.6 v2.8
  103. 103. Dev v2.6 v2.6v2.6 QA UAT v2.6 Production v2.6 v2.6v2.6 v2.7 v2.6 v2.6v2.6 v2.8 Source Control for the database data v2.6 v2.6 v2.6 v2.6 v2.6 v2.7 v2.6 v2.7 v2.6 v2.8 v2.6 v2.8
  104. 104. Dev Prod 2.6
  105. 105. Dev QA Prod 2.6
  106. 106. Dev QA UAT Prod 2.6
  107. 107. Dev QA UAT Prod Dev QA UAT 2.6 2.7
  108. 108. Dev QA UAT Prod Dev QA UAT 2.6 2.7 Dev QA UAT 2.8
  109. 109. Dev QA UAT Prod Dev QA UAT 2.6 2.7 Dev QA UAT 2.8 Data Control = Source Control for the Database
  110. 110. Dev QA UAT Dev QA UAT 2.6 2.7 Dev QA UAT 2.8 Data Control = Source Control for the Database Production Time Flow
  111. 111. Who is Kyle Hailey  1990 Oracle – 90 support – 92 Ported v6 – 93 France – 95 Benchmarking – 98 ST Real World Performance  2000 Dot.Com  2001 Quest  2002 Oracle OEM 10g Success! First successful OEM design
  112. 112. Who is Kyle Hailey  1990 Oracle – 90 support – 92 Ported v6 – 93 France – 95 Benchmarking – 98 ST Real World Performance  2000 Dot.Com  2001 Quest a  2002 Oracle OEM 10g  2005 Embarcadero – DB Optimizer
  113. 113. Who is Kyle Hailey • 1990 Oracle  90 support  92 Ported v6  93 France  95 Benchmarking  98 ST Real World Performance • 2000 Dot.Com • 2001 Quest • 2002 Oracle OEM 10g • 2005 Embarcadero  DB Optimizer • 2010 Delphix When not being a Geek - Have a little 4 year old boy who takes up all my time NoCOUG board IOUG Liaison
  114. 114. About Delphix • Founded in 2008, launched in 2010 • CEO Jedidiah Yueh (founder of Avamar: >$1B revenue)) • Based in Silicon Valley, Global Operations • 10% of Fortune 500
  115. 115. $40M $75M $850M $27,000M Storage IT Develop Business
  116. 116. Good, Cheap, Fast : choose two Fast Good Cheap
  117. 117. DevOps With Delphix 1. Efficient QA: Low cost, high utilization 2. Quick QA : Fast Bug Fix 3. Every Dev gets DB: Parallelized Dev 4. Full DB : Less Bugs 5. Fast Builds: Fast Dev, Culture of Yes

×