Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Kscope 2013 delphix

1,232 views

Published on

Delphix Database Virtualization

Published in: Technology, Spiritual
  • Be the first to comment

Kscope 2013 delphix

  1. 1. Database VirtualizationandInstant CloningKyle Haileyhttp://dboptimizer.com
  2. 2. Database Cloning ChallengeBusiness want data now.Business don’t understand DBAs.Databases getting bigger & harder to copy.Developers want more copies.Reporting wants more copies.Everyone has storage constraints.If you can’t satisfy the businessdemands your process is broken.
  3. 3. Two PartsI. Cloning TechnologyII. Accelerate your business
  4. 4. Part I : Cloning Technology3. Virtual2. Thin Provision1. Physical
  5. 5. =database1. Physical Cloning
  6. 6. ProblemDevelopersQA and UATReportsFirstcopyProduction• CERN - European Organization for NuclearResearch• 145 TB database• 75 TB growth each year• Dozens of developers want copies.
  7. 7. workaroundsDevelopersQA and UATReportsSharedSub set copyProductionMany copies
  8. 8. Physical ClonesDatabase SubsetsShared Databases
  9. 9. Subsets
  10. 10. The Production‘Wall’Classic problem is that queries thatrun fast on subsets hit the wall inproduction.Developers are unable to test againstall data
  11. 11. Shared Full
  12. 12. Shared access = Poor ProductivityDevelopers andtester get frustratedDatabases become oldand unrepresentativeof production.Requires complexscheduling andmanagement
  13. 13. Never enough environmentsAverage customer makes 12 copies of production- Charles Garry, Database Product Manager Oracle
  14. 14. Physical CopiesTime consumingTime to make copies, days to weeksRMAN backup, archive logs, copy data over, recoverMeetings , days to weeksAdmins: System, Storage ,Database ,Network,manager coordinationSpace consuming40 devs x 2.5TB production = 100TB20 report DBs x 40 TB = 800TB=> bottlenecks
  15. 15. Setup Develop
  16. 16. Setup
  17. 17. Setup DevelopQA
  18. 18. $40M$75M$850M$27,000MStorageITDevelopBusiness
  19. 19. ERP Project Failures 2011• NYC CityTime : delays $63 M => $760 M• Montclair Uni: delays sues PeopleSoft• Idaho : delays ERP cost millionsStandish : IT Project Failure Rate1994 1996 1998 2000 2002 2004 200931% 40% 28% 23% 15% 18% 24%★http://www.galorath.com/wp/software-project-failure-costs-billions-better-estimation-planning-can-help.php*http://www.pcworld.com/article/246647/10_biggest_erp_software_failures_of_2011.html
  20. 20. Clone 1 Clone 399% of blocks are IdenticalClone 2
  21. 21. 2. Thin Provision Cloning
  22. 22. Clone 1 Clone 2 Clone 3Thin Provision
  23. 23. 2. Thin Provision CloningCornerstone Technology:File System Snapshots
  24. 24. Thin Cloning• Snapshot DB Files @ point in time• Copy off production• Snapshot again• Export over NFS or FC to Host• Recovery Database
  25. 25. Netapptr-3761.pdf
  26. 26. NetappSnapManagerSnapManagerRepositoryProtectionManagerSnap DriveSnapManagerSnap MirrorFlex CloneRMANRepositoryProductionDevelopmentDBAStorageAdmin
  27. 27. NetApp Filer - DevelopmentNetApp Filer - ProductionProduction DatabaseDatabaseLunsTarget ATarget BTarget CClone 1Clone 2Clone 3Clone 4Snap mirrorSnapshot Managerfor OracleFlexcloneRepositoryDatabaseNetappSnapDriveProtectionManage
  28. 28. 2. Thin Provision Cloning
  29. 29. 2. Thin Provision Cloning
  30. 30. 3. Database Virtualization
  31. 31. Virtualization Layer38Virtualization
  32. 32. DatabaseVirtualizationAppliance(DVA)3 Clones Physical 3 Clones Virtual
  33. 33. Install Delphix on x86 hardwarex86 hardware
  34. 34. Allocate Storage to DelphixAllocateStorageAny type
  35. 35. One time backup of source databaseDatabaseProductionInstanceFile systemRMAN APIs
  36. 36. Delphix Compress DataDatabaseProductionInstanceFile systemData iscompressedtypically 1/3size
  37. 37. Incremental forever change collectionDatabaseProductionInstanceFile systemChanges are collectedautomatically foreverData older than retentionwidow freed
  38. 38. Typical ArchitectureDatabaseFile systemProductionInstanceDatabaseFile systemDevelopmentInstanceDatabaseFile systemQAInstanceDatabaseUATInstanceFile system
  39. 39. Clones share duplicate blocksDevelopmentInstanceDatabaseProductionInstanceFile systemvDatabaseQAInstanceUATInstancevDatabase vDatabaseSource Database Clone Copies of Source Database
  40. 40. Use Cases1. Development2. Recovery3. Reporting
  41. 41. 1. Development Acceleration
  42. 42. 1: Development Accelerationa) Developer each get a copy– Fast, fresh, full, frequent– Self serviceb) Branchingc) Federated
  43. 43. SourceFastSource DatabaseTarget HostVirtual DatabaseNFSFiberFiberRMAN overTCPNo Data Movement
  44. 44. SourceFreshVirtual DatabaseFiber
  45. 45. SourceFrequentVirtual DatabaseVirtual DatabaseTarget HostsVirtual DatabaseVirtual DatabaseFiber
  46. 46. Full clones
  47. 47. Self Service
  48. 48. 1 b) Branching and Rapid QA
  49. 49. dSource1 b) BranchingDeveloper VDBQA VDB
  50. 50. Devv2.6 v2.6v2.6QA UATv2.6v2.6 v2.6v2.6v2.7v2.6 v2.6v2.6v2.8v2.6v2.6v2.6v2.6v2.6v2.7v2.6v2.7v2.6v2.8v2.6v2.8
  51. 51. Devv2.6 v2.6v2.6QA UATv2.6Productionv2.6 v2.6v2.6v2.7v2.6 v2.6v2.6v2.8Source Control for the database datav2.6v2.6v2.6v2.6v2.6v2.7v2.6v2.7v2.6v2.8v2.6v2.8
  52. 52. DevProd2.6
  53. 53. DevQAProd2.6
  54. 54. DevQAUATProd2.6
  55. 55. DevQAUATProdDevQAUAT2.62.7
  56. 56. DevQAUATProdDevQAUAT2.62.7DevQAUAT2.8
  57. 57. DevQAUATProdDevQAUAT2.62.7DevQAUAT2.8Data Control = Source Control for the Database
  58. 58. DevQAUATDevQAUAT2.62.7DevQAUAT2.8Data Control = Source Control for the DatabaseProduction Time Flow
  59. 59. 1 c) Federated Cloning
  60. 60. Source2Source3Source11 c) Federated sourcesVirtual DatabaseVirtual DatabaseVirtual DatabaseVirtual Database
  61. 61. “I looked like a hero”Tony Young, CIO Informatica
  62. 62. 1. Review Development Use Casesa) Developer each get a copya) QAb) Federated
  63. 63. 2. Recovery, Testing, Forensicsa) Forensicsb) A/B testingc) Recovery
  64. 64. Source2 a) Forensic AnalysisVirtual Database
  65. 65. Source2 b) Upgrades, Patches, RAT, A/BVirtual Database
  66. 66. • Production vs Virtual– invisible index on Prod– Creating index on virtual• Flashback vs Virtual• Keep tests for compare2 b) Upgrades, Patches, RAT, A/B
  67. 67. 2 c) Recovery
  68. 68. Source2 c) Logical Recovery ProductionVirtual Database
  69. 69. Source2 c) Logical Recovery DevelopmentVirtual DatabaseVDB rolled back
  70. 70. SourceRecoveryVDBV2P
  71. 71. 2. Recovery, Testing, Forensicsa) Forensicsb) A/B testingc) Recovery : Logical and physical
  72. 72. 3: reportinga) Fast refreshesb) Temporal queriesc) Confidence testing
  73. 73. Fast Refreshes• Refresh in minutes• Without data movement• Faster , cheaper
  74. 74. Temporal Data
  75. 75. 3: reportinga) Fast refreshesa) Temporal queriesb) Confidence testing
  76. 76. Review: Use Cases1. Developmenta) Full, Fresh, Fast , Self Serveb) Branchingc) Federated2. Recovery, Testing :a) Forensicsb) Testing : A/B, upgrade, patchc) Recovery: logical, physical3. Reportinga) Fast refreshb) Temporal Datac) Confidence testing
  77. 77. over 10 times"perhaps the single largest storage consolidationopportunity history“
  78. 78. Oracle 12c
  79. 79. 80MB buffer cache ?
  80. 80. 200GBCache
  81. 81. 5000Tnxs/minLatency300ms1 5 10 20 30 60 100 200with1 5 10 20 30 60 100 200Users
  82. 82. 8000Tnxs/minLatency600ms1 5 10 20 30 60 100 200Users1 5 10 20 30 60 100 200
  83. 83. Database Virtualization

×