Kscope 2013 delphix


Published on

Delphix Database Virtualization

Published in: Technology, Spiritual
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Kyle HaileyWork for a company called DelphixWe write software that enables companies toCopy their databases in 2 minutes with almost no storage overheadWe accomplish that by taking one initial copy and sharing the duplicate blocks Across all the clones
  • want data now.don’t understand DBAs.Db bigger and harder to copy.Devswant more copies.Reporting wants more copies.Everyone has storage constraints.If you can’t satisfy the business demands your process is broken
  • What are these technologiesbenefits and drawbacksTechnology is awesomeComing of ageClonedb 3 pres @ OOW SMU OEM 12c DBaaS 12c “clone” pluggable databases”
  • Databases are largeMoving data around is hard work.Moving them takestime, resources, equipment and experienceI spent ½ my time cloning. How much do you?How many of you copy databases ? How much time does you spend on it?----1) 10 years in support - No backups, hospital companies still don’t back up dev often dev is the new prod2) Full time DBA, half time copying3) OEM wanted me to do full phys cloning4) Was a consultant always wanted database to play on
  • Prod critical for businessPerformance of prod is top priorityProtect prod from load
  • 2 options to create enough copies
  • Xxx spends 50% of time copying databases have to subset because not enough storagesubseting process constantly needs fixing modificationWhat happens when developers use subsets -- ****** -----
  • Stubhub (ebay) estimates that 20% of there production bugs arise from testing onSubsets instead of full database copies.
  • Wait orebay till next slide
  • Example at Ebay2 dozen developers have a massive shared copy of productionExample at DB3 development teams agree between themselves who is doing what testing this week as some runs destroy other teams data,If the database is shared it’s hard to get opportunity to refresh and a data get’s old
  • Only having enough equipment to support 2 or 3 environments causes massive delaysState of Colorado has a 100 projects support 3KLA tencor can only support 2 projects of a dozen
  • DB had databases which were not refreshed for 6 months+ due to refresh time and size
  • Slow downs mean bottlenecksThese bottlenecks cause failures in IT projectsI’m into eliminating bottlenecks (whether it is wait events, tuning sql or provisioning copies of dbs)
  • Development asks for a database it takes days or weeks.
  • 90% of lost developer days at customer was due to waiting for environment builds
  • Happens both for dev and QA
  • Tightening constraining resourcesCascading affect on companies.The business doesn’t know or understand this DBA workDBAs are often the hardest resource for IT to justify because they are invisibleDBAs are already being asked to do a tremendous amountDBAs are often on call 24x7DBAs are foundational.
  • http://mobile.stufffundieslike.com/2010/02/upping-the-ante-on-doctrinal-disagreements/
  • Delays cause failures*http://www.pcworld.com/article/246647/10_biggest_erp_software_failures_of_2011.html
  • MisguidedattributingRelax the constraints http://martinfowler.com/bliki/NoDBA.html
  • Fastest query is the query not run
  • Creating a thin clone on one lun easyBut how do you get it off production filer?How do you bring in new changes from the sourceHow do you purge old changes
  • Ask a customerHow long does it take to thin clone a database on Netapp?2-5 days!?2-4 hours if DBA, sys admin, storage admin were in the same room
  • Most implemented thin cloning technology I see is Netapp
  • http://partners.netapp.com/go/techontap/empower-dba.html?fmt=printCreate Luns, aggr, snapshots, clonesMirroring filesystemsExporting file systemsMounting file systems
  • Technology has existed 15+ years Why hasn’t there been more adoption ??
  • Requires expert storage admins specialized equipment scripting2-7 Days or 2-8 hours if everyone togetherCERN recently gave a presentation where they wrote almost 30,000 lines of code13k lines & 15k lines of PHP
  • Like the internetInternet existed before browserftp, bulliten boards, chat rooms, gopher, telnet etcDidn’t take off until the browserThin cloning didn’t take off until database virtualizaiton
  • Like the internet
  • Delphix GUI is what Oracle Enterprise Manager would look like if Apple had designed itI like at DelphixFrustratedSteve and Larry gave aweseom presentationsSteve jobs and Ellison ui combined forces now I have it
  • Database virtualization is to the data tier whatVMware is to the compute tier. On the compute tier VMware allows the same hardware to be shared by multiple machines. On the data tier virtualization allows the same datafiles to be shared by multiple clones allowing almost instantaneous creation of new copies of databases with almost no disk footprint.  
  • In physical worldIf 3 Copies of a database
  • Software installs an any x86 hardware uses any storage supports any Oracle OS
  • Fast = Fresh Full = Quality Many = jet pack on development-
  • Self Service
  • Source Control for the database data
  • Source Control for the database data
  • Physically independent but logically correlatedCloning multiple source databases at the same time can be a daunting task
  • One example with our customers is InformaticaWho had a project to integrate 6 databases into one central databaseThe time of the project was estimated at 12 monthsWith much of that coming from trying to orchestratingGetting copies of the 6 databases at the same point in timeLike herding cats
  • Informatical had a 12 month project to integrate 6 databases.After installing Delphix they did it in 6 months.I delivered this earlyI generated more revenueI freed up money and put it into innovationwon an award with Ventana Research for this project
  • Production vs VirtualCreating invisible index on ProdCreating index on virtualFlashback vs Virtual
  • Multiple scripted dumps or RMAN backups are used to move data today. With application awareness, we only request change blocks—dramatically reducing production loads by as much as 80%. We also eliminate the need for DBAs to manage custom scripts, which are expensive to maintain and support over time.
  •      Com - DBA dropped movie titles table, 8 hour to restore a backup     PG - operator entered  Euro instead of US $      Fid - Oracle bug caused logical corruption on Dataguard, wouldn't start,
  •    HD - developer    truncated 232 Million row table, wanted it back
  • Developer each get a copyFast, fresh, full, frequentSelf serviceQA branch from DevelopmentFederated cloning easyForensicsA/B testingRecovery : Logical and physical Development Provision and RefreshFullFreshFrequent (Many) Source control for code, data control for the database Data version per release version Federated cloning QA fork copies off to QA QA fork copies back to Dev Instant replay – set up and run destructive tests performance A/B Upgrade patching Recovery Backup 50 days in size of 1 copy, continuous data protection (use recent slide ob backup schedules full, incr,inrc,inrc, full) Restore logical recovery on prod logical recovery on Dev Debugging debug on clone instead of prod debug on data at the time of a problem Validate physical integrity (test for physical corruption)
  • Change mentality from few as possible to as many as accelerates the businessRemember Jinga ?
  • If every MB was an Inch 300,000 customers 12 copies on average 100 GB avg size PB TB GB 300000*12*100 = 360,000,000 300000*1*.3*100 = 9,000,000 351 PB e p t g 1,191,290,000 feet to moon, 132,000,000 feet around the earthe p t g m k b 15,133,979,520 inches to the moone p t g m k b 351,000,000,00015,133,979,520 inches to the moone p t g m k b 35100000000015133979520 inches to the moon
  •  HD  720TB down to   8TB     ( create 19 x 36TB VDBs )
  • Informatica – finished 2x fasterStubhub -   2 x as many releases a yearKLA-Tencore-  5 x as many projectsQA/QualityStubhub - 20% less bugs in production, found full table scan that would have been missed on subsets
  • Moral of this storyInstead of dragging behind enormous amounts of infrastructureand bureaucracy  required to provide database copiesUses db virteliminates the drag and provides power and acceleration To your companyDefining moment CompetitorsServices
  • Once Last Thinghttp://www.dadbm.com/wp-content/uploads/2013/01/12c_pluggable_database_vs_separate_database.png
  • 250 pdb x 200 GB = 50 TBEMC sells 1GB$1000Dell sells 32GB $1,000.terabyte of RAM on a Dell costs around $32,000terabyte of RAM on a VMAX 40k costs around $1,000,000.
  • http://www.emc.com/collateral/emcwsca/master-price-list.pdf    These prices obtain on pages 897/898:Storage engine for VMAX 40k with 256 GB RAM is around $393,000Storage engine for VMAX 40k with  48 GB RAM is around $200,000So, the cost of RAM here is 193,000 / 208 = $927 a gigabyte.   That seems like a good deal for EMC, as Dell sells 32 GB RAM DIMMs for just over $1,000.    So, a terabyte of RAM on a Dell costs around $32,000, and a terabyte of RAM on a VMAX 40k costs around $1,000,000.2) Most DBs have a buffer cache that is less than 0.5% (not 5%, 0.5%) of the datafile size.
  • reduces storagealleviates DBA of repetitive focus on innovationAccelerates DevelopmentEliminate bottleneck more code faster and of better quality
  • Kscope 2013 delphix

    1. 1. Database VirtualizationandInstant CloningKyle Haileyhttp://dboptimizer.com
    2. 2. Database Cloning ChallengeBusiness want data now.Business don’t understand DBAs.Databases getting bigger & harder to copy.Developers want more copies.Reporting wants more copies.Everyone has storage constraints.If you can’t satisfy the businessdemands your process is broken.
    3. 3. Two PartsI. Cloning TechnologyII. Accelerate your business
    4. 4. Part I : Cloning Technology3. Virtual2. Thin Provision1. Physical
    5. 5. =database1. Physical Cloning
    6. 6. ProblemDevelopersQA and UATReportsFirstcopyProduction• CERN - European Organization for NuclearResearch• 145 TB database• 75 TB growth each year• Dozens of developers want copies.
    7. 7. workaroundsDevelopersQA and UATReportsSharedSub set copyProductionMany copies
    8. 8. Physical ClonesDatabase SubsetsShared Databases
    9. 9. Subsets
    10. 10. The Production‘Wall’Classic problem is that queries thatrun fast on subsets hit the wall inproduction.Developers are unable to test againstall data
    11. 11. Shared Full
    12. 12. Shared access = Poor ProductivityDevelopers andtester get frustratedDatabases become oldand unrepresentativeof production.Requires complexscheduling andmanagement
    13. 13. Never enough environmentsAverage customer makes 12 copies of production- Charles Garry, Database Product Manager Oracle
    14. 14. Physical CopiesTime consumingTime to make copies, days to weeksRMAN backup, archive logs, copy data over, recoverMeetings , days to weeksAdmins: System, Storage ,Database ,Network,manager coordinationSpace consuming40 devs x 2.5TB production = 100TB20 report DBs x 40 TB = 800TB=> bottlenecks
    15. 15. Setup Develop
    16. 16. Setup
    17. 17. Setup DevelopQA
    18. 18. $40M$75M$850M$27,000MStorageITDevelopBusiness
    19. 19. ERP Project Failures 2011• NYC CityTime : delays $63 M => $760 M• Montclair Uni: delays sues PeopleSoft• Idaho : delays ERP cost millionsStandish : IT Project Failure Rate1994 1996 1998 2000 2002 2004 200931% 40% 28% 23% 15% 18% 24%★http://www.galorath.com/wp/software-project-failure-costs-billions-better-estimation-planning-can-help.php*http://www.pcworld.com/article/246647/10_biggest_erp_software_failures_of_2011.html
    20. 20. Clone 1 Clone 399% of blocks are IdenticalClone 2
    21. 21. 2. Thin Provision Cloning
    22. 22. Clone 1 Clone 2 Clone 3Thin Provision
    23. 23. 2. Thin Provision CloningCornerstone Technology:File System Snapshots
    24. 24. Thin Cloning• Snapshot DB Files @ point in time• Copy off production• Snapshot again• Export over NFS or FC to Host• Recovery Database
    25. 25. Netapptr-3761.pdf
    26. 26. NetappSnapManagerSnapManagerRepositoryProtectionManagerSnap DriveSnapManagerSnap MirrorFlex CloneRMANRepositoryProductionDevelopmentDBAStorageAdmin
    27. 27. NetApp Filer - DevelopmentNetApp Filer - ProductionProduction DatabaseDatabaseLunsTarget ATarget BTarget CClone 1Clone 2Clone 3Clone 4Snap mirrorSnapshot Managerfor OracleFlexcloneRepositoryDatabaseNetappSnapDriveProtectionManage
    28. 28. 2. Thin Provision Cloning
    29. 29. 2. Thin Provision Cloning
    30. 30. 3. Database Virtualization
    31. 31. Virtualization Layer38Virtualization
    32. 32. DatabaseVirtualizationAppliance(DVA)3 Clones Physical 3 Clones Virtual
    33. 33. Install Delphix on x86 hardwarex86 hardware
    34. 34. Allocate Storage to DelphixAllocateStorageAny type
    35. 35. One time backup of source databaseDatabaseProductionInstanceFile systemRMAN APIs
    36. 36. Delphix Compress DataDatabaseProductionInstanceFile systemData iscompressedtypically 1/3size
    37. 37. Incremental forever change collectionDatabaseProductionInstanceFile systemChanges are collectedautomatically foreverData older than retentionwidow freed
    38. 38. Typical ArchitectureDatabaseFile systemProductionInstanceDatabaseFile systemDevelopmentInstanceDatabaseFile systemQAInstanceDatabaseUATInstanceFile system
    39. 39. Clones share duplicate blocksDevelopmentInstanceDatabaseProductionInstanceFile systemvDatabaseQAInstanceUATInstancevDatabase vDatabaseSource Database Clone Copies of Source Database
    40. 40. Use Cases1. Development2. Recovery3. Reporting
    41. 41. 1. Development Acceleration
    42. 42. 1: Development Accelerationa) Developer each get a copy– Fast, fresh, full, frequent– Self serviceb) Branchingc) Federated
    43. 43. SourceFastSource DatabaseTarget HostVirtual DatabaseNFSFiberFiberRMAN overTCPNo Data Movement
    44. 44. SourceFreshVirtual DatabaseFiber
    45. 45. SourceFrequentVirtual DatabaseVirtual DatabaseTarget HostsVirtual DatabaseVirtual DatabaseFiber
    46. 46. Full clones
    47. 47. Self Service
    48. 48. 1 b) Branching and Rapid QA
    49. 49. dSource1 b) BranchingDeveloper VDBQA VDB
    50. 50. Devv2.6 v2.6v2.6QA UATv2.6v2.6 v2.6v2.6v2.7v2.6 v2.6v2.6v2.8v2.6v2.6v2.6v2.6v2.6v2.7v2.6v2.7v2.6v2.8v2.6v2.8
    51. 51. Devv2.6 v2.6v2.6QA UATv2.6Productionv2.6 v2.6v2.6v2.7v2.6 v2.6v2.6v2.8Source Control for the database datav2.6v2.6v2.6v2.6v2.6v2.7v2.6v2.7v2.6v2.8v2.6v2.8
    52. 52. DevProd2.6
    53. 53. DevQAProd2.6
    54. 54. DevQAUATProd2.6
    55. 55. DevQAUATProdDevQAUAT2.62.7
    56. 56. DevQAUATProdDevQAUAT2.62.7DevQAUAT2.8
    57. 57. DevQAUATProdDevQAUAT2.62.7DevQAUAT2.8Data Control = Source Control for the Database
    58. 58. DevQAUATDevQAUAT2.62.7DevQAUAT2.8Data Control = Source Control for the DatabaseProduction Time Flow
    59. 59. 1 c) Federated Cloning
    60. 60. Source2Source3Source11 c) Federated sourcesVirtual DatabaseVirtual DatabaseVirtual DatabaseVirtual Database
    61. 61. “I looked like a hero”Tony Young, CIO Informatica
    62. 62. 1. Review Development Use Casesa) Developer each get a copya) QAb) Federated
    63. 63. 2. Recovery, Testing, Forensicsa) Forensicsb) A/B testingc) Recovery
    64. 64. Source2 a) Forensic AnalysisVirtual Database
    65. 65. Source2 b) Upgrades, Patches, RAT, A/BVirtual Database
    66. 66. • Production vs Virtual– invisible index on Prod– Creating index on virtual• Flashback vs Virtual• Keep tests for compare2 b) Upgrades, Patches, RAT, A/B
    67. 67. 2 c) Recovery
    68. 68. Source2 c) Logical Recovery ProductionVirtual Database
    69. 69. Source2 c) Logical Recovery DevelopmentVirtual DatabaseVDB rolled back
    70. 70. SourceRecoveryVDBV2P
    71. 71. 2. Recovery, Testing, Forensicsa) Forensicsb) A/B testingc) Recovery : Logical and physical
    72. 72. 3: reportinga) Fast refreshesb) Temporal queriesc) Confidence testing
    73. 73. Fast Refreshes• Refresh in minutes• Without data movement• Faster , cheaper
    74. 74. Temporal Data
    75. 75. 3: reportinga) Fast refreshesa) Temporal queriesb) Confidence testing
    76. 76. Review: Use Cases1. Developmenta) Full, Fresh, Fast , Self Serveb) Branchingc) Federated2. Recovery, Testing :a) Forensicsb) Testing : A/B, upgrade, patchc) Recovery: logical, physical3. Reportinga) Fast refreshb) Temporal Datac) Confidence testing
    77. 77. over 10 times"perhaps the single largest storage consolidationopportunity history“
    78. 78. Oracle 12c
    79. 79. 80MB buffer cache ?
    80. 80. 200GBCache
    81. 81. 5000Tnxs/minLatency300ms1 5 10 20 30 60 100 200with1 5 10 20 30 60 100 200Users
    82. 82. 8000Tnxs/minLatency600ms1 5 10 20 30 60 100 200Users1 5 10 20 30 60 100 200
    83. 83. Database Virtualization