Your SlideShare is downloading. ×
  • Like
What is Delphix
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Now you can save presentations on your phone or tablet

Available for both IPhone and Android

Text the download link to your phone

Standard text messaging rates apply

What is Delphix

  • 1,012 views
Published

 

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,012
On SlideShare
0
From Embeds
0
Number of Embeds
2

Actions

Shares
Downloads
39
Comments
0
Likes
2

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • Interview Delphix blew me away. As a DBA I had to spend 50% of my time making copiesAfter joining Delphix, I banged on Delphix for 2 years. It worksBlog entryReinforce ideas we’ve already seen from a different perspectiveWork for a company called DelphixWe write software that enables Oracle and SQL Server customers toCopy their databases in 2 minutes with almost no storage overheadWe accomplish that by taking one initial copy and sharing the duplicate blocks Across all the clones
  • In the physical database world, 3 clones take up 3x the storage.In the virtual world 3 clones take up 1/3 the storage thanks to block sharing and compressionDelphix orchestrates access to the shared compressed blocks on the storage that Delphix manages.Delphix manages changes block access only making changed blocks visible to the database that made that block change.
  • Software installs an any x86 hardware uses any storage supports Oracle 9.2-12c, standard edition, enterprise edition, single instance and RAC on AIX, Sparc, HPUX, LINUX support SQL Server
  • EMC, Netapp, Fujitsu, Or newer flash storage likeViolin, Pure Storage, Fusion IO etc
  • Delphix does a one time only copy of the source database onto Delphix
  • Physically independent but logically correlatedCloning multiple source databases at the same time can be a daunting task
  • One example with our customers is InformaticaWho had a project to integrate 6 databases into one central databaseThe time of the project was estimated at 12 monthsWith much of that coming from trying to orchestratingGetting copies of the 6 databases at the same point in timeLike herding cats
  • Informatical had a 12 month project to integrate 6 databases.After installing Delphix they did it in 6 months.I delivered this earlyI generated more revenueI freed up money and put it into innovationwon an award with Ventana Research for this project
  • Developer each get a copyFast, fresh, full, frequentSelf serviceQA branch from DevelopmentFederated cloning easyForensicsA/B testingRecovery : Logical and physical Development Provision and RefreshFullFreshFrequent (Many) Source control for code, data control for the database Data version per release version Federated cloning QA fork copies off to QA QA fork copies back to Dev Instant replay – set up and run destructive tests performance A/B Upgrade patching Recovery Backup 50 days in size of 1 copy, continuous data protection (use recent slide ob backup schedules full, incr,inrc,inrc, full) Restore logical recovery on prod logical recovery on Dev Debugging debug on clone instead of prod debug on data at the time of a problem Validate physical integrity (test for physical corruption)
  • We talked to Presbyterian HealthcareAnd they told us that they spend 96% of their QA cycle time building the QA environmentAnd only 4% actually running the QA suiteThis happens for every QA suitemeaningFor every dollar spent on QA there was only 4 cents of actual QA value Meaning 96% cost is spent infrastructure time and overhead
  • Presbyterian when from 10 hour builds to 10 minute buildsTotal Investment in Test Environment: $2M/year10 QA engineersDBA, storage team dedicated to support testingApp, Oracle server, storage, backupsRestore load competes with backup jobsRequirements: fast data refresh, rollbackData delivery takes 480 out of 500 minute test cycle (4% value)$.04/$1.00 actual testing vs. setup
  • Because of the time required to set up QA environmentsThe actual QA tests suites lag behind the end of a sprint or code freezeMeaning that the amount of time that goes by after the introduction of a bug in code and before the bug is found increasesAnd the more time that goes by after the introduction of a bug into the codeThe more dependent is written on top of the bug Increasing the amount of code rework required after the bug is finally foundIn his seminal book that some of you may be familiar with, “Software Engineering Economics”, author Barry Boehm Introduce the computer world to the idea that the longer one delays fixing a bug in the application design lifescyleThe more expensive it is to to fix that bug and these cost rise exponentially the laterThe bug is address in the cycle
  • Not sure if you’ve run into this but I have personally experience the followingWhen I was talking to one group at Ebay, in that development group they Shared a single copy of the production database between the developers on that team.What this sharing of a single copy of production meant, is that whenever a Developer wanted to modified that database, they had to submit their changes to codeReview and that code review took 1 to 2 weeks.I don’t know about you, but that kind of delay would stifle my motivationAnd I have direct experience with the kind of disgruntlement it can cause.When I was last a DBA, all schema changes went through me.It took me about half a day to process schema changes. That delay was too much so it was unilaterally decided byThey developers to go to an EAV schema. Or entity attribute value schemaWhich mean that developers could add new fields without consulting me and without stepping on each others feat.It also mean that SQL code as unreadable and performance was atrocious.Besides creating developer frustration, sharing a database also makes refreshing the data difficult as it takes a while to refresh the full copyAnd it takes even longer to coordinate a time when everyone stops using the copy to make the refreshAll this means is that the copy rarely gets refreshed and the data gets old and unreliable
  • For example Stubhub went from 5 copies of production in development to 120Giving each developer their own copy
  • To circumvent the problems of sharing a single copy of productionMany shops we talk to create subsets.One company we talked to , RBS spends 50% of time copying databases have to subset because not enough storagesubsetting process constantly needs fixing modificationNow What happens when developers use subsets -- ****** -----
  • Stubhub (ebay) estimates that 20% of there production bugs arise from testing onSubsets instead of full database copies.
  • Stubhub estimated a 20% reduction in bugs that made it to production
  • Due to the constraints of building clone copy database environments one ends up in the “culture of no”Where developers stop asking for a copy of a production database because the answer is “no”If the developers need to debug an anomaly seen on production or if they need to write a custom module which requires a copy of production they know not to even ask and just give up.
  • State of Colorado has a 100 projects support 3KLA tencor can only support 2 projects of a dozen
  • Take for example a 9TB source databaseIt’s compressed to 3TB on Delphix.Then we still have 6TB on Delphix before we reach the size of the original database, but that 6TB is actuall 18TB of data at a 1/3 compressionWhich means we can have the original database and 18TB of changes in the size of the original databaseWe see on average about 50 days worth of back up in the size of the original databaseMultiple scripted dumps or RMAN backups are used to move data today. With application awareness, we only request change blocks—dramatically reducing production loads by as much as 80%. We also eliminate the need for DBAs to manage custom scripts, which are expensive to maintain and support over time.
  • Once Last Thinghttp://www.dadbm.com/wp-content/uploads/2013/01/12c_pluggable_database_vs_separate_database.png
  • 250 pdb x 200 GB = 50 TBEMC sells 1GB$1000Dell sells 32GB $1,000.terabyte of RAM on a Dell costs around $32,000terabyte of RAM on a VMAX 40k costs around $1,000,000.
  • http://www.emc.com/collateral/emcwsca/master-price-list.pdf    These prices obtain on pages 897/898:Storage engine for VMAX 40k with 256 GB RAM is around $393,000Storage engine for VMAX 40k with  48 GB RAM is around $200,000So, the cost of RAM here is 193,000 / 208 = $927 a gigabyte.   That seems like a good deal for EMC, as Dell sells 32 GB RAM DIMMs for just over $1,000.    So, a terabyte of RAM on a Dell costs around $32,000, and a terabyte of RAM on a VMAX 40k costs around $1,000,000.2) Most DBs have a buffer cache that is less than 0.5% (not 5%, 0.5%) of the datafile size.
  • Source Control for the database data
  • Source Control for the database data

Transcript

  • 1. What is Delphix
  • 2. kylehailey.com/delphix Use Cases What is Delphix Competition
  • 3. What is Delphix
  • 4. Three Physical Copies Three Virtual Copies Delphix
  • 5. Install Delphix on Intel commodity hardware Intel hardware
  • 6. Allocate Any Storage to Delphix Allocate Storage Any type Pure Storage + Delphix Better Performance for 1/10 the cost
  • 7. One time backup of source database Production Supports Instance Instance Instance Database File system Application Stack Virtualization Upcoming
  • 8. DxFS (Delphix) Compress Data Production Instance Instance Instance Database File system Data is compressed typically 1/3 size
  • 9. Incremental forever change collection Production Instance Instance Instance Database Changes Time Window File system • Collected incrementally forever • Old data purged
  • 10. Change collection works with snapshots Snapshot 0: Source Full Copy Snapshot 1: Changed blocks at time T1
  • 11. Snapshots immediately look like full copies as they are mapped onto the previous copy Snapshot 0 Snapshot 1
  • 12. Snapshot 0 Snapshot 1 Snapshot 2: Changed blocks at time T2
  • 13. Snapshot 0 Snapshot 2 Snapshot 1
  • 14. Purging old snapshot releases any blocks that have been since replaced, but keeps all unchanged blocks Snapshot 2 Snapshot 1 Drop Snapshot 0
  • 15. Cloning : almost no storage at creation since it is all shared. Upon usage changed blocks will add to storage used Production Instance Instance Instance Instance Instance Database Database Time Window File system
  • 16. Typical Architecture Production Dev, QA, UAT Reporting Instance Instance Instance Instance Instance Instance Instance Instance Database Database Database Database Database File system File system File system File system File system File system File system File system File system File system File system Backup Instance Instance Instance File system File system
  • 17. With Delphix Production Instance Instance Database File system Dev & QA Instance Instance Instance Instance Instance Instance Database Database Database Reporting Backup Instance Instance Database Database
  • 18. Delphix Use Cases 1. 2. 3. 4. 5. Fast, Fresh, Full Free Branching Federated Self Serve
  • 19. Fast, Fresh, Full Source Development VDB Instance Instance Time Window
  • 20. Free Instance Source Instance Instance Instance gif by Steve Karam
  • 21. Branching Source Instance branching Dev Instance Source QA branched from Dev Instance
  • 22. Federated Cloning
  • 23. Federated Source1 Instance Source1 Instance Source2 Instance Instance
  • 24. “I looked like a hero” Tony Young, CIO Informatica
  • 25. Self Service
  • 26. Use Cases 1. Development Acceleration 2. Quality 3. BI
  • 27. DevOps
  • 28. DevOps With Delphix 1. 2. 3. 4. 5. Efficient QA: Low cost, high utilization Quick QA : Fast Bug Fix Every Dev gets DB: Parallelized Dev Full DB : Less Bugs Fast Builds: Culture of Yes
  • 29. Without Delphix 1. Inefficient QA: Long Build times Build QA Test Build Time 96% of QA time was building environment $.04/$1.00 actual testing vs. setup
  • 30. : With Delphix 1. Efficient QA: Lower cost Build QA Test Build Time B u i l d T i m e QA Test 1% of QA time was building environment $.99/$1.00 actual testing vs. setup
  • 31. Rapid QA via Branching
  • 32. Without Delphix 2. QA Delays: bugs found late require more code re-work Build QA Env Sprint 3 Sprint 2 Sprint 1 X Build QA Env QA Bug Code 70 60 50 40 30 20 10 0 Cost To Correct 1 2 3 4 5 6 7 Delay in Fixing the bug Software Engineering Economics – Barry Boehm (1981) QA
  • 33. : With Delphix 2. QA Immediate: bugs found fast and fixed Build QA Env Sprint 2 Sprint 1 X Q A Build QA Env Q A Sprint 3 Bug Code QA QA Sprint 2 Sprint 1 X Bug Code Sprint 3
  • 34. Without Delphix 3. Full Copy Shared : Bottlenecks Old Unrepresentative Data Frustration Waiting
  • 35. : With Delphix 3. Private Copies: Parallelize gif by Steve Karam
  • 36. Without Delphix 4. Subsets : cause bugs
  • 37. Without Delphix 4. Subsets : cause bugs The Production ‘Wall’ Classic problem is that queries that run fast on subsets hit the wall in production. Developers are unable to test against all data
  • 38. : With Delphix 4. Full Size DB : Eliminate bugs
  • 39. Without Delphix 5. Slow Environment Builds: Slow Environment Builds: 3-6 Months to Deliver Data Developer Asks for DB Get Access Manager DBA System Admin Storage Admin approves Request system Setup DB Request storage Setup machine Allocate storage (take snapshot)
  • 40. Without Delphix 5. Slow Environment Builds: culture of no DBA Developer
  • 41. Without Delphix Never enough environments
  • 42. : With Delphix 5. Self Service: Fast, Efficient. Culture of Yes!
  • 43. Quality 1. Forensics 2. Testing 3. Recovery
  • 44. 1. Forensics: Investigate Production Bugs Development Instance Instance Time Window Anomaly on Prod Possible code bug At noon yesterday Spin up VDB of Prod as it was during anomaly
  • 45. 2. Testing : Rewind for patch and QA testing Prod Development Instance Instance Time Window Time Window
  • 46. 2. Testing: A/B Instance Test A with Index 1 Instance Instance Time Window • Keep tests for compare • Production vs Virtual – invisible index on Prod – Creating index on virtual • Flashback vs Virtual Test B with Index 2
  • 47. 3. Recovery: Surgical recover of Production Source Development Instance Instance Spin VDB up Before drop Time Window Problem on Prod Dropped Table Accidently
  • 48. 3. Recovery Surgical or Full Recovery on VDB Dev1 VDB Source Instance Instance Dev2 VDB Branched Source Time Window Dev1 VDB Time Window Instance
  • 49. 3. Recovery: Virtual to Physical Source VDB Instance Instance Spin VDB up Before drop Time Window Corruption
  • 50. 50 days of backup in size of Original 3. Recovery
  • 51. Business Intelligence
  • 52. ETL and Refresh Windows 1pm noon 10pm 8am
  • 53. ETL and DW refreshes taking longer 1pm noon 10pm 2011 2012 2013 2014 2015 8am
  • 54. ETL and Refresh Windows Globalization limits ETL windows to weekends 6am 8am 10pm 10pm 1pm noon 8am 10pm 2011 2012 2013 2014 2015 noon 9pm 8am
  • 55. ETL and DW Refreshes Prod DW & BI Instance Instance Data Guard – requires full refresh if used Active Data Guard – read only, most reports don’t work
  • 56. Fast Refreshes • Collect only Changes • Refresh in minutes Prod Instance BI DW Instance Instance ETL 24x7
  • 57. Temporal Data
  • 58. Oracle 12c
  • 59. 80MB buffer cache ?
  • 60. 200GB Cache
  • 61. with Latency Tnxs / min 5000 300 ms 1 5 10 20 30 60 100 200 Users 1 5 10 20 30 60 100 200
  • 62. Latency Tnxs / min 8000 600 ms 1 5 10 20 30 60 100 200 Users 1 5 10 20 30 60 100 200
  • 63. Five 200GB database copies are cached with : $1,000,000 1TB cache on SAN $6,000 200GB shared cache on Delphix
  • 64. Data Center Migration : clone migration 5x Source Data Copy < 1x Source Data Copy
  • 65. Data Center Migration : clone migration + source S S 5x Source Data Copy < 2 x Source Data Copy
  • 66. Data Center Migration : clone migration + source S C C C 5x Source Data Copy C S V V V < 1 x Source Data Copy V
  • 67. Consolidation Without Delphix Active Active With Delphix Idle Active Idle Active
  • 68. DBMS License Utilization • Most DBMS time spent waiting on environment setup, data loads, backups, etc. • Coupling of data and DB servers forces over-provisioning, limits utilization • Delphix can attach / detach data to server tier on-demand to drive consolidation • Potential 2X reduction in DB server footprint cuts HW and DBMS license © 2014 DELPHIX. ALL RIGHTS RESERVED. PRIVATE & CONFIDENTIAL. costs 69
  • 69. Dev v2.6 v2.6 QA v2.6 v2.6 UAT v2.6 v2.6 v2.6 v2.7 v2.6 v2.7 v2.7 v2.6 v2.6 v2.6 v2.8 v2.8 v2.6 v2.8
  • 70. Production Dev v2.6 v2.6 QA v2.6 v2.6 UAT v2.6 v2.6 v2.6 v2.7 v2.6 v2.7 v2.7 v2.6 v2.6 v2.6 v2.8 v2.8 v2.6 v2.8 Source Control for the database data
  • 71. 2.6 branching Dev Prod checkout bookmark
  • 72. 2.6 QA Dev Prod
  • 73. 2.6 UAT QA Dev Prod
  • 74. 2.6 UAT QA Dev Prod Dev QA UAT 2.7
  • 75. 2.6 2.8 UAT QA UAT QA Dev Dev Prod Dev QA UAT 2.7
  • 76. 2.6 2.8 UAT QA UAT QA Dev Dev Prod Dev QA UAT 2.7 Data Control = Source Control for the Database
  • 77. 2.6 2.8 UAT QA UAT QA Dev Dev Production Time Flow Dev QA UAT 2.7 Data Control = Source Control for the Database