Delphix and DBmaestro


Published on

Published in: Software, Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Growing almost 300% a year

    Founded in 2008, launched in 2010
    Jedidiah Yueh, President and CEO
    Founded Avamar in 1999, sold to EMC in 2006, VP Product Mgmt at EMC
    Avamar: >$1B revenue,

    150 Employees: HQ in Menlo Park, SF, Boston, DC, London, NY and Atlanta
    Growing 250% annually – 130+ customers including 100 Fortune1000 Customers
  • 80% of outages impacting mission-critical services caused by people and process issues thru 2015, with the majority of those outages (50%+) caused by change/configuration/release integration and hand-off issues (Gartner RAS Core Research Note G00208328 Ronni J. Colville, George Spafford [October 27, 2010] – Strategic Planning Assumption(s) “Top Seven Considerations for Configuration Management for Virtual and Cloud )
  • I’m not sure if you have run into these situations at your organization or
    if you can imagine some of these situations
    But here are some of the issues we at Delphix are seeing in the industry with the companies we are talking to.
    Let’s look at the 5 points in more detail

  • We talked to Presbyterian Healthcare
    And they told us that they spend 96% of their QA cycle time building the QA environment
    And only 4% actually running the QA suite
    This happens for every QA suite
    For every dollar spent on QA there was only 4 cents of actual QA value
    And that 96% cost is infrastructure time and overhead

  • Because of the time required to set up QA environments
    The actual QA tests suites lag behind the end of a sprint or code freeze
    Meaning that the amount of time that goes by after the introduction of a bug in code and before the bug is found increases
    And the more time that goes by after the introduction of a bug into the code
    The more dependent code is written on top of the bug
    Increasing the amount of code rework required after the bug is finally found

    In his seminal book that some of you may be familiar with, “Software Engineering Economics”, author Barry Boehm
    Introduce the computer world to the idea that the longer one delays fixing a bug in the software development life cycle
    The more expensive it is to to fix that bug and these cost rise exponentially as delays increase

  • Not sure if you’ve run into this but I have personally experience the following
    When I was talking to one development group at Ebay, they
    Shared a single copy of the production database between the developers on that team.
    What this sharing of a single copy of production meant, is that whenever a
    Developer wanted to modified that database, they had to submit their changes to code
    Review and that code review took 1 to 2 weeks.
    I don’t know about you, but that kind of delay would stifle my motivation

    And I have direct experience with the kind of disgruntlement it can cause.
    When I was last a DBA, all schema changes went through me.
    It took me about half a day to process schema changes. That delay was too much so it was unilaterally decided by
    They developers to go to an EAV schema. Or entity attribute value schema
    Which mean that developers could add new fields without consulting me and without stepping on each others feat.
    It also mean that SQL code as unreadable and performance was atrocious.

    Besides creating developer frustration, sharing a database
    also makes refreshing the data difficult as it takes a while to refresh the full copy
    And it takes even longer to coordinate a time when everyone stops using the copy to make the refresh
    All this means is that the copy rarely gets refreshed and the data gets old and unreliable

  • To circumvent the problems of sharing a single copy of production
    Many shops we talk to create subsets.
    One company we talked to , RBS spends 50% of time copying databases
    have to subset because not enough storage
    subsetting process constantly needs fixing modification

    Now What happens when developers use subsets -- ****** -----
  • The biggest and most pervasive problem we see is slow build times.
    In order to set up an database copy for a development environments
    Requires submitting a request to management who has to review it
    Then if the request is granted, it is passed to the DBA who has to coordinate with the Sysadmin who has to coordinate with the storage admin.
    In such a situation it makes sense that copying a large database would take a long time
    But even when we talk to someone who uses netapp storage snapshots like Electronics Art and Ariba, they said even using storage snapshot sit took
    days to weeks get a database clone copy due to the coordination between DBA, sys admin and storage admin
    At many of the customers we talk to provisioning a database clone copy takes weeks or months.
    One large global bank quotes us as taking typically 6 months to provision a database clone copy environment.

    Requirements: self service for app teams
    Requirements: end-to-end automation
    Metrics: # people, process, time for delivery

    So far we have talked about the weight of infrastructure on app delivery. Of course, to control and manage that infrastructure, firms layer on a large set of bureaucratic processes, change control, approvals, procurement, governance, etc etc. So the operational and organizational hurdles then create an even bigger drag on IT and app development.

    Here’s an example from one banking customer.

    Once the app developer puts in a request for a new development environment, there’s at least a week long wait for management approvals. Then project DBA work with the sysadmin and storage groups for capacity. If more capacity needs to be allocated, it’s 3 more days. If more needs to be purchased, weeks or months.
    If a copy of production data is needed, the process needs to wait on a production DBA, who might be busy with production issues. Recovering the database to a specific point in time and configuration can also take days.
    It is very common for two weeks to pass between a developer request and a ready environment. The process can be repeated for multiple environments, for data refreshes, and for integration across multiple systems.

    With Delphix, turns stop signs into green lights. Provisioning, refresh, rollback, and data integration happen nearly instantly and do not trigger approvals from production systems or require additional storage. That is why KLA is able to deliver 5 times the output from its SAP teams…

    Without Delphix, it’s impossible for organizations to implement the level of agile processes they desire. The management of data, and the bureaucracy of data management, slows things down too much.
  • Due to the constraints of building clone copy database environments one ends up in the “culture of no”
    Where developers stop asking for a copy of a production database because the answer is “no”
    If the developers need to debug an anomaly seen on production or if they need to write a custom module which requires a copy of production they know not to even ask and just give up.
  • I’m not sure if you have run into these situations at your organization or
    if you can imagine some of these situations
    But here are some of the issues we at Delphix are seeing in the industry with the companies we are talking to.
    Let’s look at the 5 points in more detail
  • In the physical database world, 3 clones take up 3x the storage.
    In the virtual world 3 clones take up 1/3 the storage thanks to block sharing and compression
  • installs an any Intel commodity hardware as a VM

    supports Oracle 9.2-12c, standard edition, enterprise edition, single instance and RAC on AIX, Sparc, HPUX, LINUX

    support SQL Server
  • EMC, Netapp, Fujitsu,
    Or newer flash storage like
    Violin, Pure Storage, Fusion IO etc
    Recent benchmark with Delphix and Pure Storage showed that a Flash and Delphix solution is an order of magnitude cheaper than spinning disk and more performant
  • Delphix does a one time only copy of the source database onto Delphix
  • Quote from a customer “Delphix GUI is what Oracle Enterprise Manager
    would look like if Apple had designed it”

    Delphix inter face is user friendly, polished and easy to use

  • Presbyterian when from 10 hour builds to 10 minute builds

    Total Investment in Test Environment: $2M/year
    10 QA engineers
    DBA, storage team dedicated to support testing
    App, Oracle server, storage, backups
    Restore load competes with backup jobs

    Requirements: fast data refresh, rollback

    Data delivery takes 480 out of 500 minute test cycle (4% value)
    $.04/$1.00 actual testing vs. setup

  • For example Stubhub went from 5 copies of production in development to 120
    Giving each developer their own copy
  • Stubhub estimated a 20% reduction in bugs that made it to production
  • Delphix and DBmaestro

    1. 1. Thank you for joining us, the webinar will start at: 10:00 Pacific / 12:00 Central / 13:00 East / 18:00 UK Time Manage Databases like Codebases
    2. 2. Before we start • You will be on mute for the duration of the event • We are now talking so please type a message in the Questions box in the Control Panel if you can’t hear us (please check your speakers and GoToWebinar audio settings first) • There will be a Q+A session at the end but please feel free to type your questions in the Questions box in the Control Panel in advance • A recording of the full webinar will be put up online
    3. 3. Presenters Tim O'Brien • Analyst, CTO • Gleanster Research Yaniv Yehuda @yanivyehuda • Co-Founder & CTO at DBmaestro • Presenter at world-wide conferences: ODTUG, ilOUG Kyle Hailey @kylehhailey • Technical Evangelist at Delphix • Oracle ACE, member of the OakTable Network
    4. 4. About Delphix • Founded in 2008, launched in 2010 • CEO Jedidiah Yueh (founder of Avamar: >$1B revenue)) • Based in Silicon Valley, Global Operations • 10% of Fortune 500
    5. 5. About DBmaestro • Founded in 2008, launched in 2010 • Founded by Yariv Tabac and Yaniv Yehuda • Headquartered in Israel, Global Operations
    6. 6. Manage Databases like Codebases Kyle Hailey & Yaniv Yehuda
    7. 7. The Business Need Copyright@2010, Gartner RAS Core Research 80% of unplanned downtime is due to Change 50% More than of this, half is due to human errors 40% of changes FAIL Copyright@2008, Juniper Networks, Inc.
    8. 8. Code, Databases, and Agility in 2014 May20,2014 Tim O’Brien Analyst, CTO Gleanster Research
    9. 9. Industry Trends Code in 2014  Distributed& AsynchronousNew Normal  Self-serviceCriticaltoProductivity  IncreasingVariety (fromJavascript,Java, .NET, Ruby,Python+everything) Everything is Automated
    10. 10. Industry Trends Databases in 2014  More thanjustRelational  IncreasingVariety (Oracle,Mongo)  Management? Notas Matureas Code Oddity  Big Datamarginallymore manageablethan traditionalRDBMS  “We’restillwaitingforthedatabase…”
    11. 11. Code is King Modern Enterprise  Fully distributed development  Continuous Deployment Pipelines  Everything Automated! - DevOps, PaaS, IaaS “Every Company is a Software Company” How did we get here?
    12. 12. Scale Development Open Source @ Scale  Google Chrome, Mozilla Firefox  Android, Apache httpd  Linux*, FreeBSD Characteristics  Rapid Iterations, Frequent Releases  Numerous Experiments (or Branches)  Thousands of developers  OSS @ Scale established best practices now in use.
    13. 13. Scale Development Organizations @ Scale  Facebook,Wal-Mart, eBay, Salesforce, Intuit  Goldman Sachs, Apple, Amazon,Yahoo!  Google, SSA, Ford,Toyota Characteristics  Rapid Iterations, Frequent Releases  Numerous Experiments (or Branches)  Thousands of developers, ManyTeams
    14. 14. @Scale Everything is Continuous Continuous integration and deployment Modern set ofTools Integrated with:  Source Control – Perforce, Clearcase, Git  Infrastructure-as-a-Service – Openstack  Platform-as-a-Service – AWS, Heroku  Databases? - Still an Integration Nightmare
    15. 15. Case in Point  Example Fortune 500 Relaunch  Re-architecture: More distributed, Move to a PaaS  Scope: Everything, 2Year Migration Project  Size: 1500 Developers, Several Continents, $100M+  Results?  Code: Distributed Projects, Continuous Deployment  Infrastructure: Self-service, On-demand, Agile  Database: Delays, Manual Builds, Cost Overruns
    16. 16. RootCause Analysis  Code is Agile  Developers move FAST.  Branches are fungible, disposable, personal.  Deployments managed with DevOps tools (immediately)  Traditional Databases are Not  Costly to Setup  Often require Dedicated Hardware  Little automation. No Change Management “The rest of the department is Virtual but our Databases are stuck in 1998.”
    17. 17. Worst Practices (we’reall used to)  Lack of Database Change Management:  Shared Developer Databases Results in Contention  Costly, Slow Environment Provisioning  Testing and Staging Not Synchronized with Production  Drag on Productivity and Efficiency  Opportunity lost  “We’ve accepted that the databases causes delays”
    18. 18. What We’ve Seen 1. QA: Expensive, Slow 2. Development: Bottlenecks & bugs 3. Provisioning: Delays
    19. 19. 1. QA: Expensive, Slow : Long Build times Build Time 96% of QA time was building environment $.04/$1.00 actual testing vs. setup Build Build Time QA Test QA Test Build
    20. 20. 1. QA: Expensive, Slow: bugs found late Build QA Env QA Build QA Env QA Sprint 1 Sprint 2 Sprint 3 Bug CodeX 0 10 20 30 40 50 60 70 1 2 3 4 5 6 7 Delay in Fixing the bug Cost To Correct Software Engineering Economics – Barry Boehm (1981) Build QA Env QA
    21. 21. 2. Development: Bottlenecks & bugs: Bottlenecks Frustration Waiting
    22. 22. 2. Development: Bottlenecks & bugs: subsets cause bugs
    23. 23. 2. Development: Bottlenecks & bugs: subsets cause bugs
    24. 24. 3. Provisioning Delays : weeks to months Management DBA System Admin Storage Admin Developers Submit Request Disk Capacity? Approve Request $$ (2 Weeks) Approve Request $$ (1 Week) Request Additional Storage? Provision Capacity File System Configured? Configure LUNS & Build File System Coordinate Replication w/ Infrastructure Re- Parameterize & Configure DB Mount Recovery DB to Specific PIT Begin Work Approve Request $$ (2 Weeks) (3 Days) (3 Days) (2 Days) (3 Days) (3 Days) …….1-2 Weeks of Approvals, Delays, and Provisioning…… 24 Developer Asks for DB Get Access Manager approves DBA Request system Setup DB System Admin Request storage Setup machine Storage Admin Allocate storage (take snapshot)
    25. 25. 3. Provisioning Delays : culture of no
    26. 26. What We’ve Seen 1. QA: Higher costs more code re-work 2. Development: Bottlenecks & bugs 3. Provisioning: Delays
    27. 27. Three Physical Copies Three Virtual Copies
    28. 28. Install Delphix on Commodity Intel Hardware Intel hardware
    29. 29. Allocate Any Storage to Delphix Allocate Storage Any type
    30. 30. One time backup of source database Database Production Instance File systemFile system
    31. 31. DxFS (Delphix) Compress Data Database Production Instance Data is compressed typically 1/3 size File system
    32. 32. Incremental forever change collection Database Production Instance File system Changes • Collected incrementally forever • Old data purged Time Window File system
    33. 33. Typical Architecture Production Instance Development Instance QA Instance UAT Instance Database File systemFile system Database File systemFile system Database File systemFile system Database File systemFile system
    34. 34. With Delphix Database File system Production Instance Database Development Instance Database QA Instance Database UAT Instance
    35. 35. Parallel environments Instance Instance Instance Instance Source
    36. 36. What We’ve Seen With Delphix 1. QA: Low cost, fast bug detection 2. Development: Parallelized, less bugs 3. Provisioning: Fast, Culture of Yes
    37. 37. 1. QA Efficient : Lower cost B u i l d T i m e QA Test 1% of QA time was building environment $.99/$1.00 actual testing vs. setup Build Time QA Test Build
    38. 38. 1. QA Fast : bugs found fast and fixed Sprint 1 Sprint 2 Sprint 3 Bug CodeX QA QA Build QA Env Q A Build QA Env Q A Sprint 1 Sprint 2 Sprint 3 Bug Cod e X
    39. 39. 2. Development : Parallelize gif by Steve Karam
    40. 40. 2. Development: Eliminate bugs
    41. 41. 3. Provisioning: Fast, Efficient, Self Service, Culture of Yes!
    42. 42. What We’ve Seen With Delphix 1. QA: Low cost, fast bug detection 2. Development: Parallelized, less bugs 3. Provisioning: Fast, Culture of Yes But … How do we manage database changes as code changes and automate deployment of changes?
    43. 43. Dealing with Risk  Smaller and more focused changes are easier to manage (Agile…)  Automation of repeating tasks lowers risk of (human) error  Development and Operations should work in synergy (DevOps)
    44. 44. 45 • More rapid changes • Reacting quickly to market needs • Getting ahead of competition • Fewer changes backed out • Better collaboration • Fewer defects • Ultimately better service • Happy customers • Profitability Why Continuous Delivery?
    45. 45. 46 • Team and process • Version everything • Automate your tests • Fix it, properly (no out of process changes!) • Automate your deployments Database Continuous Delivery?
    46. 46. 47 A quick poll
    47. 47. 48 • The database holds your essential information • Changes can impact the entire system • Need to be synchronized with other changes • Often overlooked Database is a Key Component
    48. 48. 49 • There is more to a database than SQL scripts – Schema structure – Code – Content and meta-content – Internal dependencies – … • Ensure that changes are not made without authorization • Ensure no out-of-process changes Reaching Inside the Database
    49. 49. 50 • Old adage but true • The database is often neglected and therefore can become the weakest link • Essential from a compliance point of view • Should be the strongest link The Weakest Link In a Chain
    50. 50. 51 Challenges… Un-coordinated Process Silos in development Deployment delays… Out-of-Process Changes Errors in production Lack of confidence in automation Code overrides Different methodologies Lack of history Missing changes wrong versions
    51. 51. 52 Two isolated Processes Version Control Process Development Process Check-Out Script Modify Script Get updated Script from DB Check-In Script Compile Script in DB Debug Script in DB ? ? ? ? A A’
    52. 52. 53 X Build Once Deploy Many Int QA Stage Prod Database Deploy Script Environment Re-Base (due to defects) Dev Dev Dev Model 1.1 1.2 1.2 1.3 1.3 1.4 1.4 1.5 1.5 1.6 1.6 1.7 1.1 1.2 1.2 1.3 1.3 1.4 1.4 1.5 1.5 1.6 1.6 1.7 1.1 1.2 1.2 1.3 1.3 1.4 1.4 1.5 1.5 1.6 1.6 1.7 Out of Process Change X X X X X ? 1.1.1 X
    53. 53. 54 Dealing with challenges… Coordinated ProcessTraceability Start in the Beginning No Out-of-Process Changes Impact Analysis Automation Task Based Development Well Defined Processes
    54. 54. What is DBmaestro TeamWork • Database Enforced Change Management + Database version control + Plugs into the ALM (change request, tickets & work items) + Database merge & change impact analysis • DevOps Solution for databases + Baseline aware deployment automation, rollback & recovery + Plugs into release management & Continuous Delivery
    55. 55. How? • Database version control – Enforced Check Out/In – Labels – Rollback/Undo – Audit trail reports • Database impact analysis – Utilizes version control repository information – 3-way analysis • Database deployment automation – API – Baselines – Conflict resolution – Customized business logic
    56. 56. 57 Version Control - One Enforced Process
    57. 57. 58 Build & Deploy On Demand * Int QA Stage Prod Database Deploy Script Environment* Execute the same script being executed at the Stage environment Re-Base (due to defects) Dev Dev Dev Model 1.1 1.2 1.2 1.3 1.3 1.4 1.4 1.5 1.5 1.6 1.6 1.7 1.1 1.4 1.4 1.7 1.1.1 1.7 1.1 1.1 File Based Version Control Out of Process Change
    58. 58. Safety Net For Deployment Automation Source vs. Target Action = No Action ≠ ? Source vs. Baseline Target vs. Baseline Action = = No Action ≠ = Deploy Changes = ≠ Protect Target ≠ ≠ Merge Changes You do not have all of the information With Baselines and 3 way analysis the unknown is now known Simple Compare & Sync Baseline aware Deployment An index exists in Target (Production) but not in source (QA). What should we do? Drop the index or not?
    59. 59. Benefits - Development • Database change repository • Follow SCM best practices (Check-Out/Check-In) • All changes are documented • Manage who can do what, where, when & why
    60. 60. Benefits - Operations • Integrated deployment engine • Business level audit • Roles & responsibilities enforcement
    61. 61. Benefits - Management • Complete visibility into changes in progress • Management reports, Audit reports • No silos
    62. 62. +
    63. 63. Time Window Instance Instance Instance Virtual DatabaseDev1 Dev2 Trunk Virtual Database Virtual Database Developer 1 modify Developer 2 modify DB VC
    64. 64. Dev2 Dev1Merge to dev1 Trunk DB VC Fork Fork Fork Fork
    65. 65. Safe and Efficient Work Process • Clone relevant number of virtual copies of the Trunk – Dev Team 1-n, Developer a-z, Hot fix environment, Etc… – Work in parallel, save time • Rely on enforced change process – Know exactly who did what, when and why • Deployment automation – Save time, mitigate risk – Continuous integration, Continuous Delivery • Deal with conflicts & merge them into the Trunk • Test before deploy to production – Pre-prod clone
    66. 66. Q&A Kyle Hailey @kylehhailey Delphix: Yaniv Yehuda @yanivyehuda DBmaestro: Tim O'Brien Gleanster:
    67. 67. Thanks! Kyle Hailey @kylehhailey Delphix: Yaniv Yehuda @yanivyehuda DBmaestro: Tim O'Brien Gleanster: