Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Data is the Constraint

615 views

Published on

Published in: Technology
  • Be the first to comment

Data is the Constraint

  1. 1. Data is the Constraint Blog: kylehailey.com • • • • • • • 20 years working w/ Oracle Oracle Ace Oaktable Conferences Training days Tech journal articles Sales meetings
  2. 2. Main blogs • Personal blog : http://kylehailey.com • Linkedin www.linkedin.com/in/kylehailey • Twitter @kylehhailey • Facebook facebook.com/kyle.hailey.73
  3. 3. Top Blog Posts • http://www.kylehailey.com/delphix/ – – – – – What is Delphix Benefits in Brief High Performance Delphix Jonathan Lewis Explains Delphix Delphix vs Netapp and EM 12c
  4. 4. Data is the constraint Three points I. Data Tax strains infrastructure II. Data Tax price is huge III. Companies unaware of Data Tax
  5. 5. I. Data Taxes your business If you can’t satisfy the business demands then your process is broken.
  6. 6. II. Data Tax hits the bottom line hard
  7. 7. III. Companies don’t think there is a Data Tax problem
  8. 8. I. Data Tax – Moving data is hard – Triple tax – Data Floods infrastructure
  9. 9. I. Data Tax : moving data is hard – Storage & Systems : capital resources – Personnel : operation expenditure – Time : delayed projects
  10. 10. I. Data Tax : Typical Architecture Production Dev, QA, UAT Reporting Instance Instance Instance Instance Instance Instance Instance Instance Database Database Database Database Database File system File system File system File system File system File system File system File system File system File system File system Backup Instance Instance Instance File system File system
  11. 11. I. Data Tax: triple data tax Sandbox Development QA UAT Production Business Intelligence Analytics Reporting Tape Backup Recovery Forensics
  12. 12. I. Data Tax : Copying data floods infrastructure
  13. 13. I. Data Tax : Data Tax floods company infrastructure 92% of the cost of business — the financial services business — is “data” www.wsta.org/resources/industry-articles Most companies have 2-9% IT spending http://uclue.com/?xq=1133 Data management is the largest portion of IT expense
  14. 14. Part II. Data Tax is Huge
  15. 15. Part II. Data Tax is Huge • Four Areas data tax hits 1. 2. 3. 4. IT Capital resources IT Operations personnel Application Development *** Business • How big is the data tax? – Measure before and after installing Delphix
  16. 16. II. Data Tax is huge : 1. IT Capital • Hardware – – – – Servers Storage Network Data center floor space, power, cooling • Example – Some customers have over 1 Petabyte duplicate data – (1000 TB, ie 1,000,000 GB )
  17. 17. II. Data Tax is huge : 2. IT Operations • Involves many people – – – – – DBAs SYS Admin Storage Admin Backup Admin Network Admin • 1000s of hours annually just for DBAs – not including all the other personnel required to supply the infrastructure necessary • Data center efforts costly and difficult – Consolidation – Migrations – Move to cloud
  18. 18. II. Data Tax is Huge : 3. App Dev quality and speed Five examples of application Data Tax impact • • • • • Inefficient QA: Higher costs of QA QA Delays : Greater re-work of code Sharing DB Environments : Bottlenecks Using DB Subsets: More bugs in Prod Slow Environment Builds: Delays “if you can't measure it you can’t manage it”
  19. 19. II. Data Tax is Huge : 3. App Dev Inefficient QA : Long Build times Build QA Test Build Time 96% of QA time was building environment $.04/$1.00 actual testing vs. setup
  20. 20. II. Data Tax is Huge : 3. App Dev QA Delays: bugs found late = more code re-work Build QA Env Sprint 3 Sprint 2 Sprint 1 X Build QA Env QA Bug Code 70 60 50 40 30 20 10 0 Cost To Correct 1 2 3 4 5 6 7 Delay in Fixing the bug Software Engineering Economics – Barry Boehm (1981) QA
  21. 21. II. Data Tax is Huge : 3. App Dev full copies cause bottlenecks Old Unrepresentative Data Frustration Waiting
  22. 22. II. Data Tax is Huge : 3. App Dev subsets cause bugs
  23. 23. II. Data Tax is Huge : 3. App Dev subsets cause bugs The Production ‘Wall’ Classic problem is that queries that run fast on subsets hit the wall in production. Developers are unable to test against all data
  24. 24. II. Data Tax is Huge : 3. App Dev Slow Environment Builds: 3-6 Months to Deliver Data Developer Asks for DB Manager approves DBA System Admin Storage Admin Get Access Request system Setup DB Request storage Setup machine Allocate storage (take snapshot)
  25. 25. II. Data Tax is Huge : 3. App Dev Slow Environment Builds Why are hand offs so expensive?
  26. 26. II. Data Tax is Huge : 3. App Dev Slow Environment Builds: culture of no DBA Developer
  27. 27. II. Data Tax is Huge : 3. App Dev Slow Environment Builds Never enough environments
  28. 28. II. Data Tax is Huge : 3. App Dev What We’ve Seen Five examples of application Data Tax impact • • • • • Inefficient QA: Higher costs QA Delays : Increased re-work Sharing DB : Bottlenecks Subset DB : Bugs Slow Environment Builds: Delays
  29. 29. II. Data Tax is Huge : 4. Business Ability to capture revenue • Business Intelligence – ETLs or data warehouse refreshes • Old data = less intelligence • Less Intelligence = Missed Revenue • Business Applications – Delays on getting applications that generate the revenue
  30. 30. II. Data Tax is Huge : 4. Business Intelligence 1pm noon 10pm 2011 2012 2013 2014 2015 8am
  31. 31. II. Data Tax is Huge : 4. Business
  32. 32. II. Data Tax is Huge : 4. Business Magnitude of business impact • $27 Billion Revenue • $1 Billion IT spend • $850M development staff • $110M IT Ops • $ 40M storage Revenue Dev IT Ops Storage 0 5000 10000 15000 20000 25000 30000
  33. 33. II. Data Tax is Huge : 4. Business More important Revenue Dev IT Ops More obvious Storage 0 5000 10000 15000 20000 25000 30000
  34. 34. II. Data Tax is Huge : How Big is the Data tax? Measure before and after Delphix w/ Fortune 500 : Median App Dev throughput increase by 2x
  35. 35. II. Data Tax is Huge : How big is the data tax ? • 10 x Faster Financial Close – Facebook 21 days down to 2 • 9x Faster BI refreshes – Bank of West from 3 weeks per refresh to 3x a week – Weekly refreshes down to 3 times a day • 2x faster Projects – – – – Informatica KLA-Tencor (5 x increase) Presbyterian Health NY Life • 20 % less bugs – Stubhub
  36. 36. II. Data Tax is Huge : Public Customer Quotes • "Delphix allowed us to shrink our project schedule from 12 months to 6 months.” – BA Scott, NYL VP App Dev • "It used to take 50-some-odd days to develop an insurance product, … Now we can get a product to the customer in about 23 days.” – Presbyterian Health • “Can't imagine working without Delphix” – Ramesh Shrinivasan CA Department of General Services
  37. 37. Part III. Companies unaware of the Data Tax
  38. 38. III. Companies unaware of the Data Tax Isn’t there technology that does that? Why do I need Delphix? Why do I need an iPhone ?
  39. 39. III. Companies unaware of the Data Tax Nobody does what we do • • • • • • NetBackup Flashback DataGuard GoldenGate SnapClone FlexClone =
  40. 40. Three Core Parts Production Instance 1 Copy Sync Snapshots Time Flow Purge Storage File System 2 Clone (snapshot) Compress Share Cache Storage Development Instance 3 Mount, recover, rename Self Service, Roles & Security Rollback & Refresh Branch & Tag
  41. 41. Netapp 1 SnapManager Repository DBA Snap Manager Protection Manager RMAN Repository Snap Manager tr-3761.pdf Flex Clone Production Snap Drive Snap Mirror Storage Admin Development Netapp Frankenstein
  42. 42. 3 Oracle EM 12c Snap Clone EM 12c Test Master Source Instance ? instance Profile • • • • • • • • Register Netapp or ZFS with Storage Credentials Install agents on a LINUX machine to manage the Netapp or ZFS storage. Register test master database Enable Snap Clone for the test master database Set up a zone – set max CPU and Memory and the roles that can see these zones Set up a pool – a pool is a set of machines where databases can be provisioned Set up a profile – a source database that can be used for thin cloning Set up a service template – init.ora values Oracle Frankenstein Linux Clone instance Agents Pool Template Zone ZFS or NetApp
  43. 43. EM 12c: Snap Clone Production Flexclone Development Flexclone Netapp Snap Manager for Oracle Other technology? • Prove it • Bake off • Customer refs
  44. 44. III. Companies unaware of the Data Tax #1 Biggest Enemy IT departments believe – – – – – have best processes that exist have latest and greatest technology nothing better exists Just the way it is Data management is a drop in the bucket of overall issues “The status quo is pre-ordained failure”
  45. 45. The Phoenix Project • • • • • IT bottlenecks Setting Priorities Company Goals Defining Metrics Fast Iterations IT version of “The Goal” by E. Goldratt • “Any improvement not made at the constraint is an illusion.” • What is the constraint ? • The DBAs and environments. • “One of the most powerful things that IT can do is get environments to development and QA when they need it”
  46. 46. III. Companies unaware of the Data Tax • Ask Questions – Delphix: we provision environments in minutes for almost not extra storage. – Customer: We already do that – Delphix: How long does it take a developer to get an environment after they ask ? – Customer: 2-3 weeks – Delphix: we do it in 2-3 minutes No other product does what we do
  47. 47. III. Companies unaware of the Data Tax • How to enlighten companies ? Ask for metrics – – – – – Batch window size for ETL How new (old) is their BI data? Number: app projects per year How long does it take a developer to get a DB copy? How long does it take QA to setup an environment • How long to rollback • How long to refresh • How many time do they run a QA cycle – How old is data in QA and DEV
  48. 48. Summary I. Companies pay a Data Tax II. Data Tax is huge III. Companies unaware of the Data Tax
  49. 49. Life with Delphix
  50. 50. Typical Architecture Production Dev, QA, UAT Reporting Instance Instance Instance Instance Instance Instance Instance Instance Database Database Database Database Database File system File system File system File system File system File system File system File system File system File system File system Backup Instance Instance Instance File system File system
  51. 51. With Delphix Production Instance Instance Database File system Dev & QA Instance Instance Instance Instance Instance Instance Database Database Database Reporting Backup Instance Instance Database Database
  52. 52. DevOps With Delphix 1. 2. 3. 4. 5. Efficient QA: Low cost, high utilization Quick QA : Fast Bug Fix Every Dev gets DB: Parallelized Dev Full DB : Less Bugs Fast Builds: Culture of Yes
  53. 53. Impact on bottom line • IT Capital expense – 90 % less storage – 30 % less licenses, floor space, servers , power • IT Operational & Personnel – 1000s of hours down to 10 • Application Development – 2x project output at higher quality • Business – 9x more Fresh BI data – 2x Faster time to market and higher quality – Increase revenue
  54. 54. Summary • Data tax IS the big gorilla. • Delphix Agile data is small & fast • With Delphix , Deliver projects – Half the Time – Higher Quality – Increase Revenue
  55. 55. kylehailey.com/delphix Use Cases What is Delphix Competition
  56. 56. What is Delphix
  57. 57. Three Physical Copies Three Virtual Copies Delphix
  58. 58. Install Delphix on x86 hardware Intel hardware
  59. 59. Allocate Any Storage to Delphix Allocate Storage Any type Pure Storage + Delphix Better Performance for 1/10 the cost
  60. 60. One time backup of source database Production Supports Instance Instance Instance Database File system Upcoming
  61. 61. DxFS (Delphix) Compress Data Production Instance Instance Instance Database File system Data is compressed typically 1/3 size
  62. 62. Incremental forever change collection Production Instance Instance Instance Database Changes Time Window File system • Collected incrementally forever • Old data purged
  63. 63. Source Full Copy Source backup from SCN 1
  64. 64. Snapshot 1 Snapshot 2
  65. 65. Snapshot 1 Snapshot 2 Backup from SCN
  66. 66. Snapshot 1 Snapshot 3 Snapshot 2
  67. 67. Snapshot 3 Snapshot 2 Drop Snapshot 1
  68. 68. Cloning Production Instance Instance Instance Instance Instance Database Database Time Window File system
  69. 69. Typical Architecture Production Instance Instance Database File system File system QA UAT Instance Instance Instance Instance Instance Instance Database Database Database File system File system File system File system File system File system File system Development
  70. 70. With Delphix Production Instance Instance Database File system Development QA UAT Instance Instance Instance Instance Instance Instance Database Database Database
  71. 71. Delphix Use Cases 1. 2. 3. 4. 5. Fast, Fresh, Full Free Branching Federated Self Serve
  72. 72. Fast, Fresh, Full Source Development VDB Instance Instance Time Window
  73. 73. Free Instance Source Instance Instance Instance gif by Steve Karam
  74. 74. Branching Source Instance branching Dev Instance checkout Source QA branched from Dev Instance bookmark
  75. 75. Federated Cloning
  76. 76. Federated Source1 Instance Source1 Instance Source2 Instance Instance
  77. 77. “I looked like a hero” Tony Young, CIO Informatica
  78. 78. Self Service
  79. 79. Use Cases 1. Development Acceleration 2. Quality 3. BI
  80. 80. DevOps
  81. 81. DevOps With Delphix 1. 2. 3. 4. 5. Efficient QA: Low cost, high utilization Quick QA : Fast Bug Fix Every Dev gets DB: Parallelized Dev Full DB : Less Bugs Fast Builds: Culture of Yes
  82. 82. 1. Efficient QA: Lower cost Build QA Test Build Time B u i l d T i m e QA Test 1% of QA time was building environment $.99/$1.00 actual testing vs. setup
  83. 83. Rapid QA via Branching
  84. 84. 2. QA Immediate: bugs found fast and fixed Build QA Env Sprint 2 Sprint 1 X Q A Build QA Env Q A Sprint 3 Bug Code QA QA Sprint 2 Sprint 1 X Bug Code Sprint 3
  85. 85. 3. Private Copies: Parallelize gif by Steve Karam
  86. 86. 4. Full Size DB : Eliminate bugs
  87. 87. 5. Self Service: Fast, Efficient. Culture of Yes!
  88. 88. Quality 1. Forensics 2. Testing 3. Recovery
  89. 89. 1. Forensics: Investigate Production Bugs Development Instance Instance Time Window Anomaly on Prod Possible code bug At noon yesterday Spin up VDB of Prod as it was during anomaly
  90. 90. 2. Testing : Rewind for patch and QA testing Prod Development Instance Instance Time Window Time Window
  91. 91. 2. Testing: A/B Instance Test A with Index 1 Instance Instance Time Window • Keep tests for compare • Production vs Virtual – invisible index on Prod – Creating index on virtual • Flashback vs Virtual Test B with Index 2
  92. 92. 3. Recovery: Surgical recover of Production Source Development Instance Instance Spin VDB up Before drop Time Window Problem on Prod Dropped Table Accidently
  93. 93. 3. Recovery Surgical or Full Recovery on VDB Dev1 VDB Source Instance Instance Dev2 VDB Branched Source Time Window Dev1 VDB Time Window Instance
  94. 94. 3. Recovery: Virtual to Physical Source VDB Instance Instance Spin VDB up Before drop Time Window Corruption
  95. 95. 3. Recovery
  96. 96. Business Intelligence
  97. 97. ETL and Refresh Windows 1pm noon 10pm 8am
  98. 98. ETL and DW refreshes taking longer 1pm noon 10pm 2011 2012 2013 2014 2015 8am
  99. 99. ETL and Refresh Windows 6am 8am 10pm 10pm 1pm noon 8am 10pm 2011 2012 2013 2014 2015 noon 9pm 8am
  100. 100. ETL and DW Refreshes Prod DW & BI Instance Instance Data Guard – requires full refresh if used Active Data Guard – read only, most reports don’t work
  101. 101. Fast Refreshes • Collect only Changes • Refresh in minutes Prod Instance BI DW Instance Instance ETL 24x7
  102. 102. Temporal Data
  103. 103. Oracle 12c
  104. 104. 80MB buffer cache ?
  105. 105. 200GB Cache
  106. 106. with Latency Tnxs / min 5000 300 ms 1 5 10 20 30 60 100 200 Users 1 5 10 20 30 60 100 200
  107. 107. Latency Tnxs / min 8000 600 ms 1 5 10 20 30 60 100 200 Users 1 5 10 20 30 60 100 200
  108. 108. $1,000,000 $6,000
  109. 109. Data Center Migration : clone migration 5x Source Data Copy < 1x Source Data Copy
  110. 110. Data Center Migration : clone migration + source S S 5x Source Data Copy < 2 x Source Data Copy
  111. 111. Data Center Migration : clone migration + source S C C C 5x Source Data Copy C S V V V < 1 x Source Data Copy V
  112. 112. Consolidation Without Delphix Active Active With Delphix Idle Active Idle Active

×