Building a General PDE Solving Framework with Symbolic-Numeric Scientific Mac...
Delphix and DBmaestro
1. Thank you for joining us, the webinar will start at:
10:00 Pacific / 12:00 Central / 13:00 East / 18:00 UK Time
Manage Databases like Codebases
2. Before we start
• You will be on mute for the duration of the event
• We are now talking so please type a message
in the Questions box in the Control Panel if you can’t
hear us (please check your speakers and
GoToWebinar audio settings first)
• There will be a Q+A session at the end but please
feel free to type your questions in the Questions
box in the Control Panel in advance
• A recording of the full webinar will be put up online
3. Presenters
Tim O'Brien
• Analyst, CTO
• Gleanster Research
Yaniv Yehuda @yanivyehuda
• Co-Founder & CTO at DBmaestro
• Presenter at world-wide conferences: ODTUG, ilOUG
Kyle Hailey @kylehhailey
• Technical Evangelist at Delphix
• Oracle ACE, member of the OakTable Network
4. About Delphix
• Founded in 2008, launched in 2010
• CEO Jedidiah Yueh (founder of Avamar: >$1B revenue))
• Based in Silicon Valley, Global Operations
• 10% of Fortune 500
5. About DBmaestro
• Founded in 2008, launched in 2010
• Founded by Yariv Tabac and Yaniv Yehuda
• Headquartered in Israel, Global Operations
7. The Business Need
Copyright@2010, Gartner RAS Core Research
80%
of unplanned
downtime is
due to Change
50%
More than
of this, half is
due to human
errors
40% of changes FAIL Copyright@2008, Juniper Networks, Inc.
9. Industry
Trends
Code in 2014
Distributed& AsynchronousNew Normal
Self-serviceCriticaltoProductivity
IncreasingVariety (fromJavascript,Java, .NET,
Ruby,Python+everything)
Everything is Automated
10. Industry
Trends
Databases in 2014
More thanjustRelational
IncreasingVariety (Oracle,Mongo)
Management? Notas Matureas Code
Oddity
Big Datamarginallymore manageablethan
traditionalRDBMS
“We’restillwaitingforthedatabase…”
11. Code is King
Modern Enterprise
Fully distributed development
Continuous Deployment Pipelines
Everything Automated! - DevOps, PaaS,
IaaS
“Every Company is a Software
Company”
How did we get here?
12. Scale
Development
Open Source @ Scale
Google Chrome, Mozilla Firefox
Android, Apache httpd
Linux*, FreeBSD
Characteristics
Rapid Iterations, Frequent Releases
Numerous Experiments (or Branches)
Thousands of developers
OSS @ Scale established best practices
now in use.
14. @Scale
Everything is
Continuous
Continuous integration and
deployment
Modern set ofTools Integrated with:
Source Control – Perforce, Clearcase, Git
Infrastructure-as-a-Service – Openstack
Platform-as-a-Service – AWS, Heroku
Databases? - Still an Integration Nightmare
15. Case in Point
Example Fortune 500 Relaunch
Re-architecture: More distributed, Move to
a PaaS
Scope: Everything, 2Year Migration Project
Size: 1500 Developers, Several Continents,
$100M+
Results?
Code: Distributed Projects, Continuous
Deployment
Infrastructure: Self-service, On-demand,
Agile
Database: Delays, Manual Builds, Cost
Overruns
16. RootCause
Analysis
Code is Agile
Developers move FAST.
Branches are fungible, disposable, personal.
Deployments managed with DevOps tools
(immediately)
Traditional Databases are Not
Costly to Setup
Often require Dedicated Hardware
Little automation. No Change Management
“The rest of the department is
Virtual but our Databases are
stuck in 1998.”
17. Worst
Practices
(we’reall used to)
Lack of Database Change Management:
Shared Developer Databases Results in Contention
Costly, Slow Environment Provisioning
Testing and Staging Not Synchronized with
Production
Drag on Productivity and Efficiency
Opportunity lost
“We’ve accepted that the databases causes
delays”
18. What We’ve Seen
1. QA: Expensive, Slow
2. Development: Bottlenecks & bugs
3. Provisioning: Delays
19. 1. QA: Expensive, Slow : Long Build times
Build Time
96% of QA time was building environment
$.04/$1.00 actual testing vs. setup
Build
Build Time
QA
Test
QA
Test
Build
20. 1. QA: Expensive, Slow: bugs found late
Build QA Env QA Build QA Env QA
Sprint 1 Sprint 2 Sprint 3
Bug CodeX
0
10
20
30
40
50
60
70
1 2 3 4 5 6 7
Delay in Fixing the bug
Cost
To
Correct
Software Engineering Economics – Barry Boehm (1981)
Build QA Env QA
30. One time backup of source database
Database
Production
Instance
File systemFile system
31. DxFS (Delphix) Compress Data
Database
Production
Instance
Data is
compressed
typically 1/3
size
File system
32. Incremental forever change collection
Database
Production
Instance
File system
Changes
• Collected incrementally forever
• Old data purged
Time Window
File system
37. What We’ve Seen With Delphix
1. QA: Low cost, fast bug detection
2. Development: Parallelized, less bugs
3. Provisioning: Fast, Culture of Yes
38. 1. QA Efficient : Lower cost
B
u
i
l
d
T
i
m
e
QA Test
1% of QA time was building environment
$.99/$1.00 actual testing vs. setup
Build Time
QA Test
Build
39. 1. QA Fast : bugs found fast and fixed
Sprint 1 Sprint 2 Sprint 3
Bug CodeX
QA QA
Build QA
Env
Q
A
Build QA
Env
Q
A
Sprint 1 Sprint 2 Sprint 3
Bug
Cod
e
X
43. What We’ve Seen With Delphix
1. QA: Low cost, fast bug detection
2. Development: Parallelized, less bugs
3. Provisioning: Fast, Culture of Yes
But …
How do we manage database changes as code
changes and automate deployment of changes?
44. Dealing with Risk
Smaller and more focused changes are easier to manage (Agile…)
Automation of repeating tasks lowers risk of (human) error
Development and Operations should work in synergy (DevOps)
45. 45
• More rapid changes
• Reacting quickly to market needs
• Getting ahead of competition
• Fewer changes backed out
• Better collaboration
• Fewer defects
• Ultimately better service
• Happy customers
• Profitability
Why Continuous Delivery?
46. 46
• Team and process
• Version everything
• Automate your tests
• Fix it, properly (no out of process changes!)
• Automate your deployments
Database Continuous Delivery?
48. 48
• The database holds your essential information
• Changes can impact the entire system
• Need to be synchronized with other changes
• Often overlooked
Database is a Key Component
49. 49
• There is more to a database than SQL scripts
– Schema structure
– Code
– Content and meta-content
– Internal dependencies
– …
• Ensure that changes are not made without
authorization
• Ensure no out-of-process changes
Reaching Inside the Database
50. 50
• Old adage but true
• The database is often neglected and therefore can
become the weakest link
• Essential from a compliance point of view
• Should be the strongest link
The Weakest Link In a Chain
51. 51
Challenges…
Un-coordinated Process
Silos in development
Deployment delays…
Out-of-Process Changes
Errors in production
Lack of confidence in automation
Code overrides
Different methodologies
Lack of history
Missing changes
wrong versions
52. 52
Two isolated Processes
Version Control Process Development Process
Check-Out
Script
Modify Script
Get updated
Script from DB
Check-In
Script
Compile
Script
in DB
Debug Script
in DB
?
?
?
?
A
A’
53. 53
X
1.11.1.11.11.21.31.41.51.61.7
Build Once Deploy Many
Int QA Stage Prod
Database Deploy Script
Environment
Re-Base (due to defects)
Dev
Dev
Dev
Model
1.1 1.2
1.2 1.3
1.3 1.4
1.4 1.5
1.5 1.6
1.6 1.7
1.11.11.41.7
1.1 1.2
1.2 1.3
1.3 1.4
1.4 1.5
1.5 1.6
1.6 1.7
1.1 1.2
1.2 1.3
1.3 1.4
1.4 1.5
1.5 1.6
1.6 1.7
Out of Process
Change
X
X
X
X
X
? 1.1.1
X
54. 54
Dealing with challenges…
Coordinated ProcessTraceability
Start in the
Beginning
No Out-of-Process Changes
Impact Analysis
Automation
Task Based Development
Well Defined Processes
55. What is DBmaestro TeamWork
• Database Enforced Change Management
+ Database version control
+ Plugs into the ALM (change request, tickets & work items)
+ Database merge & change impact analysis
• DevOps Solution for databases
+ Baseline aware deployment automation, rollback &
recovery
+ Plugs into release management & Continuous
Delivery
56. How?
• Database version control
– Enforced Check Out/In
– Labels
– Rollback/Undo
– Audit trail reports
• Database impact analysis
– Utilizes version control repository information
– 3-way analysis
• Database deployment automation
– API
– Baselines
– Conflict resolution
– Customized business logic
58. 58
1.11.21.31.41.51.61.7
Build & Deploy On Demand
*
Int QA Stage Prod
Database Deploy Script
Environment* Execute the same script
being executed at the
Stage environment
Re-Base (due to defects)
Dev
Dev
Dev
Model
1.1 1.2
1.2 1.3
1.3 1.4
1.4 1.5
1.5 1.6
1.6 1.7
1.1 1.4
1.4 1.7
1.1.1 1.7
1.1 1.1 1.11.41.7
File Based
Version Control
Out of Process
Change
1.1.11.7 1.1.11.7
59. Safety Net For Deployment Automation
Source vs.
Target
Action
= No Action
≠
?
Source vs.
Baseline
Target vs.
Baseline
Action
= = No Action
≠ = Deploy Changes
= ≠ Protect Target
≠ ≠ Merge Changes
You do not have all
of the information
With Baselines and 3 way
analysis the unknown is
now known
Simple Compare & Sync Baseline aware Deployment
An index exists in Target (Production) but not in source (QA).
What should we do? Drop the index or not?
60. Benefits - Development
• Database change repository
• Follow SCM best practices (Check-Out/Check-In)
• All changes are documented
• Manage who can do what, where, when & why
66. Safe and Efficient Work Process
• Clone relevant number of virtual copies of the Trunk
– Dev Team 1-n, Developer a-z, Hot fix environment, Etc…
– Work in parallel, save time
• Rely on enforced change process
– Know exactly who did what, when and why
• Deployment automation
– Save time, mitigate risk
– Continuous integration, Continuous Delivery
• Deal with conflicts & merge them into the Trunk
• Test before deploy to production
– Pre-prod clone
Growing almost 300% a year
Founded in 2008, launched in 2010
Jedidiah Yueh, President and CEO
Founded Avamar in 1999, sold to EMC in 2006, VP Product Mgmt at EMC
Avamar: >$1B revenue,
150 Employees: HQ in Menlo Park, SF, Boston, DC, London, NY and Atlanta
Growing 250% annually – 130+ customers including 100 Fortune1000 Customers
80% of outages impacting mission-critical services caused by people and process issues thru 2015, with the majority of those outages (50%+) caused by change/configuration/release integration and hand-off issues (Gartner RAS Core Research Note G00208328 Ronni J. Colville, George Spafford [October 27, 2010] – Strategic Planning Assumption(s) “Top Seven Considerations for Configuration Management for Virtual and Cloud )
I’m not sure if you have run into these situations at your organization or
if you can imagine some of these situations
But here are some of the issues we at Delphix are seeing in the industry with the companies we are talking to.
Let’s look at the 5 points in more detail
We talked to Presbyterian Healthcare
And they told us that they spend 96% of their QA cycle time building the QA environment
And only 4% actually running the QA suite
This happens for every QA suite
meaning
For every dollar spent on QA there was only 4 cents of actual QA value
And that 96% cost is infrastructure time and overhead
Because of the time required to set up QA environments
The actual QA tests suites lag behind the end of a sprint or code freeze
Meaning that the amount of time that goes by after the introduction of a bug in code and before the bug is found increases
And the more time that goes by after the introduction of a bug into the code
The more dependent code is written on top of the bug
Increasing the amount of code rework required after the bug is finally found
In his seminal book that some of you may be familiar with, “Software Engineering Economics”, author Barry Boehm
Introduce the computer world to the idea that the longer one delays fixing a bug in the software development life cycle
The more expensive it is to to fix that bug and these cost rise exponentially as delays increase
Not sure if you’ve run into this but I have personally experience the following
When I was talking to one development group at Ebay, they
Shared a single copy of the production database between the developers on that team.
What this sharing of a single copy of production meant, is that whenever a
Developer wanted to modified that database, they had to submit their changes to code
Review and that code review took 1 to 2 weeks.
I don’t know about you, but that kind of delay would stifle my motivation
And I have direct experience with the kind of disgruntlement it can cause.
When I was last a DBA, all schema changes went through me.
It took me about half a day to process schema changes. That delay was too much so it was unilaterally decided by
They developers to go to an EAV schema. Or entity attribute value schema
Which mean that developers could add new fields without consulting me and without stepping on each others feat.
It also mean that SQL code as unreadable and performance was atrocious.
Besides creating developer frustration, sharing a database
also makes refreshing the data difficult as it takes a while to refresh the full copy
And it takes even longer to coordinate a time when everyone stops using the copy to make the refresh
All this means is that the copy rarely gets refreshed and the data gets old and unreliable
To circumvent the problems of sharing a single copy of production
Many shops we talk to create subsets.
One company we talked to , RBS spends 50% of time copying databases
have to subset because not enough storage
subsetting process constantly needs fixing modification
Now What happens when developers use subsets -- ****** -----
The biggest and most pervasive problem we see is slow build times.
In order to set up an database copy for a development environments
Requires submitting a request to management who has to review it
Then if the request is granted, it is passed to the DBA who has to coordinate with the Sysadmin who has to coordinate with the storage admin.
In such a situation it makes sense that copying a large database would take a long time
But even when we talk to someone who uses netapp storage snapshots like Electronics Art and Ariba, they said even using storage snapshot sit took
days to weeks get a database clone copy due to the coordination between DBA, sys admin and storage admin
At many of the customers we talk to provisioning a database clone copy takes weeks or months.
One large global bank quotes us as taking typically 6 months to provision a database clone copy environment.
Requirements: self service for app teams
Requirements: end-to-end automation
Metrics: # people, process, time for delivery
So far we have talked about the weight of infrastructure on app delivery. Of course, to control and manage that infrastructure, firms layer on a large set of bureaucratic processes, change control, approvals, procurement, governance, etc etc. So the operational and organizational hurdles then create an even bigger drag on IT and app development.
Here’s an example from one banking customer.
Once the app developer puts in a request for a new development environment, there’s at least a week long wait for management approvals. Then project DBA work with the sysadmin and storage groups for capacity. If more capacity needs to be allocated, it’s 3 more days. If more needs to be purchased, weeks or months.
If a copy of production data is needed, the process needs to wait on a production DBA, who might be busy with production issues. Recovering the database to a specific point in time and configuration can also take days.
It is very common for two weeks to pass between a developer request and a ready environment. The process can be repeated for multiple environments, for data refreshes, and for integration across multiple systems.
With Delphix, turns stop signs into green lights. Provisioning, refresh, rollback, and data integration happen nearly instantly and do not trigger approvals from production systems or require additional storage. That is why KLA is able to deliver 5 times the output from its SAP teams…
Without Delphix, it’s impossible for organizations to implement the level of agile processes they desire. The management of data, and the bureaucracy of data management, slows things down too much.
Due to the constraints of building clone copy database environments one ends up in the “culture of no”
Where developers stop asking for a copy of a production database because the answer is “no”
If the developers need to debug an anomaly seen on production or if they need to write a custom module which requires a copy of production they know not to even ask and just give up.
I’m not sure if you have run into these situations at your organization or
if you can imagine some of these situations
But here are some of the issues we at Delphix are seeing in the industry with the companies we are talking to.
Let’s look at the 5 points in more detail
In the physical database world, 3 clones take up 3x the storage.
In the virtual world 3 clones take up 1/3 the storage thanks to block sharing and compression
installs an any Intel commodity hardware as a VM
supports Oracle 9.2-12c, standard edition, enterprise edition, single instance and RAC on AIX, Sparc, HPUX, LINUX
support SQL Server
EMC, Netapp, Fujitsu,
Or newer flash storage like
Violin, Pure Storage, Fusion IO etc
Recent benchmark with Delphix and Pure Storage showed that a Flash and Delphix solution is an order of magnitude cheaper than spinning disk and more performant
Delphix does a one time only copy of the source database onto Delphix
Quote from a customer “Delphix GUI is what Oracle Enterprise Manager
would look like if Apple had designed it”
Delphix inter face is user friendly, polished and easy to use
Presbyterian when from 10 hour builds to 10 minute builds
Total Investment in Test Environment: $2M/year
10 QA engineers
DBA, storage team dedicated to support testing
App, Oracle server, storage, backups
Restore load competes with backup jobs
Requirements: fast data refresh, rollback
Data delivery takes 480 out of 500 minute test cycle (4% value)
$.04/$1.00 actual testing vs. setup
For example Stubhub went from 5 copies of production in development to 120
Giving each developer their own copy
Stubhub estimated a 20% reduction in bugs that made it to production
