Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Database version control and deployment - model or migration scripts

2,525 views

Published on

For the last few years Microsoft and others have been promoting declarative, model-based database development. For many this is the way forward – design the desired state and let software work out the upgrades. Gone are the days of managing endless upgrade scripts and manual deployments.
At the same time, leaders and shakers of our industry including Jez Humble, Pramod Sadalge and Paul Stovell promote an iterative, migration script driven approach asserting that deployment scripts should be tested early and not generated by software. They often assert that a disciplines migrations approach is the only reliable way to achieve database continuous delivery.
So many presenters have opinions that one of the approaches is the “right” way and the other is the “wrong” way. However, like with most complicated problems, the truth is that it depends.
I’ll illustrate the relative strengths and limitations of each approach with a simple scenario. I’ll describe teams and databases that are better suited to a model or a migrations approach, and whether it’s possible to get the best of both worlds.
You’ll leave this session with a better understanding of what options are available to you and which is more likely to work for your team. With so many teams struggling to apply continuous delivery to their databases it’s important to consider whether the problems you are facing are because you have taken an approach which is fundamentally unsuitable for your database.

Published in: Technology
  • Be the first to comment

Database version control and deployment - model or migration scripts

  1. 1. @_AlexYates_ #SQLRelay State-Based or Migrations-Based Database Development? A Review of the Pros and Cons… Alex Yates
  2. 2. @_AlexYates_ #SQLRelay DLM Consultant workingwithdevs.com alex.yates@dlmconsultants.com AlexYates @_AlexYates_
  3. 3. @_AlexYates_ #SQLRelay @_AlexYates_ | #SQLRelay
  4. 4. @_AlexYates_ #SQLRelay Agile Scrum Lean Iterative Continuous Delivery Developers
  5. 5. @_AlexYates_ #SQLRelay Monitoring Deployment Integrity Performance DBAs
  6. 6. @_AlexYates_ #SQLRelay
  7. 7. @_AlexYates_ #SQLRelay Woah! Deployment fail?
  8. 8. @_AlexYates_ #SQLRelay #worksOnMyMachine Woah! Deployment fail?
  9. 9. @_AlexYates_ #SQLRelay A cursor?! You just CAN’T do that?!
  10. 10. @_AlexYates_ #SQLRelay #worksOnMyMachine A cursor?! You just CAN’T do that?!
  11. 11. @_AlexYates_ #SQLRelay Hey, you just dropped my hot-fix!
  12. 12. @_AlexYates_ #SQLRelay #worksOnMyMachine Hey, you just dropped my hot-fix!
  13. 13. @_AlexYates_ #SQLRelay Farm Credit Services of America (FCSA) • 100 person IT team, 14 sub-teams • Database version control inconsistent • Deployment process manual • Delivery was slow and unreliable
  14. 14. @_AlexYates_ #SQLRelay Farm Credit Services of America (FCSA) • Deployments easier to review • Delivery faster and more reliable • Standardised source control and delivery process • Automated manual deployment tasks
  15. 15. @_AlexYates_ #SQLRelay The automated deployment pipeline
  16. 16. @_AlexYates_ #SQLRelay Databases are hard • Schema changes vs existing data • Reference data vs production data • Teamwork and testing • Database drift (change outside process, e.g. production hot-fixes)
  17. 17. @_AlexYates_ #SQLRelay There’s more than one way to skin a cat
  18. 18. @_AlexYates_ #SQLRelay There’s more than one way to skin automate a cat
  19. 19. @_AlexYates_ #SQLRelay V1 V2
  20. 20. @_AlexYates_ #SQLRelay V1 V2 Migrations-based solutions
  21. 21. @_AlexYates_ #SQLRelay V1 V2 State-based solutions
  22. 22. @_AlexYates_ #SQLRelay
  23. 23. @_AlexYates_ #SQLRelay “There's nothing more reliable than keeping track of exactly the scripts you intend to run, and running them, without trying to compare state and guess.”
  24. 24. @_AlexYates_ #SQLRelay “There's nothing more reliable than keeping track of exactly the scripts you intend to run, and running them, without trying to compare state and guess.” Paul Stovell, Octopus Deploy http://docs.octopusdeploy.com/display/OD/SQL+Server+databases
  25. 25. @_AlexYates_ #SQLRelay “As soon as you have multiple changes on a single aspect of an object, ordering and the ability to detect which change needs to be made gets very complicated.”
  26. 26. @_AlexYates_ #SQLRelay “As soon as you have multiple changes on a single aspect of an object, ordering and the ability to detect which change needs to be made gets very complicated.” Gert Drapers, built DataDude https://blogs.msdn.microsoft.com/gertd/2009/06/05/declarative-database-development/
  27. 27. @_AlexYates_ #SQLRelay Migrations vs state http://workingwithdevs.com/delivering-databases-migrations-vs-state/
  28. 28. @_AlexYates_ #SQLRelay Migrations vs state http://workingwithdevs.com/delivering-databases-migrations-vs-state/
  29. 29. @_AlexYates_ #SQLRelay Migrations vs state http://workingwithdevs.com/delivering-databases-migrations-vs-state/
  30. 30. @_AlexYates_ #SQLRelay Migrations vs state http://workingwithdevs.com/delivering-databases-migrations-vs-state/
  31. 31. @_AlexYates_ #SQLRelay Migrations vs state http://workingwithdevs.com/delivering-databases-migrations-vs-state/
  32. 32. @_AlexYates_ #SQLRelay Migrations vs state http://workingwithdevs.com/delivering-databases-migrations-vs-state/
  33. 33. @_AlexYates_ #SQLRelay Migrations vs state http://workingwithdevs.com/delivering-databases-migrations-vs-state/
  34. 34. @_AlexYates_ #SQLRelay 11 2 Create view: kittenTrainersNrLdn Selects only: kittenTrainers.FullName Edit table: kittenTrainers Add column: AcceptsTigerCubs BIT Both work! Same changes, versioned and deployed two ways
  35. 35. @_AlexYates_ #SQLRelay Demo
  36. 36. @_AlexYates_ #SQLRelay 11 2 Create view: kittenTrainersNrLdn Selects only: kittenTrainers.FullName Edit table: kittenTrainers Add column: AcceptsTigerCubs BIT Both work! Same changes, versioned and deployed two ways
  37. 37. @_AlexYates_ #SQLRelay 11 Edit table: kittenTrainers Split column trainerFullName into trainerFirstName and trainerLastName But both suck. A tale of two work items… 2 Edit view: kittenTrainersNrLdn Add column: AcceptsTigerCubs
  38. 38. @_AlexYates_ #SQLRelay
  39. 39. @_AlexYates_ #SQLRelay 0124_edit_view.sql ALTER VIEW kittenTrainersNrLdn AS SELECT fullName, acceptsTigerCubs FROM kittenTrainers
  40. 40. @_AlexYates_ #SQLRelay 0124_edit_view.sql 0125_edit_table.sql ALTER VIEW kittenTrainersNrLdn AS SELECT fullName, acceptsTigerCubs FROM kittenTrainers ALTER TABLE kittenTrainers *FUN WITH STRINGS* END ALTER VIEW kittenTrainersNrLdn AS SELECT firstName, lastName FROM kittenTrainers
  41. 41. @_AlexYates_ #SQLRelay ALTER TABLE kittenTrainers *FUN WITH STRINGS* END ALTER VIEW kittenTrainersNrLdn AS SELECT firstName, lastName FROM kittenTrainers 0124_edit_view.sql 0125_edit_table.sql ! ALTER VIEW kittenTrainersNrLdn AS SELECT fullName, acceptsTigerCubs FROM kittenTrainers
  42. 42. @_AlexYates_ #SQLRelay Conflicts easily missed Changes overwritten, hard to spot Order matters Last script wins The “winding path” problem How do you fix a bug that has been deployed to some environments but not others? The problem with migrations
  43. 43. @_AlexYates_ #SQLRelay V123 kittenTrainersNrLdn.sql CREATE VIEW kittenTrai… AS SELECT fullName FROM kittenTrainers
  44. 44. @_AlexYates_ #SQLRelay V124 kittenTrainersNrLdn.sql CREATE VIEW kittenTrai… AS SELECT fullName, acceptsTigerCubs FROM kittenTrainers V123 kittenTrainersNrLdn.sql CREATE VIEW kittenTrai… AS SELECT fullName FROM kittenTrainers
  45. 45. @_AlexYates_ #SQLRelay V125 kittenTrainersNrLdn.sql CREATE VIEW kittenTrai… AS SELECT firstName, lastName FROM kittenTrainers V124 kittenTrainersNrLdn.sql CREATE VIEW kittenTrai… AS SELECT fullName, acceptsTigerCubs FROM kittenTrainers V123 kittenTrainersNrLdn.sql CREATE VIEW kittenTrai… AS SELECT fullName FROM kittenTrainers
  46. 46. @_AlexYates_ #SQLRelay V125 kittenTrainersNrLdn.sql CREATE VIEW kittenTrai… AS SELECT firstName, lastName FROM kittenTrainers V124 kittenTrainersNrLdn.sql CREATE VIEW kittenTrai… AS SELECT fullName, acceptsTigerCubs FROM kittenTrainers V123 kittenTrainersNrLdn.sql CREATE VIEW kittenTrai… AS SELECT fullName FROM kittenTrainers !
  47. 47. @_AlexYates_ #SQLRelay
  48. 48. @_AlexYates_ #SQLRelay REVISION 123 REVISION 124 kittenTrainers.sql CREATE TABLE kittenTrainers (fullName, acceptsTigerCubs) kittenTrainersNrLdn.sql CREATE VIEW kittenTrainersN… AS SELECT fullName FROM kittenTrainers kittenTrainers.sql CREATE TABLE kittenTrainers (firstName, lastName, acceptsTigetCubs) kittenTrainersNrLdn.sql CREATE VIEW kittenTrainersN… AS SELECT firstName, lastName, acceptsTigerCubs FROM kittenTrainers
  49. 49. @_AlexYates_ #SQLRelay Diff script: DROP COLUMN fullName ALTER TABLE kittenTrainers ADD FirstName NVARCHAR(50), LastName NVARCHAR(50) ALTER VIEW kittenTrainersNrLdn AS …
  50. 50. @_AlexYates_ #SQLRelay Diff script: DROP COLUMN fullName ALTER TABLE kittenTrainers ADD FirstName NVARCHAR(50), LastName NVARCHAR(50) ALTER VIEW kittenTrainersNrLdn AS … !
  51. 51. @_AlexYates_ #SQLRelay Script: DROP COLUMN fullName ALTER TABLE kittenTrai… ADD FirstName, LastName ALTER VIEW kittenTrain… AS … Need to understand your tool  It should be obvious to you that your tool won’t work What is the Plan B / override?  Because one day you’ll need it Test for data loss  Automatically (naturally) The problem with state
  52. 52. @_AlexYates_ #SQLRelay So what is better? State MigrationsVS
  53. 53. @_AlexYates_ #SQLRelay State  Easier (less control)  Better for sprocs/functions  Better for large/distributed teams  Better for frequent changes  Better for dependency nightmares  Drift: rolled back Migrations  More control (harder/needs discipline)  Better for data migrations  Better for small teams  Better for infrequent changes  Better for simple data stores  Drift: ignored VS
  54. 54. @_AlexYates_ #SQLRelay State  Easier (less control)  Better for sprocs/functions  Better for large/distributed teams  Better for frequent changes  Better for dependency nightmares  Drift: rolled back  Better for development Migrations  More control (harder/needs discipline)  Better for data migrations  Better for small teams  Better for infrequent changes  Better for simple data stores  Drift: ignored  Better for automation VS
  55. 55. @_AlexYates_ #SQLRelay Pick appropriate tooling State DevArt Schema Compare DB Maestro Migrations DbUp Flyway Hybrid
  56. 56. @_AlexYates_ #SQLRelay Let’s talk about: https://msdn.microsoft.com/en-us/library/mt204009.aspx
  57. 57. @_AlexYates_ #SQLRelay How does SSDT work? Most scripts do not require manual scripting Manual scripting causes problems
  58. 58. @_AlexYates_ #SQLRelay How does SQL Source Control work? Most scripts do not require manual scripting Manual scripting causes problems Perhaps just these ones? So only write these ones
  59. 59. @_AlexYates_ #SQLRelay How does SSDT work? https://msdn.microsoft.com/en-US/library/hh272704(v=vs.103).aspx https://msdn.microsoft.com/en-us/library/jj889461(v=vs.103).aspx << RefactorLog Pre/Post-Deploy scripts >>
  60. 60. @_AlexYates_ #SQLRelay Let’s talk about: http://www.red-gate.com/sql-source-control/
  61. 61. @_AlexYates_ #SQLRelay How does SQL Source Control work? (This diagram again) We still only need a few upgrade scripts
  62. 62. @_AlexYates_ #SQLRelay How does SQL Source Control work?
  63. 63. @_AlexYates_ #SQLRelay How does SQL Source Control work? https://documentation.red-gate.com/display/SOC5/How+migration+scripts+work c
  64. 64. @_AlexYates_ #SQLRelay Let’s talk about: http://www.red-gate.com/readyroll/
  65. 65. @_AlexYates_ #SQLRelay State  Easier (less control)  Better for sprocs/functions  Better for large/distributed teams  Better for frequent changes  Better for dependency nightmares  Drift: rolled back Migrations  More control (harder/needs discipline)  Better for data migrations  Better for small teams  Better for infrequent changes  Better for simple data stores  Drift: ignored Better for sprocs/functions Better data migrations How does ReadyRoll work?
  66. 66. @_AlexYates_ #SQLRelay How does ReadyRoll work? Programmable objects (Views, stored procedures etc) V125 myproc.sql EXEC sp_rename 'table_foo', 'table_bar‘ Migrations (Tables, reference data etc)
  67. 67. @_AlexYates_ #SQLRelay How does ReadyRoll work? Programmable objects (Views, stored procedures etc) V125 myproc.sql EXEC sp_rename 'table_foo', 'table_bar‘ Migrations (Tables, reference data etc)
  68. 68. @_AlexYates_ #SQLRelay How does ReadyRoll work? Programmable objects (Views, stored procedures etc) V125 myproc.sql EXEC sp_rename 'table_foo', 'table_bar‘ Migrations (Tables, reference data etc) https://documentation.red-gate.com/display/RR1/Programmable+Objects
  69. 69. @_AlexYates_ #SQLRelay But haven’t we been doing hybrid for years?
  70. 70. @_AlexYates_ #SQLRelay ProdTestDev
  71. 71. @_AlexYates_ #SQLRelay ProdTestDev
  72. 72. @_AlexYates_ #SQLRelay ProdTestDev S0
  73. 73. @_AlexYates_ #SQLRelay ProdTestDev S0 S0
  74. 74. @_AlexYates_ #SQLRelay ProdTestDev S0 S0 State Migrations
  75. 75. @_AlexYates_ #SQLRelay Prod Test Dev S0 “Late migrations” S1 Prod Test Dev S0 “Early migrations”
  76. 76. @_AlexYates_ #SQLRelay Early vs Late • Test deploy script early  • Easier to automate  • Developers own deployments (typically) • Optimised deployment scripts (no winding path)  • Easier to intervene  • DBAs own deployments (typically)
  77. 77. @_AlexYates_ #SQLRelay Drift
  78. 78. @_AlexYates_ #SQLRelay
  79. 79. @_AlexYates_ #SQLRelay “When making changes directly on production, you are making a decision that the delay due to poor cycle time is more expensive than the risk of making a mistake.”
  80. 80. @_AlexYates_ #SQLRelay “When making changes directly on production, you are making a decision that the delay due to poor cycle time is more expensive than the risk of making a mistake.” AlexYates, Built this slide
  81. 81. @_AlexYates_ #SQLRelay The bad stuff: • Accidental roll-backs (state) • Failed deployments (migrations) • Environment inconsistency (migrations)
  82. 82. @_AlexYates_ #SQLRelay Drift Strategies to help  Improve cycle time  Strict security policies  Monitor drift (play with DDL triggers)  Redgate DLM Dashboard (free) https://www.simple-talk.com/sql/database-administration/database-deployment-the-bits-database-version-drift/
  83. 83. @_AlexYates_ #SQLRelay Drift Strategies to help  Improve cycle time  Strict security policies  Monitor drift (play with DDL triggers)  Redgate DLM Dashboard (free) http://www.red-gate.com/products/dlm/dlm-dashboard/
  84. 84. @_AlexYates_ #SQLRelay Demo
  85. 85. @_AlexYates_ #SQLRelay Farm Credit Services of America (FCSA) • Standardised DLM processes • FCSA delivering much more efficiently • The FCSA model being rolled out at FCMA (sister org) • The team all have excellent CV’s
  86. 86. @_AlexYates_ #SQLRelay
  87. 87. @_AlexYates_ #SQLRelay Siloed sparrows suck at DLM
  88. 88. @_AlexYates_ #SQLRelay Heated hippos are closed minded
  89. 89. @_AlexYates_ #SQLRelay State  Easier (less control)  Better for sprocs/functions  Better for large/distributed teams  Better for frequent changes  Better for dependency nightmares  Drift: rolled back  Better for development Migrations  More control (harder/needs discipline)  Better for data migrations  Better for small teams  Better for infrequent changes  Better for simple data stores  Drift: ignored  Better for automation VS Clever people consider options
  90. 90. @_AlexYates_ #SQLRelay State DevArt Schema Compare DB Maestro Migrations DbUp Flyway Hybrid Hybrid tools are awesomeTools are pretty good
  91. 91. @_AlexYates_ #SQLRelay Prod Test Dev S0 “Late migrations” S1 Prod Test Dev S0 “Early migrations” It’s a balance, not a choice
  92. 92. @_AlexYates_ #SQLRelay Hunt his smug face - make pain on it
  93. 93. @_AlexYates_ #SQLRelay This stuff makes a big difference http://www.codeaperture.io/2016/09/13/how-redgate-helped-define-our-process/
  94. 94. @_AlexYates_ #SQLRelay Questions?
  95. 95. @_AlexYates_ #SQLRelay DLM Consultant workingwithdevs.com alex.yates@dlmconsultants.com Alex Yates @_AlexYates_
  96. 96. @_AlexYates_ #SQLRelay PREMIER SPONSOR GOLD SPONSORS SILVER SPONSORS BRONZE SPONSORS SUPPORTERS
  97. 97. @_AlexYates_ #SQLRelay Please give us your feedback: sqlrelay.co.uk/feedback Thank you
  98. 98. @_AlexYates_ #SQLRelay Image sources Author Source Information Chiltepinster Wikimedia Commons Mocking Bird Argument.jpg – Wikimedia Commons. This file is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported license. Source on Wikimedia Commons: “Own work” Bit Boy Flickr The elephant in the room – Flickr. This file is licensed under the Creative Commons Attribution 2.0 Generic license. Nils Rinaldi Flickr Hippo fight 2/3 – Flickr. This file is licensed under the Creative Commons Attribution 2.0 Generic license. My own collection Taken by/property of Alex Yates Kitten, “There’s more than one way to skin a cat!” Memegenerator.net Memegenerator.net I don’t always edit database. Content designed to be shared and delivered with credit to memegenerator.net. Ctrl.Alt.Design ctrla.lt Social Media share icons

×