Presented at Richmond SQL Server Users group, this presentation provides an introduction to Continuous Delivery principles, practices, tools and how to apply these to SQL Server Database. It also presents a case study and lessons learned from adopting these practices on a real project.

1. Continuous Database Delivery
David P. Moore
7/12/2018

2. 2
• Manager at CapTech Consulting
• Software/data engineer with 20+ years of experience
in a variety of roles, developing applications and data
solutions.
• Passionate about continuous improvement, seek
ways to make agile practices practical.
• Have been using SQL Server (as a Developer) since
version 7
• Play guitar (acoustic)
• I practice Karate with my son, and I’m an
Intermediate Brown Belt
About Me

3. • What is Continuous Delivery?
• A brief history of Continuous Delivery
• Continuous Delivery Principles
• Continuous Delivery Tools and Practices
• Challenges with Database Continuous Delivery
• Case Study
• Demo
Agenda3

4. 4
“the ability to get changes …
into production, or into the hands of users,
safely and quickly in a sustainable way…
by ensuring our code is always in a deployable state”
Jezz Humble
https://www.continuousdelivery.com/
What is Continuous Delivery?

5. A brief history…

6. 6 The old days…
https://en.wikipedia.org/wiki/Waterfall_model
Fred Brooks:
Large Systems Development = Tar pit

7. 7
Many fundamental agile concepts originated with Extreme Programming (XP),
by Kent Beck, Martin Fowler and others in the late 1990’s and the beginning of
the agile movement
Some XP Practices:
• Automated Unit Tests
• Integrate Often
• Make frequent small releases
http://www.extremeprogramming.org
Extreme Programming (1999)

8. 8
Principle #1:
“Our highest priority is to
satisfy the customer
through early and
continuous delivery of
valuable software.”
http://agilemanifesto.org/principles.html
Agile Manifesto (2001)

9. 9
Practices:
• Maintain a Single Source Repository
• Automate the Build
• Make Your Build Self Testing
• Commit to Mainline every day
• Every Commit Should Build the Mainline on an
Integration Machine
• Automate Deployment
Continuous Integration – Martin Fowler (2006)
https://www.martinfowler.com/articles/continuousIntegration.html

10. 10
• John Allspaw & Paul Hammond
• Explained how at Flickr the Dev & Ops teams worked together to achieve
multiple releases per day to production
• Many who heard it described it as an “Aha” moment
Velocity Conference – “10+ Deploys Per Day” (2009)
https://www.youtube.com/watch?v=LdOe18KhtT4

11. 11
Jez Humble and David Farley write the definitive guide on
Continuous Delivery
Continuous Delivery Book (2010)

12. 12
A novel that details the experience of Bill Palmer,
Head of Operations at Parts Unlimited, and his
crash course into adopting DevOps so that the
company can survive.
Introduced the Three Ways:
1. Systems Thinking (increase flow)
2. Amplify Feedback Loops
3. Culture of Continual Experimentation &
Learning
The Phoenix Project (2013)

13. 13
Culmination and collaboration and of thought leaders to bring many of these
ideas together in one comprehensive guide.
DevOps Handbook (2016)

14. 14
“DevOps is the union of people, process, and products to enable continuous
delivery of value to our end users.”
Donovan Brown, Microsoft Principal DevOps Manager
http://donovanbrown.com/post/what-is-devops
So what is DevOps?
https://en.wikipedia.org/wiki/DevO
ps

15. 15
“the ability to get changes …
into production, or into the hands of users,
safely and quickly in a sustainable way…
by ensuring our code is always in a deployable state”
Jezz Humble
https://www.continuousdelivery.com/
What is Continuous Delivery?

16. Continuous Delivery Principles

17. 17
#1: Automate Almost Everything
Computers Perform Repetitive Tasks, People Solve Problems

18. 18
#2: Keep Everything in Version Control

19. 19
#3: If It Hurts, Do It More Frequently:
Bring the Pain Forward

20. 20
#4: Build Quality In

21. 21
#5: Work in Small Batches

22. 22
#6: Everyone is Responsible

23. 23
#7: Relentlessly Pursue Continuous
Improvement

24. Continuous Delivery Pipelines,
Tools and Practices

25. 25
Commit
Change and
Push to
Version
Control
Trigger Build
Deploy to
Test
Environment
Run Tests
Approve
Release
(Manual)
Deploy to
Production
Example Simple Deployment Pipeline

26. 26
Select and use a Configuration Management tool for storing and versioning:
• Code
• Artifacts (Packages)
• Documentation
Centralized (SVN, TFVC) vs. Distributed (Git, Mercurial)
On-prem vs. Cloud (Github, Bitbucket, TFVS)
Configuration Management and Version Control

27. 27
Somewhat controversial, but proponents of
Continuous Delivery recommend not using feature
branching or long-lived branches
Instead the recommendation is to work off of the
mainline or master branch and make frequent small
commits
If branching is used, the recommendation is to
merge frequently – daily or even more frequently
What about Branching?

28. 28
Automate the Build and Deployment processes
On-Prem:
• Jenkins
• Bamboo
• Team City
• Team Foundation Server
Cloud:
• Travis CI
• Gitlab
• Visual Studio Team Services
Continuous Integration and Deployment Tools

29. 29
Continuous Integration: Developers commit and integrate their changes to
master branch (trunk) daily, triggering automated builds and automated tests
Continuous Deployment: Every change that passes automated tests is
automatically deployed to production
Continuous Integration vs. Continuous Deployment
Continuous
Integration
Continuous
Delivery
Continuous
Deployment

30. 30
• Deployment – This is a technical activity - the
process of making a change or set of changes to
the production environment
• Release – This is a business activity or decision –
whether or not to make available that set of
changes to users
Deploy vs. Release

31. 31
ITIL v.3* supports the concept of Standard Changes
Standard Changes are changes that are pre-approved, repeatable and low
risk
Many change management systems have some sort of API interface so that
Change Requests can be automated
*http://itsmtransition.com/2014/03/name-difference-standard-normal-changes-itil/
What about Change Management?

32. Challenges with Database
Continuous Delivery

33. 33
Typical (non-database) Coding to Deployment Process
Database Coding to Deployment Process
Deploying Database Code is Different
Code compile
.exe
.jar,
.dll
deploy
SQL
code
compile/
deploy

34. 34
Traditional Applications can simply replace the existing objects
Databases have to alter existing objects and maintain state
What about state?
v.2v.1
v.1 v.2

35. 35
There are primarily two ways of approaching Configuration Management for
Database Development:
State Based (object based)
• Each database object stored in its own file and added to the repository
• State of objects are compared at deployment time to generate a script with all
changes
Migration Based (aka DBdeploy pattern)
• Each change to the database is saved as a separate script
• Scripts are deployed in specific order, and the database maintains history of which
scripts have been applied
Both approaches have their own challenges and pros and cons
Configuration Management Strategies: State-Based vs. Migration
Based

36. 36
Microsoft (in partnership with Redgate) has provide tools to enable both
kinds of data development in SQL Server
• State Based : SQL Server Data Tools (SSDT)
• Migration Based: Ready Roll (with Visual Studio Enterprise 2017)

37. 37
ReadyRoll is one of
the SQL Server
Project Types in
Visual Studio 2017
Enterprise
ReadyRoll – Migration Based Database Project

38. 38
Reference Data that is reasonably small can be checked into version control
Unit Tests should have their own data contained inside the tests
A best practice is to create and maintain a set of curated test data that
provides an accurate representation of your actual production data
For other types of testing such as performance testing there will be need to be
alternative solutions, such as backup/restore
What about the Data?

39. 39
Another challenge particular to Database Continuous Delivery is Production
Drift
This is caused when a change is made manually in production (or any other
environment) that is not checked back into version control.
For example:
• Indexing
• Production hot fixes
• Security changes
There is no easy solution to this – there needs to be discipline to ensure that
all changes are checked back into version control, or take away all production
access and enforce all changes through the deployment pipeline
Production Drift

40. 40
After a database change is made, it is difficult if not impossible to rollback the
change
Instead you have to plan to rollforward with a script that will undo the changes
that were made
This requires more preparation and planning, and depending on the risk you
may chose to handle it differently
Database transactions can be used to do an auto rollback to deal with
deployment errors
What about Deployment Rollbacks?

41. 41
Each developer should have his/her own private
sandbox database to work in, isolated from all other
developers
Ideally there should be an easy to use automated
process to set up a new sandbox database
Best Practice: Sandbox Database per Developer

42. 42
Should have at least 3 environments*, although 5 is preferable
• Dev (Local/Sandbox)*
• Dev Integration
• QA*
• Pre-Prod / Staging
• Production*
Others:
• Performance Testing
• Manual/Explorative Testing
Environments

43. Case Study

44. 44
CapTech worked with a Fortune 500 insurance company to
build next generation data capabilities, with real-time data
integration feeding into a data warehouse solution
Sourced raw data from multiple operational systems databases
using both real-time replication, as well as database copy
(backup/restore)
Target system was SQL Server 2016
Data Warehouse used a custom ELT framework with metadata-
driven code generation
Case Study – Agile Data Warehouse

45. 45
• 1 to 3 Agile Teams with 5-6 developers
• Implemented Scrum (2-week sprints) as well as Scaled
Agile Framework (SaFE)
• Adopted Extreme Programming practices, including
Pair-Programming, Collective code ownership,
Automated Testing, Test Driven Development, and
Continuous Delivery
• Team supported production with prod support rotation
Methodology

46. 46
Git chosen for Version Control
Automated unit tests using tSQLt framework
Used On-Prem Team Foundation Server (TFS) for
automated build and release pipeline
• 900+ production releases in a little over a year
Technical Details

47. 47
Example App 1: Security Framework
• 80+ tables
• 900 tests
Example App 2: Data Warehouse
• 100+ tables
• 600+ views
• 1500+ stored procedures
• 200 tests
Example Applications

48. 48
• Agile and XP practices go together
• Automated testing (ideally TDD) is really important, but it is
hard to write tests for legacy code – it is better to refactor
existing code
• Keep releases small
• Branching and object-based configuration management can
be useful for coordination of changes
• Communication is key
Lessons Learned

49. Demo

50. References & Resources

51. 51 Books
Continuous Delivery
Jezz Humble, David Farley
http://amzn.to/1QBJM7k
DevOps Handbook
Gene Kim Jez Humble John Willis
Debois
http://bit.ly/devops-handbook
The Phoenix Project
Gene Kim, George Spafford, Kevin
https://itrevolution.com/book/the-phoenix-

52. 52 Videos
DevOps for Databases
Steve Jones
https://courses.edx.org/courses/course-
v1:Microsoft+DEVOPS200.6x+1T2018a/course/
Continuous Delivery
Jezz Humble
http://bit.ly/jez-cd-training
Martin Fowler – Continuous Delivery https://www.youtube.com/watch?v=aoMfbgF2D_4
Grant Fritchey - Extending DevOps to
Server
https://www.youtube.com/watch?v=bqhv0ukdpJM
Database DevOps with Redgate https://www.youtube.com/watch?v=N1lRaBxnfBY
SQL Server Data Tools for https://www.youtube.com/watch?v=9WH_r7M4QhM

53. Questions?