Kubernetes bietet viel Funktionalität, um Zero-Downtime Deployments durchzuführen. Etwas herausfordernder wird es dann, wenn der Service-Update auch mit einem Datenbank-Schema Update verbunden ist. Nebst den verschiedenen Strategien, um ein Datenbankschema in einem Zero-Downtime-Release auszurollen, lernen Sie in diesem Vortrag, wie das Datenbank-Schema sowie die Deployment-Tools in einem Container Verpackt mit der Applikation ausgerollt werden können. Somit erhalten wir ein einziges, in sich konsistentes, Helm Paket, welches den Service samt Datenbank-Schema ausrollen kann.

1. DB Schema deployment within
Kubernetes Releases
Marc Müller
Principal Consultant
marc.mueller@4tecture.ch
@muellermarc
www.4tecture.ch

3. Slide Download
https://www.4tecture.ch/events/dwx23k8sdbdeployment

4. Agenda
▪ Intro
▪ Evolving Databases
▪ K8s specific deployment
approaches
▪ Autonomous Deployment
▪ Implementing a DB schema
deployment solution
▪ SQL Server Data Tools

5. Intro
DB schema deployment with Kubernetes releases

6. Write Code
Customer use the
code
Write Code
Customer use the
code

7. 100 deployments
per day!

9. Being ready for 100 deployments a day
Fully automated process
▪ Build Automation
▪ Deployment Automation
▪ Test Automation
Small and frequent releases
▪ Reduce Complexity
▪ Daily Business
There is no place like production
▪ Testing in Production
▪ Zero Downtime
▪ Feature Flags

10. What about the DB development?
Database development
is fully integrated
No manual schema
changes
Automated deployment
of schema changes

11. Reality?
Different Teams
DB development not
integrated / manual
Schema mismatch
between dev and prod

12. Challenges
DB schema and code change
belong together
Dry-Run on (production) data
Data Migrations / Reference
Data
Zero Downtime Deployment

13. Evolving Databases
DB schema deployment with Kubernetes releases

14. Database Migrations
New version of the app = different database
schema
Many tools to diff and apply new schemas
Zero-downtime deployments is often a critical
requirement

15. Code First
DB First
Release v2
Deployment Approaches
Deploy
DB Schema
Deploy
Binaries
Prod Environment
v2
Binaries
v1
Prod Environment
v2
Binaries
v2
Release v2
Deploy
Binaries
Deploy
DB Schema
Prod Environment
v1 Binaries
v1
Binaries
v2
Factory
Prod Environment
v2 Binaries
v1
Binaries
v2
Factory

16. Where to put the fallback logic?
Database
▪ Use views / triggers to support old
schema
Advantages
▪ Old code just works during
deployment
Disadvantages
▪ Have a lot of if statement in
database logic
▪ Harder to test
Code
▪ Use factory to determine the
implementation for the current database
version
▪ Couple database version to features /
implementation
Advantages
▪ Code is easier to test
Disadvantages
▪ More complexity in code
▪ Factory / Toggles needed

17. Support Rollback Scenarios
«If you can’t get upgrade right, what leads
you to believe you could get rollback right
as well?” – Buck Hodges
Implement Rollback logic only if needed
▪ DB deployment is often complex and multi-step
▪ Hopefully never used – wasted time for implementation and testing?

18. When to run the migration?
On service startup As part of the deployment
process script
As dedicated jobs within your
application (i.e. k8s jobs)

19. Best Practices
▪ DB Frist deployment mode
▪ Easer to develop
▪ DB Migration is critical – fail fast / don’t deploy binaries
▪ No rollback – forward only
▪ Saves huge effort
▪ PR validation / staging will bring up errors before production deployment
▪ Fully automated process – fast rollout of fixes
▪ Dedicated Deployment Job
▪ Application is self-contained
▪ No dependencies to other deployment scripts
▪ Functionality of target environment

20. Kubernetes specific
deployment approaches
DB schema deployment with Kubernetes releases

21. CD
PR
Classic CI / CD Pipeline
CI
Checkout
Build
App
Run Unit
Test
Build
Dacpac
Publish
Dacpac
Publish
App
Create DB
Deploy DB
Schema
Deploy
App
QA
Deploy DB
Schema
Deploy
App
Pre-Prod
Clone
Prod DB
Deploy DB
Schema
Deploy
App
Prod
Deploy DB
Schema
Deploy
App
CI Type

22. k8s
Pod
Pod
Push Approach
CI Pipeline
CD Pipeline
Container
Registry
SQL DB SQL DB SQL DB
Pod
Pod
Pod
Pod

23. Pull Approach
k8s
Pod
Pod
CI Pipeline
CD Pipeline
Container
Registry
SQL DB SQL DB SQL DB
Pod
Pod
Pod
Pod
Git Repo
(config)
Operator

24. Push vs Pull Approaches
Push Approach
▪ Classical CI/CD
▪ Agent pushes artifacts
to target
▪ Pipeline owns
deployment logic
Pull Approach
▪ GitOps approach
▪ Observe configuration
changes and pull
application artifacts
▪ Application package
owns deployment logic

25. Current Deployment
Pod #1
v1
Pod #2
v1
Pod #3
v1
Pod #4
v1
Rolling Update 1/4
Pod #1
v1
Pod #2
v1
Pod #3
v1
Pod #4
v1
Rolling Update 2/4
Pod #1
v2
Pod #2
v2
Pod #3
v1
Pod #4
v1
Rolling Update 3/4
Pod #1
v2
Pod #2
v2
Pod #3
v2
Pod #4
v1
Rolling Update 4/4
Pod #1
v2
Pod #2
v2
Pod #3
v2
Pod #4
v2
Rolling Update

26. Migration on service start
Call “db.Database.Migrate” at startup
Problems:
▪ Every instance of the service will attempt to migrate
the database
▪ The application has permissions to perform
destructive updates to the database

27. Migration run by deployment scripts
Use Azure Pipelines / GitHub Actions to run a
deployment script before service rollout
Pro:
▪ Single and dedicated DB deployment
▪ Dedicated security principal for schema deployment
Challenges:
▪ Knowledge in pipeline, application has a dependency
to pipelines to run correctly

28. Using Jobs as part of your application
Use Kubernetes jobs and init containers / Helm chart
hooks
Pro:
▪ Dedicated job with dedicated identity / permissions
▪ Part of target environment, no external
dependencies
Challenges:
▪ More complexity

29. Automomous
Deployment
DB schema deployment with Kubernetes releases

30. Autonomous Application Packages
▪ CI/CD pipelines work great for internal services
▪ If an application package is distributed, the schema
deployment should be part of it
▪ Logic from the CI/CD pipeline is moved to the
application package
▪ CI/CD pipelines can be simplified

31. Helm Release
Deployment
Helm Release
App Pod
App
Container
Init
Container
Job
DB Migration Pod
DB Migration
Container
Service Ingress
DB

32. Security Considerations
▪ Strict security boundary between dev/test
and prod
▪ Use dedicated users for each database /
service
▪ Use dedicated users for
▪ Schema deployment with DDL
▪ Application / service with read/write permissions

33. Demo
Kubernetes Rollout

35. Implementing a DB schema
deployment solution
DB schema deployment with Kubernetes releases

36. Create a custom migration runner
▪ Independent (and app specific) tool to
run the DB migration
▪ Developed side-by-side with application
and DB schema
▪ Containerized
▪ Packaged in service deployment

37. Demo
Migration Runner

40. Define a Kubernetes Job
▪ Run your DB migration tool as a
Kubernetes job
▪ Use dedicated service identities with
corresponding permissions on database

41. Demo
Job

43. Use init containers
▪ Use init containers to wait for the
migration to successfully finish
▪ Init container will block the deployment /
execution of new application containers
without a successful deployment
▪ Dedicated permissions needed to monitor
jobs

44. Demo
Init Container

47. Publish single package
▪ Package contains all configurations and
container references to deploy and run
the application
▪ Supports any deployment paradigm /
automated and manual deployment
▪ Ideal solution to distribute your
applications at customer site

48. SQS Server Data Tools
(SSDT)
DB schema deployment with Kubernetes releases

49. SSDT - Characteristic
• SSDT Project Type for relational Database
Development
• Integrated in Visual Studio IDE
• Others: SSMS, Redgate, DDL/DML Scripts
• SSDT Advantages:
IDE
MSBuild
IntelliSense
Validation
Code Base
Consistency
Design
Compare
CI
CD

50. • Officially Supported since VS 2015
• 1:1 Database Representation
• SSDT Deployment / Prerequisites:
SSDT - Characteristic
DB Schema Migrations (Static & Dynamic SQL)
Single Pre- and Post Script Logic
Microsoft.Data.Tools.Msbuild
(NuGet)

51. SSDT - Features
• Build time validation / IntelliSense Support
• Bidirectional Scheme Comparison (SSDT  DB)
• Bidirectional Scheme Synchronization (SSDT  DB)
• Versioned migration and schemes artifact (DACPAC)
• Code-base integration / Change tracking (GIT)

52. Schema Compare
• Schema Compare
• Local Development
• Bidirectional Sync.
• Choose your
favorite IDE
• Prevent data loss:
rename in SSDT

53. Code Analysis
• Standardized Design Patterns
• Code Quality
• Reduce Code
Smells
• Supports Static-/
and Dynamic SQL
• Tables, SP, UDDT,
Views…

54. Developer Workflow
1. Create a Feature Branch
from Development
2. Publish/Deploy (F5)
Database Project
3. Develop Database
changes (Renames have
to performed in SSDT)
4. Perform a Schema
Compare from DB to
Database Project, Sync.
5. Commit > PR > Review
Local
DEV DB
Visual Studio
DB Project Git Repo
→
QA Dump
Prod Dump

55. Demo
SSDT in Visual Studio

56. SSDT is nice, but…
SSDT supports basic script extensibility
▪ Single Pre-Script
▪ Single Post-Script
Enterprise-grade migrations imply
complexity
▪ Extended Script Management is needed
▪ State Tracking of Custom Migrations
▪ «DB Version» Tracking
SSDT can easily be extended….

57. Migration History
▪
▪
▪
▪
▪
▪
▪

58. Our Learnings combined…
Features
• Configurable Setup and
Naming Convention
• Logging / Full Transactional
Scripts
• Custom execution filters
• Fully configurable Extension

59. Q & A
DB schema deployment with Kubernetes releases

60. Recap
▪ Dedicated migration runner outside the
service
▪ Use k8s functionality: jobs and init container
▪ Self-contained package, no additional
deployment logic
▪ Database Development fully integrated into
development process
▪ No manual schema changes in deployment
process

61. Thank you for your attention!
If you have any questions do not hesitate to contact us:
4tecture GmbH Marc Müller
Industriestrasse 25 Principal Consultant
CH-8604 Volketswil
+41 44 508 37 00 marc.mueller@4tecture.ch
info@4tecture.ch @muellermarc
www.4tecture.ch www.powerofdevops.com