Delivering Changes for Applications and Databases

Miguel Alho
Miguel AlhoTechnical Lead at Dixons Crphone
Delivering changes for
applications and databases
“The beginning of a beautiful friendship”
Eduardo Piairo
Miguel Alho
Delivering changes for applications and databases
@EdPiairo
https://pt.linkedin.com/in/jesuspiairo
eduardopiairo@outlook.com
http://www.eduardopiairo.com/
Eduardo Piairo, DBA & DevOps @ Celfinet
Delivering changes for applications and databases
@MytyMyky
https://pt.linkedin.com/in/miguelalho-982331
alho@miguelalho.com
http://www.miguelalho.pt/
Miguel Alho, Software Engineer @ Faro Europe
@DevOpsPorto
Is Coming
Delivering changes for applications and databases
Source
Control
Continuous
Integration
Continuous
Delivery
Database
+
Application
What’s so special about the database?
What’s so special about the database?
DLM – Database Lifecycle Management
Data management and data migration
Data monitoring
Data recovery
Typical problems we want to fix
• Databases are out of pace with application development
• Lack of traceability of database changes (changes history)
• Manual databases processes prevent the CI and CD utilization
• Lack of testability
• Database setup time for new environments
• Bugs in production
Typical problems we want to fix
Databases becoming a
bottleneck in an agile
delivery process
Typical problems we want to fix
Databases becoming an
easy target to blame
Delivering Changes for Applications and Databases
Why database automation?
• Enable control over database development
• Increase speed of response to change
• Keep a versioned “history” of database states and changes
• Reliability of the release process
• Increase release frequency though repeatability of processes
• Reduce time spent fixing bugs - automated tests
• Remove/reduce human intervention in the release process
Why database automation?
Fearless database changes
1st step: Source Control
• First step in your database deployment pipeline
• Traceability through change history
• SQL as documentation
• Shared code-base and shared process
• Enforceable standards to reduce conflicts
1st Step: Source Control
Fundamental resource:
SQL Script
Migrations-based approach vs State-based approach
• State based solutions
• Script represents the current database state
• Your source of truth is how the database should be
• Migrations based solutions
• Script represents a migration
• Migration represents how to transition to the next database version
• Your source of truth is how the database should change
Flyway
• Flyway – “Database Migrations made Easy”
http://flywaydb.org/
• Open source database migration tool
• Flyway command-line enabled
Simplicity: easy to setup, no need to install
Zero dependencies (java + jdbc)
Scripts are written in SQL
Flyway – How does it work?
• Flyway commands:
• migrate, clean, info, validate, baseline, repair
• Metadata table:
• Used to track the state of the database
• If the database is empty, flyway won’t find it and will create it
• Flyway scans the migrations directory and check migrations against the metadata table
• Migrations are sorted based on their version number and applied in order
Version Control System
Two components:
Database scripts repository
Flyway – database migrations
Rule 1: Script version (timestamp)
Rule 2: Operation type
Rule 3: Object type
Rule 4: Object name
Example: V20160728.1100__Create_TB_MyTable.sql
Demo
How about applications?
It takes two to tango
Database and Application change alignment
● Database schemas should evolve along with application evolution
○ deploy app => deploy changes / migrations
● (Integration) Test app and db changes together
○ should pass tests while developing,
○ should pass tests while promoting to new environments
● Ideally, one application per database
● Pipeline deployment only!
Integration test at the database layer
● Use TDD techniques
● Limit database integration tests
to data layer
○ faster tests!
○ validate sql and mappings
● Always test with the latest
schema
DatabaseApplication
DatabaseApplication
Demo
https://github.com/MiguelAlho/Purchases-DbMigration-sample
What we just saw
Database and Application change alignment
Fearless database changes
Challenges
• Challenge #1: managing multiple branches
• Manage merge conflicts
• The default branch is the main/master
branch and all the merges for this branch
must be done with caution
Challenges
• Challenge #2: Script expiration date
• Should the migration script have an
expiration date?
• Sometimes, in the next day, the migration
script no longer makes sense
Challenges
• Challenge #3: migration scripts stack
• How can I manage the dependencies
between different databases?
Challenges
• Challenge #4: developer sandbox
• How to supply a disposable development
environment where the database is
included?
What’s Next?
Links
● https://github.com/MiguelAlho/Purchases-DbMigration-sample
https://flywaydb.org/
● http://dbup.github.io/
● http://www.red-gate.com/products/sql-development/sql-compare/
● https://octopus.com/
○ http://docs.octopusdeploy.com/display/OD/Deploying+ASP.NET+Core+Web+Applications
● https://www.visualstudio.com/
●
Reading material
Delivering changes for applications and databases
@EdPiairo
https://pt.linkedin.com/in/jesuspiairo
eduardopiairo@outlook.com
http://www.eduardopiairo.com/
@MytyMyky
https://pt.linkedin.com/in/miguelalho-982331
alho@miguelalho.com
http://www.miguelalho.pt/
1 of 35

More Related Content

What's hot(20)

Plug   saikuPlug   saiku
Plug saiku
Skills Matter1.5K views
Evolutionary database designEvolutionary database design
Evolutionary database design
Salehein Syed346 views
LinqLinq
Linq
Ariful Haque tutul281 views
Flux and React.jsFlux and React.js
Flux and React.js
sara stanford882 views
Cross-platform interactionCross-platform interaction
Cross-platform interaction
Oleksii Duhno945 views
DMann-SQLDeveloper4ReportingDMann-SQLDeveloper4Reporting
DMann-SQLDeveloper4Reporting
David Mann307 views
Cycling for noobsCycling for noobs
Cycling for noobs
Steve Lee1.2K views
Performance metrics for a social networkPerformance metrics for a social network
Performance metrics for a social network
Thierry Schellenbach18.7K views
FashiolistaFashiolista
Fashiolista
nlwebperf577 views
AzkabanAzkaban
Azkaban
Anatoliy Nikulin1.7K views
Managing Exchange 2016 - Paul RobichauxManaging Exchange 2016 - Paul Robichaux
Managing Exchange 2016 - Paul Robichaux
Summit 7 Systems916 views
LSC@LDAPCon 2011LSC@LDAPCon 2011
LSC@LDAPCon 2011
sbahloul810 views

Similar to Delivering Changes for Applications and Databases(20)

More from Miguel Alho(6)

KanbanKanban
Kanban
Miguel Alho622 views
Web 3.0Web 3.0
Web 3.0
Miguel Alho817 views
Aspectos profissionais 3pontosAspectos profissionais 3pontos
Aspectos profissionais 3pontos
Miguel Alho460 views

Recently uploaded(20)

Liqid: Composable CXL PreviewLiqid: Composable CXL Preview
Liqid: Composable CXL Preview
CXL Forum120 views
ThroughputThroughput
Throughput
Moisés Armani Ramírez31 views
Tunable Laser (1).pptxTunable Laser (1).pptx
Tunable Laser (1).pptx
Hajira Mahmood21 views
[2023] Putting the R! in R&D.pdf[2023] Putting the R! in R&D.pdf
[2023] Putting the R! in R&D.pdf
Eleanor McHugh36 views

Delivering Changes for Applications and Databases

  • 1. Delivering changes for applications and databases “The beginning of a beautiful friendship” Eduardo Piairo Miguel Alho
  • 2. Delivering changes for applications and databases @EdPiairo https://pt.linkedin.com/in/jesuspiairo eduardopiairo@outlook.com http://www.eduardopiairo.com/ Eduardo Piairo, DBA & DevOps @ Celfinet
  • 3. Delivering changes for applications and databases @MytyMyky https://pt.linkedin.com/in/miguelalho-982331 alho@miguelalho.com http://www.miguelalho.pt/ Miguel Alho, Software Engineer @ Faro Europe
  • 5. Delivering changes for applications and databases Source Control Continuous Integration Continuous Delivery Database + Application
  • 6. What’s so special about the database?
  • 7. What’s so special about the database?
  • 8. DLM – Database Lifecycle Management Data management and data migration Data monitoring Data recovery
  • 9. Typical problems we want to fix • Databases are out of pace with application development • Lack of traceability of database changes (changes history) • Manual databases processes prevent the CI and CD utilization • Lack of testability • Database setup time for new environments • Bugs in production
  • 10. Typical problems we want to fix Databases becoming a bottleneck in an agile delivery process
  • 11. Typical problems we want to fix Databases becoming an easy target to blame
  • 13. Why database automation? • Enable control over database development • Increase speed of response to change • Keep a versioned “history” of database states and changes • Reliability of the release process • Increase release frequency though repeatability of processes • Reduce time spent fixing bugs - automated tests • Remove/reduce human intervention in the release process
  • 15. 1st step: Source Control • First step in your database deployment pipeline • Traceability through change history • SQL as documentation • Shared code-base and shared process • Enforceable standards to reduce conflicts
  • 16. 1st Step: Source Control Fundamental resource: SQL Script
  • 17. Migrations-based approach vs State-based approach • State based solutions • Script represents the current database state • Your source of truth is how the database should be • Migrations based solutions • Script represents a migration • Migration represents how to transition to the next database version • Your source of truth is how the database should change
  • 18. Flyway • Flyway – “Database Migrations made Easy” http://flywaydb.org/ • Open source database migration tool • Flyway command-line enabled Simplicity: easy to setup, no need to install Zero dependencies (java + jdbc) Scripts are written in SQL
  • 19. Flyway – How does it work? • Flyway commands: • migrate, clean, info, validate, baseline, repair • Metadata table: • Used to track the state of the database • If the database is empty, flyway won’t find it and will create it • Flyway scans the migrations directory and check migrations against the metadata table • Migrations are sorted based on their version number and applied in order
  • 20. Version Control System Two components: Database scripts repository Flyway – database migrations Rule 1: Script version (timestamp) Rule 2: Operation type Rule 3: Object type Rule 4: Object name Example: V20160728.1100__Create_TB_MyTable.sql
  • 21. Demo
  • 22. How about applications? It takes two to tango
  • 23. Database and Application change alignment ● Database schemas should evolve along with application evolution ○ deploy app => deploy changes / migrations ● (Integration) Test app and db changes together ○ should pass tests while developing, ○ should pass tests while promoting to new environments ● Ideally, one application per database ● Pipeline deployment only!
  • 24. Integration test at the database layer ● Use TDD techniques ● Limit database integration tests to data layer ○ faster tests! ○ validate sql and mappings ● Always test with the latest schema DatabaseApplication DatabaseApplication
  • 27. Database and Application change alignment Fearless database changes
  • 28. Challenges • Challenge #1: managing multiple branches • Manage merge conflicts • The default branch is the main/master branch and all the merges for this branch must be done with caution
  • 29. Challenges • Challenge #2: Script expiration date • Should the migration script have an expiration date? • Sometimes, in the next day, the migration script no longer makes sense
  • 30. Challenges • Challenge #3: migration scripts stack • How can I manage the dependencies between different databases?
  • 31. Challenges • Challenge #4: developer sandbox • How to supply a disposable development environment where the database is included?
  • 33. Links ● https://github.com/MiguelAlho/Purchases-DbMigration-sample https://flywaydb.org/ ● http://dbup.github.io/ ● http://www.red-gate.com/products/sql-development/sql-compare/ ● https://octopus.com/ ○ http://docs.octopusdeploy.com/display/OD/Deploying+ASP.NET+Core+Web+Applications ● https://www.visualstudio.com/ ●
  • 35. Delivering changes for applications and databases @EdPiairo https://pt.linkedin.com/in/jesuspiairo eduardopiairo@outlook.com http://www.eduardopiairo.com/ @MytyMyky https://pt.linkedin.com/in/miguelalho-982331 alho@miguelalho.com http://www.miguelalho.pt/