Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Delivering Changes for Applications and Databases


Published on

Presentation used at PortoData (28/07/2016) about manageing database changes in application projects. Demoed Flyway migrations for script packages, and a strategy for using DBUp and localDb for integration testing at the data layer

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Delivering Changes for Applications and Databases

  1. 1. Delivering changes for applications and databases “The beginning of a beautiful friendship” Eduardo Piairo Miguel Alho
  2. 2. Delivering changes for applications and databases @EdPiairo Eduardo Piairo, DBA & DevOps @ Celfinet
  3. 3. Delivering changes for applications and databases @MytyMyky Miguel Alho, Software Engineer @ Faro Europe
  4. 4. @DevOpsPorto Is Coming
  5. 5. Delivering changes for applications and databases Source Control Continuous Integration Continuous Delivery Database + Application
  6. 6. What’s so special about the database?
  7. 7. What’s so special about the database?
  8. 8. DLM – Database Lifecycle Management Data management and data migration Data monitoring Data recovery
  9. 9. Typical problems we want to fix • Databases are out of pace with application development • Lack of traceability of database changes (changes history) • Manual databases processes prevent the CI and CD utilization • Lack of testability • Database setup time for new environments • Bugs in production
  10. 10. Typical problems we want to fix Databases becoming a bottleneck in an agile delivery process
  11. 11. Typical problems we want to fix Databases becoming an easy target to blame
  12. 12. Why database automation? • Enable control over database development • Increase speed of response to change • Keep a versioned “history” of database states and changes • Reliability of the release process • Increase release frequency though repeatability of processes • Reduce time spent fixing bugs - automated tests • Remove/reduce human intervention in the release process
  13. 13. Why database automation? Fearless database changes
  14. 14. 1st step: Source Control • First step in your database deployment pipeline • Traceability through change history • SQL as documentation • Shared code-base and shared process • Enforceable standards to reduce conflicts
  15. 15. 1st Step: Source Control Fundamental resource: SQL Script
  16. 16. Migrations-based approach vs State-based approach • State based solutions • Script represents the current database state • Your source of truth is how the database should be • Migrations based solutions • Script represents a migration • Migration represents how to transition to the next database version • Your source of truth is how the database should change
  17. 17. Flyway • Flyway – “Database Migrations made Easy” • Open source database migration tool • Flyway command-line enabled Simplicity: easy to setup, no need to install Zero dependencies (java + jdbc) Scripts are written in SQL
  18. 18. Flyway – How does it work? • Flyway commands: • migrate, clean, info, validate, baseline, repair • Metadata table: • Used to track the state of the database • If the database is empty, flyway won’t find it and will create it • Flyway scans the migrations directory and check migrations against the metadata table • Migrations are sorted based on their version number and applied in order
  19. 19. Version Control System Two components: Database scripts repository Flyway – database migrations Rule 1: Script version (timestamp) Rule 2: Operation type Rule 3: Object type Rule 4: Object name Example: V20160728.1100__Create_TB_MyTable.sql
  20. 20. Demo
  21. 21. How about applications? It takes two to tango
  22. 22. Database and Application change alignment ● Database schemas should evolve along with application evolution ○ deploy app => deploy changes / migrations ● (Integration) Test app and db changes together ○ should pass tests while developing, ○ should pass tests while promoting to new environments ● Ideally, one application per database ● Pipeline deployment only!
  23. 23. Integration test at the database layer ● Use TDD techniques ● Limit database integration tests to data layer ○ faster tests! ○ validate sql and mappings ● Always test with the latest schema DatabaseApplication DatabaseApplication
  24. 24. Demo
  25. 25. What we just saw
  26. 26. Database and Application change alignment Fearless database changes
  27. 27. Challenges • Challenge #1: managing multiple branches • Manage merge conflicts • The default branch is the main/master branch and all the merges for this branch must be done with caution
  28. 28. Challenges • Challenge #2: Script expiration date • Should the migration script have an expiration date? • Sometimes, in the next day, the migration script no longer makes sense
  29. 29. Challenges • Challenge #3: migration scripts stack • How can I manage the dependencies between different databases?
  30. 30. Challenges • Challenge #4: developer sandbox • How to supply a disposable development environment where the database is included?
  31. 31. What’s Next?
  32. 32. Links ● ● ● ● ○ ● ●
  33. 33. Reading material
  34. 34. Delivering changes for applications and databases @EdPiairo @MytyMyky