Database Schema Evolution

1,921 views
1,672 views

Published on

Streamline your database changes by versioning your database instances and your database schema, running database instances locally and implementing database changes using migration scripts based on database refactoring patterns.

Published in: Technology
0 Comments
6 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,921
On SlideShare
0
From Embeds
0
Number of Embeds
46
Actions
Shares
0
Downloads
36
Comments
0
Likes
6
Embeds 0
No embeds

No notes for slide

Database Schema Evolution

  1. 1. Database Schema Evolution Lars Thorup ZeaLake Software Consulting August, 2013
  2. 2. Who is Lars Thorup? ● Software developer/architect ● C#, JavaScript and C++ ● Test Driven Development ● Coach: Teaching agile and automated testing ● Advisor: Assesses software projects and companies ● Founder and CEO of BestBrains and ZeaLake
  3. 3. Elevator pitch ● Don't do this ● Do this instead $ importData Error: Invalid column name 'PartnerId' $ ...??? $ importData Error: Database should be version 47 but is 44 $ migrate Migrated database from version 44 to 47 $ importData Success!
  4. 4. Keeping database and app in sync
  5. 5. Demo $ SampleData.exe migrate 32 Latest version is 73 Current version is 30 Target version is 32 Migrating up to version 31 using 31_section_track_up.sql Migrating up to version 32 using 32_action_kind_up.sql $
  6. 6. Got problems? ● Tests fail because data was changed by someone else? ● Upgrades fail because some script was missing? ● Upgrades and downgrades can't be tested reliably? ● Creating new test and development environments is a hassle? ● Who made this schema change and why? ● Applications fail because code and database are not in sync?
  7. 7. We have solutions! ● Never share a development database ● Automate database schema deployment ● Version your database schema and database instances ● Base script, up and down scripts ● Check db version on app start ● Tools: FlyWay, LiquiBase, home grown ● Database refactorings ● Migration conflicts on commit ● Data builders
  8. 8. Never share a development database ● Run development database locally ● MySQL ● Use free versions of commercial database products where available ● Oracle XE ● SQL Server Express ● Use compatible open source database products if feasible ● PostgreSQL instead of Netezza
  9. 9. Automate database schema deployment ● Recreate database from scratch ● Development boxes and CI servers ● $ mvn dbschema ● Upgrade database to newer version ● Test and production servers ● $ migrate ● Rollback database to older version ● Production servers ● $ migrate down 44
  10. 10. Version your schema and instances ● Schema described by .sql files ● Keep those .sql files under version control ● Include a SCHEMA_INFO table in the schema ● Record the database version in SCHEMA_INFO
  11. 11. Base script, up and down scripts ● 1-base-up.sql ● If you are not starting from scratch ● Contains a dump of the current schema ● Including the SCHEMA_INFO table ● 51-customer-unique-{up,down}.sql ● up: migrates the schema from version 50 to version 51 ● down: migrates the schema from version 51 to version 50 ● Tip: establish a naming convention ● List all scripts in a central file ● migrations.txt
  12. 12. Up and down scripts ● 51-customer-unique-up.sql alter table Customer add constraint NameUnique unique nonclustered (Name) on [primary] ● 51-customer-unique-down.sql alter table Customer drop constraint NameUnique
  13. 13. Check db version on app start ● Application code var expectedVersion = Config.ExpectedDatabaseVersion; var actualVersion = session.GetDBVersion(); if (expectedVersion != actualVersion) { throw new ApplicationException(string.Format( "The database needs migration (its version should be {0} but is {1})", expectedVersion, actualVersion)); }
  14. 14. Tools: FlyWay ● Open-Source, Java-based ● Plain SQL scripts
  15. 15. Tools: LiquiBase ● Open-Source, Java-based ● XML or formatted SQL scripts
  16. 16. Database refactorings ● Catalog of database refactorings ● Split Table, Merge Column, etc ● 18-replace-roles-with-privileges-up.sql alter table Participation add Privileges int constraint DF_privileges default 1 not null update Participation set Privileges=1 update Participation set Privileges=3 where UserId in (select Id from User where Email='admin') drop table UserRole
  17. 17. Migration conflicts on commit ● Commit might conflict on migrations.txt ● Merge migrations.txt, keep both ● Renumber your migration ● Test ● Commit
  18. 18. Test data builder [Test] public void GetResponseMedia() { // given var stub = new StubBuilder { Questions = new [] { new QuestionBuilder { Name = "MEDIA" }, }, Participants = new[] { new ParticipantBuilder { Name = "Lars", Votes = new [] { new VoteBuilder { Question = "MEDIA", Responses = new ResponseBuilder(new byte [] {1, 2, 3}) }, }}, }, }.Build(); var voteController = new VoteController(stub.Session); // when var result = voteController.GetResponseMedia(vote.Id, true) as MediaResult; // then Assert.That(result.Download, Is.True); Assert.That(result.MediaLength, Is.EqualTo(3)); Assert.That(TfResponse.ReadAllBytes(result.MediaStream), Is.EqualTo(new byte[] {1, 2, 3})); }
  19. 19. Read more ● http://techportal.inviqa.com/2011/01/11/database-version- control/ ● http://databaserefactoring.com/ ● http://flywaydb.org/

×