Uploaded byIASA
Database Refactoring With Liquibase
This document discusses agile database development. It promotes refactoring databases using evolutionary modeling and automated testing to allow for continuous integration of database changes. Sandboxes are used to test changes before migrating them to production. Changes are tracked using tools like Liquibase that version database schemas and manage changesets and migrations. This helps break the "curse" of databases that are difficult to change by allowing agile practices to be applied to database development.
More Related Content
Similar to Database Refactoring With Liquibase
Database Refactoring With Liquibase
- 1. AGILE DATABASE DEVELOPMENT WITH Tim Berglund June, 2009
- 2.
- 11. WE BELIEVE CODE CHANGES
- 12. WE BELIEVE CODE CHANGES
- 14. WE ACT LIKETHE DATABASE DOESN’T
- 15. WE ACT LIKETHE DATABASE DOESN’T
- 16.
- 17.
- 19.
- 20.
- 21. CURSES THE DBA WHOCHANGES IT
- 22.
- 23.
- 24. Refactoring Evolutionary Automated Data Testing Modeling Agile Database Development Source Sandboxes Control
- 25.
- 26.
- 27. REFACTORING This term occasionally gets abused!
- 28. REFACTORING A disciplined way to make small changes to your source code to improve its design, making it easier to work with.
- 29.
- 30.
- 31. REFACTORING Opportunistically improving designto accommodate change.
- 32. DATABASE REFACTORING A simple change to a database schema that improves its design while retaining both its behavioral and informational semantics.
- 33.
- 34. EVOLUTIONARY DATAMODELING NOT TRYING TO “GET IT RIGHT UP FRONT.”
- 35. EVOLUTIONARY DATA MODELING MAYBESOME BIG THOUGHTS AT FIRST
- 36. EVOLUTIONARY DATAMODELING BUILD THE SIMPLEST THING THAT CAN POSSIBLY WORK...EVERY DAY
- 37. DATABASE TESTS TDD IN THE DB
- 38.
- 39. DATABASE TESTS STOREDPROCEDURES
- 40. DATABASE TESTS STOREDPROCEDURES SQLUNIT
- 41. DATABASE TESTS STOREDPROCEDURES SQLUNIT DBUNIT
- 42. DATABASE TESTS STOREDPROCEDURES SQLUNIT DBUNIT CUSTOM HYBRIDS
- 43.
- 44. SOURCE CONTROL LIKE MOTHERTAUGHT YOU
- 45.
- 46. SOURCE CONTROL DDL EXTRACT/MIGRATION SCRIPTS REFERENCE DATA STORED PROCS/TRIGGERS
- 47.
- 50.
- 51. COMMAND-LINE TOOL JAVA/JDBC
- 52. COMMAND-LINE TOOL JAVA/JDBC CROSS-PLATFORM
- 53. COMMAND-LINE TOOL JAVA/JDBC CROSS-PLATFORM ANT, MAVEN, SPRING, GRAILS
- 55.
- 56.
- 57. CHANGELOG COMMANDS MIGRATION SCENARIOS
- 58.
- 59.
- 60. CHANGELOG SQL IN XML COHERENT GROUPS OF OPERATIONS
- 61. CHANGELOG SQL IN XML COHERENT GROUPS OF OPERATIONS TRACKED IN DATABASE
- 62. CHANGELOG <changeSet author="tlberglund (generated)" id="1236660747556-11"> <createTable tableName="scshoppingcart"> <column autoIncrement="true" name="id" type="INT"> <constraints nullable="false"/> </column> <column name="userid" type="INT"> <constraints nullable="false"/> </column> <column name="lastmodified" type="DATETIME"/> <column name="name" type="VARCHAR(50)"/> </createTable> </changeSet> <changeSet author="tlberglund" id="spot-1-1-2"> <renameTable oldTableName="scscrubrecordwarehouse" newTableName="scrub_record_log" /> </changeSet>
- 63.
- 64. LIQUIBASE COMMANDS generateChangeLog
- 65.
- 66.
- 67.
- 68.
- 69. LIQUIBASE COMMANDS diff
- 70.
- 71.
- 72. Refactoring Evolutionary Automated Data Testing Modeling Agile Database Development Source Sandboxes Control
- 73.
- 74.
- 75.
- 76.
- 77. CREATIVE ORM MAPPINGS
- 78.
- 79. BROWNFIELD DATA INTEGRATION
- 80. BROWNFIELD DATA INTEGRATION TIME ZERO MIGRATION
- 81. BROWNFIELD DATA INTEGRATION TIME ZERO MIGRATION
- 82.
- 83. BROWNFIELD TESTING INTEGRATION TEST DATA
- 84.
- 85. BROWNFIELD TESTING T0MIGRATION SCRIPTS
- 86.
- 87.
- 88. THANK YOU! TIM BERGLUND AUGUSTTECHNOLOGY GROUP, LLC http://www.augusttechgroup.com tim.berglund@augusttechgroup.com @tlberglund
- 89. PHOTO CREDITS HAMMER SHATTERINGGLASS: HTTP://WWW.FLICKR.COM/PHOTOS/WHISPERWOLF/3486270713 DODO: HTTP://WWW.INTERNATIONALDOVESOCIETY.COM/MISCSPECIES/DODO.HTM EMBALMING: HTTP://LIBRARY.THINKQUEST.ORG/C0116982/HTML%20PAGE%20FOLDER/ HMUMMIFICATION.HTM ATTACKING MUMMY: HTTP://WWW.FLICKR.COM/PHOTOS/ROONBABOON/292393932/ CREEPY MUMMY HEAD: HTTP://WWW.FLICKR.COM/PHOTOS/DR-INI/446311713/ MARTIN FOWLER: HTTP://WWW.FLICKR.COM/PHOTOS/PRAGDAVE/173640462/ SCOTT AMBLER: HTTP://WWW.AMBYSOFT.COM/SCOTTAMBLER.HTML BRENDEN FRASER: HTTP://SCRAPETV.COM/NEWS/NEWS%20PAGES/ENTERTAINMENT/UNIVERSAL-FIRE- ACCIDENTAL-NEW-MUMMY-MOVIE-DELIBERATE-SCRAPE-TV-THE-WORLD-ON-YOUR-SIDE.HTML SAND: HTTP://WWW.FLICKR.COM/PHOTOS/LEVIATHOR/207625319/ MOTHER AND CHILD: HTTP://WWW.FLICKR.COM/PHOTOS/PATRICK_Q/268149208/ BROWN FIELD: HTTP://WWW.FLICKR.COM/PHOTOS/ARTUR02/471094378/ JALOPY: HTTP://WWW.FLICKR.COM/PHOTOS/DRAGONFLEYE/3246153/ JALOPY ON RT 66: HTTP://WWW.FLICKR.COM/PHOTOS/EVOETSCH/2436023218/ PIMPMOBILE: HTTP://WWW.FLICKR.COM/PHOTOS/AIRGAP/1053594933/ JUNKED JALOPY: HTTP://WWW.FLICKR.COM/PHOTOS/JABOOBIE/61014658/ PAINTING CAR: HTTP://WWW.FLICKR.COM/PHOTOS/GARETHJMSAUNDERS/2066690016/
Editor's Notes
- #3 I&#x2019;m your presenter.
- #4 I&#x2019;m a Java developer.
- #5 I work with a lot of open source technologies.
- #6 I haven&#x2019;t written a lot of Java in the past year. Mostly these days I write Groovy.
- #7 I participate in the local development community by serving on the boards of www.denveropensource.org and www.iasadenver.org.
- #8 The August Technology Group is my consulting firm.
- #9 As software developers and architects, our worlds are filled with data.
- #10 Unfortunately, the data is not always in the condition we&#x2019;d like. We try to do a good job structuring it, but the reality is that we often fail, and even when we succeed, the business changes enough so that even our successes are short-lived.
- #11 Our tool sets and processes have developed to deal expertly with rapidly changing code.
- #12 Our tool sets and processes have developed to deal expertly with rapidly changing code.
- #13 The same tools and practices have not been applied to data. Leading to one of two problems...
- #14 The same tools and practices have not been applied to data. Leading to one of two problems...
- #15 Will it die? It is said of evolutionary systems that things that fail to adapt to change die out. If only this were the case with the enterprise database...
- #16 Rather than going extinct, the production DBAs surround it like priests, carefully filling it with embalming fluid and wrapping it in linen strips. (And possibly putting some operations personnel in the tomb to serve the database in the afterlife.)
- #17 Inability to change and overprotection lead us to this dread antipattern.
- #18 There are long-lived Enterprise applications that rely on it, so it can&#x2019;t just go away.
- #19 It isn&#x2019;t really alive either, because business needs are constantly changing, but the database can&#x2019;t change with them.
- #20 At least the production DBAs seem to think so!
- #21 We&#x2019;ll figure out how to manage changes to the database such that we can make them with confidence in a way that brings the best of developer tools to bear and is relatively friendly to DBA workflows. We&#x2019;ll do this with a mindset and a tool.
- #22 The fix is what Scott Ambler calls &#x201C;evolutionary [or agile] database development.&#x201D; This consists of five components.
- #24 Martin sez...
- #25 Martin sez...
- #30 Scott sez.... Code refactorings are really only concerned about behavior. DBs have behavior (stored procs, triggers, etc.) but also information. The database must say the same thing in the same way after the refactoring.
- #31 These are two mistakes here: one is thinking we&#x2019;re smart enough to do all this designing correctly at the outset&#x2014;we&#x2019;re not. The other is thinking that the database&#x2019;s business context is a static thing&#x2014;it isn&#x2019;t. There is no &#x201C;right&#x201D; up front, because requirements will change constantly.
- #32 Which doesn&#x2019;t mean we can&#x2019;t do a couple of days of designing at first; we can. It makes sense to try to anticipate what we can and make big, hard-to-change commitments correctly. We always expect change, though.
- #34 TDD adoption in software development is low enough, but it is virtually unheard of in database development. The tooling lags behind and the expertise is singularly rare.
- #35 There are ways to do it. The tools aren&#x2019;t what they are for TDD of code, but there are options.
- #36 There are ways to do it. The tools aren&#x2019;t what they are for TDD of code, but there are options.
- #37 There are ways to do it. The tools aren&#x2019;t what they are for TDD of code, but there are options.
- #38 There are ways to do it. The tools aren&#x2019;t what they are for TDD of code, but there are options.
- #39 Bring all the knowledge, practices, and advantages of software source control to the database. Simply control all those text files in SVN or Git like you normally would. This practice is old hat.
- #40 Every database artifact goes in to the repository.
- #41 Developers need a place to deploy refactorings when they&#x2019;re trying to get their tests to pass. This must be a local database not used by any other team member or system.
- #42 If we learn how to refactor databases from Ambler and Sandalage, Liquibase is the tool that makes it easy. It is an XML-based love poem to Scott Ambler.
- #43 Liquibase is fundamentally a command-line tool written in Java. It uses JDBC to communicate with the database, and can coexist well in a non-Java shop. I can be invoked from popular open-source build tools and frameworks.
- #44 Liquibase is fundamentally a command-line tool written in Java. It uses JDBC to communicate with the database, and can coexist well in a non-Java shop. I can be invoked from popular open-source build tools and frameworks.
- #45 Liquibase is fundamentally a command-line tool written in Java. It uses JDBC to communicate with the database, and can coexist well in a non-Java shop. I can be invoked from popular open-source build tools and frameworks.
- #46 Liquibase is fundamentally a command-line tool written in Java. It uses JDBC to communicate with the database, and can coexist well in a non-Java shop. I can be invoked from popular open-source build tools and frameworks.
- #47 We&#x2019;ll consider three aspects of Liquibase. How it stores the schema, how it interacts with the database, and how to use it in some real-world scenarios.
- #48 We&#x2019;ll consider three aspects of Liquibase. How it stores the schema, how it interacts with the database, and how to use it in some real-world scenarios.
- #49 We&#x2019;ll consider three aspects of Liquibase. How it stores the schema, how it interacts with the database, and how to use it in some real-world scenarios.
- #50 Must we rewrite our SQL in XML? We must. It&#x2019;s painful and unappetizing, but worth it! Also, there&#x2019;s a Grails plugin called Autobase that lets us do it in nice Groovy Builder syntax, which is preferable. We won&#x2019;t address Autobase in detail here, but it&#x2019;s worth looking in to.
- #51 Must we rewrite our SQL in XML? We must. It&#x2019;s painful and unappetizing, but worth it! Also, there&#x2019;s a Grails plugin called Autobase that lets us do it in nice Groovy Builder syntax, which is preferable. We won&#x2019;t address Autobase in detail here, but it&#x2019;s worth looking in to.
- #52 Must we rewrite our SQL in XML? We must. It&#x2019;s painful and unappetizing, but worth it! Also, there&#x2019;s a Grails plugin called Autobase that lets us do it in nice Groovy Builder syntax, which is preferable. We won&#x2019;t address Autobase in detail here, but it&#x2019;s worth looking in to.
- #53 The changelog is a script that builds your schema one DDL statement at a time. Each changeSet is converted into a dialect-specific SQL statement and is executed in the database, then marked as complete in a log table.
- #54 Extracts the database&#x2019;s metadata and generates a changelog. This is the first step is getting started on a database that already exists.
- #55 Since Liquibase tracks the status of all changesets in the changelog, we&#x2019;ll need to tell it that our newly extracted changelog is in sync with the actual database. This command usually shouldn&#x2019;t be executed except as the second step getting started with the tool.
- #56 Plays any new changeSets in the changeLog against the database. This is the statement you&#x2019;ll use the most during evolutionary database development with Liquibase.
- #57 Marks the database for a future rollback.
- #58 Rolls back changesets to given tag.
- #59 Supposed to document the differences between the database and the changelog. I&#x2019;ve not had good luck with this so far against MySQL and SQLServer databases.
- #63 Evolutionary Database Development is not controversial among software developers, because we are largely persuaded of the core propositions. Moreover, Liquibase seems like a reasonable tool to help. Now let&#x2019;s talk about some practices to put in place in a brownfield effort.
- #64 People are using it, it works, it has literally years of knowledge and business process encoded in it. It&#x2019;s also ugly and outdated, and its schema seems to have been built by the DBA Club after school. But you&#x2019;re replacing it, with Grails...!