Your SlideShare is downloading. ×
Trunk and branches for database configuration management
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Trunk and branches for database configuration management

9,662
views

Published on

The document describes the Trunk and Branches approach to Configuration Management for a Database project. It explains the advantages and challenges of the strategy and provides solutions and …

The document describes the Trunk and Branches approach to Configuration Management for a Database project. It explains the advantages and challenges of the strategy and provides solutions and describes the dedicated tools that significantly improved the quality and efficiency of the process.

Published in: Technology

0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
9,662
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
110
Comments
0
Likes
2
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Trunk and Branches for Database Configuration Management Trunk and Branches for Database Configuration Management Robert Berliński, support Mercedes Gavilan Document version 1.7, February 16, 2012 http://www.scmsupport.com/Table of contents:PREFACE ................................................................................................................................. 2CHALLENGES .......................................................................................................................... 2THE SOLUTION ........................................................................................................................ 2 The experience ........................................................................................................................................ 3 Repository structure ................................................................................................................................ 3 Comments in commits ............................................................................................................................ 4 Patches and Upgrades ............................................................................................................................. 4 The environment ..................................................................................................................................... 5 The workflow .......................................................................................................................................... 6CONTINUOUS INTEGRATION ................................................................................................. 7TOOLS ...................................................................................................................................... 8 The DEV environment ............................................................................................................................ 8 Building patches and an upgrade ............................................................................................................ 9CONCLUSION ........................................................................................................................ 11REFERENCES ........................................................................................................................ 11AbstractThe document describes the Trunk and Branches approach to Configuration Management for a Databaseproject. It explains the advantages and challenges of the strategy and provides solutions and describesthe dedicated tools that significantly improved the quality and efficiency of the process.Copyright (c) www.scmsupport.com 2012. All right reserved. Page 1/11
  • 2. Trunk and Branches for Database Configuration ManagementPrefaceThe document describes authors unique approach to the Configuration Management for a Databaseproject. And the document is based on the authors experience. The procedures had been successfullyapplied to an Oracle Database project with Subversion repository. It resulted in significantimprovements of procedures and deployment of dedicated tools. The tools help to automate most of thetime-consuming processes and eliminate space for human mistakes. The automation saves more than70% of Config Managers time as compared to manual process and about 90% of Administrators time.The improved efficiency allows Config Manager and Administrator to handle more projects in the sametime. It also gives the Project Manager more flexibility to plan, schedule and when required dynamicallychange the projects goals.However the work is based on Oracle and SVN, the procedures might be applied to a different databaseand/or to a different version control repositories.Authors assume that the Reader is familiar with the process of using trunk and braches within revisioncontrol repository (e.g. described in the " Trunk and Branches for Configuration Management"document").ChallengesThe trunk and branches approach within a revision control is well known for its benefits e.g. in the Javaworld. The benefits include: better control and security of the code - separates the production version from the work-in-progress version, imposes access control and manage read/write rights for each branch and the trunk, support for parallel projects - allows to separate changes between branches, e.g. distribute changes based on different delivery date or based on different functionality/skills, improves manageability for the changes and allows better assignment of tasks to employees, therefore improving the overall efficiency, flexibility to satisfied unexpected business requirements to divide scope of a project, postpone or obsolete selected parts, therefore giving better manageability in terms of time, resources and costs required for a project.The goal is to benefit from taking the same approach against a database project. But due to the nature ofa database some issues require to be addressed. Below are some of them: strategy/granularity for merging - source code merge vs. file merge, mapping variety of database objects (packages, types, procedures, model, etc.) into version control repository, strategy for handling scripts (data-modification and model-modification), strategy for handling relevant dependencies between model/objects/data changes, strategy for preserving time-order dependencies between objects that are being changed/removed.The SolutionFirst of all it is important to focus on the high level aspects of the approach. The experience shows thefollowing ways are valuable and efficient: developers check-out and commit changes to revision control repository, Config Manager collects changes from repository, builds patches and upgrade scripts; the processes are supported by dedicated tools, then Config Manager tests the patches and upgrades in test environment, Config Manager delivers the files of patches and upgrade to Business/Administrator,Copyright (c) www.scmsupport.com 2012. All right reserved. Page 2/11
  • 3. Trunk and Branches for Database Configuration Management the Administrator deploys the changes in straightforward manner by executing the delivered scripts.The experienceThe authors work with an Oracle database. The characteristics of the database are shown below.INDEX 1500TRIGGER (generated) 600TABLE 550SEQUENCE 380PACKAGE 300VIEW 50TYPE 35FUNCTION 30PROCEDURE 20Total lines of code 225000Table 1. The characteristics of database.Repository structureThe structure of SVN repository folders maps an Oracle database project.1. trunk2. branches  sal<version> o patches o scripts  database  docs  schema  dict  updates  data  dict  sc o sources  Functions  Packages  Procedures  Triggers  Types  ViewsFigure 1. The structure of SVN repository (the idea on the left and the implementation on the right).The SCRIPTS directory contains all scripts and documents required to create and configure a database,table spaces and a schema. There are also scripts creating roles and privileges and granting permissionsto project schema. The subdirectories are organized to contain the following: DATABASE – scripts that create a database and table spaces DOC – documentation that describes configuration procedures for a database; for example: startup and tuning parameters, coding standard/recommendations for PL/SQL, name convention for database objects; it is also a place to store the database physical model generated by a data modeling tool (e.g. Sybase Power Designer, which we use) as well as the html documentation for any table, etc.Copyright (c) www.scmsupport.com 2012. All right reserved. Page 3/11
  • 4. Trunk and Branches for Database Configuration Management SCHEMA – scripts that create a schema, grant user, create file system directories for database directory object (e.g. used for exports). The subfolder DICT contains data inserts for application dictionaries. UPDATES – three groups of scripts that might be written during a development process (in contrast to the scripts included in the SCHEMA and DATABASE directories meant to create a database); the three groups are: o SC (abbr. schema change) – scripts that change schema, add/modify columns in tables, add/remove indexes, etc., o DICT (abbr. dictionary) – scripts that insert and modify dictionary tables used by an application, e.g. Countries, Currencies, o DATA – script that modify user data, e.g. reorganize data according to schema changes.The SOURCES directory contains sources of the PL/SQL stored objects. Each type of the objects has itsown folder named after the objects type. The structure assumes that specification and body are storedtogether in one file. The PL/SQL allows to split these in separate files.The PATCH directory contains all patch scripts build in order to deploy updates. The changes areselected based on a range set by two revisions numbers A and B and include all changes committedbetween revisions A and B. Scripts are stored in subdirectories named after the branch name followedby the build sequence number within the branch, for example PATCH_1.1.002 means version 1.1 andpatch number 2.Experience shows, that the structure is both flexible and effective. It has been in use for the database inexample and it satisfied all needs for changes over two years of development.Comments in commitsIt is important, that every change is referenced by a unique ID. Developers must write the ID incomments when committing changes. Later the comments are pulled out from repository by anautomatic tool when building a patch. It helps to provide to Business the list of changes delivered by thegive patch.Patches and UpgradesPatches and Upgrades are sql scripts meant to deploy changes to a database via SQLPLUS: Patches deliver changes grouped during development of a branch and aims test regions. There might be many patches depending on the complexity of a project, e.g. one patch representing each following week of development. An Upgrade delivers the sum of changes included in patches and it aims the Pilot/Production environments. It is important that patches inside upgrade are ordered the same way, they were deployed in tests.A Patch includes control/meta files, sources and new scripts: readme.txt file that describes the installing procedure, usually it is: sqlplus <schema_user>/<schema_password> @patcher.sql. patcher.sql script that is the primary script to be executed by Administrator. The script triggers individual changes and it runs the versionhistory.sql script and a post-installation script. versionhistory.sql script writes changes to the Version History table. Post-installation scripts are responsible for re-compilation of the schema and might perform additional tasks depending on the nature of changes.Copyright (c) www.scmsupport.com 2012. All right reserved. Page 4/11
  • 5. Trunk and Branches for Database Configuration Management Folder for each group of the source objects that changed. The types can be: Functions, Packages, Procedures, Triggers, Types, and Views. Folders for scripts and changed dictionaries.A Patch is stored in version repository inside patch subfolder. It includes dependable source objectsfrom the sources folder and scripts from the scripts folder. The elements are copied to the patchfolder. Therefore a patch always includes everything that is required for deploying changes that belongto the scope of the patch.An upgrade is an ordered collection of patches. The order is important to preserve all dependencies. Anupgrade includes control/meta files and the patches subfolders: README.txt file that describes the installing procedure. upgrder.sh script that is the primary script to be executed by Administrator. It iterates and executes each included patch. When necessary additional scripts, e.g. newdir.sh and newdir.sql that builds file system folders and sets the relation with Oracle.The rule of thumb is, that patches should be tested in a test environment before release. The names ofpatches include ordering number. The number helps to preserve time-order when including patches in anupgrade.The environmentThe environment is built based on multiple elements.Figure 2. The work environment - elements and relations.The above diagram shows the distribution of regions in the process of Configuration Management: Starting from the left, there are three IT regions labeled as DEV1, DEV2 and DEV3. There is SVN repository placed in the central. It provides branches for each DEV region. Special Hot Fix region serves a shorter path for detection, fixes and tests in case of critical and urgent production problems. On the right are business environments: Tests reflecting the IT branches, Pilot and Production.Copyright (c) www.scmsupport.com 2012. All right reserved. Page 5/11
  • 6. Trunk and Branches for Database Configuration ManagementPlease note the straight arrows from left to right. They mark processes (committing changes, buildingand delivering patches, building and delivering upgrades) that are automated or supported by dedicatedtools. Elimination of the manual work not only saved time but also helped to eliminate many processesexposed to human errors.The workflowThe workflow starts from source code and finish with upgrade delivery to Production. The versionrepository plays key role, as it guarantees flow of data between succeeding processes. The idea is toprovide steps meant to transform the level of work starting from source code into patches and upgrades.Figure 3. The workflow.Source code changesThere is one DEV* database for each branch shared between all developers. It is used by the developersto code and test changes. After testing, developers commit changes to SVN. It is possible for a developerto work with multiple branches. And with the help of additional tools (e.g. an SVN plugin for PL/SQLDeveloper) the complexity of work with multiple branches is almost transparent for the developers.Therefore it allows the optimum assignments of tasks in different branches to developers. Whenpreparing changes the developers are responsible for resolving file-context conflicts and keeping thesources in repository in synchronization with the database. They also have to remember to commentevery committed change giving the change number provided by business. It makes possible to trackchanges later on and prepare a report of changes included in a patch. The developers are also responsiblefor retrofitting changes made in parallel branches and/or the Hot Fix region.Changes delivered by ScriptsThe developer might also prepare scripts for updating model/data. The developers doesnt commit thescripts to SVN but sends them to the Config Manager, who applies to them a special name that follows aschema for ordering and then commits them to the SVN. In this process the Config Manager mightrequest additional information from the developer who send a script about its nature and relation withother parts of the database. Thanks to that, the Config Manager keeps control of scripts and can orderthem based on the given name. It is also recommended, that the Config Manager has a databaseexperience that will allow to verify the scripts.PatchesCopyright (c) www.scmsupport.com 2012. All right reserved. Page 6/11
  • 7. Trunk and Branches for Database Configuration ManagementOn a scheduled time, Config Manager collects changes committed to SVN in a file called a patch. Apatch is a script that is used to update the database with the committed changes. Depending on the scaleof project and the number of changes patches might be build biweekly, weekly or daily.Since there might be many changes it is important to automate the process of collecting the changes andbuilding each patch. It is possible based on a few assumption: keeping track of revision numbers with reference to the previously built patches and collecting changes that belong only to the newer revisions, the source code changes are being ordered based on the type of the objects for execution within a patch file; it works for most changes and in special situation the Config Manager might apply manual changes (e.g. for type changes), the scripts are saved, committed and their names include an ordering part.In practice, when patches are build regularly and include all or most source code changes the automatedbuild of patches is possible without any need for manual edition. Usually it takes only a few minutes.After building a patch the Config Manager commits it to the SVN, tests in Unit Tests and sends tobusiness repository with a special report including the list of changes. Then Business can decide torequest from Administrator to apply the patch to a User Test region. Since the patch itself is anexecutable bash scripts and includes SQL commands, the Administrators tasks are straightforward: copy the patch file to a working directory based on Administrators preferences (since the patch is directory independent), execute the patch, verify the generated log file for potential errors, send a confirmation e-mail with the log file to a group e-mail address (including Config Manger, Project Manager, Business representatives).It is important to build the patches regularly, so they preserve ordered milestones and can be used laterfor upgrading a database. In case that there will be a need to restore/cancel a change that was included ina patch it is recommended to provide a correcting change/script and build a next patch that will cancelthe previous change.An UpgradeAn upgrade is another script built by Config Manager. The file includes all patches delivered within agiven branch. Therefore an upgrade is the only file required for upgrading Pilot/Production databases. Atthe same time Config Manager merges the changes from the branch into trunkAgain the process of building an upgrade can be automated, since the required patches have beenalready committed to the SVN branch. It is important that the order of execution of patches will followthe same order as they have been build and tested. Besides the nature of the upgrade, the administratorsprocedure with regards to deploying it remains the same.Delivery to productionThe process of production delivery is accomplished by delivering and applying an upgrade. The ConfigManager performs a merge of the branch into trunk. The merge is based on files and should include thesources, scripts, patches and upgrade.Continuous integrationCI = commit build automata + source code repository + tests automata + feedback reportCopyright (c) www.scmsupport.com 2012. All right reserved. Page 7/11
  • 8. Trunk and Branches for Database Configuration ManagementThe automation of building a database and applying patches opens the possibility for automatedcontinuous integration. It can play significant role for verifying database changes and detecting possibleissues as soon as possible, e.g. within nightly cycles.Figure 4. The Continuous Integration.The continuous integration process is currently under development and might assume both: presentationlayer Java application as well as the database. The database might be tested indirectly by performing aset of requests to the application and then verifying the responses.ToolsThe manual work with the Trunk and Branches strategy for Databases will be very time consuming.Hopefully the tools described below helps to automate most of the process and eliminate places forhuman errors.The DEV environmentThe work in developers environment is supported by: TortoiseSVN client, PL/SQL Developer with dedicated plugin for SVN.The TortoiseSVN and PL/SQL Developer are popular tools. They work just fine as expected. On theother hand the SVN plugin requires a few words of comments. It serves as a bridge between the PL/SQLDeveloper and the local SVN repository on developers machine. It automates the process of saving adatabase object (e.g. package or function)., And it automatically support multiple branches as long asConfig Manager and Developers follows the recommended naming conventions for SVN structure. Adeveloper needs to focus only on what database he/she is connected to and then the plugin guarantees,that the files will be stored in the right place. Without that it would be a real nightmares to search for theright folder in order to save a file when working with multiple branches and it could lead to manyhuman mistakes (writing files to wrong branch and/or directory).Copyright (c) www.scmsupport.com 2012. All right reserved. Page 8/11
  • 9. Trunk and Branches for Database Configuration ManagementFigure 5. An example of mapping SVN into local repository after check-out operation.Figure 6. An example of using the SVN plugin to save a Function object.Building patches and an upgradeDepending on the calendar of tests, the number of changes and their importance, Business schedulesdelivery of all the currently available changes to a test region. Before each delivery developers mustcommit their stable changes and then config manager pulls-out and collects all the changes whenbuilding a patch.Copyright (c) www.scmsupport.com 2012. All right reserved. Page 9/11
  • 10. Trunk and Branches for Database Configuration ManagementFigure 7. Patches and upgrades.Config manager uses an automated tool. The tool checks for new changes committed in a branch of aversion repository and then collects the changes in a special patch that again is saved in the same branchof the repository. Each patch is an executable script that applies the changes to a database. Automationis the key aspect in the process of building a patch. As long as all developers follow the procedures forcommitting and commenting changes.Below is an example. Lets assume there are two, new changes #1 and #2. The first added SIMPLE.sqlfunction, the second added EMP_ACTIONS.sql package. The previous patch had the 000 number andincluded revisions 20 to 21. New patch carry the 001 name and includes changes from revisions 22 to 23(the Head).23:06:56,522 [main] - #Start23:06:56,524 [main] - #Config: true23:06:56,634 [main] - #http://1....../......23:06:56,634 [main] - #35ec356c-ed03-11e0-b048-9b1cc1dbb0c923:06:56,643 [main] - # BranchDir: true23:06:56,649 [main] - # PatchDir: true23:06:56,656 [main] - # SourceDir: true23:06:56,673 [main] - # The latest patch [patch_1.1.000;0;20;21]23:06:56,687 [main] - # The new patch [patch_1.1.001;1;22;23]23:06:56,687 [main] - #Mkdir D:Patches/patch_1.1.00123:06:56,710 [main] - #revision: 23, author: null, date: Thu Feb 09 23:00:01 CET 2012, logmessage: Change#223:06:56,716 [main] - A /sal/branches/sal1_1/sources/Packages/EMP_ACTIONS.sql23:06:56,716 [main] - #Adding the Packages subfolder23:06:56,716 [main] - #Added Packages/EMP_ACTIONS23:06:56,756 [main] - #revision: 22, author: null, date: Thu Feb 09 22:59:32 CET 2012, logmessage: Change#123:06:56,762 [main] - A /sal/branches/sal1_1/sources/Functions/SIMPLE.sql23:06:56,762 [main] - #Adding the Functions subfolder23:06:56,762 [main] - #Added Functions/SIMPLE.sql23:06:56,795 [main] - #Log changes: Change#2 Change#123:06:56,795 [main] - #FinishFigure 8. An example shows the log4j output from the patch building tool.Similarly a series of patches can be collected by an automated tool and then saved as one upgrade.Upgrades are collections of tested patches that can be used to deliver changes to Pilot/Production.Copyright (c) www.scmsupport.com 2012. All right reserved. Page 10/11
  • 11. Trunk and Branches for Database Configuration ManagementConclusionThe Trunk and Branch approach successfully works for an Oracle Database project with help ofadditional procedures, scripts and dedicated tools. It required more discipline and structure from ConfigManager and from all developers. Thankfully the most time-consuming tasks performed by ConfigManager are automated. The benefits include better control of the changes, higher efficiency of work,elimination of many places for human mistakes. It also gives more flexibility to managers to planchanges and assign work to developers based on the business requirements.References Martin Fowler, Continuous Integration http://martinfowler.com/articles/continuousIntegration.html Paul M. Duval, Steve Matyas, Andrew Glover, Continuous Integration – Improving Software Quality and Reducing Risk, Addison-Wesley, 2007 JT’s Blog, CI Builds with PL/SQL http://jts-blog.com/?p=48 Wikipedia http://en.wikipedia.org/wiki/Configuration_management Wikipedia http://en.wikipedia.org/wiki/Revision_control SCM Support http://www.scmsupport.com/Copyright (c) www.scmsupport.com 2012. All right reserved. Page 11/11

×