SUNZ 2010 Version Control and Automated Deployment of Source Code in a Data Warehouse Environment Mike O’Neil MSD Data War...
Content <ul><li>Background and Context </li></ul><ul><li>Motivation </li></ul><ul><li>Change Management </li></ul><ul><li>...
Background and Context <ul><li>IAP Data Warehouse (SAS based) </li></ul><ul><li>Established 1994 </li></ul><ul><li>30+ Sou...
Motivation <ul><li>Quality improvement </li></ul><ul><li>Reduction in Failure </li></ul><ul><li>Productivity and Efficienc...
 
Change Management <ul><li>Change Management </li></ul><ul><ul><li>Policy and process </li></ul></ul><ul><li>Change Control...
Change Management <ul><li>Survey Results </li></ul><ul><li>Not all changes logged  95% </li></ul><ul><li>Changes not thoro...
Automation of Deployment <ul><li>Common feature of commercial software development </li></ul><ul><li>Continuous integratio...
IAP Automation of Deployment <ul><li>Subversion for Version Control </li></ul><ul><li>TortoiseSVN for Developer Interface ...
Subversion & TortoiseSVN <ul><li>Subversion – Database, Server, CLI </li></ul><ul><li>TortoiseSVN – Windows Explorer plug-...
Subversion
 
IAP Automation of Deployment <ul><li>Configuration “database” </li></ul><ul><ul><li>Configuration_Objects – names all obje...
 
IAP Automation of Deployment <ul><li>Reconciliation </li></ul><ul><li>Export from repository </li></ul><ul><li>Pre-deploy ...
Standards and Practice <ul><li>Typical IAP approach </li></ul><ul><li>Driver Shell </li></ul><ul><ul><li>Invoked by Contro...
Standards and Practice <ul><li>Driver Shell Script </li></ul><ul><ul><li>Export ENVAR=value </li></ul></ul><ul><ul><li>exp...
SAS Features and Functionality <ul><li>Configuration “database” in CSV files </li></ul><ul><ul><li>SAS Proc Import </li></...
Where to from here? <ul><li>Acceptance </li></ul><ul><li>Training </li></ul><ul><li>Complete repository set up </li></ul><...
Questions? <ul><li>Thank you </li></ul>[email_address]
Upcoming SlideShare
Loading in …5
×

Sunz2010 Development Version Control And Automated Deployment Of Source Code In A Data Warehouse Environment

1,668
-1

Published on

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,668
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
20
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Sunz2010 Development Version Control And Automated Deployment Of Source Code In A Data Warehouse Environment

  1. 1. SUNZ 2010 Version Control and Automated Deployment of Source Code in a Data Warehouse Environment Mike O’Neil MSD Data Warehouse Team [email_address]
  2. 2. Content <ul><li>Background and Context </li></ul><ul><li>Motivation </li></ul><ul><li>Change Management </li></ul><ul><li>Automation of Deployment </li></ul><ul><li>Subversion and TortoiseSVN </li></ul><ul><li>Standards and Practices </li></ul><ul><li>SAS Features and Functionality </li></ul><ul><li>Where to from here? </li></ul>
  3. 3. Background and Context <ul><li>IAP Data Warehouse (SAS based) </li></ul><ul><li>Established 1994 </li></ul><ul><li>30+ Source Systems </li></ul><ul><li>10,000 Information Consumers </li></ul><ul><li>UNIX and Windows SAS platforms </li></ul><ul><li>600+ Control-M scheduled jobs </li></ul><ul><li>Huge SAS Code Base </li></ul><ul><li>Frequency of Change </li></ul>
  4. 4. Motivation <ul><li>Quality improvement </li></ul><ul><li>Reduction in Failure </li></ul><ul><li>Productivity and Efficiency </li></ul><ul><li>Cost </li></ul><ul><li>Physical Environment Separation </li></ul><ul><li>Maturing process </li></ul><ul><li>Removal of emotion </li></ul><ul><li>Increasing accountability and ownership </li></ul>
  5. 6. Change Management <ul><li>Change Management </li></ul><ul><ul><li>Policy and process </li></ul></ul><ul><li>Change Control </li></ul><ul><ul><li>Recording and approval </li></ul></ul><ul><li>Version Control </li></ul><ul><ul><li>Asset Management </li></ul></ul>
  6. 7. Change Management <ul><li>Survey Results </li></ul><ul><li>Not all changes logged 95% </li></ul><ul><li>Changes not thoroughly tested 90% </li></ul><ul><li>Lack of process enforcement 85% </li></ul><ul><li>Poor communication and dissemination 65% </li></ul><ul><li>Lack of centralized process ownership 60% </li></ul><ul><li>Lack of change approval policy 50% </li></ul><ul><li>Frequent change notification after the fact 40% </li></ul>Reference: Edward Stickel quoting Harris Kern in article at http://www.itsmwatch.com/itil/article.php/11700_3367151_1
  7. 8. Automation of Deployment <ul><li>Common feature of commercial software development </li></ul><ul><li>Continuous integration </li></ul><ul><li>Regression testing </li></ul><ul><li>Configuration driven (what goes where) </li></ul><ul><li>Version Control </li></ul>
  8. 9. IAP Automation of Deployment <ul><li>Subversion for Version Control </li></ul><ul><li>TortoiseSVN for Developer Interface </li></ul><ul><li>Configuration driven </li></ul><ul><li>Deployment Application written in SAS (and UNIX shell script) </li></ul>
  9. 10. Subversion & TortoiseSVN <ul><li>Subversion – Database, Server, CLI </li></ul><ul><li>TortoiseSVN – Windows Explorer plug-in </li></ul><ul><li>Basic developer workflow: </li></ul><ul><ul><li>Check Out to create a Working Copy </li></ul></ul><ul><ul><li>Branch to manage your own stuff </li></ul></ul><ul><ul><li>Commit changes </li></ul></ul><ul><ul><li>Update when new release hits trunk </li></ul></ul><ul><ul><li>Tag when ready for release </li></ul></ul>
  10. 11. Subversion
  11. 13. IAP Automation of Deployment <ul><li>Configuration “database” </li></ul><ul><ul><li>Configuration_Objects – names all objects </li></ul></ul><ul><ul><li>Configuration_Paths – says where they go </li></ul></ul><ul><li>Owned and maintained by developers </li></ul>
  12. 15. IAP Automation of Deployment <ul><li>Reconciliation </li></ul><ul><li>Export from repository </li></ul><ul><li>Pre-deploy steps </li></ul><ul><li>Create target directory structure </li></ul><ul><li>Deploy software objects to target </li></ul><ul><li>Post-deploy steps </li></ul>
  13. 16. Standards and Practice <ul><li>Typical IAP approach </li></ul><ul><li>Driver Shell </li></ul><ul><ul><li>Invoked by Control-M </li></ul></ul><ul><ul><li>Creates environment variables </li></ul></ul><ul><ul><li>Calls Driver SAS program </li></ul></ul><ul><li>Driver SAS </li></ul><ul><ul><li>Retrieves environment variables (%sysget) </li></ul></ul><ul><ul><li>%includes SAS code to do work </li></ul></ul>
  14. 17. Standards and Practice <ul><li>Driver Shell Script </li></ul><ul><ul><li>Export ENVAR=value </li></ul></ul><ul><ul><li>export SASLIB=/lev1_3/app/sasdata </li></ul></ul><ul><ul><li>export SASCODE=/r02/app/sasprogs </li></ul></ul><ul><li>Driver SAS Program </li></ul><ul><ul><li>%let saslib = %sysget(SASLIB); </li></ul></ul><ul><ul><li>libname applib “&saslib”; </li></ul></ul><ul><ul><li>%let sascode = %sysget(SASCODE); </li></ul></ul><ul><ul><li>filename appsrc “&sascode”; </li></ul></ul><ul><ul><li>%include appsrc(worker1.sas); </li></ul></ul>
  15. 18. SAS Features and Functionality <ul><li>Configuration “database” in CSV files </li></ul><ul><ul><li>SAS Proc Import </li></ul></ul><ul><li>SVN List command –xml </li></ul><ul><ul><li>SAS xml libname engine and xml maps </li></ul></ul><ul><li>Programs that write programs </li></ul><ul><ul><li>Put “cp a.sh /r02/test/ControlM/shells”; </li></ul></ul><ul><ul><li>Put “chgrp develop /r02/test/ControlM/shells/a.sh”; </li></ul></ul><ul><ul><li>Put “chmod 2771 /r02/test/ControlM/shells/a.sh”; </li></ul></ul><ul><li>Standards and practice (using %sysget) </li></ul>
  16. 19. Where to from here? <ul><li>Acceptance </li></ul><ul><li>Training </li></ul><ul><li>Complete repository set up </li></ul><ul><li>Complete required refactoring </li></ul><ul><li>Integrate with JIRA and Remedy </li></ul><ul><li>Run under Control-M </li></ul>
  17. 20. Questions? <ul><li>Thank you </li></ul>[email_address]

×