The document describes RelayHealth's build and deployment system for a large codebase with over 7 million lines of code maintained by 12 development teams. It outlines the development life cycle which includes daily commits by developers, automated builds and testing, and weekly deployments to test and production environments. The automated build system utilizes CruiseControl.NET to continuously build releases from the code repository, run tests, and promote "green" builds to the test system. Deployments are managed through branching in the code repository and involve building, testing, and deploying code to stage and production environments without downtime. Planned changes include migrating to Team Foundation Server and tools for continuous integration and deployment.

1. RelayHealth Build System
Ron Cordell
RelayHealth Clinicals
© 2011 RelayHealth and/or its affiliates. All Rights Reserved. Proprietary and Confidential. For Internal Use Only. 1

2. RelayHealth Build System
Agenda
 Development/Deployment Life Cycle
 Local Developer Builds
 Automated Build System
 Deployment System
 Planned Changes
© 2011 RelayHealth and/or its affiliates. All Rights Reserved. Proprietary and Confidential. For Internal Use Only. 2

3. Development/Deploy Life Cycle
• 7 Million Lines of Code, 20+ services, 20+ IIS sites, 20+ DBs
• Ancilliary, 3rd Party processes
• SQL Server 2008R2, MongoDB, RabbitMQ …
• Developers run full system stack on their workstations
• 12 Dev Teams working on 1 Product/Code Base
• Production topology is somewhat different
• All development teams work on TRUNK (typically no
branching/merging)
• Code changes are deployed behind soft switches
• Deploy weekly (currently)
© 2011 RelayHealth and/or its affiliates. All Rights Reserved. Proprietary and Confidential. For Internal Use Only. 3

4. Development/Deployment Life Cycle
Code/Unit
Test/Commit
(Everyday)
Deploy to Automated
Production Build/Deploy
(Thursday) (TEST)
Automated Deployment
Build/Deploy Branch Cut
(STAGE) (Friday)
© 2011 RelayHealth and/or its affiliates. All Rights Reserved. Proprietary and Confidential. For Internal Use Only. 4

5. RelayHealth Build System
Agenda
 Development/Deployment Life Cycle
 Local Developer Builds
 Automated Build System
 Deployment System
 Planned Changes
© 2011 RelayHealth and/or its affiliates. All Rights Reserved. Proprietary and Confidential. For Internal Use Only. 5

6. Local Developer Builds -
MorningCoffee
• Clean local output and cache folders
• Turn off Windows Services and stop IIS
• SVN update
• Update SQL Server database (run delta scripts)
• Create new Mongo collections (if any)
• MSBUILD.EXE against targets file, DEBUG Profile, unsigned
• Run Ndepends Analysis, Sync switches to database
• Run Unit Tests (Nunit)
• Create all IIS site definitions
• Create all Windows service definitions
• Start all services & IIS
• WatiN Automation Tests run manually as needed
© 2011 RelayHealth and/or its affiliates. All Rights Reserved. Proprietary and Confidential. For Internal Use Only. 6

7. RelayHealth Build System
Agenda
 Development/Deployment Life Cycle
 Local Developer Builds
 Automated Build System
 Deployment System
 Planned Changes
© 2011 RelayHealth and/or its affiliates. All Rights Reserved. Proprietary and Confidential. For Internal Use Only. 7

8. Automated Build System – CC.Net
• CruiseControl .NET runs continuously on the build servers
• Performs similar steps the MorningCoffee build, broken out into
separate CruiseControl steps.
• MSBUILD -> Release Profile, Signed & Versioned Assemblies
• CC.Net adds a deployment step and a test automation step before
a build is called “green”
• Only green builds are promoted to Test System, usually twice daily.
• Build failures result in automated rollback of check-ins since the
last “green” build, including the databases.
© 2011 RelayHealth and/or its affiliates. All Rights Reserved. Proprietary and Confidential. For Internal Use Only. 8

9. CC.Net Tray Status App
© 2011 RelayHealth and/or its affiliates. All Rights Reserved. Proprietary and Confidential. For Internal Use Only. 9

10. Build Step Status
© 2011 RelayHealth and/or its affiliates. All Rights Reserved. Proprietary and Confidential. For Internal Use Only. 10

11. RelayHealth Build System
Agenda
 Development/Deployment Life Cycle
 Local Developer Builds
 Automated Build System
 Deployment System
 Planned Changes
© 2011 RelayHealth and/or its affiliates. All Rights Reserved. Proprietary and Confidential. For Internal Use Only. 11

12. Deployment System
• Code deployed to Stage once a week from latest TRUNK “green”
build.
• Code is branched for deployment in SVN
• Branch is built similarly in CC.Net for the Deployment
• Code is deployed to STAGE environment for final testing
• Automated Tests are run - BVT
• Switches as set to proper state to enable new code behavior
• QA Tests as needed
© 2011 RelayHealth and/or its affiliates. All Rights Reserved. Proprietary and Confidential. For Internal Use Only. 12

13. Deployment to Production
• STAGE testing by QA from Monday – Wednesday
• PROD deployment on Thursday
• BVT / Automated Tests run on deployment in PROD
• Switches set to correct state
• Site is never “down” for deployment
© 2011 RelayHealth and/or its affiliates. All Rights Reserved. Proprietary and Confidential. For Internal Use Only. 13

14. RelayHealth Build System
Agenda
 Development/Deployment Life Cycle
 Local Developer Builds
 Automated Build System
 Deployment System
 Planned Changes
© 2011 RelayHealth and/or its affiliates. All Rights Reserved. Proprietary and Confidential. For Internal Use Only. 14

15. Planned Changes
• Move to a complete, enterprise-hosted TFS system?
• TFS source control
• TFS Build Farms, Test Farms
• Take advantage of impact analysis, gated check-ins, traceability
• Move to a more “best of breed” model?
• Enterprise GitHub for source control
• Team City for build system (gated check-ins)
• Other tools for impact analysis, test traceability, etc.
• True Continuous Build and Deployment
• Better Test Automation
• Correct issues preventing continuous deployment
• Package Management, a la NuGet
© 2011 RelayHealth and/or its affiliates. All Rights Reserved. Proprietary and Confidential. For Internal Use Only. 15

16. Q&A
We’re Hiring!
Finance Director Product Manager
Quality Assurance Analyst Sr. Software Engineer
Sr. Software Engineer Client Services Director
Entry Level Software Engineer DevOps Developer
Software Engineer Scrum Master Agile Project Manager
Software Engineer Software QA Manager
Storage Systems Engineer Director of Product Development
Sr Software Eng (.Net Asp.Net Sql Svr
Knowledge Manager
Visio)
Applications Security Architect Web
RH Intern
Applications
Sr. Software Engineer Product Manager Agile SaaS
Software Engineer in Test Scrum Master Agile Project Manager
Software Quality Analyst Sr. Systems Administrator
Quality Engineer 4 Manager User Experience
Software Automation Engineer Quality Assurance Analyst
Sr. Software Quality Engineer SW QA Automation Engineer
Software Architect (.Net Sql Svr Asp.net) Project Manager EMR Implementations
Interaction Designer Software Quality Architect
Interface Implementation Services
Sr. Software Engineer
Manager
Program Director Security Analyst CISSP
© 2011 RelayHealth and/or its affiliates. All Rights Reserved. Proprietary and Confidential. For Internal Use Only. 16