This document outlines Jose Luis Soria's presentation on designing a release pipeline. It discusses defining components, sub-pipelines, stages and orchestration, environments, steps, automation and tooling. The key aspects are defining independent components, identifying stages to filter unfit candidates and provide feedback, automating steps, and choosing tools to support the pipeline design.

1. MADRID · NOV 21-22 · 2014
Designing a Release Pipeline
Jose Luis Soria
Continuous Improvement Manager at Ria Financial
@jlsoriat

2. MADRID · NOV 21-22 · 2014
What is a
Release Pipeline?

3. MADRID · NOV 21-22 · 2014
What is a Release Pipeline?
Automated manifestation of your delivery process.
Feedback mechanism.
Detection of unfit release candidates.
Pull system.
Useful for CD, or any other delivery model.

4. MADRID · NOV 21-22 · 2014
Pipeline design considerations
Emergent design. No BDUF.
Start early.
Start simple and evolve with the system.
Begin with the most valuable assets.
Address the bottlenecks.

5. MADRID · NOV 21-22 · 2014
#1 Define
Components

6. MADRID · NOV 21-22 · 2014

7. MADRID · NOV 21-22 · 2014
What is a (software) component? (*)
A set of artifacts (binaries, dynamic code, configuration files, other supporting files) that can be deployed and verified together without affecting other areas of the application.
(*) in a release pipeline

8. MADRID · NOV 21-22 · 2014
Choosing components
Deploy and test the smallest independent entity.
Rely on the architecture:
Logical / physical.
Layers / tenants.
See the whole.

9. MADRID · NOV 21-22 · 2014
What do we need to know?
For each component:
Meaningful name.
Description.
Priority / order (when to address it?)
Source (most likely, version control.)
Target (where it gets deployed.)
Pre-requisites.
Dependencies.
Configuration tokens.

10. MADRID · NOV 21-22 · 2014
Configuration tokens
1.Make a list of environment-dependent information.
2.Tokenize it in the configuration.
3.Gather the values for all the environments.

11. MADRID · NOV 21-22 · 2014
Component sheet

12. MADRID · NOV 21-22 · 2014
Activity: defining components
Our application consists on:
A web site built on top this technology stack:
MVC framework.
Client-side logic (HTML5, JavaScript.)
Entity model mapped to the DB using a ORM.
Data model residing in a DB.

13. MADRID · NOV 21-22 · 2014
Activity: defining components
Our application consists on:
A web site built on top this technology stack:
MVC framework.
Client-side logic (HTML5, JavaScript.)
Entity model mapped to the DB using a ORM.
Data model residing in a DB.
Key business logic resides on a web services layer.
We also maintain a mobile client for two platforms.

14. MADRID · NOV 21-22 · 2014
Activity: defining components
Our application consists on:
A web site built on top this technology stack:
MVC framework.
Client-side logic (HTML5, JavaScript.)
Entity model mapped to the DB using a ORM.
Data model residing in a DB.
Key business logic resides on a web services layer.
We also maintain a mobile client for two platforms.
For some operations we make calls to third-party web
services.

15. MADRID · NOV 21-22 · 2014
Example: component sheet

16. MADRID · NOV 21-22 · 2014
Implementation notes
It should be possible to independently build, deploy and test anything defined as a component.
You should decide how dependencies will be made available:
Source Control.
Artifact repositories.
Deployed (third-party) artifacts.
Etc.

17. MADRID · NOV 21-22 · 2014
Tooling
Version control systems
Git, Mercurial, SVN, TFS…
Artifact repositories
Artifactory, Nexus, Frog, NuGet…

18. MADRID · NOV 21-22 · 2014
#2 Identify
Sub-pipelines

19. MADRID · NOV 21-22 · 2014
Single pipeline
A single pipeline servicing all the components and teams.
May be able to detect which component has
changed and operate only on that one.

20. MADRID · NOV 21-22 · 2014
Onepipeline per component
Each component has its own pipeline.
Different pipelines may have different designs.
Individual pipelines may fan-in to a system pipeline.
More flexible but more complex.

21. MADRID · NOV 21-22 · 2014
Onepipeline per team
Each team has its own pipeline.
Different pipelines may have different designs.
Individual pipelines may fan-in to an integration pipeline.

22. MADRID · NOV 21-22 · 2014
Mixedapproach
Different teams building different components.
Keep it simple!

23. MADRID · NOV 21-22 · 2014
Implementationnotes
It is easier if you use a tool that allows to define sub-pipelines, fan-in, fan-out, etc.

24. MADRID · NOV 21-22 · 2014
#3 Define
Stages& Orchestration

25. MADRID · NOV 21-22 · 2014

26. MADRID · NOV 21-22 · 2014
Whatisa stage?
A set of steps or activities that are performed on a release candidate.
It lets any release candidate advance towards production, or discards it.
When a release candidate passes through a stage, our confidence on it is increased.
It is a source for feedback.
Frequently taken for environments (but they’re not the same)

27. MADRID · NOV 21-22 · 2014
Whatisorchestration?
It is the way we arrange the stages so release
candidates flow through them, in their way to production.

28. MADRID · NOV 21-22 · 2014
Tipsforstages& orchestration(I)
Feedback is the key. Arrange stages and
orchestration based on the feedback we need.
Stages are filters. The orchestration should be arranged to stop the pipeline if a stage fails.
Stages can contain both manual and automated steps.

29. MADRID · NOV 21-22 · 2014
Tipsforstages& orchestration(II)
Stages can be manually or automatically triggered (think about approvals.)
Automate as much as possible. Including the approvals.

30. MADRID · NOV 21-22 · 2014
Tipsforstages& orchestration(III)
Grow your pipeline wide, not long http://bit.ly/1jsNGP5
Build only once.
Use environment-agnostic binaries.
Version everything.

31. MADRID · NOV 21-22 · 2014
Whatdo weneedto know?
For each stage:
∘Meaningful name.
∘Clear goal.
∘Does it need a manual approval to be triggered?
∘Does it need a manual verification when it has finished?
∘Sources.
∘Flow (orchestration.)

32. MADRID · NOV 21-22 · 2014
Pipeline-levelorchestration(examples)
CommitManual testingReleaseMinimum pipelineFully automatedPartially automated, or manualLegend:

33. MADRID · NOV 21-22 · 2014
Pipeline-level orchestration
(examples)
Commit Acceptance testing Release
Exploratory testing
Capacity testing Security testing
User Acceptance
testing
Complex pipeline
Fully automated
Partially automated, or manual
Legend:

34. MADRID · NOV 21-22 · 2014
Stage-levelorchestration(example)
Code is builtOnce and Only Once: In the first stage. Subsequent stagesare run in parallel if configured that way. Pipeline gets triggered: When a developer does a check-in, or manuallyA new instance of the pipeline is createdGet next stage, relate it to the pipeline instance, prepare parameters, notify for monitoringAutomatically- triggered stage? Trigger stageWait for the user to trigger the stageGather stage results and notify them for monitoringSucceded? Stop the pipeline instanceAny stages left? NOYESNONOYESYES

35. MADRID · NOV 21-22 · 2014
Aboutsources
Version control.
Artifactrepositories.
Environmentlibraries.

36. MADRID · NOV 21-22 · 2014
Whichstagesdo I need?
Think about the kind of feedback you need.
Think about what should stop a release candidate to get to production.
Create a Value Stream Map.

37. MADRID · NOV 21-22 · 2014
Valuestreammap(example)
AssessmentApprovalPlanningCapacity testsReleaseAcceptance testsCodeSpecification2 days1 dayValue- added timeWaittimeDevelopment: cycle time ~ ? 3 days3 days? ?? ? 4 days1 day2 days2 weeks?? Delivery: lead time ~ ? Exploratory tests? UAT? ?

38. MADRID · NOV 21-22 · 2014
Prevalentstages: theCommitstage
Eliminate early release candidates that are unfit for production.
Close to (or the same as) a CI build.
Quick validations: build, unit testing, static analysis, etc.
Packaging.
For Continuous Delivery, it runs on each commit (no branches –feature toggles.)
For other models, decide when it gets triggered (for example, on each merge to trunk.)

39. MADRID · NOV 21-22 · 2014
Prevalentstages: theCommitstage
http://bit.ly/1jsSkwA

40. MADRID · NOV 21-22 · 2014
Prevalent stages: the Automated Acceptance Test stage
http://bit.ly/1jsSkwA

41. MADRID · NOV 21-22 · 2014
Prevalent stages: the Manual Test stage
http://bit.ly/1jsSkwA

42. MADRID · NOV 21-22 · 2014
Prevalent stages: non-Functional Testing stages
http://bit.ly/1jsSkwA

43. MADRID · NOV 21-22 · 2014
Activity: defining stages & orchestration
“We have a basic suite of automated acceptance tests that we plan to grow along with the system.”
“The team does (manual) functional testing.”

44. MADRID · NOV 21-22 · 2014
Activity: defining stages & orchestration
“We have a basic suite of automated acceptance tests that we plan to grow along with the system.”
“The team does (manual) functional testing.”
“We need to support 2,000 concurrent users.”

45. MADRID · NOV 21-22 · 2014
Implementationnotes
Choose a tool that allows to easily model and
visualize the flow.
Choose a tool that supports what you need for orchestration:
∘Approvals.
∘Validations.
∘Parallelization.
∘Alerts.
∘Etc.

46. MADRID · NOV 21-22 · 2014
#4 Define
Environments

47. MADRID · NOV 21-22 · 2014
Whatisanenvironment?
A set of servers, devices or any other resources we need in order to run and validate a release candidate in its way to production.

48. MADRID · NOV 21-22 · 2014
Tipsfordefiningenvironments
Prepare for deployment automation.
Lock down environments. Restrict access.
Different stages could target the same environment if needed.
Prepare for auto-provision.
Make environments disposable. Don’t turn them into bottlenecks.
Environments may not be tied to stages. It should be easy to point any stage to any environment.

49. MADRID · NOV 21-22 · 2014
Activity: defining environments
“We have a basic suite of automated acceptance tests that we plan to grow along with the system.”
“The team does (manual) functional testing.”
“We need to support 2,000 concurrent users.”

50. MADRID · NOV 21-22 · 2014
Implementationnotes
Use virtualization.
Use cloud-based environments.
Use tools for managing templates, configuration, auto-provision, etc.

51. MADRID · NOV 21-22 · 2014
Tooling
Phisicalmachines
Virtualization tools
VmWare, VirtualBox, Hyper-V, etc.
Containers
Docker
Cloud
Amazon, Google, Azure…
Environment definition
Vagrant, PowerShell DSC…

52. MADRID · NOV 21-22 · 2014
#5 DefineSteps

53. MADRID · NOV 21-22 · 2014
Whatisa step?
Any activity that is done in the context of a stage, that allows us to get feedback and prove the fitness of the release candidate.
Examples:
∘Deploy a component.
∘Run automated tests.
∘Run manual tests.
∘Update metrics.
∘Alert the user of some event.
∘Etc.

54. MADRID · NOV 21-22 · 2014
Tipsfordefiningsteps(I)
Consider:
The goal of the stage.
The kind of feedback you need.
Sources.
Targets (environments.)
Build and package only in the Commit stage.

55. MADRID · NOV 21-22 · 2014
Tipsfordefiningsteps(II)
Consider:
Most times, deployment is present, but not always.
(Automated) Smoke Testing should follow any deployment.
Think about both automated and manual steps.

56. MADRID · NOV 21-22 · 2014
Activity: defining steps
“We want to filter out anything producing static analysis warnings.”
“We want to try exploratory testing.”
“We may use the same environment for load testing and security testing.”

57. MADRID · NOV 21-22 · 2014
#6 Define
Automation & Tooling

58. MADRID · NOV 21-22 · 2014
Tipsforstepautomation(I)
Automate everything.
Automate everywhere (for all the environments.)
Preference for automation:
Fully automated steps.
Manually triggered automatic steps.
Manual steps.

59. MADRID · NOV 21-22 · 2014
Tipsforstepautomation(II)
Build only once.
Version everything.
This includes the automations.
Have environment lockdown in mind.

60. MADRID · NOV 21-22 · 2014
Deploymentautomationconsiderations
Deploy the same way to every environment. The target environment should be a (implicit)
parameter for the automations.
Set up (tool-agnostic) one-click deployments.
Treat configuration tokens as parameters for the automations.
Prepare for rollbacks.

61. MADRID · NOV 21-22 · 2014
Databasedeploymentconsiderations
Databasedeploymentisnotthesameas databasedevelopment.
Decide aboutthedeploymentstrategy:
Schema& Data compare.
Delta scripts (betterforContinuousDelivery.)
ORM tools(schemaupdate, migrations, etc.)

62. MADRID · NOV 21-22 · 2014
Test automationconsiderations
Q2 tests are not necessarily run through the UI.
Smoke tests may be run through the UI.
Frequently, non-functional testing can be
automated.
Leave environments and data in a known state.
A few things can’t be automated (UAT & Q3
testing.)

63. MADRID · NOV 21-22 · 2014
Whatdo weneedto know
For each step to be automated:
Automation tool or technology.
Execution model.
Parameters (at least you’ll have the configuration tokens.)
Source / target.

64. MADRID · NOV 21-22 · 2014
Aboutexecutionmodels
Native OS tool.
Agent.
Remote execution.

65. MADRID · NOV 21-22 · 2014
Activity: defining automation & tooling
“Production environment is in an isolated network”
“Operations people won’t allow us to install anything there”

66. MADRID · NOV 21-22 · 2014
Tooling
Environment provision
Puppet, Chef, Ansible
Virtualization tools (VM templates, etc.)
App deployment
Scripting (Unix shell, PowerShell, etc.)
Puppet, Chef, Ansible, VS Release Management
DB Deployment
RedGate, DBDeploy, etc.
Testing
Testing frameworks
UI automation frameworks
Non-functional testing tools

67. MADRID · NOV 21-22 · 2014
#7 Define Execution model, monitoring & metrics

68. MADRID · NOV 21-22 · 2014
Continuousdeliveryflowmodel
Pipeline instances are created on each commit. Any commit is a release candidate.
One-piece continuous flow model.
There is no way back. Any error makes the release candidate to be discarded.
Fixes are treated as new release candidates. They are run through the entire pipeline from the beginning.

69. MADRID · NOV 21-22 · 2014
Otherflowmodels
Pipeline instances are created as needed. A release candidate might comprise several commits.
Decide on the batch size. Larger batches may be cheaper but limit feedback.
Errors might be fixed in the context of the stage where they arise.

70. MADRID · NOV 21-22 · 2014
Monitoringthepipeline
Transparency.
Rely on a proper tool.
Set up alerts for key events.
Use a Cumulative Flow Diagram (CFD.)
Gather metrics and act on them.

71. MADRID · NOV 21-22 · 2014
Primarymetrics
Cycle time.
Mean Time Between Failures (MTBF.)
Mean Time To Recover (MTTR.)

72. MADRID · NOV 21-22 · 2014
Secondarymetrics
Test coverage.
Duplication of code.
Coupling.
Compilation warnings.
Code churn.
Build frequency
Etc.

73. MADRID · NOV 21-22 · 2014
Tooling
Outof thebox metricsin theorchestrationtool.
Ad-hoc reporting.
Sonar, FxCop, Ndepend, FindBugs…

74. MADRID · NOV 21-22 · 2014
#8 Plan forfutureenhancements

75. MADRID · NOV 21-22 · 2014
Examples–DevOps culture
Improvebranchingmodelifneeded.
Debuggingoptimization, symbol servers.
Canaryreleases.
Blue/greendeployments.
A/B Testing.

76. MADRID · NOV 21-22 · 2014
Examples–DevOps culture
Preventiveprofiling.
Telemetry, analytics, ApplicationPerformance Monitoring(APM.)
Proactiveresiliencyenablement(Simianarmyhttp://nflx.it/SPeTGj)

77. MADRID · NOV 21-22 · 2014
Tooling
Applicationprofilers
APM
New Relic
App Insights
Raygun
CompuwareAPM

78. MADRID · NOV 21-22 · 2014Artifact and metadata repositoriesBinaries repositoryTFS Build Drops folderReporting system (receives data from all stages) TFS Reporting – tracks Cycle Time, MTTR & MTBF Custom TFS Reporting on Work ItemsCommit stage TFS Build Definition, customized from Default templateCodeGet dependencies (package restore) BuildCommit testsContinuous integrationCode analysisBasic functional tests (manual) Version artifactsName the pipeline instance (set the build number) Update the artifact repositoryUpdate the Symbol ServerAcceptance test stage TFS Build Definition, customized from Lab template. MTM for Test Case managementChange configuration (automated) Deployment (automated) MSBuild for WPPowershell for AzureAcceptance tests (automated) MSTest, CodedUI (?) Release stage TFS Build Definition, customized from Lab templateBranch or merge to Release branch (for archiving) Deployment (automated) MSBuild for WPPowershell for AzureSmoke testing (automated) CodedUI (?) Error detection and recovery (resiliency enablement) SCOM-TFS IntegrationIntellitrace in ProductionPreemptive AnalyticsCapacity test stage TFS Build Definition, customized from Lab templateDeployment (automated) MSBuild for WPPowershell for AzureSmoke testing (automated) CodedUI (?) Performance & load testing (automated) Visual Studio perf & load testsEnvironments (automatic provision, locked down so only automated deployments are allowed) Using Lab Manager SCVMM EnvironmentsDevelopment (isolated) Local deploymentto WP and Azure emulatorsTestProductionIteration 6 Pipeline: further improvementsManualtriggerAutomatictriggerVersion Control (code and configuration) TFS Version ControlMain branch (w/ feature toggles) Release branch (for archiving) StagingDeployment & testingtriggered by Test AgentPipeline monitoringCustom solutionUAT stageUAT (manual) Acceptance criteria from TFS requirementsTFS Feedback toolManualtriggerDeployment triggeredby Test AgentUsing deploymentfrom Capacity test stageDeployment & testingtriggered by Test AgentSecurity test stageSecurity tests (manual) WACAAny additional tools? AutomatictriggerUsing deploymentfrom Capacity test stageExploratory test stage TFS Build Definition, customized from Lab templateDeployment (automated) MSBuild for WPPowershell for AzureExploratory testing (manual) MTMManualtriggerExploratoryTestDeployment & testingtriggered by Test AgentArtifact repositoryNuGet ServerSymbol ServerTFS Symbol Server

79. MADRID · NOV 21-22 · 2014
#9 Inspect & Adapt

80. MADRID · NOV 21-22 · 2014
Continuouslyimprovethepipeline
Component or architectural changes.
New skills in the team.
New resources, tools, environments.
Reserve time, and make the team accountable for improvement.

81. MADRID · NOV 21-22 · 2014
Summary
1. Define components.
2. Identifysub-pipelines.
3. Define Stages& Orchestration.
4. Define Environments.
5. Define Steps.
6. Define Automation& Tooling.
7. Define executionmodel, monitoringand metrics.
8. Plan forfutureenhancements.
9. Inspect& adapt.

82. MADRID · NOV 21-22 · 2014
Thanks!
Jose Luis Soria
ContinuousImprovementManager at Ria Financial
jlsoriat@gmail.com-@jlsoriat
http://geeks.ms/blogs/jlsoria
Slides: http://www.slideshare.net/jlsoria
http://aka.ms/releasepipeline