This document describes how a company achieved continuous delivery of software through breaking down bottlenecks, automating processes, and establishing a feature pipeline. The key steps included forming feature teams, deploying features to isolated environments, implementing automated testing and deployment, and removing dependencies on operations for releases. This enabled the goals of lower risk, faster feedback, and releasing multiple times per day while maintaining quality. Ongoing work includes further automating infrastructure configuration and deployments.

1. CONTINUO
US
DELIVERY
ALEXEY TURCHANIKOV, INFRASCALE
oleksiy.turchanikov@gmail.com

2. #1 WHAT IS THIS?
#2 HOW WE ACHIEVED THIS
#3 CONSEQUENCES
#4 DETAILS

3. WHAT IS
THIS?

4. PIPELINE
build unit test
deploy to
feature env
UI/func test manual QA
merge to
mainline
build unit test
deploy to
integration
UI/func test manual QA
deploy to staging
manual QA
(optional)
deploy to production manual QA
 Story on production as a part of DoD
 5 days per story on average
 1-3 releases per day
 2-4 hours to release hotfix
 15 isolated QA environments
feature
integration
staging production

5. GOALS
Lower risks
Visible progress, peace of mind
Fast feedback loop

6. INGREDIENTS
Fast feedback for every change
Automation
Collaboration
Continuous improvement

7. HOW WE
ACHIEVED
THIS

8. BEGINNING
Ukraine
Backend
Frontend
Desktop
Mobile
QA /
Regression
Sysadmins
Partners
Afghanistan
India
10hours,language
Product Owner
USA

9. BOTTLENECK:
COMPONENT TEAMS
CHANGE: FEATURE
TEAM

10. BOTTLENECK:
REGRESSION TESTING
CHANGE: SMALL
STORIES, AVOID
REGRESSION

11. BOTTLENECK:
DEPLOYMENT VIA IT OPS
CHANGE: SELF-SERVICE
DEPLOYMENT,
EXTERNAL
CONFIGURATION

12. BOTTLENECK: BUGS IN
TRUNK
CHANGE:
FEATURE
BRANCHES

13. BOTTLENECK: QA
ENVIRONMENT
CHANGE: VIRTUAL
ENVIRONMENTS, CM
AUTOMATION

14. BOTTLENECK:
INFRASTRUCTURE
CHANGE: DEVOPS

15. BOTTLENECK: QA
CHANGE:
AUTOMATION

16. UPCOMING AND
ONGOING
DevOps: DSC-based
configuration
Production deployments by QA
Build-Test-Deploy-Test speedup
Learning Management System

17. DETAILS

18. FEATURE PIPELINE:
BUILD, UNIT TEST
build unit test
deploy to
feature env
UI/func test manual QA
merge to
mainline
Full pipeline for every commit
Parallelized
Tests:
Only fast tests
Unit test coverage: <25%
Fail on metric changes
Output: custom package file

19. FEATURE PIPELINE:
DEPLOY
build unit test
deploy to
feature env
UI/func test manual QA
merge to
mainline
 Isolated environments, VM based on template
 One-step deployment of whole stack
 Includes DB migrations, IIS, MSMQ…
 Same package for Integration, Staging,
Production
 Environment-specific configuration in a single
file under source control + conventions
 PowerShell as a glue

20. FEATURE PIPELINE: UI /
FUNCTIONAL TESTS
build unit test
deploy to
feature env
UI/func test manual QA
merge to
mainline
 Test data auto-created
 Selenium & WebDriver for Web
 White for PC
 Priorities:
 OS-dependent functions (time saver)
 Common workflows (safety net)
 New functionality
 Optimizations:
 SSDs
 Run in parallel
 Headless browsers – no luck
 Test execution (and failure!) recorded as video

21. FEATURE PIPELINE:
MANUAL QA
build unit test
deploy to
feature env
UI/func test manual QA
merge to
mainline
Feature-level
Scope defined with developer
Immediately available for
testing

22. FEATURE PIPELINE:
STAGING, PRODUCTION
One-by-one
Same package
Partial deployment – tier,
application or web service
Deployment by developer or QA
Risk-based manual QA
deploy to staging
manual QA
(optional)
deploy to production manual QA
staging production

23. PIPELINE ON BUILD
SERVER
Jenkins
TeamCity

24. ORGANIZATION
Feature teams
Offset sprint schedules
Architecture committee
Cross-training, rotation, Lunch
& Learn, L3 support “shifts”

25. CONTROL CHART
Rally
Jira

26. CONSEQUE
NCES

27. PRODUCT
MANAGEMENT
Good:
 Clear DoD for business
 Faster integration
 No “surges” and overtimes
Watch for:
 Keep Sales, Support up to date
 Roadmap is a must
 Global (e.g. architectural) changes

28. SALES, MARKETING
Fast response to change
Virtual “Big Bang” releases

29. SUPPORT
Fast turnaround for fixes and
feature requests
2-4 hours to release hotfix
Need continuous learning

30. QUESTIONS