This document summarizes a presentation by Jesse Dowdle on massive continuous integration and light-speed iterations. Dowdle discusses how AtTask implemented agile development across 11 teams using Scrum, with daily internal releases and public releases every 2 weeks. AtTask automated testing at scale with over 800 unit tests, 3300 API tests, and 2000 UI tests running in under an hour. Dowdle describes how AtTask achieved faster release cycles through continuous integration, including transitioning from Selenium 1 to Selenium 2 to reduce test runtime from 4 hours to 30 minutes.
2. Jesse Dowdle
AtTask, Inc.
Senior manager of development for AtTask Inc. Jesse Dowdle directs the engineering efforts of
nearly a dozen agile teams in the U.S. and overseas. Successful engineering in a fast-paced
industry requires constant research in emerging technologies. Jesse drives thought leadership
on continuous delivery, development best practices, and process evolution to help AtTask ship
software at lightning speed. Happiest when he’s building great software, Jesse sees technology,
process, and people as equally important prerequisites for success. A native of Utah, where he
enjoys fast cars in the summer and snowboarding in winter, Jesse tweets at @dowdlemj
and blogs.
4. •Salt
Lake City, Utah
•Software
•Project
as a Service
Management
•Collaborative
•Salt
Work Management
Lake City, Utah
•Software
•Project
as a Service
Management
•Collaborative
Work Management
6. How we did Scrum
Sprint
Sprint
Sprint
How we did Scrum
4 WEEKS
Sprint
Sprint
Sprint
7. How we did Scrum
2 days
Demo
Retrospective
Plan
4 WEEKS
Sprint
Sprint
Sprint
How we did Scrum
2 days
Demo
Retrospective
Plan
4 WEEKS
Sprint
Sprint
Public
Release
Sprint
Public
Release
Public
Release
8. How we did Scrum
2 days
Demo
Retrospective
Plan
4 WEEKS
Sprint
Internal
Internal
Sprint
Internal
Public
Release
Internal
Internal
Sprint
Internal
Public
Release
Internal
Internal
Internal
Public
Release
Internal
Public
Release
How we did Scrum
2 days
Demo
Retrospective
Plan
4 WEEKS
Sprint
Internal
Internal
Sprint
Internal
Public
Release
Internal
Internal
Sprint
Internal
Public
Release
Internal
Internal
9. Then
Scripted scenarios
Slow, brittle
Monthly Releases
3-5 days acceptance
Manual administration
Now
Tight unit-like UI tests
Fast, Robust
Daily releases
30-45 minutes
Sign-off and go
10. What is Continuous Integration?
“Continuous Integration is a software development
practice where members of a team integrate their
work frequently, usually each person integrates at
least daily - leading to multiple integrations per day.
Each integration is verified by an automated build
(including test) to detect integration errors as
quickly as possible. Many teams find that this
approach leads to significantly reduced integration
problems and allows a team to develop cohesive
software more rapidly.” -- Martin Fowler
Components
•Version
•Build
•Test
Control System
Server
Runner
•Test
Suite
•Notification
Mechanism
11. Basic Integration
•Version
•Build
•Test
Control triggers new action
Server retrieves latest version, compiles, starts Application Stack
Runner executes tests against Application Stack
•Results
•Build
are collected and sent by notification
Server is cleaned and restored.
Basic Integration
15. Massive Test Automation
•800
Unit tests
•3300
API tests
•2000
UI tests
•55,000
hours per month
•Release
Acceptance + CI
“True” Continuous Integration
•How
often do I integrate?
•What
tests tell me I’m integrated?
•What
must I know to release?
20. Selenium Grid
Selenium IE
Grid + AtTask
Commit
Install
Selenium FF
Build Installer
Build Tests
Other tests
Unit
Integration
1/4
3/4
2/4
4/4
ATTASK’S PIPE
BUILD -- DEPLOY -- TEST -- DESTROY
HOW TO MAKE A PAPER AIRPLANE
OR, QUEUEING THEORY AS EXPRESSED BY A FOLDED PROJECTILE
Tear Down
21. Selenium Grid
Selenium IE
Grid + AtTask
Commit
Install
Selenium FF
Tear Down
Build Installer
Build Tests
Other tests
Unit
Integration
1/4
3/4
2/4
4/4
SERIAL VS PARALLEL
Selenium Grid
Selenium IE
Grid + AtTask
Commit
Install
Selenium FF
Build Installer
Build Tests
Other tests
Unit
Integration
1/4
2/4
SERIAL VS PARALLEL
3/4
4/4
Tear Down
28. Scalability
•Taking
•To
the Test Suite to massive scale
the cloud with Amazon Cloud Formations
•Dynamic
Selenium Grids
•Dynamic
AtTask Environments
Divide and Conquer
•Module
separation will allow for scale
•Test
Suites should support sharding
•Run
tests with different dependencies in parallel
29. Divide and Conquer
Selenium Grid
Selenium IE
Grid + AtTask
Commit
Selenium FF
Install
Build Installer
Build Tests
Other tests
Unit
Integration
1/4
3/4
2/4
4/4
Divide and Conquer
Unit
Integration
1/4
3/4
2/4
4/4
Tear Down
43. SLAVE
SELENIUM GRID
SLAVE
SLAVE
SELENIUM GRID
SLAVE
SLAVE
IT’S RAINING GRIDS
Visibility
•Test
results in Jenkins
•Description
•Theres
•View
•All
plugin
my commit!
plugin
the jobs on one line, oh the humanity!
•Screen-shotting
•Test
UI Tests
Age, Project Health
44. Visibility
•Test
results in Jenkins
•Description
•Theres
•View
•All
plugin
my commit!
plugin
the jobs on one line, oh the humanity!
•Screen-shotting
•Test
UI Tests
Age, Project Health
Visibility
•Test
results in Jenkins
•Description
•Theres
•View
•All
plugin
my commit!
plugin
the jobs on one line, oh the humanity!
•Screen-shotting
•Test
UI Tests
Age, Project Health
45. Visibility
•Test
results in Jenkins
•Description
•Theres
•View
•All
plugin
my commit!
plugin
the jobs on one line, oh the humanity!
•Screen-shotting
•Test
UI Tests
Age, Project Health
Visibility
•Test
results in Jenkins
•Description
•Theres
•View
•All
plugin
my commit!
plugin
the jobs on one line, oh the humanity!
•Screen-shotting
•Test
UI Tests
Age, Project Health
47. Accountability
•Claim
/ Blame test results using AtTask
CI as a Platform
•On-Demand
•Easy
demo environments
button for version control
•Visibility
•Other
into production systems
quality measurements
•Static
Analysis
•Code
Coverage
48. Staffing for Massive CI
•Get
the right skill set
•Watch
your integration points
•Technological
•Organizational
•DevOps
Adoption
•Process
/ Tool alignment
•Information
•Integration
•Scrum
Radiators
with existing tools
board
•Backlog
•Integration
with Release Management
49. CI EVOLUTION
ITS A CONTINUOUS INVESTMENT
What does it cost?
•3
FTE
•Two
human, one machine(s)
•Keeping
•Efficient
costs down
machine sizes
•Rock-solid
•Elasticity
tear down
50. Return on investment
•5
days (40 hours) to certify one release...
•10
engineers
•$40
per hour
•$20,000
•30
minutes to certify one release...
•10
engineers
•$40
per hour
•$250
To release 12 times in a year...
$240,000 TO CERTIFY
$3000 TO TEST
$160,000 FOR TWO FTE
$77,000 FOR CI
To release 24 times in a year...
$480,000 TO CERTIFY
$6000 TO TEST
$160,000 FOR TWO FTE
$77,000 FOR CI
To release 100 times in a year...
$2,000,000 TO CERTIFY
$25000 TO TEST
$160,000 FOR TWO FTE
$77,000 FOR CI
52. 100 deployments annually
$265,000
Where to go from here
•Continuous
Delivery
•Application
Lifecycle Management
•Continuous
Integration of Infrastructure
•Open
Source
54. Thank you!
•Jesse
Dowdle, Sr Manager of Development
•@dowdlemj
•linkedin.com/in/jessedowdle
•www.continuousdeliverist.com
•David
Tolley, Sr QA Automation Engineer
•@davidtolley
•linkedin.com/profile/view?id=81007074