The document discusses continuous testing and its role within Agile methodologies, emphasizing the importance of fast feedback loops in software development. It outlines key concepts such as continuous integration, deployment, and testing processes, while introducing the GROWS methodology to improve the software development life cycle. The document also highlights the benefits of collaboration between development and QA teams and provides practical exercises to foster understanding and adoption of these practices.

1. Continuous Testing
Jared Richardson
Agile Artisans
@JaredRichardson
GrowsMethod.com

2. Who am I?

3. 3
Cofounder of GROWS Methodology
Author Ship It! and Career 2.0
Screencast editor PragProg.com
2nd public signatory of the Agile Manifesto
Started AgileRTP in 2007
Coach
AgileArtisans.com
Jared Richardson

4. Who are you?

5. What’s up with the material?
Major rewrite
Introduction
Slides on USB
On Slideshare.com
Provided to conference as well

6. Who’s a User?
Already using continuous integration?
Continuous deployment?
Which tools?
Test first?

7. What would you say you do
here?
Roles?
Responsibilities?
Why are you here?
What are your tests for this session?

8. Objectives
Understand Continuous Feedback in context of a healthy SDLC
Understand the Continuous Paradigm
Understand an end-to-end Continuous workflow
Understand how to scale your efforts

9. Outline
GROWS Methodology
Continuous paradigm
Battleship!
Continuous Integration (benefits & workflow)
Continuous Testing (benefits & workflow)
Miscellaneous tips & tricks

10. Breaks
Halfway (90 minutes)?
Each hour ?
Speak up!

11. SDLC

12. Idea
Verifies
Use or
delete
Implements
Software Development Life Cycle

13. wikipedia.org/wiki/Waterfall_model#/media/File:Waterfall_model.svg
Traditional Approach

15. Requirements
Write Code
The Traditional Approach
Demo
Rewrite (60%)
“Discussion”
The architects
generate a
document and
hand it off
An isolated event,
primarily worked by
the customer proxy
Arch
Code like mad!
We’re starting
late!
Not what we meant!
Wrong arch
Ignored my docs
etc
so on

16. Demo
Write Code
Requirements
A More Agile Approach
Before
significant time
and $$ is
invested
Discuss
Include the
Golden Triad
Dev QA
Customer
Rewrite

17. Demo
Write Code
Architecture
Proof of Concept
Requirements Arch

18. Why Feedback?
Feedback is good for budgets
Cheaper
Faster
More accurate
Lower risk

19. flickr.com/photos/alancleaver/4293345631
To Err is Human
What’s your recovery plan?

20. Experimentation
Learn about software by writing software
Can’t study your way into experience
Like riding a bike
Create feedback loops
flickr.com/photos/schwar/6801765768

21. Requirements
Rewrite
Discuss
Node.js
Bootstrap
Angular.js
Pick one
SOA
Monolithic
Peer to peer
Pick one
Developers
disagree on
which architecture
is best
Continue until you can demo
Developers
disagree on
which Javascript
toolkit is best
Demo
Revise
Rewrite
Adjust
Frequent demos discover communication failures early

22. Discuss
Intentional
Experimentation
Study
Experiment
Learn

23. Decide! Learn
Learn Decide!
Two Paths

24. GROWS Methodology

25. Directed Empiricism

26. Why Bother?
https://www.mountaingoatsoftware.com/blog/agile-succeeds-three-times-more-often-than-waterfall

27. Acceptance
We’re not very good
We’re going to mess up
Mistakes are inevitable
How do we recover?

28. Continuous
Paradigm

29. Experimentation
We’re not very good
We’re going to mess up
Mistakes are inevitable
How do we recover?

30. STICK Experiments
Short
Time boxed
Informative
Concrete
Key
Specific
flickr.com/photos/peterpearson/1062279922

31. Automated Tests
Answer Questions
Is this done?
Does it still work?
Does it work somewhere besides my machine?
This is what I’m asking you to write.
Everything is checked in.

32. flickr.com/photos/bobowen/4501875907
Product or system
flickr.com/photos/30691679@N07/3808395213
Module/
Sub System
flickr.com/photos/kevino/1438502858
Class

33. Feedback Loops are
Fractal
flickr.com/photos/dancentury/4579906149

34. Agility Is...
Tighter Feedback Loops

35. Which Level?
35

36. 36
Days
Iterations
Months
Quarters
Years
Optional
Optional

37. C*/Board
37

38. VP
38

39. Director Level
39

40. Management
Level
40

41. Developer
Level
41

42. flickr.com/photos/89228431@N06/11348332905
Minimize Our Dark Times
Code Change
Validation

43. Fast feedback
leads to fast
fixes
flickr.com/photos/bogenfreund/556656621

44. Different Levels
Manual acceptance tests
Unit tests
Continuous Testing

45. Different Levels
Good at the bottom
Great at the top
Ignore the middle

46. Continuous Feedback
Continuous Integration
Continuous Delivery
Continuous Testing
Continuous Monitoring

47. Continuous deployment is controversial.
Most people ... think I’m advocating
low-quality code
or an undisciplined
cowboy-coding
development process.
Eric Ries
radar.oreilly.com/2009/03/continuous-deployment-5-eas.html

48. Language is a Tool
Use it

49. Exercise
How long is your SDLC (idea to verification)?
What are your SDLC feedback bottlenecks?
Write down 3 worst bottlenecks
Gather into groups of 5 to 7
Share and discuss
Keep your list!
Take ten minutes

50. Continuous Integration

51. Continuous Integration
Watch Code
Build product
Run tests
Publish results

52. Pros
Fast feedback
Known state
Easily consumed history
Shines a light
52

53. Cons
Slows you down
Nags you after every code change
Exposes problems
Shares issues
Forces you to deal with problems
Smoke alarm
53flickr.com/photos/117168856@N06/12486837493

54. 54
Work
Time
Perception of Problem Accumulation

55. 55
Work
Time
Actual Problem Accumulation

56. 56
Work
Time
Continuous Problem Resolution

57. Sidebar: How We Learn
Dreyfus Model of Skills Acquisition
57

58. 58
Start
Familiar
Working
Smooth
Teaching
Dreyfus
Model
of
Skills
Acquisition
Steps
Recipes
Intuition
Stuck
Principles

59. Key Points
Skills don’t map
59dilbert.com/strip/1990-09-13

60. Key Points
Beginners need steps
60

61. Steps
1. Use source code management (SCM)
2. Check in ~everything~
3. Automate the build (script)
4. Acquire a build environment (isolated)
5. Install build tools (dev tools)
61

62. Java Tooling
Publish results

63. Continuous Integration Servers
Many
Jenkins
Cruise Control
Bamboo
Pulse
TFS
63

64. Jenkins
Open source
Established
Split from Hudson
Java application
Any application server
Self contained
64

65. Example
Publish results
javac (Oracle & IBM)
message broker
etc
ws ejb deploy
message broker tools
sshexec (to scripts)
JUnit
HP's UFT
Selenium
Cucumber
Web page
RSS
Email

66. Common Scenarios
66

67. Dependent Builds
My build depends on another team’s build
No problem
We’ll trigger your build based on their results
67

68. Dependent Builds
Services
Client/GUI
Data Layer

69. Complicated Dependencies
Our software is too complicated for CI
69
flickr.com/photos/chaoticmind75/16169722270/

70. Dependent Builds
Data Layer
Data Services
Rich Client
Common Libs
Web Portal
Biz Services

71. What’s Big?
3,000 developers
300 major projects
All related
Platform to UI
Multiple branches
71

72. Common Integration Build
72

73. Chained Continuous Integration
Team 1 Team 2 Team 3
Integration
build

74. Multiple Platforms
74

75. Platform Testing
Team 1 Team 3Team 2 Team 4 Team 5
Integration Build
Platform1 Plat 3Plat 2 Plat 4 Plat 5

76. Battleship!

77. Each player
has a board
flickr.com/photos/spookyamd/6333317567
flickr.com/photos/jking89/1363983490
Place your fleet
Drop your bombs

78. Two Ways
to Play
flickr.com/photos/compacflt/22797421725
flickr.com/photos/bz3rk/2436165683
Use all your ammo quickly
Use ammo one shot at a time

79. Round 1
Start a timer
Place your fleet
Use all your “ammo”
Check with your opponent
Record hits/misses/time
Play twice
Round 2
Start a timer
Place your fleet
Fire 1 round
Check with your opponent
Continue until out of ammo
Record hits/misses/time
Ships & Ammo
1 Sub (2 dot ship)
2 Destroyers (3 dots ships)
1 Battleship (4 dot ship)
1 Aircraft carrier (5 dot ship)
35 rounds to fire

80. Results
Which way is faster?
Which spends less “budget”?
Which way is more effective?
Which way do you prefer?

81. Feedback
Faster with feedback?
Slower with feedback?
Is slower bad?

82. Efficient or Effective

83. A B C D E F G H I J
1
2
3
4
5
6
7
8
9
10
Opponent’s Fleet
A B C D E F G H I J
1
2
3
4
5
6
7
8
9
10
My Fleet
A B C D E F G H I J
1
2
3
4
5
6
7
8
9
10
Opponent’s Fleet
A B C D E F G H I J
1
2
3
4
5
6
7
8
9
10
My Fleet

84. A B C D E F G H I
1
2
3
4
5
6
7
8
9
A B C D E F G H I
1
2
3
4
5
6
7
8
9
A B C D E F G H I
1
2
3
4
5
6
7
8
9
A B C D E F G H I
1
2
3
4
5
6
7
8
9
A B C D E F G H I
1
2
3
4
5
6
7
8
9
A B C D E F G H I
1
2
3
4
5
6
7
8
9
A B C D E F G H I
1
2
3
4
5
6
7
8
9
A B C D E F G H I
1
2
3
4
5
6
7
8
9
Time Shots Misses Hits Sinks % hits % sinks
1st All at once
2nd All at once
3rd Interactive
4th Interactive
Battleship! An exercise by Tony
Brill1 2 3 4

85. Continuous Testing

86. Continuous Integration
Watch Code
Build product
Run tests
Publish results

87. Continuous Testing
Watch CI
Deploy product
Integration
tests
Publish results

88. Continuous Testing
CI CT
Compile &
unit test
Deploy &
integration test

89. Watch
Build
Unit Tests
Publish
Continuous
Integration
AgileArtisans.com

90. Watch
Build
Unit Tests
Deploy
to Dev
Integration
Tests
First
Look
Continuous
Testing
AgileArtisans.com

91. SCMDevelopers
Watch
Build
Unit Tests
Deploy
to Dev
Integration
Tests
First
Look
Continuous
Testing
On
Demand
Deploy
to QA
Manual
Testing
AgileArtisans.com

92. Continuous Testing
Build in CI (compile & unit tests)
Deploy to dev (staging)
Run integration tests

93. What does this tell us?
If it works

94. How?
By keeping fast feedback fast
Providing slow feedback slowly
Don’t tie fast feedback to slow feedback

95. Fast vs Slow
“Fast”
Integration
Tests
Compiles
& Unit Tests
Slow
Integration
Tests
5-10 minutes
30 to 60
4 to 8 hours

96. What's this look like?

97. Continuous Testing
CI CT
Compile &
unit test
Deploy &
integration test

98. Multi Platform Deploys
CI
CD Win7
CD Linux
CD OS X

99. Exercise
Remember your SDLC bottlenecks?
What could continuous integration solve?
What could continuous testing solve?
Regroup with your team and discuss
Bring your top 3 solutions
99

100. As We Explore a
Continuous World...
10
0

101. What Concepts Emerge?
10
1

102. Nuke and Pave
Reset (completely)
Start over
Completely Scripted
10
2

103. Binaries
Nuke & Pave
Delete
Recompile
Deploy
Use
10
3
flickr.com/photos/x-ray_delta_one/4152356464

104. Data
Nuke & Pave
Delete schema
Drop tables
Recreate
Restock
Rails
Liquibase
10
4
flickr.com/photos/x-ray_delta_one/4152356464

105. Rails ActiveRecord
CREATE TABLE products (
id int(11) NOT NULL auto_increment,
name varchar(255),
PRIMARY KEY (id)
);
10
5

106. Rails ActiveRecord
change_table :card_deck do |table|
tablet.remove :description, :name
tablet.string :first_name
tablet.index :part_number
tablet.rename :list_name, :last_name
end
10
6

107. Rails ActiveRecord
rake db:migrate
rake db:migrate VERSION=0
rake db:reset
10
7

108. ActiveRecord
rake db:reset
Drops schema
Recreates schema
10
8

109. Operating System
Nuke & Pave
Virtualize
Recreate
10
9
flickr.com/photos/x-ray_delta_one/4152356464

110. Steps for a New OS
Install VirtualBox
Install Vagrant
vagrant init hashicorp/precise32
vagrant up
11
0

111. Benefits
Scriptable
Reproducible environment
Solid results
Fast results
Repeatable results

112. When?
Day one
Easiest

113. “If you don’t do it on Day One,
you probably won’t”
Andy Hunt
113andy.pragprog.com

114. Easy?
No
Difficult
Necessary

115. Costs Move to Zero
Once scripted…
What changes?

116. How do We Improve Our
Feedback Loops?

117. Developers
Devs write
code
normally
SCM
Code is shared
via a source
code
management
tool
Continuous
Testing
Providing
immediate
feedback and a
shared
environment
for a QA “First
Look”
Triggered
Deploy
to QA
QA team
members
trigger this
deploy
Manual
Testing
A stable
environment for
manual and
exploratory testing

118. SCMDevelopers
Watch
Build
Unit Tests
Deploy
to Dev
Integration
Tests
First
Look
Continuous
Testing
On
Demand
Deploy
to QA
Manual
Testing
Rapid feedback
Pair Dev & QA

119. First Look
• QA can see what’s been coded immediately
• Differences in opinions can be discussed quickly
• Fewer problems during the larger testing cycles
• Keeps QA & Dev working together as partners
• Minimizes feedback for the previous iteration
• Faster feedback for QA

120. Continuous Testing
• Includes QA in the continuous feedback cycle
• Retains a traditional QA stable testing environment
• Dev & QA should both review all tests (unit & int.)
• QA requests trigger the QA env. deployment

121. Alternatives

122. flickr.com/photos/lightsinthedark/5810724330/
Work in a vacuum

123. Builds
Deploys
Tests
flickr.com/photos/pmillera4/931792447
4
Ruthlessly Automate

124. Be Lazy!
You have builds scripted
CI servers running
Tests written
Things deploy
What else can you do?
flickr.com/photos/ofernandezberrios/2720567252

125. Dev
Environments
QA
Staging
Prod
5 minutes
Integration
Tests
8:15am
8:20am
8:50am
8:25am
8:30am
8:35am
8:40am
8:45am

126. Traditionally…
Someone needs the app
Notices it’s down
URGENT!
FIX!
Fire drill
flickr.com/photos/kaibara/5332191131

127. Detect Problems
Before it’s urgent
Detect
Notify
Fix

128. Wait
5 minutes
Integration
Tests
Continuous
Monitoring
Notify

129. Feedback Loops
Identify
Reduce
Minutes
Not days
Never weeks

130. Months
flickr.com/photos/parkstreetparrot/9764446493

131. Steps
flickr.com/photos/jalalspages/842122500

132. Steps
Continuous Integration (aka compiles)
flickr.com/photos/jalalspages/842122500

133. Steps
Continuous Integration (aka compiles)
Test coverage numbers (aka unit tests)
flickr.com/photos/jalalspages/842122500

134. Steps
Continuous Integration (aka compiles)
Test coverage numbers (aka unit tests)
Continuous deployment (aka deploys to dev/test)
flickr.com/photos/jalalspages/842122500

135. Steps
Continuous Integration (aka compiles)
Test coverage numbers (aka unit tests)
Continuous deployment (aka deploys to dev/test)
Continuous testing (aka integration tests)
flickr.com/photos/jalalspages/842122500

136. Steps
Continuous Integration (aka compiles)
Test coverage numbers (aka unit tests)
Continuous deployment (aka deploys to dev/test)
Continuous testing (aka integration tests)
Continuous monitoring (test against environs)
flickr.com/photos/jalalspages/842122500

137. Culture Change
Fundamental change in development

138. Steps
Automate builds
Continuous Integration
Add unit tests
Automate deployments
Continuous Delivery
Add integration tests
flickr.com/photos/untitlism/25474234
65

139. Your Move

140. Exercise
At lunch
Continuous Testing
Two 3x5 cards
3 reasons for
3 barriers to adoption
Groups of 5 to 7
Brainstorm on solutions
flickr.com/photos/curtisperry/6119618990

141. Exercise
Useful or not...
Will you pursue or not...
Continuous Integration
Continuous Testing
Continuous Monitoring
Test automation
How will these ideas help or hurt your org?
Write down your answers and pass them up

143. GrowsMethod.com
AgileArtisans.com
jared@GrowsMethod.com

145. An Addendum

146. Test Automation
Unit tests
Acceptance Tests

147. Unit Tests
One class
One method
Fast
Illuminating

148. Integration Tests
High level
Lots of coverage
No detail

149. Continuous Feedback
Large amounts of code
Small amounts of change
Tests run after every change
Requires frequent code checkins

150. What's Better?
Unit or Integration?
150

151. Best Is...
Automated
Reliable
Frequently executed
151

152. Test Lifecycle
Set up state
Use product
Verify results
(Clean up)
152

153. Set Your Teams Up
Builds work
Functionality works
Biggest risk?
Requirements

154. Developers Testers
Customers
The Team

155. Grid Driven Development
Remove technical barriers
Everyone speaks “Excel”

156. Testing Strategies
Test First is best
Defect Driven Testing
Blitzkrieg Testing

157. Tips
Open standards...
Junit or Junit compatible
Avoid vendor lock in
Glue tools
Time matters....
Fast servers
Incremental change

158. Script Tips
Save scripts in source code management
Modular
Basic encapsulation
One target => One function

159. Script Targets
all depends_on clean, compile, test-unit
clean # deletes build artifacts
compile # builds the product (Ruby?)
test-unit # run your tests
deploy_locally # to your dev environment
deploy_dev (?) # to a shared dev environment
test-integration # run your integration tests

160. Resources
JUnit JUnit.org
Jenkins Jenkins-ci.org
BDD Dannorth.net/introducing-bdd
Cucumber Cukes.info
Continuous Deployment en.wikipedia.org/wiki/Continuous_delivery

161. Devops Resources
VagrantUp.com
GetChef.com/chef
PuppetLabs.com
Ansible.com
VirtualBox.org

162. Data Resources
Guides.RubyOnRails.org/active_record_basics.html
Liquibase.org
FlywayDB.org

163. Additional Content
Material for you to review later…

164. Release Planning
Releases contain features
How many are ready to ship?
164

165. Releases contain features
165
Team
Release
Date
Release
Theme
#
Features
#
Stories
Stories
Complete
%
Done
1
2
3

166. Releases contain features
Team Release Features Stories
Stories
Complete
% Done
A 7/1 25 392 17 0.23
B 7/15 13 208 104 0.5
C 7/12 96 2,400 150 0.16
D 12/30 476 ?? 234 ?

167. Release date
Burn Up Chart
167
Time
Features
Features in
the release
Required
rate of work

168. Will we hit our date?
168
Time
Features
ACTUAL
rate of work

169. Burn Up Chart
169
Time
Features
ACTUAL
rate of work

170. Feature Creep
170
Time
Features
ACTUAL
rate of work

171. Observe Progress
On track
Behind
Unrealistic
171

172. The Solution
Reduce features
DO NOT focus on increased work
172

173. Good or Not?
173
Time
Features
ACTUAL
rate of work

174. What’s the Fix?
174
Time
Features
How do I
make the team
work harder?

175. 175
Longer
Hours
Work Faster!Even
Later
More
Mistakes
Lots of
Debugger
Time

176. What’s the Fix?
176
Time
Features

177. What’s the Fix?
177
Time
Features

178. What’s the Fix?
178
Time
Features

179. What Happens to Overloaded Planes?
179
flickr.com/photos/jameski/5322886899