Agile practices, although applicable to a variety of situations, are most commonly applied to IT projects, generally for software development. Can you apply agile methods to just part of a software implementation project? Todd Jones presents this case study where agile techniques were applied to the testing phase of a multiyear, multimillion-dollar IT program that included replacing a legacy system, new software development, creation of a new enterprise data model and document management solution, and complex financial balancing. After briefly describing the challenges faced by the organization, Todd covers each guiding principle from the Agile Manifesto, describes the practical approach he used to implement agile techniques, and shows information radiators that were used to track velocity and demonstrate progress to program stakeholders. Todd outlines the key benefits this brought to the business as well as lessons the team learned along the way. Leave with a greater appreciation for the testing complexity of a large-scale software implementation, a better understanding of agile practices and lessons learned, and practical techniques and tracking tools you can apply to your current project.

1. AT5
Concurrent Session
11/9/17 11:30 AM
Put Agile to the Test: A Case Study for
Test Agility on a Large IT Project
Presented by:
Todd Jones
Westfield Insurance
Brought to you by:
350 Corporate Way, Suite 400, Orange Park, FL 32073
888---268---8770 ·· 904---278---0524 - info@techwell.com - https://www.techwell.com/

2. Todd Jones
Westfield Insurance
Todd Jones has spent his entire career with Westfield Insurance—as a developer,
architect, testing leader, and project manager, having led dozens of business and
technology projects with a proven track record of satisfied sponsors and team
members. He led the testing project team for the case study—a $90 MM
program comprised of six projects to replace an aging legacy system. An active
member of the Project Management Institute, he serves on the board of the
Northeast Ohio Chapter. He was recognized last year for his contributions to the
profession with the Dr. Harold Kerzner Award for Project Management
Excellence. Connect with Todd on LinkedIn.

3. 10/16/17
1
Todd R Jones
PMP, CSM, PMI-ACP
Putting Agile to the Test:
A Case Study for Test Agility on a Large IT Project
©2017 Todd R Jones
Who am I?
Todd R Jones
PMP, CSM, PMI-ACP

4. 10/16/17
2
Learning Objectives
• Understand how agile techniques can be applied to the testing
portion of a large project - via a case study
• Appreciate the scope, challenges, approach, artifacts, benefits
and lessons learned for our project
• Be able to apply agile techniques and tracking tools to projects
in your work environment
Setting the Stage

5. 10/16/17
3
Challenges
• Testing unable to start as planned due to missed
development complete milestones
• Internally, multiple test systems configured differently
• Vendor test systems did not align to our test environments
• Questionable stability of the System Integration Test (SIT)
environment
• Changing business requirements – lack of traceability
• Mix of IT and Business testers
• Communicating progress was not consistent
So what did we do?
• Threw everything out the nearest window and told management
we were going Agile!

6. 10/16/17
4
Actually we did adopt key agile principles…
Note: No windows
were broken in the
delivery of this
project!
• Development was not complete on time for testing to start
as planned
• Internally, multiple test systems configured differently
• Vendor test systems did not align to our test environments
• Questionable stability of the System Integration Test (SIT)
environment
• Changing business requirements – lack of traceability
• IT testers and Business testers
• Communicating progress was not consistent
Given these Challenges, what principles
from the Agile Manifesto can you apply?
Break into small groups and discuss for 5-6 minutes.

7. 10/16/17
5
The meeting (re)starts in
minute
1
Small Group Exercise
The meeting (re)starts in
seconds
45
Small Group Exercise

8. 10/16/17
6
The meeting (re)starts in
seconds
30
Small Group Exercise
The meeting (re)starts in
seconds
15
Small Group Exercise

9. 10/16/17
7
The meeting (re)starts
NOW!
Small Group Exercise
• Development was not complete on time for testing to start
as planned
• Internally, multiple test systems configured differently
• Vendor test systems did not align to our test environments
• Questionable stability of the System Integration Test (SIT)
environment
• Changing business requirements – lack of traceability
• IT testers and Business testers
• Communicating progress was not consistent
Given these Challenges, what principles
from the Agile Manifesto can you apply?

10. 10/16/17
8
Expected AVG Closed
Avg Days
to Fix
Closed
Avg Days
to Fix
Closed
Avg Days
to Fix
Closed
Avg Days
to Fix
Closed
Avg Days
to Fix
Closed
Avg Days
to Fix
Closed
Fatal 1.0 3.9 7.0 2.1 1.0 13.0 1.0 6.0 1.0 5.0 0.0 NA 0.0 NA 0.0
Critical 3.0 7.5 154.0 7.1 12.0 11.4 3.0 3.7 8.0 6.3 4.0 6.5 0.0 NA 5.0
All Projects Defect
Count
CMS Services Payments
Test Case Execution and Business Scenarios Test Case Execution and Business Scenarios
Plan Defect % Plan Defect %
SIT 2
Current Defect
Details
SIT 2 Critical / Fatal Defect Trend
Defect Quality Defect Quality
Blocked Scenarios
Graduated To Plan
9
Scenarios Graduated
20 315
Scenarios Graduated
Test Cases
Blocked by
Incomplete
Development
1 47 3 0
of 24 of 22 of 22
Scenarios Graduated
19 15
Graduated To Plan Graduated To Plan
38% 86% 71%
Test Case Execution and Business Scenarios
Blocked
Scenarios
Test Cases
Blocked by
Incomplete
Development
Blocked
Scenarios
Test Cases
Blocked by
Incomplete
Development
ECF Reporting PODS
Plan Defect %
Actual Defect % Actual Defect % Actual Defect %
6% 5% 1%
Legacy
Avg Days to Fix
NA
10.4
All Projects
7/18/2014 - There are 3 blocked scenarios: #3 (def 8237),
#7 (def 8281), and #21 (def 8285).
7/16/2014 - There are 5 blocked scenarios: #3 (def 8237),
#7 (def 8281), #12 (def 8291), #16 (def 8291), and #21
(def 8285).
7/14/2014 - There are 5 blocked scenarios: #3 (def 8237),
#7 (def 8281), #12 (def 8291), #16 (def 8291), and #21
(def 8285).
Actual Defect % to Plan Actual Defect % to Plan Actual Defect % to Plan
0
13
20% 40% 10%
Run 1 Run 2
Defect Quality
Run 3
SIT 2 - as of 7/18
Defects found in SIT 2
Daily - Open SIT Count Weekly - All SIT Defects Turnaround Analysis - Last Run 3/13/2014
0%0%0%1%1%1%1%1%
2%2%
10%
18%18%18%
23%
29%
33%
42%
30%30%30%30%30%31%32%32%32%32%
35%36%36%
0%
10%
20%
30%
40%
50%
60%
17-Feb
18-Feb
19-Feb
20-Feb
21-Feb
22-Feb
23-Feb
24-Feb
25-Feb
26-Feb
27-Feb
28-Feb
1-Mar
2-Mar
3-Mar
4-Mar
5-Mar
6-Mar
7-Mar
8-Mar
9-Mar
10-Mar
11-Mar
12-Mar
13-Mar
14-Mar
15-Mar
16-Mar
17-Mar
18-Mar
19-Mar
20-Mar
21-Mar
Plan
Actual
4%
6%7%8%8%8%8%
10%
12%
17%
21%
23%23%23%
27%
31%
35%
39%
43%43%43%
45%
47%
49%
52%
54%54%54%
56%57%57%
59%
63%63%63%63%
65%
68%68%
70%70%70%70%71%72%
74%75%75%75%75%
77%78%79%79%79%79%79%79%80%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
24-Mar
26-Mar
28-Mar
30-Mar
1-Apr
3-Apr
5-Apr
7-Apr
9-Apr
11-Apr
13-Apr
15-Apr
17-Apr
19-Apr
21-Apr
23-Apr
25-Apr
27-Apr
29-Apr
1-May
3-May
5-May
7-May
9-May
11-May
13-May
15-May
17-May
19-May
21-May
23-May
25-May
27-May
29-May
Plan
Actual
0
9%
14%
17%
20%
29%
37%
40%
46%
52%
58%
68%68%68%
72%
77%79%81%83%83%83%83%
84%84%85%86%
87%87%87%87%87%87%87%89%
0
0.2
0.4
0.6
0.8
1
1.2
15-Jun
16-Jun
17-Jun
18-Jun
19-Jun
20-Jun
21-Jun
22-Jun
23-Jun
24-Jun
25-Jun
26-Jun
27-Jun
28-Jun
29-Jun
30-Jun
1-Jul
2-Jul
3-Jul
4-Jul
5-Jul
6-Jul
7-Jul
8-Jul
9-Jul
10-Jul
11-Jul
12-Jul
13-Jul
14-Jul
15-Jul
16-Jul
17-Jul
18-Jul
19-Jul
20-Jul
21-Jul
22-Jul
Plan
Actual
1,561
15
126
1,240
TC Not Started
TC Running
TC Failed
TC Completed
0
500
1000
1500
2000
2500
2031
109
TC Started
Defects
0
500
1000
1500
2000
2500 2219
23
TC Started
Defects
0
200
400
600
800
1000
1200
1400
1381
80
TC Started
Defects
328
4 89
1,938
TC Not Started
TC Running
TC Failed
TC Completed
2624
4
2,211
TC Not Started
TC Running
TC Failed
TC Completed
0
5
10
15
20
25
30
35
40
45
Remaining Open
New Open Today
Newly Removed Today
Progress was measured by delivering software
free of critical defects.
Progress was measured by delivering software
free of critical defects.
Fatal
Critical
All Projects Defect
Count
SIT 2
Current Defect
Details
0
13
Daily - Open SIT Count

11. 10/16/17
9
Progress was measured by delivering software
free of critical defects.
Expected AVG Closed
Avg Days
to Fix
Closed
Avg Days
to Fix
Closed
Avg Days
to Fix
Closed
Avg Days
to Fix
Closed
Avg Days
to Fix
Closed
Avg Days
to Fix
Closed
1.0 3.9 7.0 2.1 1.0 13.0 1.0 6.0 1.0 5.0 0.0 NA 0.0 NA 0.0
3.0 7.5 154.0 7.1 12.0 11.4 3.0 3.7 8.0 6.3 4.0 6.5 0.0 NA 5.0
CMS Services Payments ECF Reporting PODSLegacy
Avg Days to Fix
NA
10.4
All Projects
Weekly - All SIT Defects Turnaround Analysis - Last Run 3/13/2014
Progress was measured by delivering software
free of critical defects.
SIT 2 Critical / Fatal Defect Trend
0
5
10
15
20
25
30
35
40
45
Remaining Open
New Open Today
Newly Removed Today

12. 10/16/17
10
Progress was measured by delivering software
free of critical defects.
Graduated To Plan
Scenarios Graduated
3 0
of 2215
71%
Test Case Execution and Business Scenarios
Blocked
Scenarios
Test Cases
Blocked by
Incomplete
Development
Plan Defect %
Actual Defect %
1%
Actual Defect % to Plan
10%
Defect Quality
Run 3
0
9%
14%
17%
20%
29%
37%
40%
46%
52%
58%
68%68%68%
72%
77%79%81%83%83%83%83%84%84%85%86%87%87%87%87%87%87%87%89%
0
0.2
0.4
0.6
0.8
1
1.2
15-Jun
16-Jun
17-Jun
18-Jun
19-Jun
20-Jun
21-Jun
22-Jun
23-Jun
24-Jun
25-Jun
26-Jun
27-Jun
28-Jun
29-Jun
30-Jun
1-Jul
2-Jul
3-Jul
4-Jul
5-Jul
6-Jul
7-Jul
8-Jul
9-Jul
10-Jul
11-Jul
12-Jul
13-Jul
14-Jul
15-Jul
16-Jul
17-Jul
18-Jul
19-Jul
20-Jul
21-Jul
22-Jul
Plan
Actual
0
500
1000
1500
2000
2500 2219
23
TC Started
Defects
2624
4
2,211
TC Not Started
TC Running
TC Failed
TC Completed
Working software was deployed regularly (almost
daily) for business validation.

13. 10/16/17
11
Adapting to changing requirements throughout
the testing phase drove success.
Image courtesy of Sira Anamwong at FreeDigitalPhotos.net
Developers, testers, and business sat and worked
together during the entire testing phase.

14. 10/16/17
12
Majority of the testing team was co-located in the
‘SITuation Room.’
How have you facilitated
co-location when working
with a geographically
dispersed team?
Testing team was motivated and empowered to
deliver.

15. 10/16/17
13
Team sustained constant pace for the duration of
the SIT phase (work hard, play hard).
• Team events
– Monthly dinners
– Wine / sports bar events
– Country line dancing
– Celebrate ‘life events’
– Exercise together
– Remote vs on-site
– ‘Wake’ for departing team
members
Customer satisfaction was the highest priority.
Metrics Chart
was updated
daily and
posted on
multiple walls in
the building

16. 10/16/17
14
Business scenarios and test cases were prioritized.
Lower priority work was not done until later.
Defect
ID
Top 40
Rank Status Severity Priority Summary
Project
Assignment
8194 1 Fix Failed 1-Fatal 1-High SIT: Occurance Incrementer is not generated correclty for policy in SIT Legacy
Remediation
7788 2 Fix Failed 2-Critical 1-High ISO-Medicare Report - Mapping of Required Fields- Target Close 7/10 CMS
8326 3 Referred 2-Critical 1-High CMS Not able to reopen subrogation at a feature level and need to in order to
continue to process recoveries if needed
Legacy
Remediation
8208 4 Referred 2-Critical 1-High Completing FNOL receiving WebSeal error Services
8190 5 Fix Failed 2-Critical 1-High Copart: XML invoice for lot # 41315924 resulted in Copart logging the following error:
http_url_post_raw error ERROR-41315924201406261357
CMS
8139 6 Fix Failed 2-Critical 1-High Incomplete and incorrect Claim Rep Profile data loaded to CMS Services
8223 7 Referred 2-Critical 1-High SIT: Backfeed events are not generated for Payments on Feature 04 on claim
0001504603
CMS
4503 8 Fixed 3-Moderate 1-High Accounting request to capture originally-entered issue date for manual payments for
1099 reporting
Payments
8066 9 Fix Failed 2-Critical 1-High SIT environment of Claim Center has logged out user ID sittest069 three times while
actively updating Claim Center screens. Occurred 6/20/14 between 7:15am and
9:15am
XSA
8215 10 Referred 2-Critical 1-High Claims Inquiry- reserve offsets are displaying in Claims Inquiry and should not Legacy
Remediation
Testing team reviewed and re-grouped at the end
of each sprint to adjust to changes.

17. 10/16/17
15
Benefits – put the ‘icing on the cake’
Overcoming the Challenges
1. Testing unable to start as planned
due to missed development
complete milestones
2. Internally, multiple test systems
configured differently
3. Vendor test systems did not align to
our test environments
4. Questionable stability of the System
Integration Test (SIT) environment
5. Changing business requirements –
lack of traceability
6. Mix of IT and Business testers
7. Communicating progress was not
consistent
" Prioritized product delivery into
sprints; focused testing on
individual insurance products
" Standardized where possible;
documented variances
" Phased delivery – started in test
and moved to SIT
" Sprint 0 to validate SIT prior to test
start; SIT under release mgt control
" Alignment of BSA and Test Lead to
review changes and test findings
" Paired together; co-mentoring
" Developed testing dashboard and
published daily; wall charts visibility

18. 10/16/17
16
Lessons Learned from Testing
• More consistent application of risk-based testing to prioritize
most important work
• Expanded usage of automated testing (grew from build testing
to full scenario automation)
• Prescribed order for type of testing: functional, then end-to-end,
then business scenario
• Consistency of test cases; more normalized size of test cases to
effectively track progress
• Daily cross-functional defect triage provided instant perspective
of the quality/stability of the total codebase and capacity across
development and testing teams for resolution and re-test
Questions?

19. 10/16/17
17
Thank you!
www.linkedin.com/in/toddrjones
Name | Todd R Jones
PMP, CSM, PMI-ACP
Email | toddrjones@westfieldgrp.com
memberservices@pmineo.org
©2017 Todd R Jones
Published as part of…