In this presentation you will learn about how the testing process and continuous quality improvements are aligned to the scrum process in a large software project. We hope that our hands -on experience will give you inspiration on how to tailor the test process in an agile environment. The project has been running for more than two years, with six successful releases to end users. We would like to share our experiences with managing test processes in a large scrum project – our do’s and don’ts, our success stories and also our lessons learned. The project is the largest scrum project in Norway to date. The project scope is to implement system support for managing a new pension reform for all inhabitants in Norway that are members of the pension fund, and replacing existing system due to outdated technology. Approximately 750 000 project hours will be spent and between 100-180 people are involved in the project: thirteen scrum teams, plus two project management and acceptance testing teams, and one business expert team. Each scrum team contains all the knowledge and expertise needed for developing high quality software: Scrum master, business expert, technical architect, UX designer, developers, build/deploy responsible, and of course, dedicated test resources. Each software delivery in this project contains five sprints. Each sprint is three weeks, followed by acceptance testing before the delivery is shipped. Test driven development is used in all levels of development, from unit tests all the way up to functional system testing. All test levels up to system integration testing is performed during the development sprint by the scrum teams. We tried to automate UI tests, but this was not successful. However, tests in all other levels are successfully automated, and after each delivery, a fully automated regression test suite is shipped with the code.

1. 
 www.steria.no
© Steria
Test process aligned to scrum
– Experiences from project PERFORM
Kristina L. Tangen
Test Manager – Steria’s deliveries to PERFORM

2.  Facts about PERFORM
 The largest and most important
project ever for the State Pension
fund
 Government funded
 About 750 000 project hours spent
and 100-180 persons involved
 Implements system support for
managing new pension reform
 Replacing legacy system due to
outdated technology
 Agile development methodology -
Scrum

3.  Organization
Customer
Subproject
Business
Supplier
Subproject
development
Roll-out
Business
modeling team
Application
architect team
Maintaining
Environments
team
Acceptance
criteria
Functional test Approval test
Analysis and
design
SPK Acceptance test
Accenture Deployment
Steria
Project director - Project manager
Subproject
Architecture
Subproject
Test

4.  Scrum team
Scrum master
Technical
architect
Test
responsible
Developers/testers
Usability expert
Functional
architect
Product
Owner

5.  Project lifecycle
 3 weeks iterations
 3 deliveries a year
 4 years duration

6. 
Development cycle
Responsibility: Development teams
Test Strategy - PERFORM
 Unit test
 Integration test –
Continuous integration
 Functional System test
 Functional Integration test
 Approval test
 Acceptance test
 Production test

7. 
Acceptance criteria
Iteration n - 1 Iteration n Iteration n + 1
Execute System Integration tests
Test process - development
Define and
execute
unit and
integration tests
Define and refine
functional test
conditions
Define and
execute
Functional
system tests

8.  Unit testing and integration testing
 Whitebox testing
techniques
 Automated tests
 Tools: Junit, Flexunit,
DBunit, Jmock and
checklists
 Continuous integration
 Hudson

9.  Functional test - System testing
 System testing is performed by the development
teams
 All user stories have functional test criteria
 At least one functional test per user story
 Automating tests should be considered
 Tools used on this level: Fitnesse and Quality Center

10.  Functional test – System testing
 Test conditions logged in
Quality Center as
requirements
 All test conditions covered by
at least one functional test
 All test conditions approved
by a business analyst

11.  Test automation – Functional test

12.  Functional Integration testing
 Planning and executing these tests is a team responsibility
 Planning takes place in the iteration before the test is executed
 Continuous Functional Integration testing during the whole development
cycle
 Development and test environments are separate
Plan acceptance
criteria
Iteration n - 1 Iteration n Iteration n + 1
Run System Integration
tests
Write and run
unit and
integration tests
Write and modify
functional test
conditions
Write and run
Functional
system tests
Plan System
integration test

13.  Controlling test quality
 After each iteration a quality check
of delivered user stories takes place
 Test:
▬ Test coverage
▬ Functional tests – both positive and
negative (destructive) are performed
▬ All code delivered is deployed to the
System Integration Test environment
▬ All planned functional tests executed
▬ Reporting – Quality Center and check
lists

14.  Test process experiences
 Early test focus from all project participants
 Test focus throughout the whole development cycle
 All project participants responsible for test execution
 Continuous functional system test
 Continuous functional integration test
 Stable test environments for acceptance testing
 Bugs are found and corrected early
 Continuous quality verification
 Disadvantages?

15.  Thank you for your attention!