RSA Scandinavia implemented a new test model to standardize testing across outsourced development projects. The model uses a risk-based approach and V-model framework. It defines requirements for test planning, design, execution, reporting, and responsibilities between RSA and suppliers. The implementation involved communicating the new model, providing training, and integrating it into project and contracting processes. Today, the model is used for all projects and is helping to streamline quality monitoring, reporting, and knowledge sharing across the organization and its suppliers.

1. Customer Testing & Quality in
Outsourced Development
- A Story from an Insurance Company
Version : 1.0
Created by: Gitte Oberbossel

2. Agenda
• RSA Scandinavia
• Background for the new test model
• RSA test model
• Implementation
• Today
2
OK, now you do exactly as I
tell you!

3. RSA Scandinavia
 Short facts
 Part of RSA Insurance PLC UK
 Scandinavia's third largest non-life
insurance company
 42 offices in Sweden, Denmark,
Norway and Finland
 Offers non-life insurance in the
Personal and Commercial segments
 2.5 million Personal customers and
200,000 Commercial customers
 4,000 employees
3
• Locations
• RSA Scandinavia is located within
Denmark, Sweden and Norway
• Brands we are known for:

4. RSA Scandinavia – Seen from a testing perspective
• Our area of expertise is insurance
− Most employees involved in testing have little or no knowledge of testing and testing
models when they are first assigned to a project.
− Test managers can be in-house insurance employees or they can be consultants.
− We have a small test centre that works as process owner, and also supports and
attends in quality evaluation.
• We are not an IT house
− The majority of the company does not think in or use IT terminology and processes –
we use business terminology, and talk about business processes and market
advances.
− The solutions we implement need to work all the time, or we lose a lot of money.
• Major parts of development and maintenance are outsourced
− We have decided to outsource most of our IT development and technical infrastructure
to various suppliers, both within development and maintenance.
− In some cases one supplier develops and another is responsible for maintenance.
− Sometimes one project has multiple suppliers.
4

5. We need a way to run test handling that fits our world
• Outsourcing development and maintenance of solutions demanded new
requirements of testing and quality ensuring within the company.
It also raised a lot of questions:
− How do we ensure that suppliers perform a test which provides a solution that
is not filled with defects and has high quality?
− What are the responsibilities with regard to test activities between supplier and
customer?
− How do we ensure effective testing without delays due to misunderstandings
between supplier and tester?
− What are the test criteria to the supplier, and how should they report these?
− How do we ensure that test material used by one supplier for development can
be re-used by another supplier for maintenance testing in the future?
− How is defect handling, test reporting etc. best done between supplier and
customer?
5

6. We need a way to run test handling that fits our world
• We need a test model that:
− Is fit to handle suppliers:
• Setting test and quality requirements for the supplier.
− Resembles Lego:
• Easy to use and quickly understood – everybody knows it already in some form
and can put it together to match different types of solutions to be tested.
− Has build-in quality ensuring and easy reporting.
− Is fit to handle many different:
• Types of projects / maintenance:
• New developments / changes of existing system.
• Standard solutions.
• Technical – Infrastructure.
• Sizes of tests.
6

7. 7
Agenda
• RSA Scandinavia
• Background for the new test model
• RSA test model
− Overall
− Test policy
− Overall test strategy
• Implementation
• Today
Making a product that matches the
people using it
Results
Budget
Quality
Time
Risk

8. RSA test model
8

9. RSA test model – mandatory models within the model
− Risk-based testing:
• Defining test scope
• Risk handling
− Test estimations
− The use of different test types
− Test design
− Test review
− Re-use of test material
− Defect handling
− Test reporting
9
The main RSA test model is made up of detailed models within:
It’s like riding a bicycle
– Once you learn how, it works the same way each time.
Only the type of bicycle and the equipment changes!

10. Points to be followed by RSA and supplier
• Test contract appendix:
− Responsibility.
− Test scope requirements:
• Amount of testing.
• Test type to be performed.
− Requirements for test approach:
• Example: Defined test design approach
is according to RSA test model, to
ensure reuse possibility between
Supplier and RSA-Suppliers.
− Test meetings.
10

11. Points to be followed by RSA and supplier
• Test deliveries according to project model
gates:
− Test to be performed.
− Documents.
• Test exit criteria.
− Example of one exit criteria: 100% of
severity 1 defects are solved (fixed and
tested ok)
• Suspension criteria and resumption
requirements.
− Example of one suspension criteria: When
the agreed deliverables (e.g. from a
supplier) from previous test phase turns to
be not kept during the next test phase.
• Test data and environments.
11 Based on the god Old V-Model

12. Setup between RSA and supplier and responsibility
Test activity, and who to perform each activity, is defined in the
model for both RSA and supplier!
12 Well defined roles and responsibility

13. RSA test policy
• The RSA Scandinavia test policy is the “law book” of testing within
RSA Scandinavia and applies to both RSA and supplier test activity.
• RSA have, within the test policy, defined the purpose of testing which:
− Reduces risk for RSA and the customer.
− Ensures solutions are:
• Delivered with agreed quality.
• Deployed with known quality.
• Deployed without creating problems in existing production:
• Regression test and new solutions can work with existing technical solutions.
− Ensures that things work according to defined requirements.
13

14. Test policy – testing principles
• Test before deployment to production.
• Test according to RSA Scandinavia testing standard:
− All tests are reviewed, measured and signed off according to standard.
− All reporting of test results is done according to test control gates.
• Tests are performed by educated tester and test manager according to RSA
Scandinavia test education standard.
• Test scope of a solution is defined in the overall test planning phase – and is a
priority!
• Focus on re-using test material.
• All tests must use test tools when mapping requirements to test cases, test
design, test execution, test defect handling and test measurement.
14

15. Test model – focus on standard
15
• The standard is overall defined by the policy and in detail in the test models and
overall test strategy, thereby enabling RSA and suppliers to have:
− Same overall test process.
− Same deliveries within each gate.
− Same requirements overall for test scopes.
− Same milestones.
− Same exit criteria.
− Same templates.
− Same method of documenting test cases.
• Benefits:
− Streamlines the monitoring, quality handling and quality reporting.
− Can handle many suppliers at the same time.
− Enables re-use and sharing of test material across RSA and suppliers.
− You do not need to known each contract, just the test model.

16. RSA Scandinavia test quality model
16

17. RSA Scandinavia V-model
17

18. Test performed by RSA / supplier
18
RSA IT – Only test execution if internal development

19. Control and quality evaluation within each main test activity
For both RSA and supplier, within the overall test planning and each test phase,
there are standard exit criteria for:
− What to deliver.
AND
− What and who to approve – (Within the project and by RSA Test Centre).
Shown main test activity within each test phase and the overall test planning
19

20. Test policy – entry and exit criteria
20
• The exit criteria and acceptance criteria are set to ensure a satisfactory result;
ensuring it is possible to proceed with the next test phase, and in the end go
to production with the required quality:
− Defined overall in the RSA test policy.
− Details documented in the overall test strategies.
Makes it possible to control that we are on track
and have the correct quality!

21. Test policy – entry and exit criteria
• Examples of exit criteria for projects:
• Test case/test checklist has been executed with the following results:
- 100% of priority 1 test cases/test checklists have been successfully executed
- 99%-80% of priority 2 test cases/test checklists have been successfully executed
- 79%-60% of priority 3 test cases/test checklists have been successfully executed
- 59%-00% of priority 4 test cases/test checklists have been successfully executed
• Defects corrected with the following results:
- 100% of severity 1 defects are solved (fixed and tested ok)
- 100% of severity 2 defects are solved (fixed and tested ok)
- Maximum outstanding of severity 3 defects are 0-10 errors
- Maximum outstanding of severity 4 defects are 0-20 errors
- Maximum outstanding of severity 5 defects are 0-30 errors
21
Exit criteria are used to define and ensure that quality is okay before
continuing to the next phase – because bad quality in one phase has an
impact on the next phase!

22. Test policy – entry and exit criteria
22
• Amount of testing for each defined, prioritized test scope:
• For each defined and prioritized test scope, the mandatory amount of testing
within each priority is as defined in the table:

23. Test policy – entry and exit criteria
23
Test execution phase Scope to be tested according to defined priority
Priority 1 Priority 2 Priority 3 Priority 4
Unit test 100%-94% 94%-80% 79%-50% 49%-20%
Component Integrations test 100%-94% 94%-80% 79%-50% 49%-20%
System test 100%-94% 94%-80% 79%-50% 49%-20%
System Integrations test 100%-94% 94%-80% 79%-50% 49%-20%
Acceptance test 100%-94% 94%-80% 79%-50% 49%-20%
Deployment test 100%-94% 94%-80% 79%-50% 49%-20%
• Amount of testing for each defined, prioritized test scope.
• Defined set of rules for what we mean by each priority:
• In practice -> Use the risk-based test model to define priority.
• All test cases within a scope have the same priority, thereby making it possible
to monitor and use it as an exit criteria.

24. Test policy – RSA test reporting policy
• It is policy to keep a close focus on test quality at:
− Overall level,
− Release level,
− Within development of each solution.
To ensure that it is possible to be pro-active as soon as it is detected
“That there is a risk that the defined quality and/or supplier contract
requirements will not be met!”
• It is therefore policy, and a mandatory requirement for tests of all solutions, to
keep focus on test quality, and to report on risk and quality:
− During all test phases, both for RSA and supplier test phases.
− A final test report documenting the quality is produced at the exit of each test
phase.
24

25. 25
Agenda
• RSA Scandinavia
• Background for the new test model
• RSA test model
− Overall
− Test policy
− Overall test strategy
• Implementation
• Today
Making a product that matches the
person using it
Results Risk
Budget
Quality
Time

26. Overall test strategy – introduction
• All testing within projects is run according to the RSA Scandinavia overall test
strategy for projects – both for RSA and supplier:
− This document contains the overall test strategy used by all projects, and it defines
the things that are special to the project, e.g. test meetings and test scope.
26

27. Overall test strategy – project types
• For projects, the detailed test process is split because there are different test
focuses and amounts of test. Therefore, to ensure effective testing, the test
template for planning, designing and reporting is tailored to match specific needs.
• Projects are split into 3 types:
27

28. Overall test strategy – test steering
• Steering of all test phases and ensuring test quality is upheld by using
RSA Scandinavia Project Excel Sheet for RSA or supplier:
− For each test phase there is one sheet documenting:
• What test activity to perform.
• Test entry and exit criteria.
• What reviews to perform and who should approve them.
− The excel sheet is updated with a status throughout test activity.
− There is one excel sheet to be filled in by RSA and one to be filled in by the
supplier.
28

29. 29
Agenda
• RSA Scandinavia
• Background for the new test model
• RSA test model
• Implementation
• Today

30. Implementation
• Used for all projects initiated since 1 January 2011, and currently being
determined for Maintenance and Incidents.
• What we did when we started:
− The test model was fully implemented into the RSA project process and the
contract negotiations process.
− Communication – a lot of it – at all levels of the organisation.
− Education:
• Internal courses in RSA made up of small modules.
− Support:
• Each project, when starting up, is appointed someone from the test centre
assigned to give support.
− High focus on approval of each test phase:
• Working with release management to ensure that the known non-approved tests
from projects are not being approved for production.
30

31. Today – status on key areas after implementation
• Contract with supplier:
− Standard test appendix is an automated part of the contract for suppliers.
• Support:
− Projects have learned to contact RSA Test Centre for support.
• Review and approval:
− Test review and approval are becoming a natural part of projects.
• Education:
− We have already educated many and are still running courses internally.
− New test manager consultants within testing receive a 1-2 hour introductory session.
• Overall reporting:
− Now running for releases and is on the way for IT management.
31

32. TIPS
• Tailor the communication to the people you are talking to:
− Have standard presentations that match different audiences, thereby ensuring all in the
test centre are able to tell the same story.
• Write down the model – without making it too IT nerdy 
• Reviews should be used for everything:
− When you are within the IT world and a test expert, you easily forget that the users of
the model are not. Therefore, a review is the best way to ensure that everything is
understood.
• Engage in team work with all process owners that interact with the test model.
32

33. TIPS
• Align and build the test model into the already existing models in the
company, and do it in cooperation with existing model owners.
• Education is important – It should be ongoing throughout the year.
• Support and help is necessary.
• High focus on the supplier and whether they are upholding the contract.
• We are a performance company, therefore, define SMART goals within testing.
33

34. THANK YOU
From Gitte Oberbossel
Firm email gio@codan.dk
Privat email goberbossel@gmail.com