EuroSTAR Software Testing Conference 2009 presentation on Regression Testing in a Migration Project by Wim Demey. See more at conferences.eurostarsoftwaretesting.com/past-presentations/

1. Copyright © CTG, Inc.
Organize regression testing in
companywide migration project
Challenge or nightmare?
Eurostar 2009 – Wim Demey

2. Copyright © CTG, Inc.
Agenda
 Context
 Project
 Challenges
 Lessons learned
 Questions

3. Copyright © CTG, Inc.
Context
 Similar project in timeframe of 2 years
 Customers within same sector
 Organisation
Business
Unit A
Business
Unit B
Business
Unit D
ITS
Infrastructure ApplicationsDivision A
Business
Unit …
Division …
Project

4. Copyright © CTG, Inc.
Context
 Project mode -> service mode
Project mode
Initiate Define Design Build Implement Close
- Project methodology needs to be followed
- A lot of deliverables (project + specific test deliverables)
+ Test strategy
+ Test requirements matrix
+ Test scenarios
+ Test reports (intermediate – progress – final)
+ Defect management
Project leader
Test coordinator
Test team
Service mode
Business
Unit B
Service
Delivery
Manager
Business
Unit …
Business
Unit A
SLA based support for application(s)
Maintenance budget

5. Copyright © CTG, Inc.
Context
 Test awareness within organisation
 Pool of test resources // separate team
 Mainly test coordinator profiles
 Methodology forces projects to use pool/test team
 Especially for large projects
 Being compliant with methodology requires a lot of time
 Internal/external audits
 Difference in focus on testing
 Infrastructure <-> application development
 Separate test environments
Development Validation Production

6. Copyright © CTG, Inc.
Context
 Work station concept -> standard environment
 By default no admin rights
 Default software for every user
 Additional software/rights on demand and dependent of business unit
Application software
Default software
Operating System
(SP’s, hotfixes)

7. Copyright © CTG, Inc.
Agenda
 Context
 Project
 Content
 Test approach
 Challenges
 Lessons learned
 Questions

8. Copyright © CTG, Inc.
Project >> Content
Initiate
WHO HAS INITIATED PROJECT?
-> Infrastructure
-> Similar project methodology as
application development
WHAT IS PROJECT SCOPE?
-> Default software upgrade
(e.g. Office2007, IE7, SapGui 7.2)
-> Front-end project
-> No upgrade of OS
STRUCTURE OF PROJECT TEAM?
-> Infrastructure architect team has lead
-> Organized in tracks
-> Involvement of test coordinator
(build <> define phase)
WHAT IS IMPACT FOR USERS?
-> All business units (company wide)
-> All workstation types (12.000 – 3.400)
-> Way of migration
(upgrade <> re-install)
Define Design Build Implement Close

9. Copyright © CTG, Inc.
Project >> Content
 What is new in Office2007?
Ribbon (with tabs -> menus)
Office button
New and xml based formats
-> .docx, xlsx, .accdb
-> possibility to save as PDF

10. Copyright © CTG, Inc.
Project >> Content
 What is impact of software upgrades?
 Huge number of Office related applications
 Mainly Access applications
 Heavy Excel files (VBA, macro,…)
 New behaviour of Word <> regulations
 Compatability of third party software with new Office formats
 New software versions as prerequisite for ongoing projects
 Design decisions
 Out of the box installation <-> Office configured to needs of BU’s
 Increase security-level (VBA, macro) in Office and IE
 What about applications in project mode?

11. Copyright © CTG, Inc.
Project >> Test approach
WHO PERFORMS THE TESTS?
-> SDM + team
-> Business
-> (support by internal test team)
HOW IS TESTING ORGANIZED?
-> Mini test cycle
(ad hoc, on preliminary build)
-> Major test cycle (3m -> 6 months)
(frozen build)
WHICH TEST ENVIRONMENT IS USED?
-> Sandbox idea (=virtual pc)
-> SDM is responsible for installation of
application + apps/test data
WHAT IS ROLE OF TEST COORDINATOR?
-> Follow up testing by SDM’s
-> Follow-up defects/issues
-> Ensure tight schedules are followed
-> Deliver required test documents
MAIN GOAL  REGRESSION TESTING
What works today, still works tomorrow on migrated software
SDM = Service Delivery Manager

12. Copyright © CTG, Inc.
Project >> Test approach
 Overview test environment
 Sandbox contains clean installation of components
CORP LCORP: XPLV90000xx
- Internet Explorer
- Outlook Repository
Install applications
admin
Remote Desktop Connection
Production account
Test account
ValidationProduction

13. Copyright © CTG, Inc.
Agenda
 Context
 Project
 Content
 Test approach
 Challenges
 How to apply test approach?
 How to coordinate/motivate SDM organisation?
 How to gather/report information -> project?
 Does test environment fit the needs?
 Can we keep business out of scope?
 Lessons learned
 Questions

14. Copyright © CTG, Inc.
Challenge 1: Test approach
Late involvement of
test coordinator
Late involvement of
test coordinator
Very rough estimations
of test effort
Very rough estimations
of test effort
Confusion about what is
in the project budget
Confusion about what is
in the project budget
No official/central list
of applications to test
No official/central list
of applications to test
Project methodology /
test deliverables not
100% fit for project
Project methodology /
test deliverables not
100% fit for project
“Tour of SDMs”
-> Public - individual
-> Get overview of portfolio
-> Planning
-> Roles + responsibilities
-> Deliverables
-> Support
Methodology adapted
-> Pragmatic but structure
is essential
-> Risk application test matrix
-> Test scenario matrix
-> MoSCoW principle
Clear out objections
-> Refine estimations
-> Provide enough information
-> Escalation to steering
committee

15. Copyright © CTG, Inc.
Challenge 1: Test approach
 Risk & Application Test Matrix
Sap GUI 7.0 IE 7.0 Word 2007 Excel 2007 PowerPoint 2007 Outlook 2007 Access 2007
0001 APPLICATION A SDM X N/A X N/A N/A N/A N/A X
0002 APPLICATION B SDM X X N/A X X N/A N/A N/A
COMPONENTSID Application SDM
Impact Severity Risk
exposure
Priority Overall Status
1 - Low 3 - High 3 Should Not started
3 - High 2 - Medium 6 Must Preparation
Link TSM Comment

16. Copyright © CTG, Inc.
Challenge 2: Test coordination
No dedicated test
resources available for
this project
No dedicated test
resources available for
this project
No test cases availableNo test cases available No Office2007
knowledge to adapt
application(s)
No Office2007
knowledge to adapt
application(s)
Daily activities interfere
with test execution
Daily activities interfere
with test execution
For some applications
business has to perform
tests
For some applications
business has to perform
tests
Technical support
-> Analysis of Office2007 issues
-> Document solutions
-> Assist developers during
implementation
-> Functional mailbox for business
Pragmatic follow up
-> Planning on weekly basis
-> Network driven approach per SDM
-> Test scenario template
-> Ad hoc meetings
-> Simple web based defect form

17. Copyright © CTG, Inc.
 Test scenario matrix
Challenge 2: Test coordination

18. Copyright © CTG, Inc.
How detailed must info
be collected/reported?
How detailed must info
be collected/reported?
Main focus on
planning/milestones
Main focus on
planning/milestones
No time to read long
reports
No time to read long
reports
Which feedback do we
give to SDM/business?
Which feedback do we
give to SDM/business?
Several documents as
input for reporting
Several documents as
input for reporting
Challenge 3: Info gathering & reporting
Dashboard
-> Overview of progress (total, per SDM)
-> Overview of defects (type, severity,
trends)
-> Small VBA code to gather
automatically status update
-> All information in one sight
-> Basis for status meeting with project
team
Test news letters
-> Only sent out if relevant & necessary
-> Information about general issues &
solutions
-> Changes in test strategy
-> Status about builds
-> Keep it as short as possible

19. Copyright © CTG, Inc.
Challenge 3: Info gathering & reporting

20. Copyright © CTG, Inc.
Mismatch between
theory and practice
Mismatch between
theory and practice
Test machines needed
in production
environment
Test machines needed
in production
environment
Teams create own test
environment
Teams create own test
environment
Some applications could
not be tested remote
Some applications could
not be tested remote
Need to travel around
with test environment
Need to travel around
with test environment
Security causes a lot of
problems
Security causes a lot of
problems
Physical machines
-> Test accounts
-> Separate container in Active Directory
-> Availability managed by test
coordinator
-> Production like install of applications
-> Staged by project team
-> Pick up & return concept
Security level reviewed
-> Decreased to current level
-> Alternative of exception policy doesn’t
work
-> Digital signing/increased security
outscoped to separate project
Challenge 4: Test environment

21. Copyright © CTG, Inc.
Unknown number of
applications developed
by business
Unknown number of
applications developed
by business
Often very critical
applications
Often very critical
applications
No support by
ITS/helpdesk
No support by
ITS/helpdesk
No experience /
knowledge to solve
issues
No experience /
knowledge to solve
issues
Problem of resources
even higher than SDM
teams
Problem of resources
even higher than SDM
teams
Not always clear what is
expected
Not always clear what is
expected
Challenge 5: Business
Active involvement
-> Same roadshows as for SDM
-> Inventory made + prioritisation
-> Critical applications are taken over by SDM
-> Technical support
-> Test scenario template provided
-> Same defect tool + procedure
-> Business representatives are SPOC

22. Copyright © CTG, Inc.
Agenda
 Context
 Project
 Content
 Test approach
 Challenges
 Lessons learned
 Some facts & figures
 Conclusions
 Questions

23. Copyright © CTG, Inc.
Lessons learned >> Facts & figures
 Applications
 1052 ITS supported applications identified (420)
 650 to test (Must, Should, Could)
 402 rest (Won’t) -> not impacted by components
 All applications tested in major test cycle (3 months)
 One team had 6 months time due to number of applications (420)
 Issues
 180 issues reported (both test cycles)
 Mainly Excel + IE7 related (70% of all issues)
 Compatibility issues with third-party software (e.g. SAS/SAP)
 REMARK: only for ITS supported applications
 No need to rebuild applications from scratch
 Some issues led to delayed migration of users/divisions

24. Copyright © CTG, Inc.
Lessons learned >> Conclusions

25. Copyright © CTG, Inc.
Lessons learned >> Conclusions

26. Copyright © CTG, Inc.
Questions / Remarks
Wim Demey (wim.demey@ctg.com)