EuroSTAR Software Testing Conference 2008 presentation on Testing - An Olympic Challenge by Atos Origin. See more at conferences.eurostarsoftwaretesting.com/past-presentations/

1. Atos, Atos and fish symbol, Atos Origin and fish symbol, Atos Consulting, and the fish itself are registered trademarks of Atos Origin SA. September 2006
© 2006 Atos Origin
Testing an Olympic Challenge
Eurostar, The Hague, 12 November 2008
Magnus Alvarsson

2. 2 Major Events
The Olympic Challenge
» There is no second chance
» Proven solutions
» Reliable solutions
» Meet the Users’
expectations
» World Wide Visibility
» Large IT Deployment
» Multiple partners, suppliers and
technology

3. 3 Major Events
Agenda
» Introduction
» The Olympic Challenge
» Framework
» Project Timeline
» Organization
» Testing
» Strategy
» Testing environments
» Process

4. 4 Major Events
Objective
» Ensure stable Games systems that:
» Conform to functional requirements
» Satisfy or exceed agreed service levels
» Support the business of the customers
AND:
» Ensure the integration of:
» The numerous and varied Games
systems
» The partners, people and processes
supporting these systems
Such that ….
» The systems and business processes
complement each other to ensure a
seamless delivery of service to our
customers

5. 5 Major Events
The Olympic Challenge - Scope
» Atos Origin Systems
- 9 Core Games Management Systems (CGS)
- 9 Info Diffusion Systems (IDS)
- The Olympic Family Intranet in 2 or 3 languages
- A large number of interfaces between these systems
» On Venue Results
- One On Venue Result System per sport (15/36)
- One Timing and Scoring System per Sport and/or Venue
- One Scoreboard per Venue
- Minimum of 1 TV-Graphics solution per sport
» External Interfaces
- On Venue Results
- 8 World News Press Agencies
- 20 Internet Data Feeds clients (Rights Holding Broadcasters, Media, IF’s and
NOC’s)
- Weather
- Transportation
- Biography and Background Provider(s)
- Background Check

6. 6 Major Events
Integration of Partners, People, Processes
» Organizing Committee
» Timing and Scoring / On Venue Results Provider
» Hardware providers
» Network
» Servers
» Desk tops
» Cameras
» Score boards / Video boards
» Mobile devices
» Software Providers
» Databases
» Operating Systems
» Office suites
» Etc
The Olympic Challenge – Scope (Cont’d)

7. 7 Major Events
» Operations
» Software Distribution Solution
» Call Management System
» Monitoring
» etc
» External Stakeholders
» International Olympic Committee
» International Sporting Federations
» Rights-holding Broadcasters and News Agencies
» OCOG Functional Areas
» etc
The Olympic Challenge – Scope (cont’d)

8. 8 Major Events
The Olympic Challenge - Scope
INTEGRATION
People, Partners
& Processes
• Integration of
sponsors/partners
• Customers (VANOC,
NBC, WNPAs etc)
• Atos TS, Ops, MEP
• Roles & Resp.
• Policies &
Procedures
• Communication
• Project / Risk / Issue
Management
• Omega – On-Venue
Results Systems
• TV Graphics :
Olympic Broadcast
Services
• Scoreboard
• RCMP, Immigration
• Data providers
Non-Atos SystemsCore Games
Management Systems
• Sport Entries &
Qualification
• Accreditation
• Workforce
• Medical
• Arrivals &Departures
• Transportation
• Commentator
Information
System
• INFO 2010
• Internet Data Feed
• Results Data Feed
to WNPAs
• Print Distribution
System
• Central Repository
System
Information
Diffusion Systems
Testing / Integration Test Lab

9. 9 Major Events
Project Timeline
2006
Design
 Planning
 Business Process
Reviews
 GAP Process
 Functional
Requirements
 Interface Definition
 Software development
 Lab build
 Staff build-up
 Software deliveries
Test Tools Acquisition
Test Case preparation
Build
 Connectivity Tests
 Functional Tests
 Sport/Test Events
 User Accesptance
 System Tests
 Multi Sport Tests
Test
 Technical Rehearsals
 Games TOC operational
 Applications in production
 Venue Deployment
 Maximum staff numbers
Operate
Feb 2010
Olympic Games
2010
40
100
200
1,200Resources

10. 10 Major Events
The Challenges
» Challenges
» Applications from suppliers worldwide
» Synchronizing deliveries and compatible code
» Management and control of:
- defects
- testing environments
- the numerous & varied Integration Test Lab activities

11. 11 Major Events
The Answers
» The answers
» A purpose-built Test Lab (500 m2; 5000 ft2)
» A dedicated Testing team (80+ people including partners)
» A comprehensive multi-faceted test program
» Strict test procedures and defect tracking
» > 200,000+ hours of testing; > 9,000+ test cases: happy v unhappy path
» Support from Technical Services and Operations
» Communication & Coordination
Testing: the single most important success
factor!

12. 12 Major Events
Objective of Integration testing
Ensure that:
» the Games systems work together seamlessly
» Fulfill the required functionality and service levels
To successfully manage and support the Olympic and Paraolympic
Games.

13. 13 Major Events
To achieve the objective…
» Test functionality of different systems
» Availability test
» Verify operating procedures of the systems
» Test systems under extreme situations
» Focus the tests on data flowing end-to-end
through the systems.
» Simulate games

14. 14 Major Events
Integration Test Program
» Pre-Delivery Testing of:
» Supplier Development
» Supplier “Connectivity”
» Supplier Technical
» Integration Testing of:
» Delivery Acceptance
» Stand-alone
» Interface
» End-to-End
» Regression

15. 15 Major Events
Integration Test Program
Integration Testing of:
» Delivery Acceptance
» Stand-alone
» Interface
» End-to-End
» Regression

16. 16 Major Events
Integration Test Program
» System Testing (3 cycles
of > 20 days each per
application)
» Load
» Stress
» Fault Tolerance
» etc
» Acceptance Testing
» Test Cases supplied
by the IOC and the
Sports Federations or
the end users

17. 17 Major Events
Integration Test Program
» Test Events
» Every Sport
» Elite level competition
» Real Games conditions
» 1 year before
» Multi-sport Tests
» Three cycles
» Subset of Games schedule each
time
» Overall system performance and
functionality
» 2 months period

18. 18 Major Events
Integration Test Program
» Technical Rehearsal (TR1 & TR2)
» Two cycles
» Subset of Games schedule
each time
» Operational readiness of
systems and people
» 10 days period

19. 19 Major Events
Technical Rehearsal main objectives
Technical rehearsals primary focus is to:
» Practice and evaluate:
» operational policies and procedures
applied during Games
» communication processes
» Establish relationships
» Verify roles and responsibilities
» Test staff utilizing systems/applications
in a Games time environment
Beijing example
 35 Sports
 39 Venues
 500 Scenarios
 2000 Staff

20. 20 Major Events
Test Organization
Integrated team made up with the following groups:
» Integration Test Board - Atos Origin / OCOG / Omega
Oversees, Highest Point of Escalation
» Integration Test Manager and Test Management Team - Atos Origin
Definer & Coordinator of testing program & lab
» Test Group Leaders - Atos Origin / OCOG
Test plan managers & delivery managers, escalation point for Test Team Leaders
» Test Team Leaders - Atos Origin / OCOG
Managers of test activities and defects by Application, Sport, Intel, Unix, Security
» Testers - Atos Origin / OCOG
Perform tests, log and track defects

21. 21 Major Events
Test Organization (cont’d)
A set of integration testing plans, policies and procedures ensuring quality and
consistency in the testing process, including:
» Integration Testing Strategy
» Integration Test Plan
» Defect Tracking Process
» Software Delivery Procedure
» Change Management Process
» Test plans for each sport/system
» etc

22. 22 Major Events
Test Lab Characteristics
» Production like environment
» Tools used for testing activities
» Test Management tool (Quality Center)
- Test Cases and Scenarios
- Defect Tracking
» Test automation tool (QuickTest Professional)
- Test scripts
» Load and Performance testing tools
(Load Runner / Win Runner)
» Staff
» 35 Atos Origin
» 30 VANOC
» 6 Omega
» 10 Others

23. 23 Major Events
» Physical Infrastructure
» 300 PCs
» 100 laptops
» 50 printers
» 32 switches
» 450 servers
(30 ITL, 420 TER)
Customer Engagement
Test Lab Characteristics

24. 24 Major Events
Test Cell

25. 25 Major Events
Criteria for Environment Strategy
» Meet objectives stated in the Integration Test
Strategy
» Integration and operation activities can be
smoothly executed.
» Minimize the total number of environments
» Consider the needs for conducting several
continuous tests in parallel without delays.
» Use experience gained from previous
Olympic Games.

26. 26 Major Events
Environments
» All applications are installed on all environments.
» Three environments must be production like
» Three environments must have the Games-time configuration
» The rest environments will meet the minimum requirements of the Integration team
» Four environments will be available during the Games,
Date
Environment J F M A M J J A S O N D J F M A M J J A S O N D J F M
PDC
E2E
T1
T2
SDC
PAR
PPP
MEP BCN
Paralympic Testing
Para PreProduction
Testing
Testing Test Events System / MSTs TRs Games
Testing
Testing
2008 2009 2010
Systems Test / DR DR
Homologation PreProduction

27. 27 Major Events
Integration Control
Keys to Integration Testing success for big scale projects:
1. Correlation with Project Plan
2. Defect Tracking Process
3. Control of Software Deliveries
4. Change Management Process
5. Clear Testing Strategy
6. Partner management/Stakeholder agreement
7. Support Tools
8. Common policies & procedures
 To minimise risk, the overall IT
system and applications must be
exhaustively tested

28. Atos, Atos and fish symbol, Atos Origin and fish symbol, Atos Consulting, and the fish itself are registered trademarks of Atos Origin SA. September 2006
© 2006 Atos Origin
Questions and Answers
Atos Origin Major Events
Av. Diagonal, 210-218
08018 Barcelona (Spain)
www.atosorigin.com
Magnus Alvarsson
+1 778 328 3636
magnus.alvarsson@atosorigin.com