Testing in the Extremes by Atos

2. Julien Bensaid
Vice President &
Head of Global Testing Practice , Atos
Marta Sanfeliu
Chief Integrator, Winter Olympics Sochi
General Manager, Olympics for Atos
Speakers

3. Barcelona 1992
• 3 Billion Television viewers
• 169 Participating Nations
London 2012
• 8 Billion Connected Devices
• 204 Participating Nations
…from a single physical event in ‘92
to the most connected event in history
Rio 2016
• Cloud based infrastructure
• 206 Qualified Nations
Evolution of the Games…

4. Scale - Rio 2016
4 BN Worldwide viewers
37 Competition venues
300,000 Accreditations
30,000 Media Personnel
15,000+ Athletes
3,500 Technology Staff
70,000 Volunteers
200,000 Testing Hours
80 Applications & Systems
500 Servers
>$500M Technology Budget

5. Total Security Events
Athens ‘04
Beijing ‘08
London ‘12
Sochi ‘14
80 M
201 M
255 M
322 M
219 Security Events Every Second at Sochi !

6. Culture
Procedures
Language Laws
LocationPeople

7. • Fixed price contract with the IOC
• Multi-year games contract
• Dealing with many unknowns
• No option of missing deadlines
• 302 events timed to a fraction of a second
• Readiness includes systems & people

8. • Zero Email Initiative
• Use of Social Media – BlueKiwi
• Dedicated Technical Operation Center
• Alerts and Communication drills
• Global delivery teams in 13 locations
• Multi language adaptation
• Bridging cultures

10. November 2012
The first team
member arrived
in Rio after
London
2016
2012 2013 2014 2015 2016
Olympics
5 – 21 Aug 2016
Paralympics
7 – 18 Sept. 2016
Our Countdown to Rio 2016
200,000
hours of
testing
5 Testers
50-80 Testers
200 Testers
Onsite Staff

11. A dedicated 1000 sq meters (10,800 sq
feet) test lab for the Olympics
54 Test Cells – 36 for sports, cells for
S/W distribution & Games Management
Tests by local teams where the event
is going to be hosted
Ability to run applications in “Shadow”
mode during trials
Real sport events and Shadow events to
test performance of people and systems
How do you simulate real life situations during
testing?

12. Hybrid methodology - Agile for development and
Waterfall for Integration
9000 test cases grouped by sports, events and
systems. E.g. Track and Field has 700 Test Cases
Tools – ALM, Performance Center, UFT,
Acunetix (Security)
Virtualization for external interfaces
A script-less Hybrid Automation framework for UFT
increases reusability
What tools and processes are used to deliver a zero
defect games?
• User Acceptance Tests
• Homologation Tests
• Multi Sport Tests
• Technical Rehearsals
• Delivery Acceptance Test
• End To End Test
• Technical Test
• Test Events
Types of Testing Conducted

13. Tests across iOS, Android & Windows platforms
Predefined checklist for navigation, visibility by
resolution, need for scrolling etc.
Usability tests by sports and events specialist
Usability and Performance deviations are
treated as defects
Platforms developed in HTML 5 & CSS 3 for
increased user responsiveness
How do you ensure consistent user experience for 4
Billion viewers across devices and operating
systems?

14. A core team with previous Olympics experience. Testers chosen via a rigorous
selection process
Sports specialists, Major event technologists and Language experts part of the
testing team
Reusable Test Scenario Repository to reduce effort for test design
A structured SIAM process to interface with 3rd parties and external agencies
Continuous learning and retrospection
How do you ensure quality with a new 200+ team
every 2 years?
Test Factory Approach

16. It’s not just the Olympics that Test in the
Extremes
Create Extreme Security focus by simulating
breaches and disaster scenarios in Security Testing
labs to provide peace of mind in sharing data
Extreme Reliability through enterprise grade
Performance Testing toolsets, assuring 100%
uptime for mission critical applications to
protect your brand
Leverage tools, labs, utilities & partners - e.g. in
Service Virtualization - to create Extreme Agility
and lower TCO