Case Study: Embratel Delivers the Gold Medal for Infrastructure Monitoring at Rio 2016

1. World®
’16
Case Study: Embratel Delivers the Gold Medal
for Infrastructure Monitoring at Rio 2016
Wanderson Coelho de Souza - Operations Manager - Embratel
DO4X121S
DEVOPS – AGILE OPERATIONS

2. 2 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
© 2016 CA. All rights reserved. All trademarks referenced herein belong to their respective companies.
The content provided in this CA World 2016 presentation is intended for informational purposes only and does not form any type
of warranty. The information provided by a CA partner and/or CA customer has not been reviewed for accuracy by CA.
For Informational Purposes Only
Terms of this Presentation

3. 3 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
Abstract
In 2016, Rio de Janeiro will become the first South American city to host the Olympic and
Paralympic Games. Capitalizing on its experience hosting the Pan American Games in 2007, and
the World Cup in 2014, the ‘Marvelous City’ surpassed Chicago, Tokyo and Madrid. The project is
budgeted at Brazilian Real 28.8bn ($7.6bn). This presentation will share the experience of how to
plan and operate the Olympic Infrastructure using CAU Unified Infrastructure Management.
There will be 10 types of service, including data centers, internet access, high-capacity data,
voice, mobility, CATV and wifi. During the event, Rio de Janeiro will host representatives from
206 countries and millions of visitors, and it is estimated that more than 21,000 journalists will
cover the event – all of which required an unprecedented telecoms and IT infrastructure. The Rio
2016 Olympic Games will use more than 60,000 network points connecting all the Olympic
arenas, and approximately 18,000 fixed lines, 16,000 SIM cards, three million SMS messages and
15,000 points of CATV. The Olympic backbone (370km) is equivalent to 971 laps of a standard
football pitch, or the length of 6,800 Olympic swimming pools.
Wanderson
Coelho de
Souza
Operations Manager
Embratel

4. 4 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
Agenda
OLYMPICS GAMES INFRASTRUCTURE REQUIREMENT OVERVIEW
ROLE OF CA UNIFIED INFRASTRUCTURE MANAGEMENT (CA UIM)
KEY DASHBOARDS
MOBILE APPLICATION USE CASE
1
2
3
4

5. 3
2
4
1
BARRA
DEODORO
MARACANÃ
COPACABANA

6. OLYMPIC GAMES 42
PARALIMPIC GAMES 23
65 CHAMPIONSHIPS44 Test events
4 Ceremonies (openings and closures)
100 DAYS Torch relay across the country
CHAMPIONSHIPS IN 12 DAYS
CHAMPIONSHIPS IN 17 DAYS
+

7. 4.8 B world wide viewers in 220 countries
30 K journalists
70 K volunteers
15 K athletes 206 countries
9 M tickets
300 K accreditations

8. All the admin networks, games and web hosting
were hosted in Embratel data centers and
monitored with CA UIM technology
1600 servers (physical/virtual)
40 network DC devices
1 Pb used data
54 M on real-time games processing
The whole environment monitored 24x7

9. Olympic venues infrastructure
33,380 hardware modules
113,472 LAN ports
440 UCS servers
5,159 wifi access points
177 security devices (firewalls, IPS, IDS)

10. What was hosted at the datacenter
ü Access control core equipment of all venues (ACS)
ü Controlling access to the APs of all venues
ü Access control for all users of the games, for wired and
wireless networks (ISE)
ü DHCP server / DNS / VPN
ü Firewalls, IPS, IDS, balancers traffic
ü Admin Rio2016 network (servers / storage / backup)
ü Games network - game information disclosure system
(servers / storage / backup)

11. During the games, approximately 2,200 tickets
were opened for the datacenter support team

12. SLA/Games Time Response Time Fix Time
Severidade 1 5 min (90%) 1 hr (90%)
Severidade 2 5 min (90%) 2 hr (90%)
Severidade 3 30 min (90%) 4 hr (90%)
Severidade 4 1 hr (90%) 6 hr (90%)
Service Request 1 hr (90%) 8 hr (90%)
Aggressive SLA levels were agreed to cover
normal operating and critical periods
• Events test
• Technical simulations
• Games time

13. CA Unified Infrastructure Management
Used to monitor the status of all game servers and network devices in venues
for Rio 2016
CA UIM was used because it is a
scalable, easy-to-use, cloud resilient,
and powerful IT monitoring tool and
provides 360 degree visibility into
systems and infrastructure
performance

14. CA Unified Infrastructure Management (CA UIM)
Provides possibility of controlling and monitoring the following parameters:
ü System Monitoring
ü %CPU usage, processor queue length, processor time
ü Disk usage (% and Mb), disk time, disk transfers/read/write per sec, paging
ü Memory usage (% and Mb)
ü Windows services, Windows event logs
ü Scanning directories, files
ü Network Monitoring
ü SNMP queries are sent towards network equipment (switches and firewalls)
ü Interface traffic from network equipment
ü ICMP ping of all the devices, servers and VMs
ü Application Monitoring
ü URL availability
ü Jboss/JAVA
ü Application processes (CPU usage, memory usage, threads);
ü Application services
ü SQL/Oracle

15. Applications
Support Team
Creation of interactive dashboards with CA UIM
was key to proactive issue detection/prevention

16. The badges below represent some of the many
portals supported
Note that one of the badges has a major (orange) alarm; the team at TOC
24x7 with the help of the right screens and alerts in front of them, was able
to verify with agility and act quickly to solve the problem

17. With CA UIM, it was possible to verify the
metrics info through the main manager interface

18. MIBs have been defined for monitoring (CPU,
interfaces, uptime, etc.)
All applied in SNMP communities, including the load balancer appliances

19. Probes and thresholds were essential in games
time, providing assertive role in identifying the
root cause and problem resolution
VMware
VMax
Load Average
Database
It was possible to
monitor the
Vmotions and DRSs
of VMs.
Identify disk failures,
generating alerts, even
before the
identification in the
storage tool itself
1-5 minutes, identify
quickly the growth of
the application queues
Transactions log and
increase of table space

20. Statistics from Aug. 5 to Aug. 17
ü Mobile App
ü Total visitors: 12,768,127
ü Total page views: 781,877,340
ü Overall
ü Total traffic in the Rio 2016 backbone: 1,794 PBs
ü Total internet traffic: 632 TBs
ü Unique devices in the network: 155,226
ü Peak of simultaneously connected wifi devices: 14,520
ü Peak of internet traffic: 5.35 Gbps

21. CA UIM helped track user experience issues with
mobile application
ü The mobile application launch to follow the torch relay
ü Internet statistics showed that every year the internet consumption has undergone access via desktop
to mobile devices; in March 2012 the number of users was 2,336,000 in December 2015 around 3.366
million and estimated growth of approximately 51% to games time
ü Mobile application launch for games time
ü With the new application, it was possible to track the results of online games, receive breaking news,
information on Venues, athletes and monitoring of the entire event, using mobile

22. Load Balance Traffic Internal Network Traffic
Oracle Memory Utilization All Errors @ CDN
Continous CA UIM data sharing w/app dev from
Aug. 10-14 resolved several performance issues

23. 23 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
Summary benefits with CA UIM
BETTER GAMES AND
APP
EXPERIENCE
SINGLE VIEW INTO
RIO OLYMPICS IT
INFRASTRUCTURE
IMPROVED
MOBILE
APPLICATION
PERFORMANCE
PROACTIVE ISSUE
RESOLUTION
RAPID ROOT
CAUSE AND
TRIAGE
Tickets Resolved
During Rio Olympics
2200

24. 24 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
Thank you.
Stay connected at communities.ca.com

25. 25 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
DevOps – Agile Ops
For more information on DevOps – Agile Ops, please visit:
http://cainc.to/wYXSg6