Cisco uses ThousandEyes to monitor cloud services and gain visibility into network stability and troubleshooting. ThousandEyes helps Cisco reduce the mean time to troubleshoot issues by 43% and the mean time to restore services by 8%. ThousandEyes has successfully helped Cisco resolve issues with WebEx, Salesforce, firewalls in India, and support apps in India. Cisco's goal is to run ThousandEyes on Cisco routers to improve monitoring.

1. Cisco IT and ThousandEyes
How Cisco IT Gains Visibility into Cloud Service Stability and
Troubleshooting
Andrea Di Lecce
IT Technical Project Manager,Cisco Systems
03/2016

2. Why ThousandEyes ?

3. Cisco Confidential 3© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Why ThousandEyes?
Cisco’s Requirements
• Once off the Cisco network, the
“Cloud” is basically a black box
• Ping and traceroute have limited
capability and no ability to alert or
keep historical information
• Growing importance and criticality of
Cloud solutions requires a
comprehensive solution !
Cisco’s Goals
• Monitor a growing suite of Cloud
solutions and end-to-end network
health – latency, packet loss, Web
transactions, BGP reachability
• Constant monitoring and alerting
from critical, strategic network
locations

4. Cisco’s Deployment of
ThousandEyes

5. Cisco Confidential 5© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Cisco’s EnterpriseAgent Deployment
• Strategic enterprise agent
placement at high-priority
sites
• Call centers
• iPoPs
• High-priority sales
sites
• Business-critical Cloud
and internal services
monitored
• Salesforce
• WebEx
• TAC tools via
Akamai

6. Cisco Confidential 6© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Cisco’s Cloud Agent Usage
• Business-critical internal
services monitored from
ThousandEyes’ Cloud
agents
• WebEx
• TAC tools via
Akamai
• BGP reachability of our
service-containing public
IP address ranges from
Cloud agents

7. Success Stories & Lessons
Learned

8. Cisco Confidential 8© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Metrics Definition
• Mean Time to Troubleshoot (MTTT)
• The time it takes from the start of the incident to when the Engineer has narrowed down the source of the issue
• Mean Time to Restore (MTTR)
• The time it takes from the start of the incident to when the service is restored
• For incidents with external providers, the timeline after handoff is beyond Cisco’s control

9. Cisco Confidential 9© 2013-2014 Cisco and/or its affiliates. All rights reserved.
ThousandEyesSuccess Story - Business Outcomes
1. Reduce Mean Time to Troubleshoot (MTTT) for
applicable network events by 43% (measured)
2. Reduce Mean Time to Restore (MTTR) for applicable
network events by 8% (measured)
NOTE: Reducing MTTT and MTTR reduces Engineering time and total outage time.

10. Cisco Confidential 10© 2013-2014 Cisco and/or its affiliates. All rights reserved.
ThousandEyes Success Story - WebEx
• Business case for ThousandEyes: The issue was automatically detected by the program, which also pinpointed that the packet loss was occurring internal
to the Cisco network. This allowed Engineers to concentrate their troubleshooting efforts on the device in question, and resolve the issue quickly.
• Troubleshooting: It was determined via ThousandEyes that there was packet loss into the WebEx service from an internal network device.
• Resolution: Within 90 minutes, the Engineers resolved the case, because they knew where the packet loss was occurring.

11. Cisco Confidential 11© 2013-2014 Cisco and/or its affiliates. All rights reserved.
ThousandEyes Success Story - Salesforce
• Business case for ThousandEyes: The issue was automatically detected by the program, which also pinpointed that the packet loss was occurring external
to the Cisco network. This allowed Engineers to perform only the basic internal network checks, and then hand off to Salesforce to fix their network !
• Troubleshooting: It was determined via ThousandEyes that there was packet loss into salesforce.com from two India sites.
• Resolution: Within one hour, the P2 case was handed to Salesforce.com for investigation.
• The issue was isolated to a saturated Level3 ISP link in Salesforce's network and resolved.

12. Cisco Confidential 12© 2013-2014 Cisco and/or its affiliates. All rights reserved.
ThousandEyes Success Story – India Firewall
• Business case for ThousandEyes: The issue was automatically detected by ThousandEyes, which also pinpointed that the packet loss was occurring on a
specific device within the Cisco network. This allowed Engineers to resolve the issue quickly by failing over to the redundant gateway.
• Troubleshooting: Alert from ThousandEyes detected packet loss on our India corporate gateway, which affected all India sites.
• Resolution: Within one hour, the P1 case was identified with root cause, which was 100% CPU utilization on the corporate gateway device.
• The issue was resolved by failing over to the backup corporate gateway.

13. Cisco Confidential 13© 2013-2014 Cisco and/or its affiliates. All rights reserved.
ThousandEyes Success Story – India Support Apps
• Business case for ThousandEyes: The issue was proactively detected by ThousandEyes, and troubleshooting with ThousandEyes pinpointed the root
cause. Since the issue was intermittent, ThousandEyes prevented multiple subsequent P2 outages.
• Troubleshooting: Alert from ThousandEyes detected packet loss from a TAC site to the Cisco TAC apps portal (served by Akamai).
• Resolution: ThousandEyes indicated a problematic link between Bharti and Akamai. This prompted Akamai to remove the server from rotation, which
immediately restored services.

14. Cisco Confidential 14© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Lessons Learned
1. Email alert volume was extremely high due to transient
network issues
• Solution: Configure alerts only if event occurs more than
2x in a row
1. VirtualBox VM did not auto-start after software update and
reload of Mac Mini box
• Solution: Procedure provided by ThousandEyes

15. Looking Toward the Future

16. Cisco Confidential 16© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Looking Toward the Future
• ThousandEyes has certified their application to run on the Cisco 4451 ISR Service Container
• Cisco Design is currently testing this for deployment on our network !
• Our goal is to install ThousandEyes on the service containers on our existing WAN routers
No more Mac Minis !

17. Cisco Confidential 17© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Looking Toward the Future
• ThousandEyes is being integrated with our Network Operations standard alerting system
• ThousandEyes is working on certifying its application on the Cisco ASR 1000 Service Container

18. Thank you.