SlideShare a Scribd company logo
1 of 18
Download to read offline
Cisco IT and ThousandEyes
How Cisco IT Gains Visibility into Cloud Service Stability and
Troubleshooting
Andrea Di Lecce
IT Technical Project Manager,Cisco Systems
03/2016
Why ThousandEyes ?
Cisco Confidential 3© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Why ThousandEyes?
Cisco’s Requirements
• Once off the Cisco network, the
“Cloud” is basically a black box
• Ping and traceroute have limited
capability and no ability to alert or
keep historical information
• Growing importance and criticality of
Cloud solutions requires a
comprehensive solution !
Cisco’s Goals
• Monitor a growing suite of Cloud
solutions and end-to-end network
health – latency, packet loss, Web
transactions, BGP reachability
• Constant monitoring and alerting
from critical, strategic network
locations
Cisco’s Deployment of
ThousandEyes
Cisco Confidential 5© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Cisco’s EnterpriseAgent Deployment
• Strategic enterprise agent
placement at high-priority
sites
• Call centers
• iPoPs
• High-priority sales
sites
• Business-critical Cloud
and internal services
monitored
• Salesforce
• WebEx
• TAC tools via
Akamai
Cisco Confidential 6© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Cisco’s Cloud Agent Usage
• Business-critical internal
services monitored from
ThousandEyes’ Cloud
agents
• WebEx
• TAC tools via
Akamai
• BGP reachability of our
service-containing public
IP address ranges from
Cloud agents
Success Stories & Lessons
Learned
Cisco Confidential 8© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Metrics Definition
• Mean Time to Troubleshoot (MTTT)
• The time it takes from the start of the incident to when the Engineer has narrowed down the source of the issue
• Mean Time to Restore (MTTR)
• The time it takes from the start of the incident to when the service is restored
• For incidents with external providers, the timeline after handoff is beyond Cisco’s control
Cisco Confidential 9© 2013-2014 Cisco and/or its affiliates. All rights reserved.
ThousandEyesSuccess Story - Business Outcomes
1. Reduce Mean Time to Troubleshoot (MTTT) for
applicable network events by 43% (measured)
2. Reduce Mean Time to Restore (MTTR) for applicable
network events by 8% (measured)
NOTE: Reducing MTTT and MTTR reduces Engineering time and total outage time.
Cisco Confidential 10© 2013-2014 Cisco and/or its affiliates. All rights reserved.
ThousandEyes Success Story - WebEx
• Business case for ThousandEyes: The issue was automatically detected by the program, which also pinpointed that the packet loss was occurring internal
to the Cisco network. This allowed Engineers to concentrate their troubleshooting efforts on the device in question, and resolve the issue quickly.
• Troubleshooting: It was determined via ThousandEyes that there was packet loss into the WebEx service from an internal network device.
• Resolution: Within 90 minutes, the Engineers resolved the case, because they knew where the packet loss was occurring.
Cisco Confidential 11© 2013-2014 Cisco and/or its affiliates. All rights reserved.
ThousandEyes Success Story - Salesforce
• Business case for ThousandEyes: The issue was automatically detected by the program, which also pinpointed that the packet loss was occurring external
to the Cisco network. This allowed Engineers to perform only the basic internal network checks, and then hand off to Salesforce to fix their network !
• Troubleshooting: It was determined via ThousandEyes that there was packet loss into salesforce.com from two India sites.
• Resolution: Within one hour, the P2 case was handed to Salesforce.com for investigation.
• The issue was isolated to a saturated Level3 ISP link in Salesforce's network and resolved.
Cisco Confidential 12© 2013-2014 Cisco and/or its affiliates. All rights reserved.
ThousandEyes Success Story – India Firewall
• Business case for ThousandEyes: The issue was automatically detected by ThousandEyes, which also pinpointed that the packet loss was occurring on a
specific device within the Cisco network. This allowed Engineers to resolve the issue quickly by failing over to the redundant gateway.
• Troubleshooting: Alert from ThousandEyes detected packet loss on our India corporate gateway, which affected all India sites.
• Resolution: Within one hour, the P1 case was identified with root cause, which was 100% CPU utilization on the corporate gateway device.
• The issue was resolved by failing over to the backup corporate gateway.
Cisco Confidential 13© 2013-2014 Cisco and/or its affiliates. All rights reserved.
ThousandEyes Success Story – India Support Apps
• Business case for ThousandEyes: The issue was proactively detected by ThousandEyes, and troubleshooting with ThousandEyes pinpointed the root
cause. Since the issue was intermittent, ThousandEyes prevented multiple subsequent P2 outages.
• Troubleshooting: Alert from ThousandEyes detected packet loss from a TAC site to the Cisco TAC apps portal (served by Akamai).
• Resolution: ThousandEyes indicated a problematic link between Bharti and Akamai. This prompted Akamai to remove the server from rotation, which
immediately restored services.
Cisco Confidential 14© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Lessons Learned
1. Email alert volume was extremely high due to transient
network issues
• Solution: Configure alerts only if event occurs more than
2x in a row
1. VirtualBox VM did not auto-start after software update and
reload of Mac Mini box
• Solution: Procedure provided by ThousandEyes
Looking Toward the Future
Cisco Confidential 16© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Looking Toward the Future
• ThousandEyes has certified their application to run on the Cisco 4451 ISR Service Container
• Cisco Design is currently testing this for deployment on our network !
• Our goal is to install ThousandEyes on the service containers on our existing WAN routers
No more Mac Minis !
Cisco Confidential 17© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Looking Toward the Future
• ThousandEyes is being integrated with our Network Operations standard alerting system
• ThousandEyes is working on certifying its application on the Cisco ASR 1000 Service Container
Thank you.

More Related Content

What's hot

Optimizing and Troubleshooting Digital Experience for a Hybrid Workforce
Optimizing and Troubleshooting Digital Experience for a Hybrid WorkforceOptimizing and Troubleshooting Digital Experience for a Hybrid Workforce
Optimizing and Troubleshooting Digital Experience for a Hybrid Workforce
ThousandEyes
 

What's hot (20)

Introduction to ThousandEyes
Introduction to ThousandEyesIntroduction to ThousandEyes
Introduction to ThousandEyes
 
Adopting SD-WAN With Confidence: How To Assure and Troubleshoot Internet-base...
Adopting SD-WAN With Confidence: How To Assure and Troubleshoot Internet-base...Adopting SD-WAN With Confidence: How To Assure and Troubleshoot Internet-base...
Adopting SD-WAN With Confidence: How To Assure and Troubleshoot Internet-base...
 
Getting Started with ThousandEyes Proof of Concepts
Getting Started with ThousandEyes Proof of ConceptsGetting Started with ThousandEyes Proof of Concepts
Getting Started with ThousandEyes Proof of Concepts
 
Getting Started with ThousandEyes
Getting Started with ThousandEyesGetting Started with ThousandEyes
Getting Started with ThousandEyes
 
Monitoring End User Experience with Endpoint Agent
Monitoring End User Experience with Endpoint AgentMonitoring End User Experience with Endpoint Agent
Monitoring End User Experience with Endpoint Agent
 
Realise True Business Value .pdf
Realise True Business Value .pdfRealise True Business Value .pdf
Realise True Business Value .pdf
 
Troubleshooting Webex and Microsoft Teams with ThousandEyes
Troubleshooting Webex and Microsoft Teams with ThousandEyesTroubleshooting Webex and Microsoft Teams with ThousandEyes
Troubleshooting Webex and Microsoft Teams with ThousandEyes
 
Optimizing and Troubleshooting Digital Experience for a Hybrid Workforce
Optimizing and Troubleshooting Digital Experience for a Hybrid WorkforceOptimizing and Troubleshooting Digital Experience for a Hybrid Workforce
Optimizing and Troubleshooting Digital Experience for a Hybrid Workforce
 
Introduction to ThousandEyes
Introduction to ThousandEyesIntroduction to ThousandEyes
Introduction to ThousandEyes
 
Getting Started With ThousandEyes Proof of Concepts: End User Digital Experience
Getting Started With ThousandEyes Proof of Concepts: End User Digital ExperienceGetting Started With ThousandEyes Proof of Concepts: End User Digital Experience
Getting Started With ThousandEyes Proof of Concepts: End User Digital Experience
 
How to Evaluate, Rollout and Operationalize Your SD-WAN Projects
How to Evaluate, Rollout and Operationalize Your SD-WAN ProjectsHow to Evaluate, Rollout and Operationalize Your SD-WAN Projects
How to Evaluate, Rollout and Operationalize Your SD-WAN Projects
 
Getting Started with ThousandEyes Proof of Concepts
Getting Started with ThousandEyes Proof of ConceptsGetting Started with ThousandEyes Proof of Concepts
Getting Started with ThousandEyes Proof of Concepts
 
emea_cisco_live_webinar_150623.pptx
emea_cisco_live_webinar_150623.pptxemea_cisco_live_webinar_150623.pptx
emea_cisco_live_webinar_150623.pptx
 
Getting Started with ThousandEyes Proof of Concepts
Getting Started with ThousandEyes Proof of ConceptsGetting Started with ThousandEyes Proof of Concepts
Getting Started with ThousandEyes Proof of Concepts
 
Understanding Cisco’s Next Generation SD-WAN Solution with Viptela
Understanding Cisco’s Next Generation SD-WAN Solution with ViptelaUnderstanding Cisco’s Next Generation SD-WAN Solution with Viptela
Understanding Cisco’s Next Generation SD-WAN Solution with Viptela
 
Cisco Security portfolio update
Cisco Security portfolio updateCisco Security portfolio update
Cisco Security portfolio update
 
Introduction to ThousandEyes
Introduction to ThousandEyesIntroduction to ThousandEyes
Introduction to ThousandEyes
 
Realize True Business Value With ThousandEyes
Realize True Business Value With ThousandEyesRealize True Business Value With ThousandEyes
Realize True Business Value With ThousandEyes
 
F5 Web Application Security
F5 Web Application SecurityF5 Web Application Security
F5 Web Application Security
 
Dynatrace
DynatraceDynatrace
Dynatrace
 

Similar to Cisco IT and ThousandEyes

Similar to Cisco IT and ThousandEyes (20)

E magic case study
E magic case studyE magic case study
E magic case study
 
EMEA.23.02.23_Top_Outages_of_2022_Webinar_Slides.pptx
EMEA.23.02.23_Top_Outages_of_2022_Webinar_Slides.pptxEMEA.23.02.23_Top_Outages_of_2022_Webinar_Slides.pptx
EMEA.23.02.23_Top_Outages_of_2022_Webinar_Slides.pptx
 
The Top Outages of 2022: Analysis and Takeaways
The Top Outages of 2022: Analysis and TakeawaysThe Top Outages of 2022: Analysis and Takeaways
The Top Outages of 2022: Analysis and Takeaways
 
ICS case studies v2
ICS case studies v2ICS case studies v2
ICS case studies v2
 
What is ThousandEyes Webinar
What is ThousandEyes WebinarWhat is ThousandEyes Webinar
What is ThousandEyes Webinar
 
The Top Outages of 2022: Analysis and Takeaways
The Top Outages of 2022: Analysis and TakeawaysThe Top Outages of 2022: Analysis and Takeaways
The Top Outages of 2022: Analysis and Takeaways
 
The Enterprise Unlocked webinar
The Enterprise Unlocked webinarThe Enterprise Unlocked webinar
The Enterprise Unlocked webinar
 
Getting Started with ThousandEyes Proof of Concepts
Getting Started with ThousandEyes Proof of ConceptsGetting Started with ThousandEyes Proof of Concepts
Getting Started with ThousandEyes Proof of Concepts
 
Cisco Connect 2018 Singapore - En06 jason pernell
Cisco Connect 2018 Singapore - En06 jason pernellCisco Connect 2018 Singapore - En06 jason pernell
Cisco Connect 2018 Singapore - En06 jason pernell
 
2021 01-27 reducing risk of ransomware webinar
2021 01-27 reducing risk of ransomware webinar2021 01-27 reducing risk of ransomware webinar
2021 01-27 reducing risk of ransomware webinar
 
Cyberdyne systems (2)
Cyberdyne systems (2)Cyberdyne systems (2)
Cyberdyne systems (2)
 
EMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? WebinarEMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? Webinar
 
Securely Connecting Your Customers to Their Cloud-Hosted App – In Minutes
Securely Connecting Your Customers to Their Cloud-Hosted App – In MinutesSecurely Connecting Your Customers to Their Cloud-Hosted App – In Minutes
Securely Connecting Your Customers to Their Cloud-Hosted App – In Minutes
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
 
Cisco Connect 2018 Philippines - introducing cisco dna assurance
Cisco Connect 2018 Philippines - introducing cisco dna assuranceCisco Connect 2018 Philippines - introducing cisco dna assurance
Cisco Connect 2018 Philippines - introducing cisco dna assurance
 
Broad Sky July Webinar Faast Failover
Broad Sky July Webinar Faast FailoverBroad Sky July Webinar Faast Failover
Broad Sky July Webinar Faast Failover
 
Optimizing and Troubleshooting Digital Experience for a Hybrid Workforce
Optimizing and Troubleshooting Digital Experience for a Hybrid WorkforceOptimizing and Troubleshooting Digital Experience for a Hybrid Workforce
Optimizing and Troubleshooting Digital Experience for a Hybrid Workforce
 
EMEA Optimizing and Troubleshooting Digital Experience for a Hybrid Workforce
EMEA Optimizing and Troubleshooting Digital Experience for a Hybrid WorkforceEMEA Optimizing and Troubleshooting Digital Experience for a Hybrid Workforce
EMEA Optimizing and Troubleshooting Digital Experience for a Hybrid Workforce
 
SP Virtual Managed Services (VMS) for Intelligent WAN (IWAN)
SP Virtual Managed Services (VMS) for Intelligent WAN (IWAN)SP Virtual Managed Services (VMS) for Intelligent WAN (IWAN)
SP Virtual Managed Services (VMS) for Intelligent WAN (IWAN)
 
Anti Hack Solution
Anti Hack Solution Anti Hack Solution
Anti Hack Solution
 

More from ThousandEyes

Enhancing Worker Digital Experience: A Hands-on Workshop for Partners
Enhancing Worker Digital Experience: A Hands-on Workshop for PartnersEnhancing Worker Digital Experience: A Hands-on Workshop for Partners
Enhancing Worker Digital Experience: A Hands-on Workshop for Partners
ThousandEyes
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
ThousandEyes
 

More from ThousandEyes (20)

How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
How To Troubleshoot Collaboration Apps for the Modern Connected Worker
How To Troubleshoot Collaboration Apps for the Modern Connected WorkerHow To Troubleshoot Collaboration Apps for the Modern Connected Worker
How To Troubleshoot Collaboration Apps for the Modern Connected Worker
 
Enhancing Worker Digital Experience: A Hands-on Workshop for Partners
Enhancing Worker Digital Experience: A Hands-on Workshop for PartnersEnhancing Worker Digital Experience: A Hands-on Workshop for Partners
Enhancing Worker Digital Experience: A Hands-on Workshop for Partners
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
 
Assure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyesAssure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyes
 
New ThousandEyes Product Features and Release Highlights: March 2024
New ThousandEyes Product Features and Release Highlights: March 2024New ThousandEyes Product Features and Release Highlights: March 2024
New ThousandEyes Product Features and Release Highlights: March 2024
 
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedInOutage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
 
Assure Patient and Clinician Digital Experiences with ThousandEyes for Health...
Assure Patient and Clinician Digital Experiences with ThousandEyes for Health...Assure Patient and Clinician Digital Experiences with ThousandEyes for Health...
Assure Patient and Clinician Digital Experiences with ThousandEyes for Health...
 
AMER Introduction to ThousandEyes Webinar
AMER Introduction to ThousandEyes WebinarAMER Introduction to ThousandEyes Webinar
AMER Introduction to ThousandEyes Webinar
 
New ThousandEyes Product Features and Release Highlights: February 2024
New ThousandEyes Product Features and Release Highlights: February 2024New ThousandEyes Product Features and Release Highlights: February 2024
New ThousandEyes Product Features and Release Highlights: February 2024
 
The Top Outages of 2023: Analyses and Takeaways
The Top Outages of 2023: Analyses and TakeawaysThe Top Outages of 2023: Analyses and Takeaways
The Top Outages of 2023: Analyses and Takeaways
 
Enhancing SaaS Performance: A Hands-on Workshop for Partners
Enhancing SaaS Performance: A Hands-on Workshop for PartnersEnhancing SaaS Performance: A Hands-on Workshop for Partners
Enhancing SaaS Performance: A Hands-on Workshop for Partners
 
The Top Outages of 2023: Analysis and Takeaways
The Top Outages of 2023: Analysis and TakeawaysThe Top Outages of 2023: Analysis and Takeaways
The Top Outages of 2023: Analysis and Takeaways
 
The Top Outages of 2023: Analysis and Takeaways
The Top Outages of 2023: Analysis and TakeawaysThe Top Outages of 2023: Analysis and Takeaways
The Top Outages of 2023: Analysis and Takeaways
 
ThousandEyes Enterprise Digital Workshop - Spanish
ThousandEyes Enterprise Digital Workshop - SpanishThousandEyes Enterprise Digital Workshop - Spanish
ThousandEyes Enterprise Digital Workshop - Spanish
 
ThousandEyes Enterprise Digital Workshop - German
ThousandEyes Enterprise Digital Workshop - GermanThousandEyes Enterprise Digital Workshop - German
ThousandEyes Enterprise Digital Workshop - German
 
ThousandEyes Enterprise Digital Workshop
ThousandEyes Enterprise Digital WorkshopThousandEyes Enterprise Digital Workshop
ThousandEyes Enterprise Digital Workshop
 
Introduction to ThousandEyes and Meraki MX for Partners
Introduction to ThousandEyes and Meraki MX for PartnersIntroduction to ThousandEyes and Meraki MX for Partners
Introduction to ThousandEyes and Meraki MX for Partners
 
Level-up Your Cloud Visibility Into AWS With ThousandEyes
Level-up Your Cloud Visibility Into AWS With ThousandEyesLevel-up Your Cloud Visibility Into AWS With ThousandEyes
Level-up Your Cloud Visibility Into AWS With ThousandEyes
 

Recently uploaded

Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
FIDO Alliance
 

Recently uploaded (20)

AI mind or machine power point presentation
AI mind or machine power point presentationAI mind or machine power point presentation
AI mind or machine power point presentation
 
Intro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptxIntro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptx
 
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsContinuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
 
Vector Search @ sw2con for slideshare.pptx
Vector Search @ sw2con for slideshare.pptxVector Search @ sw2con for slideshare.pptx
Vector Search @ sw2con for slideshare.pptx
 
Overview of Hyperledger Foundation
Overview of Hyperledger FoundationOverview of Hyperledger Foundation
Overview of Hyperledger Foundation
 
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
 
Introduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDMIntroduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDM
 
Simplifying Mobile A11y Presentation.pptx
Simplifying Mobile A11y Presentation.pptxSimplifying Mobile A11y Presentation.pptx
Simplifying Mobile A11y Presentation.pptx
 
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
 
Design Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptxDesign Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptx
 
Design and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data ScienceDesign and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data Science
 
JavaScript Usage Statistics 2024 - The Ultimate Guide
JavaScript Usage Statistics 2024 - The Ultimate GuideJavaScript Usage Statistics 2024 - The Ultimate Guide
JavaScript Usage Statistics 2024 - The Ultimate Guide
 
Navigating the Large Language Model choices_Ravi Daparthi
Navigating the Large Language Model choices_Ravi DaparthiNavigating the Large Language Model choices_Ravi Daparthi
Navigating the Large Language Model choices_Ravi Daparthi
 
UiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overviewUiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overview
 
JohnPollard-hybrid-app-RailsConf2024.pptx
JohnPollard-hybrid-app-RailsConf2024.pptxJohnPollard-hybrid-app-RailsConf2024.pptx
JohnPollard-hybrid-app-RailsConf2024.pptx
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
Stronger Together: Developing an Organizational Strategy for Accessible Desig...
Stronger Together: Developing an Organizational Strategy for Accessible Desig...Stronger Together: Developing an Organizational Strategy for Accessible Desig...
Stronger Together: Developing an Organizational Strategy for Accessible Desig...
 
2024 May Patch Tuesday
2024 May Patch Tuesday2024 May Patch Tuesday
2024 May Patch Tuesday
 
Working together SRE & Platform Engineering
Working together SRE & Platform EngineeringWorking together SRE & Platform Engineering
Working together SRE & Platform Engineering
 
How to Check CNIC Information Online with Pakdata cf
How to Check CNIC Information Online with Pakdata cfHow to Check CNIC Information Online with Pakdata cf
How to Check CNIC Information Online with Pakdata cf
 

Cisco IT and ThousandEyes

  • 1. Cisco IT and ThousandEyes How Cisco IT Gains Visibility into Cloud Service Stability and Troubleshooting Andrea Di Lecce IT Technical Project Manager,Cisco Systems 03/2016
  • 3. Cisco Confidential 3© 2013-2014 Cisco and/or its affiliates. All rights reserved. Why ThousandEyes? Cisco’s Requirements • Once off the Cisco network, the “Cloud” is basically a black box • Ping and traceroute have limited capability and no ability to alert or keep historical information • Growing importance and criticality of Cloud solutions requires a comprehensive solution ! Cisco’s Goals • Monitor a growing suite of Cloud solutions and end-to-end network health – latency, packet loss, Web transactions, BGP reachability • Constant monitoring and alerting from critical, strategic network locations
  • 5. Cisco Confidential 5© 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco’s EnterpriseAgent Deployment • Strategic enterprise agent placement at high-priority sites • Call centers • iPoPs • High-priority sales sites • Business-critical Cloud and internal services monitored • Salesforce • WebEx • TAC tools via Akamai
  • 6. Cisco Confidential 6© 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco’s Cloud Agent Usage • Business-critical internal services monitored from ThousandEyes’ Cloud agents • WebEx • TAC tools via Akamai • BGP reachability of our service-containing public IP address ranges from Cloud agents
  • 7. Success Stories & Lessons Learned
  • 8. Cisco Confidential 8© 2013-2014 Cisco and/or its affiliates. All rights reserved. Metrics Definition • Mean Time to Troubleshoot (MTTT) • The time it takes from the start of the incident to when the Engineer has narrowed down the source of the issue • Mean Time to Restore (MTTR) • The time it takes from the start of the incident to when the service is restored • For incidents with external providers, the timeline after handoff is beyond Cisco’s control
  • 9. Cisco Confidential 9© 2013-2014 Cisco and/or its affiliates. All rights reserved. ThousandEyesSuccess Story - Business Outcomes 1. Reduce Mean Time to Troubleshoot (MTTT) for applicable network events by 43% (measured) 2. Reduce Mean Time to Restore (MTTR) for applicable network events by 8% (measured) NOTE: Reducing MTTT and MTTR reduces Engineering time and total outage time.
  • 10. Cisco Confidential 10© 2013-2014 Cisco and/or its affiliates. All rights reserved. ThousandEyes Success Story - WebEx • Business case for ThousandEyes: The issue was automatically detected by the program, which also pinpointed that the packet loss was occurring internal to the Cisco network. This allowed Engineers to concentrate their troubleshooting efforts on the device in question, and resolve the issue quickly. • Troubleshooting: It was determined via ThousandEyes that there was packet loss into the WebEx service from an internal network device. • Resolution: Within 90 minutes, the Engineers resolved the case, because they knew where the packet loss was occurring.
  • 11. Cisco Confidential 11© 2013-2014 Cisco and/or its affiliates. All rights reserved. ThousandEyes Success Story - Salesforce • Business case for ThousandEyes: The issue was automatically detected by the program, which also pinpointed that the packet loss was occurring external to the Cisco network. This allowed Engineers to perform only the basic internal network checks, and then hand off to Salesforce to fix their network ! • Troubleshooting: It was determined via ThousandEyes that there was packet loss into salesforce.com from two India sites. • Resolution: Within one hour, the P2 case was handed to Salesforce.com for investigation. • The issue was isolated to a saturated Level3 ISP link in Salesforce's network and resolved.
  • 12. Cisco Confidential 12© 2013-2014 Cisco and/or its affiliates. All rights reserved. ThousandEyes Success Story – India Firewall • Business case for ThousandEyes: The issue was automatically detected by ThousandEyes, which also pinpointed that the packet loss was occurring on a specific device within the Cisco network. This allowed Engineers to resolve the issue quickly by failing over to the redundant gateway. • Troubleshooting: Alert from ThousandEyes detected packet loss on our India corporate gateway, which affected all India sites. • Resolution: Within one hour, the P1 case was identified with root cause, which was 100% CPU utilization on the corporate gateway device. • The issue was resolved by failing over to the backup corporate gateway.
  • 13. Cisco Confidential 13© 2013-2014 Cisco and/or its affiliates. All rights reserved. ThousandEyes Success Story – India Support Apps • Business case for ThousandEyes: The issue was proactively detected by ThousandEyes, and troubleshooting with ThousandEyes pinpointed the root cause. Since the issue was intermittent, ThousandEyes prevented multiple subsequent P2 outages. • Troubleshooting: Alert from ThousandEyes detected packet loss from a TAC site to the Cisco TAC apps portal (served by Akamai). • Resolution: ThousandEyes indicated a problematic link between Bharti and Akamai. This prompted Akamai to remove the server from rotation, which immediately restored services.
  • 14. Cisco Confidential 14© 2013-2014 Cisco and/or its affiliates. All rights reserved. Lessons Learned 1. Email alert volume was extremely high due to transient network issues • Solution: Configure alerts only if event occurs more than 2x in a row 1. VirtualBox VM did not auto-start after software update and reload of Mac Mini box • Solution: Procedure provided by ThousandEyes
  • 16. Cisco Confidential 16© 2013-2014 Cisco and/or its affiliates. All rights reserved. Looking Toward the Future • ThousandEyes has certified their application to run on the Cisco 4451 ISR Service Container • Cisco Design is currently testing this for deployment on our network ! • Our goal is to install ThousandEyes on the service containers on our existing WAN routers No more Mac Minis !
  • 17. Cisco Confidential 17© 2013-2014 Cisco and/or its affiliates. All rights reserved. Looking Toward the Future • ThousandEyes is being integrated with our Network Operations standard alerting system • ThousandEyes is working on certifying its application on the Cisco ASR 1000 Service Container