SlideShare a Scribd company logo
1 of 4
Download to read offline
Reasons why major Content Providers and Websites need an IP routing looking
glass
Author: Peter Ehiwe

Last updated: 28 March, 2013

Abstract

This document tries to justify why leading content providers like Facebook, Twitter, Google, Amazon
need to setup BGP looking glass at least for the good of the internet operations.

Justification

I am presenting a compelling reason(s) why these popular websites need to have a looking glass set up
in their network, some of the benefits to the content providers will be:

       Reduced number of Fault tickets handled by their NOC team.
       Increased productivity as the NOC team can focus on more pressing network infrastructure
        issues.
       It will be good for the general operations of the internet.
       Increased traffic to your website.
       Ultimately more revenue as the traffic to websites increase.

I have seen several cases where a content provider having a looking glass will aid the troubleshooting
efforts of ISP network engineers; however I will use the most interesting scenario encountered so far to
explain further.
Scenario

               F.com




                                                                                    Upstream 3(U3)
             F.com Edge router




                LYNX(An internet
               exchange in London)                                                  Upstream 2(U2)




               ISP X Router
                in London                                                           Upstream 1(U1)

                       LONDON




                                                                                        ISP M
           ISP X Router in Nigeria

                                                                                                   Full BGP Feed
                              NIGERIA
                                                                   Default route
                   Internet                          ISP Z                            Customer Y
                 Exchange in
                    Nigeria



Problem Statement

Customer Y reported to ISP M that it cannot open www.F.com which is a popular social networking
service provided by F company.

Troubleshooting

From experience, I like to troubleshoot these kinds of fault tickets on Layers 3, 4, 7 of the OSI model in
that particular order.

At Layer 3: Ping and traceroute to the IP address of the content site will suffice. Remember Traceroute is
unidirectional, so you need to have forward and reverse traces to get the full picture.

At Layer 4: Telnet to the website IP address on port 80 should suffice; if you are unlucky  you may
need to look at MSS issues or even MTU issues (at Layer 2)
At Layer 7: Look at the user browser for proxy settings, cookies, malware.

Most times the trouble tickets are usually resolved at Layer 3 or Layer 4 of the OSI model .

So back to the scenario, ping to the website failed from the edge router of Customer Y and forward trace
showed timeouts at certain IP address within the administrative control of F.com, similar tests from
other ISPs within the region networks yielded successful result to www.F.com , after this test there is
100% certainty that the root cause si routing related.

As aforementioned, another troubleshooting step is to perform at Layer 3 is forward and reverse traces.
Forward trace to F.com IP address showing outbound traffic goes from Customer Y -> ISP M -> U1 -> U2
->U3 -> F.com.

But how do you get reverse trace from F.com? Easiest way would be to check their looking glass but you
can’t because F.com doesn’t have a looking glass, so the engineer the “next best thing” and tries reverse
trace from different Tier 1 ISPs around the world closest to www.F.com .

The flaw with the above approach is most of the Tier 1’s will show a reverse trace similar to U3->U2-
>U1->ISP M->Customer Y . This presents a false representation that both forward and reverse path
routing is symmetric and could lead the engineer on the wrong troubleshooting path.

The last, longer and more accurate option is to contact the Content provider directly for first

a) A reverse trace to Customer Y IP address (if you are lucky you will get a response from them  ),

b) the next thing to do is to identify the AS PATH from F.com to Customer Y, this can be obtained from
the BGP control plane information from the content provider.

The reverse trace output can help identify routing loops and in some cases BCP 38 enforcement (look
out for !X or administratively prohibited messages) .

Requesting for reverse trace revealed bad routing loops or bad routing within ISPX and BCP 38
enforcement by ISP M for source address validation which ensured that ISP M didn’t provide free transit
for F.com to Customer Y. The BGP as path information provided enabled us see why the loop was
occurring and ultimately the root cause of the issue.

Root cause

The root cause was due to route leaking of the prefixes learned from the internet exchange in Nigeria to
the public internet by ISP X ( Making ISP X a potential provider of free transit for F.com to Customer Y ,
this scenario was prevented by ISP M which performs source IP address validation for its transit
customers hence denying IP packet with source address from F.com destined to Customer Y coming
from its connection to ISP X) , this caused reachability issues for Customer Y because ISP X has a direct
peering with F.com at the internet exchange LYNX in London , therefore BGP prefers to route back from
F.com to Customer Y via LYNX due to shorter AS path ultimately causing return traffic to be dropped by
ISP M since ISP X prefers to reach Customer Y via its direct connection to ISP M .
Solution:

ISP X fixed the Route leaking and the issue was resolved

Conclusion:

Without the cooperation and information provided by F.com NOC, the problem will have been left
unsolved. Since I passionately believe in Internet Transparency, I am using this medium to reach out to
these leading content providers and social networking sites to create publicly available routing looking
glasses that will include all Edge Routers in their network. This looking glass will also have useful
network tools like Ping, Traceroute and appropriate verification commands to show relevant BGP
information.

For the ISPs that provide the eyeballs for these major websites , the implementation on the looking
glass by the major websites will help improve response time to our valuable customers , because rather
than waiting for a response from the concerned website NOC team , ISP Engineers can easily get the
information from the Looking glass of the content provider.

Looking glasses are easy to setup and a lot of open source code already exist on the internet, so setting
this up shouldn’t be an issue for major websites and content providers.



* Fictitious names are used in this document for representation of a real life event.

More Related Content

Viewers also liked

[A d206-9] pp.41-48-pencarian_jalur_terpendek_untuk_robot_micromouse_dengan
[A d206-9] pp.41-48-pencarian_jalur_terpendek_untuk_robot_micromouse_dengan[A d206-9] pp.41-48-pencarian_jalur_terpendek_untuk_robot_micromouse_dengan
[A d206-9] pp.41-48-pencarian_jalur_terpendek_untuk_robot_micromouse_denganTeguh Santoso
 
Laboratorio formez modello di gestione del recupero crediti per le pubbliche ...
Laboratorio formez modello di gestione del recupero crediti per le pubbliche ...Laboratorio formez modello di gestione del recupero crediti per le pubbliche ...
Laboratorio formez modello di gestione del recupero crediti per le pubbliche ...Salvatore [Sasa'] Barresi
 
Mid term presentation-chimulwa
Mid term presentation-chimulwaMid term presentation-chimulwa
Mid term presentation-chimulwachimulwa
 
The Autism Spectrum – A Guide to the World of Autistic Disorders
The Autism Spectrum – A Guide to the World of Autistic DisordersThe Autism Spectrum – A Guide to the World of Autistic Disorders
The Autism Spectrum – A Guide to the World of Autistic DisordersShema Kolainu - Hear Our Voices
 
Technlogoy Action Plan
Technlogoy Action PlanTechnlogoy Action Plan
Technlogoy Action PlanRubberman
 
Who am i presentation
Who am i presentationWho am i presentation
Who am i presentationbdapple2
 
Star Chart Presentation
Star Chart PresentationStar Chart Presentation
Star Chart Presentationljack2011
 
Learning & decelopment in HP
Learning & decelopment in HPLearning & decelopment in HP
Learning & decelopment in HPtomthin2805
 
Route object white paper update1jun252012
Route object white paper update1jun252012Route object white paper update1jun252012
Route object white paper update1jun252012peterehiwe
 
生涯規劃 養一
生涯規劃 養一生涯規劃 養一
生涯規劃 養一Fuhan Hu
 
La Programmazione Regionale Calabrese In Campo Ambientale 2007 2013 [S.Barresi]
La Programmazione Regionale Calabrese In Campo Ambientale 2007 2013 [S.Barresi]La Programmazione Regionale Calabrese In Campo Ambientale 2007 2013 [S.Barresi]
La Programmazione Regionale Calabrese In Campo Ambientale 2007 2013 [S.Barresi]Salvatore [Sasa'] Barresi
 
Ann devlin edld5352
Ann devlin edld5352Ann devlin edld5352
Ann devlin edld5352annmariedev
 
Technlogoy Action Plan
Technlogoy Action PlanTechnlogoy Action Plan
Technlogoy Action PlanRubberman
 
Musical invaders
Musical invadersMusical invaders
Musical invadersvigarilla
 
第十章 休閒生活
第十章 休閒生活第十章 休閒生活
第十章 休閒生活Fuhan Hu
 

Viewers also liked (17)

[A d206-9] pp.41-48-pencarian_jalur_terpendek_untuk_robot_micromouse_dengan
[A d206-9] pp.41-48-pencarian_jalur_terpendek_untuk_robot_micromouse_dengan[A d206-9] pp.41-48-pencarian_jalur_terpendek_untuk_robot_micromouse_dengan
[A d206-9] pp.41-48-pencarian_jalur_terpendek_untuk_robot_micromouse_dengan
 
Final trees copy
Final trees   copyFinal trees   copy
Final trees copy
 
Laboratorio formez modello di gestione del recupero crediti per le pubbliche ...
Laboratorio formez modello di gestione del recupero crediti per le pubbliche ...Laboratorio formez modello di gestione del recupero crediti per le pubbliche ...
Laboratorio formez modello di gestione del recupero crediti per le pubbliche ...
 
Mid term presentation-chimulwa
Mid term presentation-chimulwaMid term presentation-chimulwa
Mid term presentation-chimulwa
 
The Autism Spectrum – A Guide to the World of Autistic Disorders
The Autism Spectrum – A Guide to the World of Autistic DisordersThe Autism Spectrum – A Guide to the World of Autistic Disorders
The Autism Spectrum – A Guide to the World of Autistic Disorders
 
Technlogoy Action Plan
Technlogoy Action PlanTechnlogoy Action Plan
Technlogoy Action Plan
 
Who am i presentation
Who am i presentationWho am i presentation
Who am i presentation
 
Star Chart Presentation
Star Chart PresentationStar Chart Presentation
Star Chart Presentation
 
Learning & decelopment in HP
Learning & decelopment in HPLearning & decelopment in HP
Learning & decelopment in HP
 
Route object white paper update1jun252012
Route object white paper update1jun252012Route object white paper update1jun252012
Route object white paper update1jun252012
 
生涯規劃 養一
生涯規劃 養一生涯規劃 養一
生涯規劃 養一
 
La Programmazione Regionale Calabrese In Campo Ambientale 2007 2013 [S.Barresi]
La Programmazione Regionale Calabrese In Campo Ambientale 2007 2013 [S.Barresi]La Programmazione Regionale Calabrese In Campo Ambientale 2007 2013 [S.Barresi]
La Programmazione Regionale Calabrese In Campo Ambientale 2007 2013 [S.Barresi]
 
Ann devlin edld5352
Ann devlin edld5352Ann devlin edld5352
Ann devlin edld5352
 
Technlogoy Action Plan
Technlogoy Action PlanTechnlogoy Action Plan
Technlogoy Action Plan
 
Musical invaders
Musical invadersMusical invaders
Musical invaders
 
第十章 休閒生活
第十章 休閒生活第十章 休閒生活
第十章 休閒生活
 
2
22
2
 

Recently uploaded

Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)Kaya Weers
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality AssuranceInflectra
 
Assure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyesAssure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyesThousandEyes
 
WomenInAutomation2024: AI and Automation for eveyone
WomenInAutomation2024: AI and Automation for eveyoneWomenInAutomation2024: AI and Automation for eveyone
WomenInAutomation2024: AI and Automation for eveyoneUiPathCommunity
 
Generative AI - Gitex v1Generative AI - Gitex v1.pptx
Generative AI - Gitex v1Generative AI - Gitex v1.pptxGenerative AI - Gitex v1Generative AI - Gitex v1.pptx
Generative AI - Gitex v1Generative AI - Gitex v1.pptxfnnc6jmgwh
 
Manual 508 Accessibility Compliance Audit
Manual 508 Accessibility Compliance AuditManual 508 Accessibility Compliance Audit
Manual 508 Accessibility Compliance AuditSkynet Technologies
 
Glenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security ObservabilityGlenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security Observabilityitnewsafrica
 
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Alkin Tezuysal
 
Kuma Meshes Part I - The basics - A tutorial
Kuma Meshes Part I - The basics - A tutorialKuma Meshes Part I - The basics - A tutorial
Kuma Meshes Part I - The basics - A tutorialJoão Esperancinha
 
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfSo einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfpanagenda
 
All These Sophisticated Attacks, Can We Really Detect Them - PDF
All These Sophisticated Attacks, Can We Really Detect Them - PDFAll These Sophisticated Attacks, Can We Really Detect Them - PDF
All These Sophisticated Attacks, Can We Really Detect Them - PDFMichael Gough
 
React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...Karmanjay Verma
 
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...Jeffrey Haguewood
 
UiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to HeroUiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to HeroUiPathCommunity
 
Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Farhan Tariq
 
QCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architecturesQCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architecturesBernd Ruecker
 
Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Hiroshi SHIBATA
 
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...itnewsafrica
 
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better StrongerModern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better Strongerpanagenda
 
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...Nikki Chapple
 

Recently uploaded (20)

Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
 
Assure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyesAssure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyes
 
WomenInAutomation2024: AI and Automation for eveyone
WomenInAutomation2024: AI and Automation for eveyoneWomenInAutomation2024: AI and Automation for eveyone
WomenInAutomation2024: AI and Automation for eveyone
 
Generative AI - Gitex v1Generative AI - Gitex v1.pptx
Generative AI - Gitex v1Generative AI - Gitex v1.pptxGenerative AI - Gitex v1Generative AI - Gitex v1.pptx
Generative AI - Gitex v1Generative AI - Gitex v1.pptx
 
Manual 508 Accessibility Compliance Audit
Manual 508 Accessibility Compliance AuditManual 508 Accessibility Compliance Audit
Manual 508 Accessibility Compliance Audit
 
Glenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security ObservabilityGlenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security Observability
 
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
 
Kuma Meshes Part I - The basics - A tutorial
Kuma Meshes Part I - The basics - A tutorialKuma Meshes Part I - The basics - A tutorial
Kuma Meshes Part I - The basics - A tutorial
 
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfSo einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
 
All These Sophisticated Attacks, Can We Really Detect Them - PDF
All These Sophisticated Attacks, Can We Really Detect Them - PDFAll These Sophisticated Attacks, Can We Really Detect Them - PDF
All These Sophisticated Attacks, Can We Really Detect Them - PDF
 
React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...
 
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
 
UiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to HeroUiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to Hero
 
Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...
 
QCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architecturesQCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architectures
 
Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024
 
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
 
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better StrongerModern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
 
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
 

Why major content providers need an ip looking glass

  • 1. Reasons why major Content Providers and Websites need an IP routing looking glass Author: Peter Ehiwe Last updated: 28 March, 2013 Abstract This document tries to justify why leading content providers like Facebook, Twitter, Google, Amazon need to setup BGP looking glass at least for the good of the internet operations. Justification I am presenting a compelling reason(s) why these popular websites need to have a looking glass set up in their network, some of the benefits to the content providers will be:  Reduced number of Fault tickets handled by their NOC team.  Increased productivity as the NOC team can focus on more pressing network infrastructure issues.  It will be good for the general operations of the internet.  Increased traffic to your website.  Ultimately more revenue as the traffic to websites increase. I have seen several cases where a content provider having a looking glass will aid the troubleshooting efforts of ISP network engineers; however I will use the most interesting scenario encountered so far to explain further.
  • 2. Scenario F.com Upstream 3(U3) F.com Edge router LYNX(An internet exchange in London) Upstream 2(U2) ISP X Router in London Upstream 1(U1) LONDON ISP M ISP X Router in Nigeria Full BGP Feed NIGERIA Default route Internet ISP Z Customer Y Exchange in Nigeria Problem Statement Customer Y reported to ISP M that it cannot open www.F.com which is a popular social networking service provided by F company. Troubleshooting From experience, I like to troubleshoot these kinds of fault tickets on Layers 3, 4, 7 of the OSI model in that particular order. At Layer 3: Ping and traceroute to the IP address of the content site will suffice. Remember Traceroute is unidirectional, so you need to have forward and reverse traces to get the full picture. At Layer 4: Telnet to the website IP address on port 80 should suffice; if you are unlucky  you may need to look at MSS issues or even MTU issues (at Layer 2)
  • 3. At Layer 7: Look at the user browser for proxy settings, cookies, malware. Most times the trouble tickets are usually resolved at Layer 3 or Layer 4 of the OSI model . So back to the scenario, ping to the website failed from the edge router of Customer Y and forward trace showed timeouts at certain IP address within the administrative control of F.com, similar tests from other ISPs within the region networks yielded successful result to www.F.com , after this test there is 100% certainty that the root cause si routing related. As aforementioned, another troubleshooting step is to perform at Layer 3 is forward and reverse traces. Forward trace to F.com IP address showing outbound traffic goes from Customer Y -> ISP M -> U1 -> U2 ->U3 -> F.com. But how do you get reverse trace from F.com? Easiest way would be to check their looking glass but you can’t because F.com doesn’t have a looking glass, so the engineer the “next best thing” and tries reverse trace from different Tier 1 ISPs around the world closest to www.F.com . The flaw with the above approach is most of the Tier 1’s will show a reverse trace similar to U3->U2- >U1->ISP M->Customer Y . This presents a false representation that both forward and reverse path routing is symmetric and could lead the engineer on the wrong troubleshooting path. The last, longer and more accurate option is to contact the Content provider directly for first a) A reverse trace to Customer Y IP address (if you are lucky you will get a response from them  ), b) the next thing to do is to identify the AS PATH from F.com to Customer Y, this can be obtained from the BGP control plane information from the content provider. The reverse trace output can help identify routing loops and in some cases BCP 38 enforcement (look out for !X or administratively prohibited messages) . Requesting for reverse trace revealed bad routing loops or bad routing within ISPX and BCP 38 enforcement by ISP M for source address validation which ensured that ISP M didn’t provide free transit for F.com to Customer Y. The BGP as path information provided enabled us see why the loop was occurring and ultimately the root cause of the issue. Root cause The root cause was due to route leaking of the prefixes learned from the internet exchange in Nigeria to the public internet by ISP X ( Making ISP X a potential provider of free transit for F.com to Customer Y , this scenario was prevented by ISP M which performs source IP address validation for its transit customers hence denying IP packet with source address from F.com destined to Customer Y coming from its connection to ISP X) , this caused reachability issues for Customer Y because ISP X has a direct peering with F.com at the internet exchange LYNX in London , therefore BGP prefers to route back from F.com to Customer Y via LYNX due to shorter AS path ultimately causing return traffic to be dropped by ISP M since ISP X prefers to reach Customer Y via its direct connection to ISP M .
  • 4. Solution: ISP X fixed the Route leaking and the issue was resolved Conclusion: Without the cooperation and information provided by F.com NOC, the problem will have been left unsolved. Since I passionately believe in Internet Transparency, I am using this medium to reach out to these leading content providers and social networking sites to create publicly available routing looking glasses that will include all Edge Routers in their network. This looking glass will also have useful network tools like Ping, Traceroute and appropriate verification commands to show relevant BGP information. For the ISPs that provide the eyeballs for these major websites , the implementation on the looking glass by the major websites will help improve response time to our valuable customers , because rather than waiting for a response from the concerned website NOC team , ISP Engineers can easily get the information from the Looking glass of the content provider. Looking glasses are easy to setup and a lot of open source code already exist on the internet, so setting this up shouldn’t be an issue for major websites and content providers. * Fictitious names are used in this document for representation of a real life event.