1.
Reasons why major Content Providers and Websites need an IP routing lookingglassAuthor: Peter EhiweLast updated: 28 March, 2013AbstractThis document tries to justify why leading content providers like Facebook, Twitter, Google, Amazonneed to setup BGP looking glass at least for the good of the internet operations.JustificationI am presenting a compelling reason(s) why these popular websites need to have a looking glass set upin their network, some of the benefits to the content providers will be:  Reduced number of Fault tickets handled by their NOC team.  Increased productivity as the NOC team can focus on more pressing network infrastructure issues.  It will be good for the general operations of the internet.  Increased traffic to your website.  Ultimately more revenue as the traffic to websites increase.I have seen several cases where a content provider having a looking glass will aid the troubleshootingefforts of ISP network engineers; however I will use the most interesting scenario encountered so far toexplain further.

2.
Scenario F.com Upstream 3(U3) F.com Edge router LYNX(An internet exchange in London) Upstream 2(U2) ISP X Router in London Upstream 1(U1) LONDON ISP M ISP X Router in Nigeria Full BGP Feed NIGERIA Default route Internet ISP Z Customer Y Exchange in NigeriaProblem StatementCustomer Y reported to ISP M that it cannot open www.F.com which is a popular social networkingservice provided by F company.TroubleshootingFrom experience, I like to troubleshoot these kinds of fault tickets on Layers 3, 4, 7 of the OSI model inthat particular order.At Layer 3: Ping and traceroute to the IP address of the content site will suffice. Remember Traceroute isunidirectional, so you need to have forward and reverse traces to get the full picture.At Layer 4: Telnet to the website IP address on port 80 should suffice; if you are unlucky  you mayneed to look at MSS issues or even MTU issues (at Layer 2)

3.
At Layer 7: Look at the user browser for proxy settings, cookies, malware.Most times the trouble tickets are usually resolved at Layer 3 or Layer 4 of the OSI model .So back to the scenario, ping to the website failed from the edge router of Customer Y and forward traceshowed timeouts at certain IP address within the administrative control of F.com, similar tests fromother ISPs within the region networks yielded successful result to www.F.com , after this test there is100% certainty that the root cause si routing related.As aforementioned, another troubleshooting step is to perform at Layer 3 is forward and reverse traces.Forward trace to F.com IP address showing outbound traffic goes from Customer Y -> ISP M -> U1 -> U2->U3 -> F.com.But how do you get reverse trace from F.com? Easiest way would be to check their looking glass but youcan’t because F.com doesn’t have a looking glass, so the engineer the “next best thing” and tries reversetrace from different Tier 1 ISPs around the world closest to www.F.com .The flaw with the above approach is most of the Tier 1’s will show a reverse trace similar to U3->U2->U1->ISP M->Customer Y . This presents a false representation that both forward and reverse pathrouting is symmetric and could lead the engineer on the wrong troubleshooting path.The last, longer and more accurate option is to contact the Content provider directly for firsta) A reverse trace to Customer Y IP address (if you are lucky you will get a response from them  ),b) the next thing to do is to identify the AS PATH from F.com to Customer Y, this can be obtained fromthe BGP control plane information from the content provider.The reverse trace output can help identify routing loops and in some cases BCP 38 enforcement (lookout for !X or administratively prohibited messages) .Requesting for reverse trace revealed bad routing loops or bad routing within ISPX and BCP 38enforcement by ISP M for source address validation which ensured that ISP M didn’t provide free transitfor F.com to Customer Y. The BGP as path information provided enabled us see why the loop wasoccurring and ultimately the root cause of the issue.Root causeThe root cause was due to route leaking of the prefixes learned from the internet exchange in Nigeria tothe public internet by ISP X ( Making ISP X a potential provider of free transit for F.com to Customer Y ,this scenario was prevented by ISP M which performs source IP address validation for its transitcustomers hence denying IP packet with source address from F.com destined to Customer Y comingfrom its connection to ISP X) , this caused reachability issues for Customer Y because ISP X has a directpeering with F.com at the internet exchange LYNX in London , therefore BGP prefers to route back fromF.com to Customer Y via LYNX due to shorter AS path ultimately causing return traffic to be dropped byISP M since ISP X prefers to reach Customer Y via its direct connection to ISP M .

4.
Solution:ISP X fixed the Route leaking and the issue was resolvedConclusion:Without the cooperation and information provided by F.com NOC, the problem will have been leftunsolved. Since I passionately believe in Internet Transparency, I am using this medium to reach out tothese leading content providers and social networking sites to create publicly available routing lookingglasses that will include all Edge Routers in their network. This looking glass will also have usefulnetwork tools like Ping, Traceroute and appropriate verification commands to show relevant BGPinformation.For the ISPs that provide the eyeballs for these major websites , the implementation on the lookingglass by the major websites will help improve response time to our valuable customers , because ratherthan waiting for a response from the concerned website NOC team , ISP Engineers can easily get theinformation from the Looking glass of the content provider.Looking glasses are easy to setup and a lot of open source code already exist on the internet, so settingthis up shouldn’t be an issue for major websites and content providers.* Fictitious names are used in this document for representation of a real life event.