Bangladesh Network Operators Group, profile picture
Uploaded byBangladesh Network Operators Group
PDF, PPTX2,674 views

Traffic Engineering for CDNs

The document discusses how content delivery networks (CDNs) use traffic engineering and DNS-based mapping to direct user requests to optimal edge servers. It provides examples of Akamai's global infrastructure and mapping techniques. Key points made include that CDNs have no backbone and operate as independent clusters, and that BGP configurations between networks may not achieve the intended traffic steering results for CDNs. Cooperation between networks and CDNs is suggested to optimize traffic patterns.

Embed presentation

Download as PDF, PPTX
Traffic Engineering for CDNs Matt Jansen Akamai Technologies BDNOG3
©2012 AKAMAI | FASTER FORWARDTM The world’s largest on-demand, distributed computing platform delivers all forms of web content and applications The Akamai Intelligent Platform Typical daily traffic: •  More than 2 trillion requests served •  Delivering over 25 Terabits/second •  15-30% of all daily web traffic The Akamai Intelligent Platform: 170,000+ Servers 2,000+ Locations 102+ Countries 1,300+ Networks 700+ Cities
©2012 AKAMAI | FASTER FORWARDTM When content is requested from CDNs, the user is directed to the optimal server to serve this user There’s 2 common ways to do that: •  anycast: the content is served from the location the request is received (easy to build, requires symmetric routing to work well) •  DNS based: the CDN decides where to best serve the content from based on the resolver it receives the request from, and replies with the optimal server How CDNs Work
©2012 AKAMAI | FASTER FORWARDTM Users querying a DNS-based CDNs will be returned different A (and AAAA) records for the same hostname depending on the resolver the request comes from This is called “mapping” The better the mapping, the better the CDN How DNS based CDNs Work
©2012 AKAMAI | FASTER FORWARDTM Example of Akamai mapping • Notice the different A records for different locations: [NYC]% host www.symantec.com www.symantec.com CNAME e5211.b.akamaiedge.net. e5211.b.akamaiedge.net. A 207.40.194.46 e5211.b.akamaiedge.net. A 207.40.194.49 [Boston]% host www.symantec.com www.symantec.com CNAME e5211.b.akamaiedge.net. e5211.b.akamaiedge.net. A 81.23.243.152 e5211.b.akamaiedge.net. A 81.23.243.145 DNS based Mapping Example
©2012 AKAMAI | FASTER FORWARDTM CDNs use multiple criteria to choose the optimal server • These include standard network metrics: • Latency • Throughput • Packet loss • as well as internal ones such as: • CPU load on the server • HD space • network utilization How DNS based CDNs Work
©2012 AKAMAI | FASTER FORWARDTM DNS based Mapping 1)  end-user requests www.example.com from ISP NS 2)  ISP NS recursively (multiple iterations) looks up www.example.com being referred to authoritative CDN NS (by CNAME) 3)  ISP NS asks authoritative CDN NS 4)  CDN NS looks up IP of requestor (ISP NS) and replies with IP of optimal cluster to serve content (closest cluster for that ISP) 5)  ISP NS replies to end-user who 6)  requests content from local Cluster end-user ISP NS 1.2.3.4 root/tld/intermediate NS (recursive lookup until reaching authoritative NS) CDN NS Closest cluster 5.6.7.8 example.com? example.com? a212.g.akamai.net 5.6.7.8 1 3 6 2 NS 1.2.3.4? best cluster = 5.6.7.8 4 5
©2012 AKAMAI | FASTER FORWARDTM Types of CDN Deployments (simplified) end-user ISP Internet IX Transit Embedded cluster in ISP Colocated Cluster with IX Infrastructure Cluster with Transit Backend Mapping system
©2012 AKAMAI | FASTER FORWARDTM DNS based CDNs don’t necessarily have a backbone Those clusters can be standalone ‘Islands’ with no connectivity between them! DNS based CDNs usually don’t announce large blocks of address space as this will only be the local servers •  it is not uncommon to see a single /24 from Akamai at an IX •  you will receive a different set of prefixes on each peering This does not mean you will not see a lot of traffic •  how many servers do you need to serve 10g nowadays? Typical DNS based Topology
©2012 AKAMAI | FASTER FORWARDTM •  no single cluster can accommodate all content •  caches get more efficient with with size •  some content requires specialized servers only present in Infrastructure clusters •  some content is only present in specific geographies •  CDNs might prefer on-net cluster over peering Why don’t I get all CDN Traffic locally?
©2012 AKAMAI | FASTER FORWARDTM Performance •  getting as close as possible to the end-user (removing intermediate ASNs) decreasing latency/increasing troughput Burstability •  for large event peaks direct connectivity to multiple networks allows for higher burstability than a single connection to a transit provider Redundancy •  Serve as overflow and backup for embedded on-net clusters Scale •  Large deployment at an IX can serve traffic that does not ‘fit’ into smaller embedded clusters Why CDNs peer
©2012 AKAMAI | FASTER FORWARDTM Performance •  ISP’s end-users benefit from direct connectivity Competitive Advantages •  improving performance over competitors •  additional revenue from downstreams Cost Reduction •  Save on transit bill and potential backbone costs Redundancy •  Serve as overflow and backup for embedded on-net clusters Why ISPs peer with CDNs
©2012 AKAMAI | FASTER FORWARDTM Transit Peer •  Akamai does not have a backbone, each IX instance is independent •  Cluster uses transit to fetch content origin •  Content is served to peers over the IX •  BGP session serves 2 purposes: •  Route traffic strictly within the local instance •  Tell our system which prefixes this cluster is allowed to serve •  New prefixes being picked up by the system can take up to 24hrs Origin Server IX Content Typical large IX deployment Origin
Scenario 1: prefix withdrawal
©2012 AKAMAI | FASTER FORWARDTM Attempting to shift traffic to another path ISP peers with CDN over 2 IXs •  wants to shift traffic from A to B due to reduce backbone traffic •  Typical BGP based techniques: 1)  AS-Path prepending 2)  MEDs 3)  more/less specific announcements None of the above have the desired effect! •  no network between clusters, any BGP parameters are only locally relevant •  Might have undesired effect of shifting traffic to a upstream/competitor on the same IX end-user ISP A IX A IX B
©2012 AKAMAI | FASTER FORWARDTM Problem solved… end-user IX B ISP withdraws some of their prefixes over IX A •  Traffic falls back to transit on the same cluster (provided that still has the best performance) •  this might result in them receiving the traffic in another location in their network so they’re happy Transit ISP A IX A
©2012 AKAMAI | FASTER FORWARDTM …but not for long within 24hrs •  The CDN backend system processes the fact that they don’t receive their prefixes at the IX A cluster •  Traffic switches to another cluster where they do receive them (this might be one of their upstreams they have a PNI with in the same city) •  Traffic comes back into their network again at the same router! Upstream of ISP A end-user Transit ISP A IX A
©2012 AKAMAI | FASTER FORWARDTM Issues •  BGP parameters only locally relevant •  Delayed effect of BGP announcements on Mapping System •  CDNs typically see your prefixes in many different locations
©2012 AKAMAI | FASTER FORWARDTM Better solution •  Talk to the CDN if you have issues with their traffic in a specific location •  You can work together to achieve the result you’re looking for •  Get a local embedded cluster
Scenario 2: more specific Route Announcement
©2012 AKAMAI | FASTER FORWARDTM •  ISP A is multi-homed to Transit Providers AS2002 and AS3003 •  Transit Provider AS2002 peers with CDN •  Transit Provider AS3003 does not peer with CDN •  CDN sends traffic to ISP A via Transit Provider AS2002 ISP A AS1001 CDN AS20940 Transit Provider AS2002 Transit Provider AS3003 100.100.96.0/20 100.100.96.0/20 0.0.0.0/0 Transit Provider AS4003 Consistent Announcements IX
©2012 AKAMAI | FASTER FORWARDTM •  ISP A would like to balance traffic between the two upstream providers •  ISP A prepends, then applies MED to Transit Provider AS2002. This has no effect on CDN traffic. •  Eventually, ISP A de-aggregates the /20 and advertises more specific routes to Transit Provider AS3003 •  What will happen? Loadbalancing
©2012 AKAMAI | FASTER FORWARDTM •  ISP A announces more specific routes to Transit Provider AS3003 •  Transit Provider AS3003 announces new /24 to AS2002 •  CDN IX router does not have a full-table, so traffic continue route to the /20 of AS2002 •  ISP A is happy with the balanced traffic on dual Transit Providers ISP A AS1001 CDN AS20940 Transit Provider AS2002 Transit Provider AS3003 100.100.96.0/20 AS2002 Routing Table 100.100.100.0/24 AS3003 AS1001 100.100.99.0/24 AS3003 AS1001 100.100.96.0/20 AS1001 CDN AS20940 Routing Table 100.100.96.0/20 AS2002 AS1001 0.0.0.0/0 AS4003 100.100.96.0/20 0.0.0.0/0 Transit Provider AS4003 Loadbalancing works... IX Peering
©2012 AKAMAI | FASTER FORWARDTM •  Lost of revenue for Transit Provider AS2002 even though their peering/backbone is utilized •  What happens if AS2002 does not like the traffic from one peer to the other? …but
©2012 AKAMAI | FASTER FORWARDTM Transit provider filters traffic •  In order to get rid of traffic between peers, Transit Provider AS2002 implements an ACL on IX port facing AS3003 •  Traffic gets blackholed, ISP A’s eyeballs don’t receive traffic anymore! ISP A AS1001 CDN AS20940 Transit Provider AS2002 Transit Provider AS3003 100.100.96.0/20 100.100.96.0/20 ACL 0.0.0.0/0 Transit Provider AS4003 hostname AS2002-R1 ! interface TenGigabitEthernet1/1 ip access-group 101 out ! access-list 101 deny ip any 100.100.100.0 0.0.0.255 access-list 101 deny ip any 100.100.99.0 0.0.0.255 access-list 101 permit ip any any IX Peering
©2012 AKAMAI | FASTER FORWARDTM •  CDN observes ISP A end-users are unable to access some websites •  CDN stops serving unreliable prefixes received from Transit Provider AS2002, traffic shifts from IX to Transit Provider AS4003 •  ISP A can access all websites happily •  Transit Provider AS2002 loses revenue ISP A AS1001 CDN AS20940 Transit Provider AS2002 Transit Provider AS3003 100.100.96.0/20 100.100.96.0/20 ACL 0.0.0.0/0 Transit Provider AS4003 Unintended Result IX Peering
©2012 AKAMAI | FASTER FORWARDTM Issues •  Don’t assume a full-table on any device on the internet •  Filtering traffic results in: •  short term traffic blackholing! •  long term widthdrawal of traffic resulting in revenue loss
©2012 AKAMAI | FASTER FORWARDTM •  AS2002 filters the specific prefixes instead of the actual traffic •  work with upstreams and/or CDN for finetuning •  Get Transit Provider AS3003 to peer with CDN directly ;) •  Get a local embedded cluster ISP A AS1001 CDN AS20940 Transit Provider AS2002 Transit Provider AS3003 100.100.96.0/20 100.100.96.0/20 Filter Specific route 100.100.99.0/24 100.100.100.0/24 0.0.0.0/0 Transit Provider AS4003 neighbor PEER-GROUP prefix-list DENY-SPECIFIC in ! ip prefix-list DENY-SPECIFIC seq 5 deny 100.100.100.0/24 ip prefix-list DENY-SPECIFIC seq 10 deny 100.100.99.0/24 ip prefix-list DENY-SPECIFIC seq 100 permit 0.0.0.0/0 le 32 Better solutions IX Peering
©2012 AKAMAI | FASTER FORWARDTM •  ISP A is single homed to Transit Provider AS2002 •  ISP A obtains a /24 from Transit Provider AS2002’s address space •  all works well ISP A AS1001 CDN AS20940 Transit Provider AS2002 100.100.97.0/24100.100.96.0/20 100.100.96.0/20 0.0.0.0/0 Transit Provider AS4003 100.100.97.0/24 CDN AS20940 Routing Table 100.100.96.0/20 AS2002 100.100.97.0/24 AS2002 AS1001 0.0.0.0/0 AS4003 100.100.97.0/24 Another variation of this Scenario IX
©2012 AKAMAI | FASTER FORWARDTM •  ISP A moves to new Transit Provider AS3003, but keeps using his previously assigned prefix 100.100.96.0/24 •  CDN keeps serving traffic to ISP A via Transit Provider AS2002 due to the /20 being received there ISP A AS1001 CDN AS20940 Transit Provider AS2002 100.100.97.0/24 100.100.96.0/20 100.100.96.0/20 0.0.0.0/0 Transit Provider AS4003 100.100.97.0/24 CDN AS20940 Routing Table 100.100.96.0/20 AS2002 0.0.0.0/0 AS4003 Transit Provider AS3003 Provider Change Peering IX
©2012 AKAMAI | FASTER FORWARDTM •  Lost of revenue for Transit Provider AS2002 even though their peering/backbone is utilized •  What happens if AS2002 does not like the traffic from one peer to the other? …but
©2012 AKAMAI | FASTER FORWARDTM •  In order to get rid of traffic between peers, Transit Provider AS2002 implements an ACL on IX port facing AS3003 •  Traffic gets blackholed, ISP A’s eyeballs don’t receive traffic anymore! ISP A AS1001 CDN AS20940 Transit Provider AS2002 100.100.97.0/24 100.100.96.0/20 100.100.96.0/20 0.0.0.0/0 Transit Provider AS4003 100.100.97.0/24 CDN AS20940 Routing Table 100.100.96.0/20 AS2002 0.0.0.0/0 AS4003 Transit Provider AS3003 ACL hostname AS2002-R1 ! interface TenGigabitEthernet1/1 ip access-group 101 out ! access-list 101 deny ip any 100.100.97.0 0.0.0.255 access-list 101 permit ip any any Transit provider filters traffic IX Peering
©2012 AKAMAI | FASTER FORWARDTM •  CDN observes ISP A end-users are unable to access some websites •  CDN stops serving unreliable prefixes received from Transit Provider AS2002, traffic shifts from IX to Transit Provider AS4003 •  ISP A can access all websites happily •  Transit Provider AS2002 loses revenue ISP A AS1001 CDN AS20940 Transit Provider AS2002 100.100.97.0/24 100.100.96.0/20 100.100.96.0/20 0.0.0.0/0 Transit Provider AS4003 100.100.97.0/24 CDN AS20940 Routing Table 100.100.96.0/20 AS2002 0.0.0.0/0 AS4003 Transit Provider AS3003 ACL hostname AS2002-R1 ! interface TenGigabitEthernet1/1 ip access-group 101 out ! access-list 101 deny ip any 100.100.97.0 0.0.0.255 access-list 101 permit ip any any Unintended Result Peering IX
©2012 AKAMAI | FASTER FORWARDTM Issues •  Don’t assume a full-table on any device on the internet •  If you do announce a prefix others expect you to be able to serve traffic to all of it •  Don’t allow customers to use your PA space as PI •  Filtering traffic results in: •  short term traffic blackholing! •  long term widthdrawal of traffic resulting in revenue loss
©2012 AKAMAI | FASTER FORWARDTM •  Deaggregate the /20 if you can’t/won’t serve all of it •  Only announce address space you will serve traffic to •  Get a local embedded cluster ISP A AS1001 CDN AS20940 Transit Provider AS2002 100.100.97.0/24 100.100.96.0/24 0.0.0.0/0 Transit Provider AS4003 100.100.97.0/24 CDN AS20940 Routing Table 100.100.96.0/24 AS2002 100.100.98.0/23 AS2002 100.100.100.0/22 AS2002 100.100.104.0/21 AS2002 0.0.0.0/0 AS4003 Transit Provider AS3003 100.100.98.0/23 100.100.100.0/22 100.100.104.0/21 Better solutions IX Peering
Scenario 3: Split Route Announcement
©2012 AKAMAI | FASTER FORWARDTM •  ISP A is multi-homed to Transit Providers AS2002 and AS3003 •  Transit Provider AS2002 peers with CDN •  Transit Provider AS3003 does not peer with CDN •  ISP A announces different prefix to different ISP •  ISP A can access full internet ISP A AS1001 CDN AS20940 Transit Provider AS2002 Transit Provider AS3003 100.100.96.0/20 100.100.96.0/22 100.100.100.0/22 0.0.0.0/0 Transit Provider AS4003 CDN AS20940 Routing Table 100.100.96.0/22 AS2002 AS1001 100.100.100.0/22 AS2002 AS1001 0.0.0.0/0 AS4003 Split announcement IX
©2012 AKAMAI | FASTER FORWARDTM •  End Users are using IP Addresses 100.100.96.0/22, 100.100.100.0/22, 100.100.104.0/22, 100.100.108.0/22 •  End Users are using ISP A NS 100.100.100.100 •  CDN receives the NS Prefix 100.100.100.0/22 from AS2002 and maps the traffic for ISP A to this cluster •  100.100.96.0/22 100.100.100.0/22 traffic is routed via AS2002 while 100.100.104.0/22 100.100.108.0/22 traffic falls back to default route via AS4003, AS3003 ISP A AS1001 CDN AS20940 Transit Provider AS2002 Transit Provider AS3003 100.100.96.0/20 0.0.0.0/0 Transit Provider AS4003 ISP A AS1001 End User IP: 100.100.96.0/24 End User IP: 100.100.108.0/24 DNS: 100.100.100.100 100.100.96.0/22 100.100.100.0/22 Split announcement IX
©2012 AKAMAI | FASTER FORWARDTM •  This can work perfectly fine •  But the path via the transit providers AS4003 & AS3003 might not be as good as the direct peering, 100.100.100.108.0/22 end users could have significantly worse performance •  What will ISP A do if the user complain? Differing performance
©2012 AKAMAI | FASTER FORWARDTM •  ISP A swaps the route announcements •  Both 100.100.96.0/22 and 100.100.108.0/22 are routed via AS2002 and end-users have the same performance •  The end-user is happy and closes the ticket ISP A AS1001 CDN AS20940 Transit Provider AS2002 Transit Provider AS3003 100.100.96.0/20 0.0.0.0/0 Transit Provider AS4003 ISP A AS1001 End User IP: 100.100.96.0/24 End User IP: 100.100.108.0/24 DNS: 100.100.100.100 100.100.96.0/22 100.100.108.0/22 Problem solved… IX
©2012 AKAMAI | FASTER FORWARDTM 24hrs later: •  CDN no longer receives NS prefix 100.100.100.0/22 from AS2002 •  CDN maps the traffic of ISP A to Cluster B (where they see AS3003’s prefixes) instead of Cluster A (which only peers with AS2002) •  ISP A will receive the traffic from a completely different source potentially all via AS3003 now negating all the TE efforts DO NOT split nameserver and end-user prefixes when traffic engineering …but
Scenario 4: (Attempted) Content Filtering
©2012 AKAMAI | FASTER FORWARDTM ISPs do receive requests from Government organizations to filter specific content The default action is often to block a specific source IP If this content is hosted by a CDN this will not do what you expect! Content filtering what it WILL DO: what it will NOT: blackhole random content to your end-users filter any specific content get your cluster suspended because of observed loss and all traffic served from upstream
©2012 AKAMAI | FASTER FORWARDTM •  Standard BGP traffic engineering will not have the expected results •  Changes in announcements may have a delayed effect •  Where mapping is based on NS, splitting nameserver and end-user prefixes over different providers will have unexpected effects •  Not all clusters have a full table •  splitting more specific announcements over different links can cause unintended behavior •  Announcing prefixes with holes results in blackholing traffic •  Talk to your CDN partners for finetuning traffic •  DO NOT filter traffic by IP Summary
©2012 AKAMAI | FASTER FORWARDTM Matt Jansen mj@akamai.com as20940.peeringdb.com Questions?

More Related Content

PPT
Direccionamiento de red IPv4, subnetting
byfillescas
 
PPTX
Comparison of existing cni plugins for kubernetes
byAdam Hamsik
 
PDF
[KubeCon EU 2022] Running containerd and k3s on macOS
byAkihiro Suda
 
PDF
LF_DPDK17_Serverless DPDK - How SmartNIC resident DPDK Accelerates Packet Pro...
byLF_DPDK
 
PDF
Kubernetes Networking
byCJ Cullen
 
PPTX
Tutorial: Using GoBGP as an IXP connecting router
byShu Sugimoto
 
ODP
Introduction to Nginx
byKnoldus Inc.
 
PDF
MikroTik Firewall : Securing your Router with Port Knocking
byAkbar Azwir, MM, PMP, PMI-SP, PSM I, CISSP
 
Direccionamiento de red IPv4, subnetting
byfillescas
 
Comparison of existing cni plugins for kubernetes
byAdam Hamsik
 
[KubeCon EU 2022] Running containerd and k3s on macOS
byAkihiro Suda
 
LF_DPDK17_Serverless DPDK - How SmartNIC resident DPDK Accelerates Packet Pro...
byLF_DPDK
 
Kubernetes Networking
byCJ Cullen
 
Tutorial: Using GoBGP as an IXP connecting router
byShu Sugimoto
 
Introduction to Nginx
byKnoldus Inc.
 
MikroTik Firewall : Securing your Router with Port Knocking
byAkbar Azwir, MM, PMP, PMI-SP, PSM I, CISSP
 

What's hot

PDF
What's Coming in CloudStack 4.19
byShapeBlue
 
ODP
Kubernetes Architecture
byKnoldus Inc.
 
PPTX
Introducing NGINX App Protect (Japanese Webinar)
byNGINX, Inc.
 
PPTX
Software defined networking(sdn) vahid sadri
byVahid Sadri
 
PPTX
Scaleway Approach to VXLAN EVPN Fabric
byScaleway
 
PDF
Deploying CloudStack and Ceph with flexible VXLAN and BGP networking
byShapeBlue
 
PDF
Introduction to Kong API Gateway
byYohann Ciurlik
 
PDF
Using eBPF for High-Performance Networking in Cilium
byScyllaDB
 
PDF
Building Microservices with gRPC and NATS
byShiju Varghese
 
PPTX
Vxlan control plane and routing
byWilfredzeng
 
PPTX
Chapter 16 : inter-vlan routing
byteknetir
 
PDF
Introduzione a Docker (Maggio 2017) [ITA]
byValerio Radice
 
PDF
GCP Best Practices for SRE Team
byHuynh Thai Bao
 
PDF
CCNP Security-VPN
bymohannadalhanahnah
 
PPTX
OVN - Basics and deep dive
byTrinath Somanchi
 
PDF
macvlan and ipvlan
bySuraj Deshmukh
 
PDF
How VXLAN works on Linux
byEtsuji Nakai
 
PDF
Keycloak Single Sign-On
byRavi Yasas
 
PDF
Mikrotik the dude
byAchmad Mardiansyah
 
PDF
Mikrotik advanced
byguest8423a64e
 
What's Coming in CloudStack 4.19
byShapeBlue
 
Kubernetes Architecture
byKnoldus Inc.
 
Introducing NGINX App Protect (Japanese Webinar)
byNGINX, Inc.
 
Software defined networking(sdn) vahid sadri
byVahid Sadri
 
Scaleway Approach to VXLAN EVPN Fabric
byScaleway
 
Deploying CloudStack and Ceph with flexible VXLAN and BGP networking
byShapeBlue
 
Introduction to Kong API Gateway
byYohann Ciurlik
 
Using eBPF for High-Performance Networking in Cilium
byScyllaDB
 
Building Microservices with gRPC and NATS
byShiju Varghese
 
Vxlan control plane and routing
byWilfredzeng
 
Chapter 16 : inter-vlan routing
byteknetir
 
Introduzione a Docker (Maggio 2017) [ITA]
byValerio Radice
 
GCP Best Practices for SRE Team
byHuynh Thai Bao
 
CCNP Security-VPN
bymohannadalhanahnah
 
OVN - Basics and deep dive
byTrinath Somanchi
 
macvlan and ipvlan
bySuraj Deshmukh
 
How VXLAN works on Linux
byEtsuji Nakai
 
Keycloak Single Sign-On
byRavi Yasas
 
Mikrotik the dude
byAchmad Mardiansyah
 
Mikrotik advanced
byguest8423a64e
 

Viewers also liked

PDF
Inter-AS MPLS VPN Deployment
byBangladesh Network Operators Group
 
PDF
Resource Public Key Infrastructure (RPKI)
byBangladesh Network Operators Group
 
PDF
Community Tools to Fight Against DDoS
byBangladesh Network Operators Group
 
PDF
Dot BD Domain and Shared Registry Model- A Policy Proposal
byBangladesh Network Operators Group
 
PDF
Securing Asterisk: A practical approach
byBangladesh Network Operators Group
 
PDF
OpenStack Cloud Administration Through Live Demonstration
byBangladesh Network Operators Group
 
PDF
Converged & Efficient Licensing Framework
byBangladesh Network Operators Group
 
PDF
IPv6 Address & Deployment Planning
byBangladesh Network Operators Group
 
PDF
ICANN Engagement Update
byBangladesh Network Operators Group
 
PDF
APNIC Policy Update
byBangladesh Network Operators Group
 
PDF
bdNOG Conference Report
byBangladesh Network Operators Group
 
PDF
Broadband for Digital Bangladesh & recommendation from ISPAB
byBangladesh Network Operators Group
 
PDF
ISOC Engagement Activities
byBangladesh Network Operators Group
 
PDF
Shikkhok.com, An ISIF awarded project
byBangladesh Network Operators Group
 
PDF
Awareness of Children Internet Addiction
byBangladesh Network Operators Group
 
PDF
EDNS0 Client-Subnet for DNS Based CDNs
byBangladesh Network Operators Group
 
PDF
Challenges of L2 NID Based Architecture for vCPE and NFV Deployment
byBangladesh Network Operators Group
 
PDF
APNIC42 Announcement
byBangladesh Network Operators Group
 
PDF
bdCERT Activities Update
byBangladesh Network Operators Group
 
PDF
Best Current Operational Practice (BCOP) - Updates from around the world
byBangladesh Network Operators Group
 
Inter-AS MPLS VPN Deployment
byBangladesh Network Operators Group
 
Resource Public Key Infrastructure (RPKI)
byBangladesh Network Operators Group
 
Community Tools to Fight Against DDoS
byBangladesh Network Operators Group
 
Dot BD Domain and Shared Registry Model- A Policy Proposal
byBangladesh Network Operators Group
 
Securing Asterisk: A practical approach
byBangladesh Network Operators Group
 
OpenStack Cloud Administration Through Live Demonstration
byBangladesh Network Operators Group
 
Converged & Efficient Licensing Framework
byBangladesh Network Operators Group
 
IPv6 Address & Deployment Planning
byBangladesh Network Operators Group
 
ICANN Engagement Update
byBangladesh Network Operators Group
 
APNIC Policy Update
byBangladesh Network Operators Group
 
bdNOG Conference Report
byBangladesh Network Operators Group
 
Broadband for Digital Bangladesh & recommendation from ISPAB
byBangladesh Network Operators Group
 
ISOC Engagement Activities
byBangladesh Network Operators Group
 
Shikkhok.com, An ISIF awarded project
byBangladesh Network Operators Group
 
Awareness of Children Internet Addiction
byBangladesh Network Operators Group
 
EDNS0 Client-Subnet for DNS Based CDNs
byBangladesh Network Operators Group
 
Challenges of L2 NID Based Architecture for vCPE and NFV Deployment
byBangladesh Network Operators Group
 
APNIC42 Announcement
byBangladesh Network Operators Group
 
bdCERT Activities Update
byBangladesh Network Operators Group
 
Best Current Operational Practice (BCOP) - Updates from around the world
byBangladesh Network Operators Group
 

Similar to Traffic Engineering for CDNs

PPTX
BGP and Traffic Engineering with Akamai
byInternet Society
 
PDF
EDNS0 Client-Subnet for DNS based CDNs by Matt Jansen
byMyNOG
 
PDF
16 (IDNOG01) EDNS0 / How CDNS works by Matt Jansen
byIndonesia Network Operators Group
 
PDF
Akamai company profile
byrahulp9999
 
PPT
CONTENT DELIVERY NETWORK
bySaif Muttair
 
PDF
Embedded CDNs in 2023
byMyNOG
 
PDF
PLNOG 6: Christian Kaufmann - How Akamai delivers your packets - the insight
byPROIDEA
 
PPT
Cdn
byAravindharamanan S
 
PPT
Unit VI Overlays
bysangusajjan
 
PDF
Misc: The Internet Story - How Data Travels, Transit Works, and the Role of C...
by3G4G
 
PPTX
Content Delivery Network
byFaisal Akbar
 
PDF
Traffic Engineering for CDNs
byMyNOG
 
PPTX
Traffic Engineering for CDNs by Matt Jansen [APRICOT 2015]
byAPNIC
 
PPTX
40 - IDNOG03 - Bob Lau (Akamai) - BGP and Traffic Engineering
byIndonesia Network Operators Group
 
PDF
PLNOG 6: Christian Kaufmann - How Akamai delivers your packets - the insight
byPROIDEA
 
DOCX
CDN
bySN Chakraborty
 
PPT
Cdn cs6740
byAravindharamanan S
 
PPTX
Contents Localization Initiatives to get better User Experience
byBangladesh Network Operators Group
 
PPT
Lec13 cdn
byAravindharamanan S
 
PPT
Slides cao
byAravindharamanan S
 
BGP and Traffic Engineering with Akamai
byInternet Society
 
EDNS0 Client-Subnet for DNS based CDNs by Matt Jansen
byMyNOG
 
16 (IDNOG01) EDNS0 / How CDNS works by Matt Jansen
byIndonesia Network Operators Group
 
Akamai company profile
byrahulp9999
 
CONTENT DELIVERY NETWORK
bySaif Muttair
 
Embedded CDNs in 2023
byMyNOG
 
PLNOG 6: Christian Kaufmann - How Akamai delivers your packets - the insight
byPROIDEA
 
Cdn
byAravindharamanan S
 
Unit VI Overlays
bysangusajjan
 
Misc: The Internet Story - How Data Travels, Transit Works, and the Role of C...
by3G4G
 
Content Delivery Network
byFaisal Akbar
 
Traffic Engineering for CDNs
byMyNOG
 
Traffic Engineering for CDNs by Matt Jansen [APRICOT 2015]
byAPNIC
 
40 - IDNOG03 - Bob Lau (Akamai) - BGP and Traffic Engineering
byIndonesia Network Operators Group
 
PLNOG 6: Christian Kaufmann - How Akamai delivers your packets - the insight
byPROIDEA
 
CDN
bySN Chakraborty
 
Cdn cs6740
byAravindharamanan S
 
Contents Localization Initiatives to get better User Experience
byBangladesh Network Operators Group
 
Lec13 cdn
byAravindharamanan S
 
Slides cao
byAravindharamanan S
 

More from Bangladesh Network Operators Group

PDF
Optimizing DNS Performance in Kubernetes: Challenges and Best Practices
byBangladesh Network Operators Group
 
PPTX
Understanding Universal Acceptance (UA) and Technical Challenges
byBangladesh Network Operators Group
 
PDF
Taming the Tide: Autoscaling Kubernetes for Cost-Effective Load Balancing
byBangladesh Network Operators Group
 
PDF
DNSSEC Implementation Journey at Prime Bank’s Domain
byBangladesh Network Operators Group
 
PDF
IPv6 Mostly Experience at APRICOT by Yoshinobu Matsuzaki (IIJ)
byBangladesh Network Operators Group
 
PDF
DNS & DNSSEC operational best practices - Sleep better at night with KINDNS i...
byBangladesh Network Operators Group
 
PDF
DNS & DNSSEC operational best practices - Sleep better at night with KINDNS i...
byBangladesh Network Operators Group
 
PDF
DDoS Mitigation Strategies by Md. Abdul Awal
byBangladesh Network Operators Group
 
PDF
A Day in the Life of IPv6 Scanning by Matsuzaki ʻmazʼ
byBangladesh Network Operators Group
 
PDF
Network Efficiency:The LLM Advantage on network infrastructures
byBangladesh Network Operators Group
 
PDF
Automating ISP Networks Using Ansible and IPAM as a Source of Truth [SoT]
byBangladesh Network Operators Group
 
PDF
DNSSEC Deployment for .BD SLDs by Abdul Awal
byBangladesh Network Operators Group
 
PDF
Hybrid Mesh Firewall: Network firewall revolution
byBangladesh Network Operators Group
 
PPTX
The Internet Service Providers and Connectivity Providers of ICANN
byBangladesh Network Operators Group
 
PDF
OFFENSIVE OPERATIONS : THE ANATOMY OF A NETWORK TAKEOVER
byBangladesh Network Operators Group
 
PDF
Optics101 for non-Optical (IP) folks by Tashi Phuntsho
byBangladesh Network Operators Group
 
PDF
Fast Reroute in SR-MPLS by Md Abdullah Al Naser
byBangladesh Network Operators Group
 
PDF
Mental Health and Workplace Culture in Tech:A Personal Perspective
byBangladesh Network Operators Group
 
PPTX
Integration of AI and GenAI in Education and beyond
byBangladesh Network Operators Group
 
PPTX
Strengthening Cyber Security with Tools and Human Expertise
byBangladesh Network Operators Group
 
Optimizing DNS Performance in Kubernetes: Challenges and Best Practices
byBangladesh Network Operators Group
 
Understanding Universal Acceptance (UA) and Technical Challenges
byBangladesh Network Operators Group
 
Taming the Tide: Autoscaling Kubernetes for Cost-Effective Load Balancing
byBangladesh Network Operators Group
 
DNSSEC Implementation Journey at Prime Bank’s Domain
byBangladesh Network Operators Group
 
IPv6 Mostly Experience at APRICOT by Yoshinobu Matsuzaki (IIJ)
byBangladesh Network Operators Group
 
DNS & DNSSEC operational best practices - Sleep better at night with KINDNS i...
byBangladesh Network Operators Group
 
DNS & DNSSEC operational best practices - Sleep better at night with KINDNS i...
byBangladesh Network Operators Group
 
DDoS Mitigation Strategies by Md. Abdul Awal
byBangladesh Network Operators Group
 
A Day in the Life of IPv6 Scanning by Matsuzaki ʻmazʼ
byBangladesh Network Operators Group
 
Network Efficiency:The LLM Advantage on network infrastructures
byBangladesh Network Operators Group
 
Automating ISP Networks Using Ansible and IPAM as a Source of Truth [SoT]
byBangladesh Network Operators Group
 
DNSSEC Deployment for .BD SLDs by Abdul Awal
byBangladesh Network Operators Group
 
Hybrid Mesh Firewall: Network firewall revolution
byBangladesh Network Operators Group
 
The Internet Service Providers and Connectivity Providers of ICANN
byBangladesh Network Operators Group
 
OFFENSIVE OPERATIONS : THE ANATOMY OF A NETWORK TAKEOVER
byBangladesh Network Operators Group
 
Optics101 for non-Optical (IP) folks by Tashi Phuntsho
byBangladesh Network Operators Group
 
Fast Reroute in SR-MPLS by Md Abdullah Al Naser
byBangladesh Network Operators Group
 
Mental Health and Workplace Culture in Tech:A Personal Perspective
byBangladesh Network Operators Group
 
Integration of AI and GenAI in Education and beyond
byBangladesh Network Operators Group
 
Strengthening Cyber Security with Tools and Human Expertise
byBangladesh Network Operators Group
 

Recently uploaded

PPTX
Role of VMware and Cloud Computing in CyberSecurity.pptx
byalehanoor1325
 
PPTX
Computer Networks 01[1 using all terms].pptx
bywaqasahmedbhatti633
 
PPTX
SriLank SLT LTE Network Sharing V2.pptx
bynthuang
 
PDF
Micro project .pdf drone technology Report
byRenitaMamgain
 
PPT
Memory Organization In Assembly Language
byumairmuhammad0409
 
PDF
Hire-NET-Developers. Dot Net Developers
byDigisoft Solution
 
PDF
Here are the winners of KALIDASA SAMMAN.
byglcppro
 
PDF
Xnspy vs FlexiSPY: 2025 Monitoring Comparison
byMichael Carter
 
PPTX
Bibhav Singh.pptx it's very useful for us it's in very good and very nice
byabhishekjaiswal5th33
 
PPTX
Ideathon template.pptx it is a ideathon template
bykonav71133
 
PPTX
Abhishek Jaiswal.ppt.pptx for you it is very useful for me and I am sharing w...
byabhishekjaiswal5th33
 
PDF
tokiohotellistening.pdf for tokiohotelfans
byrafaelgombos1234
 
PPTX
ChatGPT on education positively snd negatively
by88rtgpscxx
 
PDF
WShell Gameplay and Games of Games of Games
byzramirezmtz9
 
PDF
WShell Gameplay and Games of Games of Games
byzramirezmtz9
 
PDF
ARTIFICIAL INTELLIGENCE DEVELOPMENT SERVICE
bydavidjohansen1597
 
PPT
html-forms in web development-courseICT.
bymadz33
 
PPT
Lecture_3 Network Securityyyeyeyyeeyyeywy.ppt
bysawsan202
 
PPTX
COĞRAFYA ödevi 70 sayfa güzeldir hayırlı olsun
bydtilter
 
PPTX
PRITHVI theater, A true artist. Newzfeaturez.pptx
bysc05digital
 
Role of VMware and Cloud Computing in CyberSecurity.pptx
byalehanoor1325
 
Computer Networks 01[1 using all terms].pptx
bywaqasahmedbhatti633
 
SriLank SLT LTE Network Sharing V2.pptx
bynthuang
 
Micro project .pdf drone technology Report
byRenitaMamgain
 
Memory Organization In Assembly Language
byumairmuhammad0409
 
Hire-NET-Developers. Dot Net Developers
byDigisoft Solution
 
Here are the winners of KALIDASA SAMMAN.
byglcppro
 
Xnspy vs FlexiSPY: 2025 Monitoring Comparison
byMichael Carter
 
Bibhav Singh.pptx it's very useful for us it's in very good and very nice
byabhishekjaiswal5th33
 
Ideathon template.pptx it is a ideathon template
bykonav71133
 
Abhishek Jaiswal.ppt.pptx for you it is very useful for me and I am sharing w...
byabhishekjaiswal5th33
 
tokiohotellistening.pdf for tokiohotelfans
byrafaelgombos1234
 
ChatGPT on education positively snd negatively
by88rtgpscxx
 
WShell Gameplay and Games of Games of Games
byzramirezmtz9
 
WShell Gameplay and Games of Games of Games
byzramirezmtz9
 
ARTIFICIAL INTELLIGENCE DEVELOPMENT SERVICE
bydavidjohansen1597
 
html-forms in web development-courseICT.
bymadz33
 
Lecture_3 Network Securityyyeyeyyeeyyeywy.ppt
bysawsan202
 
COĞRAFYA ödevi 70 sayfa güzeldir hayırlı olsun
bydtilter
 
PRITHVI theater, A true artist. Newzfeaturez.pptx
bysc05digital
 

Traffic Engineering for CDNs

  • 1.
    Traffic Engineering forCDNs Matt Jansen Akamai Technologies BDNOG3
  • 2.
    ©2012 AKAMAI |FASTER FORWARDTM The world’s largest on-demand, distributed computing platform delivers all forms of web content and applications The Akamai Intelligent Platform Typical daily traffic: •  More than 2 trillion requests served •  Delivering over 25 Terabits/second •  15-30% of all daily web traffic The Akamai Intelligent Platform: 170,000+ Servers 2,000+ Locations 102+ Countries 1,300+ Networks 700+ Cities
  • 3.
    ©2012 AKAMAI |FASTER FORWARDTM When content is requested from CDNs, the user is directed to the optimal server to serve this user There’s 2 common ways to do that: •  anycast: the content is served from the location the request is received (easy to build, requires symmetric routing to work well) •  DNS based: the CDN decides where to best serve the content from based on the resolver it receives the request from, and replies with the optimal server How CDNs Work
  • 4.
    ©2012 AKAMAI |FASTER FORWARDTM Users querying a DNS-based CDNs will be returned different A (and AAAA) records for the same hostname depending on the resolver the request comes from This is called “mapping” The better the mapping, the better the CDN How DNS based CDNs Work
  • 5.
    ©2012 AKAMAI |FASTER FORWARDTM Example of Akamai mapping • Notice the different A records for different locations: [NYC]% host www.symantec.com www.symantec.com CNAME e5211.b.akamaiedge.net. e5211.b.akamaiedge.net. A 207.40.194.46 e5211.b.akamaiedge.net. A 207.40.194.49 [Boston]% host www.symantec.com www.symantec.com CNAME e5211.b.akamaiedge.net. e5211.b.akamaiedge.net. A 81.23.243.152 e5211.b.akamaiedge.net. A 81.23.243.145 DNS based Mapping Example
  • 6.
    ©2012 AKAMAI |FASTER FORWARDTM CDNs use multiple criteria to choose the optimal server • These include standard network metrics: • Latency • Throughput • Packet loss • as well as internal ones such as: • CPU load on the server • HD space • network utilization How DNS based CDNs Work
  • 7.
    ©2012 AKAMAI |FASTER FORWARDTM DNS based Mapping 1)  end-user requests www.example.com from ISP NS 2)  ISP NS recursively (multiple iterations) looks up www.example.com being referred to authoritative CDN NS (by CNAME) 3)  ISP NS asks authoritative CDN NS 4)  CDN NS looks up IP of requestor (ISP NS) and replies with IP of optimal cluster to serve content (closest cluster for that ISP) 5)  ISP NS replies to end-user who 6)  requests content from local Cluster end-user ISP NS 1.2.3.4 root/tld/intermediate NS (recursive lookup until reaching authoritative NS) CDN NS Closest cluster 5.6.7.8 example.com? example.com? a212.g.akamai.net 5.6.7.8 1 3 6 2 NS 1.2.3.4? best cluster = 5.6.7.8 4 5
  • 8.
    ©2012 AKAMAI |FASTER FORWARDTM Types of CDN Deployments (simplified) end-user ISP Internet IX Transit Embedded cluster in ISP Colocated Cluster with IX Infrastructure Cluster with Transit Backend Mapping system
  • 9.
    ©2012 AKAMAI |FASTER FORWARDTM DNS based CDNs don’t necessarily have a backbone Those clusters can be standalone ‘Islands’ with no connectivity between them! DNS based CDNs usually don’t announce large blocks of address space as this will only be the local servers •  it is not uncommon to see a single /24 from Akamai at an IX •  you will receive a different set of prefixes on each peering This does not mean you will not see a lot of traffic •  how many servers do you need to serve 10g nowadays? Typical DNS based Topology
  • 10.
    ©2012 AKAMAI |FASTER FORWARDTM •  no single cluster can accommodate all content •  caches get more efficient with with size •  some content requires specialized servers only present in Infrastructure clusters •  some content is only present in specific geographies •  CDNs might prefer on-net cluster over peering Why don’t I get all CDN Traffic locally?
  • 11.
    ©2012 AKAMAI |FASTER FORWARDTM Performance •  getting as close as possible to the end-user (removing intermediate ASNs) decreasing latency/increasing troughput Burstability •  for large event peaks direct connectivity to multiple networks allows for higher burstability than a single connection to a transit provider Redundancy •  Serve as overflow and backup for embedded on-net clusters Scale •  Large deployment at an IX can serve traffic that does not ‘fit’ into smaller embedded clusters Why CDNs peer
  • 12.
    ©2012 AKAMAI |FASTER FORWARDTM Performance •  ISP’s end-users benefit from direct connectivity Competitive Advantages •  improving performance over competitors •  additional revenue from downstreams Cost Reduction •  Save on transit bill and potential backbone costs Redundancy •  Serve as overflow and backup for embedded on-net clusters Why ISPs peer with CDNs
  • 13.
    ©2012 AKAMAI |FASTER FORWARDTM Transit Peer •  Akamai does not have a backbone, each IX instance is independent •  Cluster uses transit to fetch content origin •  Content is served to peers over the IX •  BGP session serves 2 purposes: •  Route traffic strictly within the local instance •  Tell our system which prefixes this cluster is allowed to serve •  New prefixes being picked up by the system can take up to 24hrs Origin Server IX Content Typical large IX deployment Origin
  • 14.
  • 15.
    ©2012 AKAMAI |FASTER FORWARDTM Attempting to shift traffic to another path ISP peers with CDN over 2 IXs •  wants to shift traffic from A to B due to reduce backbone traffic •  Typical BGP based techniques: 1)  AS-Path prepending 2)  MEDs 3)  more/less specific announcements None of the above have the desired effect! •  no network between clusters, any BGP parameters are only locally relevant •  Might have undesired effect of shifting traffic to a upstream/competitor on the same IX end-user ISP A IX A IX B
  • 16.
    ©2012 AKAMAI |FASTER FORWARDTM Problem solved… end-user IX B ISP withdraws some of their prefixes over IX A •  Traffic falls back to transit on the same cluster (provided that still has the best performance) •  this might result in them receiving the traffic in another location in their network so they’re happy Transit ISP A IX A
  • 17.
    ©2012 AKAMAI |FASTER FORWARDTM …but not for long within 24hrs •  The CDN backend system processes the fact that they don’t receive their prefixes at the IX A cluster •  Traffic switches to another cluster where they do receive them (this might be one of their upstreams they have a PNI with in the same city) •  Traffic comes back into their network again at the same router! Upstream of ISP A end-user Transit ISP A IX A
  • 18.
    ©2012 AKAMAI |FASTER FORWARDTM Issues •  BGP parameters only locally relevant •  Delayed effect of BGP announcements on Mapping System •  CDNs typically see your prefixes in many different locations
  • 19.
    ©2012 AKAMAI |FASTER FORWARDTM Better solution •  Talk to the CDN if you have issues with their traffic in a specific location •  You can work together to achieve the result you’re looking for •  Get a local embedded cluster
  • 20.
    Scenario 2: morespecific Route Announcement
  • 21.
    ©2012 AKAMAI |FASTER FORWARDTM •  ISP A is multi-homed to Transit Providers AS2002 and AS3003 •  Transit Provider AS2002 peers with CDN •  Transit Provider AS3003 does not peer with CDN •  CDN sends traffic to ISP A via Transit Provider AS2002 ISP A AS1001 CDN AS20940 Transit Provider AS2002 Transit Provider AS3003 100.100.96.0/20 100.100.96.0/20 0.0.0.0/0 Transit Provider AS4003 Consistent Announcements IX
  • 22.
    ©2012 AKAMAI |FASTER FORWARDTM •  ISP A would like to balance traffic between the two upstream providers •  ISP A prepends, then applies MED to Transit Provider AS2002. This has no effect on CDN traffic. •  Eventually, ISP A de-aggregates the /20 and advertises more specific routes to Transit Provider AS3003 •  What will happen? Loadbalancing
  • 23.
    ©2012 AKAMAI |FASTER FORWARDTM •  ISP A announces more specific routes to Transit Provider AS3003 •  Transit Provider AS3003 announces new /24 to AS2002 •  CDN IX router does not have a full-table, so traffic continue route to the /20 of AS2002 •  ISP A is happy with the balanced traffic on dual Transit Providers ISP A AS1001 CDN AS20940 Transit Provider AS2002 Transit Provider AS3003 100.100.96.0/20 AS2002 Routing Table 100.100.100.0/24 AS3003 AS1001 100.100.99.0/24 AS3003 AS1001 100.100.96.0/20 AS1001 CDN AS20940 Routing Table 100.100.96.0/20 AS2002 AS1001 0.0.0.0/0 AS4003 100.100.96.0/20 0.0.0.0/0 Transit Provider AS4003 Loadbalancing works... IX Peering
  • 24.
    ©2012 AKAMAI |FASTER FORWARDTM •  Lost of revenue for Transit Provider AS2002 even though their peering/backbone is utilized •  What happens if AS2002 does not like the traffic from one peer to the other? …but
  • 25.
    ©2012 AKAMAI |FASTER FORWARDTM Transit provider filters traffic •  In order to get rid of traffic between peers, Transit Provider AS2002 implements an ACL on IX port facing AS3003 •  Traffic gets blackholed, ISP A’s eyeballs don’t receive traffic anymore! ISP A AS1001 CDN AS20940 Transit Provider AS2002 Transit Provider AS3003 100.100.96.0/20 100.100.96.0/20 ACL 0.0.0.0/0 Transit Provider AS4003 hostname AS2002-R1 ! interface TenGigabitEthernet1/1 ip access-group 101 out ! access-list 101 deny ip any 100.100.100.0 0.0.0.255 access-list 101 deny ip any 100.100.99.0 0.0.0.255 access-list 101 permit ip any any IX Peering
  • 26.
    ©2012 AKAMAI |FASTER FORWARDTM •  CDN observes ISP A end-users are unable to access some websites •  CDN stops serving unreliable prefixes received from Transit Provider AS2002, traffic shifts from IX to Transit Provider AS4003 •  ISP A can access all websites happily •  Transit Provider AS2002 loses revenue ISP A AS1001 CDN AS20940 Transit Provider AS2002 Transit Provider AS3003 100.100.96.0/20 100.100.96.0/20 ACL 0.0.0.0/0 Transit Provider AS4003 Unintended Result IX Peering
  • 27.
    ©2012 AKAMAI |FASTER FORWARDTM Issues •  Don’t assume a full-table on any device on the internet •  Filtering traffic results in: •  short term traffic blackholing! •  long term widthdrawal of traffic resulting in revenue loss
  • 28.
    ©2012 AKAMAI |FASTER FORWARDTM •  AS2002 filters the specific prefixes instead of the actual traffic •  work with upstreams and/or CDN for finetuning •  Get Transit Provider AS3003 to peer with CDN directly ;) •  Get a local embedded cluster ISP A AS1001 CDN AS20940 Transit Provider AS2002 Transit Provider AS3003 100.100.96.0/20 100.100.96.0/20 Filter Specific route 100.100.99.0/24 100.100.100.0/24 0.0.0.0/0 Transit Provider AS4003 neighbor PEER-GROUP prefix-list DENY-SPECIFIC in ! ip prefix-list DENY-SPECIFIC seq 5 deny 100.100.100.0/24 ip prefix-list DENY-SPECIFIC seq 10 deny 100.100.99.0/24 ip prefix-list DENY-SPECIFIC seq 100 permit 0.0.0.0/0 le 32 Better solutions IX Peering
  • 29.
    ©2012 AKAMAI |FASTER FORWARDTM •  ISP A is single homed to Transit Provider AS2002 •  ISP A obtains a /24 from Transit Provider AS2002’s address space •  all works well ISP A AS1001 CDN AS20940 Transit Provider AS2002 100.100.97.0/24100.100.96.0/20 100.100.96.0/20 0.0.0.0/0 Transit Provider AS4003 100.100.97.0/24 CDN AS20940 Routing Table 100.100.96.0/20 AS2002 100.100.97.0/24 AS2002 AS1001 0.0.0.0/0 AS4003 100.100.97.0/24 Another variation of this Scenario IX
  • 30.
    ©2012 AKAMAI |FASTER FORWARDTM •  ISP A moves to new Transit Provider AS3003, but keeps using his previously assigned prefix 100.100.96.0/24 •  CDN keeps serving traffic to ISP A via Transit Provider AS2002 due to the /20 being received there ISP A AS1001 CDN AS20940 Transit Provider AS2002 100.100.97.0/24 100.100.96.0/20 100.100.96.0/20 0.0.0.0/0 Transit Provider AS4003 100.100.97.0/24 CDN AS20940 Routing Table 100.100.96.0/20 AS2002 0.0.0.0/0 AS4003 Transit Provider AS3003 Provider Change Peering IX
  • 31.
    ©2012 AKAMAI |FASTER FORWARDTM •  Lost of revenue for Transit Provider AS2002 even though their peering/backbone is utilized •  What happens if AS2002 does not like the traffic from one peer to the other? …but
  • 32.
    ©2012 AKAMAI |FASTER FORWARDTM •  In order to get rid of traffic between peers, Transit Provider AS2002 implements an ACL on IX port facing AS3003 •  Traffic gets blackholed, ISP A’s eyeballs don’t receive traffic anymore! ISP A AS1001 CDN AS20940 Transit Provider AS2002 100.100.97.0/24 100.100.96.0/20 100.100.96.0/20 0.0.0.0/0 Transit Provider AS4003 100.100.97.0/24 CDN AS20940 Routing Table 100.100.96.0/20 AS2002 0.0.0.0/0 AS4003 Transit Provider AS3003 ACL hostname AS2002-R1 ! interface TenGigabitEthernet1/1 ip access-group 101 out ! access-list 101 deny ip any 100.100.97.0 0.0.0.255 access-list 101 permit ip any any Transit provider filters traffic IX Peering
  • 33.
    ©2012 AKAMAI |FASTER FORWARDTM •  CDN observes ISP A end-users are unable to access some websites •  CDN stops serving unreliable prefixes received from Transit Provider AS2002, traffic shifts from IX to Transit Provider AS4003 •  ISP A can access all websites happily •  Transit Provider AS2002 loses revenue ISP A AS1001 CDN AS20940 Transit Provider AS2002 100.100.97.0/24 100.100.96.0/20 100.100.96.0/20 0.0.0.0/0 Transit Provider AS4003 100.100.97.0/24 CDN AS20940 Routing Table 100.100.96.0/20 AS2002 0.0.0.0/0 AS4003 Transit Provider AS3003 ACL hostname AS2002-R1 ! interface TenGigabitEthernet1/1 ip access-group 101 out ! access-list 101 deny ip any 100.100.97.0 0.0.0.255 access-list 101 permit ip any any Unintended Result Peering IX
  • 34.
    ©2012 AKAMAI |FASTER FORWARDTM Issues •  Don’t assume a full-table on any device on the internet •  If you do announce a prefix others expect you to be able to serve traffic to all of it •  Don’t allow customers to use your PA space as PI •  Filtering traffic results in: •  short term traffic blackholing! •  long term widthdrawal of traffic resulting in revenue loss
  • 35.
    ©2012 AKAMAI |FASTER FORWARDTM •  Deaggregate the /20 if you can’t/won’t serve all of it •  Only announce address space you will serve traffic to •  Get a local embedded cluster ISP A AS1001 CDN AS20940 Transit Provider AS2002 100.100.97.0/24 100.100.96.0/24 0.0.0.0/0 Transit Provider AS4003 100.100.97.0/24 CDN AS20940 Routing Table 100.100.96.0/24 AS2002 100.100.98.0/23 AS2002 100.100.100.0/22 AS2002 100.100.104.0/21 AS2002 0.0.0.0/0 AS4003 Transit Provider AS3003 100.100.98.0/23 100.100.100.0/22 100.100.104.0/21 Better solutions IX Peering
  • 36.
    Scenario 3: SplitRoute Announcement
  • 37.
    ©2012 AKAMAI |FASTER FORWARDTM •  ISP A is multi-homed to Transit Providers AS2002 and AS3003 •  Transit Provider AS2002 peers with CDN •  Transit Provider AS3003 does not peer with CDN •  ISP A announces different prefix to different ISP •  ISP A can access full internet ISP A AS1001 CDN AS20940 Transit Provider AS2002 Transit Provider AS3003 100.100.96.0/20 100.100.96.0/22 100.100.100.0/22 0.0.0.0/0 Transit Provider AS4003 CDN AS20940 Routing Table 100.100.96.0/22 AS2002 AS1001 100.100.100.0/22 AS2002 AS1001 0.0.0.0/0 AS4003 Split announcement IX
  • 38.
    ©2012 AKAMAI |FASTER FORWARDTM •  End Users are using IP Addresses 100.100.96.0/22, 100.100.100.0/22, 100.100.104.0/22, 100.100.108.0/22 •  End Users are using ISP A NS 100.100.100.100 •  CDN receives the NS Prefix 100.100.100.0/22 from AS2002 and maps the traffic for ISP A to this cluster •  100.100.96.0/22 100.100.100.0/22 traffic is routed via AS2002 while 100.100.104.0/22 100.100.108.0/22 traffic falls back to default route via AS4003, AS3003 ISP A AS1001 CDN AS20940 Transit Provider AS2002 Transit Provider AS3003 100.100.96.0/20 0.0.0.0/0 Transit Provider AS4003 ISP A AS1001 End User IP: 100.100.96.0/24 End User IP: 100.100.108.0/24 DNS: 100.100.100.100 100.100.96.0/22 100.100.100.0/22 Split announcement IX
  • 39.
    ©2012 AKAMAI |FASTER FORWARDTM •  This can work perfectly fine •  But the path via the transit providers AS4003 & AS3003 might not be as good as the direct peering, 100.100.100.108.0/22 end users could have significantly worse performance •  What will ISP A do if the user complain? Differing performance
  • 40.
    ©2012 AKAMAI |FASTER FORWARDTM •  ISP A swaps the route announcements •  Both 100.100.96.0/22 and 100.100.108.0/22 are routed via AS2002 and end-users have the same performance •  The end-user is happy and closes the ticket ISP A AS1001 CDN AS20940 Transit Provider AS2002 Transit Provider AS3003 100.100.96.0/20 0.0.0.0/0 Transit Provider AS4003 ISP A AS1001 End User IP: 100.100.96.0/24 End User IP: 100.100.108.0/24 DNS: 100.100.100.100 100.100.96.0/22 100.100.108.0/22 Problem solved… IX
  • 41.
    ©2012 AKAMAI |FASTER FORWARDTM 24hrs later: •  CDN no longer receives NS prefix 100.100.100.0/22 from AS2002 •  CDN maps the traffic of ISP A to Cluster B (where they see AS3003’s prefixes) instead of Cluster A (which only peers with AS2002) •  ISP A will receive the traffic from a completely different source potentially all via AS3003 now negating all the TE efforts DO NOT split nameserver and end-user prefixes when traffic engineering …but
  • 42.
    Scenario 4: (Attempted)Content Filtering
  • 43.
    ©2012 AKAMAI |FASTER FORWARDTM ISPs do receive requests from Government organizations to filter specific content The default action is often to block a specific source IP If this content is hosted by a CDN this will not do what you expect! Content filtering what it WILL DO: what it will NOT: blackhole random content to your end-users filter any specific content get your cluster suspended because of observed loss and all traffic served from upstream
  • 44.
    ©2012 AKAMAI |FASTER FORWARDTM •  Standard BGP traffic engineering will not have the expected results •  Changes in announcements may have a delayed effect •  Where mapping is based on NS, splitting nameserver and end-user prefixes over different providers will have unexpected effects •  Not all clusters have a full table •  splitting more specific announcements over different links can cause unintended behavior •  Announcing prefixes with holes results in blackholing traffic •  Talk to your CDN partners for finetuning traffic •  DO NOT filter traffic by IP Summary
  • 45.
    ©2012 AKAMAI |FASTER FORWARDTM Matt Jansen mj@akamai.com as20940.peeringdb.com Questions?