• Save
Starting and running an IXP
Upcoming SlideShare
Loading in...5
×
 

Starting and running an IXP

on

  • 838 views

A talk from RIPE NCC Regional Meeting in Dubrovnik, 2011...

A talk from RIPE NCC Regional Meeting in Dubrovnik, 2011
This presentation is a from a second part of Running an IXP Workshop. See
http://www.ripe.net/ripe/meetings/regional-meetings/dubrovnik-2011/workshops-and-tutorials for full programme.

Statistics

Views

Total Views
838
Slideshare-icon Views on SlideShare
838
Embed Views
0

Actions

Likes
1
Downloads
0
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Starting and running an IXP Starting and running an IXP Presentation Transcript

    • Starting and running an IXP All that stuff around the switch Some guidelinesMatjaž Straus Istenič, SIX/ARNES 1
    • Agenda • all that stuff around the switch • practical examples – addressing – configuration examples – guidelines and hints for membersMatjaž Straus Istenič, 8.9.2011 2/417 2
    • Stuff around the switch • proper location with many fibre providers –a building with one single provider is a bad idea • different fibre paths inside of the building • power supplies and grounding • cooling system • physical security • staff, support, remote hands • good and accurate documentationMatjaž Straus Istenič, 8.9.2011 3 3
    • Stuff around the switch (cont.) • monitoring and alarming • ticketing system • mailing lists • web portal • best current practices and knowledge base • contracts, SLAs, billing, ... • planning for a collocation/datacenterMatjaž Straus Istenič, 8.9.2011 4 4
    • The powerMatjaž Straus Istenič, 8.9.2011 5 5
    • The power • allocate up to 20 kW per rack • actual usage 5 kW - 10 kW per rack • dual separate circuit breaker for each rack • power supply redundancy – dual feed from electrical distribution company – separate dual UPS system N+1 and PDU – diesel generator • cooling equipment is independently dual powered, including chillers • how much power does datacenter use – monitoring on UPS, on PDU – monitoring total on main branch circuit • typicaly the load will double in 5 yearsMatjaž Straus Istenič, 8.9.2011 6 6
    • Cooling • full redundancy of cooling system – two different power grids – separate piping – chiller redundancy – room units redundancy • hot/cold isle – reduce air mixing – cold aisle with barriers made of metal, plastic or fiberglass – use blanking panels on the cabinets without servers • no need for double floor – run network cabling over the top of the cabinets – "in row" cooling • recommended temperature in cold isle is between 23 - 25 °C • cooling system rating must be 1.3 x IT load rating • make sure that the space will allow for future growth – for more cooling capacity and redundancy if required • Power usage effectiveness (PUE = Total Facility Power/IT Equipment Power) – typical PUE is 2.0 or higherMatjaž Straus Istenič, 8.9.2011 7 7
    • Matjaž Straus Istenič, 8.9.2011 8 8
    • Fire protection • sensing the smoke/fire type ✔ ✗ • very sensitive • early warning • more expensive aspiration sensor • single point of electrical • plastic air ducting under the instalation ceiling must be installed • targeted sensing is possible • cheaper • less sensitive optical sensor • can be used as confirmation • each sensor needs its own for fast aspiration sensors cableMatjaž Straus Istenič, 8.9.2011 9 9
    • Fire protection • extinguishing fire Gaseous fire extinguishing system All are considered safe for breathing after release, although, products of burning plastics are always dangerous! type active substance ✔ ✗ Inergen • big storage requirements displacement - mixture of gases, displaces • totally natural • high pressure (200 or 300 bar) air with “air” with less • environmentaly • computer room needs big exhaust vents of air neutral • bug rush of gas at release causes dust and oxygen objects to lift • small storage area • has some effect on environment Novec 1230 • stored as fluid • expensive - chemical bonding, cooling • very small greenhouse • stored under pressure (40/50 bar) chemical gas footprint action • small storage area • being phased out FM200 (phasing out) - chemical bonding • small greenhouse gas • has some ozone depletion impact footprint • stored under pressure (40/50 bar) • totally natural • water in computer room is not a good idea ;-) cooling water mist • environmentaly • possible condensation on cold surfaces neutralMatjaž Straus Istenič, 8.9.2011 10 10
    • Examples and guidelines • addressing • port configuration • guidelines for membersMatjaž Straus Istenič, 8.9.2011 11 11
    • Examples: addressing • a single subnet taken from independent address space – member address is assigned per location • address schema at SIX 2001:7f8:46:0:L:N::<AS>/64 L = 0 at location 1 91.220.194.n/24 L = 1 at location 2 n = n1 = 2..99 at location 1 N = 0 for a single router, n = n1 + 100 = 102..199 otherwise N = 1, 2, ... at location 2 AS = member AS in decimal n = 1, 101 for route-reflectors AS = 51988 for RRs - diverse lower 24 bits which form solicited-node mcast addressMatjaž Straus Istenič, 8.9.2011 12 12PI for IPv4, IPv6 Assignment for an Internet Exchange Point (IXP) for IPv6 (see RIPE-460 and RIPE-451/policy).
    • Examples: port configuration • access port on Cisco 4900M interface GigabitEthernet2/24 class-map match-any IPv4_traffic switchport access vlan <N> match protocol ip switchport mode access class-map match-any IPv6_traffic switchport nonegotiate match protocol ipv6 switchport port-security [maximum 2] ! load-interval 30 policy-map COUNTER_IPv4_IPv6 storm-control broadcast level 1.00 class IPv4_traffic storm-control action shutdown police cir 32000 spanning-tree portfast conform-action transmit spanning-tree bpduguard enable exceed-action transmit service-policy input COUNTER_IPv4_IPv6 violate-action transmit service-policy output LIMIT-QUEUE-200 class IPv6_traffic ! police cir 32000 conform-action transmit exceed-action transmit violate-action transmit ! policy-map LIMIT-QUEUE-200 class class-default queue-limit 200 !Matjaž Straus Istenič, 8.9.2011 13 13
    • Examples: port configuration • interconnecting ports – aggregated to EtherChannel with LACP – maximal MTU interface TenGigabitEthernet1/1 interface Port-channel48 switchport access vlan <N> switchport switchport mode access switchport access vlan <N> switchport nonegotiate switchport mode access mtu 9198 switchport nonegotiate load-interval 30 mtu 9198 channel-protocol lacp bandwidth 10000000 channel-group 48 mode active ! ! port-channel load-balance src-dst-ip interface TenGigabitEthernet1/2 switchport access vlan <N> switchport mode access switchport nonegotiate mtu 9198 load-interval 30 channel-protocol lacp channel-group 48 mode active !Matjaž Straus Istenič, 8.9.2011 14 14
    • Guidelines for members • access port configuration • BGP – routing considerations – MD5 authentication – filtering announcements – control received prefixes – control advertised prefixesMatjaž Straus Istenič, 8.9.2011 15 15
    • Example: access port configuration • turn off anything but IP and ARP – no redirects ! example for Cisco IOS ! – no vendor proprietary interface TenGigabitEthernet3/3 ip address x.y.z.w 255.255.255.0 ip access-group IxIncoming in protocols like CDP ip access-group IxOutgoing out no ip redirects no ip proxy-arp – no broadcast ipv6 address 2001:.../64 ipv6 enable ipv6 traffic-filter IxIncoming6 in – no IPv6 RA ipv6 traffic-filter IxOutgoing6 out ipv6 nd reachable-time 300000 ipv6 nd ra suppress –! ICMP unreachables are no ipv6 redirects storm-control broadcast level 1.00 used in PMTU discovery ! no cdp enableMatjaž Straus Istenič, 8.9.2011 16 16
    • Multiple locations • routing considerations – localize traffic – minimize traffic between locations IXP @location 1 IXP @location 2 A A IXP LAN B interconnect B C DMatjaž Straus Istenič, 8.9.2011 17 17
    • Examples: two members prefer one location – the importance of next-hop self in iBGP –a member should use next-hop self in his iBGP sessions to avoid using the IX interconnect link ISP A ISP B R1 R3 eBGP A-B .1 .3 iBGP A iBGP B location 1 (backup) IXP LAN AS B AS A location 2 (primary) .17 .20 eBGP A-B ISP A ISP B ✔ preffered path R2 R4 ✗ avoid this pathMatjaž Straus Istenič, 8.9.2011 18 18Consider AS A and B. They have established two eBGP sessions - R1 - R3 and R2 - R4. We want to localize traffic between AS A and B perlocation, whenever possible. A and B agree to use location 2 for their primary peering and location 1 for backup. They use BGP attributesto achieve that (AS-path prepend at backup location 1, for example). Traffic that arrives from AS A to R1 (which is on a backup location)should be sent to R2 via A’s infrastructure, and then from R2 to R4. Therefore “next-hop self” is needed for iBGP R1-R2. Without “next-hopself”, R1 would learn B’s prefixes with next-hop R4 (.20) and forward traffic for B directly to R4 (directly connected) via IXP LAN -->adding additional load to interconnecting link.
    • Examples: members on both locations – prefixes are marked according to the location where they are being announced Im marking my prefixes I prefer prefixes – adjusting with 65432:1 65432:1 ISP A ISP B the metric R1 eBGP A-B R3 .1 .3 B P A- iBGP A e BG iBGP B location 1 IXP LAN AS B AS A location 2 eB GP A -B .17 .20 eBGP A-B ISP A ISP B R2 R4 ✔ preffered path Im marking my prefixes I prefer prefixes ✗ avoid this path with 65432:2 65432:2Matjaž Straus Istenič, 8.9.2011 19 19Consider AS A and B, again. Member at location 1 should prefer prefixes marked with BGP community for this location (65432:1 for example).It is advisable to adjust metric and not alter local-preference to achive that. Local-preference has higher priority than many other route-selection criteria, even higher than AS-path length, therefore adjusting metric is more appropriate. See some examples on the next slide.
    • Examples: members on both locations – again,next-hop self in iBGP Im marking my prefixes I prefer prefixes with 65432:1 65432:1 ISP A ISP B R1 R3 eBGP A-B .1 .3 B P A- iBGP A e BG iBGP B destination announced by location 1 IXP R4 only LAN AS B AS A location 2 eB GP A -B .17 .20 eBGP A-B ISP A ISP B R2 R4 ✔ preffered path Im marking my prefixes I prefer prefixes ✗ avoid this path with 65432:2 65432:2Matjaž Straus Istenič, 8.9.2011 20 20Next-hop self is also needed for iBGP -- for example, some B’s prefixes are announced from R4 only. Traffic from R1 should travers A’sinfrastructure to reach R4 (via R2) and should not be directly forwarded to R4 (.20).
    • Examples: localization • Cisco IOS ! router R3 at location 1 ! router R4 at location 2 ip community-list 61 permit 65432:1 ip community-list 62 permit 65432:2 ! ! route-map AnnounceToIX permit 10 route-map AnnounceToIX permit 10  set community 65432:1  set community 65432:2 ! ! route-map AcceptFromIX permit 10 route-map AcceptFromIX permit 10  ! this location  ! this location  match community 61  match community 62 route-map AcceptFromIX permit 20 route-map AcceptFromIX permit 20  ! other location - worse metric  ! other location - worse metric  set metric +1  set metric +1 ! ! router bgp <member-AS> router bgp <member-AS> template peer-policy IX template peer-policy IX route-map AcceptFromIX in route-map AcceptFromIX in route-map AnnounceToIX out route-map AnnounceToIX out next-hop-self next-hop-self send-community send-community ! ! address-family ipv4|6 address-family ipv4|6 neighbor <R1> inherit peer-policy IX neighbor <R1> inherit peer-policy IX neighbor <R2> inherit peer-policy IX neighbor <R2> inherit peer-policy IX ! !Matjaž Straus Istenič, 8.9.2011 21 21
    • Examples: localization • Juniper JUNOS /* router at location 1 */ /* router at location 2 */ protocols { protocols {     bgp {     bgp {         local-as <member-AS>;         local-as <member-AS>;         group Ix {         group Ix {             type external;             type external;             import [ LocalizeTraffic AcceptFromIx ];             import [ LocalizeTraffic AcceptFromIx ];             export AnnounceToIx;             export AnnounceToIx;         }         }     }     } } } policy-options { policy-options {     policy-statement AcceptFromIx {     policy-statement AcceptFromIx {         <member policy at receive>         <member policy at receive>     }     }     policy-statement AnnounceToIx {     policy-statement AnnounceToIx {         term Localize {         term Localize {             then {             then {                 community set IxLocation1;                 community set IxLocation2;                 next term;                 next term;             }             }         }         }         <member policy for announcements>         <member policy for announcements>     }     }     policy-statement LocalizeTraffic {     policy-statement LocalizeTraffic {         term LocalTraffic {         term LocalTraffic {             from community IxLocation1;             from community IxLocation2;             then next policy;             then next policy;         }         }         term OtherTraffic {         term OtherTraffic {             then {             then {                 metric add 1;                 metric add 1;             }             }         }         }     }     }     community IxLocation1 members 65432:1;     community IxLocation2 members 65432:2; } }Matjaž Straus Istenič, 8.9.2011 22 22
    • Examples: next-hop self, no redirects • A wants to get the traffic from B and send it to C • B should not send traffic eBGP A-B directly to C ISP A ISP C ISP B R1 R5 AS C R3 – next-hop self eBGP A-C in eBGP also .1 .2 .3 -C iBGP iBGP A – no ICMP GP eB location 1 IXP redirects location 2 LAN AS B AS A .17 .20 eBGP A-B ✔ preffered path ISP A R2 ISP B R4 ✗ avoid this pathMatjaž Straus Istenič, 8.9.2011 23 23Member C peers with A, but not with B.Consider that C announces prefix x to A with next-hop = R5 (.2) and let us assume, that A readvertises x to B. A want traffic towards x tobe forwarded to him, so he changes the next-hop to itself (to .1 at and to .17 at R2). A also disables ICMP redirect messages to avoidsending them to B when forwarding a packet for x back on the same interface where the packet came from. This also prevents traffic from Bfor x to be forwarded directly to C (which has no peering with B).
    • Example: BGP filters router bgp 65432 template peer-policy IX6 route-map AcceptFromIx in route-map AnnounceToIx out filter-list 200 out prefix-list FROM-IX-PREFIX6 in prefix-list TO-IX-PREFIX6 out next-hop-self soft-reconfiguration inbound remove-private-as maximum-prefix 1500 send-community exit-peer-policy ! neighbor <...> remote-as 65000 address-family ipv6 neighbor <...> inherit peer-policy IX6 neighbor <...> filter-list 166 in ! ipv6 prefix-list FROM-IX-PREFIX6 seq 5 deny ::/0 ipv6 prefix-list FROM-IX-PREFIX6 seq 10 deny <our-prefix>/32 if you decide to block small prefixes, ipv6 prefix-list FROM-IX-PREFIX6 seq 15 deny <our-prefix>/32 ge 33 for example, less than /56 ipv6 prefix-list FROM-IX-PREFIX6 seq 15 deny ::/0 ge 57 ipv6 prefix-list FROM-IX-PREFIX6 seq 25 permit ::/0 ge 1 ! ipv6 prefix-list TO-IX-PREFIX6 seq 5 permit <our-prefix>/32 ipv6 prefix-list TO-IX-PREFIX6 seq 10 permit <custumer1>/32 ipv6 prefix-list TO-IX-PREFIX6 seq 15 permit <customer2>/48 ... ! ip as-path access-list 166 permit ^(65000_)+$ ip as-path access-list 166 permit ^(65000_)+(65001_)+$ ip as-path access-list 166 permit ^(65000_)+(65002_)+$ ! ip as-path access-list 200 permit ^$ ip as-path access-list 200 permit ^(<our-custumer1-AS>_)+$ ip as-path access-list 200 permit ^(<our-customer2-AS>_)+$Matjaž Straus Istenič, 8.9.2011 24 24
    • Goodies • looking-glass router • route-server (reflector) • graphs – public – or members only – or private • meetings :-)Matjaž Straus Istenič, 8.9.2011 25 25