SIX and some best practices for running an IXP

1,945 views

Published on

Presented at ISOC's Serbian Open Exchange – IXP Workshop, December 4-5, 2013

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,945
On SlideShare
0
From Embeds
0
Number of Embeds
24
Actions
Shares
0
Downloads
41
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

SIX and some best practices for running an IXP

  1. 1. SIX and some best practices for running an IXP All that stuff around the switch Matjaž Straus Istenič, SIX, ARNES matjaz.straus@arnes.si sreda, 04. december 13 1
  2. 2. About me Matjaž Straus Istenič, 8.9.2011 sreda, 04. december 13 2 2
  3. 3. About me Matjaž Straus Istenič, 8.9.2011 sreda, 04. december 13 2 2
  4. 4. About me Triglav, 2864 m Matjaž Straus Istenič, 8.9.2011 sreda, 04. december 13 2 2
  5. 5. Agenda • Slovenian Internet Exchange - SIX • all that stuff around the switch • practical examples – addressing – best practices – configuration examples for the IXP – configuration examples, guidelines and hints for members Matjaž Straus Istenič, 8.9.2011 sreda, 04. december 13 2/417 3
  6. 6. Slovenia has tradition - not only in breweries SIX - operated by ARNES since 1994 photo: http://www.pivo-lasko.si/ Matjaž Straus Istenič, SEE2, Macedonia, 4/2013 sreda, 04. december 13 4 4
  7. 7. SIX - the history • started in february 1994 - two members • 1995: two more members • 1996: two more... – and the big Telecom • 1997 ... 2002: more alternative providers • 2000: second location, interconnect with the first at 2001 • 2003: third location (LIX, decommissioned 3/2012) • 2006: first IPv6 at SIX • 2009: new location at Ljubljana Technology Park Matjaž Straus Istenič, 8.9.2011 sreda, 04. december 13 5 5
  8. 8. SIX - the forbidden graphs 30 SIX members 27 24 • • • 26 members > 50% with 10 G most with IPv6 21 18 15 12 9 6 3 0 Matjaž Straus Istenič, 8.9.2011 sreda, 04. december 13 1994 1996 1998 2000 2002 2004 2006 2008 2010 2012 6 6
  9. 9. SIX - the IXP technology Matjaž Straus Istenič, 8.9.2011 sreda, 04. december 13 7 7
  10. 10. SIX - today • L2 • two locations • *national • Cisco 4500X • bird route server • IXP manager and portal *note: one cross-border link Matjaž Straus Istenič, 8.9.2011 sreda, 04. december 13 8 8
  11. 11. Stuff around the switch • proper location with many fibre providers –a building with one single provider is a bad idea • different fibre paths inside of the building • power supplies and grounding • cooling system • physical security • staff, support, remote hands • good and accurate documentation Matjaž Straus Istenič, 8.9.2011 sreda, 04. december 13 9 9
  12. 12. Stuff around the switch (cont.) • monitoring and alarming • ticketing system • mailing lists • web portal • best current practices and knowledge base • contracts, SLAs, billing, ... • planning for a collocation/datacenter Matjaž Straus Istenič, 8.9.2011 sreda, 04. december 13 10 10
  13. 13. The power Matjaž Straus Istenič, 8.9.2011 sreda, 04. december 13 11 11
  14. 14. The power • allocate up to 20 kW per rack • actual usage 5 kW - 10 kW per rack • dual separate circuit breaker for each rack • power supply redundancy – dual feed from electrical distribution company – separate dual UPS system N+1 and PDU – diesel generator (redundant) • cooling equipment is independently dual powered, including chillers • how much power does datacenter use – – • monitoring on UPS, on PDU monitoring total on main branch circuit typicaly the load will double in 5 years Matjaž Straus Istenič, 8.9.2011 sreda, 04. december 13 12 12
  15. 15. Cooling • full redundancy of cooling system – – separate piping – chiller redundancy – • two different power grids room units redundancy hot/cold isle – – cold aisle with barriers made of metal, plastic or fiberglass – • reduce air mixing use blanking panels on the cabinets without servers no need for double floor – run network cabling over the top of the cabinets – "in row" cooling • recommended temperature in cold isle is between 23 - 25 °C • cooling system rating must be 1.3 x IT load rating • make sure that the space will allow for future growth – • for more cooling capacity and redundancy if required Power usage effectiveness (PUE = Total Facility Power/IT Equipment Power) – typical PUE is 2.0 or higher Matjaž Straus Istenič, 8.9.2011 sreda, 04. december 13 13 13
  16. 16. Matjaž Straus Istenič, 8.9.2011 sreda, 04. december 13 14 14
  17. 17. sreda, 04. december 13 15
  18. 18. Fire protection • sensing the smoke/fire type aspiration sensor optical sensor Matjaž Straus Istenič, 8.9.2011 sreda, 04. december 13 ✔ ✗ • very sensitive • early warning • single point of electrical • more expensive • air ducting under the ceiling • cheaper • can be used as confirmation • less sensitive • each sensor needs its own instalation • targeted sensing is possible for fast aspiration sensors must be installed cable 16 16
  19. 19. Fire protection • extinguishing fire Gaseous fire extinguishing system All are considered safe for breathing after release, although, products of burning plastics are always dangerous! type displacement of air Inergen - mixture of gases, displaces air with “air” with less oxygen ✔ active substance chemical action cooling • • totally natural environmentaly neutral • • • • Novec 1230 - chemical bonding, cooling • • • small storage area stored as fluid very small greenhouse gas footprint • • • has some effect on environment expensive stored under pressure (40/50 bar) FM200 (phasing out) - chemical bonding • • small storage area small greenhouse gas footprint • • • being phased out has some ozone depletion impact stored under pressure (40/50 bar) water mist • • totally natural environmentaly neutral • • water in computer room is not a good idea ;-) possible condensation on cold surfaces Matjaž Straus Istenič, 8.9.2011 sreda, 04. december 13 ✗ big storage requirements high pressure (200 or 300 bar) computer room needs big exhaust vents big rush of gas at release causes dust and objects to lift 17 17
  20. 20. Fire protection • extinguishing fire Gaseous fire extinguishing system All are considered safe for breathing after release, although, products of burning plastics are always dangerous! type displacement of air Inergen - mixture of gases, displaces air with “air” with less oxygen ✔ active substance chemical action cooling • • totally natural environmentaly neutral • • • • Novec 1230 - chemical bonding, cooling • • • small storage area stored as fluid very small greenhouse gas footprint • • • has some effect on environment expensive stored under pressure (40/50 bar) FM200 (phasing out) - chemical bonding • • small storage area small greenhouse gas footprint • • • being phased out has some ozone depletion impact stored under pressure (40/50 bar) water mist • • totally natural environmentaly neutral • • water in computer room is not a good idea ;-) possible condensation on cold surfaces Matjaž Straus Istenič, 8.9.2011 sreda, 04. december 13 ✗ big storage requirements high pressure (200 or 300 bar) computer room needs big exhaust vents big rush of gas at release causes dust and objects to lift 17 17
  21. 21. Examples and guidelines • addressing • IXP port configuration • guidelines for members • goodies Matjaž Straus Istenič, 8.9.2011 sreda, 04. december 13 18 18
  22. 22. Examples: addressing • a single subnet taken from independent address space – • member address is assigned per location address schema at SIX 91.220.194.n/24 n = n1 = 2..99 at location 1 n = n1 + 100 = 102..199 at location 2 n = 1, 101 for route-reflectors Matjaž Straus Istenič, 8.9.2011 sreda, 04. december 13 2001:7f8:46:0:L:N::<AS>/64 L = 0 at location 1 L = 1 at location 2 N = 0 for a single router, otherwise N = 1, 2, ... AS = member AS in decimal AS = 51988 for route-server - diverse lower 24 bits which form solicited-node mcast address 19 19
  23. 23. Examples: IXP port configuration • access port on Cisco 4500X interface TenGigabitEthernet1/6 description -- member (AS...) -switchport access vlan <x> switchport mode access switchport nonegotiate switchport port-security load-interval 30 datalink flow monitor FlowMonitor-L2 input storm-control broadcast level 1.00 storm-control action shutdown no cdp enable spanning-tree portfast spanning-tree bpduguard enable service-policy input INPUT-200M-EF ! policy-map LIMIT-QUEUE-200 class class-default queue-limit 200 ! Matjaž Straus Istenič, 8.9.2011 sreda, 04. december 13 flow record StandardFlow-L2 match datalink mac source address input match datalink mac destination address input collect interface input collect interface output collect counter bytes long collect counter packets collect timestamp sys-uptime first collect timestamp sys-uptime last ! flow exporter FlowExporter destination <x.y.z.w> vrf mgmtVrf source FastEthernet1 transport udp <port> template data timeout 60 ! flow monitor FlowMonitor-L2 record StandardFlow-L2 exporter FlowExporter cache timeout active 60 ! 20 20
  24. 24. Examples: IXP port configuration • interconnecting ports – aggregated – maximal to EtherChannel with LACP MTU interface TenGigabitEthernet1/1 interface Port-channel48 switchport access vlan <N> description -- IX-trunk -switchport mode access switchport switchport nonegotiate switchport access vlan <N> mtu 9198 switchport mode access load-interval 30 switchport nonegotiate datalink flow monitor FlowMonitor-L2 input mtu 9198 channel-protocol lacp bandwidth 10000000 channel-group 48 mode active load-interval 30 ! datalink flow monitor FlowMonitor-L2 input interface TenGigabitEthernet1/2 flowcontrol receive on switchport access vlan <N> ! switchport mode access port-channel load-balance src-dst-ip switchport nonegotiate mtu 9198 load-interval 30 datalink flow monitor FlowMonitor-L2 input channel-protocol lacp channel-group 48 mode active ! Matjaž Straus Istenič, 8.9.2011 sreda, 04. december 13 21 21
  25. 25. Guidelines for members • the bads (proxy ARP, redirects) • access port configuration • BGP – routing considerations (next-hop, localization) – safety (MD5 authentication) – policy (filtering announcements) – control received prefixes – control advertised prefixes – RPKI Matjaž Straus Istenič, 8.9.2011 sreda, 04. december 13 22 22
  26. 26. Proxy ARP in action • incident at AMS-IX, DE-CIX, ... - proxy ARP enabled - router has no IP address from the peering LAN - router has a default route or - router has a more specific route for the peering LAN reference: Maksym Tulyuk, Wolfgang Tremmel, reported at RIPE63 http://ripe63.ripe.net/presentations/ Presenter Name, Date sreda, 04. december 13 23 23
  27. 27. ARP hijacking • no RS, full BGP mesh between R2, R3 in R4 • normal situation R1 MAC A R2 MAC B R2 is at B R3 is at C R4 is at D R1 is at A R3 is at C R4 is at D BGP BGP R1 is at A R2 is at B R3 is at C R1 is at A R2 is at B R4 is at D BGP R4 MAC D R3 MAC C source: ARP Hijacking Mitigation, 19th Euro-IX Forum, 10/2011, Steven Bakker @ams-ix.net Presenter Name, Date sreda, 04. december 13 24 24
  28. 28. ARP hijacking (2/8) • R1 send bogus ARP replies R1 MAC A R2 is at B R3 is at C R4 is at D R2 MAC B R1 is at A R3 is at C R4 is at D R2 is at A R3 is at A R4 is at A BGP BGP R1 is at A R2 is at B R3 is at C R1 is at A R2 is at B R4 is at D BGP R4 MAC D R3 MAC C source: ARP Hijacking Mitigation, 19th Euro-IX Forum, 10/2011, Steven Bakker @ams-ix.net Presenter Name, Date sreda, 04. december 13 25 25
  29. 29. ARP hijacking (3 /8) • ARP cache poisioned • BGP down • traffic stops R1 MAC A R2 MAC B R1 is at A R2 is at A R3 is at A R1 is at A R3 is at A R4 is at A R1 is at A R2 is at A R4 is at A R4 MAC D R3 MAC C source: ARP Hijacking Mitigation, 19th Euro-IX Forum, 10/2011, Steven Bakker @ams-ix.net Presenter Name, Date sreda, 04. december 13 26 26
  30. 30. ARP hijacking (4/8) • hijacker R1 isolated • ARP caches recover with BGP packets R2 MAC B • BGP up • traffic normalizes after a few minutes R3 is at C R4 is at D BGP BGP R2 is at B R3 is at C R2 is at B R4 is at D BGP R4 MAC D R3 MAC C source: ARP Hijacking Mitigation, 19th Euro-IX Forum, 10/2011, Steven Bakker @ams-ix.net Presenter Name, Date sreda, 04. december 13 27 27
  31. 31. What if route-server is being used? • RS, partial BGP between RS and R2, R3 • normal situation R1 MAC A R2 MAC B R2 is at B R3 is at C RS is at D R1 is at A R3 is at C RS is at D BGP R1 is at A R2 is at B R3 is at C R1 is at A R2 is at B RS is at D BGP RS MAC D R3 MAC C source: ARP Hijacking Mitigation, 19th Euro-IX Forum, 10/2011, Steven Bakker @ams-ix.net Presenter Name, Date sreda, 04. december 13 28 28
  32. 32. ARP hijacking (6/8) • R1 send bogus ARP replies R1 MAC A R2 is at B R3 is at C RS is at D R2 MAC B R1 is at A R3 is at C RS is at D R2 is at A R3 is at A RS is at A BGP R1 is at A R2 is at B R3 is at C R1 is at A R2 is at B RS is at D BGP RS MAC D R3 MAC C source: ARP Hijacking Mitigation, 19th Euro-IX Forum, 10/2011, Steven Bakker @ams-ix.net Presenter Name, Date sreda, 04. december 13 29 29
  33. 33. ARP hijacking (7/8) • ARP cache poisioned • BGP with RS down • traffic stops R1 MAC A R2 MAC B R1 is at A R2 is at A R3 is at A R1 is at A R3 is at A RS is at A R1 is at A R2 is at A RS is at A RS MAC D R3 MAC C source: ARP Hijacking Mitigation, 19th Euro-IX Forum, 10/2011, Steven Bakker @ams-ix.net Presenter Name, Date sreda, 04. december 13 30 30
  34. 34. ARP hijacking (8/8) • hijacker R1 isolated • ARP caches partially recover with BGP packets R1 MAC A • BGP up • R2 MAC B traffic is being blackholed – R2 and R3 still have bogus entries for each other – R1 je na A R3 je na A RS je na D BGP outage can last for hours R2 je na B R3 je na C R1 je na A R2 je na A RS je na D BGP RS MAC D R3 MAC C source: ARP Hijacking Mitigation, 19th Euro-IX Forum, 10/2011, Steven Bakker @ams-ix.net Presenter Name, Date sreda, 04. december 13 31 31
  35. 35. ARP Sponge R2 MAC B • update ARP caches • mitigates unknown unicast • 3 ARP update methods: • spoofed unsolicited ARP reply • spoofed gratuitous ARP query • spoofed ARP request (dve muve jednim udarcem ;-)) R3 is at C R2 is at B R3 is at C in case of unknown unicast: - the unknown is here R2 is at B R3 MAC C from to message sponge B reply: R3 is at C sponge B request: where is R3? - tell R3 at C sponge B request: where is R2? - tell R3 at C B C reply: R2 is at B source: ARP Hijacking Mitigation, 19th Euro-IX Forum, 10/2011, Steven Bakker @ams-ix.net Presenter Name, Date sreda, 04. december 13 32 32
  36. 36. Tuđe nećemo, ... 2001:db8::/48 R2 doesn’t want to receive any traffic from R3 B R2 eB GP B ne xth op (2 00 1: db 8: :/4 8 • R2 peers with R1 but not with R3 )= • A next-hop(2001:db8::/48) = A or B? R1 Presenter Name, Date sreda, 04. december 13 eBGP R3 33 33
  37. 37. Tuđe nećemo, ... 2001:db8::/48 R2 doesn’t want to receive any traffic from R3 B R2 eB GP B ne xth op (2 00 1: db 8: :/4 8 • R2 peers with R1 but not with R3 )= • A next-hop(2001:db8::/48) = A or B? R1 Presenter Name, Date sreda, 04. december 13 eBGP R3 33 33
  38. 38. Tuđe nećemo, ... 2001:db8::/48 R2 doesn’t want to receive any traffic from R3 B R2 eB GP B ne xth op (2 00 1: db 8: :/4 8 • R2 peers with R1 but not with R3 )= • A next-hop(2001:db8::/48) = A or B? R1 eBGP R3 ✔ preffered path ✗ path to avoid Presenter Name, Date sreda, 04. december 13 33 33
  39. 39. Tuđe nećemo, ... 2001:db8::/48 R2 doesn’t want to receive any traffic from R3 B R2 eB GP B ne xth op (2 00 1: db 8: :/4 8 • R2 peers with R1 but not with R3 )= • with next-hop self at R1 next-hop for 2001:db8::/48 at R3 is A, not B ✔ next-hop self in eBGP Presenter Name, Date sreda, 04. december 13 A next-hop(2001:db8::/48) = A or B? R1 eBGP R3 ✔ preffered path ✗ path to avoid 33 33
  40. 40. ... svoje ne damo! 2001:db8::/48 R2 B ne xth op (2 eB GP 8: :/4 8 )= B R1 doesn’t want to redirect any traffic to R2 00 • R1 receives traffic and sends it back via the same port 1: db • A next-hop(2001:db8::/48) = A or B? R1 eBGP R3 ✔ preffered path ✗ path to avoid Presenter Name, Date sreda, 04. december 13 34 34
  41. 41. ... svoje ne damo! 2001:db8::/48 R2 B ne xth op (2 eB GP 8: :/4 8 )= B R1 doesn’t want to redirect any traffic to R2 00 • R1 receives traffic and sends it back via the same port 1: db • ICMP redirect messages should not be sent ✔ no ip redirects Presenter Name, Date sreda, 04. december 13 A next-hop(2001:db8::/48) = A or B? R1 eBGP R3 ✔ preffered path ✗ path to avoid 34 34
  42. 42. Unreachables and PMTU discovery 1 9 Presenter Name, Date sreda, 04. december 13 00 0 00 5 ICMP 3/4 Packet too big, fragmentation required and DF flag set 35 35
  43. 43. Unreachables and PMTU discovery ICMP unreachables are always sent for IPv4 9 00 0 1 00 5 ICMP 3/4 Packet too big, fragmentation required and DF flag set note: In IPv6, ICMP Packet-too-big message is not an “Unreachable” Presenter Name, Date sreda, 04. december 13 35 35
  44. 44. Example: member port configuration • turn off anything but IP and ARP – no proxy ARP – no redirects – no vendor proprietary protocols like CDP – no broadcasts – no IPv6 RA – ICMP unreachables are used in PMTU discovery in IPv4 Matjaž Straus Istenič, 8.9.2011 sreda, 04. december 13 ! example for Cisco IOS ! interface TenGigabitEthernet3/3 ip address x.y.z.w 255.255.255.0 ip access-group IxIncoming in ip access-group IxOutgoing out no ip redirects no ip proxy-arp ipv6 address 2001:.../64 ipv6 enable ipv6 traffic-filter IxIncoming6 in ipv6 traffic-filter IxOutgoing6 out ipv6 nd reachable-time 300000 ipv6 nd ra suppress no ipv6 redirects storm-control broadcast level 1.00 no cdp enable ! 36 36
  45. 45. Multiple locations • routing considerations – localize traffic – minimize traffic between locations IXP @location 1 IXP @location 2 A A B C Matjaž Straus Istenič, 8.9.2011 sreda, 04. december 13 IXP LAN interconnect B D 37 37
  46. 46. Multiple locations • routing considerations – localize traffic – minimize traffic between locations IXP @location 1 IXP @location 2 A A B C Matjaž Straus Istenič, 8.9.2011 sreda, 04. december 13 IXP LAN interconnect B D 37 37
  47. 47. Multiple locations • routing considerations – localize traffic – minimize traffic between locations IXP @location 1 IXP @location 2 A A B C Matjaž Straus Istenič, 8.9.2011 sreda, 04. december 13 IXP LAN interconnect B D 37 37
  48. 48. Localization tt w o lo ca tio ns 2001:db8::/48 ne next-hop(2001:db8::/48) = R2 iBGP m em be ra R2 B xt- ho p(2 00 1:d b8 ::/4 8 eB )= B GP A R3 m em R1 eBGP lo ber ca a tio t o n ne next-hop(2001:db8::/48) = A Matjaž Straus Istenič, 8.9.2011 sreda, 04. december 13 38 38
  49. 49. Localization tt w o lo ca tio ns 2001:db8::/48 ne next-hop(2001:db8::/48) = R2 iBGP m em be ra R2 B xt- ho p(2 00 1:d b8 ::/4 8 eB )= B GP A R3 m em R1 eBGP lo ber ca a tio t o n ne next-hop(2001:db8::/48) = A Matjaž Straus Istenič, 8.9.2011 sreda, 04. december 13 38 38
  50. 50. Localization 2001:db8::/48 tt w o lo ca tio ns use BGP communities R2 ne next-hop(2001:db8::/48) = R2 I’m marking my prefixes with community for blue location iBGP m em be ra • B xt- ho p(2 00 1:d b8 ::/4 8 eB )= B GP A Matjaž Straus Istenič, 8.9.2011 sreda, 04. december 13 m I’m marking my prefixes with community for red location R3 em R1 eBGP lo ber ca a tio t o n ne next-hop(2001:db8::/48) = A I preffer prefixes with red community 38 38
  51. 51. Examples: localization • Cisco IOS ! router at location 1 ip community-list 61 permit 65432:1 ! route-map AnnounceToIX permit 10  set community 65432:1 ! route-map AcceptFromIX permit 10  ! this location  match community 61 route-map AcceptFromIX permit 20  ! other location - worse metric  set metric +1 ! router bgp <member-AS> template peer-policy IX route-map AcceptFromIX in route-map AnnounceToIX out next-hop-self send-community ! address-family ipv4|6 neighbor <R1> inherit peer-policy IX neighbor <R2> inherit peer-policy IX ! Matjaž Straus Istenič, 8.9.2011 sreda, 04. december 13 ! router at location 2 ip community-list 62 permit 65432:2 ! route-map AnnounceToIX permit 10  set community 65432:2 ! route-map AcceptFromIX permit 10  ! this location  match community 62 route-map AcceptFromIX permit 20  ! other location - worse metric  set metric +1 ! router bgp <member-AS> template peer-policy IX route-map AcceptFromIX in route-map AnnounceToIX out next-hop-self send-community ! address-family ipv4|6 neighbor <R1> inherit peer-policy IX neighbor <R2> inherit peer-policy IX ! 39 39
  52. 52. Examples: localization • Juniper JUNOS /* router at location 1 */ protocols {     bgp {         local-as <member-AS>;         group Ix {             type external;             import [ LocalizeTraffic AcceptFromIx ];             export AnnounceToIx;         }     } } policy-options {     policy-statement AcceptFromIx {         <member policy at receive>     }     policy-statement AnnounceToIx {         term Localize {             then {                 community set IxLocation1;                 next term;             }         }         <member policy for announcements>     }     policy-statement LocalizeTraffic {         term LocalTraffic {             from community IxLocation1;             then next policy;         }         term OtherTraffic {             then {                 metric add 1;             }         }     }     community IxLocation1 members 65432:1; } Matjaž Straus Istenič, 8.9.2011 sreda, 04. december 13 /* router at location 2 */ protocols {     bgp {         local-as <member-AS>;         group Ix {             type external;             import [ LocalizeTraffic AcceptFromIx ];             export AnnounceToIx;         }     } } policy-options {     policy-statement AcceptFromIx {         <member policy at receive>     }     policy-statement AnnounceToIx {         term Localize {             then {                 community set IxLocation2;                 next term;             }         }         <member policy for announcements>     }     policy-statement LocalizeTraffic {         term LocalTraffic {             from community IxLocation2;             then next policy;         }         term OtherTraffic {             then {                 metric add 1;             }         }     }     community IxLocation2 members 65432:2; } 40 40
  53. 53. BGP • authenticate (secure) BGP session • filter announcements • sanity checks • a must-read – BGP Operations and Security http://tools.ietf.org/id/draft-jdurand-bgp-security-02.txt – Internet Exchange Route Server Operation http://tools.ietf.org/html/draft-ietf-grow-ix-bgp-route-server-operations-01 Matjaž Straus Istenič, 8.9.2011 sreda, 04. december 13 41 41
  54. 54. Example: BGP filters router bgp 65432 • remove your own communities <your-as>:<community> template peer-policy Ix6 • accept only communities that are meaningful for you route-map AcceptFromIx in • respect “no-export” route-map AnnounceToIx out • do not remove other communities for no reason filter-list 200 out prefix-list FromIx6 in prefix-list ToIx6 out • properly mark your prefixes next-hop-self remove-private-as • limit the number of accepted prefixes maximum-prefix 1000 (beware of the full routing table!) send-community ! template peer-session Ix6 • authenticate with MD5 password <default_key> • TTL security (optional) ttl-security hops 1 update-source Vlan50 ! neighbor <...> remote-as 65000 address-family ipv6 neighbor <...> inherit peer-policy Ix6 neighbor <...> inherit peer-session Ix6 neighbor <...> password <another_key> neighbor <...> filter-list 100 in ... ! Matjaž Straus Istenič, 8.9.2011 sreda, 04. december 13 42 42
  55. 55. Example: prefix filters • beware of the default route! router bgp 65432 • desi se i najboljima ;-) template peer-policy Ix6 filter-list 200 out • do not accept your own prefixes - they should stay at home prefix-list FromIx6 in • do not accept too specific prefixes prefix-list ToIx6 out • SIX policy: /8 .. /25 for IPv4, /16 .. /48 for IPv6 ... • block martians exit-peer-policy • announce your own and nothing else ! neighbor <...> remote-as 65000 address-family ipv6 neighbor <...> inherit peer-policy Ix6 neighbor <...> filter-list 100 in ! ipv6 prefix-list FromIx6 seq 5 deny ::/0 ipv6 prefix-list FromIx6 seq 10 deny <our-prefix>/32 ipv6 prefix-list FromIx6 seq 15 deny <our-prefix>/32 ge 33 ipv6 prefix-list FromIx6 seq 15 deny ::/0 ge 48 ipv6 prefix-list FromIx6 seq 20 deny 2002::/16 ge 17 ipv6 prefix-list FromIx6 seq 99 permit 2000::/3 ge 4 ! ipv6 prefix-list ToIx6 seq 5 permit <our-prefix>/32 ipv6 prefix-list ToIx6 seq 10 permit <customer1>/32 ipv6 prefix-list ToIx6 seq 15 permit <customer2>/48 ... ! ip as-path access-list 100 permit ^(65000_)+$ ip as-path access-list 100 permit ^(65000_)+.*(65001_)+$ ip as-path access-list 100 permit ^(65000_)+.*(65002_)+$ ! ip as-path access-list 200 permit ^$ ip as-path access-list 200 permit ^(<our-customer1-AS>_)+$ ip as-path access-list 200 permit ^(<our-customer2-AS>_)+$ Matjaž Straus Istenič, 8.9.2011 sreda, 04. december 13 43 43
  56. 56. Register your route objects $ whois -h whois.ripe.net -- '-i or AS2107' | grep ^route route: 109.127.192.0/18 route: 141.255.192.0/18 route: 149.62.64.0/18 route: 153.5.0.0/16 route: 164.8.0.0/16 route: 164.8.0.0/17 route: 164.8.128.0/17 route: 164.8.128.0/20 route: 178.172.0.0/17 route: 185.13.52.0/22 route: 193.138.1.0/24 route: 193.138.2.0/24 route: 193.2.0.0/16 route: 194.249.0.0/16 route: 212.235.128.0/17 route: 88.200.0.0/17 route: 92.244.64.0/19 route: 95.87.128.0/18 route6: 2001:1470::/29 route6: 2001:1470::/32 Matjaž Straus Istenič, 8.9.2011 sreda, 04. december 13 44 44
  57. 57. Example: What is registered at RIPE? • peval – list of ASNs at the end of the AS-PATH – list of prefixes $ peval -h whois.ripe.net -protocol ripe -no-as AS-ARNES ((AS28933 AS2121 AS51988 AS42909 AS12785 AS50195 AS2107 )) $ peval -h whois.ripe.net -protocol ripe 'afi ipv6 AS-ARNES' ({2A00:1600::/32, 2A00:1368::/32, 2001:1470::/32, 2001:7F8:46::/48, 2001:67C:64::/48, 2001:678:4::/48, 2001:678:5::/48}) Matjaž Straus Istenič, 8.9.2011 sreda, 04. december 13 45 45
  58. 58. Example: What is registered at RIPE? • public whois servers $ whois -h filtergen.level3.net -- "-v6 RIPE::RS-ARNES-HOSTED" Prefix list for policy RIPE::RS-ARNES-HOSTED = RIPE::RS-ARNES-HOSTED 2001:503:c27::/48 2001:503:231d::/48 2001:658:4::/48 2001:658:5::/48 2001:67c:44::/48 2001:7f8:46::/48 Matjaž Straus Istenič, 8.9.2011 sreda, 04. december 13 46 46
  59. 59. Register your peering information Matjaž Straus Istenič, 8.9.2011 sreda, 04. december 13 47 47
  60. 60. Register your peering information Matjaž Straus Istenič, 8.9.2011 sreda, 04. december 13 https://www.peeringdb.com/ 47 47
  61. 61. RPKI-based BGP route origin validation Matjaž Straus Istenič, 8.9.2011 sreda, 04. december 13 48 48
  62. 62. RPKI-based BGP route origin validation https://certification.ripe.net/ Matjaž Straus Istenič, 8.9.2011 sreda, 04. december 13 48 48
  63. 63. Goodies • route server (reflector) • IXP manager • looking-glass router • graphs – public – or – or • members only private meetings :-) Matjaž Straus Istenič, 8.9.2011 sreda, 04. december 13 49 49
  64. 64. Route server • use route server from day 1 • SIX uses bird - http://bird.network.cz • how it works • goodies – enforced policy – community based prefix filtering – “automatic” Presenter Name, Date sreda, 04. december 13 localization 50 50
  65. 65. Bird at SIX BGP R2 RT R1 RT R1 POLICY BGP R2 POLICY R1 R2 RS member 1 R101 POLICY BGP R101 Presenter Name, Date sreda, 04. december 13 R101 RT MASTER RIB member 2 R102 POLICY R102 RT BGP R102 51 51
  66. 66. Bird at SIX All the magic happens here! BGP R2 RT R1 RT R1 POLICY BGP R2 POLICY R1 R2 RS member 1 R101 POLICY BGP R101 Presenter Name, Date sreda, 04. december 13 R101 RT MASTER RIB member 2 R102 POLICY R102 RT BGP R102 51 51
  67. 67. Bird at SIX All the magic happens here! BGP All valid routes are here. R2 RT R1 RT R1 POLICY BGP R2 POLICY R1 R2 RS member 1 R101 POLICY BGP R101 Presenter Name, Date sreda, 04. december 13 R101 RT MASTER RIB member 2 R102 POLICY R102 RT BGP R102 51 51
  68. 68. Bird at SIX All the magic happens here! BGP All valid routes are here. R2 RT R1 RT R1 POLICY BGP R2 POLICY R1 R2 RS member 1 R101 POLICY BGP R101 RT MASTER RIB member 2 R102 POLICY R102 RT BGP R101 R102 Policy (pipe) filters received routes, marks them, adjusts preference according to location, filters advertised routes. Presenter Name, Date sreda, 04. december 13 51 51
  69. 69. Route server • improved (enforced) security – filtering based on routing registry – matching on prefix and origin AS – blocks martians – blocks default – blocks more specifics Presenter Name, Date sreda, 04. december 13 27 52 52
  70. 70. Route server • improved (enforced) security – filtering based on routing registry – matching on prefix and origin AS – blocks martians – blocks default – blocks more specifics Presenter Name, Date sreda, 04. december 13 is ho the u w for yo e is ibl ur It yo s on ty of sp ri ! re cu rk se netwo 27 52 52
  71. 71. Example: route server - custom filtering • based on BGP communities description community extended community Prevent announcement of a prefix to a peer 0:peer-as soo:0:peer-as Announce a route to a certain peer 51988:peer-as soo:51988:peer-as Prevent announcement of a prefix to all peers 0:51988 soo:0:51988 Presenter Name, Date sreda, 04. december 13 53 53
  72. 72. Example: route server - localization • we adjust the route preference according to AS_PATH length BGP R2 RT R1 RT R1 POLICY BGP R2 POLICY R1 R2 RS member 1 R101 POLICY BGP Presenter Name, Date sreda, 04. december 13 R101 R101 RT MASTER RIB member 2 R102 POLICY R102 RT BGP R102 54 54
  73. 73. Example: route server - localization • we adjust the route preference according to AS_PATH length import from member RT/BGP to master RIB: preference = 100; if bgp_path.len > 50 then preference = 0; else preference = 100 - ( 2 * bgp_path.len ); export from master RIB to member RT/BGP: if same_location() then preference = preference + 1; BGP R2 RT R1 RT R1 POLICY BGP R2 POLICY R1 R2 RS member 1 R101 POLICY BGP Presenter Name, Date sreda, 04. december 13 R101 R101 RT MASTER RIB member 2 R102 POLICY R102 RT BGP R102 54 54
  74. 74. IXP Manager • portal and RS manager Presenter Name, Date sreda, 04. december 13 55 55
  75. 75. IXP Manager • portal and RS manager https://github.com/inex/IXP-Manager/wiki Presenter Name, Date sreda, 04. december 13 55 55
  76. 76. Looking glass Presenter Name, Date sreda, 04. december 13 screenshot from NLNOG RING http://lg.ring.nlnog.net/ 56 56
  77. 77. Looking glass https://github.com/sileht/bird-lg/ Presenter Name, Date sreda, 04. december 13 screenshot from NLNOG RING http://lg.ring.nlnog.net/ 56 56
  78. 78. Looking glass • ...or, at least, route-collector arnes@rarnes6.re0:rc> show route aspath-regex 2107.* active-path terse table inet6.0 inet6.0: 132 destinations, 330 routes (132 active, 0 holddown, 0 hidden) Restart Complete + = Active Route, - = Last Active, * = Both A * * * * * * Destination 2001:678:4::/48 2001:678:5::/48 2001:1470::/29 2001:1470::/32 2a00:1600::/32 2a00:d440::/29 P B B B B B B Prf 170 170 170 170 170 170 Metric 1 1 1 1 1 1 1 Metric 2 0 0 0 0 0 0 Next hop AS path >2001:7f8:46:0:1::2107 2107 >2001:7f8:46:0:1::2107 2107 >2001:7f8:46:0:1::2107 2107 >2001:7f8:46:0:1::2107 2107 >2001:7f8:46:0:1::2107 2107 >2001:7f8:46:0:1::2107 2107 42909 42909 I I 50195 58046 I I I I https://github.com/sileht/bird-lg/ Presenter Name, Date sreda, 04. december 13 screenshot from NLNOG RING http://lg.ring.nlnog.net/ 56 56
  79. 79. Graphs Presenter Name, Date sreda, 04. december 13 57 57
  80. 80. Graphs Presenter Name, Date sreda, 04. december 13 57 57
  81. 81. Graphs Presenter Name, Date sreda, 04. december 13 57 57
  82. 82. Graphs Presenter Name, Date sreda, 04. december 13 57 57
  83. 83. Graphs Presenter Name, Date sreda, 04. december 13 57 57
  84. 84. Graphs ta da ct le ph as ol ra C an d g you c an as ch mu Presenter Name, Date sreda, 04. december 13 57 57
  85. 85. Meet the community Presenter Name, Date sreda, 04. december 13 CC EssjayNZ/flickr 58 58
  86. 86. Thank you! sreda, 04. december 13 59

×