Building an Internet Exchange Point - Technical Checklist

This talk is about building an internet exchange points in regions where there is noexchange, or where an exchange project has previously failed. The idea is to spark a discussion afterwards.

The most important thing a technical manager at a start-up IXP must do, is plan the project. An InternetExchange point is simple to build from a technical pointof view, but in order to be successful, scalable, andmaintain a good reputation, you must plan every aspectof the technology and service.This presentation covers many of the technologyrequirements that you must consider when starting andbuilding a new or young Internet Exchange Point.

An internet exchange is a technical solution (a switch) to a technical problem (needtoo many cross-connects). If starting from scratch, it is essentially important to builda community between potential peers. Invite them to regular meetings in your city, and explain the beneﬁts of peering.Ensure that individuals in the room agree in principal to peering. Show them the Euro-IX IXP video: http://www.youtube.com/watch?v=a5837LcDHfE If you can not get people around a table, and on a mailing list, and send you a letter ofintent, you wont ever get them on the switch..

Administration Legal formation – Association ? Commercial ? Not for proﬁt ? Let the initialparticipants decide, they are also the advocates you have today. You need some kind of legal structure (or sponsor who has a legal structure) to signcontracts, and get numbering resources (more shortly.) What kind of rules do you want ? - Keep them simple so that there is a low barrier to entry - enforce technical rules that keep the exchange clean - consider appropriate business rules – pay bills on time, support the exchange’sevents and general meetings - constitution / board – elected or appointed ? If elected, consider political inﬂuence.

Define the offering carefully, as this will haveconsequences for your configuration. You should dothis before you buy a switch so that you know whichfeatures you need to supportWill you offer a single peering LAN, or will you offerpeers the opportunity to have private VLANs betweenother customers ? If you offer multiple VLANs then youshould make sure that your switch can support enoughVLANs for your expected growth. If you do offermultiple VLANs, then will you allow your customers tointerlink their own infrastructure? This would mean that- if you had multiple locations - you were filling up yourinter-site links with non-peering traffic. It also meansthat you would be offering a transport service - does thiscompete with your intended customers ? Is it ok for yourcustomers to sell each other transit via the exchange ?

Number resources : - No PI from ﬁnal /8 - Proposal of mine to allow PI from ﬁnal /8 for IXPs - v6 PI needs a published connection policy (explain to ripe how networks canconnect, fair connection policy, at least 3 participants). - ASN needs multihomed, routing policy. - Probably request 16 bit asn for your collector, this needs maximum connectivity.But support 32 bit asns on your collector) All number resources need an End User contract, and a payment of €50 per year.

Also an important part of the service offering is whetheryou will offer Jumbo frames transport at the exchange.This is popular because it enables services betweenmembers, such as : - storage - dsl aggregationBut you need to consider whether to allow it on thepeering LAN, or insist that users use a closed user groupfor Jumbo features.You should check whether jumbo frames can beconﬁgured on a per-port, or whether it is a globalsetting. If you offer Jumbo for any service, then all inter-switch links need to be Jumbo, which means that if youuse an ethernet based service to glue the exchange

If you are building a service that spans several facilities,or expanding your IXP into new sites, then you may wishto think about the resilient design of your peering LAN.Typically, exchanges build loops and then configure alayer two resilience protocol to shut down a single link inthat loop to prevent an ethernet storm.You may need to build a complex topology due to thelocation of your facility (and fibre availability), or in orderto make use of resilient fibre providers in a region. Inthis scenario, you may wish to consider a resilienceprotocol like MRP which can have multiple/overlappingrings configured.There are no significant differences between how youwould design a layer 2 IXP network, compared with howyou would design a resilient ISP network - inspect the

Port security is essential to the health of your new internetexchange point.It is simple for ISPs to accidentally create a loop facing theInternet Exchange point. Where an ISP transports internetexchange points from different regions to their router in a certaintown, it is also simple for them to bridge together multipleexchange points. Further, it is easy for the ISP to mistakenlyexpose their own layer-2 to the exchange. Easy for telco to loopfibre to exchange.Port-security guards against all of these failure modes by limitingthe number of MAC addresses that a switch will add to the CAMtable, on any particular port. - Only configured on customer facing ports, not inter-switch link - Typical to see ‘1’ MAC only, but some exchanges permit 2, toallow smooth customer router-swap, or layer 2 keepaliveprotocol (urgh!) - Exceed modes can be ‘drop frame’ (polite, but can be harderto debug), or ‘shut port’ (firm and effective..., make sure you

Here’s a photo of LONAP’s 6500 - not necessarilyadvocated as an exchange switch - but has done a goodjob for us for some time.Cut through vs Store and Forward - Mixing port speeds on cut-through switching can cause buffering problems.Mac learning - Dynamic, and conﬁguredDHCP / Spanning Tree ﬁlter (BPDU Guard)Layer 2 / 3 inspection ?SNMP and Management featuresPort mirroringUpcoming 40GE / 100GE supportResilient core components - power supplies, supervisorsFlowUnknown Unicast controlLink AggregationOptic Locking - Support for WDM opticsVlan TranslationDOM / optical monitoring - avoid DC trips to test cables!

The internet exchange will have a shared fate with thecolocation providers that host the switch : - Successful IXP makes a colocation desirable - IXP will rely on excellent colocation, so that service isreliable and customers are happyEuropean IXP model is usually one of independence fromcolo - very successful. However, this relies on the goodpartnership between IXP and Colocation. Must stressvalue of good partnership, ensure they understand whatyou do.It is easier to set expectations with the colocation at thebeginning of the project rather than when the project isrolled out.

Install a collector at your Internet Exchange Point. A collector is a BGP speaking router which : - Announces the internet exchange’s own routes - Has a unique, public ASN - Peers with all of your customers, mandatoryThis allows you to test your customers’ BGPconﬁguration before putting them in the ‘wild’ for theﬁrst time, and it also gives you a perspective to monitorthe health of your customers (and by extension, theexchange), and a perspective to understand yourcustomers’ issues when trying to support them. Youcan run other monitoring systems, e.g. test forprohibited traffic, via the collector (more on this topicsoon).

Monitoring is important. I recommend that youmonitor : - Availability of switches, collector - Availability of member ports, bgp sessions - Route servers (if offered) [ Example - Nagios ] - Traffic on exchange LAN (excellent for marketingpurposes!) - Traffic on customer port - Dropped frames / congestion on customer port - Dropped frames / congestion on ISL [ Example -Cacti ] - Latency between exchange infrastructure andcustomer/member port [ Example - Smokeping ]

Plan for a dense cabling environment. A successfulexchange is the aim of everyone here, but what will youdo if you are managing 200, 300, or 400 cables ? Ormore ?Pre-cable what can be done in the rack, and plan a cabledatabase. Will you manage the cabling or the endusers ? If you manage the cabling, then you have thepower to resolve faults quickly. This may cost a lot ofmoney, however.This is not a lesson in keeping cables tidy, the messageis to plan for a big environment, before you have aproblem with scale.

Internet Exchange users adore s-flow, because it givesthem insight into their top peers at an internet exchange(and possibly move peers to PNI, backup exchanges).Running an s-flow service is also of benefit to theinternet exchange, because it gives you an insight intowho is peering with who for marketing reasons, alsoallows you to ensure that the main flows are onprotected Interswitch links, and also look for missedpeering opportunities.

Should you offer a service level agreement ? - If an association, then SLA credits come out of member funds anyway – you needto charge them more in the ﬁrst place, in order to credit some. - If commercial, then this is just a business decision. How to measure ? - Port availability ? What if member causes outage ? - Switch availability ? Remember to ‘map’ SLA of colo power onto your SLA. ISLavailability / load ? Discards ?

Communication is very important. Run mailing lists : - operational-participant – so that participants can inform each other when there isnews. - operational-announce – so that the exchange can alarm participants about veryimpacting work - membership (if association) for company business - commercial – share business opportunities. Run separate mailing lists, because in larger organisations, different people will needto see different messages. Run your technical support through a ticketing system like RT, Zendesk, Kayako,because you need to track queries and never lose them.

A route server is a great thing for an internet exchange point to offer. It is a platform where peers can exchange preﬁxes with a large number of networksin a single BGP session (called Multilateral peering – MLP), so great for newmembers, and great for IXPs who have a large number of open peering networks on asingle LAN. Achieve a reliable service by operating two route-servers on different route-serverplatforms, but ensure they are both conﬁgured in the same way. Some exchanges have a mandatory MLP policy – this prevents many networks with aselective peering policy from connecting.

In a traditional peering LAN, BGP and trafﬁc is exchange directly between routers ofwilling peers. On a large LAN, this leads to a large administrative overhead for thepeers.

With multilateral peering, a single BGP session gives access to a large number ofpeers, but the trafﬁc still passes between participant routers, and NOT the exchangeroute server. This means that the exchange need to provision a service that can handle the BGPstate and behaviour only, not one which can handle the trafﬁc volumes.

Some networks instruct the route server to filter prefixes between route-serverparticipants. This is particularly useful when a route server user has a bgp customeron the exchange and does not want the customer to learn their on-net prefixes overthe exchange for free. Here it is extremely important to run a separate RIB (i.e. routing table) for eachrouting policy (the easy way to do that is run a different RIB for every connectedASN) so that you do not prevent shadowing. If one BGP best path is filtered by aparticipant, this should not prevent shadowing of the prefix to other route-serverparticipants. Filtering here is in the outbound direction – “do not show my prefixesto peer XXX”. This is normally influenced by communities attached to prefixes whensent to the route-server. For a wealth of information on this topic see the EIX-WGwebsite and the Euro-IX website, particularly slides by Nick Hilliard.

This is the rough process which I recommend for turningup new customers/members.Cabling - who is responsible for cabling ? Customer /Exchange / Colo ?When connected, add the port to the Quarantine VLAN(more in a moment), so that tests can occurWhen quarantine is complete, connect the customer tothe main peering LANAs soon as connection is complete, add to monitoringsystemsMake sure you ANNOUNCE new connections, so thatother participants know to turn up peers.

Install a Quarantine VLAN. I recommend that you put thecollector on this VLAN (as well as the production VLANtoo), so that you can sanity check every single newcustomer’s behaviour : - They do not trip port-security - They are not emitting banned link state protocols, e.g.OSPF - They are not emitting banned layer 2 traffic, like CDP/decnet... - They have to speak to you when service commences(check BGP in real time)Get them to conﬁgure the collector BGP session whenthey are in the quarantine VLAN. It will not establish,but this will have the effect of putting some real trafficonto the interface, so that you can check for good

It is extremely easy to offer IPv6 as an internet exchangepoint. - Get resources - Add to collector - Encourage customers to peerYou can derive customer address space from as numberor customer id number, e.g. 2001:7f8:xx::[id]:1 - id canbe AS or other identiﬁer. xx is assigned by RIPE (Internetexchange assignments come from this /32, each IXPgets a /48 each.)“Same service level” means you monitor the same thingsas you do for v4 (sessions, etc.)Used to be trendy to deploy v4 and v6 peering in

Congestion is the exchange point’s enemy. It manifests asdropped frames. It can normally occur in two places : - Member ports - give them tools to monitor their portutilisation, and be proactive about contacting customersabout upgrade paths. Offer affordable upgrade paths(Nx1GE, fractional 10GE, as well as 1GE->10GE). Congestioncan also appear in the customer’s backbone, if they aretransporting the peering port from one city to another, orusing a third party metro ethernet carrier to deliver the port.The way that your member reacts (upgrade, move trafficaway, shout) will depend on the local culture - i.e. do theyeven care that they are congesting ? Is a congestedexchange port better than a severely congested transit port ?In an ideal environment, member ports will be congestionfree.

When your exchange starts to scale, you may wish to look at installingan ARP sponge.This is an automated software tool which catches ARP packets for IPaddresses that are offline. The AMS-IX have an open source arp-sponge which is based on the number of requests for a hwaddress in agiven period.

Thank you for listening / reading. :-)Other sources of help and advice.RIPE EIX-Working Group. EURO-IX.

http://www.andyd.net	117
http://www.linkedin.com	3
http://static.slidesharecdn.com	2
http://translate.googleusercontent.com	1

Building an Internet Exchange Point - Technical Checklist

by Andy Davidson

Accessibility

Categories

Upload Details

Usage Rights

4 Embeds 123

Statistics

Building an Internet Exchange Point - Technical Checklist Document Transcript