The document outlines the technical and operational aspects of Internet Exchange Points (IXPs), highlighting their importance for local internet development and cost-effective network interconnections. It discusses various models for IXP implementation, governance, and operation, while emphasizing the need for a tailored approach based on local conditions. The document also showcases the benefits and challenges of IXPs, particularly in developing economies, and the role of organizations like APNIC in supporting their establishment.

1. 1
1 v1.0
Technical and Operational Aspects
of Regional Internet Exchange Model
Che-Hoo Cheng
APNIC
2024-11-05

2. 2 v1.0
2
Agenda
• Objectives of Peering, and Benefits of IXPs
• IXP Implementation Steps
• Operation/Business/Governance Models for IXPs
• IXP Development Work of APNIC
• Final Remarks
• Q&A

3. 3 v1.0
3
Disclaimer
• High-level knowledge exchange only
o Not so technical
• No “One Size Fits All”
o Many different “recipes” for running an IXP
• Just to provide hints, not answers
• Cannot cover all scenarios here because of limited time
• Supposed to be interactive

4. 4
4 v1.0
Objectives of Peering
and Benefits of IXPs

5. 5 v1.0
5
How Does Internet Operate?
• Internet is a network of networks
o Composed of networks of ISPs and users
• User networks connect to ISPs
• Small ISPs connect to large ISPs
• Various networks interconnected with one another to form Internet

6. 8 v1.0
8
Transit Provider /
Internet Gateway
(Upstream)
Global Internet
Downstream
Customer
Customer
routes only
Routes of
the whole world
All customer routes
Ordinary Transit Model – Internet Gateway
Google
eBay
APNIC

7. 10 v1.0
10
Transit Provider A
(Upstream)
Transit Provider B
(Upstream)
Downstream
Customer
Downstream
Customer
Downstream
Customer
Downstream
Customer
Routes of A and
its customers
Routes of B and
its customers
Ordinary Peering Model
Google
eBay
APNIC

8. 11 v1.0
11
Peering in General
• ASes are interconnected/peered one another at Internet exchanges points (IXPs)
or privately using BGP protocol
• For higher performance, lower latency and lower cost
• Usually no settlement between peers and circuit cost is shared
For mutual benefits
• Local-to-local traffic do NOT need to route through overseas
• Important to local Internet development
• Improved resilience
Local Peering

9. 13 v1.0
13
Full Mesh for Peering
10C2 = 45 circuits

10. 15 v1.0
15
What is Internet eXchange Point (IXP)?
• A shared physical network infrastructure over which various ASes can do easy
peering with one another
One physical connection to IXP used for interconnections with multiple networks
More cost-effective and scalable, especially with more participants
IXP serves Internet Service Providers (ISPs), Cloud Service Providers, Content
Providers, Content Delivery Network (CDN) Providers, Research & Education
(R&E) Networks and Government/Enterprise Networks

11. 16 v1.0
16
Simplest IXP Topology
IXP Switch
Border Router
of ISP A
Border Router
of ISP B
Border Router
of ISP D
Border Router
of ISP C

12. 19 v1.0
19
Main Benefits of IXP
• Keep local traffic local
o Important to local Internet development
• Easy interconnection and direct traffic exchange among participating networks
o Reduced cost – cheaper connectivity
o Enhanced network performance – faster speed
o Reduced latency – lower delay
o Improved resilience – more backup paths
• Help encourage development of more local content and local applications
o Helps local data centre business and other businesses
• Everybody is benefited
o The gain for each may be different but all will gain
• Often considered as Critical Internet Infrastructure locally, regionally or globally

13. 20 v1.0
20
Examples of Pacific Islands
• Far from any other places
• External connectivity very expensive
• Small markets because of small population
• Usually just a few ISPs but not usually interconnected locally
• Local traffic across ISPs usually routed through US or Australia
• Local IXP very much needed for helping Internet development
• Observed immediate benefits on Day 1 of set-up of Fiji-IXP
o Much improved latency and high volume of traffic
o And improved resilience
• Small land-locked economies have more or less similar issues

14. 21
Before Fiji-IXP was
set up
• A Fijian ISP in Suva accessing
content at the University of the
South Pacific in Suva
• Packet travels > 25,000km
• Physical distance < 10km
• Adding long latency
• Possibly high jitter too
• Using expensive submarine
capacity
• Return path had similar issue
• SOLVED after Fiji-IXP was set up
• Resilience improved as well

15. 22 v1.0
22
Value and Attractiveness of an IXP
• Proportional to number of different networks (ASNs) connected and amount
of traffic exchanged
• Snowball effect after reaching critical mass
o Initial period usually the hardest
▸ Most take wait-and-see approach
▸ Market needs determine growth
o Gradually have good mix of networks of different types
▸ E.g. Eyeballs vs Content
• Good quality service useful and important

16. 23
23 v1.0
IXP Implementation Steps

17. 24 v1.0
24
IXPs are Layer-2 Networks
• Switched Ethernet
o One physical connection for interconnections with multiple networks
o Only routers are allowed to connect to the switching fabric directly
• IXP participants can do direct Bilateral Peering (BLPA) over the layer 2
infrastructure anytime
• With Route Server added to the layer 2 infrastructure, IXP participants can also do
Multilateral Peering (MLPA) for easier interconnections among everybody
o Traffic exchange is direct and not going through the route server
• Those called themselves “IX” but serving layer-3 services are considered as
transit providers
o Note that IXPs, transit providers and data centres are not the same things

18. 25 v1.0
25
Evolution
• IXP development is an evolutionary process done step by step
• It can be improved over time, but picking the right initial neutral organisation
& governance model and a neutral site at the start is important for future
success
• Some Local IXPs can evolve into Regional IXPs

19. 26 v1.0
26
Simplest IXP Topology
IXP Switch
Border Router
of ISP A
Border Router
of ISP B
Border Router
of ISP D
Border Router
of ISP C

20. 27 v1.0
27
Possible Steps for IXP Development
• Can be gradual, step by step (some steps can be skipped or be done at the same time)
• Layer-2 network is the bare minimal
o Can use private IP addresses if small amount of participants
• Public IP addresses next
o Legal entity issue
• Site resilience is IMPORTANT while equipment resilience is also included
o Has to have site resilience sooner or later
• Route server(s) with ASN follows
o RPKI/ROV consideration
• Other value added services
o DNS: Root / TLDs / Recursive
o Shared CDN Caches

21. 29 v1.0
29
IXP Topology with Minimal Resilience
IXP Switch x 2
(at same location or different locations)
Border Router
of ISP A
Border Router
of ISP B
Border Router
of ISP D
Border Router
of ISP C

22. 30 v1.0
30
Adding Route Server for Multilateral Peering
Routes
Redistribute
Routes to All
RS

23. 31 v1.0
31
Route Origin Validation (ROV) at IXP
– via Route Server for Improved Routing Security
Validated
cache
Validator
RPKI-to-Router (RTR)
Routes
Tagged/Filtered
Routes
RS

24. 35 v1.0
35
Scalability and Resilience Issues
• IXPs were supposed to have no packet loss in its infrastructure
o And with very low latency and very good resilience too
• Become an issue when IXP grow beyond one switch
o Due to not enough ports or expanding to multiple sites
• Inter-switch links are the risk
o Over-subscription has to be minimised
• Also need to minimise single point of failure
o For improved resilience

25. 36 v1.0
36
IXP Topology with Minimal Resilience
IXP Switch x 2
(at same location or different locations)
Border Router
of ISP A
Border Router
of ISP B
Border Router
of ISP D
Border Router
of ISP C

26. 37 v1.0
37
Spine
Switch
Spine
Switch
Leaf
Switch
Leaf
Switch
Leaf
Switch
n x 100GE/10GE
Inter-Switch
Links
n x 100GE/10GE
Inter-Switch
Links
ISP ISP ISP ISP ISP ISP ISP
Spine
Switch
Spine
Switch
Leaf
Switch
Leaf
Switch
100GE/10GE/GE
Links
100GE/10GE/GE
Links
Leaf
Switch
Spine-and-Leaf Architecture for Beyond 2 Switches

27. 40 v1.0
40
IXP Manager Helps Manage “Everything”
• Open-source software
• Highly recommended to use it to manage your IXP
• Helps manage:
o Provisioning
o Monitoring
o Switches
o Route Servers
o PeeringDB / IXP-DB / IRRDB
o RPKI / MANRS
o And many more (Refer to https://docs.ixpmanager.org under “Features”)

28. 45
45 v1.0
Operation/Business/Governance
Models for IXPs

29. 46 v1.0
46
Government-led vs Industry-led
Subsidized vs Self-financed
Commercial vs Non-profit
IXP Models
• No one single model which can suit all situations
• Relative Neutrality is important
Developed economies vs Developing economies

30. 47 v1.0
47
Commercial vs Non-Profit
• Commercial set-up is free to do anything
o No need to care about neutrality too much
o IXP may be a service to help other business
• Non-profit set-up tends to be more cautious
o Neutrality is more important, at least to the target participants
o Tend to be more independent from individual participants
o Tend to offer fewer services

31. 48 v1.0
48
Developed Economies
• IXPs are business
o Even for not-for-profit set-up
o Less government involvement
• Multiple IXPs
o Keen competition
• But if they cannot keep intra-economy traffic local, someone needs to step
up
o Government? Industry group? Customer pressure?

32. 49 v1.0
49
IXPs and Data Centres in Developed Economies
• They are natural partners
• Common situation in advanced metro cities
o Multiple IXPs in one Data Centre
▸ A lot of data centres have their IXPs
o One IXP in multiple Data Centres
▸ The same layer-2 broadcast domain
▸ Circuit cost is a burden to the IXP
▸ A lot of telco’s have their IXPs
o Healthy competition would be good
▸ Customers have choices
▸ Also for better resilience

33. 50 v1.0
50
Developing Economies
• Some do not have any IXPs yet
• Local traffic does not stay local
o A lose-lose situation for everybody
• IXPs can help Internet development a lot
o Better to be non-for-profit set-up
o May need to start with subsidized model
o May not be a business at all
o Help from government is mostly needed
o Active participation of the biggest players is also very important
• For smaller developing economies, there should not be too many IXPs

34. 51 v1.0
51
Governance for Non-Profit IXPs
• Multi-stakeholder bottom-up approach is an approach with good
acceptance by the community
o E.g. membership-based model
• Government support is also important
o At the very least, should get the proper license
• Be as inclusive as possible in order to provide maximum benefits to the
whole community which it serves
• Should be fair and consistent to every participant or member
• Should be open and transparent as much as possible

35. 52 v1.0
52
Membership-Based Model for Non-Profit IXPs
• Networks connected can join as members
o Open membership vs closed membership
o Mandatory membership vs optional membership
• Full Members with voting rights vs Associate Members without voting rights
o Licensed ISPs vs others
o Local legal entities vs overseas legal entities
• Governance by the Board elected
o Policy decisions
o Financial responsibilities
• Government’s role
o A facilitator at the very least

36. 53 v1.0
53
Neutral Locations
• May choose one of the followings as starting point:
o University
o Landing Station
o Technology Park
o Carrier Neutral Commercial Data Center
o Government Data Center
• Having multiple carrier options with easy access is important
• Should maintain neutrality continuously
• Expansion to multiple sites within the same metro area can be done on Day 1 or
be done gradually coupled with growth
o This also helps improve neutrality as more options are provided

37. 54 v1.0
54
Anchor Participants
• It is important to have a few Anchor Participants to start with to help attract
other participants to join
• IXP has no value to attract anyone if no participants
o The more participants, the higher the value for the IXP to attract other participants

38. 55 v1.0
55
Geography
• IXP usually is NOT expanded beyond a metro area so as to avoid
competing with IXP participants and to maintain neutrality
o And simple port charging model can be used
• Usually start with the biggest city first and gradually set up separate
infrastructure in other bigger cities one by one

39. 56 v1.0
56
To Add Value
• Domain Name Infrastructure: DNS infrastructure is very important to Internet
operations so Root/TLD DNS server instance(s) should be connected directly to
IXP for direct peering in order to benefit all participants for better DNS
performance and resilience
• Shared CDN Caches: Connecting cache servers of popular content to the IXP will
help everyone save bandwidth, but the cost of the transit bandwidth needed for
cache-fill has to be properly shared by the ISPs benefited
o Different CDN providers have different supported models
o Need to think about long-term sustainability
• NOTE: Transit for the above should NOT be used for providing usual transit
service to IXP participants so as to maintain neutrality

40. 57 v1.0
57
Financial Model for Non-Profit IXPs
• Seed money?
o Perhaps from subsidies and/or sponsorship/donation
• IXPs need money to operate continuously
o Need to have a long-term sustainable finance model for full cost recovery of CapEx and OpEx
▸ All cost should be covered
o Should not forget about the limited lifetime of the equipment used so must save money for
future major equipment upgrade say once every 5 years
▸ By setting up a reserve fund
• Subsidies, sponsorship or volunteers support may not be stable
• Those networks which are benefited should all contribute in a fair way
o A charging model should be devised to help achieve that
• A good financial model will help sustain the IXP operations in long term

41. 65 v1.0
65
Government Funding for Non-Profit IXPs?
• More needed during infancy stage of IXP development
o Government usually can only provide one-off funding support
• For long-term, IXPs need to have a long-term sustainable financial model
o Better be together with bottom-up industry-led governance for IXP
o Align with bottom-up multi-stakeholder approach

42. 67 v1.0
67
Regional IXP?
• It can be an evolution from a successful local IXP facilitating good local peering
o Will attract more overseas ISPs to connect
• It is more for overseas participants to come to the IXP instead of having the IXP to
expand out to them
• It is usually coupled with a number of positive driving factors
o Competitive environment/pricing for international circuits
o Competitive environment/pricing for local circuits
o Competitive environment/pricing for data centres
o Open and competitive environment -> Submarine cable hub + Data centre hub + Popular IXPs
-> Regional Internet hub
• Good quality services with good resilience are also critical
o There can be multiple regional IXPs/hubs in a region

43. 70 v1.0
70
IXP across Multiple Cities / Economies?
• Affect neutrality?
o Generally considered as competing with participants which provide services across
the same set of locations
• Bad for non-profit IXPs targeting all kinds of networks or providers?
o Those that see competition may not join and then it may affect the goal of “keeping
local traffic local”
o Commercial IXPs can take this business risk especially if this may help their other
business

44. 71 v1.0
71
IXP across Multiple Cities / Economies??
• If more than 2 locations, what is the topology?
o Star or Ring?
• May not be good at Layer-2
o Backup circuit(s) at Layer-2?
o Not easy to manage such a large layer-2 network across long-distance
▸ Riskier to operate
o Or have to use more expensive technologies such as EVPN over VxLAN
• How to cover the cost of the long-distance bandwidth?

45. 72 v1.0
72
Charging for Long-Distance Bandwidth?
• Local IXPs usually charge fixed charge by port speed
o Simple and of low risk
• How to cover the cost of the long-distance / international bandwidth if it is
involved?
o How to be fair? Charge by usage?
o How to make sure it covers the cost fully?
o Somewhat similar to providing shared CDN caches to IXP participants but even more
complicated
▸ Multi-way traffic on possibly multiple circuits of different cost

46. 74 v1.0
74
Remote Layer-2 Connections to IXP?
• More and more common nowadays
o Some even from >1,000km away
• Using fibre-only connection is much easier, with much fewer issues
o But up to 70-80km only
• Some IXPs even help you to connect to oversea IXPs with one physical port
o One physical Ethernet connection to multiple networks/IXPs
• IXPs cannot afford to not support remote connections
o As they want to have more business, sometimes through resellers
o May even partner with overseas IXPs for providing remote connection service to them
• It is still for participants to come to the IXP instead of having the IXP to expand out

47. 75 v1.0
75
Some Lessons Learnt from Some Economies
• Government led IXP has outsourced the operations to one of the ISPs
o Other ISPs complain that they have hard time connecting to the IXP
o The IXP has became not neutral enough
• Government led IXP having CDN caches installed cannot get the ISPs to
use the CDN caches because of disagreement on the cost sharing
arrangement
o Lose-lose for everybody
• Industry led IXP installed at government data centre had power outages
multiple times
o Need better resilience including site resilience

48. 77
77 v1.0
IXP Development Work of APNIC

49. 78 v1.0
78
IXP Development Work of APNIC
• APNIC strongly believes IXPs help Internet development
o After all, IXPs serve and benefit APNIC members
o In fact, IXPs need IP addresses and ASNs and so are APNIC members themselves
• Do more on helping those developing economies
o Especially those which do not have any IXP yet
o Or those which their only IXP is not functioning well
• Training and Technical Assistance work primarily
o Not just for IXP operators but also for IXP participants
o Also help talk to major stakeholders to convince them of the benefits of having a local IXP while
maintaining neutrality
o May need help of Community Trainers and Consultants from time to time
• Having been supporting IXP development in Fiji, PNG, Vanuatu, Mongolia, Bhutan,
Myanmar, Thailand, Pakistan, Maldives, India and others

50. 79 v1.0
79
IXP Development Package of APNIC
• Training & Technical Assistance primarily
o Technical & non-technical
• Other possible support items (on case-by-case basis according to individual
needs):
o Equipment support along with APNIC Foundation
o Route Server
o ROV & IPv6 deployment support
o IXP Manager
o Root Server anycast instance
o RIPE Atlas Anchor
• APIX & MANRS Memberships are recommended to all IXPs
o Up-to-date and accurate records on PeeringDB and IXP-DB are recommended as well

51. 80 v1.0
80
Other Help & Support by APNIC
• APNIC also provides help & support to:
o APIX
o Peering Asia
o Peering Forums hosted by not-for-profit IXPs
o NOGs (which IXPs usually support)
• And, APNIC Foundation sponsors:
o PeeringDB
o IXP-DB
o IXP Manager

52. 81
81 v1.0
Final Remarks

53. 82 v1.0
82
Final Remarks
• IXPs will continue to play a key role for easier interconnections among
networks
o Especially for developing economies
o But IXP is NOT a magic wand to solve all the issues
▸ Collaborative spirit is
• Need to find a suitable model for long-term sustainability
• Relative neutrality is important
o So have to maintain it as much as possible
• After all, “Keeping Local Traffic Local” is the most important thing for the
whole country/economy

54. 83
83 v1.0
Q&A

55. 84 v1.0
84