Building your own anycast network allows you to make your services faster and more reliable. The key steps are to get IP addresses and autonomous system numbers, set up points of presence in data centers around the world, establish BGP routing, and monitor performance to optimize traffic routing. Issues like suboptimal routing due to BGP or outages from route flaps require working with internet service providers and active management of the network.

1. How to build How to build
your own anycastyour own anycast
serviceservice
(and why)(and why)
Marian MarinovMarian Marinov
mm@siteground.commm@siteground.com
Chief System ArchitectChief System Architect
Head of the DevOps departmentHead of the DevOps department

2. First let's start with First let's start with
what are what are
Anycast and BGP :)Anycast and BGP :)

3. ● In IP there are a few types of IP requests:
– unicast - IP to IP
– broadcast - IP to the whole prefix
(IPv6 doesn't have broadcast)
– multicast - IP to multiple Ips
– anycast - IP to the closest instance of IP

4. Unicast

5. Broadcast

6. Multicast

7. Anycast
AS13335
1.1.1.1
AS13335
1.1.1.1

8. How does your ISP finds, to How does your ISP finds, to
which instance to connect?which instance to connect?
route: 1.1.1.0/24
origin: AS13335
via 12.12.1.2 on eth4.21 [cogent 2018-04-10 20:58:00]
BGP.as_path: 174 13335 13335
via 13.13.1.2 on eth4.56 [delta 2018-04-13 17:22:11]
BGP.as_path: 197216 31287 13335 13335
via 14.14.1.2 on eth4.3565 [he_net 2018-04-13 17:22:02]
BGP.as_path: 6939 13335 13335
via 15.15.1.2 on eth4.356 [marla 2018-04-10 20:58:01
from 15.15.1.3]
BGP.as_path: 174 13335 13335

9. Why would you build Why would you build
your own anycast?your own anycast?
● Building your own CDN
● Making your services faster for
local clients
● Reducing the attack surface for
your site and services and
increasing reliability

10. Budget
* Does Toni needs to know?
* My company gave me these $$$

11. Now, what's the plan?Now, what's the plan?
1. Get IPv4/v6 prefixes
2. Setup your PoPs
3. Get ASNs
4. Setup BGP sessions
5. Announce your prefix
6. Setup some services

12. How can I get IPs and ASN?How can I get IPs and ASN?
There are two ways to get those:
­ the LIR way
­ the sponsoring LIR way

14. the LIR waythe LIR way
* Register as a LIR
­ prepare all company documents
­ for RIPE and ARIN
­ LACNIC, Afrinic and APNIC
are considerably slower
­ around 2000 euro per year

15. the LIR waythe LIR way
* Request an IPv4 prefix
­ you will receive only one /22 prefix with
your first request
­ you have to request an ASN. You will need
the ASNs of your peering ISPs
­ get the ASNs of your datacenters
­ every request takes about a week to be
serviced

16. sponsoring LIR waysponsoring LIR way
* Find a sponsoring LIR
* Request ASN and IPv4/v6 prefixes

17. Choose PoPsChoose PoPs
* First analyze your current traffic
* Find your top 10 countries, sources
of traffic
* Find a DataCenter that will provide
you Vms/containers and give you BGP
* The most important part here is to
try to keep the list of suppliers
small

18. BGP SetupBGP Setup
1. Setup the VMs with your favourite
distribution
2. Install a BGP speaker
­ Quagga
­ BIRD
­ OpenBGPd
­ exabgpd
3. Prepare your announcements

19. BGP SetupBGP Setup
1. Announce one /24 prefix from one
DataCenter
2. Check if everything goes as
expected
3. Announce the same /24 prefix from
second location
4. Verify that everything works
correctly
5. Announce the prefix from all
locations

20. BGP SetupBGP Setup
* In some cases the ISP/DC would want
to setup the .1 address on their
router
* Announce to the ISP/DC the whole /24
prefix and also the /32 prefix that
you use to connect to that router

21. Monitoring of this setupMonitoring of this setup
* I advise you to setup Headers on all
web servers, that will tell you which
servers you actually hit
X­Location: EUX­Location: EU
* You can also use services like
­ https://tools.keycdn.com/ping
­ or RIPE­Atlas

22. BGP IssuesBGP Issues
* There will be inevitably BGP issues
like
­ traffic from North America ends up
in Mexico or Brazil
­ traffic from Asia ends up in Africa

23. BGP IssuesBGP Issues
* Work with your ISPs to setup
community routing
­ setup prepends for the upstreams of
your ISPs
­ but setup smaller prepends for the
local ISPs
* This can be the hardest part of all

24. BGP IssuesBGP Issues
* Monitor your BGP announces for flaps
­ route flaps can STOP some of your
anycast regions
­ also route flaps can lead to
extremely sub­optimal routing

25. Managing your PoPsManaging your PoPs
* My advise would be to use TincVPN
and connect all nodes to your office
­ use Tinc or any other Mesh VPN
­ then use the VPN network for
management
­ if possible, order the VMs with two
separate NICs

26. Managing your PoPsManaging your PoPs
* DNS
* NTP
* Web Content Delivery
­ keep in mind that flaps are a big
problem for any TCP based service,
such as HTTP/1.1 ­ HTTP/2
­ somehow you need to synchronize
the content between the nodes

27. Marian MarinovMarian Marinov
mm@siteground.commm@siteground.com