1. Anycast all the things
Load balacing and redundancy in your network
FrOSCon 14 Network and Automation Track
Maximilian Wilhelm
1 / 35

2. Agenda
1. Who's who
2. Load balancing concepts
1. DNS Round robin
2. Layer 4-7 LBs
3. Anycast
3. How to Anycast
1. Requirements
2. Key elements
3. Routing protocols
4. Anycast in practice
5. Outlook
2 / 35

3. Who's who Maximilian Wilhelm
Networker
OpenSource Hacker
Fanboy of
(Debian) Linux
ifupdown2
Occupation:
By day: Senior Infrastructure Architect, Uni Paderborn
By night: Infrastructure Archmage, Freifunk Hochstift
In between: Freelance Solution Architect for hire
Contact
@BarbarossaTM
max@sdn.clinic
3 / 35

4. Who's who
Concepts
Load balancing concepts
4 / 35

5. Who's who
Concepts
Why load balancing?
Availability
Hardware will fail
Software bugs
Fat fingers
Maintenance
Scalability
Maybe users like your service (base line)
Christmas shopping (peaks)
5 / 35

6. Who's who
Concepts
Now what?
Availability
Make sure at least one service node is working
Scalability
Make sure enough service nodes are working
Users don't care about your infrastructure.
They care about their user experience.
6 / 35

7. Who's who
Concepts
How to achieve that?
DNS Round Robin
Load balancer appliance(s) / software
Anycast
A combination of those
7 / 35

8. Who's who
Concepts
Multiple A / AAAA entries
Balancing by DNS replies
pseudo randomly sorted by DNS
server
DNS Round Robin
8 / 35

9. Who's who
Concepts
DNS Round Robin
Pros
Easy to set up
Cons
All IPs have to be reachable
Maintenance is hard
Beware of stupid clients
Slow reaction times
Due to TTL of records
Broken caching in resolvers ignoring small TTLs
9 / 35

10. Who's who
Concepts
DNS Round Robin
Availabilty
Node failure might be noticed by users
DNS caching may prolong failures
Scalability
Add more DNS records
10 / 35

11. Who's who
Concepts
Some appliance (or cluster there of)
Terminate service IP(s) from clients
perspective
Connection to real backend via
NAT
TCP proxy
Application level proxy
(HAproxy, nginx, ...)
.21 .22 .23
.17
194.107.206.16/28
LB
194.107.206.8
Layer 4-7 Load balancers
11 / 35

12. Who's who
Concepts
Layer 4-7 Load balancers
Pros
LB appliance can hide complexity
May provide point and click management
LB can manipulate on a per-request basis
Cons
risk of vendor lock-in
LB may become bottle neck
LB config may become to complex to maintain
Backend systems may lose information about client IPs
12 / 35

13. Who's who
Concepts
Layer 4-7 Load balancers
Availabilty
Have LB cluster
Have multiple backends
Scalabilty
Add more backends
Maybe scale appliance vertically
13 / 35

14. Who's who
Concepts
Just a bunch of backend servers
have the service IP(s)
configured
On a virtual interface
Terminate service IP(s) from clients
perspective
.21
.22 .23
.17
194.107.206.16/28
194.107.
206.8
194.107.
206.8
194.107.
206.8
Anycast
14 / 35

15. Who's who
Concepts
Anycast
Pros
Scales horizontally at line-rate
Easy to set up (after losing fear)
Cons
No way to steer where client connection terminates
15 / 35

16. Who's who
Concepts
Anycast
Availabilty
Have multiple backends
Scalabilty
Have as much backends as needed and some spare
Have enough network bandwidth
16 / 35

17. Who's who
Concepts
Anycast
How to Anycast
17 / 35

18. Who's who
Concepts
Anycast
Requirements / Ingredients
At least two services nodes
A spare IP from a different subnet
Router which support ECMP
A dynamic routing protocol
A mechanism to check if a service node is healthy
18 / 35

19. Who's who
Concepts
Anycast
OSPF or IS-IS
Potentially more well-known
No (good) way to filter prefixes
BGP
Potentially less well-known
Potentially higher learning curve
Filtering is part of protocol design
Choice of routing protocol
19 / 35

20. Who's who
Concepts
Anycast
Service healthchecker
Each node has to determine it's health
Annouce service IP accordingly
Check definition should be present for monitoring anyway
Why not just reuse it locally?
20 / 35

21. Who's who
Concepts
Anycast
Practice Anycast in practice
21 / 35

22. Who's who
Concepts
Anycast
Practice
Three web servers
Debian Linux
nginx
simple static web page
One router
Cisco Catalyst 3750
Assuming BGP on router is
configured
.21
.22 .23
.17
194.107.206.16/28
Example
22 / 35

23. Who's who
Concepts
Anycast
Practice
Con gure Virtual Anycast IP
Set up a dummy interface on each webserver
The same IP everywhere
#
# /etc/network/interfaces (ifupdown2 style)
#
auto anycast_srv
iface anycast_srv
link-type dummy
#
address 194.107.206.8/32
Bring up the interface
ifup anycast_srv
23 / 35

24. Who's who
Concepts
Anycast
Practice
Setting up Bird
We'll use Bird Internet Routing Daemon
Simple basic setup up
Use regular host IPv4 IP as Router ID
Default protocol device
#
# /etc/bird/bird.conf
#
# Change this into your BIRD router ID. It's a world-wide unique identification
# of your router, usually one of router's IPv4 addresses.
router id 194.107.206.21;
# The Device protocol is not a real routing protocol. It doesn't generate any
# routes and it only serves as a module for getting information about network
# interfaces from the kernel.
protocol device {
}
See last year's FrOSCon OSPF and BGP talks for details
24 / 35

25. Who's who
Concepts
Anycast
Practice
Learn Anycast IP in bird
Extend config with a direct protocol
protocol direct anycast_srv {
interface "anycast_srv";
}
Reload bird configuration
birdc configure
Check bird routing table
root@srv01:~# birdc show route
BIRD 1.6.3 ready.
194.107.206.8/32 dev anycast_srv [anycast_srv 16:18:29] * (240)
25 / 35

26. Who's who
Concepts
Anycast
Practice
RUN BGP
Export Anycast IP to our router
protocol bgp dr_dc_01 {
import none;
export where net = 194.107.206.8/32;
local as 39225;
neighbor 194.107.206.17 as 39225;
}
Reload bird configuration
birdc configure
26 / 35

27. Who's who
Concepts
Anycast
Practice
RUN BGP
Check protocol status
root@srv01:~# birdc show protocol dr_dc_01
BIRD 1.6.3 ready.
name proto table state since info
dr_dc_01 BGP master up 01:11:35 Established
Check exported prefixes
root@srv01:~# birdc show route export dr_dc_01
BIRD 1.6.3 ready.
194.107.206.8/32 dev anycast_srv [anycast_srv 16:18:30] * (240)
27 / 35

28. Who's who
Concepts
Anycast
Practice
Setting up the health check
We'll use anycast-healtchecker
Setting up global config
# /etc/anycast-healthchecker/anycast-healthchecker.conf
[DEFAULT]
interface = anycast_srv
check_interval = 3
check_timeout = 2
check_disabled = false
on_disabled = withdraw
ip_check_disabled = false
[daemon]
ipv4 = true
ipv6 = true
bird_conf = /var/lib/anycast-healthchecker/anycast-prefixes-v4.conf
bird6_conf = /var/lib/anycast-healthchecker/anycast-prefixes-v6.conf
bird_variable = ANYCAST_PREFIXES
bird6_variable = ANYCAST_PREFIXES
[...]
dummy_ip_prefix = 10.189.200.255/32
dummy_ip6_prefix = 2001:db8::1/128
[...]
28 / 35

29. Who's who
Concepts
Anycast
Practice
Setting up the health check
Setting up the check for our web service (trivial version)
# /etc/anycast-healthchecker/check.d/webserver.conf
[webserver_v4]
check_cmd = wget -O/dev/null -q http://localhost/
ip_prefix = 194.107.206.8/32
Let's get it started
systemctl restart anycast-healthchecker.service
Check the bird config snippet
root@srv01:~# cat /var/lib/anycast-healthchecker/anycast-prefixes-v4.conf
# Generated 2019-08-10 01:30:37.375957 by anycast-healthchecker (pid=6825)
# 10.189.200.255/32 is a dummy IP Prefix. It should NOT be used and REMOVED [...]
define ANYCAST_PREFIXES =
[
10.189.200.255/32,
194.107.206.8/32
];
29 / 35

30. Who's who
Concepts
Anycast
Practice
Glueing it all together
include "/var/lib/anycast-healthchecker/anycast-prefixes-v4.conf";
protocol bgp dr_dc_01 {
import none;
export where net ~ ANYCAST_PREFIXES;
local as 39225;
neighbor 194.107.206.17 as 39225;
}
And again, reconfigure bird
birdc configure
Now what happens when nginx goes down?
30 / 35

31. Who's who
Concepts
Anycast
Practice
Outlook
Outlook
31 / 35

32. Who's who
Concepts
Anycast
Practice
Outlook
Why no anycasting LBs?
HAproxy for the win
Production set up at Uni PB
As well as pure Anycast for
Kerberos KDCs
.21 .22 .23
.17
194.107.206.16/28
LB
194.107.206.8
LB
194.107.206.8
Combining forces
32 / 35

33. Who's who
Concepts
Anycast
Practice
Outlook
Let's scale the shit out of it
There can be lot's of LBs and lot's of backends
All of this can be distributed (anycasted) over the globe
CloudFlare, Facebook, ...
Even with DNS-RR
Or more intelligent solutions
Every CDN out there
33 / 35

34. Who's who
Concepts
Anycast
Practice
Outlook
Links
Further Reading
BGP basics
https://blog.sdn.clinic/2018/09/froscon-13-network-track/
Anycast with Cisco Nexus 7000 and Debian Linux
https://blog.sdn.clinic/2018/02/anycasted-services-with-debian-bird-anycast-
healthchecker-and-cisco-nexus-7000/
34 / 35

35. Who's who
Concepts
Anycast
Practice
Outlook
Links
Questions?
Questions?
35 / 35