Technical Debt: An Anycast Story discusses Cloudflare's use of anycast routing to direct traffic to their global network of data centers. The document summarizes how Cloudflare resolved issues from technical debt in their anycast configuration by automating configuration changes across their entire network using SaltStack in under two minutes with minimal customer impact. It describes their extensive internal and external monitoring using tools like Prometheus, Grafana, RIPE Atlas and looking glasses to track the effects of the global configuration change.

1. Technical Debt: An Anycast Story
Mircea Ulinic
Cloudflare, London
APRICOT 2019
Daejeon, South Korea

2. Mircea Ulinic
● Network software engineer at Cloudflare
● Member and maintainer at NAPALM Automation
● SaltStack contributor of the year 2017
● O’Reilly author
● OpenConfig representative
● https://mirceaulinic.net/
mirceaulinic
@mirceaulinic
2

3. ● How big?
○ 7+ million zones/domains
○ 100+ billion DNS queries/day
■ Largest
■ Fastest
■ 35% of the Internet queries
■ Now also a resolver (1.1.1.1)
○ 10% of the web requests
● 150+ anycast locations globally
○ 74 countries (and growing)
○ Many hundreds of network devices
Cloudflare
3

4. ● Anycast introduction
● Our technical debt
● Configuration changes using Saltstack
● Change monitoring
Agenda
4

5. 5

6. Our Anycast Network
● ± 250 IPv4 prefixes
● ± 15 IPv6 prefixes
● Announced globally (150+ locations)
6

7. 7

8. Technical Debt
8

9. Technical Debt
● Few Tier 1 transit providers
● Prepends to steer traffic to proper location (± 10 PoPs)
● Eventually normalized globally
History
9

10. Technical Debt
Issues
10

11. Technical Debt
● Authoritative DNS targeted (eg. 173.245.58.0/24)
● Single location attracts all traffic due to missing prepends
Incidents
11

12. Technical Debt
● RPKI
● Shorter AS-path
● /24 everything
Solutions
12

13. Technical Debt
● Staggered deployment (6 stages)
● As quickly as possible globally
● Extensive internal and external monitoring
Resolution
13

14. Technical Debt
Change
14

15. Global Rollout
15

16. Global Rollout
● Automation and orchestration
● Open source
● Python, Jinja2 & YAML
● Highly scalable
● Very fast
● Vendor neutral
● Across our fleet: servers and network equipment
Saltstack
16

17. Global Rollout
● Make sure we know what we’re changing
● Adjust configuration if needed
● Add confidence
Prechecks
17

18. Global Rollout
Actual Change
18
● All in Python (config generation)
● Both Junos and EOS
● Concurrently
● Done globally within
± 2 minutes

19. Global Rollout
● Internal metrics
● External metrics
Metrics
19

20. Global Rollout
● Stored in Clickhouse or Prometheus
● Visualized with Grafana
● Flows
● SNMP data
● Request data
Internal metrics
20

21. Global Rollout
● Developed at Yandex
● Column-oriented DBMS
● Open source
● 3 PB on disk
● 100 Gbps insertion
Clickhouse
21

22. Global Rollout
● Stores flow data
● Stores request data
Clickhouse
22
Clickhouse query
Result

23. Global Rollout
● Time-series database
● Monitoring platform
● Open source
Prometheus
23

24. Global Rollout
● Prepend length
● PromQL
Prometheus
24
Begin rollout
Rollout complete
All external looking-glasses

25. Global Rollout
● Analytics
● Time-series visualization
● Multitude of plugins
● Open source
Grafana
25

26. Internal Metrics
● RPS / colo
● Traffic shift during change
● Real-time information
26
Global Rollout

27. Global Rollout
● Stored in Prometheus or raw ingestion
● Visualized with Grafana
● RIPE Atlas
● Looking glasses
External metrics
27

28. Global Rollout
● Global probes
● Ping, Traceroute, DNS query
● REST API
● Determine routing before and after change
RIPE Atlas
28

29. Global Rollout
● Routeviews
● AS57335 (http://dfz.watch/) looking glasses (Thanks Aaron!)
● IX looking glasses (We need more! With APIs!)
● Collect / scrape into Prometheus
● Visualize with Grafana
Looking Glasses
29

30. Global Rollout
● Scrape metrics
● BeautifulSoup for scraping
● Aggregate data
Looking Glasses
30

31. Global Rollout
● Insert into Prometheus
● python_client
Looking Glasses
31

32. Global Rollout
Looking Glasses
32
● Grafana visualization
● Track change in
near-real-time

33. Global Rollout
Combined
33
● Traffic
● Prepends
Perturbance
Delay

34. Global Rollout
Takeaways
34

35. Global Rollout
● Negligible customer impact
● Route fluctuations for ± 2 minutes
● Took 1 hour to complete change, 2 days to prepare
● Instantly detected and resolved minor issues
● Heavily reliant on open source tooling and community
Takeaways
35

36. Questions
?
mircea@cloudflare.com
36