Presentation by Mircea Ulinic at APRICOT 2019 on Wednesday, 27 February 2019.

1. Technical Debt: An Anycast Story
Mircea Ulinic
Cloudflare, London
APRICOT 2019
Daejeon, South Korea

2. Mircea Ulinic
● Network software engineer at Cloudflare
● Member and maintainer at NAPALM Automation
● SaltStack contributor of the year 2017
● O’Reilly author
● OpenConfig representative
● https://mirceaulinic.net/
mirceaulinic
@mirceaulinic
2

3. ● How big?
○ 7+ million zones/domains
○ 100+ billion DNS queries/day
■ Largest
■ Fastest
■ 35% of the Internet queries
■ Now also a resolver (1.1.1.1)
○ 10% of the web requests
● 150+ anycast locations globally
○ 74 countries (and growing)
○ Many hundreds of network devices
Cloudflare
3

4. ● Anycast introduction
● Our technical debt
● Configuration changes using Saltstack
● Change monitoring
Agenda
4

5. 5

6. Our Anycast Network
● ± 250 IPv4 prefixes
● ± 15 IPv6 prefixes
● Announced globally (150+ locations)
6

7. 7

8. Technical Debt
8

9. Technical Debt
● Few Tier 1 transit providers
● Prepends to steer traffic to proper location (± 10 PoPs)
● Eventually normalized globally
History
9

10. Technical Debt
Issues
10

11. Technical Debt
● Authoritative DNS targeted (eg. 173.245.58.0/24)
● Single location attracts all traffic due to missing prepends
Incidents
11

12. Technical Debt
● RPKI
● Shorter AS-path
● /24 everything
Solutions
12

13. Technical Debt
● Staggered deployment (6 stages)
● As quickly as possible globally
● Extensive internal and external monitoring
Resolution
13

14. Technical Debt
Change
14

15. Global Rollout
15

16. Global Rollout
● Automation and orchestration
● Open source
● Python, Jinja2 & YAML
● Highly scalable
● Very fast
● Vendor neutral
● Across our fleet: servers and network equipment
Saltstack
16

17. Global Rollout
● Make sure we know what we’re changing
● Adjust configuration if needed
● Add confidence
Prechecks
17

18. Global Rollout
Actual Change
18
● All in Python (config generation)
● Both Junos and EOS
● Concurrently
● Done globally within
± 2 minutes

19. Global Rollout
● Internal metrics
● External metrics
Metrics
19

20. Global Rollout
● Stored in Clickhouse or Prometheus
● Visualized with Grafana
● Flows
● SNMP data
● Request data
Internal metrics
20

21. Global Rollout
● Developed at Yandex
● Column-oriented DBMS
● Open source
● 3 PB on disk
● 100 Gbps insertion
Clickhouse
21

22. Global Rollout
● Stores flow data
● Stores request data
Clickhouse
22
Clickhouse query
Result

23. Global Rollout
● Time-series database
● Monitoring platform
● Open source
Prometheus
23

24. Global Rollout
● Prepend length
● PromQL
Prometheus
24
Begin rollout
Rollout complete
All external looking-glasses

25. Global Rollout
● Analytics
● Time-series visualization
● Multitude of plugins
● Open source
Grafana
25

26. Internal Metrics
● RPS / colo
● Traffic shift during change
● Real-time information
26
Global Rollout

27. Global Rollout
● Stored in Prometheus or raw ingestion
● Visualized with Grafana
● RIPE Atlas
● Looking glasses
External metrics
27

28. Global Rollout
● Global probes
● Ping, Traceroute, DNS query
● REST API
● Determine routing before and after change
RIPE Atlas
28

29. Global Rollout
● Routeviews
● AS57335 (http://dfz.watch/) looking glasses (Thanks Aaron!)
● IX looking glasses (We need more! With APIs!)
● Collect / scrape into Prometheus
● Visualize with Grafana
Looking Glasses
29

30. Global Rollout
● Scrape metrics
● BeautifulSoup for scraping
● Aggregate data
Looking Glasses
30

31. Global Rollout
● Insert into Prometheus
● python_client
Looking Glasses
31

32. Global Rollout
Looking Glasses
32
● Grafana visualization
● Track change in
near-real-time

33. Global Rollout
Combined
33
● Traffic
● Prepends
Perturbance
Delay

34. Global Rollout
Takeaways
34

35. Global Rollout
● Negligible customer impact
● Route fluctuations for ± 2 minutes
● Took 1 hour to complete change, 2 days to prepare
● Instantly detected and resolved minor issues
● Heavily reliant on open source tooling and community
Takeaways
35

36. Questions
?
mircea@cloudflare.com
36