This document discusses Cloudflare's global network and the process of building and managing over 80 points of presence (PoPs). It outlines Cloudflare's global distributed network and anycast routing system, and describes the benefits this architecture provides including performance, resiliency, and mitigating attacks. It then details the strategic planning, challenges, and solutions involved in continuously deploying new PoPs around the world at a rate of one per week.

1. Jimmy Halim IDNOG3
jhalim@cloudflare.com Jakarta, 28 July 2016
Building & Managing 80+ PoPs

2. Overview of CloudFlare

3. ● 4+ million zones/domains
● 43+ billion DNS queries/day
● How?
○ Orange cloud
○ Global distributed network
in 80+ locations
Still growing fast!
○ Anycast routing
Protect and accelerate any website online

4. Benefit of orange cloud
● Direct visitors to the nearest entry point
○ Fast!
■ Lesser hops
■ Reduced latency
■ Improved performance
● Save bandwidth!
○ Lesser requests to origin
■ Typically 50% of the resources on any given
web page are cacheable
○ Mitigate malicious visitors or DDoS
■ Stop them before get to the origin web server
● Resiliency
○ 80+ locations!

5. Grey cloud vs orange cloud

6. Building like crazy

7. 1 new PoP per week!

8. Strategic Planning
● Agreement/Negotiation
● Location
○ Peering Exchanges
○ Cost
○ Support
● Size
○ Traffic analysis
■ Number of Racks
■ Equipment types
■ Transits/Peering Exchanges
● How many?
● How big are the pipes?

9. Challenges
● Installation
○ Regulation
■ Import policy
○ Transits
■ Different carriers have different setup/policies
○ Language barriers
● Human factors
○ Configuration errors!
■ Anycast
● Traffic turnup
○ How to ensure it is not impacting
■ No outages please!

10. Solutions
● Out of band network is a must!
○ Acting as last resort
○ Upgrade/downgrade
○ Maintenances
● Configuration template
○ Auto configuration
■ Anycast!
○ Peer review
● Global Network Engineering
○ Round the clock deployment
■ Reduced bottleneck

11. Testing with providers
● Circuit testing
○ Point to point extended ping test
■ Test all physical ports
○ Failover Testing
■ Redundancy
● Do not create a blackhole instead!
● Use testing prefix
○ Global versus domestic
■ RIPE Atlas measurement
■ Public route servers
○ Good related BGP configuration
■ It does what is supposed to do

12. Traffic Turnup
● Do not send all prefixes at 1 go!
○ Start with few prefixes
○ Check the routing to these few prefixes
■ Global traffic analysis
● No big drop of traffic in other location
● Traffic comes from the right countries
○ Monitor for 24 hours
■ Confirms there are no anomalies observed
● On the new location
● Globally
○ Announce all prefixes
■ In batches
■ Repeat the same steps above!

13. Traffic Turnup
● Get the providers to be involved
○ Especially if it is a single homed
○ Inform them the schedule
■ Get them to understand what to expect
■ Troubleshoot and fix the problem faster!
○ Their users might be able to see problem faster

14. Managing 80+ PoPs

15. ● 80+ locations
● 500+ transit/exchange ports
● 500+ network equipments
● Uncountable alerts!
Challenges

16. Building Resilience Network
● Stable hardware and software
● Automatic configuration template/peer review
● Solid monitoring system
● Network automation
● Global network engineering

17. Hardware and Software
● Proper evaluation and testing
○ Fits requirement
○ Bugs free
○ Scalable
● Global standardization
○ Same models of hardware
○ Same software codes
● No mass software upgrade!
○ Small PoP first
○ Deploy in batches

18. Solid Monitoring System
● Reduced unwanted alerts
○ Only gets relevant alerts
○ Silence PoP/ports during maintenances
● Monitor the performance of transit providers
○ Detects packet loss on their backbone
○ Provides automatic related traceroutes
○ Actions based on severity
■ Disabling the PoP automatically
■ Disabling traffic on related transit provider automatically
■ Suggests on actions to do

19. Alerts Channel and Dashboard

20. Alerts Channel and Dashboard

21. Alerts Channel and Dashboard

22. Alerts Channel and Dashboard

23. Network Automation
● Open source recipe: napalm-salt
ripe72-NetworkAutomation-SaltandNAPALM-MirceaUlinic-CloudFlare

24. Network Automation | NAPALM-Salt (examples)
● salt "edge*" net.cli "show version"
● salt -G "os:junos" net.cli "show chassis hardware”
● salt -G "os:iosxr" net.arp
● salt-run net.find [target_device]
● salt-run net.find [mac_address]
● salt-run bgp.neighbors [bgp_asn]
● salt [target_device] [anycast.disable | anycast.enable]
● salt [target_device] [transit.disable | transit.enable]
[transit_name]

25. Network Automation | NAPALM-Salt (examples)

26. Global Network Engineering
● Follow the sun approach
○ San Francisco -> Singapore -> London -> San Francisco
● Doing all stuffs
○ Technical operations
○ Network engineering
○ Network expansion projects
○ New PoPs deployment
○ Peering stuffs
● Very fast response to network issues and escalation

27. Statistics

30. Indonesian’ Statistics

31. Q&A