SGNOG5 2017

1. Jimmy Lim SGNOG 5
jhalim@cloudflare.com Singapore, 5 September 2017
Managing Global Distributed Network

2. Cloudflare in a glance

3. This is the new style.
● 4+ million zones/domains
● 43+ billion DNS queries/day
● How?
○ Orange cloud
○ Global distributed network
in 120+ locations
Still growing fast!
○ Anycast routing
Protect and accelerate any website online

4. Benefit of orange cloud
● Direct visitors to the nearest entry point
○ Fast!
■ Lesser hops
■ Reduced latency
● Save bandwidth!
○ Lesser requests to origin
○ Mitigate DDoS
● Resiliency
○ 120+ locations!

5. Orange cloud vs grey cloud

6. Building like crazy

7. 1 new PoP per week!

8. Challenges
● 120+ locations
● 500+ transit/exchange ports
● 500+ network equipments
● Countless alerts!

9. Managing 120+ PoPs

10. Building Resilience Network
● Stable hardware and software
● Automatic configuration template/peer review
● Solid monitoring system
● Global network engineering
● Network automation

11. Hardware and Software
● Proper evaluation and testing
○ Fits requirement
○ Bugs free
○ Scalable
● Global standardization
○ Same models of hardware
○ Same software codes
● No mass software upgrade!
○ Small PoP first
○ Deploy in batches

12. Solid Monitoring System
● Reduced unwanted alerts
○ Only gets relevant alerts
○ Silence PoP/ports during maintenances
● Monitor the performance of transit providers
○ Detects packet loss on their backbone
○ Provides automatic related traceroutes
○ Actions based on severity
■ Disabling the PoP automatically
■ Disabling traffic on related transit provider automatically
■ Suggests on actions to do

13. Alerts Channel and Dashboard

14. Alerts Channel and Dashboard

15. Global Network Engineering
● Follow the sun approach
○ San Francisco -> Singapore -> London -> San Francisco
● Doing all stuffs
○ Technical operations
○ Network engineering
○ Network expansion projects
○ New PoPs deployment
○ Peering stuffs
● Very fast response to network issues and escalation

16. Network Automation
● Open source recipe: napalm-salt
Apricot'2017 Network Automation by Salt+Napalm by Mircea Ulinic

17. Some daily usage examples
● salt [target_device] net.cli "junos_show_command"
● salt-run net.find [target_device]
● salt-run bgp.neighbors [bgp_asn]
● salt [target_device] [anycast.disable | anycast.enable]
● salt [target_device] [transit.disable | transit.enable]
[transit_name]

18. Net finder salt runner example

19. BGP salt runner example

20. Get advertised routes to particular BGP peer

21. PeeringDB lookup on IXPs info

22. BGP peering configuration push

23. Global configuration push
● salt [target_device] net.load_template [config_template]
○ Configuration template with Jinja2 format
○ Dynamic configuration loader
■ Flexible
○ Allow dry-run
○ Example: salt -N junos-cdn-edges net.load_template test-lar

24. Configuration template file

25. Global configuration push using template

26. Peering Update in Asia

27. In August 2015
● AKL-IX (Auckland)
● APE (Auckland)
● BBIX (Tokyo, Osaka, Singapore)
● Equinix (Hong Kong, Osaka, Singapore, Sydney, Tokyo)
● HKIX (Hong Kong)
● JPIX (Tokyo, Osaka)
● JPNAP (Tokyo, Osaka)
● Megaport (Auckland, Singapore, Sydney)
● PIPE (Sydney)

28. Addition of Peering Exchange as for now
● AMS-IX (Hong Kong)
● KINX (Incheon)
● MYIX (Kuala Lumpur)
● NSW-IX (Sydney)
● SGIX (Singapore)
● VIC-IX (Melbourne)
● WAIX (Perth)

29. Globally
● Registered in 150 public peering exchange points
● About 120 of them are up and running!
● More details: https://www.peeringdb.com/asn/13335
● On-going 100G deployment
● Peering Contact: peering@cloudflare.com

30. Q&A