Many of the most common uses of the Internet today weren’t envisioned when it was created. In many ways, the success of the Internet and the TCP/IP protocol once envisioned by DARPA is pushing it to the limits. As a result, ensuring high-performance for end-users is complicated. Join Cloudflare experts for a talk that will describe the depth of these problems -- ranging from how routing breaks, to how shortage of IP space (under IPv4) hurts performance, to route leaks -- and how these issues lead to congestion and poor performance. They'll also discuss an approach to solving these challenges given the constraints.

1. Unreliable Internet Nick Wondra @ Cloudflare
Martin J. Levy @ Cloudflare
November 2017

2. Todays agenda
● Introduction (Tim Fong)
○ Why does the Internet sometime “misbehave” when it
comes to delivering applications?
○ What are some ways to solve this?
● Martin J. Levy (20 min)
○ The Internet and how it’s tied together
○ BGP and topology
○ Testing (example of tools and techniques)
● Nick Wondra (20 min)
○ Approaches to solve the problem
○ Examples of mechanisms in place
● Summary (5 min)
● Audience Q/A (10 min)

3. The Internet and how it’s tied together
Martin J. Levy

4. The Internet
● Technically – a somewhat complex subject
○ The Internet is a collection of networks
○ No network stands alone (all interconnected)
○ Robustness can be created
○ Multi-homing (more than one transit/path)
○ Peering between “like” networks
○ Diversity (physical and logical)
○ Nothing is static!
● Internet was developed for something different
● Many types of data (and data layers)
● TCP/UDP vs FTP/HTTP/SMTP vs TLS vs XML/JSON

5. The Internet - just how complex? (hint: very!)
This is the representation of
a single network (a medium
sized telco) and its
interconnections globally to
various other backbone
networks.
A full diagram would have
upwards of 60,000
independent networks
depicted on a single
diagram, which is hard to
follow.

6. Glueing the Internet together - BGP routing
● The IETF specified a protocol (BGP4) that can handle:
○ Massive routing tables
○ CIDR routing (ability to specify IP network address plus a network size)
○ IPv4 & IPv6
○ Rules for routing internally within a network
○ Rules for routing to an external network
○ Much more!
● BGP in real-life is used by every network on the Internet
○ Every destination on the globe exists within the BGP global routing tables
○ Everything is public, visible, exposed, and recorded

7. What works? What breaks? What’s the fix?
● There’s no steady state within the Internet
○ The path from A to Z is forever changing. Sometime for the better
○ The BGP routing protocol has to address many factors:
■ Physical interruptions (a fiber break)
■ Planned maintenance (upgrades to facilities or services)
■ Increases in capabilities - for example, a new undersea cable
■ Third party “hiccups”
○ Commercial agreements (and disagreements)
■ Purchasing from a different Internet service provider
■ Ending contracts and changing service providers
● What we do know is that it keeps our network engineers busy!

8. The Internet - it keeps on growing
A new undersea is laid
between the African
coast and the
Seychelles island
(replacing a satellite
connection)

9. The Internet - When it breaks, it breaks!
This is an example of what happens each-and-every day all
around the globe. The physical layer of the Internet is fragile.
All those bright spray-painted lines you see on a street (before
someone digs it up) is meant to stop this from happening.
It doesn’t!

10. Protocol stack - what’s above the physical layer
● Layers provide capabilities
○ Application - the end-users view
○ Transport - HTTP & TLS
○ Internet - IP and routing
○ Data link - that fiber in the ground
● Each layer has its possible failures
Physical
Data Link
Network
Transport
Session
Presentation
Application
OSI Network Model
Data Link
Internet
Transport
Application
The TCP/IP Model

11. Distance, Latency, Variable Paths, and more
150 msec
70 msec
230 msec
400 msec
● Speed of light
○ Very constant![1]
● Distance ~= Hops
○ Reliability decreases
● Variable paths
○ Redundancy .vs.
○ Non-deterministic
● Variable providers
○ Sometimes useful
[1] https://www.quora.com/What-is-precisely-the-speed-of-light-in-fiber-optics

12. Monitoring tools and more
● Beyond ping - or what’s really happening to your packets?
[1] http://bgp.he.net/
[2] http://atlas.ripe.net/
[3] http://stat.ripe.net/

13. Approaches to solve the problem
Nick Wondra

14. Change the model of the Internet?
● Address the content, not the server
○ Content centric networking, et al
○ Route requests based on content location
○ Content is decentralized, moves through the network
● Requires changes deep in the protocol stack
○ … but lots of investment built into current infrastructure

15. Change the core Internet protocols?
● Can we build a better BGP?
○ Low-level distance and performance metrics may not translate to
application performance
○ Many networks = many systems to change
● Can we build a better transport?
○ TCP and UDP deeply ingrained in end-user systems and network
middleboxes (firewalls, LBs, WAN optimizers, etc)

16. ● Evolve new solutions on top of existing frameworks
○ Solve for problems in the malleable network layers
○ Example: TLS 1.3
■ More secure and faster (fewer RTTs)
○ Example: TCP+HTTP => UDP+QUIC
■ Speed: connection establishment, session multiplexing
■ Resilience: congestion control, forward error correction
■ Flexibility: connection migration
● The challenge is distribution
○ Clients and servers must opt-in
Evolution, not revolution

17. ● Cloudflare has Points of Presence (PoPs) across the globe
○ PoPs close to every Internet user and server
○ Transit/peering with multiple networks at every PoP
○ Proxies 10% of all web requests
● Global Internet performance and reliability monitoring
○ Real-time feedback as data traverses the network
○ Can “test” network paths that BGP wouldn’t use
○ Use performance metrics that matter to web applications (TTFB,
response time)
Value of a large global network

18. Global footprint = path control
● Force routing paths by pinning to intermediate PoPs
150ms
200ms

19. ● Evolution inside the network, transparent to client and server
Global footprint = distribution channel
TCP+HTTPTCP+HTTP UDP+QUIC

20. Summary
Nick Wondra & Martin J. Levy

21. Summary, Questions, and Thank You!
Martin J. Levy - Network Strategy
@Cloudflare
@mahtin
Nick Wondra - Systems Engineer
@Cloudflare
@nickwondra

22. Appendix

23. Additional Reading (via Cloudflare blog)
● Argo & Warp:
○ https://blog.cloudflare.com/argo/
○ https://blog.cloudflare.com/the-making-of-cloudflare-warp/
● Railgun:
○ https://blog.cloudflare.com/cacheing-the-uncacheable-cloudflares-railgun-
73454/
● Load Balancing:
○ https://blog.cloudflare.com/introducing-load-balancing-intelligent-failover-with-
cloudflare/
● TLS:
○ https://blog.cloudflare.com/introducing-tls-client-auth/