The physical, logical, and economic topology of the Internet is ever changing due to new network to network interconnections, the planned or accidental disconnection of existing interconnections, and the ever growing needs for more network capacity. As a multi-Terabit traffic source, Fastly must regularly evaluate its needs for network capacity against performance and operating costs. This talk will examine the capacity planning models, scalability concerns, and economic drivers used to make key business decisions regarding connectivity, emphasizing the needs for metrics collection, analysis, and constant iterative tuning involving a distributed, international team.

1. Name of Presentation
Iterative Traffic Engineering
in Changing Internet Economics
Tom Daly
VP, Infrastructure
tjd@fastly.com

2. Name of Presentation
• Provide a framework to make complex bandwidth
purchasing decisions:
– Business needs
– Traffic throughput and performance
– Direct cost management and cost optimization
– Staff size and cost
• Interest in:
– Internet Economics Megatrends
– Disruptive Events
– Metrics and Tools
• Creation of a long term bandwidth acquisition strategy
Goals

3. Name of Presentation
• Industry experience: 15+ years of datacenter and connectivity
acquisition deals
• I work for a CDN: Fastly
– We use a lot of bandwidth
– Therefore, we acquire much bandwidth
– Partnerships with Sales / Deal Desk, Finance, Marketing, and
Engineering
• Internet T/E techniques: DNS, BGP, and IP Anycast
– And related measurement techniques: Performance Monitoring, Real
User Monitoring (RUM), TCP Flow Analysis
• Education:
– BS ECE – Worcester Polytechnic Institute
– MBA – Bentley University
Presenter Qualifications

4. Name of Presentation
• Pricing:
– Don’t ask for pricing at volume; I honor non-disclosure
agreements.
– IP Transit costs are normalized to USD$10/Mbps/MRC for
services delivered in carrier neutral datacenter.
– IP Peering costs are normalized to USD $5000/10G Port/MRC.
• Net Neutrality:
– I/My employer have a duty to deliver content bits on behalf of our
customers in an performance optimal and efficient manner.
• Opinions and views expressed are my own; not the
viewpoint of Fastly, Inc.
Assumptions

5. Name of Presentation
• IP Transit: Paid access to the full Internet routing table.
• Peering: Access to a subset of Internet routes via agreed upon
interconnection.
– Settlement Free Peering: Free and balanced.
– Paid Peering: Dominant party pays
• Internet Exchange Point (IXP): Peering across a shared Layer 2
switch domain
• Cloud Interconnect Service: Peering across an allocated Layer 2
switch domain
• Private Network Interconnect (PNI): Peering across a physical
circuit
Nomenclature

6. Name of Presentation
The Challenge: Growth and Scale

7. Name of Presentation
The Challenge: Growth and Scale

8. Name of Presentation
The Challenge: Growth and Scale
$$$

9. Name of Presentation
• 10Gb Ethernet is mainstream; 100GbE is accessible
• Net Neutrality rulings in US and EU
– Highly varying benefits and downside between US and EU
• The “dumb” pipes problem: Transport and IP services
increasingly commoditized
• Resultantly; the global IP Transit market compresses 16-
18% per year, depending on region
• Movements to private datacenter/cloud interconnection
• Massive improvements to measurement and monitoring
Industry Shifting Economic Trends

10. Name of Presentation
Capitalize the Open Market Economy

11. Name of Presentation
Question: Transit vs. Peering?

12. Name of Presentation
Phase 1: Transit

13. Name of Presentation
Transit Pricing at Scale

14. Name of Presentation
Transit Pricing at Scale
“Yolo…just go peer some traffic”

15. Name of Presentation
Phase 2: Transit + Some Offload
IP Transit
Peering
Cloud Interconnect

16. Name of Presentation
Peering Offload Modeling (20Gbps)

17. Name of Presentation
Peering Offload Modeling (20Gbps)

18. Name of Presentation
Transit Pricing – Compressing 17% YoY

19. Name of Presentation
Transit Pricing – Compressing 17% YoY
No consideration for
peering staff overhead or
peering externalities

20. Name of Presentation
Transit Pricing – Compressing 17% YoY
No consideration for
peering staff overhead or
peering externalities
2x Peering Coordinators
3x NOC
---------------------------------
$425,000/yr + Benefits

21. Name of Presentation
Peering Path Connection Complexity

22. Name of Presentation
Peering Offload Modeling (40Gbps)

23. Name of Presentation
Peering Offload Modeling (40Gbps)

24. Name of Presentation
• IXP outages
• Congestion through IXP ports
• Unbalanced peers won’t upgrade PNIs
• “de-peering” between interconnected
networks
• Changes to routing and geo-location
degrade or improve performance
Disruptive Peering Events and Behaviors

25. Name of Presentation
Single Month Outage (40Gbps)

26. Name of Presentation
Question: Capacity Planning + Cost Models
What we know
The Future!

27. Name of Presentation
Regression Analysis - Excel

28. Name of Presentation
Regression Analysis – Peering Prediction

29. Name of Presentation
Scenario Analysis: Expected Peering Productivity

30. Name of Presentation
Question: 10Gb vs. 100Gb Ethernet
12x10Gb Ports
- 12x 10GBase-LR Optics:
$150 each = $1,800
- 12x SMF Cross Connects:
$350/MRC = $50,400/ARC
- Total: $52,200/1 YR

31. Name of Presentation
100GbE Scenario
12x10Gb Ports
- 12x 10GBase-LR Optics:
$150 each = $1,800
- 12x SMF Cross Connects:
$350/MRC = $50,400/ARC
- Total: $52,200/1 YR
- Consider line card ports 2x100Gb Ports
- 2x 100GBase-LR4 Optics:
$5600 each = $11,200
- 2x SMF Cross Connects:
$350/MRC = $8,400/ARC
- Total: $19,600/1 YR
- Consider line card ports

32. Name of Presentation
Eggs? Why are they in one basket?
12x10Gb Ports
- 12x 10GBase-LR Optics:
$150 each = $1,800
- 12x SMF Cross Connects:
$350/MRC = $50,400/ARC
- Total: $52,200/1 YR
2x100Gb Ports
- 2x 100GBase-LR4 Optics:
$5600 each = $11,200
- 2x SMF Cross Connects:
$350/MRC = $8,400/ARC
- Total: $19,600/1 YR
Reliability Modeling?
- Afford to loose 10G at a time?
- Afford to loose 100G at time?

33. Name of Presentation
An incomplete story…
Performance Matters...
Volumetric cost improvements…

34. Name of Presentation
Latency and Bandwidth Delay Product

35. Name of Presentation
ms and BDP: Doubled Packet Loss

36. Name of Presentation
ms and BDP: Halved Latency

37. Name of Presentation
“You can't manage what
you can't measure.”
– Peter F. Drucker
Social Ecologist;
Management Consultant;
Professor

38. Name of Presentation
Detailed Measurements
• Synthetic Agents: Latency, packet loss, and
path validation
• RUM: Geolocation, Recursive DNS divergence
and positioning, /latency/
• NetFlow / sFlow: Bandwidth by Remote ASN
and Next-Hop Peer
• Linux TCPExt: Retransmission and Packet Loss
• ICMP Ping: Position confirmation; triangulation

39. Name of Presentation
RIPE Atlas – https://atlas.ripe.net

40. Name of Presentation
Real User Monitoring (RUM)

41. Name of Presentation
NetFlow / sFlow Analysis

42. Name of Presentation
Linux TCP Extensions

43. Name of Presentation
ICMP Ping – “Fastly Style”

44. Name of Presentation
Methodology and Take Away(s)

45. Name of Presentation
• Early Stage: Just buy transit!
– Too small and too inefficient to peer
– Work on tooling to help you peer later by
analyzing traffic patterns
– Automate automate automate your routine
peering and monitoring workloads
Iterative Traffic Engineering (1)

46. Name of Presentation
• Mid Stage: Identify growth plans and potential
– Regression analysis across 18-24 months
data
– Identify and connect strategic / high value
peers
– Use cost or performance scenario analysis
– Note: Not all networks are good peers; see
risks
Iterative Traffic Engineering (2)

47. Name of Presentation
• Ongoing: Constant evolution
– Transit providers apprised of growth plan
– Contract duration adjusted to long term risk of market
price compression
– Peering load adjusted to team side
– Measure, measure, measure
Iterative Traffic Engineering (3)

48. Name of Presentation
Questions?
Tom Daly
VP, Infrastructure
tjd@fastly.com