Using SD-WAN over the internet required understanding latency variation on different paths, This presents results compairing various global city pairs and how latency varies, along with a comparison to AWS backbone performance for the same paths.

1. Project: Measuring
Internet Core Variability
Steve Garson, SD-WAN Experts
https://www,sd-wan-experts.com

2. Latency Variation and SD-WAN

3. Methodology: Tools used
TOOL DESCRIPTION
Cedexis • Cedexis Measures Internet Health
• Collects 14 billion RUM data points per day
• Data collection methodology
Catchpoint • Catchpoint is a leading monitoring system
• Data collection methodology
• We set up tests from last mile agents to Speedtest last mile servers
Speedtest • Ookla/Speedtest is a leading last mile performance tool
• Used Catchpoint to test against Speedtest servers

4. Methodology: Definition of “Response”
Response = Send + Wait
• Excludes all one-time setup/negotiations like DNS and Connect
• Response is a better measure of real Internet response than Ping
DNS SERVER
WEB SERVER
CLIENT CLIENT1. DNS Lookup 2. Connect 3. TLS 4. Send 5. Wait 6. Load
a b a b c a a b c
RESPONSE

5. Part 1
Calculate the Core

6. Methodology
Calculate CORE by subtracting the last mile
FIRST
MILE
MIDDLE
MILE
LAST
MILE
X
Y
Core
Response
Core
Response Long Haul
(Cedexis)
Response Last Mile
(Catchpoint/Speedtest) =-

7. Results
Long Haul (Cedexis) Last Mile (Catchpoint/Speedtest) Core
End User Location End User ISP Origin Server
Median
(ms)
SD
(ms)
Variance
(ms2)
Median
(ms)
SD
(ms)
Variance
(ms2)
Median
(ms)
SD
(ms)
Diff Variances
(ms2)
Bangalore Atria SJC AWS 224 125 15625 3 5.88 34.57 221 125 15590
Bangalore Atria London AWS 148 92 8464 3 5.88 34.57 145 92 8429
Bangalore Atria Tokyo AWS 119 99 9801 3 5.88 34.57 116 99 9766
Bangalore Atria Sydney AWS 295 106 11236 3 5.88 34.57 292 106 11201
DC Cox SJC AWS 101 119 14161 16 6.08 36.97 85 119 14124
DC Cox Tokyo AWS 186 112 12544 16 6.08 36.97 170 112 12507
DC Cox Sydney AWS 264 163 26569 16 6.08 36.97 248 163 26532
Tokyo Ucom SJC AWS 105 102 10404 4 7.46 55.65 101 102 10348
Tokyo Ucom London AWS 205 98 9604 4 7.46 55.65 201 98 9548
Tokyo Ucom Sydney AWS 191 68 4624 4 7.46 55.65 187 68 4568
London BT SJC AWS 174 123 15129 10 12 144.00 164 122 14985
London BT Tokyo AWS 271 160 25600 10 12 144.00 261 160 25456
London BT Sydney AWS 348 182 33124 10 12 144.00 338 182 32980
Melbourne Singtel/Optus SJC AWS 198 92 8464 6 5.2 27.04 192 92 8437
Melbourne Singtel/Optus London AWS 337 191 36481 6 5.2 27.04 331 191 36454
SF Comcast London AWS 166 134 17956 17 9.08 82.45 149 134 17874
SF Comcast Virginia AWS 89 103 10609 17 9.08 82.45 72 103 10527

8. Why the Problem in the Middle Mile?
• $$$ 20X* investment
• paid for by customers
• $ 6X* investment
• least cost peering and routing
• $$$ 50X* investment
• paid for by customers
FIRST
MILE
LAST
MILE
* Source: Akamai
Telia
Carrier
GTT
LEVEL 3

9. Example: Core & Last Mile Traceroutes
FIRST
MILE
MIDDLE
MILE
LAST
MILE

10. Part 2
Measure the Core

11. Methodology
Directly measure CORE and compare
FIRST
MILE
MIDDLE
MILE
LAST
MILE
Response Core
(Catchpoint)

12. Middle Mile (Core) Median (ms) SD (ms)
City Backbone Origin Server Part 1 Part 2 Part 1 Part 2
Bangalore Tata SJC AWS 221 224 125 153
Bangalore Tata London AWS 145 153 92 50
Bangalore Tata Tokyo AWS 116 114 99 66
Bangalore Tata Sydney AWS 292 300 106 154
DC Comcast SJC AWS 85 74 119 35
DC Comcast Tokyo AWS 170 191 112 118
DC Comcast Sydney AWS 248 250 163 346
Tokyo NTT SJC AWS 101 109 102 70
Tokyo NTT London AWS 201 230 98 81
Tokyo NTT Sydney AWS 187 110 68 84
London BT SJC AWS 164 181 122 228
London BT Tokyo AWS 261 271 160 273
London BT Sydney AWS 338 336 182 346
Melbourne Telstra SJC AWS 192 168 92 9
Melbourne Telstra London AWS 331 307 191 1725
SF Level3 London AWS 149 145 134 68
SF Level3 Virginia AWS 72 76 103 71
Results
This one is interesting.
Telstra is doing something
special with SJC AWS

13. Part 3
Measure AWS to AWS

14. Methodology
Directly measure CORE and AWS to AWS on-net
FIRST
MILE
MIDDLE
MILE
LAST
MILE
Response Core
(Catchpoint)

15. Part 2 Backbone - AWS Part 3 AWS – AWS Part 2 Backbone – AWS Part 3 AWS - AWS
Backbone Agent AWS Agent Origin Server Median (ms) Median (ms) SD (ms) SD (ms)
Tata Mumbai AWS SJC AWS 224 240 152.86 6.18
Tata Mumbai AWS London AWS 153 113 50.13 3.63
Tata Mumbai AWS Tokyo AWS 114 121 65.78 4.41
Tata Mumbai AWS Sydney AWS 300 228 154.17 9.43
Comcast DC AWS SJC AWS 74 61 34.54 9.91
Comcast DC AWS Tokyo AWS 191 172 118.08 7.63
Comcast DC AWS Sydney AWS 250 205 345.50 5.22
NTT Tokyo AWS SJC AWS 109 113 69.81 6.70
NTT Tokyo AWS London AWS 230 247 81.13 6.19
NTT Tokyo AWS Sydney AWS 110 104 83.60 11.78
BT London AWS SJC AWS 181 137 227.80 10.51
BT London AWS Tokyo AWS 271 247 272.98 122.24
BT London AWS Sydney AWS 336 281 345.50 16.87
Telstra Sydney AWS SJC AWS 168 147 9.36 10.59
Telstra Sydney AWS London AWS 307 280 1725.37 16.77
Level3 SJC AWS London AWS 145 140 68.11 52.87
Level3 SJC AWS Virginia AWS 76 63 71.17 91.82
DC Azure SJC AWS 73 9.47
Results
Poor stability
AWS-to-AWS on-net
is significantly more
consistent/reliable
AWS-to-AWS on-net has better/lower latency

16. Conclusions
• Internet variability is most dramatic in the core
• On high latency paths, to provide the stable performance
needed by latency sensitive applications, use a private
network such as:
• SD-CORE accessed via internet VPN to local POPs, or
• MPLS, which requires fiber or wire connections
• SD-WAN using two internet connections can mitigate much
of the public internet variability, depending on overall path

17. Conclusions
• Understand that latency varies on the internet
• Look at your application requirements for latency and packet
loss
• If a median latency is 70ms with a SD of 34 ms is adequate
for your application performance, you don’t need MPLS with
two internet circuits.
• If median latency for a path is 307 ms with an SD 1,725 ms,
you might not want to depend purely on the internet
• There are differences in internet backbones.

18. Thank You
https://www.sd-wan-experts.com