This document discusses the problem of bufferbloat on the internet. It explains that bufferbloat occurs when routers and switches have too much buffering capacity, which leads to increased latency. The causes of bufferbloat include increased bandwidth usage from applications, large TCP initial windows, and routers being configured to avoid dropping packets. Solutions presented are queue management techniques like CoDel and PIE that aim to control latency while maintaining high throughput. The document notes that while awareness of bufferbloat has increased, legacy equipment and congestion on backbones continue to cause issues.

1. BufferbloatBufferbloat
Why is the Internet slow?Why is the Internet slow?
Stephen HemmingerStephen Hemminger
stephen@networkplumber.orgstephen@networkplumber.org
Limelight Networks May 2015

3. BufferbloatBufferbloat
● What is the problemWhat is the problem
● What causes the problemWhat causes the problem
● SolutionsSolutions
– DemonstrationDemonstration
● Current statusCurrent status
– The GoodThe Good
– The BadThe Bad
– The UglyThe Ugly

4. Network PerformanceNetwork Performance
Throughput
Mbits/sec
Latency
Round Trip Time
Fairness

5. How bad is it?How bad is it?

6. Why is worse now?Why is worse now?
● No longer in FTP universeNo longer in FTP universe
● TCP Initial Window Size = 10TCP Initial Window Size = 10
● Advanced TCP algorithmsAdvanced TCP algorithms
● Network Offload packet trainsNetwork Offload packet trains
● Bad benchmarksBad benchmarks
● Router/switch memory is cheapRouter/switch memory is cheap
● Everyone fears dropping packetsEveryone fears dropping packets

7. Head of Line blockingHead of Line blocking

8. Queuing Theory BasicsQueuing Theory Basics
averagetimeinqueue
utilization
servicerate
1 utilization
Graphic courtesy Sprint, Apricot 2004
From Fred Baker: Bufferbloat!

9. Netalyzr: DownstreamNetalyzr: Downstream

10. Netalyzr: UpstreamNetalyzr: Upstream

12. TCP throughput dynamicsTCP throughput dynamicsIncreasingMeasurableThroughput
Increasing TCP Window
“knee” “cliff”
Bottleneck Capacity
Queue
Depth
meanthroughput
effectivewindow
meanroundtriptime
From Fred Baker: Bufferbloat!

13. Blame LinuxBlame Linux
● Windows XPWindows XP
– Maximum window 64kMaximum window 64k
● Windows 7Windows 7
– Bandwidth limit to 80 mbitsBandwidth limit to 80 mbits
● AndroidAndroid
– Receive window limitedReceive window limited

14. Blame the customerBlame the customer
● Customers callCustomers call
supportsupport
● Applications areApplications are
using moreusing more
bandwidthbandwidth
● Block and chargeBlock and charge

16. Why Queueing?Why Queueing?

17. Priority Queue: TheoryPriority Queue: Theory

18. Priority Queue: RealityPriority Queue: Reality

19. Hierarchical Token BucketHierarchical Token Bucket

20. Random Early DetectRandom Early Detect

21. Explicit Congestion NotificationExplicit Congestion Notification
PACKETS

22. Ideal Active Queue ManagementIdeal Active Queue Management
● FairFair
– All flows get some bandwidthAll flows get some bandwidth
● SimpleSimple
– No tuningNo tuning
● Easy to deployEasy to deploy
– No special hardware, no protocol changesNo special hardware, no protocol changes
● ReasonableReasonable
– Won't create multi-second latencyWon't create multi-second latency

23. Stocastic Fair QueueStocastic Fair Queue

24. CodelCodel

25. PIEPIE

26. CakeCake
Priority
Fair Queue
CoDel

27. Back to Reality

28. Benchmark issuesBenchmark issues
● BadBad
– Bytes/secBytes/sec
– Packes/secPackes/sec
– LatencyLatency
● GoodGood
– Throughput + LatencyThroughput + Latency
– Multiple connectionsMultiple connections
– Real not simulatedReal not simulated

29. Better toolsBetter tools
● DSLreport'sDSLreport's
– http://www.dslreports.com/speedtest/http://www.dslreports.com/speedtest/
● ICSI netalyzrICSI netalyzr
– http://netalyzr.icsi.berkeley.edu/http://netalyzr.icsi.berkeley.edu/
● Real time Response Under LoadReal time Response Under Load
– https://github.com/tohojo/netperf-wrapperhttps://github.com/tohojo/netperf-wrapper
– Flent: The Flexible Network TesterFlent: The Flexible Network Tester

30. DSLReports speedtestDSLReports speedtest
5 sec delay!!

31. FIFO – the defaultFIFO – the default
Høiland-Jørgensen T., Battling Bufferbloat
Ping 1sec!
Upload
Download

32. Stochastic Fair QueueStochastic Fair Queue
Høiland-Jørgensen T., Battling Bufferbloat

33. Controlled Delay - codelControlled Delay - codel
Høiland-Jørgensen T., Battling Bufferbloat

34. Fair Queue Controlled DelayFair Queue Controlled Delay
Høiland-Jørgensen T., Battling Bufferbloat

35. Bufferbloat statusBufferbloat status

36. Bufferbloat: The GoodBufferbloat: The Good
● Industry awarenessIndustry awareness
● Standards progressStandards progress
● Active researchActive research
● Open Source SolutionsOpen Source Solutions

37. Once you start looking,Once you start looking,
Bufferbloat can be everywhereBufferbloat can be everywhere
● EdgeEdge
– Home routerHome router
● ProviderProvider
● NetworkNetwork
● AcceleratorsAccelerators
– CachesCaches
– Load balancers,...Load balancers,...
● ServersServers

38. How is Bufferbloat Solved on Linux?How is Bufferbloat Solved on Linux?
● Queuing disciplinesQueuing disciplines
– Codel, PIE, ...Codel, PIE, ...
● Linux internalLinux internal
– BQL, FQ, PacingBQL, FQ, Pacing
● Enable good defaultsEnable good defaults
– ecnecn
– fq_codelfq_codel

39. Bufferbloat: the BadBufferbloat: the Bad
● Legacy equipmentLegacy equipment
– OEM focus on cost not AQMOEM focus on cost not AQM
● Congestion on backboneCongestion on backbone
– It's not a bug it's a feature?It's not a bug it's a feature?
– Problem is political not technicalProblem is political not technical

40. Bufferbloat: the UglyBufferbloat: the Ugly
● WirelessWireless
– AggregationAggregation
– MulticastMulticast
– Bad benchmarksBad benchmarks
– Proprietary firmwareProprietary firmware

41. Questions?

42. Thank you
Stephen Hemminger
@networkplumber
stephen@networkplumber.org

43. Bufferbloat resources
Bufferbloat.net: http://bufferbloat.net
Email Lists: http://lists.bufferbloat.net
CeroWrt: http://www.bufferbloat.net/projects/cerowrt
Other talks: http://mirrors.bufferbloat.net/Talks
Jim Gettys Blog – http://gettys.wordpress.com
A big thanks to the bloat mailing list, Jim, Kathie, Van, Dave,
Eric, ISC, and all the other the CeroWrt/OpenWrt contributors