The document discusses a 5-year study of internet traffic from an ISP network. It measured over 247 billion flows from 15,000 households between 2013-2017. The key findings were: 1) Daily traffic consumption per household doubled from 2014 to 2017 for ADSL connections and increased even more for FTTH connections. 2) Heavy users accounted for a large portion of total traffic and increased their consumption over time. 3) The study was able to track usage of specific services like Facebook and Netflix by identifying domains in traffic flows.

1. Five Years at the Edge:
Watching Internet from the ISP Network
Martino Trevisan, Danilo Giordano, Idilio Drago,
Marco Mellia, Maurizio Munafò

2. Five Years at the Edge:
Watching Internet from the ISP Network
Martino Trevisan, Danilo Giordano, Idilio Drago,
Marco Mellia, Maurizio Munafò

3. The Goals
2
“The Internet is the first thing that humanity has
built that humanity doesn't understand, the largest
experiment in anarchy that we have ever had.”

4. The Goals
2
“The Internet is the first thing that humanity has
built that humanity doesn't understand, the largest
experiment in anarchy that we have ever had.”
Eric Schmidt – ex Google Exec. Chairman

5. The Questions
1. How much does the user cost?
3

6. The Questions
1. How much does the user cost?
2. How users' habits changed?
3

7. The Questions
1. How much does the user cost?
2. How users' habits changed?
3. How the protocols changed?
3

8. The Questions
1. How much does the user cost?
2. How users' habits changed?
3. How the protocols changed?
4. How the infrastructure changed?
3

9. Measurement Setup
4

10. Measurement Setup
5 k households
Download 100 Mbps
Upload 10 Mbps
10 k households
Download 4-20 Mbps
Upload 1 Mbps
4

11. Measurement Setup
5 k households
Download 100 Mbps
Upload 10 Mbps
10 k households
Download 4-20 Mbps
Upload 1 Mbps
4

12. Measurement Setup
5 k households
Download 100 Mbps
Upload 10 Mbps
10 k households
Download 4-20 Mbps
Upload 1 Mbps
4

13. Measurement Setup
5 k households
Download 100 Mbps
Upload 10 Mbps
10 k households
Download 4-20 Mbps
Upload 1 Mbps
4

14. Measurement Setup
5 k households
Download 100 Mbps
Upload 10 Mbps
10 k households
Download 4-20 Mbps
Upload 1 Mbps
5 Years 04/2013 – 10/2017
247 Billions flows
31.9 TB of compressed data
4
*
Data at https://smartdata.polito.it/five-years-at-the-edge-watching-internet-from-the-isp-network/

15. Measurement Setup
5 Years 04/2013 – 10/2017
247 Billions flows
31.9 TB of compressed data
4

16. Measurement Setup
5
• Process traffic in real-time
• Statistics computed at the end of the flow
5 Years 04/2013 – 10/2017
247 Billions flows
31.9 TB of compressed data

17. Measurement Setup
5
• Process traffic in real-time
• Statistics computed at the end of the flow
c_ip c_port c_pkts c_bytes s_ip s_port s_pkts s_bytes Protocol
12.132.54.94* 1197 1 18938 87.250.137.92 443 1 992221 HTTP
Tstat*Client IP address is anonymized
5 Years 04/2013 – 10/2017
247 Billions flows
31.9 TB of compressed data

18. TLS/SNI
Facebook.com
Measurement Setup
5
• Process traffic in real-time
• Statistics computed at the end of the flow
• Implements DN-Hunter
• To Track DNS conversations
• To tag the network flows with the FQDN
c_ip c_port c_pkts c_bytes s_ip s_port s_pkts s_bytes Protocol
12.132.54.94* 1197 1 18938 87.250.137.92 443 1 992221 HTTP
Tstat
FQDN
Facebook.com
DN-Hunter
*Client IP address is anonymized
HTTP Host:
--
5 Years 04/2013 – 10/2017
247 Billions flows
31.9 TB of compressed data

19. TLS/SNI
Facebook.com
Measurement Setup
5
• Process traffic in real-time
• Statistics computed at the end of the flow
• Implements DN-Hunter
• To Track DNS conversations
• To tag the network flows with the FQDN
c_ip c_port c_pkts c_bytes s_ip s_port s_pkts s_bytes Protocol
12.132.54.94* 1197 1 18938 87.250.137.92 443 1 992221 HTTP
Tstat
FQDN
Facebook.com
DN-Hunter
*Client IP address is anonymized
HTTP Host:
--
Jon Postel – RFC 791
“A name indicates what we seek.
An address indicates where it is.
A route indicates how we get there.”
5 Years 04/2013 – 10/2017
247 Billions flows
31.9 TB of compressed data

20. TLS/SNI
Facebook.com
Measurement Setup
5
• Process traffic in real-time
• Statistics computed at the end of the flow
• Implements DN-Hunter
• To Track DNS conversations
• To tag the network flows with the FQDN
c_ip c_port c_pkts c_bytes s_ip s_port s_pkts s_bytes Protocol
12.132.54.94* 1197 1 18938 87.250.137.92 443 1 992221 HTTP
Tstat
FQDN
Facebook.com
DN-Hunter
*Client IP address is anonymized
HTTP Host:
--
When FQDN
Current facebook.com, fbcdn.com
Before fbstatic-a.akamaihd.net
Jon Postel – RFC 791
“A name indicates what we seek.
An address indicates where it is.
A route indicates how we get there.”
5 Years 04/2013 – 10/2017
247 Billions flows
31.9 TB of compressed data

21. TLS/SNI
Facebook.com
Measurement Setup
5
• Process traffic in real-time
• Statistics computed at the end of the flow
• Implements DN-Hunter
• To Track DNS conversations
• To tag the network flows with the FQDN
c_ip c_port c_pkts c_bytes s_ip s_port s_pkts s_bytes Protocol
12.132.54.94* 1197 1 18938 87.250.137.92 443 1 992221 HTTP
Tstat
FQDN
Facebook.com
DN-Hunter
*Client IP address is anonymized
*
Full list at https://smartdata.polito.it/five-years-at-the-edge-watching-internet-from-the-isp-network/
HTTP Host:
--
When FQDN
Current facebook.com, fbcdn.com
Before fbstatic-a.akamaihd.net
Jon Postel – RFC 791
“A name indicates what we seek.
An address indicates where it is.
A route indicates how we get there.”
Regexp* Service
facebook.com, fbcdn.com,
ˆfbstatic-[a-z].akamaihd.net$
Facebook
netflix.com, nflxvideo.net Netflix
5 Years 04/2013 – 10/2017
247 Billions flows
31.9 TB of compressed data

22. TLS/SNI
Facebook.com
Measurement Setup
5
• Process traffic in real-time
• Statistics computed at the end of the flow
• Implements DN-Hunter
• To Track DNS conversations
• To tag the network flows with the FQDN
c_ip c_port c_pkts c_bytes s_ip s_port s_pkts s_bytes Protocol
12.132.54.94* 1197 1 18938 87.250.137.92 443 1 992221 HTTP
Tstat
FQDN
Facebook.com
DN-Hunter
*Client IP address is anonymized
*
Full list at https://smartdata.polito.it/five-years-at-the-edge-watching-internet-from-the-isp-network/
HTTP Host:
--
When FQDN
Current facebook.com, fbcdn.com
Before fbstatic-a.akamaihd.net
Regexp* Service
facebook.com, fbcdn.com,
ˆfbstatic-[a-z].akamaihd.net$
Facebook
netflix.com, nflxvideo.net Netflix
5 Years 04/2013 – 10/2017
247 Billions flows
31.9 TB of compressed data

23. TLS/SNI
Facebook.com
Measurement Setup
5
• Process traffic in real-time
• Statistics computed at the end of the flow
• Implements DN-Hunter
• To Track DNS conversations
• To tag the network flows with the FQDN
c_ip c_port c_pkts c_bytes s_ip s_port s_pkts s_bytes Protocol
12.132.54.94* 1197 1 18938 87.250.137.92 443 1 992221 HTTP
Tstat
FQDN
Facebook.com
DN-Hunter
*Client IP address is anonymized
*
Full list at https://smartdata.polito.it/five-years-at-the-edge-watching-internet-from-the-isp-network/
HTTP Host:
--
When FQDN
Current facebook.com, fbcdn.com
Before fbstatic-a.akamaihd.net
Regexp* Service
facebook.com, fbcdn.com,
ˆfbstatic-[a-z].akamaihd.net$
Facebook
netflix.com, nflxvideo.net Netflix
5 Years 04/2013 – 10/2017
247 Billions flows
31.9 TB of compressed data

24. 1. How much does the user cost?
6

25. Daily Traffic Consumption
7
0.01
0.10
1.00
10 kB
100 kB
1 MB
10 MB
100 MB
1 GB
10 GB
100 GB
CCDF
ADSL, 2014
0.01
0.10
1.00
10 kB
100 kB
1 MB
10 MB
100 MB
1 GB
10 GB
100 GB
CCDF
ADSL, 2014
Download
Upload
• CCDF of Daily traffic consumption per household

26. Daily Traffic Consumption
7
0.01
0.10
1.00
10 kB
100 kB
1 MB
10 MB
100 MB
1 GB
10 GB
100 GB
CCDF
ADSL, 2014
0.01
0.10
1.00
10 kB
100 kB
1 MB
10 MB
100 MB
1 GB
10 GB
100 GB
CCDF
ADSL, 2014
Download
Upload
0.01
0.10
1.00
10 kB
100 kB
1 MB
10 MB
100 MB
1 GB
10 GB
100 GB
CCDF
ADSL, 2014
ADSL, 2017
0.01
0.10
1.00
10 kB
100 kB
1 MB
10 MB
100 MB
1 GB
10 GB
100 GB
CCDF
ADSL, 2014
ADSL, 2017
• CCDF of Daily traffic consumption per household

27. Daily Traffic Consumption
7
0.01
0.10
1.00
10 kB
100 kB
1 MB
10 MB
100 MB
1 GB
10 GB
100 GB
CCDF
ADSL, 2014
0.01
0.10
1.00
10 kB
100 kB
1 MB
10 MB
100 MB
1 GB
10 GB
100 GB
CCDF
ADSL, 2014
Download
Upload
0.01
0.10
1.00
10 kB
100 kB
1 MB
10 MB
100 MB
1 GB
10 GB
100 GB
CCDF
ADSL, 2014
ADSL, 2017
0.01
0.10
1.00
10 kB
100 kB
1 MB
10 MB
100 MB
1 GB
10 GB
100 GB
CCDF
ADSL, 2014
ADSL, 2017
Heavy Usage
Heavy Usage
• CCDF of Daily traffic consumption per household

28. Daily Traffic Consumption
7
0.01
0.10
1.00
10 kB
100 kB
1 MB
10 MB
100 MB
1 GB
10 GB
100 GB
CCDF
ADSL, 2014
0.01
0.10
1.00
10 kB
100 kB
1 MB
10 MB
100 MB
1 GB
10 GB
100 GB
CCDF
ADSL, 2014
Download
Upload
0.01
0.10
1.00
10 kB
100 kB
1 MB
10 MB
100 MB
1 GB
10 GB
100 GB
CCDF
ADSL, 2014
ADSL, 2017
0.01
0.10
1.00
10 kB
100 kB
1 MB
10 MB
100 MB
1 GB
10 GB
100 GB
CCDF
ADSL, 2014
ADSL, 2017
• 2014 vs 2017
• CCDF of Daily traffic consumption per household

29. Daily Traffic Consumption
7
0.01
0.10
1.00
10 kB
100 kB
1 MB
10 MB
100 MB
1 GB
10 GB
100 GB
CCDF
ADSL, 2014
0.01
0.10
1.00
10 kB
100 kB
1 MB
10 MB
100 MB
1 GB
10 GB
100 GB
CCDF
ADSL, 2014
Download
Upload
0.01
0.10
1.00
10 kB
100 kB
1 MB
10 MB
100 MB
1 GB
10 GB
100 GB
CCDF
ADSL, 2014
ADSL, 2017
0.01
0.10
1.00
10 kB
100 kB
1 MB
10 MB
100 MB
1 GB
10 GB
100 GB
CCDF
ADSL, 2014
ADSL, 2017
• 2014 vs 2017
• ADSL 2x Download and Upload
• CCDF of Daily traffic consumption per household

30. Daily Traffic Consumption
7
0.01
0.10
1.00
10 kB
100 kB
1 MB
10 MB
100 MB
1 GB
10 GB
100 GB
CCDF
ADSL, 2014
0.01
0.10
1.00
10 kB
100 kB
1 MB
10 MB
100 MB
1 GB
10 GB
100 GB
CCDF
ADSL, 2014
Download
Upload
0.01
0.10
1.00
10 kB
100 kB
1 MB
10 MB
100 MB
1 GB
10 GB
100 GB
CCDF
ADSL, 2014
ADSL, 2017
0.01
0.10
1.00
10 kB
100 kB
1 MB
10 MB
100 MB
1 GB
10 GB
100 GB
CCDF
ADSL, 2014
ADSL, 2017
• 2014 vs 2017
• ADSL 2x Download and Upload
• CCDF of Daily traffic consumption per household

31. Daily Traffic Consumption
7
0.01
0.10
1.00
10 kB
100 kB
1 MB
10 MB
100 MB
1 GB
10 GB
100 GB
CCDF
ADSL, 2014
0.01
0.10
1.00
10 kB
100 kB
1 MB
10 MB
100 MB
1 GB
10 GB
100 GB
CCDF
ADSL, 2014
Download
Upload
0.01
0.10
1.00
10 kB
100 kB
1 MB
10 MB
100 MB
1 GB
10 GB
100 GB
CCDF
ADSL, 2014
ADSL, 2017
0.01
0.10
1.00
10 kB
100 kB
1 MB
10 MB
100 MB
1 GB
10 GB
100 GB
CCDF
ADSL, 2014
ADSL, 2017
FTTH, 2014
FTTH, 2017
0.01
0.10
1.00
10 kB
100 kB
1 MB
10 MB
100 MB
1 GB
10 GB
100 GB
CCDF
ADSL, 2014
ADSL, 2017
0.01
0.10
1.00
10 kB
100 kB
1 MB
10 MB
100 MB
1 GB
10 GB
100 GB
CCDF
ADSL, 2014
ADSL, 2017
FTTH, 2014
FTTH, 2017
• CCDF of Daily traffic consumption per household
• FTTH vs ADSL

32. Daily Traffic Consumption
7
0.01
0.10
1.00
10 kB
100 kB
1 MB
10 MB
100 MB
1 GB
10 GB
100 GB
CCDF
ADSL, 2014
0.01
0.10
1.00
10 kB
100 kB
1 MB
10 MB
100 MB
1 GB
10 GB
100 GB
CCDF
ADSL, 2014
Download
Upload
0.01
0.10
1.00
10 kB
100 kB
1 MB
10 MB
100 MB
1 GB
10 GB
100 GB
CCDF
ADSL, 2014
ADSL, 2017
0.01
0.10
1.00
10 kB
100 kB
1 MB
10 MB
100 MB
1 GB
10 GB
100 GB
CCDF
ADSL, 2014
ADSL, 2017
FTTH, 2014
FTTH, 2017
0.01
0.10
1.00
10 kB
100 kB
1 MB
10 MB
100 MB
1 GB
10 GB
100 GB
CCDF
ADSL, 2014
ADSL, 2017
0.01
0.10
1.00
10 kB
100 kB
1 MB
10 MB
100 MB
1 GB
10 GB
100 GB
CCDF
ADSL, 2014
ADSL, 2017
FTTH, 2014
FTTH, 2017
• CCDF of Daily traffic consumption per household
• FTTH vs ADSL
• 25% + Download, 2x Upload

33. Daily Traffic Consumption
7
0.01
0.10
1.00
10 kB
100 kB
1 MB
10 MB
100 MB
1 GB
10 GB
100 GB
CCDF
ADSL, 2014
0.01
0.10
1.00
10 kB
100 kB
1 MB
10 MB
100 MB
1 GB
10 GB
100 GB
CCDF
ADSL, 2014
Download
Upload
0.01
0.10
1.00
10 kB
100 kB
1 MB
10 MB
100 MB
1 GB
10 GB
100 GB
CCDF
ADSL, 2014
ADSL, 2017
0.01
0.10
1.00
10 kB
100 kB
1 MB
10 MB
100 MB
1 GB
10 GB
100 GB
CCDF
ADSL, 2014
ADSL, 2017
FTTH, 2014
FTTH, 2017
0.01
0.10
1.00
10 kB
100 kB
1 MB
10 MB
100 MB
1 GB
10 GB
100 GB
CCDF
ADSL, 2014
ADSL, 2017
0.01
0.10
1.00
10 kB
100 kB
1 MB
10 MB
100 MB
1 GB
10 GB
100 GB
CCDF
ADSL, 2014
ADSL, 2017
FTTH, 2014
FTTH, 2017
• CCDF of Daily traffic consumption per household
• FTTH vs ADSL
• 25% + Download, 2x Upload
Heavy Upload

34. Traffic Consumption Increase
• Average download per user per day
8

35. Traffic Consumption Increase
• Average download per user per day
• Increases at constant rate
8

36. Traffic Consumption Increase
• Average download per user per day
• Increases at constant rate
8
• Does it hold at every hour of the day?

37. Traffic Consumption Increase
• Average download per user per day
• Increases at constant rate
8
• Does it hold at every hour of the day?
• Yes!

38. Traffic Consumption Increase
• Average download per user per day
• Increases at constant rate
8
• Does it hold at every hour of the day?
• Yes!
• Late night traffic increases
•automatic download of updates, machine-generated traffic
like IoT devices
•Larger for FTTH due to different profiles

39. Traffic Consumption Increase
• Average download per user per day
• Increases at constant rate
8
• Does it hold at every hour of the day?
• Yes!
• Late night traffic increases
•automatic download of updates, machine-generated traffic
like IoT devices
•Larger for FTTH due to different profiles
• During prime time FTTH has a higher increase
• Video streaming content in higher definition

40. Traffic Consumption Increase
• Average download per user per day
• Increases at constant rate
8
• Does it hold at every hour of the day?
• Yes!
• Late night traffic increases
•automatic download of updates, machine-generated traffic
like IoT devices
•Larger for FTTH due to different profiles
• During prime time FTTH has a higher increase
• Video streaming content in higher definition
The same user consume twice the traffic

41. 2. How users' habits changed?
9

42. 2013/07
2014/01
2014/07
2015/01
2015/07
2016/01
2016/07
2017/01
2017/07
Peer-To-Peer
Ebay
Amazon
SnapChat
Telegram
WhatsApp
Skype
Spotify
Adult
Netflix
YouTube
LinkedIn
Twitter
Instagram
Facebook
DuckDuckGo
Bing
Google
0
10
20
30
40
50
60
70
80
90
100
Popularity[%]
Service Popularity
• % of active households accessing a given service
10

43. 2013/07
2014/01
2014/07
2015/01
2015/07
2016/01
2016/07
2017/01
2017/07
Peer-To-Peer
Ebay
Amazon
SnapChat
Telegram
WhatsApp
Skype
Spotify
Adult
Netflix
YouTube
LinkedIn
Twitter
Instagram
Facebook
DuckDuckGo
Bing
Google
0
10
20
30
40
50
60
70
80
90
100
Popularity[%]
Service Popularity
• % of active households accessing a given service
• Only Real Visits
10

44. 2013/07
2014/01
2014/07
2015/01
2015/07
2016/01
2016/07
2017/01
2017/07
Peer-To-Peer
Ebay
Amazon
SnapChat
Telegram
WhatsApp
Skype
Spotify
Adult
Netflix
YouTube
LinkedIn
Twitter
Instagram
Facebook
DuckDuckGo
Bing
Google
0
10
20
30
40
50
60
70
80
90
100
Popularity[%]
Service Popularity
• % of active households accessing a given service
• Only Real Visits
10

45. 2013/07
2014/01
2014/07
2015/01
2015/07
2016/01
2016/07
2017/01
2017/07
Peer-To-Peer
Ebay
Amazon
SnapChat
Telegram
WhatsApp
Skype
Spotify
Adult
Netflix
YouTube
LinkedIn
Twitter
Instagram
Facebook
DuckDuckGo
Bing
Google
0
10
20
30
40
50
60
70
80
90
100
Popularity[%]
Service Popularity
• % of active households accessing a given service
• Only Real Visits
10

46. 2013/07
2014/01
2014/07
2015/01
2015/07
2016/01
2016/07
2017/01
2017/07
Peer-To-Peer
Ebay
Amazon
SnapChat
Telegram
WhatsApp
Skype
Spotify
Adult
Netflix
YouTube
LinkedIn
Twitter
Instagram
Facebook
DuckDuckGo
Bing
Google
0
10
20
30
40
50
60
70
80
90
100
Popularity[%]
Service Popularity
• % of active households accessing a given service
• Only Real Visits
10
15 % è 45 %
0.3 % è 0.3 %

47. 2013/07
2014/01
2014/07
2015/01
2015/07
2016/01
2016/07
2017/01
2017/07
Peer-To-Peer
Ebay
Amazon
SnapChat
Telegram
WhatsApp
Skype
Spotify
Adult
Netflix
YouTube
LinkedIn
Twitter
Instagram
Facebook
DuckDuckGo
Bing
Google
0
10
20
30
40
50
60
70
80
90
100
Popularity[%]
Service Popularity
• % of active households accessing a given service
• Only Real Visits
10
15 % è 45 %
0.3 % è 0.3 %

48. What is the traffic load imposed by web services?
11

49. • Popularity tops in 2016
What is the traffic load imposed by web services?
11

50. • Popularity tops in 2016
• Per-user traffic drops mid 2016
What is the traffic load imposed by web services?
11

51. What is the traffic load imposed by web services?
12

52. What is the traffic load imposed by web services?
12
• Popularity almost reached saturation

53. What is the traffic load imposed by web services?
12
• Popularity almost reached saturation
• Per-user traffic now 10 MB per day

54. What is the traffic load imposed by web services?
12
• Popularity almost reached saturation
• Per-user traffic now 10 MB per day
• Christmas and New Year’s Eve peaks

55. What is the traffic load imposed by web services?
13

56. What is the traffic load imposed by web services?
13
• Popularity still growing

57. What is the traffic load imposed by web services?
13
• Popularity still growing
• Per-user traffic still exploding

58. What is the traffic load imposed by web services?
13
• Popularity still growing
• Per-user traffic still exploding
Snapchat, WhatsApp and Instagram are mostly
used from mobiles
Those are taking a predominant role also at home

59. What is the traffic load imposed by web services?
• Average traffic per-client per day
14

60. What is the traffic load imposed by web services?
• Average traffic per-client per day
14

61. What is the traffic load imposed by web services?
• Average traffic per-client per day
• Video auto-play enabled in 2014
14

62. What is the traffic load imposed by web services?
• Average traffic per-client per day
• Video auto-play enabled in 2014
• April traffic from 35 MB to 60+ MB
14

63. What is the traffic load imposed by web services?
• Average traffic per-client per day
• Video auto-play enabled in 2014
• April traffic from 35 MB to 60+ MB
• After video auto-play stopped
14

64. What is the traffic load imposed by web services?
• Average traffic per-client per day
• Video auto-play enabled in 2014
• April traffic from 35 MB to 60+ MB
• After video auto-play stopped
• July daily traffic topped at 90 MB
• 2.5 times March 2014!
14

65. What is the traffic load imposed by web services?
• Average traffic per-client per day
• Video auto-play enabled in 2014
• April traffic from 35 MB to 60+ MB
• After video auto-play stopped
• July daily traffic topped at 90 MB
• 2.5 times March 2014!
14
Big players owning servers and clients
can dramatically impact traffic consumption
complicating the management of ISP networks

66. 3. How the protocols changed?
15

67. 2013/04
2013/07
2013/10
2014/01
2014/04
2014/07
2014/10
2015/01
2015/04
2015/07
2015/10
2016/01
2016/04
2016/07
2016/10
2017/01
2017/04
2017/07
2017/10
0
10
20
30
40
50
60
70
80
90
100
Share[%]
FB-ZERO
SPDY
HTTP/2
TLS
QUIC
HTTP
A B C D E F
Web Protocols Over Five Years
• Traffic share for several Web protocols
16

68. 2013/04
2013/07
2013/10
2014/01
2014/04
2014/07
2014/10
2015/01
2015/04
2015/07
2015/10
2016/01
2016/04
2016/07
2016/10
2017/01
2017/04
2017/07
2017/10
0
10
20
30
40
50
60
70
80
90
100
Share[%]
FB-ZERO
SPDY
HTTP/2
TLS
QUIC
HTTP
A B C D E F
Web Protocols Over Five Years
• Traffic share for several Web protocols
16
YouTube from HTTP
to HTTPS

69. 2013/04
2013/07
2013/10
2014/01
2014/04
2014/07
2014/10
2015/01
2015/04
2015/07
2015/10
2016/01
2016/04
2016/07
2016/10
2017/01
2017/04
2017/07
2017/10
0
10
20
30
40
50
60
70
80
90
100
Share[%]
FB-ZERO
SPDY
HTTP/2
TLS
QUIC
HTTP
A B C D E F
Web Protocols Over Five Years
• Traffic share for several Web protocols
16
YouTube from HTTP
to HTTPS
Google start
QUIC

70. 2013/04
2013/07
2013/10
2014/01
2014/04
2014/07
2014/10
2015/01
2015/04
2015/07
2015/10
2016/01
2016/04
2016/07
2016/10
2017/01
2017/04
2017/07
2017/10
0
10
20
30
40
50
60
70
80
90
100
Share[%]
FB-ZERO
SPDY
HTTP/2
TLS
QUIC
HTTP
A B C D E F
Web Protocols Over Five Years
• Traffic share for several Web protocols
16
YouTube from HTTP
to HTTPS
Google start
QUIC
Probe updates
to identify SPDY

71. 2013/04
2013/07
2013/10
2014/01
2014/04
2014/07
2014/10
2015/01
2015/04
2015/07
2015/10
2016/01
2016/04
2016/07
2016/10
2017/01
2017/04
2017/07
2017/10
0
10
20
30
40
50
60
70
80
90
100
Share[%]
FB-ZERO
SPDY
HTTP/2
TLS
QUIC
HTTP
A B C D E F
Web Protocols Over Five Years
• Traffic share for several Web protocols
16
YouTube from HTTP
to HTTPS
Google start
QUIC
Probe updates
to identify SPDY
Google Disabled
QUIC

72. 2013/04
2013/07
2013/10
2014/01
2014/04
2014/07
2014/10
2015/01
2015/04
2015/07
2015/10
2016/01
2016/04
2016/07
2016/10
2017/01
2017/04
2017/07
2017/10
0
10
20
30
40
50
60
70
80
90
100
Share[%]
FB-ZERO
SPDY
HTTP/2
TLS
QUIC
HTTP
A B C D E F
Web Protocols Over Five Years
• Traffic share for several Web protocols
16
SPDY traffic
to HTTP/2
YouTube from HTTP
to HTTPS
Google start
QUIC
Probe updates
to identify SPDY
Google Disabled
QUIC

73. 2013/04
2013/07
2013/10
2014/01
2014/04
2014/07
2014/10
2015/01
2015/04
2015/07
2015/10
2016/01
2016/04
2016/07
2016/10
2017/01
2017/04
2017/07
2017/10
0
10
20
30
40
50
60
70
80
90
100
Share[%]
FB-ZERO
SPDY
HTTP/2
TLS
QUIC
HTTP
A B C D E F
Web Protocols Over Five Years
• Traffic share for several Web protocols
16
SPDY traffic
to HTTP/2
YouTube from HTTP
to HTTPS
Google start
QUIC
Probe updates
to identify SPDY
Google Disabled
QUIC FB-Zero
deployment

74. 2013/04
2013/07
2013/10
2014/01
2014/04
2014/07
2014/10
2015/01
2015/04
2015/07
2015/10
2016/01
2016/04
2016/07
2016/10
2017/01
2017/04
2017/07
2017/10
0
10
20
30
40
50
60
70
80
90
100
Share[%]
FB-ZERO
SPDY
HTTP/2
TLS
QUIC
HTTP
A B C D E F
Web Protocols Over Five Years
• Traffic share for several Web protocols
16
SPDY traffic
to HTTP/2
YouTube from HTTP
to HTTPS
Google start
QUIC
Probe updates
to identify SPDY
Google Disabled
QUIC FB-Zero
deployment

75. 2013/04
2013/07
2013/10
2014/01
2014/04
2014/07
2014/10
2015/01
2015/04
2015/07
2015/10
2016/01
2016/04
2016/07
2016/10
2017/01
2017/04
2017/07
2017/10
0
10
20
30
40
50
60
70
80
90
100
Share[%]
FB-ZERO
SPDY
HTTP/2
TLS
QUIC
HTTP
A B C D E F
Web Protocols Over Five Years
• Traffic share for several Web protocols
16
SPDY traffic
to HTTP/2
YouTube from HTTP
to HTTPS
Google start
QUIC
Probe updates
to identify SPDY
Google Disabled
QUIC FB-Zero
deployment
90%
25%

76. 2013/04
2013/07
2013/10
2014/01
2014/04
2014/07
2014/10
2015/01
2015/04
2015/07
2015/10
2016/01
2016/04
2016/07
2016/10
2017/01
2017/04
2017/07
2017/10
0
10
20
30
40
50
60
70
80
90
100
Share[%]
FB-ZERO
SPDY
HTTP/2
TLS
QUIC
HTTP
A B C D E F
Web Protocols Over Five Years
• Traffic share for several Web protocols
16
SPDY traffic
to HTTP/2
YouTube from HTTP
to HTTPS
Google start
QUIC
Probe updates
to identify SPDY
Google Disabled
QUIC FB-Zero
deployment
90%
25%
Big players owning servers and clients
are free to deploy experiments on the web
complicating the network management

77. 4. How the infrastructure changed?
17

78. CDN Monitoring Methodology
• Study CDNs impact
• Content access latency as Users’ Quality of
Experience metric
18

79. CDN Monitoring Methodology
• Study CDNs impact
• Content access latency as Users’ Quality of
Experience metric
18

80. CDN Monitoring Methodology
• Study CDNs impact
• Content access latency as Users’ Quality of
Experience metric
• Focus on the Round Trip Time (RTT)
18

81. CDN Monitoring Methodology
• Study CDNs impact
• Content access latency as Users’ Quality of
Experience metric
• Focus on the Round Trip Time (RTT)
18

82. CDN Monitoring Methodology
• Study CDNs impact
• Content access latency as Users’ Quality of
Experience metric
• Focus on the Round Trip Time (RTT)
18
segment

83. CDN Monitoring Methodology
• Study CDNs impact
• Content access latency as Users’ Quality of
Experience metric
• Focus on the Round Trip Time (RTT)
18
ack
segment

84. CDN Monitoring Methodology
• Study CDNs impact
• Content access latency as Users’ Quality of
Experience metric
• Focus on the Round Trip Time (RTT)
18
RTT
ack
segment

85. CDN Monitoring Methodology
• Study CDNs impact
• Content access latency as Users’ Quality of
Experience metric
• Focus on the Round Trip Time (RTT)
• RTT from the probe to servers only
• Core Network Delay
18
RTT
ack
segment
Core
network
delay

86. CDN Monitoring Methodology
• Study CDNs impact
• Content access latency as Users’ Quality of
Experience metric
• Focus on the Round Trip Time (RTT)
• RTT from the probe to servers only
• Core Network Delay
• Excluding RTT from clients to the probe
• Access network delay
18
RTT
ack
segment
Access
network
delay
Core
network
delay

87. CDN Monitoring Methodology
• Study CDNs impact
• Content access latency as Users’ Quality of
Experience metric
• Focus on the Round Trip Time (RTT)
• RTT from the probe to servers only
• Core Network Delay
• Excluding RTT from clients to the probe
• Access network delay
18
RTT
ack
segment
Access
network
delay
Core
network
delay
…Because access
delay is negligible…
… right?

88. The birth of the sub-millisecond Caching
• Compare RTT Distributions
• April 2014 vs April 2017
19
0.0
0.2
0.4
0.6
0.8
1.0
0.1 1 10 100
CDF
RTT [ms]
Facebook, 2014

89. The birth of the sub-millisecond Caching
• Compare RTT Distributions
• April 2014 vs April 2017
19
0.0
0.2
0.4
0.6
0.8
1.0
0.1 1 10 100
CDF
RTT [ms]
Facebook, 2014
3

90. The birth of the sub-millisecond Caching
• Compare RTT Distributions
• April 2014 vs April 2017
19
0.0
0.2
0.4
0.6
0.8
1.0
0.1 1 10 100
CDF
RTT [ms]
Facebook, 2014
3
!
20

91. The birth of the sub-millisecond Caching
• Compare RTT Distributions
• April 2014 vs April 2017
19
0.0
0.2
0.4
0.6
0.8
1.0
0.1 1 10 100
CDF
RTT [ms]
Facebook, 2014
0.0
0.2
0.4
0.6
0.8
1.0
0.1 1 10 100
CDF
RTT [ms]
Facebook, 2014
Facebook, 2017
3
!
20

92. The birth of the sub-millisecond Caching
• Compare RTT Distributions
• April 2014 vs April 2017
19
0.0
0.2
0.4
0.6
0.8
1.0
0.1 1 10 100
CDF
RTT [ms]
Facebook, 2014
0.0
0.2
0.4
0.6
0.8
1.0
0.1 1 10 100
CDF
RTT [ms]
Facebook, 2014
Facebook, 2017
3
!
"
20

93. The birth of the sub-millisecond Caching
• Compare RTT Distributions
• April 2014 vs April 2017
19
0.0
0.2
0.4
0.6
0.8
1.0
0.1 1 10 100
CDF
RTT [ms]
YouTube, 2014
0.0
0.2
0.4
0.6
0.8
1.0
0.1 1 10 100
CDF
RTT [ms]
Facebook, 2014
0.0
0.2
0.4
0.6
0.8
1.0
0.1 1 10 100
CDF
RTT [ms]
Facebook, 2014
Facebook, 2017
3
!
"
20

94. The birth of the sub-millisecond Caching
• Compare RTT Distributions
• April 2014 vs April 2017
19
0.0
0.2
0.4
0.6
0.8
1.0
0.1 1 10 100
CDF
RTT [ms]
YouTube, 2014
0.0
0.2
0.4
0.6
0.8
1.0
0.1 1 10 100
CDF
RTT [ms]
Facebook, 2014
0.0
0.2
0.4
0.6
0.8
1.0
0.1 1 10 100
CDF
RTT [ms]
Facebook, 2014
Facebook, 2017
3
!
"
20 3
"

95. The birth of the sub-millisecond Caching
• Compare RTT Distributions
• April 2014 vs April 2017
19
0.0
0.2
0.4
0.6
0.8
1.0
0.1 1 10 100
CDF
RTT [ms]
YouTube, 2014
0.0
0.2
0.4
0.6
0.8
1.0
0.1 1 10 100
CDF
RTT [ms]
Facebook, 2014
0.0
0.2
0.4
0.6
0.8
1.0
0.1 1 10 100
CDF
RTT [ms]
Facebook, 2014
Facebook, 2017
3
!
"
20
0.0
0.2
0.4
0.6
0.8
1.0
0.1 1 10 100
CDF
RTT [ms]
YouTube, 2014
YouTube, 2017
3
"

96. The birth of the sub-millisecond Caching
• Compare RTT Distributions
• April 2014 vs April 2017
19
0.0
0.2
0.4
0.6
0.8
1.0
0.1 1 10 100
CDF
RTT [ms]
YouTube, 2014
0.0
0.2
0.4
0.6
0.8
1.0
0.1 1 10 100
CDF
RTT [ms]
Facebook, 2014
0.0
0.2
0.4
0.6
0.8
1.0
0.1 1 10 100
CDF
RTT [ms]
Facebook, 2014
Facebook, 2017
3
!
"
20
0.0
0.2
0.4
0.6
0.8
1.0
0.1 1 10 100
CDF
RTT [ms]
YouTube, 2014
YouTube, 2017
3
#
"
0.37
The CDN is inside
the PoP!
0.5

97. The birth of the sub-millisecond Caching
• Compare RTT Distributions
• April 2014 vs April 2017
19
0.0
0.2
0.4
0.6
0.8
1.0
0.1 1 10 100
CDF
RTT [ms]
YouTube, 2014
0.0
0.2
0.4
0.6
0.8
1.0
0.1 1 10 100
CDF
RTT [ms]
Facebook, 2014
0.0
0.2
0.4
0.6
0.8
1.0
0.1 1 10 100
CDF
RTT [ms]
Facebook, 2014
Facebook, 2017
3
!
"
20
0.0
0.2
0.4
0.6
0.8
1.0
0.1 1 10 100
CDF
RTT [ms]
YouTube, 2014
YouTube, 2017
3
#
"
0.37
The CDN is inside
the PoP!
CDNs are making sub-millisecond internet possible
New headache for the ISPs: host/manage CDNs inside their network0.5

98. The Server Infrastructure
• Server IP evolution over time
• Red: the IP address was dedicated
• Blue: the IP address
• no dot: the IP address was not contacted in that day
20

99. The Server Infrastructure
• Server IP evolution over time
• Red: the IP address was dedicated
• Blue: the IP address
• no dot: the IP address was not contacted in that day
20
Always used a totally
dedicated infrastructure

100. The Server Infrastructure
• Server IP evolution over time
• Red: the IP address was dedicated
• Blue: the IP address
• no dot: the IP address was not contacted in that day
20
Always used a totally
dedicated infrastructure
In 2015 starts migrating to a
dedicated infrastructure

101. The Server Convergence
• Which Autonomous Systems are involved?
21

102. The Server Convergence
• Which Autonomous Systems are involved?
YouTube PoP integration
during 2015
21

103. The Server Convergence
• Which Autonomous Systems are involved?
YouTube PoP integration
during 2015
Migration to Facebook infrastructure
started in 2014 ended in 2015
21

104. The Server Convergence
• Which Autonomous Systems are involved?
YouTube PoP integration
during 2015
Migration to Facebook infrastructure
started in 2014 ended in 2015
21
Content providers deploy their own infrastructures
with thousands of IP addresses

105. Conclusion & Future Work
• Q1: Households daily traffic more than doubled
• Q2: Quick rise and sudden death of social applications
• Able to generate massive amount of data
• Q3: Big players experiment new protocols (& imposing
standards)
• Q4: The infrastructure keeps concentrating and specializing
22

106. Conclusion & Future Work
• Q1: Households daily traffic more than doubled
• Q2: Quick rise and sudden death of social applications
• Able to generate massive amount of data
• Q3: Big players experiment new protocols (& imposing
standards)
• Q4: The infrastructure keeps concentrating and specializing
• Future Work
• Web portal to make this data playable
• New analytics for…
22