The document discusses the differences and optimizations in HTTP/2 compared to HTTP/1.x, focusing on resource prioritization, browser rendering processes, and the handling of fonts and images. It highlights the role of various browser behaviors, server configurations, and TCP mechanisms in improving performance, while also addressing challenges like TCP congestion control and OCSP latency. Additionally, it emphasizes the importance of using proper content delivery networks (CDNs) and tuning server settings for better web performance.

1. HTTP/2 in Practice
Patrick Meenan
@PatMeenan

2. Outline
• Browser
• HTTP/1.x vs HTTP/2
• Resource Prioritization
• Network
• TCP Buffering
• Network Buffering
• Server
• Future

3. The Browser

4. https://developers.google.com/web/fundamentals/performance/critical-rendering-path/render-tree-construction

6. Basic Parser Rules
•Process 1 token at a time
•Stylesheets Block Render
•Non-Async Script tags block Parser/DOM until:
•Pending Stylesheets have loaded
•Script has loaded

7. Browser Events
•DOM Content Loaded
• When the parser reaches the end of the document
•Load
• When all of the document resources have finished loading

31. Late-Discovered Resources
• Fonts
• Background Images
• Script-injected content
• @import

32. (Simplified) Example
•1 HTML
•1 Stylesheet
•4 Scripts (2 blocking in the head)
•1 Web Font
•13 Images (5 visible)

33. In-browser Prioritization (Chrome)
HTML CSS
Script 1 Script 2
Image 1
VeryHigh
High
Medium
Async 1 Async 2Low
VeryLow
Image 2 Image 3 Image …

34. In-browser Prioritization (Chrome)
HTML CSS
Script 1 Script 2
Image 1
VeryHigh
High
Medium
Async 1 Async 2Low
VeryLow
Image 2 Image 3 Image …
Font

35. HTML CSS
Script 1 Script 2
Image 6
VeryHigh
High
Medium
Async 1 Async 2Low
VeryLow
Image 7 Image 8 Image …
Font
Image 1 Image 2 Image 3 Image 4 Image 5

36. HTTP/1.x Prioritization
• 6 Connections per origin
• Pick next-highest for each origin as a connection becomes available

37. HTMLVeryHigh
High
Medium
Low
VeryLow

38. HTML CSS
Script 1 Script 2
Image 1
VeryHigh
High
Medium
Async 1 Async 2Low
VeryLow
Image 2 Image 3 Image …

39. HTML CSS
Script 1 Script 2
Image 1
VeryHigh
High
Medium
Async 1 Async 2Low
VeryLow
Image 2 Image 3 Image …

40. HTML CSS
Script 1 Script 2
Image 1
VeryHigh
High
Medium
Async 1 Async 2Low
VeryLow
Image 2 Image 3 Image …

41. HTML CSS
Script 1 Script 2
Image 1
VeryHigh
High
Medium
Async 1 Async 2Low
VeryLow
Image 2 Image 3 Image 4 Image …

42. HTML CSS
Script 1 Script 2
Image 1
VeryHigh
High
Medium
Async 1 Async 2Low
VeryLow
Image 2 Image 3 Image 4 Image …

43. HTML CSS
Script 1 Script 2
Image 6
VeryHigh
High
Medium
Async 1 Async 2Low
VeryLow
Image 7 Image 8 Image …
Font
Image 1 Image 2 Image 3 Image 4 Image 5

44. HTML CSS
Script 1 Script 2
Image 6
VeryHigh
High
Medium
Async 1 Async 2Low
VeryLow
Image 7 Image 8 Image …
Font
Image 1 Image 2 Image 3 Image 4 Image 5

45. HTML CSS
Script 1 Script 2
Image 6
VeryHigh
High
Medium
Async 1 Async 2Low
VeryLow
Image 7 Image 8 Image …
Font
Image 1 Image 2 Image 3 Image 4 Image 5

46. HTML CSS
Script 1 Script 2
Image 6
VeryHigh
High
Medium
Async 1 Async 2Low
VeryLow
Image 7 Image 8 Image …
Font
Image 1 Image 2 Image 3 Image 4 Image 5

47. HTTP/2 Prioritization
•All requests sent to server immediately
•Priorities specified in dependency tree
•Any “stream” can depend on another stream
•Peers can be weighted
•Priority changes communicated with a PRIORITY
frame

48. https://developers.google.com/web/fundamentals/performance/http2/

49. Optimal Script Loading

50. Optimal Loading
Start Render
Visually Complete

52. Worst-Case
Start Render
Visually Complete

55. HTTP/2
HTTP/1.x

56. Browser Differences

57. Chrome – Linear List
https://github.com/quicwg/wg-materials/raw/master/interim-19-05/priorities.pdf

58. Firefox - Groups
https://github.com/quicwg/wg-materials/raw/master/interim-19-05/priorities.pdf

59. Safari - Weighted
https://github.com/quicwg/wg-materials/raw/master/interim-19-05/priorities.pdf

60. Edge
¯_(ツ)_/¯

65. Prioritizing across
connections

66. Cross-connection prioritization
•3rd-parties
•Domain sharding
•www.example.com
•static1.example.com
•static2.example.com

67. Prioritizing across connections

68. HTTP/2 Connection coalescing
•New domain resolution includes IP of active connection
•Active connection cert includes new domain
•Still pay DNS lookup cost
• ORIGIN frame/CERTIFICATE frame
•Multi-CDN

69. Edge/Origin
HTTP Load Balancer
(HA Proxy, Netscaler, etc)
HTTP/2 Broken
(and fixed) Here

70. Edge/Origin
HTTP Load Balancer
(HA Proxy, Netscaler, etc)
HTTP/2 Broken
(and fixed) Here

71. Edge/Origin
Layer 4 Load Balancer
HTTP/2 Broken
(and fixed) Here

72. Edge/Origin
CDN
HTTP/2 Broken
(and fixed) Here

73. Edge/Upstream
Best Effort, Data sent as available

74. ishttp2fastyet.com

75. External Fonts (before)

76. Same-origin fonts (wtf)
Start Render

77. Same-origin fonts (correct)
Start Render

78. https://twitter.com/zachleat/status/1055219667894259712

79. Testing for prioritization
https://www.webpagetest.org/http2priorities.html?image=
<url-encoded-image-URL>

80. Warm-up Connection

81. Request 30
below-the-fold
Images
(low priority)

82. Delay until 2 images complete

83. Sequentially Request 2
in-viewport images
(high priority)

84. Interrupts in-flight responses

85. Ideal response time would
match single-image download

86. When it goes wrong

87. Queued behind in-flight
Low-priority requests

88. Round Robin?

89. ¯_(ツ)_/¯

92. Fast Responses
Data Interleaving?

93. ishttp2fastyet.com
NO – 9 of 25 pass

94. The Good
• Akamai
• CDNsun
• Cloudflare
• Fastly

95. The Bad
• Amazon Cloudfront
• Cachefly
• CDNetworks
• ChinaCache
• Edgecast
• Highwinds
• Incapsula
• Instart Logic
• KeyCDN
• LeaseWeb CDN
• Level 3
• Limelight
• Medianova
• Netlify
• Rocket CDN
• StackPath/NetDNA/MaxCDN
• Jetpack CDN
• Yottaa
• Zenedge

96. The Ugly (cloud load balancers)
• Amazon AWS
• Google Cloud
• Microsoft Azure

97. Testing/Throttling
https://calendar.perfplanet.com/2016/testing-with-realistic-networking-conditions/

98. Renderer Renderer
Browser Process

99. Renderer Renderer
Browser Process

100. Renderer Renderer
Browser Process
Packet-
level
Shaper

102. HTTP Header

103. Performance Monitoring
• Understand what (if any) traffic-shaping is
used
• Watch out for (avoid):
•Dev Tools
•Lighthouse
•Puppeteer
•Proxy

105. WHY it Goes Wrong

106. TCP
2 13 4 5 6 7 5 6 7 8

107. TCP
2 13 4 5 6 7 5 6 7 8
Receive Buffer Send BufferBDP

108. BDP

109. BDP
Round Trip Time : 100 ms

110. BDP
Round Trip Time : 100 ms
(divide by 1000 to get seconds)
= 0.100 second rtt

111. BDP
Bandwidth : 8 Mbps

112. BDP
Bandwidth : 8 Mbps
(divide by 8 bits per byte)
= 1 MB per second

113. BDP
1 MB per second * 0.100 second RTT
= 100 KB BDP

114. Bandwidth Latency (RTT) BDP
400 Kbps 400 ms (400 / 8) KB * (400 / 1000) s 20 KB
1.6 Mbps 150 ms (1.6 / 8) MB * (150 / 1000) s 30 KB
8 Mbps 100 ms (8 / 8) MB * (100 / 1000) s 100 KB
50 Mbps 25 ms (50 / 8) MB * (25 / 1000) s 1.6 MB
1 Gbps 100 ms (1000 / 8) MB * (100 / 1000) s 12.5 MB
10 Gbps 200 ms (10 / 8) GB * (200 / 1000) s 250 MB

115. Congestion Window
2 13 4 5 6 7 5 6 7 8
Receive Buffer Send BufferBDP
MIN(RCV, BDP, SEND)

116. TCP Send Buffer
net.ipv4.tcp_wmem = 10240 87380 12582912
Min Default Max

117. TCP Send buffer

118. TCP Auto Tuning
• Disabled if explicit socket buffer is set
• Starts at the Default
• Grows towards max as needed

119. TCP_NOTSENT_LOWAT
• Specify additional buffer above BDP
• Signals socket available only when send buffer drops
below threshold

120. TCP_NOTSENT_LOWAT
net.ipv4.tcp_notsent_lowat = 16384

121. Over-buffering (TCP)
• Impact on reprioritization
• Server metrics (send time artificially low)

122. TCP Congestion Control
• Start with initial Congestion Window (CWND)
• Ramp-up until network is “full”
• Reduce
• lather, rinse, repeat

123. TCP Congestion Control
• Start with initial Congestion Window (CWND)
• Ramp-up until network is “full”
• Reduce
• lather, rinse, repeat

124. Bufferbloat

125. Loss-based congestion control

126. BBR

129. BBR starvation
https://blog.apnic.net/2017/05/09/bbr-new-kid-tcp-block/

130. Initial Congestion Window

131. TCP Slow Start

132. TCP Slow Start
• Start with Congestion Window (CW) = Initial Window (IW)
• Used to default to 4. 10 More common. Measured in packets.
• 10 ~ 15,000 bytes (assuming 1500 MTU)
• Increase CW by 1 for every
• Doubles every RTT
• Round Trips = log2(BDP / IW)

133. Bandwidth Latency (RTT) BDP Slow Start
Round Trips
Slow Start
Time
400 Kbps 400 ms 20 KB 1 400 ms
1.6 Mbps 150 ms 30 KB 1 150 ms
8 Mbps 100 ms 100 KB 3 300 ms
50 Mbps 25 ms 1.6 MB 7 175 ms
1 Gbps 100 ms 12.5 MB 16 1.6 s
10 Gbps 200 ms 250 MB 19 3.8 s
Assuming Initial Congestion Window 10

134. Query Initial Congestion Window

135. Configuring IW
Set on a per-route basis:
sudo ip route change default via 192.168.1.1 dev eth0 proto static initcwnd 10

136. Server Buffers
TLSTCP HTTP/2
HTTP TCP
HTTP TCP
fd

137. Upstream Limits
• 100+ simultaneous inbound requests
• Back-end request limits?

138. 10 second read delay

139. Server HTTP/2 implementations
• H2o, Apache good
• Nginx – not so much
• https://trac.nginx.org/nginx/ticket/1763

140. istlsfastyet.com

141. Online Certificate Status Protocol (OCSP)
Stop the world and query the OCSP server
● DNS lookup
● TCP connect
● Wait for server response
What if the OCSP check times out, gets blocked, etc? See “Revocation still doesn’t work.”
Has this certificate been revoked?

142. ● Chrome blocks on EV certs only
● Other browsers may block on all (FF)
Eliminating OCSP latency
OCSP endpoint
Use OCSP stapling!
1. Server retrieves the OCSP response
2. Server “staples” response to certificate
3. Client verifies stapled response

143. TLS handshake with stapled OCSP response...
$> openssl s_client -connect example.com:443 -tls1 -tlsextdebug -status
OCSP Response Data:
OCSP Response Status: successful (0x0)
Response Type: Basic OCSP Response
Version: 1 (0x0)
Responder Id: C = IL, O = StartCom Ltd., CN = StartCom Class 1 Server OCSP Signer
Produced At: Feb 18 17:53:53 2014 GMT
Responses:
Certificate ID:
Hash Algorithm: sha1
Issuer Name Hash: 6568874F40750F016A3475625E1F5C93E5A26D58
Issuer Key Hash: EB4234D098B0AB9FF41B6B08F7CC642EEF0E2C45
Serial Number: 0B60D5
Cert Status: good
Stapled OCSP means no blocking!
OCSP stapling increases certificate size! Is this a problem for your site? Better check.

144. How many RTTs does your certificate incur?
● Average certificate chain depth: 2-3 certificates
● Average certificate size: ~1~1.5KB
● Plus OCSP response…
● Many cert chains overflow the old TCP (4 packet) CWND
● Upgrade your servers to use IW10!
3+ RTT TLS handshake due to 2 RTT cert?

145. Check your server, you may be surprised...
● Capture a tcpdump of your handshake and check the exchange
● Some servers will pause on “large certificates” until they get an ACK for the
first 4KB of the certificate (doh!)
nginx <1.5.6, HAProxy <1.5-dev22 incur extra RTT, even w/ IW10!

146. 1-RTT non-resumed handshake with TLS False Start
Client sends application data
immediately after “Finished”.
● Eliminates 1RTT
● No protocol changes...
● Only timing is affected
In practice…
● Some servers break (ugh)
● Hence, opt-in behavior...

147. Deploying False Start...
● Chrome and Firefox
Chrome and Firefox
● ALPN advertisement - e.g. “http/1.1”
● Forward secrecy ciphersuite - e.g. ECDHE
Safari
● Forward secrecy ciphersuite
Internet Explorer
● Blacklist + timeout
● If handshake fails, retry without False Start
TL;DR: enable ALPN advertisement and forward secrecy to get 1RTT handshakes.

148. Ingredients for a 1-RTT TLS experience…
1. False Start = 1-RTT handshake for new visitors
a. New users have to perform public-key crypto handshake
1. Session resumption = 1-RTT handshake for returning visitors
a. Plus, we can skip public-key crypto by reusing previous parameters
1. OCSP Stapling
a. No OCSP blocking to verify certificate status
1. False Start + Session Resumption + OCSP stapling
a. 1-RTT handshake for new and returning visitors
b. Returning visitors can skip the public-key crypto

149. What’s wrong with this picture?
300ms RTT, 1.5Mbps...
● It’s a 2-RTT time to first byte!
Large records are buffered, which delays processing!
● It’s a 2-RTT handshake… we know better!
o At least there is no OSCP overhead!

150. TLS record size + latency gotchas...
This record is split across 8 TCP packets
TLS allows up to 16KB of application per record
● New connection + 16KB record = CWND overflow and an extra RTT
● Lost or delayed packet delays processing of entire record

151. Optimizing record size…
1. Google servers implement dynamic record sizing
a. New connections start with 1400 byte records (aka, single MTU)
b. After ~1MB is sent, switch to 16K records
c. After ~1s of inactivity, reset to 1400 byte records
1. Most servers don’t optimize this case at all...
a. HAProxy recently landed dynamic sizing patch - yay!
b. Nginx landed ssl_buffer_size: static override - better, but meh...
c. Cloudflare released patch for Nginx dynamic record sizing
TL;DR: there is no “perfect record size”. Adjust dynamically.

152. Quick sanity check...
theory is great, but does this all work in practice?

153. Tuning Nginx TLS Time To First Byte (TTTFB)
● Pre 1.5.7: bug for 4KB+ certs, resulting in 3RTT+ handshakes
● 1.7.1 added ssl_buffer_size: 4KB record size remove an RTT
● 1.7.1 with NPN and forward secrecy → 1RTT handshake
https://www.igvita.com/2013/12/16/optimizing-nginx-tls-time-to-first-byte/

154. https://hpbn.co/https://www.igvita.com/
@igrigorik

155. What to do?
•Use a “good” CDN
•Or…

156. What to do?
• Set reasonable default TCP send buffer sizes
• Enable TCP_NOTSENT_LOWAT
• Enable BBR
• Use a Web Server with good prioritization support

157. Linux
net.core.wmem_max = 250000000
net.ipv4.tcp_wmem = 10240 102400 250000000
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_notsent_lowat = 16384
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr

159. HTTP/2 PUSH
• Disable it.
• May be automatically triggered by “preload” headers

160. HTTP/2 PUSH – Only works when
• There is a back-end that is slow to generate HTML
• You know HOW slow it is going to be
• You know exactly how much data you can send
without delaying the HTML

161. Preloading
LayoutFonts
<link rel="preload" href="//www.i.cdn.cnn.com/.a/fonts/cnn/3.7.2/cnnsans-
regular.woff2" as="font" type="font/woff2" crossorigin="anonymous">

162. Preload/Appcache Chrome Bugs
Preloaded Fonts Block CSS
(blocking render)
https://andydavies.me/blog/2019/02/12/preloading-fonts-and-the-puzzle-of-priorities/

163. https://twitter.com/AndyDavies/status/1060558550492155909

164. Resource Unbundling
• Avoid temptation to stop bundling JS/CSS
• Image sprites?
• Lose cross-resource compression efficiency
• Browser overhead with lots of requests
• Check cache
• Cross-process IPCs
• Ad-blocker/AV overhead
https://engineering.khanacademy.org/posts/js-packaging-http2.htm

165. Server-initiated prioritization

166. HTTP/3 (and QUIC)
• UDP-based (port 443)
• Moves TCP logic into application layer
• Less OS-level failure modes
• More application responsibility
• Moves loss recovery from per-connection to per-stream
• Will NEVER reach 100% availability
• Treat it as a progressive enhancement

167. HTTP/3 Prioritization?
• Currently replicating HTTP/2 tree
• Join the discussion
•https://httpwg.org/

168. Thank You
Patrick Meenan
pmeenan@webpagetest.org
@PatMeenan

169. We’re Hiring!
https://www.facebook.com/careers/