Slides from the talk on HighLoad++'18 conference.

1. QUIC Illustrated
Alexander Krizhanovsky
Tempesta Technologies, Inc.
ak@tempesta-tech.com

2. Tempesta Technologies
Custom software development in:
●
high performance network traffic processing
e.g. WAF mentioned in Gartner magic quadrant
https://www.ptsecurity.com/ww-en/products/af/
●
Databases
e.g. MariaDB SQL System-Versioned Tables
https://mariadb.com/kb/en/library/system-versioned-tables/
https://mariadb.com/conference/session/querying-data-previous-
point-time
Developing Tempesta FW – open source Linux
Application Delivery Controller (ADC)

3. Tempesta FW:
Application Delivery Controller (ADC)
https://www.netdevconf.org/2.1/session.html?krizhanovsky
HTTP(S) reverse proxy
filtering
●
HTTP DDoS mitigation
●
Web Application Firewall
built into the TCP/IP stack
up to 1.8M HTTP RPS
on 4 cores

4. Disclamer
We’re sceptic about QUIC…
https://github.com/tempesta-tech/tempesta/issues/724
...but I did my best to figure out why QUIC is good
The talk isn’t about QUIC benefits, there are many other talks (see
references)
...instead it’s about how does it work
Not a comprehensive description…
...instead, just how to learn and debug the protocol

5. Why QUIC?
QUIC is ~7% of Internet traffic, 98% of them to Google
Has QUIC: Google, Amazon, Fastly, LiteSpeed Technologies
Adopting: CloudFlare, Mellanox (UDP offload on NICs)
https://www.netdevconf.org/0x12/session.html?udp-segmentation-offload
Middleboxes slowly learn about QUIC
Highlights:
●
Performance: no OSI layers - each layer knows about each other
●
A UDP-based TCP replacement
●
no head-of-line blocking
●
0-RTT handshakes

6. Current state
Still in draft state (IETF meeting 103, Nov 6-7, 2018)
$ grep TBD *.txt|wc -l
23
Several server implementations
Chrome seems the only usable client
Wireshark knows about QUIC

7. Head-of-line blocking (the long story)
HTTP/2 solves HTTP/1 HoL blocking
...so no need many TCP connections
...so 1 TCP connection introduces HoL
blocking
...so multi-stream QUIC replaces TCP
BTW: SCTP solves HoL problem for 11 years
https://en.wikipedia.org/wiki/Stream_Control_Tr
ansmission_Protocol#Features
SCTP is implemented by many libraries and OS
kernels

8. Why not SCTP and/or DTLS?
QUIC FAQ for Geeks
https://docs.google.com/document/d/1lmL9EF6qKrk7gbazY8bIdvq3Pno2
Xj_l_YShP40GLQE/edit
SCTP and DTLS were not designed to minimize latency, and this is
significantly apparent even during the connection establishment phases.
Several of the techniques that QUIC is experimenting with would be
difficult technically to incorporate into existing standards. As an
example, each of these other protocols require several round trips to
establish a connection, which is at odds with our target of 0-RTT
connectivity overhead.
Middleboxes alre also against updating standards

9. Why not TCP Fast Open + TLS 1.3?
TLS 1.3 (used by QUIC anyway): 0/1-RTT handshakes
TCP Fast Open (RFC 7413, default in Linux): 0-RTT
●
does not detect duplicate SYN segments (RFC 7413 6.1)
●
1st
data segment size <= MSS (RFC 7413 6.2)
●
TCP HoL blocking still exists
source: https://lwn.net/Articles/508865/

10. QUIC architecture
Some HTTP/2 stuff goes to
QUIC layer:
●
compression
●
streams
●
framing

11. QUIC & TLS 1.3
draft-ietf-quic-applicability-03.txt
draft-ietf-quic-tls-16.txt: ”Rather than a strict layering,
these two protocols are co-dependent”
TLS record = QUIC packet (no need for TLS dynamic records)

12. QUIC in the wild
Chrome, ver. >= 63
google-chrome
--enable-quic
# tcpdump -i wlp1s0 -X -s0 -nn -vvv
udp port 443 and host www.google.com

13. QUIC in Chrome://net-internals

14. Connection upgrade: TCP → QUIC
GET https://www.google.com/images/errors/robot.png
tcpdump -i wlp1s0 [-X -s0 -nn -q] host www.google.com
192.168.1.67.43588 > 64.233.162.104.443: TCP
64.233.162.104.443 > 192.168.1.67.43588: TCP
... (TCP handshake, TLS handshake, the image)
192.168.1.67.54258 > 64.233.162.147.443: UDP, length 1350
0x0000: 4500 0562 b348 4000 4011 dcda c0a8 0143 E..b.H@.@......C
0x0010: 40e9 a293 d3f2 01bb 054e db6d 0d48 76c7 @........N.m.Hv.
0x0020: 9a51 b1ed 0f51 3034 3301 57be c208 f1de .Q...Q043.W..…
0x0030: a1e2 1b46 d338 a001 0514 4348 4c4f 1000 ...F.8....CHLO..
0x0040: 0000 5041 4400 0604 0000 534e 4900 1404 ..PAD.....SNI…
...
Almost everything is encrypted :(

15. Alternate service mappings
Refresh https://www.google.com/images/errors/robot.png
tcpdump -i wlp1s0 [-X -s0 -nn -q] host www.google.com
192.168.1.67.51462 > 64.233.162.104.443: UDP
64.233.162.104.443 > 192.168.1.67.51462: UDP
0x0000: 4500 0562 8326 4000 4011 664f c0a8 0143 E..b.&@.@.fO...C
0x0010: adc2 dc67 c906 01bb 054e dcbe 0dd7 029f ...g.....N...…
0x0020: 6f34 96ef 4751 3034 3301 f1b1 9b27 e50e o4..GQ043....'..
...

16. Chrome net-export

17. Connection upgrade: HTTP header Alt-Svc
draft-ietf-quic-http-16.txt, RFC 7838
ALTSVC HTTP/2 frame also can be used
Format seems changing
"headers": [
":status: 200",
"accept-ranges: bytes",
"content-type: image/png",
...
"alt-svc: quic=":443"; ma=2592000; v="44,43,39,35""
],

18. Fallback to TCP
Some middleboxes frop UDP (draft-ietf-quic-applicability-03.txt)
# iptables -A OUTPUT -p udp --dport 443 -j DROP
# tcpdump -i wlp1s0 -nn -q host www.google.com
192.168.1.67.54512 > 64.233.165.103.443: tcp 78
192.168.1.67.54512 > 64.233.165.103.443: tcp 46
…

19. Packet headers
Long header – to establish connection contexts
●
Initial, handshake, retry
Short header – after that
Version negotiation – for unsupported version in ClientHello

20. QUIC long header
{D,S}CIL = {Destination,Source} Connection ID Length
Version=0: version negotiation w/ list of supported versions
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+
|1| type(7) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Version (32) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|DCIL(4)|SCIL(4)|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Destination Connection ID (0/32..144) ...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Source Connection ID (0/32..144) ...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Length (i) ...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Packet Number (8/16/32) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

21. QUIC short header
Connection ID
●
Survive NAT rebindings of UDP ports
●
Connection migration (draft-deconinck-quic-multipath-01.txt)
Bits description: “this section should be removed”
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+
|0|K|1|1|0|R R R|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Destination Connection ID (0.144) ...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Packet Number (8/16/32) ...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

22. QUIC header example (ClientHello)
Connection ID: x7bf28b4776840752 = 8931354150076417874
Padding to 1200 bytes to mitigate asymmetric DDoS attacks
IP 0x0000: 4500 0562 c6b0 4000 4011 c69e c0a8 0143 E..b..@.@......C
UDP, type 0x0010: 40e9 a567 c858 01bb 054e 0aed 0d7b f28b @..g.X...N...{..
version 0x0020: 4776 8407 5251 3034 3301 86eb 710d 6f62 ..t,.Q043...q.ob
ClientHello0x0030: a224 675b a416 a001 0400 4348 4c4f 1a00 .$g[......CHLO..
0x0040: 0000 5041 4400 0604 0000 534e 4900 6201 ..PAD.T...SNI.b.
0x0050: 0000 5354 4b00 9801 0000 5645 5200 9c01 ..STK.....VER...
...
0x00c0: 0000 2d2d 2d2d 2d2d 2d2d 2d2d 2d2d 2d2d ..--------------
padding ...
(x406) 0x04c0: 2d2d 2d2d 2d2d 7777 772e 676f 6f67 6c65 ------www.google
0x04d0: 6c65 2e63 6f6d 5130 3433 01e8 8160 9292 le.comQ043...`..

23. QUIC header example (ClientHello)
Connection ID: x7bf28b4776840752 = 8931354150076417874
Padding to 1200 bytes to mitigate asymmetric DDoS attacks
IP 0x0000: 4500 0562 c6b0 4000 4011 c69e c0a8 0143 E..b..@.@......C
UDP, type 0x0010: 40e9 a567 c858 01bb 054e 0aed 0d7b f28b @..g.X...N...{..
version 0x0020: 4776 8407 5251 3034 3301 86eb 710d 6f62 ..t,.Q043...q.ob
ClientHello0x0030: a224 675b a416 a001 0400 4348 4c4f 1a00 .$g[......CHLO..
0x0040: 0000 5041 4400 0604 0000 534e 4900 6201 ..PAD.T...SNI.b.
0x0050: 0000 5354 4b00 9801 0000 5645 5200 9c01 ..STK.....VER...
...
0x00c0: 0000 2d2d 2d2d 2d2d 2d2d 2d2d 2d2d 2d2d ..--------------
padding ...
(x406) 0x04c0: 2d2d 2d2d 2d2d 7777 772e 676f 6f67 6c65 ------www.google
0x04d0: 6c65 2e63 6f6d 5130 3433 01e8 8160 9292 le.comQ043...`..

24. QUIC handshake
Crypto tags
●
CCS – Common Certificate Set
●
AEAD – authentication & encryption algorithm
●
KEXS – key exchange method
●
…and many others

25. QUIC: 0-RTT resumption
draft-ietf-quic-applicability-03.txt: 2 data copies are possible (~TFO)
0-RTT (draft-ietf-quic-tls-16.txt)
●
Protection with earlier or handshake keys
ClientHello
(0-RTT Application Data) -------->
ServerHello
{EncryptedExtensions}
{Finished}
<-------- [Application Data]
{Finished} -------->
[Application Data] <-------> [Application Data]

26. Streams
A “message” abstraction
Separate control streams from application data streams
A packet may include frames for different streams
A UDP datagram may include several packets
Length-encoded integers (in most significant bits)
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+
|Stream Type (8)|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Length (i) ...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Type (8) | Frame Payload (*) ...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

27. Stream life
Somewhat close to TCP connections
RST → RST_STREAM frame
FIN → CONNECTION_CLOSE or
APPLICATION_CLOSE frames
PING → keepalive probe
BLOCKED,STREAM_BLOCKED →
zero window announcement
o
| Create Stream (Sending)
| Create Bidirectional Stream (Receiving)
v
+-------+
| Ready | Send RST_STREAM
| |-----------------------.
+-------+ |
| |
| Send STREAM / |
| STREAM_BLOCKED |
| |
| Create Bidirectional |
| Stream (Receiving) |
v |
+-------+ |
| Send | Send RST_STREAM |
| |---------------------->|
+-------+ |
| |
| Send STREAM + FIN |
v v
+-------+ +-------+
| Data | Send RST_STREAM | Reset |
| Sent |------------------>| Sent |
+-------+ +-------+
| |
| Recv All ACKs | Recv ACK
v v
+-------+ +-------+
| Data | | Reset |
| Recvd | | Recvd |
+-------+ +-------+

28. Packet and stream frame example
192.168.1.67.50494 > 64.233.165.103.443: [udp sum ok] UDP, length 318
0x0000: 4500 015a 88e6 4000 4011 0871 c0a8 0143 E..Z..@.@..q...C
0x0010: 40e9 a567 c53e 01bb 0146 6b09 0c7b f28b @..g.>...Fk..{..
0x0020: 4776 8407 5203 80b8 139c 8545 7533 d5a1 Gv..R......Eu3..

29. HTTP headers compression
QPACK (~HPACK in HTTP/2, draft-ietf-quic-qpack-03.txt)
●
Shares Huffman encoding tables among asynchronous streams
●
Static & dynamic tables
●
Uses designated stream for the synchronization

30. QUIC: packet loss
draft-ietf-quic-recovery-16.txt
RTO, TLP, Fast & early retransmit, {S,F}ACK loss
recovery
ACK frames: for packets, retransmitted packet has a
new number
●
retransmitted frames with the same offset and
length (like TCP)
●
like TCP: delayed piggybacked ACKs
●
Like SACK: ACK blocks (no reneging)
Explicit Congestion Notification (ECN) [RFC3168]

31. Forward Error Correction
draft-pardue-quic-http-mcast-03.txt
no ACKs in multicast QUIC
gQUIC, removed in IETF, TBD: reconstruct 1 packet from 10

32. QUIC: congestion control
Chromium is a monster: BBR, CUBIC, PRR, slow start etc.
draft-ietf-quic-recovery-16.txt
NewReno (cwnd in bytes), slow start
Packets pacing
●
Packetization delays to bundle multiple frames
Frames
MAX_DATA, MAX_STREAM_DATA

33. Security considerations
Volumetric DDoS: opaque UDP traffic just like UDP flood
●
Middlebox filtration: ClientHello + Connection ID tracking
Aplification DDoS: minimal packet length for ClientHello
Stream fragmentation & reassembly → memory overcommit

34. QUIC in the kernel
User-space for rapid prototyping
The sendfile() problem (solved by kTLS for TLS)
● setsockopt(sd, SOL_UDP, UDP_ULP, "quic", sizeof("quic") - 1);
setsockopt(sd, SOL_QUIC, QUIC_CTX, &quic_ctx, sizeof(quic_ctx));
●
recvmsg() / sendmsg() - read/write frames vector
High CPU usage (ACK & Ko copies to user-space)
System wide memory accounting for all processes and connections
NIC acceleration (crypto offload, UDP segmentation offload)
First simple implementation (TBD): handshakes are in user-space

35. Open source implementations
Apache Traffic Server,
https://github.com/apache/trafficserver/tree/quic-latest (18KLoC)
H2O, https://github.com/h2o/quicly (9KLoC)
LiteSpeed Client Library, https://github.com/litespeedtech/lsquic-client (68KLoC)
https://github.com/ngtcp2/ngtcp2 (36KLoC)
Chromium, https://github.com/chromium/chromium (130KLoC)
Mozilla, https://github.com/mcmanus/mozquic (15KLoC)
NetApp, https://github.com/NTAP/quant (Netmap, 11KLoc)
https://github.com/private-octopus/picoquic (33KLoC)

36. References: documents
Current IETF drafts: https://datatracker.ietf.org/wg/quic/
QUIC Working Group: https://quicwg.org/
Discussions & open issues: https://github.com/quicwg/base-drafts/issues
Known implementations:
https://github.com/quicwg/base-drafts/wiki/Implementations

37. References: good to read & watch
“The QUIC Transport Protocol:Design and Internet-Scale Deployment”,
https://static.googleusercontent.com/media/research.google.com/en//pubs/archiv
e/46403.pdf
“QUIC: Developing and Deploying a TCP Replacement for the Web”,
https://www.netdevconf.org/0x12/session.html?quic-developing-and-deploying-a-
tcp-replacement-for-the-web
QUIC FAQ for Geeks,
https://docs.google.com/document/d/1lmL9EF6qKrk7gbazY8bIdvq3Pno2Xj_l_Y
ShP40GLQE/edit
https://en.wikipedia.org/wiki/QUIC
Thoughts on how to support QUIC in (lib)curl,
https://github.com/curl/curl/wiki/QUIC

38. Thanks!
Web-site: http://tempesta-tech.com
Tempesta FW: https://github.com/tempesta-tech/tempesta
Blog: http://natsys-lab.blogspot.com
E-mail: ak@tempesta-tech.com