Uncovering performance regressions in the TCP SACKs vulnerability fixes In early July 2019, Databricks noticed some Apache Spark workloads regressing by as much as 6x. In this talk, we'll discuss how we traced these regressions back to the Linux kernel and the fixes for the TCP SACKs vulnerabilities. We will explain the symptoms we were seeing, walk through how we debugged the TCP connections, and dive into the Linux source to uncover the root cause. Speaker: Chris Stevens (Databricks) Chris Stevens is a software engineer at Databricks where he works on the reliability, scalability, and security of Apache Spark clusters. His work focuses on auto-scaling compute, auto-scaling storage, node initialization performance, and node health monitoring. Prior to Databricks, Chris founded the Minoca OS project, where he built a POSIX compliant, general purpose OS - from scratch - to run on resource constrained device. He got his start at Microsoft working on the Windows kernel team, porting the Windows boot environment from BIOS to UEFI.

1. Adventures with TCP SACKs
SF Big Analytics
1
Chris Stevens
November 12, 2019

2. 2
A kernel developer in Big Data
● ~4 years working on the Windows NT kernel
● ~4 years building Minoca OS - https://github.com/minoca/os
● ~2 years at Databricks working on Apache Spark resource management

3. 3
Road Map
● A failing benchmark
● TCP SACKs explained
● Debugging TCP connections
● Root cause in the Linux Kernel

4. 4
6x slower benchmark

5. 5
6x slower benchmark
spark
.range(10 * BILLION)
.write
.format(“parquet”)
.mode(“overwrite”)
.save(“dbfs:/tmp/test”)

6. 6
Task Skew in the Spark UI

7. 7
Socket timeouts in the logs
org.apache.spark.SparkException:
... truncated ...
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception:
Your socket connection to the server was not read from or written
to within the timeout period. Idle connections will be closed.

8. 8
TCP SACKs Vulnerabilities
● June 17, 2019: CVE-2019-11477, CVE-2019-11478 & CVE-2019-11479
● Discovered by Jonathan Looney (Netflix Information Security)
● Remote attacker can ...
○ Trigger a kernel panic (CVE-2019-11477):
■ BUG_ON(tcp_skb_pcount(skb) < pcount);
○ Cause resource exhaustion (CVE-2019-11478, CVE-2019-11479).
● Mitigations:
○ Disable TCP SACKs
■ Set /proc/sys/net/ipv4/tcp_sack to 0
○ Update Linux kernel
■ Databricks updated on June 24, 2019

9. 9
TCP Protocol
Wikipedia (GNU Free Doc License)

10. 10
TCP without SACKs
Sender Receiver
Segment 1
Segment 1 Received
Segment 2
Segment 3
Segment 4 X
Segments [1-2] Received
Segment 3
Segment 4
Segments [1-4] Received

11. 11
TCP with SACKs
Sender Receiver
Segment 1
Segment 1 Received
Segment 2
Segment 3
Segment 4 X
Segments [1-2, 4] Received
Segment 3
Segments [1-4] Received

12. 12
TCP SACKs Attack
Sender Receiver
Seq = 1, Length = 8
Seq = 9, Length = 8
...
Seq = 801, Length = 8
MSS = 48
SACK = [9-16, 25-32,..., 801-808]

13. 13
TCP SACKs Resource Exhaustion
● SACK processing fragments Linux socket buffers
● Increases resource utilization to handle fragmentation (DoS)
sk_buff
nr_frags = 17
frags[MAX_SKB_FRAGS]
skb_shared_info
Frag 1
Frag 2
Frag 2
Frag 17
...
struct skb_frag_struct
{
Struct page *page;
__u32 page_offset;
__u32 size
}

14. 14
TCP SACKs Resource Exhaustion
● SACK processing fragments Linux socket buffers
● Increases resource utilization to handle fragmentation (DoS)
sk_buff
nr_frags = 1
frags[MAX_SKB_FRAGS]
skb_shared_info
Frag 1
Frag 2
Frag 2
Frag 17
...
sk_buff
nr_frags = 17
frags[MAX_SKB_FRAGS]
sk_shared_info

15. 15
TCP SACKs Regressions
● CVE-2019-11478
● June 17, 2019
○ tcp: tcp_fragment() should apply sane memory limits
● June 21, 2019
○ tcp: refine memory limit test in tcp_fragment()
● July 19, 2019
○ tcp: be more careful in tcp_fragment()
○ Ubuntu updates not available until Sept, 2019

16. 16
Is TCP SACKs the culprit?
● Benchmark regression aligned with Databricks’ June 24 kernel update.
● Impacted many versions of the Databricks Runtime (Apache Spark fork).
● Reverting just the kernel update fixed the benchmark.
○ No other platform updates at fault.

17. 17
Debugging the live repro

18. 18
Debugging the live repro
~# netstat
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp6 816 99571 ip-10-0-144-224.u:42464 s3-us-west-2-r-w.:https CLOSE_WAIT
Wikipedia (GNU Free
Doc License)

19. 19
Failed Retransmissions
~# ss -a -t -m -o --info
CLOSE-WAIT 816 99571 ::ffff:10.0.144.224:42464
::ffff:52.218.234.25:https timer: (on,1min37sec,13)
skmem:(r9792,rb236184,t0,tb46080,f732,w103971,o0,bl0) sack cubic wscale:8,7
rto:120000 backoff:13 rtt:0.379/0.055 ato:40 mss:1412 cwnd:1 ssthresh:23
bytes_acked:1398838 bytes_received:14186 segs_out:1085 segs_in:817 send
29.8Mbps lastsnd:592672 lastrcv:552644 lastack:550640 pacing_rate 4759.3Mbps
unacked:80 lost:24 sacked:56 rcv_space:26883
● Retransmission is enabled
● It’s made 13 retransmission attempts.
● It will try another retransmission in 1 minute 37 seconds.
● No packets sent in the last 9 minute 51 seconds.

20. 20
Connection Timeout Explained
● net.ipv4.tcp_retries2 defaults to 15
● TCP_RTO_MIN defaults to 200 milliseconds
● TCP_RTO_MAX defaults to 120 seconds
● Retransmission timeout is 15 mins 24 seconds.

21. 21
TCP Tracing
sudo tcpdump -nnS net 52.218.0.0/16 -w awstcp.pcap
● 22,138 bytes were dropped (SACK Left Edge minus ACK)
● 40 seconds of silence (expected 7 retransmissions)

22. 22
TCP Tracing
sudo tcpdump -nnS net 52.218.0.0/16 -w awstcp.pcap
● Sender enters CLOSE_WAIT when FIN is received.
● Unacknowledged 22,138 bytes were never retransmitted
● 77,433 bytes were SACK’d.
● 22,138 + 77,433 = 99,571 (from netstat Send-Q).
● The SACK’d bytes were never freed.

23. 23
Two Questions
● Why were the retransmissions failing?
● Why were the SACK’d bytes not freed?

24. 24
tcp_fragment()
if (unlikely((sk->sk_wmem_queued >> 1) > sk->sk_sndbuf)) {
NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPWQUEUETOOBIG);
return -ENOMEM;
}
if (unlikely((sk->sk_wmem_queued >> 1) > sk->sk_sndbuf &&
skb != tcp_send_head(sk))) {
NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPWQUEUETOOBIG);
return -ENOMEM;
}
Fixing the vulnerability: patch
First regression fix: patch
~# nstat | grep
“TcpExtTCPWqueueTooBig”
TcpExtTCPWqueueTooBig 179
0.0

25. 25
Retransmission Failures
int __tcp_retransmit_skb(struct sock *sk, struct sk_buff *skb)
{
...
if (skb->len > cur_mss) {
if (tcp_fragment(sk, skb, cur_mss, GFP_ATOMIC))
return -ENOMEM; /* We’ll try again later. */
}
...
}
● cur_mss = 1412 (from ss)
● TCP Segmentation Offloading is enabled, so skb->len should be greater than 1412.
● TCPRETRANSFAIL = 143

26. 26
Retransmission Failures
● “tcp_sendmsg() and tcp_sendpage() might overshoot sk_wmem_queued by about one
full [TCP Segmentation Offload] skb (64KB size).” - Linux commit message.
● sk->sk_wmem_queued = 103971 (from ss)
● sk->sk_sndbuf = 46080 (from ss)
● 103971 >> 1 = 51985 > 46080.
if (unlikely((sk->sk_wmem_queued >> 1) > sk->sk_sndbuf &&
skb != tcp_send_head(sk))) {
NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPWQUEUETOOBIG);
return -ENOMEM;
}

27. 27
ACK Failures
tcp_ack
-tcp_sacktag_write_queue
-tcp_sacktag_walk
-tcp_shift_skb_data
-tcp_match_skb_to_sack
-tcp_fragment
● TCP fragmentation is invoked if a socket buffer does not completely overlap with the SACK’d region.
● tcp_shift_skb_dataaborts if underlying fragmentation fails.

28. 28
tcp_fragment()
limit = sk->sk_sndbuf + 2 * SKB_TRUESIZE(GSO_MAX_SIZE);
if (unlikely((sk->sk_wmem_queued >> 1) > limit &&
skb != tcp_rtx_queue_head(sk) &&
skb != tcp_rtx_queue_tail(sk))) {
NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPWQUEUETOOBIG);
return -ENOMEM;
}
Third time’s a charm: patch
● sk->sk_wmem_queued = 103971 (from ss)
● sk->sk_sndbuf = 46080 (from ss)
● limit = 46080 + 2 * 65536 = 177152
○ Adds 128KB overhead to account for overshooting sk_wmem_queued by 64KB.
● 103971 >> 1 = 51985 < 177152.

29. 29
Lessons Learned
● Consider the whole stack
○ We’ve hit 2 other, unrelated Linux bugs since.
● Benchmarks are just numbers; logging is vital.
● Know the right tools.
○ ss was a lifesaver.

30. 30
Thanks!
Chris Stevens - linkedin.com/in/chriscstevens