The document discusses TCP performance for mobile applications. It describes the networking stack with layers for the application, session, transport, network, and data link. It then provides examples of packet captures for a TCP connection between localhost ports, showing the handshake, data transfer, and connection termination.

1. TCP Performance
For Mobile Applications
Vladimir Kirillov
@darkproger

2. Networking Stack

3. Application
Session
Transport
Network
Data Link

4. Application HTTP
Session TLS
Transport TCP
Network IP
WiFi Edge
Data Link
3G LTE

5. Application HTTP NSURLConnection
Session TLS OpenSSL
Transport TCP SOCK_STREAM
Network IP kernel
WiFi Edge
Data Link hardware
3G LTE
Level Protocol API / Implementation

6. HTTP NSURLConnection ptrace
TLS OpenSSL socket
dtrace
API
TCP SOCK_STREAM
IP kernel bpf(4)
WiFi Edge
hardware
3G LTE
Protocol API / Implementation Introspection

7. capturing iPhone traffic
% udid=$(system_profiler SPUSBDataType
| awk '/iPhone/{go=1} /Serial/ {if (go) print $3; go=0}')
276cb9530201bcehelloworldcd55560ed015d00
% rvictl -s $udid
Starting device 276cb9530201bcehelloworldcd55560ed015d00
[SUCCEEDED]
% ifconfig rvi0
rvi0: flags=3005<UP,DEBUG,LINK0,LINK1> mtu 0

8. tcpdump -i lo0 -w t.pcap -s0 &
nc -l 5000 &
echo hello | nc localhost 5000
kill %1

9. # tcpdump -r t.pcap -nnvv -tttt -K 'tcp port 5000'
2012-11-24 12:23:35.511134 IP6 (hlim 64, next-header TCP (6) payload length: 44) ::1.51734 > ::1.5000: Flags [ S ], seq 453038127, win
65535, options [mss 16324,nop,wscale 4,nop,nop,TS val 303407352 ecr 0,sackOK,eol], length 0
2012-11-24 12:23:35.511175 IP6 (hlim 64, next-header TCP (6) payload length: 20) ::1.5000 > ::1.51734: Flags [ R .], seq 0, ack 453038128,
win 0, length 0
2012-11-24 12:23:35.511226 IP (tos 0x0, ttl 64, id 8400, offset 0, flags [DF], proto TCP (6), length 64)
S
127.0.0.1.51735 > 127.0.0.1.5000: Flags [ ], seq 2527137802, win 65535, options [mss 16344,nop,wscale 4,nop,nop,TS val 303407352 ecr
0,sackOK,eol], length 0
2012-11-24 12:23:35.511276 IP (tos 0x0, ttl 64, id 58311, offset 0, flags [DF], proto TCP (6), length 64)
127.0.0.1.5000 > 127.0.0.1.51735: Flags S
[ .], seq 494520280, ack 2527137803, win 65535, options [mss 16344,nop,wscale 4,nop,nop,TS val
303407352 ecr 303407352,sackOK,eol], length 0
2012-11-24 12:23:35.511287 IP (tos 0x0, ttl 64, id 47796, offset 0, flags [DF], proto TCP (6), length 52)
127.0.0.1.51735 > 127.0.0.1.5000: Flags [.], seq 1, ack 1, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 0
2012-11-24 12:23:35.511298 IP (tos 0x0, ttl 64, id 52186, offset 0, flags [DF], proto TCP (6), length 52)
127.0.0.1.5000 > 127.0.0.1.51735: Flags [.], seq 1, ack 1, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 0
2012-11-24 12:23:35.511332 IP (tos 0x0, ttl 64, id 31417, offset 0, flags [DF], proto TCP (6), length 58)
127.0.0.1.51735 > 127.0.0.1.5000: Flags [P .], seq 1:7, ack 1, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 6
2012-11-24 12:23:35.511351 IP (tos 0x0, ttl 64, id 29060, offset 0, flags [DF], proto TCP (6), length 52)
127.0.0.1.51735 > 127.0.0.1.5000: Flags [F.], seq 7, ack 1, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 0
2012-11-24 12:23:35.511354 IP (tos 0x0, ttl 64, id 4019, offset 0, flags [DF], proto TCP (6), length 52)
127.0.0.1.5000 > 127.0.0.1.51735: Flags [.], seq 1, ack 7, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 0
2012-11-24 12:23:35.511367 IP (tos 0x0, ttl 64, id 20879, offset 0, flags [DF], proto TCP (6), length 52)
127.0.0.1.5000 > 127.0.0.1.51735: Flags [.], seq 1, ack 8, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 0
17 packets captured
2012-11-24 12:23:35.511378 IP (tos 0x0, ttl 64, id 59633, offset 0, flags [DF], proto TCP (6), length 52)
127.0.0.1.51735 > 127.0.0.1.5000: Flags [F.], seq 7, ack 1, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 0
2012-11-24 12:23:35.511388 IP (tos 0x0, ttl 64, id 56794, offset 0, flags [DF], proto TCP (6), length 52)
127.0.0.1.5000 > 127.0.0.1.51735: Flags [F.], seq 1, ack 8, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 0

10. # tcpdump -r t.pcap -nnvv -tttt -K 'tcp port 5000'
2012-11-24 12:23:35.511134 IP6 (hlim 64, next-header TCP (6) payload length: 44) ::1.51734 > ::1.5000: Flags [ S ], seq 453038127, win
65535, options [mss 16324,nop,wscale 4,nop,nop,TS val 303407352 ecr 0,sackOK,eol], length 0
2012-11-24 12:23:35.511175 IP6 (hlim 64, next-header TCP (6) payload length: 20) ::1.5000 > ::1.51734: Flags [ R .], seq 0, ack 453038128,
win 0, length 0
2012-11-24 12:23:35.511226 IP (tos 0x0, ttl 64, id 8400, offset 0, flags [DF], proto TCP (6), length 64)
S
127.0.0.1.51735 > 127.0.0.1.5000: Flags [ ], seq 2527137802, win 65535, options [mss 16344,nop,wscale 4,nop,nop,TS val 303407352 ecr
0,sackOK,eol], length 0
2012-11-24 12:23:35.511276 IP (tos 0x0, ttl 64, id 58311, offset 0, flags [DF], proto TCP (6), length 64)
127.0.0.1.5000 > 127.0.0.1.51735: Flags S
[ .], seq 494520280, ack 2527137803, win 65535, options [mss 16344,nop,wscale 4,nop,nop,TS val
303407352 ecr 303407352,sackOK,eol], length 0
2012-11-24 12:23:35.511287 IP (tos 0x0, ttl 64, id 47796, offset 0, flags [DF], proto TCP (6), length 52)
127.0.0.1.51735 > 127.0.0.1.5000: Flags [.], seq 1, ack 1, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 0
2012-11-24 12:23:35.511298 IP (tos 0x0, ttl 64, id 52186, offset 0, flags [DF], proto TCP (6), length 52)
127.0.0.1.5000 > 127.0.0.1.51735: Flags [.], seq 1, ack 1, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 0
2012-11-24 12:23:35.511332 IP (tos 0x0, ttl 64, id 31417, offset 0, flags [DF], proto TCP (6), length 58)
127.0.0.1.51735 > 127.0.0.1.5000: Flags [P length 6
.], seq 1:7, ack 1, win 9186, options [nop,nop,TS val 303407352 ecr 303407352],
2012-11-24 12:23:35.511351 IP (tos 0x0, ttl 64, id 29060, offset 0, flags [DF], proto TCP (6), length 52)
127.0.0.1.51735 > 127.0.0.1.5000: Flags
2012-11-24 12:23:35.511354 IP (tos 0x0, ttl
127.0.0.1.5000 > 127.0.0.1.51735: Flags
64, id 4019, offset 0, flags [DF], proto TCP (6), length 52) ^^
[F.], seq 7, ack 1, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 0
[.], seq 1, ack 7, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 0
2012-11-24 12:23:35.511367 IP (tos 0x0, ttl
127.0.0.1.5000 > 127.0.0.1.51735: Flags % stat -f %z t.pcap
64, id 20879, offset 0, flags [DF], proto TCP (6), length 52)
[.], seq 1, ack 8, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 0
17 packets captured 1306
2012-11-24 12:23:35.511378 IP (tos 0x0, ttl 64, id 59633, offset 0, flags [DF], proto TCP (6), length 52)
127.0.0.1.51735 > 127.0.0.1.5000: Flags [F.], seq 7, ack 1, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 0
2012-11-24 12:23:35.511388 IP (tos 0x0, ttl 64, id 56794, offset 0, flags [DF], proto TCP (6), length 52)
127.0.0.1.5000 > 127.0.0.1.51735: Flags [F.], seq 1, ack 8, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 0

11. % tcptrace t.pcap
17 packets seen, 17 TCP packets traced
elapsed wallclock time: 0:00:00.001344, 12648 pkts/sec analyzed
trace file elapsed time: 0:00:00.000305
TCP connection info:
1: localhost:52132 - localhost:5000 (a2b) 1> 1< (reset)
2: localhost:52133 - localhost:5000 (c2d) 8> 7< (complete) (reset)

12. % tcptrace -o2 -l t.pcap
...
adv wind scale: 4 adv wind scale: 4
req sack: Y req sack: Y
sacks sent: 0 sacks sent: 0
urgent data pkts: 0 pkts urgent data pkts: 0 pkts
urgent data bytes: 0 bytes urgent data bytes: 0 bytes
mss requested: 16344 bytes mss requested: 16344 bytes
max segm size: 6 bytes max segm size: 0 bytes
min segm size: 6 bytes min segm size: 0 bytes
avg segm size: 5 bytes avg segm size: 0 bytes
max win adv:146976 bytes max win adv: 146976 bytes
min win adv:146976 bytes min win adv: 146976 bytes
zero win adv: 0 times zero win adv: 0 times
avg win adv: 146976 bytes avg win adv: 122480 bytes
initial window:6 bytes initial window: 0 bytes
initial window: 1 pkts initial window: 0 pkts
ttl stream length: 6 bytes ttl stream length: 1 bytes
missed data: 0 bytes missed data: 1 bytes
truncated data: 0 bytes truncated data: 0 bytes
truncated packets: 0 pkts truncated packets: 0 pkts
data xmit time: 0.000 secs data xmit time: 0.000 secs
idletime max: 0.1 ms idletime max: 0.0 ms
throughput: 27027 Bps throughput: 0 Bps

13. endpoint endpoint
SO_RCVBUF SO_SNDBUF
SO_SNDBUF SO_RCVBUF

14. endpoint endpoint
SO_RCVBUF SEG SO_SNDBUF
SO_SNDBUF SEG SO_RCVBUF

15. 2 * LATENCY = RTT
endpoint LATENCY endpoint
SO_RCVBUF SEG SO_SNDBUF
BANDWIDTH
SO_SNDBUF SEG SO_RCVBUF

16. • Time from one endpoint to another
• Each connection spans multiple links
• edge
• ethernet
Latency • fiber
• wifi
• latency = sum (lat foreach link)
• RTT = 2 * latency

17. • Number of bytes a link can handle
Bandwidth • bw = min (bw foreach link)

18. Bandwidth
Delay BDP = RTT * BANDWIDTH
Product

19. client server
SO_RCVBUF receiver window SO_SNDBUF
SO_SNDBUF sender window SO_RCVBUF

20. client server
SO_RCVBUF SEG receiver window SEG SO_SNDBUF
SO_SNDBUF sender window
SEG SEG SO_RCVBUF

21. client server
SO_RCVBUF SEG receiver window
SEG SEG SEG SEG SEG SO_SNDBUF
SO_SNDBUF SEG receiver windowSEG
sender window
SEG SEG
SEG SEG SEG SEG SO_RCVBUF

22. TCP • stateful
• ordered
byte • reliable
stream • managed

23. HTTP
paired
has state
TLS
no state
TCP
IP

24. % openssl s_client -
showcerts -connect
google.com:443
CONNECTED(00000003)
depth=1 /C=US/O=Google Inc/CN=Google Internet Authority
verify error:num=20:unable to get local issuer certificate
TLS
verify return:0
---
Certificate chain
0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/
CN=*.google.com
i:/C=US/O=Google Inc/CN=Google Internet Authority
...
1 s:/C=US/O=Google Inc/CN=Google Internet Authority
i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority
...
SSL handshake has read 2388 bytes and written 316 bytes

25. % openssl s_client -showcerts -
connect ciklum.com:443
CONNECTED(00000003)
ciklum.net
depth=0 /C=UA/OU=Domain Control Validated/CN=*.
verify error:num=20:unable to get local issuer certificate
verify return:1 lol :)
depth=0 /C=UA/OU=Domain Control Validated/CN=*.ciklum.net
verify error:num=27:certificate not trusted
verify return:1
TLS depth=0 /C=UA/OU=Domain Control Validated/CN=*.ciklum.net
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
0 s:/C=UA/OU=Domain Control Validated/CN=*.ciklum.net
i:/O=AlphaSSL/CN=AlphaSSL CA - G2
...
Server certificate
subject=/C=UA/OU=Domain Control Validated/CN=*.ciklum.net
issuer=/O=AlphaSSL/CN=AlphaSSL CA - G2
SSL handshake has read 1854 bytes and written 328 bytes

26. HTTP 1 RTT
TLS 3 RTTs
TCP 1 RTT
IP
It takes 4 RTTs to serve a HTTPS request

27. HTTP 1 RTT
TLS 3 RTTs
TCP 1 RTT
IP
It takes 4 RTTs to serve a HTTPS request

28. TCP Reliability

29. client server
SO_RCVBUF SEG receiver window
SEG SEG SEG SEG SEG SO_SNDBUF
A A A A
SO_SNDBUF C receiver window
sender window
C SEG SEG
C C SO_RCVBUF
K K K K

30. AirPort router router
client Express server
SO_RCVBUF SEG receiver window
SEG SEG SEG SEG SEG SO_SNDBUF
A A A A
SO_SNDBUF C receiver window
sender window
C SEG SEG
C C SO_RCVBUF
K K K K
router

31. AirPort router router
client Express server
SO_RCVBUF SEG receiver window
SEG SEG SEG SEG SEG SO_SNDBUF
A A A A
SO_SNDBUF C receiver window
sender window
C SEG SEG
C C SO_RCVBUF
K K K K
router
retransmit on timeout

32. TCP Congestion Control

33. client server
SEG SEG SEG receiver
SEG
SO_RCVBUF
SEG SEG SEG SEG
SEG windowSEG
SEG SO_SNDBUF
SO_SNDBUF sender window SO_RCVBUF

34. AirPort router router
client Express server
SO_RCVBUF SEG receiver window
SEG SEG SEG SEG SEG SO_SNDBUF
A A A
SO_SNDBUF C receiver window
sender window
C C SO_RCVBUF
K K K
overloaded
router
^^^ What congestion control is actually designed for

35. AirPort router router
client Express server
S S S S
SO_RCVBUF E E E E SEG receiver window
SEG SEG SEG SEG SEG SO_SNDBUF
G G G G
A A A A
SO_SNDBUF C sender window
receiverSEG
C window C
SEG C SO_RCVBUF
K K K K
router
^^^ What actually happens on mobile devices

36. • Nagle algorithm
while (1)
write(fd, “5”, 1);
TCP (telnet syndrom)
Artifacts
Delayed ACK
http://www.stuartcheshire.org/
papers/NagleDelayedAck/

37. TCP •SO_OOBINLINE
Artifacts •TCP URG

38. API Issues

39. Async NSURLConnection
API Issues UIScrollView
CFRunLoopAddCommonMode

40. client server
SEG SEG
SEG
SEG SEG
SO_RCVBUF
SO_RCVBUF SEG SEG SEG SEG
SEG receiver window SO_SNDBUF
SO_SNDBUF sender window SO_RCVBUF

41. More • Additive Increase
Congestion • Multiplicative Decrease
Control • Slow Start

43. • setsockopt(2)
• adjust window size
• socket buffer sizes
• TCP_NODELAY
iOS / Unix • etc
•getsockopt(2)
• monitoring
• low-latency responding to socket events
• do not let the buffer stay full

44. • Responsive UI
• Bad example: Instagram
Speedup • Bad example:VK
• best UI
• worst reliability

45. • TCP Fast Open
• Linux 3.6
• HAProxy
• TCP/NC
Speedup • TCP and math (maths)
• http://dspace.mit.edu/openaccess-
disseminate/1721.1/58796
• TCP Westwood+

46. • TLS False Start
• TLS NPN
Speedup • Next Protocol Negotiation
• HTTP Pipelining
• SPDY

47. kthxbai
@darkproger
http://kirillov.im