Проблемы использования TCP в мобильных приложениях. Владимир Кириллов

TCP Performance
For Mobile Applications
Vladimir Kirillov
@darkproger

Application

Session

Transport

Network

Data Link

Application HTTP

Session TLS

Transport TCP

Network IP

WiFi Edge
Data Link
3G LTE

Application HTTP (Http|NS)URLConnection

Session TLS OpenSSL

Transport TCP SOCK_STREAM

Network IP kernel
WiFi Edge
Data Link hardware
3G LTE

Level Protocol API / Implementation

gdb
HTTP (Http|NS)URLConnection
ptrace
TLS OpenSSL
socket dtrace
API
TCP SOCK_STREAM

bpf(4)
IP kernel
LSF
WiFi Edge
hardware
3G LTE

Protocol API / Implementation Introspection

capturing iPhone traffic

% udid=$(system_profiler SPUSBDataType
| awk '/iPhone/{go=1} /Serial/ {if (go) print $3; go=0}')
276cb9530201bcehelloworldcd55560ed015d00

% rvictl -s $udid
Starting device 276cb9530201bcehelloworldcd55560ed015d00
[SUCCEEDED]
% ifconfig rvi0
rvi0: flags=3005<UP,DEBUG,LINK0,LINK1> mtu 0

capturing Android traffic

# adb connect 192.168.56.100
# adb shell
shell@android:/ $ su
Test prop
su allows access thanks to
androVM.su.bypass property
shell@android:/ # tcpdump -i eth1

tcpdump -i lo0 -w t.pcap -s0 &
nc -l 5000 &
echo hello | nc localhost 5000
kill %1

# tcpdump -r t.pcap -nnvv -tttt -K 'tcp port 5000'

2012-11-24 12:23:35.511134 IP6 (hlim 64, next-header TCP (6) payload length: 44) ::1.51734 > ::1.5000: Flags [ S ], seq 453038127, win
65535, options [mss 16324,nop,wscale 4,nop,nop,TS val 303407352 ecr 0,sackOK,eol], length 0

2012-11-24 12:23:35.511175 IP6 (hlim 64, next-header TCP (6) payload length: 20) ::1.5000 > ::1.51734: Flags [ R .], seq 0, ack 453038128,
win 0, length 0

2012-11-24 12:23:35.511226 IP (tos 0x0, ttl 64, id 8400, offset 0, flags [DF], proto TCP (6), length 64)

S
127.0.0.1.51735 > 127.0.0.1.5000: Flags [ ], seq 2527137802, win 65535, options [mss 16344,nop,wscale 4,nop,nop,TS val 303407352 ecr
0,sackOK,eol], length 0

127.0.0.1.5000 > 127.0.0.1.51735: Flags S
[ .], seq 494520280, ack 2527137803, win 65535, options [mss 16344,nop,wscale 4,nop,nop,TS val
303407352 ecr 303407352,sackOK,eol], length 0
127.0.0.1.51735 > 127.0.0.1.5000: Flags [.], seq 1, ack 1, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 0

127.0.0.1.51735 > 127.0.0.1.5000: Flags [P .], seq 1:7, ack 1, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 6
127.0.0.1.51735 > 127.0.0.1.5000: Flags [F.], seq 7, ack 1, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 0

17 packets captured

# tcpdump -r t.pcap -nnvv -tttt -K 'tcp port 5000'

2012-11-24 12:23:35.511134 IP6 (hlim 64, next-header TCP (6) payload length: 44) ::1.51734 > ::1.5000: Flags [ S ], seq 453038127, win
65535, options [mss 16324,nop,wscale 4,nop,nop,TS val 303407352 ecr 0,sackOK,eol], length 0

2012-11-24 12:23:35.511175 IP6 (hlim 64, next-header TCP (6) payload length: 20) ::1.5000 > ::1.51734: Flags [ R .], seq 0, ack 453038128,
win 0, length 0


S
127.0.0.1.51735 > 127.0.0.1.5000: Flags [ ], seq 2527137802, win 65535, options [mss 16344,nop,wscale 4,nop,nop,TS val 303407352 ecr
0,sackOK,eol], length 0

127.0.0.1.5000 > 127.0.0.1.51735: Flags S
[ .], seq 494520280, ack 2527137803, win 65535, options [mss 16344,nop,wscale 4,nop,nop,TS val
303407352 ecr 303407352,sackOK,eol], length 0

127.0.0.1.51735 > 127.0.0.1.5000: Flags [P length 6
.], seq 1:7, ack 1, win 9186, options [nop,nop,TS val 303407352 ecr 303407352],
127.0.0.1.51735 > 127.0.0.1.5000: Flags
2012-11-24 12:23:35.511354 IP (tos 0x0, ttl
127.0.0.1.5000 > 127.0.0.1.51735: Flags
64, id 4019, offset 0, flags [DF], proto TCP (6), length 52) ^^
[F.], seq 7, ack 1, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 0

[.], seq 1, ack 7, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 0
2012-11-24 12:23:35.511367 IP (tos 0x0, ttl
127.0.0.1.5000 > 127.0.0.1.51735: Flags % stat -f %z t.pcap
64, id 20879, offset 0, flags [DF], proto TCP (6), length 52)
[.], seq 1, ack 8, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 0

17 packets captured 1306

% tcptrace t.pcap
17 packets seen, 17 TCP packets traced
elapsed wallclock time: 0:00:00.001344, 12648 pkts/sec analyzed
trace file elapsed time: 0:00:00.000305
TCP connection info:

1: localhost:52132 - localhost:5000 (a2b) 1> 1< (reset)
2: localhost:52133 - localhost:5000 (c2d) 8> 7< (complete) (reset)

% tcptrace -o2 -l t.pcap
...

adv wind scale: 4 adv wind scale: 4
req sack: Y req sack: Y

sacks sent: 0 sacks sent: 0
urgent data pkts: 0 pkts urgent data pkts: 0 pkts
urgent data bytes: 0 bytes urgent data bytes: 0 bytes

mss requested: 16344 bytes mss requested: 16344 bytes
max segm size: 6 bytes max segm size: 0 bytes
min segm size: 6 bytes min segm size: 0 bytes
avg segm size: 5 bytes avg segm size: 0 bytes

max win adv:146976 bytes max win adv: 146976 bytes

min win adv:146976 bytes min win adv: 146976 bytes

zero win adv: 0 times zero win adv: 0 times
avg win adv: 146976 bytes avg win adv: 122480 bytes

initial window:6 bytes initial window: 0 bytes
initial window: 1 pkts initial window: 0 pkts
ttl stream length: 6 bytes ttl stream length: 1 bytes

missed data: 0 bytes missed data: 1 bytes
truncated data: 0 bytes truncated data: 0 bytes
truncated packets: 0 pkts truncated packets: 0 pkts
data xmit time: 0.000 secs data xmit time: 0.000 secs
idletime max: 0.1 ms idletime max: 0.0 ms
throughput: 27027 Bps throughput: 0 Bps

endpoint endpoint
SO_RCVBUF SO_SNDBUF

SO_SNDBUF SO_RCVBUF

endpoint endpoint
SO_RCVBUF SEG SO_SNDBUF

SO_SNDBUF SEG SO_RCVBUF

2 * LATENCY = RTT

endpoint LATENCY endpoint
SO_RCVBUF SEG SO_SNDBUF

BANDWIDTH
SO_SNDBUF SEG SO_RCVBUF

• Time from one endpoint to another
• Each connection spans multiple links
Latency
• latency = sum (lat foreach link)
• RTT = 2 * latency

• Number of bytes a link can handle
Bandwidth
• bw = min (bw foreach link)

Bandwidth
Delay BDP = RTT * BANDWIDTH

Product

client server
SO_RCVBUF receiver window SO_SNDBUF

SO_SNDBUF sender window SO_RCVBUF

client server
SO_RCVBUF SEG receiver window SEG SO_SNDBUF

SO_SNDBUF sender window
SEG SEG SO_RCVBUF

client server
SO_RCVBUF SEG receiver window
SEG SEG SEG SEG SEG SO_SNDBUF

SO_SNDBUF SEG receiver windowSEG
sender window
SEG SEG
SEG SEG SEG SEG SO_RCVBUF

TCP • stateful
• ordered
byte • reliable
stream • managed

HTTP
paired
TLS
has state
TCP
no state
IP

HTTP
TLS
SYN
TCP 1 RTT SYN,ACK
ACK
IP

"Oh, a SSL certiﬁcate warning.

I'll read it carefully and
understand the possible
TLS implications before
proceeding.”

-- no User, ever.

"Oh, a SSL library.

I'll understand carefully its
semantics and will not break
TLS authentication.”

-- unknown developer.

% openssl s_client -
showcerts -connect
TLS internet.velcom.by:443

CONNECTED(00000003)
depth=3 Thawte Premium Server CA

verify error:num=19:self signed certificate in
certificate chain
verify return:0

Certificate chain
0 s:/C=BY/ST=Minsk/L=Minsk/O=FE Velcom/
CN=internet.velcom.by
i:/C=US/O=Thawte, Inc./CN=Thawte SSL CA

-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----

1 s:/C=US/O=Thawte, Inc./CN=Thawte SSL CA
i:/C=US/O=thawte, Inc./OU=Certification Services
Division/OU=(c) 2006 thawte, Inc. - For authorized use
only/CN=thawte Primary Root CA

...

2 s:/C=US/O=thawte, Inc./OU=Certification Services
Division/OU=(c) 2006 thawte, Inc. - For authorized use
only/CN=thawte Primary Root CA
i:/C=ZA/ST=Western Cape/L=Cape Town/O=Thawte
Consulting cc/OU=Certification Services Division/
CN=Thawte Premium Server CA/emailAddress=premium-
server@thawte.com

...

3 s:/C=ZA/ST=Western Cape/L=Cape Town/O=Thawte
server@thawte.com
i:/C=ZA/ST=Western Cape/L=Cape Town/O=Thawte
server@thawte.com

...

Server certificate
subject=/C=BY/ST=Minsk/L=Minsk/O=FE Velcom/
CN=internet.velcom.by
issuer=/C=US/O=Thawte, Inc./CN=Thawte SSL CA

SSL handshake has read 4736 bytes and written 328 bytes

% openssl s_client -showcerts -
connect ciklum.com:443
CONNECTED(00000003)

depth=0 /C=UA/OU=Domain Control Validated/CN=*. ciklum.net
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 /C=UA/OU=Domain Control Validated/CN=*.ciklum.net
verify error:num=27:certificate not trusted

TLS verify return:1
depth=0 /C=UA/OU=Domain Control Validated/CN=*.ciklum.net
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
0 s:/C=UA/OU=Domain Control Validated/CN=*.ciklum.net
i:/O=AlphaSSL/CN=AlphaSSL CA - G2
...
Server certificate
subject=/C=UA/OU=Domain Control Validated/CN=*.ciklum.net
issuer=/O=AlphaSSL/CN=AlphaSSL CA - G2

SSL handshake has read 1854 bytes and written 328 bytes

HTTP SYN
SYN,ACK
TLS 2 RTTs ACK, ClientHello
ServerHello, Certiﬁcate
TCP 1 RTT ClientKEX, ChangeCipherSpec
ChangeCipherSpec,Finished
IP

SYN
HTTP 1 RTT SYN,ACK
ACK, ClientHello
TLS 2 RTTs ServerHello, Certiﬁcate
ClientKEX, ChangeCipherSpec
TCP 1 RTT ChangeCipherSpec,Finished
GET
OK
IP

It takes 4 RTTs to serve a HTTPS request

client server

A A A A
SO_SNDBUF C receiver window
sender window
C SEG SEG
C C SO_RCVBUF
K K K K

AirPort router router
client Express server

A A A A
sender window
C SEG SEG
C C SO_RCVBUF
K K K K
router


A A A A
sender window
C SEG SEG
C C SO_RCVBUF
K K K K
router

retransmit on timeout (~200ms)

client server
SEG SEG SEG receiver
SEG
SO_RCVBUF
SEG SEG SEG SEG
SEG windowSEG
SEG SO_SNDBUF



A A A
sender window
C C SO_RCVBUF
K K K

overloaded
router

^^^ What congestion control is actually designed for

Crappy Wi-Fi

S S S S
SO_RCVBUF E E E E SEG receiver window
G G G G
A A A A
SO_SNDBUF C sender window
receiverSEG
C window C
SEG C SO_RCVBUF
K K K K
router

^^^ What actually happens on mobile devices

• Nagle algorithm
while (1)
write(fd, “5”, 1);
TCP (telnet syndrom)
Artifacts
Delayed ACK
http://www.stuartcheshire.org/
papers/NagleDelayedAck/

•SO_OOBINLINE
TCP
•TCP URG
Artifacts •RFC 6093

Async NSURLConnection
API Issues UIScrollView
CFRunLoopAddCommonMode

client server
SEG SEG
SEG
SEG SEG
SO_RCVBUF
SO_RCVBUF SEG SEG SEG SEG
SEG receiver window SO_SNDBUF


Congestion
• Additive Increase
Avoidance Multiplicative Decrease
•
• Slow Start
TCP Reno

# cat /proc/sys/net/ipv4/tcp_slow_start_after_idle
1
# cat /proc/sys/net/ipv4/tcp_no_metrics_save
0

# echo 0 > /proc/sys/net/ipv4/tcp_slow_start_after_idle
# echo 1 > /proc/sys/net/ipv4/tcp_no_metrics_save

# find /proc/sys/net/ipv4 | grep cong | xargs -tn1 cat

cat /proc/sys/net/ipv4/tcp_allowed_congestion_control
cubic reno
cat /proc/sys/net/ipv4/tcp_available_congestion_control
cubic reno
cat /proc/sys/net/ipv4/tcp_congestion_control
cubic

# ip route show
default via 192.168.56.1 dev eth1 initcwnd 10 initrwnd 10

• setsockopt(2)
• adjust window size
• socket buffer sizes
• TCP_NODELAY (Nagle)
Sockets • etc
•getsockopt(2)
• monitoring
• low-latency responding to socket events
• do not let the buffer stay full

getsockopt(SOL_TCP, TCP_INFO)
ESTAB 0 176 10.1.1.1:22 10.1.1.2:61984 users:(("sshd",18989,3))
!
mem:(r0,w1168,f2928,t0)

ts sack bic wscale:4,5 rto:280

rtt:56.25/7.5 ato:40 cwnd:8 ssthresh:7

send 1.6Mbps rcv_rtt:50 rcv_space:14480

#include <linux/tcp.h>

iproute2

for _i in $(seq 10);
ssh -f thailand cat

for _i in $(seq 10);
ssh
-o 'ControlMaster yes'
-f thailand cat

• Instagram
Responsive • VK
UI • best UI
• worst reliability

• TCP Fast Open
Steroids • Linux 3.6
• HAProxy

• TCP/NC
• TCP and math (maths)
Steroids
• http://dspace.mit.edu/openaccess-
disseminate/1721.1/58796

• TCP Westwood+ (LFN)
• TCP Veno (Wi-Fi)
Scheduling, • http://www.apan.net/meetings/
Algorithms honolulu2004/materials/engineering/
APAN_ppt.pdf
•CONF_TCP_CONG_VENO

• TLS False Start
• TLS NPN
Steroids • Next Protocol Negotiation
• HTTP Pipelining
• SPDY

• https://github.com/proger/iproute2
ss -I
• https://github.com/proger/captcp
Research • tcptrace
• tcpﬂow
• monitoring

kthxbai
@darkproger

http://kirillov.im

Проблемы использования TCP в мобильных приложениях. Владимир Кириллов

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (7)

Similar to Проблемы использования TCP в мобильных приложениях. Владимир Кириллов

Similar to Проблемы использования TCP в мобильных приложениях. Владимир Кириллов (20)

More from Anthony Marchenko

More from Anthony Marchenko (17)

Проблемы использования TCP в мобильных приложениях. Владимир Кириллов