Tcpip

TCP / IP
Detailed for the masses

Hi
 Julien PAULI
 Programming in PHP since early 2000s
 PHP Internals hacker and trainer
 PHP 5.5/5.6 Release Manager
 Working at SensioLabs in Paris - Blackfire
 Writing PHP tech articles and books
 http://phpinternalsbook.com
 @julienpauli - github.com/jpauli - jpauli@php.net
 Like working on OSS such as PHP :-)

TOC
 We can't explain "the network" , in an hour or
so (sorry)

TOC
 We can't explain the network , in an hour
 Even if that latter is not that hard to understand
 In fact, it is easy to spot it in its whole, but long

TOC
 Go further by yourself with highly
recommanded books =>

TOC (the real one)
 Reminder on OSI
 Brief presentation of IP : Internet Protocol
 Deep dive into TCP
 Conclusion

TOC (detailed)
 Deep dive into TCP
 The need of reliability on top of IP
 ARQ: Automatic Repeat reQuest and acknowledgments
 TCP header overview
 Pipelined reliable Service model
 Connected model and FSM
 Data Flow, segmentation, MSS and sliding Window
mechanism
 Congestion control and Avoidance algorithms
 Tuning TCP on Linux
 Wireshark examples

Everything is under control
 Every concept is
 Logical (common sense)
 Maths oriented
 Fully understandable (engineer capabilities assumed)
 TCP is not that hard, it is purely logical
 TCP is not that huge, it mixes several concepts
 TCP is a protocol, a binary language talked
between several hosts
 TCP is technically really beautiful

Mantra
 TCP/IP runs the Internet since 1970, aka 90% of it
(very huge part of it)
 TCP runs over another protocol, usually over IP
 TCP/IP used to be one layer, and one protocol
 They got separated to respect OSI
 IP is layer 3 , TCP is layer 4 : they are used together
 TCP/IP understanding helps a lot in designing other
(Layer 7) protocols
 A simple TCP implementation requires only several
hundreds LOC (C lang)

TCP/IP model
 Just a conceptual simplification of OSI
 OSI 5-6-7 and OSI 1-2 are commonly merged

Vocabulary (only)
 Every layer adds its infos
 Every layer carries some data
 But we used to give different wordings to them
Frame
Datagram
Segment (also packet)
payload, packet, … ?

IP
 Internet protocol
 Sends a datagram from place A to place B
 A could be several thousands Km far from B
 Datagram will be routed, through routers (packet-
switched network)
 from 1 router, to several dozens, hundreds
 Datagram may or may not arrive at destination
 Datagrams may arrive in different order
 "Best effort" protocol
 Do what you can to route info, as fast as possible
 Do not bother to "check" anything, just route, and go

IP Forward
 Routers analyze the destination address field
 They route the IP datagram on connected
networks
 Datagram may get destroyed (router overload)
 Datagram may get lost (iterconnexion problem)
 Datagram may get altered, falsified (noise)
 Datagram may arrive good, but the host is
down, or cant process it

Typical routing scenario
1
2
3
4

1
34
2

1
3
4

1
3
4
?

IP notes
 IP is a best effort protocol
 Many scenarios may be part of a failure
 Datagrams can be destroyed / lost
 Datagrams can be misordered
1234 ? 1 34

Remember
 IP is not reliable
 IP is a network protocol : it does what it can to route
packets from A to B , that's all
 IP offers no delivery garanty
 IP route traffic but doesnt route service
 When a datagram reaches a host, which
process should handle it ?

UDP
 Basically, UDP adds ports to IP
 It then allows multiplexing data streams
 When a datagram arrives, UDP layer processes
it and distributes it to the right host process by
reading the destination port

UDP
 UDP is light : only 8 bytes overhead per
message
 UDP does not provide
 Reliability
 Error correction mechanisms
 Connected service

TCP is more complex than UDP
 Transmission Control Protocol
 That means : control the transmission :-p
 TCP is order of magnitudes more complex than
UDP
 That's not hard to do given the simplicity of UDP
 TCP brings a lot of intelligence
 UDP has none at all

TCP overview
 TCP runs on top of IP
 TCP provides
 Full duplex connection oriented data exchanges
 Bytes can go in the 2 ways - Senders may be receivers
 PointToPoint connection
 No broadcast , no multicast , only 2 points
 Service multiplexing
 Through ports
 Stream data transfer
 No need to prepare the data to a specific form before sending

TCP overview
 … TCP provides
 Reliability
 data sent will be checked and ACKed
 Order
 data will be treated in the same order its been sent
 Flow control
 data will arrive whatever the processing speed at either
side*
 Congestion avoidance
 data will arrive whatever the middleware network load*
* : assuming L3 is not down

TCP header
 Variable length :
 20 bytes minimum
 60 bytes maximum (with all options field)

TCP header
Transmission control
Congestion controlProtocol control bits
Service multiplexingService multiplexing

TCP header overview
 Very verbose, embeds many informations
 Min 20 bytes, min 2.5x > UDP header
 Carries ports, like UDP (con multiplexing)
 Carries a checksum, like UDP
 Carries Sequence number and ACK number
 The most important things in TCP

The concept of reliability
 You are host "sender"
 You send a message to host "receiver"
 How can you be sure message arrived at
destination ?

ACK makes reliability
 The answer is to have receiver ACK sender's
message
1 1
ack 1ack 1

Reliability
 What happens if B doesn't ACK ?
 What happens if B's ACK is lost in network ?
1 1
ack 1

ARQ
 Automatic Repeat reQuest
 Sender re-sends the exact same message
 … and still waits until ACK
1 1
ack 1
1 1
ack 1ack 1

TCP :-)
 You just understood the very main baseline
behind TCP
 TCP is a connected protocol
 sender and receiver keep communicating
with each other
 receiver ACK messages received from sender

It all starts by a TCP
connection

TCP connection
 Before exchanging data, hosts must handshake
 Like in phone conversation
 Sen ->Rec "Hello ?"
 Rec ->Sen "Hello !"
 Sen ->Rec "Hi, right, let's start talking now"
 This is called the 3 way handshake
 Because it effectively requires 3 message exchanges
 Before any data exchange

TCP 3-way handshake
SYN
SYN
ACK
ACK
ACK bit
SYN bit
active opener
connect()
active opener
connect()
passive opener
listen()

Communication negotiation
 When in SYN, hosts will also exchange protocol
details
 Those are stuck in the options of the header
 "Do you support feature XXX" ?
 "Yes I do, we will use it"
 "No I dont, never use it for this communication"
options

TCP options negociation
 MSS is mandatory (IP MTU negociation)
 Prevents hawful IP fragmentation (end-to-end only)
 Window Scale factor is used as of 2017
 "SACK" is really commonly used

Let's disconnect
 We are now connected
 Disconnection is also a matter of message
exchanges
 But remember the full-duplex bi-directionnal
feature of TCP
 Connection must be closed in both ways
 From A to B
 And from B to A
 (Or the opposite : it doesn't matter)

TCP full closing
 One host (A or B) wishes to disconnect
 FIN - ACK and FIN - ACK
FIN
ACK
FIN
active closer
passive closer
ACK
FIN bitFIN bit

TCP half-closing
 Connection could be half closed
 Host A cant send any data to host B anymore
 But host B still can send data to host A
 Until host B initiates a FIN from its side
 This is uncommon , but still can happen
 close() : full close
 shutdown() : half close
 The host closing the connection will free
resources on its side
 It may then be interesting in some scenarios

TCP half-closing
 One host (A or B) wishes to disconnect
 Sends a FIN
 Waits for the corresponding ACK
 A can't send anymore data to B now
 But will still ACK it !
 But B still can send data to A
 Until itself sends a FIN , and receives the ACK
FIN
ACK
A B

Remember
 A TCP connection is uniquely identified by the
quadruplet
 Source IP address
 Source Port
 Destination IP address
 Destination Port
 Those 4 infos together, are unique quadruplet
 A TCP connection needs a preamble handshake
 A TCP connection needs a preamble close dialogue
to close
 And it may be "half closed"

Remember
 A TCP connection is between 2 and only 2 hosts
 One can be seen as the "client"
 The other as the "server"
 This is purely conceptual
 I prefer talking about "source" and "destination"
 Or "host A" and "host B"
 client/server , is what will be done at layer 7, with TCP
 Connection will run in both directions (full duplex)
 Any part, can initiate a close, whenever it wants

Reminder : ACK
 Ok with that ?
1
ack 1
2
ack 2
...

This is highly problematic !
 Guess the problem ?

RTT
 Round Trip Time - The time for a packet
 To be generated
 Create the segment
 Compute the checksum
 Add the header to the segment
 To be sent - routed
 Router1 → Router 2 → Router 3 - - - - - - → Router 42 - - - - ?
 To be received by destination
 NIC will interrupt CPU
 To be treated by destination
 Destination will extract header
 Check the checksum
 Done

waiting for ACK
 If you wait during RTT for an ACK …
 You highly underuse the network bandwidth
1
ack 1
t

Sliding Window
 Don't send just one packet at a time , but
several !
 Wait until ACKed
 Re-send if not
 Go further
 This is the crucial concept of TCP sliding
Window

Multiple packets on the way
 TCP sliding window
1234
ack 1 ack 2 ack 3 ack 4

TCP Sliding Windows Demo
 The best link about Sliding Window :
 http://www2.rad.com/networks/2004/sliding_win
dow/

TCP's heart in just one picture

Seq numbers and ACK
 TCP is a stream oriented protocol
 Layer 7 send() to a TCP socket some bytes
 TCP shrinks those bytes to some segment sizes
 The current segment size depends on
 MSS announced in SYN phase
 Total data volume to transfer
 TCP starts sending segments

Seq numbers
 The Sequence number indicates where we are
in the sent stream from our sender view of
the connection
 The ACK number indicates what we expect
next from the receiver view of the
connection
 We always ACK bytes, not segment
numbers
 We can use cumulative ACK

Example
 A got 750 bytes to send to B
 We'll assume A and B have connected
successfully
 Let's say segment size is 100 bytes
 Let's say window size is 500 bytes
A B
750 bytes

Seq and ACK in byte stream
 A receives from B
 ACK = 101 meaning B received bytes 1-100 and now expects 101
 A received from B ack = 301 so far
 That means that A received contiguously up to byte 300
 That means that A is expecting byte 301 now
seq = 1
len = 100
seq = 101
len = 100
seq = 201
len = 100
seq = 301
len = 100
seq = 401
len = 100
ack = 101ack = 201 ack = 301

Seq and ACK in byte stream
ack = 101
seq = 301
len = 100
seq = 401
len = 100
ack = 401
seq = 501
len = 100
seq = 601
len = 100
ack = 201 ack = 301
 Host A retransmits missing ACK segments
 Host A got as last ACK=601 from B
ack = 501 ack = 601

Seq ACK
 Remember ACK show what host expects next
 Here, B received up until 601, but not 601
ack = 601ack = 401ack = 501

Seq ACK
seq = 601
len = 100
seq = 701
len = 50
ack = 701
seq = 701
len = 50
ack = 751
ack = 701

Done !
 We just transfered bytes with TCP
 We managed to cope with lost segments
 Remember TCP pairs keep communicating
together
 Sender retransmit what receiver missed

Selective ACK (SACK)
 ACK system is nice but got a drawback
 By design, TCP resends every segments from
the point which has been lost
 TCP does not ACK each received segment
 But TCP ACKS the most recent contiguous
received bytes
 SACK is about ACKing segments
 To better detect holes, and only resend lost segments

TCP default ACK drawbacks
 The answer from B to A is 101 followed by 3 times 201
 That means that B has received the first 2 segments OK
 And that B received a total of 4 segments
 Where are the holes ? Which segments have been lost ?
 Segment 201 ? 301 ? 401 ?
 By default ; TCP must resend from 201 to 501 (3 segments)
seq = 1
len = 100
seq = 101
len = 100
seq = 201
len = 100
seq = 301
len = 100
seq = 401
len = 100
ack = 101ack = 201 ack = 201ack = 201

SACK in a word
 SACK is an option that allows ACKs to signal a
range of noncontiguous data received so
far
 The sender then has a better idea of which
segments have been received by the receiver,
or not
 SACK is a TCP Option (RFC 2018)
 That must be negociated in the handshake
 Both A and B must support it to use it

SACK example
seq = 1
len = 100
seq = 101
len = 100
seq = 201
len = 100
seq = 301
len = 100
seq = 401
len = 100
ack = 101ack = 201 ack = 201
sack = 301
ack = 201
sack = 301-401
seq = 201
len = 100
ack = 501

End-to-end Congestion control
Window size adjustment

Sliding Window size
 The Window size is how many bytes the
receiving part is able to proceed
 This is typically the side of its buffer
 It is also how maximum bytes can be in flight
seq = 645
len = 1000
seq = 1645
len = 1000
2000 bytes in flight

Sliding Window size
 When host B receives those, it sticks the
segments together in a buffered stream
 It effectively recreates the sent stream on its side
 TCP pushes up the bytes to layer 7
 Now, B layer 7 app must receive that buffer
recv()
tcp buffertcp buffer
layer 7 buffer

Sliding Window size
 What if B processes the bytes too slowly ?
 By reading f.e 10 bytes only from the received stream
 B's TCP receive buffer will saturate
 Because B doesn't process it fast enough at layer 7
 If A keeps sending, B won't be able to handle the
traffic
 A congestion will appear
 Resources will be wasted, as well as network bandwidth
 Packets will be lost and need re-send
recv(10)
tcp buffertcp buffer
layer 7 buffer

Window size advertisement
 The receiver advertises its window size to
the sender depending on its saturation
seq = 42
len = 1000
seq = 1042
len = 1500
seq = 2542
len = 1500
ack = 1042
win = 500
seq = 1042
len = 100
seq = 1142
len = 100
seq = 1242
len = 100
seq = 1342
len = 100
seq = 1442
len = 100
4000
bytes
500
bytes

Window size advertisement
 The receiver advertizes its receiving capabilities
to the sender
ack = 1042
win = 500
ouch, please don't send me more than 500 bytes as of now !
Window size

Window 0
 Eventually, the receiver will completely saturate
 It will advsertise a 0 Window
ack = 1042
win = 0
Please, stop any transfer

Network congestion detection
and avoidance

What do you do when you
detect congestion ?

Types of congestions
 So far, we've seen how hosts can advertize their
capabilities
 The receiving host advertises the sending host on
its receiving capabilities
 By adjusting the sliding window size
 That is, the number of bytes allowed in flight
 This way, the sender will never saturate the receiver
 But how to detect network (middle) congestion,
and what to do in such cases ?

The need to detect congestions
 Remember that TCP is blind about IP traffic*
 TCP must continuously guess the IP layer state from
PTP
 TCP must measure and evaluate continuously
 RTT
 Dup ACKs, meaning packet lost
 Advertized window size
 And TCP must run specific algorithms in case of
congestion detection
*: congestion signals exist (IP ECN)

Network Congestion
123 2
 Detecting network congestion is easy
 Lost packets (dup ACKs)
 TTL increasing
 But getting congestion to smoothly resorb it,
without improving it, is a challenge
 TCP must do its best to smoothly use the network
 TCP must guess, and act on the unknown network
state

Network congestion avoidance
algorithms RFC 2001
 TCP Slow Start
 Congestion Avoidance mode (Reno, Tahoe, NewReno,
Westwood)
 Fast Retransmit
 Fast Recovery
 Algorithms implemented in modern TCP stacks
 Compute an "image" of the underlying network
 By taking some measures
 Act and adapt accordingly to reduce congestion

TCP Slow Start
 When you start sending, send only one MSS, even
if the receiving window is larger
 When receiving the ACK, send one more MSS
 Even if the window is even larger
 When receiving the 2 ACKs, send two more MSS
 Even if the window is even larger
 This is exponential growth
 Etc, until congestion detected, or reaching a
special threshold called ssthresh
 This prevents a new connection on the network
from suddenly flooding the link

TCP Slow Start (mss = 1000)
seq = 1
len = 500
seq = 501
len = 500
ack = 501
win = 5000
seq = 1001
len = 1000
seq = 2001
len = 1000
ack = 3001
win = 7000
seq = 3001
len = 1000
seq = 4001
len = 1000
seq = 5001
len = 1000
seq = 6001
len = 1000
ack = 1001
win = 5000
1000
bytes
2000
bytes
4000
bytes

Slow Start from RFC
Beginning transmission into a network with unknown
conditions requires TCP
to slowly probe the network to determine the available
capacity, in order to avoid
congesting the network with an inappropriately large burst of
data. The slow start
algorithm is used for this purpose at the beginning of a
transfer, or after repairing
loss detected by the retransmission timer

Congestion Avoidance algo
 When network congestion is detected, drop down
the sender window
 Even if the advertized recv window is larger
 Usually, it is divided by two
 Then, don't slow start, but use a smoother algo
 Also after Slow Start, when reaching ssthresh
 Only send one more MSS per ACK received (linear
growth)
 Instead of doubling them each time (exponential growth)

TCP Slow Start and Congestion
avoidance
 Start exponentialy (assume bandwidth is
available)
 Continue linearly (assume full bandwidth will be
reached soon)

TCP congestion avoidance
 We send 3000 bytes on the wire
 We got a double ACK, meaning packets got lost
 ACK = 2001 means packet 2001 has been lost
 As we detect a packet loss, we will now send
half the amount of bytes : 1500
seq = 1
len = 1000
seq = 1001
len = 1000
ack = 1001
win = 5000
seq = 2001
len = 1000
ack = 2001
win = 5000
3000
bytes
ack = 2001
win = 5000

 Only one ACK, for two segments sent
 Again : something got lost (ack 2501 is expected)
 Even if we already divided the payload by two
 We must again, divide it by two : 750 bytes
 Look at the advertised window
 It keeps growing, meaning receiver is up, and wants more
bytes
 But the network keeps dropping them : it is congestionned
seq = 1001
len = 1000
ack = 2001
win = 7000
seq = 2001
len = 500
1500
bytes

 By sending smaller segments, TCP managed to make
the network deliver them
 Remember
 The more data in flight, the more network has to be strong
 It has little to do with the number of packets in flight !
 Routers have finite-length buffers
 If exceeded : they start dropping
seq = 1001
len = 200
ack = 1201
win = 9000
ack = 1401
win = 9000
seq = 1201
len = 200
seq = 1401
len = 200
seq = 1551
len = 150
ack = 1701
win = 9000
750
bytes

Fast Retransmit
 Sender analyzes duplicate ACK from receiver,
and tries to guess the « holes » in the stream
 Eventually helped by SACK
 It then retransmits the missing segments
immediately
 Without waiting for its segment expiration timer
 This is fast retransmit algorithm

Fast Retransmit
 Here it is clear segment 301 is missing
 and segment 401 is received
 Don’t wait for the 301 segment timer to expire,
resend !
seq = 1
len = 100
ack = 101
seq = 101
len = 100
seq = 201
len = 100
seq = 301
len = 100
seq = 401
len = 100
ack = 201 ack = 301 ack = 301

Fast Retransmit
seq = 1
len = 100
ack = 101
seq = 101
len = 100
seq = 201
len = 100
seq = 301
len = 100
seq = 401
len = 100
ack = 201 ack = 301 ack = 301
seq = 301
len = 100
ack = 501

You can feel those algorithms,
everyday
 You have already felt such algorithms
 Share a L2 switch with a friend – both plugin
 Plug in a route to Internet
 You start a download. You get full bandwidth
 Your friend (same L2) starts a download
 … What happens to you ?
 What happens to him ?

TCP is magic
 TCP continuously tries to be fair and to share the
overall Layer 3 bandwidth between every hosts,
whatever it is
 Every TCP connection will auto-regulate
 Leaving equal traffic to siblings
 Assuming no QOS rule
 As the Layer3 link gets saturated, every Layer 4
connection slows down to not over saturate the link
 As the layer3 link gets unsaturated, every
connection re-accelerate, until full bandwidth is
reached

Thank you TCP
 Isn’t that terribly magic and fascinating protocol ?
 Invented in 70's
 Vint Cerf - Bob Kahn
 Normalized in 1981
 Still under development
 Networks evolve, and are faster and faster
 The need of newer algorithms
 Lots of RFC
 Changes are pushed in Kernels and Windows
versions/updates
 Be up to date

Bandwidth calculation
 Theoretical BW calculation is very easy
 You have a window size of 70Kb
 Your RTT is 30ms
 Then your max bandwidth will be
 Assuming no packet loss
BW = WinSize / RTT
70*1024/30*10^-3 = 18,2Mbps

Improving Bandwidth
 Reduce RTT
 Get better paths (better peering)
 Increase WinSize
 Sysctl under Linux
BW = WinSize / RTT

Optimal Window size
 If you got a 1Gbps link
 And your RTT is 30ms
 Then you should set Window size to
 If you set less : you’ll underuse the network
 If you set more : you’ll saturate and drop bandwidth
 Take care of available memory on your host
 You should also enable SACK
BW = WinSize / RTT
WinSize = BW * RTT
1*1024^3 * 30*10^-3 = 3.84MB

Linux TCP
 Man tcp
 Sysctl -a | grep tcp
 MANY parameters
 Be careful, don’t break the network, share it
net.ipv4.tcp_abort_on_overflow = 0
net.ipv4.tcp_adv_win_scale = 1
net.ipv4.tcp_allowed_congestion_control = cubic reno
net.ipv4.tcp_app_win = 31
net.ipv4.tcp_autocorking = 1
net.ipv4.tcp_available_congestion_control = cubic reno
net.ipv4.tcp_base_mss = 1024
net.ipv4.tcp_challenge_ack_limit = 1000
net.ipv4.tcp_congestion_control = cubic
net.ipv4.tcp_dsack = 1
net.ipv4.tcp_early_retrans = 3
net.ipv4.tcp_ecn = 2
net.ipv4.tcp_ecn_fallback = 1
net.ipv4.tcp_fack = 1
net.ipv4.tcp_fastopen = 1

Linux TCP sockets
 SO_SNDBUF / SO_RCVBUF
 Sizes of TCP socket buffers
 TCP_NODELAY
 Do no buffer segments, send them asap (disable Nagle)
 Usually needed for stream applications (HTTP etc...)
 TCP_CORK (TCP_NOPUSH)
 Do only push full segments (MSS)
 SO_KEEPALIVE
 Enable TCP keepalive probes
 TCP_MAXSEG
 Limit MSS

Linux TCP
https://github.com/torvalds/linux/blob/master/net/i
pv4/tcp.c

TCP performances
 Computing the checksum costs a lot to CPU
 We nowadays transfer at GB or 10Gb rates
 Several Ghz of processing are needed for such rates
 TCP segmentation costs on both Tx and Rx
 Memory movements

TCP offload
 Offloading , is dedicating a part of the job to the NIC
 TOE : TCP Offload Engine
 A full TCP stack embed into a chip on the NIC
 Linux never accepted that
 TSO
 TCP Segmentation offload : offload the segmentation at Tx
Level
 LRO
 Large receive offload : offload the de-segmentation at Rx
Level
 Chk
 Compute the checksum

TCP offload at the NIC level
> jpauli@840G3:~$ sudo ethtool -k enp0s31f6
Features for enp0s31f6:
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp6-segmentation: on
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on

Tcpip

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Tcpip

Similar to Tcpip (20)

More from julien pauli

More from julien pauli (20)

Recently uploaded

Recently uploaded (20)

Tcpip