This document summarizes Linux TCP/IP tuning techniques for optimizing real-time communication. It discusses adjusting interrupts, transmission/receiving queues, socket buffer sizes, protocol parameters, port ranges, shell limits, packet marking, traffic control, connection tracking, and more. The goal is to configure these various network parameters together to ensure high availability, media quality with low delay, jitter and packet loss for signaling and media packets. Proper testing and measurement tools are also needed to validate any tuning changes.

1. Linux TCP/IP Tuning
17 March 2015
Presented by
Binan AL Halabi
Master in Communications
For Real Time Communication
Blog: voipmagazine.wordpress.com/

2. Why Tuning ?

Service Availability (Signaling)

Media Quality (Low Delay, Low Loss and Low Jitter)
* Low Jitter = Packets arrives in a nice steady stream
(not bursts)
Signaling
/Media
Packets

3. Interrupts Tuning
Interrupts Distribution

Fair: All CPU Cores have the same priority.

Using “ irqbalance” daemon.

Problem: Time is needed to load the interrupt
handler to the cache of the fresh CPU core.

Not Fair:

Interrupt Affinity = CPU Cores (As a Bit Mask)
that handle the interrupts from a Specific
Device.

4. Interrupt Coalescing
Some NIC drivers support interrupt coalescing
(multiple interrupts can be coalesced into one
kernel interrupt).

5. Transmission & Receiving
Queues

“txqueuelen”: Transmission Queue Length related to
the interface. The kernel stores the departing frames in
this queue (the frames are not loaded into the NIC’s
buffer yet).

The “netdev_max_backlog” is the corresponding
receive queue.

6. Socket Buffers Sizes

Sizes allowed to be allocated by the application

Read/Receive Buffer: rmem_max and rmem_default

Write /Send Buffer: wmem_max and wmem_default

The values are in Byte.

7. Protocol-Specific Parameters
Example

” TIME_WAIT” Interval (tcp_fin_timeout)
– TCP socket state is “closing”
– Waiting to ensure the ACK (of respective FIN) it
sent was received.

Small value is recommended for high loaded
servers (e.g. 1-10 seconds)

8. Port Range &
Number Of File Handles
 Port Range “net.ipv4.ip_local_port_range”

“fs.file-max” is the maximum number of file
handles on the entire system

9. SHELL Limits
“ulimit” is used to limit the resources allocated to the
user, among other things, the number of files that
can be opened by the user:
• The user can change its limit:
# ulimit -n New_Value
• The root set the range for the user's
value. “/etc/security/limits.conf”

10. Packet Marking & Kernel
Traffic Control (TC/TCNG)

Packets Marking using IPtables: target “MARK”
with the option “--set-mark”

Example: Mark RTP packets with 6:
# iptables -t mangle -A PREROUTING -p udp -m
udp --dport 16384:32768 -i eth0 -j MARK
--set-mark 6

11. Kernel Traffic Control
(TC/TCNG)

Bind Marks with Defined Classes
Example: Makes packets marked with 6, to be processed
by the 1:10 class:
# tc filter add dev eth1 protocol ip parent 1:0 prio 1
handle 6 fw flowid 1:10

Define the classes (class based queueing)
# tc class add dev eth1 parent 1:1 classid 1:10 htb
rate XMbps ceil YMbps burst 10k

12. Kernel Connection Tracking
System
Conntrack Example:
Udp 17 20 src=192.168.1.2 dst=192.168.1.5
sport=137 dport=1025 [UNREPLIED]
src=192.168.1.5 dst=192.168.1.2 sport=1025
dport=137 use=1

13. Connection Tracking
“nf_conntrack” Hash Table

Error : “nf_conntrack: table full, dropping packet

Change The value: net.netfilter.nf_conntrack_buckets
And net.netfilter.nf_conntrack_max

14. What We Need ?
Testing Lab:

Case Definition

Traffic Generator

Measurement Tools

Results Documentation

15. Virtual Network Node

The network node is going virtual

Follow the packet in its path in the
system:

Policy

Network Virtualization Solution's QoS
Support

16. And Much More Parameters
But Remember the Whole Picture
(All Parameters Together)