Go gives us net.Dial and net.Listen for sending and receiving data at Layer 4. Now you will see how to send and receive raw packets directly to and from the NIC at Layer 1 to get timestamp information from timestamping-enabled NICs and when packets enter and leave the Linux kernel. Capturing these timestamps allows us to get better granularity when measuring latency and jitter instead of relying on time.Now() in userspace where that is subject to additional time introduced by the OS and Go runtime schedulers.

1. Brought to you by
Capturing NIC and Kernel
Tx/Rx Timestamps for
Packets in Go
Blain Smith
Staff Software Engineer at Rocket Science

2. Blain Smith
Staff Software Engineer | Professor of Rocket Systems, Rocket Science
■ 10+ years building AAA online game services (AmongUs,
Unity, WB, Riot, 2K)
■ Low level networking and distributed systems
■ Competitive powerlifter and strongman

3. ■ Multiplayer Games Engineering Specialists!
■ Led, advised or invested in many of the industry’s largest
companies including Unity, PUBG, Multiplay, Vivox among
others as well as building solutions for the biggest games
out there.
■ Publishers around the globe trust us and our expert team to
design and deliver planet-scale online services for the
world’s biggest games such as Rocket League, League of
Legends, DOOM, and many more!

4. 3 Basic Measurements
There are 3 basic measurements we want to know about given any 2 computers
on a network.
■ Latency - time it takes for a packet move from computer A to computer B
■ Jitter - variation of latency over a given timeframe
■ Packet Loss - percentage of packets that were lost over a given timeframe

5. ping
> ping rocketscience.gg
PING rocketscience.gg (76.76.21.21) 56(84) bytes of data.
64 bytes from 76.76.21.21 (76.76.21.21): icmp_seq=1 ttl=119 time=24.6 ms
64 bytes from 76.76.21.21 (76.76.21.21): icmp_seq=2 ttl=119 time=22.2 ms
64 bytes from 76.76.21.21 (76.76.21.21): icmp_seq=3 ttl=119 time=24.6 ms
64 bytes from 76.76.21.21 (76.76.21.21): icmp_seq=4 ttl=119 time=23.0 ms
64 bytes from 76.76.21.21 (76.76.21.21): icmp_seq=5 ttl=119 time=23.9 ms
^C
--- rocketscience.gg ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4012ms
rtt min/avg/max/mdev = 22.184/23.666/24.618/0.935 ms

6. ping with Go
// using https://github.com/go-ping/ping
pinger, err := ping.NewPinger("rocketscience.gg")
if err != nil {
panic(err)
}
pinger.Count = 3
err = pinger.Run()
if err != nil {
panic(err)
}
stats := pinger.Statistics() // get send/receive/duplicate/rtt stats

7. HTTP Timestamp with Go
// http_server.go
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
log.Println(time.Now().UnixNano(), r.Method, r.URL.String())
w.WriteHeader(http.StatusOK)
io.Copy(w, r.Body)
})
http.ListenAndServe(":8080", nil)
// Client
> curl -d 'testing' localhost:8080
Testing
// Server Logs
2022/05/23 19:41:44 1653349304362572711 POST /

8. TCP Timestamp with Go
// tcp_server.go
ln, _ := net.Listen("tcp", ":8080")
for {
conn, _ := ln.Accept()
go func(conn net.Conn) {
log.Println(time.Now().UnixNano(), conn.RemoteAddr().String())
io.Copy(conn, conn)
}(conn)
}
// Client
> netcat localhost 8080
hi
hi
// Server Logs
2022/05/23 20:52:31 1653353551565391099 [::1]:53648

9. UDP Timestamp with Go
// udp_server.go
conn, _ := net.ListenPacket("udp", ":8080")
for {
data := make([]byte, 1024)
n, src, _ := conn.ReadFrom(data)
log.Println(time.Now().UnixNano(), src.String())
conn.WriteTo(data[:n], src)
}
// Client
> netcat -u localhost 8080
hi
hi
// Server Logs
2022/05/23 20:57:43 1653353863894400940 [::1]:48194

10. Problems with time.Time in Go
Every time you call tcpconn.(Read|Write) or udpconn.(ReadFrom|WriteTo)
your Go process makes a Linux system call send or recv via the kernel so it can
put the data into the NIC queue to be sent out onto or read in from the wire. These
system calls and queues all take time.

11. Sample Ping Sequence
■ Latency = Rx - Tx in
userspace
■ Includes kernel time (green)
● Go runtime context switching
● OS context switching
■ Includes NIC queue (blue)
● Tx/Rx buffers
■ We want just wire time
(orange)

12. Packet Sequence with Control Messages
■ Full Latency = Rx - Tx in userspace
■ Kernel Tx: is when packet makes it into the
kernel from userspace
■ NIC Tx is when the packet gets placed onto
the wire (if supported by hardware)
■ Kernel Rx is when the packet enters the
kernel from the NIC
■ NIC Rx is when the packet comes in from the
wire into the NIC

13. AF_PACKET
■ Requires we manually construct each layer of our packet and wrap our data
(Layer 1, 2, 3, and 4) manually
■ The great github.com/mdlayher/socket library makes working with raw
sockets very idiomatic
socket, err := socket.Socket(unix.AF_PACKET, unix.SOCK_RAW, 0, "socket", nil)
socket.ReadFrom(...)
socket.WriteTo(...)

14. Manually Creating Packet []byte
■ Use github.com/google/gopacket
■ Allows you to encode/decode packet layers
l1 := layers.Ethernet{SrcMAC: smac, DstMAC: dmac EthernetType: layers.EthernetTypeIPv4}
l2 := layers.IPv4{Version: 4, TTL: 64, Protocol: layers.IPProtocolUDP, SrcIP: saddr, DstIP: daddr}
l3 := layers.UDP{SrcPort: layers.UDPPort(sport), DstPort: layers.UDPPort(dport)}
l4 := []byte("this is my application data")
pkt := gopacket.NewSerializeBuffer()
gopacket.SerializeLayers(pkt, nil, l1, l2, l4, gopacket.Payload(l4))
fullpkt := pkt.Bytes()
socket.Write(fullpkt)

15. SO_TIMESTAMPING for Control Messages
Control messages are extra information provided by the Linux kernel about the socket.
Timestamp information is transmitted with these control messages using specific types.
Accessing these control messages are available with Resvmsg on a socket to "receive a
message" from the kernel once SO_TIMESTAMPING is enabled on the socket.
https://www.kernel.org/doc/html/latest/networking/timestamping.html
data := make([]byte, 1024)
cmsg := make([]byte, 1024)
// read incoming data from the RX queue (normally conn.Read)
socket.Recvmsg(data, cmsg, 0)
// read outgoing data from the TX queue (sent with conn.Write)
socket.Recvmsg(data, cmg, unix.MSG_ERRQUEUE)

16. Parsing the Timestamps in Control Messages
Socket kernel and NIC timestamp information can be parsed from these control
messages.
msgs, _ := unix.ParseSocketControlMessage(cmsg)
if msg.Header.Level != unix.SOL_SOCKET && msg.Header.Type != SocketOptTimestamping {
return ts, fmt.Errorf("no timestamp control messages")
}
var ts SocketTimestamps
ts.Software = time.Unix(int64(binary.LittleEndian.Uint64(msg.Data[0:])),
int64(binary.LittleEndian.Uint64(msg.Data[8:]))) // kernel timestamp
ts.Hardware = time.Unix(int64(binary.LittleEndian.Uint64(msg.Data[32:])),
int64(binary.LittleEndian.Uint64(msg.Data[40:]))) // NIC timestamp (if supported by NIC hardware)

17. Use blainsmith/fast_afpacket
■ Implements net.PacketConn
■ Sets up the AF_PACKET & SO_TIMESTAMPING options for you
■ Offer convenient RecvTxTimestamps() & RecvRxTimestamps()
// create a socket and bind it directly to the interface
// named eth0 and send/recv all IPv4/v6 TCP/UDP traffic!
iface, _ := net.InterfaceByName("eth0")
socket, _ := fastafpacket.Listen(iface, unix.SOCK_RAW, unix.ETH_P_ALL, nil)

18. Packet Sender/Receiver
for range ticker.C {
packet, _ := encodePacket(srcmac, srcip, srcport, dstmac, dstip, dstport, data)
_, _ = conn.WriteTo(packet, &fastafpacket.Addr{HardwareAddr: dstmac})
}
for {
packet := make([]byte, 1024)
_, _, ts, _ := conn.RecvTxTimestamps(packet)
log.Println(“tx”, ts.Hardware.UnixNano(), ts.Software.UnixNano())
}
// tx 1652138203507654453 1652138202507654453
for {
packet := make([]byte, 1024)
n, _, ts, _ := conn.RecvRxTimestamps(packet)
log.Println(“rx”, ts.Hardware.UnixNano(), ts.Software.UnixNano(), ts.Hardware.Sub(time.Now()))
}
// rx 1652138204508141040 1652138203508141040 -423.796µs

19. Use Cases
■ One-way latency NIC A -> NIC B
● Asymmetric measurements
● NIC B -> NIC A might be different
■ Observability into time being spent processing packets
● in/out kernel
● in/out NIC queues
■ Wire only
● Subtract kernel and NIC timestamps from userspace timestamps
● Removes jitter introduced from context switching (both Go runtime and OS) and NIC queuing
■ Target specific NICs
● Target which NIC the packet goes out and which destination NIC on the receiving side
● Expose slow NIC queues when comparing to other NICs on the same machine

20. Timestamp Delays
Once we have kernel and NIC timestamp information we can calculate the delays
between each of the layers.
■ tx.Software.Sub(userspaceTxTime) = outgoing kernel delay
■ tx.Hardware.Sub(userspaceTxTime) = outgoing NIC delay
■ rx.Software.Sub(userspaceRxTime) = incoming kernel delay
■ rx.Hardware.Sub(userspaceRxTime) = incoming NIC delay

21. Timestamping Support
■ Kernel timestamping should be able for most, but hardware timestamps
depends on NIC hardware
■ Mellanox (owned by NVIDIA) NICs have support
■ Use ethtool to check your own hardware
> ethtool -T eth0
Time stamping parameters for eth0:
Capabilities:
hardware-transmit (SOF_TIMESTAMPING_TX_HARDWARE)
software-transmit (SOF_TIMESTAMPING_TX_SOFTWARE)
hardware-receive (SOF_TIMESTAMPING_RX_HARDWARE)
software-receive (SOF_TIMESTAMPING_RX_SOFTWARE)
software-system-clock (SOF_TIMESTAMPING_SOFTWARE)
hardware-raw-clock (SOF_TIMESTAMPING_RAW_HARDWARE)
PTP Hardware Clock: 0
Hardware Transmit Timestamp Modes:
off (HWTSTAMP_TX_OFF)
on (HWTSTAMP_TX_ON)

22. Further Reading
■ AF_PACKET: https://man7.org/linux/man-pages/man7/packet.7.html
■ SO_TIMESTAMPING: https://www.kernel.org/doc/html/latest/networking/timestamping.html
■ One-way Delay: https://api.semanticscholar.org/CorpusID:5433866
■ Clock and Time Sync: https://arxiv.org/abs/2106.16140
■ NTP: http://www.ntp.org

23. Brought to you by
Blain Smith
@blainsmith
blainsmith.com
rocketscience.gg