This document analyzes the timing performance of a Linux-based CAN-to-CAN gateway designed for interconnecting microcontrollers, particularly in automotive applications. It discusses the impact of various factors such as traffic load, gateway configuration, and kernel version on latency, presenting findings from extensive experiments. The results indicate that the SocketCAN-based gateway is a reliable kernel-space solution for CAN bus routing, outperforming user-space alternatives.

1. Timing Analysis of a Linux-Based
CAN-to-CAN Gateway
Michal Sojka1 , Pavel Píša1 , Ondˇej Špinka1 ,
r
Oliver Hartkopp2 , Zdenek Hanzálek1
ˇ
1 Czech Technical University in Prague
Faculty of Electrical Engineering
2 Volkswagen Group Research
13th Real-Time Linux Workshop
October 22, Prague
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 1 / 26

2. Outline
Motivation
Testbed Description
Analysis Results
Conclusions
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 2 / 26

3. Motivation
CAN Bus
Intended to interconnect microcontrollers for short
distances.
Network with deterministic medium access ⇒ real-time
applications.
Heavily used in automotive industry.
Bit rates up to 1 Mbit/s.
Every frame can carry up to 8 bytes of data.
Frames are identified by frame IDs (11 or 29 bits).
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 3 / 26

4. Motivation
Rapid prototyping with CAN
Goal: Quickly interconnect devices/subsystems from
multiple vendors.
Frame ID clashes cause problems – devices cannot be
connected together.
A need for CAN-to-CAN gateways to route (and modify)
messages between multiple buses.
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 4 / 26

5. Motivation
SocketCAN
Open-source project, official CAN-bus support for Linux
kernel
Initiated by Volkswagen research – first open source
project where Volkswagen participates
Core developers also from a few other companies
Contributors from many other people
In mainline Linux since 2.6.25
Based on standard Linux networking infrastructure
http://socketcan.berlios.de
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 5 / 26

6. Motivation
SocketCAN-based Gateway
Routes CAN frames between multiple CAN buses.
Frames can be modified (ID, length, data)
Modifications: SET, AND, OR, XOR, CRC
Frames can be filtered based on ID/mask pairs.
Implemented as a Linux kernel module
Easy to use:
cangw -A -s can0 -d can1
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 6 / 26

7. Motivation
Motivation of our Work
Adding gateways to real-time network can violate real-time
requirements (deadlines).
How are the routed frames delayed by the gateway?
What influences the delays (latencies)?
Traffic patterns,
Additional load,
Gateway configuration,
Linux kernel version.
Where are the limits of the SocketCAN-based gateway?
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 7 / 26

8. Motivation
Motivation of our Work
Adding gateways to real-time network can violate real-time
requirements (deadlines).
How are the routed frames delayed by the gateway?
What influences the delays (latencies)?
Traffic patterns,
Additional load,
Gateway configuration,
Linux kernel version.
Where are the limits of the SocketCAN-based gateway?
Nice way to measure temporal properties of the Linux
kernel.
Some of our results apply not only for CAN, but also for
Linux networking subsystem in general.
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 7 / 26

9. Testbed Description
Testbed Setup
Gateway – PowerPC MPC5200 (Busybox)
PC – Pentium 4, Kvaser PCI quad-CAN SJA1000-based
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 8 / 26

10. Testbed Description
Testbed Setup
Gateway – PowerPC MPC5200 (Busybox)
PC – Pentium 4, Kvaser PCI quad-CAN SJA1000-based
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 8 / 26

11. Testbed Description
Measured Latencies
CAN bus 0 msg 1
CAN gateway
(Linux)
CAN bus 1 msg 1'
time
GW latency Duration
Total latency
RX timestamp 1 RX timestamp 2
Same clock for both
timestamps
SO_TIMESTAMPNS
socket option
Clock source = TSC
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 9 / 26

12. Testbed Description
Tested Scenarios
Traffic patterns:
one message at a time,
50% bus load,
flood (almost 100% bus load)
Additional loads:
CPU load – hackbench
Ethernet load – ping -f -s 6000
Versions:
Kernel: 2.6.33, 2.6.33-rt29, 2.6.36, 3.0.4, 3.0.4-rt14
Latest SocketCAN from SVN (rev 1199 + fixes)
All combinations of the above.
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 10 / 26

13. Analysis Results
The Simplest GW Configuration
Experiment:
Frames with payload of 2 bytes
Message transmission time: 60 µs
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 11 / 26

14. Analysis Results
Presentation of Results – Histogram
4000 Histogram
What is the
Frames
3000
2000
1000 worst-case latency?
0
0 20 40 60 80 100 GW latency [µs]
10000
8000 Cumulative histogram
Frames
6000
4000
2000
0
0 50 100 150 200
10000
8000 Backward cumulative histogram
Frames
6000
4000
2000
0
0 50 100 150 200
10000
Latency profile
1000
Frames
100
10
1
0 50 100 150 200
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 12 / 26

15. Analysis Results
Presentation of Results – Cumulative Histogram
4000 Histogram
What is the
Frames
3000
2000
1000 worst-case latency?
0
0 20 40 60 80 100 GW latency [µs]
10000
8000 Cumulative histogram How many frames
Frames
6000
4000 had latency greater
2000
0 then 60 µs?
0 50 100 150 200
10000
8000 Backward cumulative histogram
Frames
6000
4000
2000
0
0 50 100 150 200
10000
Latency profile
1000
Frames
100
10
1
0 50 100 150 200
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 12 / 26

16. Analysis Results
Presentation of Results – Backward-Cumulative H.
4000 Histogram
What is the
Frames
3000
2000
1000 worst-case latency?
0
0 20 40 60 80 100 GW latency [µs]
10000
8000 Cumulative histogram How many frames
Frames
6000
4000 had latency greater
2000
0 then 60 µs?
0 50 100 150 200
10000
8000 Backward cumulative histogram Backward-
Frames
6000
4000 cumulative
2000
0 histogram.
0 50 100 150 200
10000
Latency profile
1000
Frames
100
10
1
0 50 100 150 200
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 12 / 26

17. Analysis Results
Presentation of Results – Latency Profile
4000 Histogram
What is the
Frames
3000
2000
1000 worst-case latency?
0
0 20 40 60 80 100 GW latency [µs]
10000
8000 Cumulative histogram How many frames
Frames
6000
4000 had latency greater
2000
0 then 60 µs?
0 50 100 150 200
10000
8000 Backward cumulative histogram Backward-
Frames
6000
4000 cumulative
2000
0 histogram.
0 50 100 150 200
10000
Latency profile
1000 Logarithmic scale.
Frames
100
10 Answer: 10 frames
1
0 50 100 150 200
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 12 / 26

18. Analysis Results
Presentation of Results – Multiple plots
4000 No load
What is the
Frames
3000 CPU load
2000
1000 worst-case latency?
0
0 20 40 60 80 100 GW latency [µs]
10000
8000 No load How many frames
Frames
6000 CPU load
4000 had latency greater
2000
0 then 60 µs?
0 50 100 150 200
10000
8000 No load Backward-
Frames
6000 CPU load
4000 cumulative
2000
0 histogram.
0 50 100 150 200
10000
No load
1000 Logarithmic scale.
Frames
CPU load
100
10 Answer: 10 frames
1
0 50 100 150 200
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 12 / 26

19. Analysis Results
Latency Profile vs. Time Plot
10000
140
120
GW latency [µs]
1000
Latency profile
100
[frames]
100 80
60
10 40
20
1 0
0 20 40 60 80 100 140
120 2.5 3 3.5 4 4.5 5 5.5
GW latency [µs] Experiment time [s]
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 13 / 26

20. Analysis Results
Measurement Precision
Our test-bed (PC) versus CANalyzer.
1000
Latency profile
[frames]
100
10 PC
CANalyzer
1
0 50 TX- 100 150 200 250
duration=84
Total latency [µs]
Difference 10 µs is sufficient for us.
Systematic error?
CANanalyzer has only resolution of 10 µs.
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 14 / 26

21. Analysis Results
Measurement Precision
Influence of Ethernet load generator on measurement precision. No GW involved!
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 15 / 26

22. Analysis Results
Measurement Precision
Influence of Ethernet load generator on measurement precision. No GW involved!
10000
No load
Ethernet load
Latency profile
1000
[frames]
100
10
1
0 31 50 100 150 200 250 300
TX+RX overhead [µs]
i.e. tkernel RX - tbefore send - tTX duration
TX+RX overhead (interfaces can0 and can1 on the PC)
Desired result: vertical lines at the same position
The error is much smaller than latencies observed for the
gateway.
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 15 / 26

23. Analysis Results
Batched Processing of Frames
10000
One frame at a time
Flood
1000
Latency profile
[frames]
100
10
1
0 20 40 60 80 100 120 140
GW latency [µs]
Soft-IRQ processes packets in batches (if possible).
Decreases average latency.
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 16 / 26

24. Analysis Results
Effect of Loading the Gateway – CPU
10000
No load
CPU load
1000
Latency profile
[frames]
100
10
1
0 50 100 150 200 250 300
GW latency [µs]
Hackbench
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 17 / 26

25. Analysis Results
Effect of Loading the Gateway – Ethernet
10000
No load
CPU load
1000 Ethernet load
Latency profile
[frames]
100
10
1
0 250 500 750 1000 1250 1500 1750 2000 2250 2500 2750 3000
GW latency [µs]
ping -f -s 60000
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 18 / 26

26. Analysis Results
Frame Filtering
Different number of filters, only the last one matches
10000
List length 1
List length 128
Latency profile
1000 List length 256
List length 384
[frames]
100 List length 512
List length 768
List length 1024
10 List length 1536
List length 2048 Two filter implementations:
1
0 200 400 600 800 1000
list traversal, array indexing
2048 SFF filters, different frame IDs More than 80 list entries ⇒
10000
Frame ID 1 Frame ID 1536
the delay increases faster
Frame ID 256 Frame ID 2048
Latency profile
1000
Frame ID 512 (cache trashing)
[frames]
Frame ID 768
100 Frame ID 1024
10
1
0 200 400 600 800 1000
GW latency [µs]
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 19 / 26

27. Analysis Results
Frame Modifications
10000
No modifications
2 modifications
4 modifications
4 modifications + XOR checksum
1000 4 modifications + CRC8 checksum
Latency profile
[frames]
100
10
1
0 50 100 150 200
GW latency [µs]
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 20 / 26

28. Analysis Results
Differences between Kernels – non-rt
10000
2.6.33.7
2.6.36.2
3.0.4
1000
Latency profile
[frames]
100
10
1
0 20 40 60 80 100 120 140
GW latency [µs]
The overhead increases over time
make oldconfig
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 21 / 26

29. Analysis Results
Differences between Kernels – rt-preempt
10000
2.6.33.7
2.6.36.2
3.0.4
2.6.33.7-rt29
1000
Latency profile
[frames]
100
10
1
0 50 100 150 200 250 300 350
GW latency [µs]
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 22 / 26

30. Analysis Results
Differences between Kernels – rt-preempt
10000
2.6.33.7
2.6.36.2
3.0.4
2.6.33.7-rt29
1000 3.0.4-rt14
3.0.7-rt20
Latency profile
[frames]
100
10
1
0 50 100 150 200 250 300 350
GW latency [µs]
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 22 / 26

31. Analysis Results
Problem (bug?) in rt-preempt kernels
GW kernel 3.0.4-rt14, Traffic oneatatime, Load none
10000
Frame id 0
Frame id 255
1000 Frame id 511
Latency profile
Frame id 767
[frames]
100
10
1
0.5 1 2 5 10 20 50
GW latency [ms]
Huge latencies in rt-preempt kernel. Conditions: 2048 EFF
filters, GW kernel: 3.0.4-rt14, traffic: one frame at a time,
load: none, payload: 2 bytes.
Appears also in 2.6.33.7-rt29
High latency repeats periodically every 1 second
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 23 / 26

32. Analysis Results
Problem (bug?) in rt-preempt kernels
GW kernel 3.0.4-rt14, Traffic oneatatime, Load none
100
GW latency [ms]
10
1
Frame ID 511
0.1
0 1 2 3 4 5 6 7 8
Experiment time [s]
Huge latencies in rt-preempt kernel. Conditions: 2048 EFF
filters, GW kernel: 3.0.4-rt14, traffic: one frame at a time,
load: none, payload: 2 bytes.
Appears also in 2.6.33.7-rt29
High latency repeats periodically every 1 second
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 23 / 26

33. Analysis Results
User-Space Gateway
Traffic: one at a time
10000
Kernel GW
Latency profile
1000 Userspace GW
[frames]
100
10
1
0 100 200 300 400 500
GW Latency [µs]
Traffic: flood
10000
Kernel GW
Latency profile
1000 User GW
[frames]
Kernel GW, 33-rt
100 User GW, 33-rt
10
1
0.01 0.1 1 10 100 1000 10000 100000
GW latency [ms]
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 24 / 26

34. Analysis Results
Results processing
More than 600 experiments
HTML-based navigation on results
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 25 / 26

35. Analysis Results
Results processing
More than 600 experiments
HTML-based navigation on results
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 25 / 26

36. Conclusions
Conclusion
SocketCAN-based gateway is a reliable solution for CAN
bus routing.
Kernel-space solution performs better than a user-space
one.
There are things to avoid:
Combining CAN traffic with Ethernet traffic.
Use of too many filters.
The gateway will be merged in 3.2 kernel.
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 26 / 26

37. Conclusions
Conclusion
SocketCAN-based gateway is a reliable solution for CAN
bus routing.
Kernel-space solution performs better than a user-space
one.
There are things to avoid:
Combining CAN traffic with Ethernet traffic.
Use of too many filters.
The gateway will be merged in 3.2 kernel.
Nice way of measuring Linux kernel temporal properties
from outside.
https://rtime.felk.cvut.cz/can/benchmark/2.1/
https://rtime.felk.cvut.cz/gitweb/can-benchmark.git
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 26 / 26

38. Conclusions
Conclusion
SocketCAN-based gateway is a reliable solution for CAN
bus routing.
Kernel-space solution performs better than a user-space
one.
There are things to avoid:
Combining CAN traffic with Ethernet traffic.
Use of too many filters.
The gateway will be merged in 3.2 kernel.
Nice way of measuring Linux kernel temporal properties
from outside.
https://rtime.felk.cvut.cz/can/benchmark/2.1/
https://rtime.felk.cvut.cz/gitweb/can-benchmark.git
Questions?
Michal Sojka CAN-to-CAN Gateway Timing Analysis RTLWS13 26 / 26