Time-sensitive networking (TSN) is taking off in just about every market. Robotic arms, motion controller, even vehicles and automobiles have a need for both time sync (coordinated events) and timeliness (timely events) to improve existing real-time capabilities. Combining TSN with the real-time behavior of VxWorks operating system can enable developers to design distributed over deterministic network real-time control systems. Join us for a demonstration and live QA with our expert Andrei. He will show you how to run a TSN application on VxWorks as a KVM (Kernel-based Virtual Machine) guest in parallel with the best effort Linux OS on the same System-on-Chip (SoC). We will feature the decoupling of the control functionality from the sensing and actuation functionality, which usually run together on the embedded device, and move the control part to the intelligent edge.

1. 2 0 2 1 W I N D R I V E R , A L L R I G H T S R E S E R V E D
Andrei Kholodnyi
https://github.com/razr
https://discourse.ros.org/u/razr
https://www.linkedin.com/in/akholodnyi/
T
e
c
h
n
o
l
o
g
y
O
f
f
i
c
e
TSN apps running on VxWorks RTOS
as a KVM guest at the intelligent edge

2. Agenda
2
0
2
1
W
I
N
D
R
I
V
E
R
,
A
L
L
R
I
G
H
T
S
R
E
S
E
R
V
E
D
Why a separation of control
from sensor and actuation
functionality is important for the
intelligent edge solution
1
Why TSN and RTOS are
essential parts of the
intelligent edge story.
2
How to properly configure
TSN, Linux and VxWorks
KVM guest to run a OPC/UA
TSN example
3

3. 2
0
2
1
W
I
N
D
R
I
V
E
R
,
A
L
L
R
I
G
H
T
S
R
E
S
E
R
V
E
D
EDGE – WHY DO WE NEED TSN AND REAL-TIME?
Edge Cloud
Devices Customer premises Access
network
~2 ms <5 ms 1-3 ms
Edge compute
~5-20 ms
0-25 km 25-100 km
Centralized Data Center
100s-1000s km

4. 2
0
2
1
W
I
N
D
R
I
V
E
R
,
A
L
L
R
I
G
H
T
S
R
E
S
E
R
V
E
D
Edge use case : decoupling control from sensing and
actuation
VM
Real-time Control
VxWorks
OPC/UA
Sensing Control Actuation
Edge Real-time Control
VxWorks
TSN
OPC/UA
VM
Linux
OPC/UA
ML MW
Edge Virtualization
VxWorks
OPC/UA
Sensing Actuation
TSN
Control ML Apps
Real-time device
TSN

5. 2
0
2
1
W
I
N
D
R
I
V
E
R
,
A
L
L
R
I
G
H
T
S
R
E
S
E
R
V
E
D
Demo Configuration: Edge real time control
QEMU VM
Core 0
OpenIL Linux RT KVM
OPC/UA
TSN
VxWorks
Unikernel
PubSub
Tx
Core 1 Core 2 Core 3
LS1046ARDB
i210
Core 0
OPC/UA
TSN
OpenIL Linux
PubSub Rx
Core 1
LS1028ARDB
eno0
TSN
Linux Apps
https://github.com/open62541/open62541/tree/master/examples/pubsub_realtime/vxworks
Edge Node
Edge Device
PCI passthrough
192.168.10.46 192.168.10.28

6. 2
0
2
1
W
I
N
D
R
I
V
E
R
,
A
L
L
R
I
G
H
T
S
R
E
S
E
R
V
E
D
• Virtualization
• Linux host
• QEMU
• Real-time guest OS
• PCI pass-through
• i210 card passed through to
the guest
DEMO – Virtualization
QEMU VM
Core 0
OpenIL Linux RT KVM
OPC/UA
TSN
VxWorks
Unikernel
PubSub
Tx
Core 1 Core 2 Core 3
LS1046ARDB
i210
Linux Apps
Edge Node
dev=0001:01:00.0
echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
echo vfio-pci > /sys/bus/pci/devices/$dev/driver_override echo $dev >
/sys/bus/pci/drivers/vfio-pci/bind
~# lspci -k 0001:01:00.0 Ethernet controller: Intel Corporation I210
Gigabit Network Connection (rev 03) Subsystem: Intel Corporation
Ethernet Server Adapter I210-T1 Kernel driver in use: vfio-pci

7. 2
0
2
1
W
I
N
D
R
I
V
E
R
,
A
L
L
R
I
G
H
T
S
R
E
S
E
R
V
E
D
• VxWorks kernel library
• librsl.a
• UP mode
• Real-time kernel application
• tsnperf
DEMO – Real-time unikernel
Core 0
OPC/UA
TSN
VxWorks
Unikernel
PubSub
Tx
Core 1 Core 2 Core 3
LS1046ARDB
i210
Linux Apps
Edge Node
$ cat ~/stash/projects/TO/rsl_apps/tsnperf/Makefile
APP = tsnPerf
all: $(APP)
$(APP): $(APP).o main.o Makefile
$(RSL_SDK)/krnl/librsl.a
rsl build $(APP).o main.o -o $(APP)
What is VxWorks?
• it is a proprietary hard real-time OS, POSIX PSE52
• 64 bits on ARM, Intel, MIPS, PowerPC, and RISC-V
• Kernel and user space separation, user space optional
• C/C++17, possible to develop kernel C++ modules and user apps, and
recently Rust and Python
• Safety certifiable: DO-178, ISO 26262, IEC 61508
• Toolchain LLVM 11, Dinkumware C/C++ libs
• Proprietary build system. CMake, autotools support for apps, e.g.
ROS2
• Kernel shell
• Eclipse, and Visual Studio Code - modern IDEs, Windows/Linux hosts

8. 2
0
2
1
W
I
N
D
R
I
V
E
R
,
A
L
L
R
I
G
H
T
S
R
E
S
E
R
V
E
D
▪ Isolate CPU 2 and 3 to run QEMU
rcu_nocbs=2,3 nohz_full=2,3
isolcpus=2,3 irqaffinity=0,1
audit=0 watchdog=0 skew_tick=1
▪ No CPU frequency scaling
▪ Other settings
DEMO – RT_PREEMPT
QEMU VM
Core 0
OpenIL Linux RT KVM
OPC/UA
TSN
VxWorks
Unikernel
PubSub
Tx
Core 1 Core 2 Core 3
LS1046ARDB
i210
Linux Apps
Edge Node
CONFIG_NO_HZ_FULL=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_PREEMPT_RT=y
CONFIG_HZ_1000=y
# CONFIG_CPU_FREQ is not set

9. 2
0
2
1
W
I
N
D
R
I
V
E
R
,
A
L
L
R
I
G
H
T
S
R
E
S
E
R
V
E
D
▪ taskset -c 2 /* run on CPU2 */
▪ -enable-kvm /* no emulation */
▪ -smp 1 /* one CPU only */
▪ -nic none /* no nic */
▪ -name debug-threads=on /* name threads */
▪ -device vfio-pci,host=$dev /* passthrough */
DEMO – QEMU
QEMU VM
Core 0
OpenIL Linux RT KVM
OPC/UA
TSN
VxWorks
Unikernel
PubSub
Tx
Core 1 Core 2 Core 3
LS1046ARDB
i210
Edge Node
taskset -c 2 /work/qemu/installArm/bin/qemu-system-aarch64 -cpu host
-enable-kvm -smp 1 -name debug-threads=on -nic none -machine
type=virt,gic-version=host -nographic -m 256 -overcommit mem-
lock=on -append "fs(0,0)host:vxWorks h=192.168.10.28
e=192.168.10.46 u=user pw=invalid f=0x01 o=gei0" -kernel
/root/tsnPerf -device vfio-pci,host=$dev

10. 2
0
2
1
W
I
N
D
R
I
V
E
R
,
A
L
L
R
I
G
H
T
S
R
E
S
E
R
V
E
D
▪ Move vCore process to the CPU 3
▪ Let other QEMU threads on the CPU 2
DEMO – pin vCore to pCore
QEMU VM
Core 0
OpenIL Linux RT KVM
OPC/UA
TSN
VxWorks
Unikernel
PubSub
Tx
Core 1 Core 2 Core 3
LS1046ARDB
i210
Edge Node
QEMU_PID=`pidof qemu-system-aarch64`
user@LS1046ARDB-Ubuntu:~$ pstree -t $QEMU_PID
qemu-system-aar─┬─{CPU 0/KVM}
├─{qemu-system-aar}
└─{signalfd_compat}
VCPU_PID=`ps -aL |grep -E "(kvm-vcpu|CPU.*KVM)"|awk '{print $2}’
VCPU_CORE=3
taskset -c -p $VCPU_CORE $VCPU_PID
QEMU
thread #1
QEMU
thread #n

11. 2
0
2
1
W
I
N
D
R
I
V
E
R
,
A
L
L
R
I
G
H
T
S
R
E
S
E
R
V
E
D
-> ptpdAdd "gei0", 2, 2, 1, 0, 254, 0, 1
-> ptpdStart
-> ptpdStatusShow "gei0"
Interface name : gei0
Now in state : PTP_SLAVE
Demo Configuration: PTP
QEMU VM
Core 0
OpenIL Linux RT KVM
PTP slave
VxWorks
Unikernel
Core 1 Core 2 Core 3
LS1046ARDB
i210
Core 0
PTP master
OpenIL Linux
Core 1
LS1028ARDB
eno0
PTP
https://github.com/open62541/open62541/tree/master/examples/pubsub_realtime/vxworks
Edge Node
PCI passthrough
192.168.10.46 192.168.10.28
Edge Device
sudo daemonize -E BUILD_ID=dontKillMe -o /var/log/ptp4l.log -e
/var/log/ptp4l.err.log /usr/bin/taskset -c 1 chrt 90 /usr/sbin/ptp4l -i eno0 -2
-mq -f /etc/ptp4l_cfg/gPTP.cfg --step_threshold=1 --fault_reset_interval=0
--announceReceiptTimeout=10 --transportSpecific=1
root@LS1028ARDB-Ubuntu:~# tail -f /var/log/ptp.log
ptp4l[186906.248]: selected local clock 5ecf89.fffe.ebf722 as best master
ptp4l[186906.248]: port 1: assuming the grand master role

12. 2
0
2
1
W
I
N
D
R
I
V
E
R
,
A
L
L
R
I
G
H
T
S
R
E
S
E
R
V
E
D
Demo Configuration: TSN (finally)
QEMU VM
Core 0
OpenIL Linux RT KVM
OPC/UA
TSN
VxWorks
Unikernel
PubSub
Tx
Core 1 Core 2 Core 3
LS1046ARDB
i210
Core 0
OPC/UA
TSN
OpenIL Linux
PubSub Rx
Core 1
LS1028ARDB
eno0
TSN
https://github.com/open62541/open62541/tree/master/examples/pubsub_realtime/vxworks
Edge Node
Edge Device
PCI passthrough
192.168.10.46 192.168.10.28
QEMU
thread #1
QEMU
thread #n
-> tsnConfig("gei", 4, 0, 0, "/romfs/62541.json")
value = 0 = 0x0
-> open62541PubTSNStart("gei", 4, 0, 0, 1)
./tsnPerfLxRx -o "01:00:5E:00:00:01" -I "eno0" -v 3000 -n 1 2>&1 | tee
tsnPerfLxRx.log

13. 2
0
2
1
W
I
N
D
R
I
V
E
R
,
A
L
L
R
I
G
H
T
S
R
E
S
E
R
V
E
D
DEMO – Results

14. 2
0
2
1
W
I
N
D
R
I
V
E
R
,
A
L
L
R
I
G
H
T
S
R
E
S
E
R
V
E
D
▪ The Edge needs determinism, TSN,
WTSN, and 5G
▪ The Edge requires real-time and
virtualization
▪ A careful configuration of the edge
system including TSN is needed to
fulfill real-time requirements
▪ VxWorks is a good candidate as a
guest OS at the edge, providing
necessary real-time features,
unikernel, and TSN protocols support
DEMO – Conclusion VM
Edge Real-time Control
VxWorks
TSN
OPC/UA
VM
Linux
OPC/UA
ML MW
Edge Virtualization
VxWorks
OPC/UA
Sensing Actuation
TSN
Control ML Apps
Real-time device
TSN