ROS 2 Foxy with Eclipse Cyclone DDS | Philly ROS Meetup July 20th 2020

ROS 2 Foxy with Eclipse Cyclone DDS
makes robots Easier, Smaller, Faster
Philly ROS Meetup July 20th 2020
bit.ly/eclipsecyclone for this presentation and links
Joe Speed
@JoeSpeeds
joespeed
jspeed12
Erik Boasson
too busy for twitter
erik-boasson-21344912
eboasson
@EclipseCyclone

Leading Edge COMPUTING adlinktech.com

ROS1 created in 2007 by Willow Garage as a platform for robotics
research
• Moved into stewardship by the Open Source Robotics Foundation
• Vibrant ecosystem of tools, algorithms and drivers for constructing robots
• Much used for research, education, prototypes
ROS2 is a rewrite of ROS1 driven by Open Robotics, first released 2017
• Addresses the weaknesses in the architecture of ROS1
• Maturing nicely, ecosystem is migrating to it
• Already the choice for mobile robotics, gaining ground in industrial robotics
The briefest of recaps: ROS1 & ROS2

- 115 engineers, equivalent to $8M
- Completeness
- Quality
- Performance
- Security
- Navigation
- No more releases of ROS “classic”
- aws.amazon.com/blogs/robotics/ros-2-foxy-fitzroy-robot-development
ROS 2 Foxy - Now is the time to switch

Who is Eclipse Cyclone DDS?

“Eclipse Cyclone DDS is a superior architecture”
- Automotive CTO
• This open source contributed DDS is very special
• Makes ROS 2 robots fast, small, reliable
• Tier-1 ROS middleware built into ROS 2 Foxy
• Open Robotics is evaluating for ROS 2 default
• Built into Autoware autonomous driving
• Industry is helping it become real-time & safe
bit.ly/eclipsecyclone

iot.eclipse.org/adopters

a) Eclipse IoT open source
project
b) very performant DDS
implementation
c) Easy, Small and Fast
d) Loves github issues & PRs!
e) Led by IoT mad scientist &
2nd gen DDS inventor Erik
Boasson eboasson
What is Eclipse Cyclone DDS?

Pure C code
Run-time library has minimal dependencies
Compact
• code size can go down to ~0.5MB
• latency testing needs ~0.5MB of heap memory
(macOS)
• Configuring for minimal footprint does reduce
throughput (~0.5x) & increase latency a bit (~5µs)
Platforms
• Supported: Linux, Windows, macOS, FreeRTOS,
x86, Arm, Jetson
• Planned: QNX, possibly VXWorks and others
• Works: Solaris 2.6, OpenIndiana (Erik & Morgan are sentimental)
Footprint of Eclipse Cyclone DDS

tested the RMWs,
selected Eclipse Cyclone DDS
“The (ROS 2) RAM usage goes from 150MB to 15MB and we basically
had no late and no lost messages. This is a huge improvement in
terms of performance simply changing the DDS implementation”

Problem Statement
~1000 robots in the same network
~35 topics from each robot
~5 different subnetworks across the globe
We want to be able to individually access any robot's
messages from a remote debugging machine

Addressing Scalability
Proposed solution 1:
Use default DDS configuration and a different namespace for
each robot
Problems:
ALL robots discover each other!
Slow/unreliable discovery, network saturation, multicast requires
forwarding between subnets

Solution 1 + ROS_DOMAIN_ID
Problems:
Limited numbers of domain IDs. What about assignment?
Still considerable network usage and multicast limitations

Disable multicast discovery.
When a remote machine wants to connect to a robot, add the
robot’s IP to the initial peers list (unicast)
Problems:
The user needs direct access to the DDS configuration. This can
be solved by new APIs.
✓

Solution 3 with CycloneDDS
ROBOT_IP=10.22.22.90 ros2 topic list
✓

Erik eboasson added domainTag to DDSI 2.3 spec
and implemented iRobot’s use case in
CycloneDDS: Find Roomba by serial # among
~1,000 robots on network
DDSI domainTag things talk if domain + tag match
//CycloneDDS/Domain/Discovery/Tag
String extension for domain id that
remote participants must match to be
discovered. The default value is: "”
Solution 4 with Eclipse Cyclone DDS
XYZ270011101230600001
✓

github.com/irobot-ros/ros2-performance
on your RPi2B or other machine
on The Construct’s ROS Dev Studio http://www.rosject.io/l/ebd6221/
Try iRobot’s test with Eclipse Cyclone DDS

+ = extremely fast ROS!

Chassis Systems Control | CC-AD/ESW1 | 2019-11-22
© Robert Bosch GmbH 2019. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
architecture A for ROS 2 and Autoware.auto
cyclonedds + iceoryx
22
„Cyclone DDS with iceoryx built-in
„ Benefits
‒ Full DDS (and QoS) support for IPC
‒ Standalone DDS product with full DDS API
‒ Fast track for ROS
„ Challenges
‒ AUTOSAR Adaptive support
‒ Adaptations in Cyclone for integrating iceoryx
communication APIs
?

Chassis Systems Control | CC-AD/ESW1 | 2019-11-22
© Robert Bosch GmbH 2019. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
architecture B for AUTOSAR Adaptive
iceoryx + cyclonedds
23
„APIs on top of iceoryx, cyclonedds network gateway
„ Benefits
‒ AUTOSAR Adaptive API binding for iceoryx available
‒ lot experience with gateways in iceoryx (SOME/IP,
ADLINK’s OpenSplice DDS, ROS1, AMQP)
‒ Good migration path for automotive projects
‒ No cyclonedds dependency in executables (freedom from interference)
‒ Fast track for AUTOSAR
„ Challenges
‒ QoS support for IPC when using ROS2 API
‒ Code generator for conversion / serialization to DDS in gateway
‒ No native DDS API
‒ Different data models in AUTOSAR and ROS/DDS
communication APIs

Bosch’s official feedback*
* not a joke, this is what Bosch sent to Eclipse Cyclone DDS contributors

Eclipse OpenADx

high velocity sensor fusion
… because race car

Suspension Analysis / Tire Modeling
CAN Bus over LTE
Intel RealSense
Embedded ML
ADLINK ROScube Jetson Xavier

Demo - Vehicle Dynamic (AI Driver Model)
trained driver model, can currently
uses a combination of neural
networks to predict with 85%
accuracy:
• Steering Angle
• Brake Position
• Throttle Position
• Gear and Speed

AMRs with Eclipse Cyclone DDS
mobile vision
“camera on wheels”
Education
Inspection
100kg payloads

Professional Sports with Eclipse Cyclone DDS

“Eclipse Cyclone DDS has given the fast
& precise control of our ROS 2 baseball
pitching robot to accurately recreate
any throw from any pitcher”
- Rowan Ferrabee
CTO Trajekt Sports

Eclipse Cyclone DDS is built into
components of military vehicles with
32-bit microcontrollers
Intra-vehicle communication!
Inter-vehicle communication?
Couldn’t say who, so here is a
picture of an old jeep
microcontrollers running Eclipse Cyclone DDS
source: Franklin D. Roosevelt Presidential Library
and Museum, Hyde Park, New York.

otokarThis is one of those vehicles

”multicast autotune” – not
really, but sort of !"
Sped Nav2 WiFi bringup
while others learned how
“just works” in hospital with
terrible multipath issues, no
config needed
WiFi happy Eclipse Cyclone DDS
image source: edmtips.com/how-to-autotune-properly

<images from Pack Expo>
Worked reliably in Last Vegas Convention Center
despite 666 WiFi APs “a devilish number” !

Vendor benchmarks?!?
Source dharmapunxnyc.blogspot.com/2013/08/dont-believe-hype.html

apt install ros-foxy-rmw-cyclonedds-cpp
(or eloquent)
(or dashing)
RMW_IMPLEMENTATION=rmw_cyclonedds_cpp
bit.ly/eclipsecyclone for this presentation and links
iRobot ROS 2 performance ROScon `19 slides
iRobot ROS 2 performance test tool for mobile robots
ApexAI/performance_test RMW and DDS evaluation tool
ros2/rmw_cyclonedds try it

Eclipse Cyclone DDS 
 Erik Boasson

Eclipse Cyclone DDS
• Full-featured DDS implementation
• QoS support: nearly complete
• Content ﬁltering/querying: planned
• Durable data: work-in-progress, partial support later this year
• DDS Security: available in master, to be released shortly
• DDS XTypes: underway, support for key features expected in the fall

Eclipse Cyclone DDS
• Fast(est) and small(est) DDS
• it was the fastest(*) by a good margin, then the others woke up
• (*) disputed by the other DDS implementors, certainly no longer true in all scenarios
• The only C-based open-source DDS implementation
• Smaller run-time footprint, fewer platform dependencies
• People are using Cyclone on FreeRTOS on a ARM Cortex-M4 with a only few MB of memory
• ADLINK’s OpenSplice DDS is in C, but support for it has been dropped in Foxy

Cyclone DDS in ROS 2
• Tier 1 since Foxy release — Yay!
• DDS Security and some performance improvements to be released shortly in
Foxy via rolling update
• Actively tracks ROS 2 development
• e.g., ﬁrst RMW to support incompatible QoS notiﬁcations
• More often than not, test failures turn out to be wrong assumptions in ROS test
code
• Some designs prompted by Fast-RTPS limitations are bad for Cyclone DDS

Fast — Gigabit
2x Ubuntu 16.04, Intel(R) Xeon(R) CPU E3-1270 V2 @ 3.50GHz (2012), connected via GbE

Fast — Loopback
2007 Macbook Pro 15”, 2.9GHz, macOS 10.15.5, loopback 
(some tweaks: max message size, receive buffer size, asynchronous delivery)
1k 5k 10k 50k 100k 500k 1M 5M 10M
0 50 100 150
10
100
1000
104
Mb/s
MBP loopback (async)
used net bw DDS 1s avg DDS 10s avg
rexmit (110Mb) discarded

Small — Cyclone and default RMW
$ size librmw_fastrtps_cpp.so librmw_fastrtps_shared_cpp.so libfastrtps.so.2
libfastcdr.so.1 | gawk '{print}$1~/^[0-9]+/{S+=$4}END{print S}'
text data bss dec ﬁlename
192609 7760 26440 226809 librmw_fastrtps_cpp.so
258978 4544 42536 306058 librmw_fastrtps_shared_cpp.so
7575437 109496 283696 7968629 libfastrtps.so.2
72047 1640 8 73695 libfastcdr.so.1
8575191
$ RMW_IMPLEMENTATION=rmw_fastrtps_cpp ros2 launch
demo_nodes_cpp talker_listener.launch.py & sleep 3 ; python3 ps_mem.py
-p "`pgrep talker`,`pgrep listener`" ; kill %
…
… [talker]: Publishing: 'Hello World: 1'
… [listener]: I heard: [Hello World: 1]
Private + Shared = RAM used Program
5.0 MiB + 4.7 MiB = 9.7 MiB talker
5.4 MiB + 4.6 MiB = 10.0 MiB listener
---------------------------------
19.6 MiB
$ size librmw_cyclonedds_cpp.so libddsc.so.0 | gawk '{print}$1~/^[0-9]+/
{S+=$4}END{print S}'
text data bss dec ﬁlename
467443 8976 424 476843 librmw_cyclonedds_cpp.so
1069917 47741 8472 1126130 libddsc.so.0
1602973
$ RMW_IMPLEMENTATION=rmw_cyclonedds_cpp ros2 launch
demo_nodes_cpp talker_listener.launch.py & sleep 3 ; python3 ps_mem.py
-p "`pgrep talker`,`pgrep listener`" ; kill %
…
---------------------------------
9.6 MiB

Small — ROS 2 and “native”
$ ddsperf pub 1Hz & ddsperf sub & sleep 3 ; python3
ps_mem.py -d -p "`pgrep -d, ddsperf`" ; kill %1 %2
…
[52385] 1.000 size 12 total 1 lost 0 delta 1 lost 0 rate 0.00 kS/s 0.00
Mb/s (0.00 kS/s 0.00 Mb/s)
Mb/s (0.00 kS/s 0.00 Mb/s)
Mb/s (0.00 kS/s 0.00 Mb/s)
880.0 KiB + 912.5 KiB = 1.8 MiB ddsperf
888.0 KiB + 912.5 KiB = 1.8 MiB ddsperf
---------------------------------
3.5 MiB
$ RMW_IMPLEMENTATION=rmw_cyclonedds_cpp
ros2 launch demo_nodes_cpp
talker_listener.launch.py & sleep 3 ; python3
ps_mem.py -p "`pgrep talker`,`pgrep listener`" ; kill %
…
---------------------------------
9.6 MiB

Challenges for DDS in robotics
• A very different environment than where DDS grew up
• more dynamic (roving robots entering a cage of Faraday such as an elevator
— things like that)
• dependence on really large messages: point clouds of 10MB are not uncommon,
and DDS people used to think 100kB was large!
• people focussed on developing the robots, rather than on pleasing the middleware
• often running everything inside a single machine, rather than over the network
• … and if there’s a network, it is often Wi-Fi

Moving forward: throughput & latency
• Good progress on transparent use of shared memory
• integrating Eclipse Iceoryx
• we’re not interested in a “shared memory transport”: that only gives you maybe
20%
• we’re interested in using the shared memory as a store of messages
• allows zero-copy communication
• pure shared memory within the machine for many cases
• automatic fall-back to loopback if QoS cannot otherwise be supported

Moving forward: scalability
• DDS discovery can be a burden, especially over Wi-Fi
• Only real solution is to not discover everything
• We’re developing a discovery service that allows discovering only matching endpoints
• This is the second version, building on Eclipse Zenoh (zenoh.io)
• The ﬁrst version was in Java and used, e.g., in Singapore for connecting smart lampposts
• Will also provide transparent bridging of DDS across networks
• Disclosure: I am one of the originators of Zenoh, this work is being done by friends in Paris

Moving forward: multiple networks
• Cyclone DDS today operates over a single network interface
• Will be adding support for operating over multiple interfaces simultaneously
• easier out-of-the-box use with containers, multiple interface, &c.
• support for load-balancing
• support for redundant networking

Maximising throughput
• Default config doesn’t max out UDP datagram sizes
• more overhead, less to retransmit on loss
• loopback benefits from larger messages, GbE doesn’t
• ≤ 0.6 default: 4kB, current master: 15kB
• Tweak: General/MaxMessageSize
• limit to 65500B or it won’t always fit in a UDP datagram
• (macOS, master, Linux is qualitatively similar; some retransmitting because of buffer overruns)
10MB sample
100kB sample
1kB sample
10 20 30 40 50
msg size [kB]
5
10
15
20
throughput [Gb/s]

Maximising throughput
• UDP socket receive buffer sizes do matter
• overrunning the buffer means retransmits (reliable) / lost messages (best-effort)
• Defaults
• Linux defaults to limiting to ~400kB, macOS to ~800kB; Cyclone to 1MB
• even without tweaking, it works well (some retransmits in preceding graph)
• Rule of thumb: make it large enough to easily contain the largest sample
• echo 33554432 > /proc/sys/net/core/rmem_max
• Internal/SocketReceiveBufferSize = 30M
• Better yet: don't send such large samples!

Multicast vs unicast
• If multicast enabled, Cyclone uses it when > 1 receiver IP address
• This is not configurable, but is something under investigation
• 1 receiver IP address: always unicast
• Multicast is enabled by default, except when a Wi-Fi interface is selected
• On Wi-Fi multicast well enough for initial node discovery, but not much else
• Ubuntu loopback interface claims to not support multicast (“ifconfig lo multicast”
enables it)
• Topics can be mapped to multicast addresses in configuration

Multicast (2)
• General/AllowMulticast conﬁgures whether it is allowed:
• false: disallowed
• spdp: allowed only for initial participant/node discovery
• true: enabled
• Discovery/SPDPMulticastAddress is discovery address (default: 239.255.0.1)
• also default data multicast address, override with Discovery/DefaultMulticastAddress
• if IGMP is used, multicasts only reach nodes subscribed to that IP address
• if you want no interference, pick a different multicast address: port numbers are ignored in
routing

Multicast (3)
• Topics (with wildcards) can be mapped to different multicast addresses in conﬁg:
<Partitioning> 
<NetworkPartitions> 
<NetworkPartition Name="ssm" Address="232.3.1.4"/> 
<NetworkPartition Name="x" Address=“224.225.1.4"/> 
<NetworkPartition Name="y" Address="224.226.0.4;8;4;${NODEIDX:-0}"/> 
</NetworkPartitions> 
<PartitionMappings> 
<PartitionMapping DCPSPartitionTopic=“rt/camera*" NetworkPartition="ssm"/> 
<PartitionMapping DCPSPartitionTopic=“*pointcloud*” NetworkPartition="y"/> 
<PartitionMapping DCPSPartitionTopic="*.*" NetworkPartition="x"/> 
</PartitionMappings> 
</Partitioning>
• source-speciﬁc multicast (SSM) is supported, the “y” one needs a bit more explanation …

Multicast (4)
• 224.226.0.4;8;4;${NODEIDX:-0} means:
• use 224.226.0.4 as a base address
• use the 4 bits at positions 8 .. 11 for identifying individual nodes
• expand NODEIDX environment variable to get the index of this node (0 if not set) 
(environment variable expansion is performed throughout the conﬁguration)
• gets dynamically expanded to address just the subscribed nodes (O(2n) groups!)
• switches can typically handle up to 7 or 8 nodes this way before ﬂooding the network
• e.g.:
• if 0 and 1 are subscribed, a multicast is sent to 224.226.3.4
• if 1 and 3 are subscribed, a multicast is sent to 224.226.6.4

Unicast discovery
• If multicast disabled for initial node discovery
• set Discovery/ParticipantIndex = auto
• add machines to ping in Discovery/Peers, e.g.,:
<Peer address=“apple”/> 
<Peer address=“banana”/> 
<Peer address=“cherry”/> 
<Peer address=“durian”/>
• The “Participant index” bit is not very user-friendly …

Retransmit on slow networks
• Taken from PR#558
• Macbook Pro to a (cheap) AP then via 100Mb/s Ethernet to an RPi3, out-of-the-box
• Packet loss at 100kB and 1MB messages but negligible received duplicates (brown)
• Retransmits explode at 10MB messages, with signiﬁcant duplication
• caused by retransmits often taking more than 100ms to ﬁll the gap
• increasing Internal/NackDelay helps (it never delays the 1st retransmit)

Retransmit on slow networks (2)
• For laughs:
• same setup but good AP
• RPi3 can’t cope, Fast Ethernet is too slow for the WiFi
• that is, a very large part of the packets gets dropped in the network
• publisher uses more bandwidth for retransmits than data!
• I don’t know how to tune Cyclone for this …
100 1k 10k 100k 1M 10M
0 100 200 300 400 500 600
0.10
1
10
100
1000
Mb/s
new ap

Updating Cyclone in your ROS 2 build
• A full source build always works, but updating a binary install is possible, too
• use an “overlay workspace”, i.e., create a workspace, clone cyclonedds and
rmw_cyclonedds packages
• do colcon build —cmake-args -DCMAKE_BUILD_TYPE=Release
• “. install/local_setup.bash” to point to the newly built ones
• For “cyclonedds”: use the master branch
• For “rmw_cyclonedds” use the branch matching the ROS 2 release you are using
• for Dashing & Eloquent, that’s the “dashing-eloquent” branch

No communication at all
• Usually caused by any of:
• Wrong network interface
• No/ﬂakey multicast support (see a few slides back)
• Firewall blocking trafﬁc

No communication: network interface
• General/NetworkInterfaceAddress selects the interface
• specify an interface name
• IP address of the machine
• IP address of the network (i.e., mask out the host part)
• e.g.: with vmnet8: […] inet 172.16.93.1 netmask 0xffffff00, any of
• General/NetworkInterfaceAddress = vmnet8
• General/NetworkInterfaceAddress = 172.16.93.1
• General/NetworkInterfaceAddress = 172.16.93.0

Firewall blocking traffic
• There is a rule for computing the port numbers, but it takes a ton of parameters
• https://github.com/eclipse-cyclonedds/cyclonedds/blob/master/docs/manual/
conﬁg.rst#controlling-port-numbers
• short version: domain id 0 with multicast:
• 7400 and 7401 are used for multicast, random port numbers used for receiving unicasts
• if you set Discovery/ParticipantIndex to “auto” (same as with unicast discovery!)
• 7410 + 10k and 7411 + 10k used for receiving unicast data, k is the “participant index”
• data is currently always sent from a random port number

Sanity checking
• “ddsperf” tool is quite a practical way
• included in the packages installed by ROS 2
• “ddsperf -h” explains the options 
“ddsperf sanity” set ups many-to-many ping at 1Hz and is an easy way to check
• “ddsperf pub” publishes small messages as fast as possible 
“ddsperf pub 10Hz size 10k burst 100” sends bursts of 100 messages of 1kB each,
at 10 bursts/s
• “ddsperf sub” subscribes and shows the throughput
• several more modes and options …

Interference from other users in the network
• Use a different domain id
• supported directly in ROS 2
• doesn’t separate out multicast traffic
• only a limited range of domain ids available (32 bits in theory, but they also map to UDP port numbers)
• Use a different domain tag
• a configurable string included in initial discovery, nodes only communicate if they have the same tag
• doesn’t separate out multicast traffic
• e.g., Discovery/Tag = fruit
• Use a different set of multicast addresses
• be sure to only do that in combination with either (or both) of the above

How to change Cyclone settings?
• Set CYCLONEDDS_URI environment variable
• point it to XML config files
• embed configuration snippets in “abbreviated” XML
• e.g.: “CYCLONEDDS_URI=‘<Tr><V>config</><Out>stdout</></>’” 
to dump the configuration to stdout on startup
• comma-separated
• Configuration files can specify settings that apply to all domain ids or just to a single one
• this can be useful if you want to use multiple domains in your system
• XSD with descriptions:
• https://raw.githubusercontent.com/eclipse-cyclonedds/cyclonedds/master/etc/cyclonedds.xsd

Configuration example
$ cat cyclonedds.xml
<?xml version="1.0" encoding="UTF-8" ?> 
<CycloneDDS xmlns="https://cdds.io/config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://cdds.io/config https://raw.githubusercontent.com/eclipse-
cyclonedds/cyclonedds/master/etc/cyclonedds.xsd"> 
<Domain id="any"> 
<General> 
<NetworkInterfaceAddress>vmnet8</NetworkInterfaceAddress> 
<AllowMulticast>false</AllowMulticast> 
<MaxMessageSize>65000B</MaxMessageSize> 
</General> 
<Discovery> 
<Tag>fruit</Tag> 
<ParticipantIndex>auto</ParticipantIndex> 
<Peers> 
<Peer address="apple"/> 
<Peer address="banana"/> 
<Peer address="cherry"/> 
<Peer address="durian"/> 
</Peers> 
</Discovery> 
<Internal> 
</Internal> 
<Tracing> 
<Verbosity>finest</Verbosity> 
<OutputFile>cdds.log.${CYCLONEDDS_PID}</OutputFile> 
</Tracing> 
</Domain> 
</CycloneDDS> 
$ export CYCLONEDDS_URI=file://$PWD/cyclonedds.xml

If all else fails …
• Wireshark … but wireshark is not so pleasant …
• Traces
• Crazy level of detail in the traces, but this slows stuff down
• I’ve been told no-one knows how to read them but me …
• Ping me on GitHub :)

ROS 2 Foxy with Eclipse Cyclone DDS | Philly ROS Meetup July 20th 2020

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to ROS 2 Foxy with Eclipse Cyclone DDS | Philly ROS Meetup July 20th 2020

Similar to ROS 2 Foxy with Eclipse Cyclone DDS | Philly ROS Meetup July 20th 2020 (20)

Recently uploaded

Recently uploaded (20)

ROS 2 Foxy with Eclipse Cyclone DDS | Philly ROS Meetup July 20th 2020