Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Improving User Experience with Ubiquitous QuickBoot
Jeff Tranter, ICS
Andrew McNaughton, Ubiquitous AI
2
Towards End-User Extensibility in Marine Robotics
Charles Cross
CTO & Co-Founder
charles@missionrobotics.us
The Problem
Technology in the field of marine robotics is currently extremely fragmented.
Unlike in many other fields of rob...
Our Team
Highly cross-functional with extensive experience in marine, aerospace, and ground systems
4
Brian Grau
CEO
Mecha...
The Mission
Unify and Expand Marine Technology
Interoperability and extensibility can be achieved through the use of open ...
ROV and AUV Systems as a Case-Study
ROV (Remotely Operated Vehicle):
● Traditional name within the industry for what you m...
Past Lessons Learned
Over time, one of the biggest lessons learned working in this space is that there is
no one-size-fits-...
Who are the Users?
Three customer segments that we are working to serve:
● Developers
● Create novel technology and focus ...
“How do I…”
Both in our discussions with customers and potential customers, as well within
popular forums and communities,...
Some Underlying Causes
Some of the reasons these questions come up time and again include…
● Lack of shared comm/API stand...
Where does that leave us?
The results of these issues include...
● Vendors creating walled gardens
● “All of our products ...
A Path Forward
We believe that one of the most effective solutions to the aforementioned issues
is to adopt open standards...
DDS Quick Summary
● Completely peer-to-peer
● No server/broker required for communication between applications/machines
● ...
DDS Quick Summary
14
ROS2 Quick Summary
● Introduces a layer of abstraction around the middleware
● Simplifies development for users
● Reduces D...
Common Robotic System Architecture Example
16
● One or more microcontrollers
○ Often running an RTOS
○ Real-time tasks
○ F...
Putting it all Together - The MR Approach
Hardware Development Approach
● Drop-in replacement for commercially available v...
Putting it all Together - The MR Approach
Software Development Approach
● Provide a system that works out of the box with ...
Hardware Overview
Features:
● Integrated motherboard
● Recent generation of MCU
● Support for various App Processor models...
20
21
22
Software Overview
Features:
● Docker-based embedded Linux platform
● Read-only Host OS running application containers
● Ma...
Developer Tools
24
● Docker based dev & deployment environment
○ Simplifies building, testing, and deploying software
○ Iso...
25
MicroROS / DDS XRCE
ROS2 / DDS
Options for Extensibility
26
There are five key paths to extend the system:
● Certain plug-and-play features are already bu...
“How do I…”
“… Add more cameras?”
Two main approaches:
● If the camera is already supported by mrOS, plug it in and configu...
28
“How do I…”
“... create a new sensor/actuator that can be integrated?”
If the device can be controlled using I/O available...
30
● Arduino sketch sends messages to App Processor
○ Messages conform to a simple JSON-based API
■ Contains unique messag...
“How do I…”
“... modify the pilot application to do or show X?”
● Introspect the fields and convert them into a dynamic mod...
Custom User Components
While the app comes pre-packaged with common components, we want to
provide a way for the user to c...
Resources
● ROS2 Resources and Documentation
● https://docs.ros.org/en/galactic/
● https://micro.ros.org/
● DDS Tutorials ...
How to Add Actuator/Sensor Data and Associated
UI to a Closed-Source Qt/QML App?
34
Qt App
“Ground Station”
(closed source...
What is Qt?
35
Application development framework (not just user interface)
Two user interface frameworks: widgets (desktop...
Ways to Extend a Closed-Source Qt App
QPluginLoader
QJSEngine (formerly Qt Script)
Dynamic QML
Loader
Binding
Connections
...
Loader Loads QML User Interface at Run Time
37
Defining C++ Properties for QML at Compile Time
38
PROPERTY Super Macro
39
QQmlPropertyMap Defines Properties at Run Time
40
Sample App
41
Object.h
42
Object.cpp
43
main.cpp
44
Main.qml
45
qml/Boolean.qml
46
qml/Number.qml
47
Thank you!
48
Any questions?
Upcoming SlideShare
Loading in …5
×

0

Share

Download to read offline

Leveraging Open Standards to Build Highly Extensible Autonomous Systems

Download to read offline

This webinar is for software developers and product leaders interested in using open standards to build extensible autonomous or robotic systems.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Be the first to like this

Leveraging Open Standards to Build Highly Extensible Autonomous Systems

  1. 1. Improving User Experience with Ubiquitous QuickBoot Jeff Tranter, ICS Andrew McNaughton, Ubiquitous AI
  2. 2. 2 Towards End-User Extensibility in Marine Robotics Charles Cross CTO & Co-Founder charles@missionrobotics.us
  3. 3. The Problem Technology in the field of marine robotics is currently extremely fragmented. Unlike in many other fields of robotics, such as unmanned aerial and ground systems, there are few, if any, vehicle platforms, software frameworks, and commonly used standards upon which to build. The result? ● Excessive reinventing of the wheel ● “Build it from scratch” or “Settle for less” decision-making ● Higher development costs and barriers to entry ● Products and systems that do not interoperate ● Slower pace of innovation and a 10-20 year lag behind similar fields 3
  4. 4. Our Team Highly cross-functional with extensive experience in marine, aerospace, and ground systems 4 Brian Grau CEO Mechanical Engineer and production specialist who has been building and shipping ROV systems around the world for the last ten years. Charles Cross CTO Walt Holm Director of Engineering Software Engineer specializing in robotics with an emphasis on creating modular, extensible, and connected systems. Electrical Engineer with a diverse background solving instrumentation challenges in aerospace, marine, and medical fields. John Clauss Operations Lifetime of marine operations experience, including extensive archaeological work, particularly in the Lake Tahoe area.
  5. 5. The Mission Unify and Expand Marine Technology Interoperability and extensibility can be achieved through the use of open standards and the creation of electronics and software solutions that are inherently designed with these goals in mind. Empower Domain Experts, Fuel Innovation We want companies and individuals to spend their time answering important questions and solving unique problems, not fighting with their tools or otherwise having to waste their time and talent solving previously solved problems. Remove Distance as a Barrier By leveraging advances in both autonomy and connectivity, we seek to move past the traditional models of one-to-one and many-to-one operations, instead opting for a future where one operator can direct many systems, even those halfway around the world. 5
  6. 6. ROV and AUV Systems as a Case-Study ROV (Remotely Operated Vehicle): ● Traditional name within the industry for what you might call a manually piloted underwater drone ● Many size-classes and specifications, from backpackable observation-class to ship-bound work-class ● Within the last 10 years, great strides in the creation of low-cost vehicles (under $10K) AUV (Autonomous Underwater Vehicle): ● Often long, torpedo-shaped, in form factor ● Typically used for long range surveys ● Generally expensive, with few “low-cost” solutions 6 Riptide™ AUV WHOI’s REMUS AUV Trident™ Underwater Drone Customized BlueROV™ WHOI’s Jason ROV
  7. 7. Past Lessons Learned Over time, one of the biggest lessons learned working in this space is that there is no one-size-fits-all solution when it comes to the many tasks that underwater vehicles perform. For any particular job you may need: ● A different vehicle type (ROV vs AUV) ● A different thruster configuration/vehicle layout ● Different actuator and sensor payloads ● Tethered vs untethered system ● Operation in turbid vs clear water ● Large carrying/lifting capacity ● Zero to many cameras arranged in various ways ● Hot-swappable power/batteries or power-from-topside ● Etc... 7
  8. 8. Who are the Users? Three customer segments that we are working to serve: ● Developers ● Create novel technology and focus on innovation ● New vehicle concepts ● New sensor, actuator, and communication products ● New software capabilities (computer vision, ML, etc) ● Target specific problem domains (i.e. coral reef restoration, seafloor mapping, etc.) ● System Integrators ● Integrate existing off-the-shelf products into bespoke, ready-to-go vehicles ● Often target specific customers, industries, and geographic regions ● Understand their needs very well! ● Are often not in the business of electronics or software development ● End-Users ● Own and/or use the end vehicle/robot to perform a task or service (inspection, salvage, etc.) ● Likely to extend and upgrade their systems by purchasing COTS products 8
  9. 9. “How do I…” Both in our discussions with customers and potential customers, as well within popular forums and communities, we hear often hear the same questions posed: “How do I…” ● … add more cameras? ● … add a new sensor/actuator to the vehicle? ● … create a new sensor/actuator that can be integrated? ● … deploy custom software to the vehicle? ● … modify the pilot application to do or show X? ● … synchronize the recorded data in time? ● … use a different microcontroller/embedded computer? ● … display data on multiple monitors and machines? ● … communicate with the MCU from the app processor? ● … control X programmatically? 9
  10. 10. Some Underlying Causes Some of the reasons these questions come up time and again include… ● Lack of shared comm/API standards across the ecosystem ● Historical constraints and limitations from project origins ● Extensibility and developer friendliness are not core user stories ● Existing solutions are designed around manual 1-to-1 operations ● Communication and power distribution underwater is tough Unfortunately, the answer users often get back is: “Everything is open source, so you should be able to read and modify it to suit your needs” There is no consistent, general approach to adding new capabilities, because the systems are not modular or flexible enough. Also, though open source, there is a steep learning curve! 10
  11. 11. Where does that leave us? The results of these issues include... ● Vendors creating walled gardens ● “All of our products work together” ● Often black boxed and still provide no path for user development ● Higher cost tier ● Adopters of open source stacks trudging forward ● Hacks and workarounds to extend and improve systems ● Reqs and conventions of non-marine systems a lasting constraint ● Improvements outside the conventions cannot be upstreamed ● Technology becoming fragmented ● Everyone develops in isolation, taping systems together orthogonally ● Ex. Sonar systems communicating over entirely separate lines with different apps 11
  12. 12. A Path Forward We believe that one of the most effective solutions to the aforementioned issues is to adopt open standards and frameworks that have become common across the rest of the robotics industry. 12 ● DDS (Data Distribution Service) ● An open middleware standard for real-time, publish-subscribe, peer-to-peer communication. ● ROS2 (Robot Operating System) ● From ros.org: “...a flexible framework for writing robot software. It is a collection of tools, libraries, and conventions that aim to simplify the task of creating complex and robust robot behavior across a wide variety of robotic platforms.” ● Built on top of DDS
  13. 13. DDS Quick Summary ● Completely peer-to-peer ● No server/broker required for communication between applications/machines ● Supports dynamic discovery of peers within a system/network (no need to specify endpoints!) ● Topic-oriented communication ● A topic consists of a name and a type description ● Datawriters publish data samples to topics ● Datareaders subscribe to topics ● Datawriters and Datareaders are loosely coupled - Datawriters don’t care who or how many subscribe ● Quality of Service ● Applications must specify not only what data is available, but also how it is to be transmitted ● Datawriters and Datareaders must agree upon Quality of Service (QOS) policy to be used ● QOS policies specify aspects like reliability, historical caching, late-joiner behavior, and more ● Communication fully specified by combination of Data Types, Topics, and QOS ● Software written by two or more parties can interoperate with well-defined behavior ● Encourages modular system design and discourages vendor lock 13
  14. 14. DDS Quick Summary 14
  15. 15. ROS2 Quick Summary ● Introduces a layer of abstraction around the middleware ● Simplifies development for users ● Reduces DDS-specific learning curve by providing rational defaults and patterns for common use-cases ● Provides utilities, libraries, and data types common to many robotics use-cases ● Scheduling ● Multi-threading ● Data recording and logging ● Coordinate systems and transformations ● Simulation tools ● Common data types (geometry, point clouds, images, GPS, time, common sensor data, and much more) ● Command line utilities for introspection and debugging ● Etc… ● Allows users to package and share their software components ● Nodes - individual applications that can be composed to form complex behaviors and systems ● Libraries - reusable library elements that can be used across ROS projects ● Messages - Data type specifications used between nodes 15
  16. 16. Common Robotic System Architecture Example 16 ● One or more microcontrollers ○ Often running an RTOS ○ Real-time tasks ○ Flexible I/O interfaces ○ Sensor/actuator control ○ Sub-MCUs ● One primary App Processor ○ Often running Linux ○ Soft real-time tasks ○ Compute intensive tasks ○ High-bandwidth I/O ○ System coordination ● One or more remote C&C devices ○ Laptop, mobile device, etc. ○ Control & monitoring tasks
  17. 17. Putting it all Together - The MR Approach Hardware Development Approach ● Drop-in replacement for commercially available vehicle platforms/ecosystems ● Designed for ROVs and AUVs, but adaptable to other types of systems ● Dramatically improved image quality, latency, and increased camera count ● Improved processing performance (support CV/ML and other intensive tasks) ● Additional space in enclosure for accessories ● Improved overall ease of use; reduced manual labor ● Expanded I/O and flexible power architecture to support add-ons ● Enabling of onboard data and telemetry recording 17
  18. 18. Putting it all Together - The MR Approach Software Development Approach ● Provide a system that works out of the box with little to no configuration ● Support existing components within the ecosystem (sensors, cameras, etc) ● Build out complete OS distribution that takes care of common needs ○ Application scheduling ○ Recording & Playback ○ Software and Firmware Updates ○ Plug-and-play discovery/management of common devices (like cameras) ○ Fault and system resource monitoring ○ Remote access ○ Configuration Management (OS, Network, Vehicle settings, User Software settings, etc) ● Implement all external interfaces and APIs using DDS and ROS2 ● Provide an SDK and complete development environment that allows day-1 deployments ● Create a flexible GUI application that supports user extensibility 18
  19. 19. Hardware Overview Features: ● Integrated motherboard ● Recent generation of MCU ● Support for various App Processor models ● New cameras ● Superior low-light performance ● 130ms glass-to-glass latency standard config ● 70ms glass-to-glass ultralow-latency mode ● High-speed onboard storage ● Additional space for user hardware ● Additional connectorized I/O interfaces ● Flexible power management architecture ● 3.3V, 5V, and 12V buses ● Switchable rails for failsafing/load-shedding ● RTC for tracking real-world time ● Low-power warning system 19
  20. 20. 20
  21. 21. 21
  22. 22. 22
  23. 23. Software Overview Features: ● Docker-based embedded Linux platform ● Read-only Host OS running application containers ● Main container running Mission Robotics OS ● Additional containers dedicated to private User software ● OTA updates and remote fleet management capabilities ● DDS and ROS2 communication ● Vehicle and Client software implement both standalone DDS and ROS2 APIs ● mrSDK* available to simplify and accelerate development of user applications (both vehicle and client) ● Board/driver support for latest Nvidia platforms, designed for autonomous vehicles ● Hardware acceleration, computer vision, and machine learning features ● Complete set of services providing standard functionality ● Can be disabled by users if they wish to implement their own 23 *placeholder name
  24. 24. Developer Tools 24 ● Docker based dev & deployment environment ○ Simplifies building, testing, and deploying software ○ Isolation of components from different vendors ● CI/CD ○ Full build, test, and deploy pipeline ready on day one ● CLI tools ● Example codebase ○ Python & C++ DDS & ROS2 starter applications ○ Arduino & STM32F example code for MCU apps ● Remote vehicle connectivity and monitoring
  25. 25. 25 MicroROS / DDS XRCE ROS2 / DDS
  26. 26. Options for Extensibility 26 There are five key paths to extend the system: ● Certain plug-and-play features are already built-in ● V4L2 and other supported cameras are automatically detected and brought up (configurable) ● MCUs that implement supported protocols are automatically bridged to the DDS bus ● Explicitly supported devices with mrOS* drivers automatically brought up ● Create and deploy custom ROS2/DDS applications to the vehicle user containers ● Or any applications/protocols, if desired ● Create and deploy custom ROS2/DDS topside applications that interact via the public API ● Create plugins for the Qt piloting application ● Create standalone MCU + sensor/actuator devices that are tunneled to both the DDS bus and Qt app *placeholder name
  27. 27. “How do I…” “… Add more cameras?” Two main approaches: ● If the camera is already supported by mrOS, plug it in and configure if needed ● Camera Supervisor will automatically bring driver up for camera ● Camera data will be made available over ROS2/DDS ● Qt App will automatically discover new camera and allow streaming/control ● If the camera is not already supported, user can do one of two things: ● Write a plugin library for the Camera Supervisor to allow it to bring the camera up ● Must implement way to find/access the device, acquire frames, change settings ● Write a standalone ROS2/DDS application that handles the camera on their own ● By following the established type/topic API conventions, it should just work 27
  28. 28. 28
  29. 29. “How do I…” “... create a new sensor/actuator that can be integrated?” If the device can be controlled using I/O available on App Processor, write a new ROS2/DDS node. If more I/O is required, user can use an additional MCU, such as an Arduino: ● Write an Arduino sketch that can interact with the sensor/actuator ● Add some code that implements the public API over the serial interface ● We provide a ready to go library and examples to make this easy ● Two API paths: ● Use MicroROS/DDS XRCE to implement API directly on the existing databus ● Use a simplified JSON-based API that will be forwarded directly to the Qt app As an example, let’s consider the user adding a custom light module via Arduino... 29
  30. 30. 30 ● Arduino sketch sends messages to App Processor ○ Messages conform to a simple JSON-based API ■ Contains unique message name, fields, field types, and field values ● MCU supervisor running on App Processor detects user MCU ○ A bridge service is created for each user MCU ● Bridge directly forwards the messages to Qt App on special DDS topic ○ Also creates an “input” topic for messages to travel back to user’s MCU
  31. 31. “How do I…” “... modify the pilot application to do or show X?” ● Introspect the fields and convert them into a dynamic model representing the “backend” for the device ● Backend contains a property map for input values and output values ● Backend provides hints about the data it contains to choose an appropriate display widget ● Signals emitted upon value changes received ● Use the model as a data source/sink for frontend elements ● User can configure the Pilot activity or any customizable screen to adjust the layout ● The new data sources/sinks will be shown, along with “suggested display elements” based on hints ● App is pre-packaged with a number of common components: labels, dials, sliders, buttons, etc. ● Upon instantiating the display component based on configuration, inputs and outputs will be connected 31
  32. 32. Custom User Components While the app comes pre-packaged with common components, we want to provide a way for the user to create an entirely new widget/component that can be created during runtime without the need to compile anything and without any prior knowledge of the user’s data types built into the app. ● User creates a plugin containing: ● One or more QML file that encompasses a “Component” ● A metadata file that describes the signals/slots of the component and any accepted hints ● This plugin is loaded via a utility in the Qt app ● Or simply place the plugin folder into a specified location where the app will find it ● The new component is made available within the layout config ● At this point, user can drop the new component into various screens ● The signals/slots will be automatically connected, allowing real-time bidirectional updates 32
  33. 33. Resources ● ROS2 Resources and Documentation ● https://docs.ros.org/en/galactic/ ● https://micro.ros.org/ ● DDS Tutorials and Learning Materials ● http://www.laas.fr/files/SLides-A_Corsaro.pdf ● https://www.dre.vanderbilt.edu/~schmidt/PDF/dds-sos.pdf ● https://www.youtube.com/watch?v=GGqcrccWfeE ● ROS2, micro-ROS, and DDS in aerial drone systems (PX4/Dronecode) ● https://www.youtube.com/watch?v=lZ8crGI16qA 33
  34. 34. How to Add Actuator/Sensor Data and Associated UI to a Closed-Source Qt/QML App? 34 Qt App “Ground Station” (closed source) Actuators Sensors Vehicle QML User Interface User-Defined Controls User-Defined Environment Telemetry Commands
  35. 35. What is Qt? 35 Application development framework (not just user interface) Two user interface frameworks: widgets (desktop) and QML (touch) Open Source Cross platform (Windows, Linux, Mac, Android, iOS...) “Write once, compile anywhere” Triple licenced: GPL, LGPL, and commercial (recommended) C++ but other language bindings exist e.g. Python Actively developed by The Qt Company with services provided by ICS (ics.com)
  36. 36. Ways to Extend a Closed-Source Qt App QPluginLoader QJSEngine (formerly Qt Script) Dynamic QML Loader Binding Connections QQmlPropertyMap (dynamic properties) 36
  37. 37. Loader Loads QML User Interface at Run Time 37
  38. 38. Defining C++ Properties for QML at Compile Time 38
  39. 39. PROPERTY Super Macro 39
  40. 40. QQmlPropertyMap Defines Properties at Run Time 40
  41. 41. Sample App 41
  42. 42. Object.h 42
  43. 43. Object.cpp 43
  44. 44. main.cpp 44
  45. 45. Main.qml 45
  46. 46. qml/Boolean.qml 46
  47. 47. qml/Number.qml 47
  48. 48. Thank you! 48 Any questions?

This webinar is for software developers and product leaders interested in using open standards to build extensible autonomous or robotic systems.

Views

Total views

240

On Slideshare

0

From embeds

0

Number of embeds

95

Actions

Downloads

4

Shares

0

Comments

0

Likes

0

×