Robotprogrammatie: enkele lessen uit de praktijk, trends en uitdagingen

Robotprogrammatie: enkele
lessen uit de praktijk, trends
en uitdagingen
Eric Demeester
Faculteit Industriële Ingenieurswetenschappen
Technologiecampus Diepenbeek
Onderzoeksgroep ACRO
CEVORA IT event, 25 oktober 2015

Robotprogrammatie: enkele lessen uit de praktijk
Outline
2
• ACRO activities: robots with vision and a plan
o Mobile, assistive robots with vision and a plan
o Agricultural robots with vision and a plan
o Industrial robots with vision and a plan
• Programming robots: learnt lessons
o The RADHAR project as an example
o Requirements
o Solutions
• Trends and challenges for the future

Overview ACRO activities
3
• Profibus & profinet, PLC (UCLL)
o Training for operators in industry
o Service and consultancy regarding industrial networks
• Vision and robotics (KU Leuven)
o Vision: choice of cameras, optics, lighting, image processing
o State estimation and machine learning, e.g.:
• Object recognition/classification and 6D pose estimation
• Mobile robot localisation and mapping
• Estimation of human intentions by robots
o Sensor-based decision making and planning, e.g.:
• Collision-free trajectory planning for mobile robots and industrial
manipulators
• Shared human-machine control
Remark: ACRO is part of Faculty of Engineering Technology => research closer to
industry (higher TRL levels)

Outline
4
o Requirements
o Solutions

Introduction
5
• Semi-autonomous robotic wheelchairs
o Motivation:
• Many elderly and disabled people suffer from a reduced mobility
• Electric wheelchairs enhance this mobility, but manoeuvring an
electric wheelchair is often difficult and time-consuming
• Consequences: accidents, frustration, dependence on others,
reduced social contact, reduced self-esteem, lower quality of life
o Solution? Equip electric wheelchairs with sensors and
computing power
=> Combine the strengths of both human (global planning) and
computer (fine motion control)
Collaborations/cases with: Permobil, Invacare, HMCI, Ottobock, BlueBotics, National MS Centre
Belgium, Windekind + various research institutes

Research questions
6
• Semi-autonomous robots should be able to answer
three questions:
1. Where am I?

Research questions
7
2. Where am I going?
o Key assumption: user should not be required to
communicate explicitly the desired navigation assistance
o Consequence: user (navigation) plans or intentions are
hidden and should be estimated from uncertain user signals
and sensor signals
o This is the problem of plan (intention) recognition:
Plan recognition is the problem of inferring the goal of
an actor and his plan to achieve this goal, based on a
sequence of actions performed by the actor.

Research questions
8
3. How should I get there?
o Even if it is known where the user would like to drive to, it
remains unclear how the manoeuvre should be executed
jointly by human and robot
o This is the problem of shared control:
The situation in which the control of a device is
shared between one or more users and one or more
robotic controllers.

Research questions
9
o Results with different interfaces: standard joystick, switch
interface, brain-computer interface, haptick joystick

Outline
10
o Requirements
o Solutions

Autonomous apple picking
11
• Introduction:
o It is hard to find personnel for fruit picking
o High wages, low prices for fruit on the global market
o Orchard management: selective spraying/fertilization
• Research questions:
o Autonomously determine the fruit’s position
o Autonomously pick the fruit without damaging the fruit nor the
tree, keeping the stem on the apple
o Feasibility study: is it possible to perform this operation with
vision and a robot arm?
o Can it be performed fast enough?
o And many other ... (logistics, wheather conditions, fruit
selection on the spot, etc.)

Autonomous fruit picking
12
• Current state:
This image cannot currently be displayed.

Outline
13
o Requirements
o Solutions

Random bin picking (IWT Tetra RaPiDo)
14
• Introduction:
• Nowadays, objects are typically fed to robots in a well-defined,
mechanised manner, which takes time and money to setup
• Alternative: random bin picking:
• Make a 3D scan of a bin with randomly positioned objects
• Recognise and estimate pose of objects
• Compute a collision-free trajectory taking size of robot, gripper, objects,
environment into account
• Research goals:
o Build an open demonstrator
o Evaluate existing open source code
o Focus on integration of vision and robotics
o Speed up of object detection and path planning
o Evaluate potential of novel sensors and algorithms

Random bin picking (IWT Tetra RaPiDo)
15
• Current state:
o Bin picking setup with sheet-of-light working
o Analysis of randomised path generation techniques
Collaborations/cases with: ABB, KUKA, Sick, Materialise, Ceratec, Egemin, cards PLM
solutions, Clock-O-Matic, Dewilde Engineering, Vision++, Exmore Benelux, Flanders Food,
Intrion, Optidrive, PEC, Intermodalics, Rabbit, Robberechts, Robosoft, Robomotive, Sedac
Meral, SoftKinetic, Beltech, Borit, Meditech, Sirris, Phaer,Gibas, Octinion, Dana, Van Hool

Flexible robot welding (IWT VIS SmartFactory)
16
• Introduction:
o Current trends:
• globalisation, individualised products, small lot sizes =>
production industry should become highly flexible
• High wages, lack of skilled labour, global competion => production
industry should become highly automated
• Factory of the future: combine flexibility and automation by using
novel technologies: machine vision, path planning software, force
control, intelligent transport
• Research goals: zero ramp-up & auto-programming
• Recognise and determine the pose of objects to be welded using
2D or 3D vision,
• perform quality checks before and after welding
• Computation of optimal (time, energy) trajectories for robots
taking into account: robot geometry, kinematics and dynamics,
singularities, components mounted on the robot, environment

Flexible robot welding (IWT VIS SmartFactory)
17
• Current state:

Vision-based wood waste sorting
18
• Introduction:
o SME innovation project for NV Gielen
o Business model: grind wood waste to wood chips and sell
o Problem: percentage MDF: 15% to 50% => price of wood
chips very low
o Solution: separate MDF from wood
• Research questions:
o How to separate MDF from wood?
• Not possible using only infrared reflection, density, colour
• => classify using vision?
o How to classify it accurately enough?
o How to separate it fast enough?

Vision-based wood waste sorting
19
• Solution:
o Determine several features based on colour, texture, shape
o Train multilayer perceptrions to classify wood and MDF
o Onsorted input (45-55% MDF) to sorted output with 5%
MDF in wood output and 2% wood in MDF output
o Capacity: 7.5 tons/hour

Outline
20
o Requirements
o Solutions

The RADHAR project as an example
21
Vision: Robotic ADaptation to Humans Adapting to Robots
heterogeneous user groups
with (time)varying skills
dynamic, 3D environments
robots adapting to humans’
signals and responses
Requires life-long
adaptation between
two interacting
learning systems
(human & machine)

22
Consortium
2 user
groups
NMSC, Nationaal
Multiple Sclerosis
Centrum V.Z.W.
Windekind,
school for
children with
disability
3 companies
3
universities
1 research
institute

23
General overview of the framework

24
Developed hardware:

Outline
25
o Requirements
o Solutions
o Problems encountered

Requirements/challenges
26
o Hardware level:
• Be able to deal with different sensors, wheelchairs, interfaces,
people
• Real-time (predictable, fast) control even if data-intensive
o Algorithmic level:
• Easily transfer algorithms to other platforms
• Safety
• Calibration
• Debugging
o Robotics-related challenges:
• How to program the robot to perform what it should do?
• Environment: uncertain/unknown, changing, 3D, soft
• Robot: kinematics and dynamics, slippage, castors, size
• Concurrent tasks/processes/threads

Requirements/challenges
27
o System level: large-scale collaborative research and
development
• Several research groups across Europe develop software
• Portability, different platforms (Windows/Linux, Java/C++)
• Little time during integration weeks (tests on the hardware)
• Runtime robustness: stopping/crashing a module of a running
robot application should not result into an overall
stopping/crashing of the robot application
• Runtime flexibility: be able to change algorithms of modules at
runtime, extend the robot application with new modules at run-
time, probe ingoing and outgoing data of modules at runtime.

Outline
28
o Requirements
o Solutions

Solutions (lessons learnt)
29
1. Hardware abstraction and algorithmic level:
• Use of object-oriented programming (C C++):
• C++ has more features => more complex, but more readable and less
maintenance cost
• C++ might be slightly slower
• Timely reaction, real-time control is very important =>
maintainability, reusability is sometimes ignored (nicely
encapsulated software runs slower)
• => lesson learnt: avoid optimisation until it is needed
• use of GPU results were not that spectacular (< 10x improvement)
• ... sometimes just use a faster PC

30
• Use of design patterns,
• e.g. to abstract implementation details away from the functionality (e.g.
Factory Method): => easily adopt “higher-level” code on different
wheelchairs, user interfaces, sensors, ...

31
• Component-based software engineering:
• Compose sw from off-the-shelf and custom-built components
• Well-defined external interface that hides its internals, independently
developed from where it is going to be used, clear specification of what
it requires and provides and depends on => it can be composed
• 5C’s principle of separation of concerns separating the communication,
computation, coordination, configuration, and composition aspects in
the overall software functionality. Design patterns exist to decouple
these aspects …
- Composition: group entities together, model interactions (= an art)
- Computation: algorithmic part (the “useful” part of the sw)
- Configuration: change settings of a system
- Coordination: how do entities work together, life-cycle FSM
- Communication

32
• Evolution towards a more complex but very structured and motivated
Composition Pattern as the basic building block
Vanthienen, Klotzbücher, Bruyninckx, "The 5C-based architectural Composition Pattern: lessons
learned from re-developing the iTaSC framework for constraint-based robot programming", Journal
of Software Engineering for Robotics, pp. 17-35, May 2014.

33
2. Robotics related level:
• Probabilistic robotics: explicitly model uncertainty on all
information sources, and use that uncertainty when taking
decisions
• For intention estimation (plan recognition)
• For shared control

Plan recognition
34
• How to represent intentions?
1. We assume: users wish to reach a certain end pose pe (xe,
ye, θe) with an end velocity ve (ve, ωe)”, e.g.:

Plan recognition
35
2. Furthermore, we assume that users have a certain mental
trajectory in mind to arrive at an end state, e.g.:
i1

Plan recognition
36
• Intention estimation scheme:
1. Generate plan hypotheses i based on local paths or paths to
learned end poses or end poses indicated on an a priori
map.

Plan recognition
37
2. A probability distribution is maintained over the set of
possible user plans i. Initially, a uniform distribution is
adopted;
3. This distribution is updated every time new user signals uk
are obtained according to Bayes’ rule.

Plan recognition
38
• Example of plan recognition performance:

Shared control
39
• The probability distribution over user plans may be
multi-modal
• Nevertheless, decisions should be made at each
time instant regarding navigation assistance
• 3 approaches have been proposed to make these
decisions:
o Maximum Likelihood (ML)
o Maximum A Posteriori (MAP)
o Greedy Partially Observable Markov Decision Process
(greedy POMDP)

Shared control
40
• Maximum likelihood versus Maximum a posteriori
versus POMDP

Shared control
41
• Example of shared control performance:
o Benchmark test: visit the goal locations 4 – 9 – 6 – 11 – 2 –
14 – 5 – 10 and back to position 4
o Execute this in user control mode, in ML shared control
mode, in POMDP shared control mode
o In shared control, the wheelchair can drive farther than in
user control mode, with the danger of making wrong
decisions and driving too far

Shared control
42
• Example of shared control performance:

43
3. System level:
• we adopted ROS (Robot Operating System), an open-source
middleware initiative that allows:
• Easy and +/- efficient data exchange
• Tools (rosbags) for data collection and replay
• Dynamic configuration of components’ (nodes) parameters
• Interactive GUI for data visualization
• ... And other (multi-language support – C++/Python), support for
distributed systems

44
• We developed a module base class, built on top of ROS
• Used to share common functionality between modules (reuse)
• This implemented a finite-state-machine: stop/start a component, be
robust against errors, ...
• Health monitor: each component sends a heart-beat message to
let a central controller know about the wellbeing of a component
• The central controller sends a heart-beat to a hardware watchdog
• A GUI allows to display the health state of all modules, to
activate/deactivate modules
• This worked for us, but:
• Required lots of integration work and programming
• Will probably be duplicated by others

Outline
45
o Requirements
o Solutions

Trends and challenges for the future
46
• Probabilistic programming
o Trend towards probabilistic robots (due to many successes)
o Can we incorporate these probabilistic and machine learning
concepts into the programming language?
• E.g. prob distributions as new data type, automatically tune code
using built-in learning functions (program fails => indicate the
correct behaviour and learn from this example)
o Initial successes: much fewer lines of code; e.g. Darpa grand
challenge (autonomous car): 100.000 lines of code
o But: how can it be debugged (randomness plays a crucial
role)

47
• Insurance and liability
o Robots should be safe, e.g. implementing obstacle avoidance
but how reliable are these? Bugs keep popping up, in
unpredictable way (e.g. memory leaks)
o E.g. by feeding random input or pseudo-random input and
see if the system crashes - a sort of automated unit tests
rather than manually crafted unit tests
• Currently, typically only “nominal” behaviour is evaluated; for
safety “worst-case” behaviour is important
• Reliable communication

48
• Plug-and-play
o there are no standard methods for connecting sensors,
motors, actuators, cameras and other components to robots
• E.g. plug a device in, detect the device, install a driver, know what
can be performed with it, ...
o It’s starting (Orocos, ROS, OpenCV), but this requires still
much programming at lower levels
• Software deployment support:
o Robot “apps” should be made to work easily on different
platforms, have example programs available and good
documentation

Robotprogrammatie: enkele lessen uit de praktijk 49
Dank voor uw aandacht!
Eric Demeester
eric.demeester _at_ kuleuven.be
tel: +32 (0)11 27 88 15

Robotprogrammatie: enkele lessen uit de praktijk, trends en uitdagingen

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (6)

Similar to Robotprogrammatie: enkele lessen uit de praktijk, trends en uitdagingen

Similar to Robotprogrammatie: enkele lessen uit de praktijk, trends en uitdagingen (20)

Recently uploaded

Recently uploaded (20)

Robotprogrammatie: enkele lessen uit de praktijk, trends en uitdagingen