This presentation is about a lecture I gave within the "Software systems and services" immigration course at the Gran Sasso Science Institute, L'Aquila (Italy): http://cs.gssi.infn.it/. http://www.ivanomalavolta.com

1. Ivano Malavolta
Assistant professor
Vrije Universiteit Amsterdam
Mission planning of
autonomous quadrotors
VRIJE
UNIVERSITEIT
AMSTERDAM

2. Hello
Empirical software engineering
+ Software Architecture + MDE
+
applied to
Complex systems
Autonomous robots
Mobile-enabled systems
If you think good architecture is expensive,
try bad architecture.
... Brian Foote and Joseph Yoder

3. Roadmap
Background
The platform
Flight plan synthesis
Adaptation at run-time
Support for any kind of robot
Conclusions

4. Background

5. High costs
team training and transportation
operating costs
Safety
significant risks (e.g., fire, earthquake, etc.)
Timing and endurance
exhausting shifts
activities stopped at night
Civilian missions today
INTUITION
A team of autonomous
quadrotors might work
together with staff

6. Special kind of helicopter with:
• high stability
• omni-directional
• smaller fixed-pitch rotors
à safer than classical helicopters
• simple to design and construct
• relatively inexpensive
http://goo.gl/FJFS5l
What is a drone for us?

7. Many civilian missions can be executed either by flying, ground or water robots
Using robots for civilian missions

8. Civilian missions can be executed by multiple robots
à lower mission completion time
à fault-tolerance
à highly-specialized robots
All the robots perform their actions to fulfil the common goal
of the mission
however...
common goal
Multi-robot missions

9. On-site operators must be expert of all the types of used
robots
in terms of dynamics, hardware capabilities, etc.
On-site operators have to simultaneously control a large
number of robots during the mission execution
Robots provide very low-level APIs and very basic
primitives
error-prone development
task-specific robots
no reuse
These issues ask for
• abstraction
• automation
Challenges

10. Application scenario
Mission
to monitor the CO2 level in a geographical area by monitoring
it along a grid composed of cells of size 12x12 meters
Contextual entities
obstacles
emergency areas
no fly zones
home
NO FLY ZONE

11. MDE allows all stakeholders to focus on models of the
mission with concepts that are:
• closer to the application domain
• independent from the specific robot technologies
• enabling automation à autonomous robots
http://mdse-book.com
Model-driven engineering for robotics

12. The platform

13. Mission modeling
Mission
Context
Map
MML
BL
Behavior specification
Detailed flight plan synthesis
Robot type
models
Mission
Execution
Engine
this layer is extensible
RL

14. Mask complexity
• usable by non-technical experts
• domain-specific concepts
• separation of concerns
Independence w.r.t. the types of robots
• behaviour generation capability
Reuse
• context models
• robot models
Robots must be autonomous
Principles Mission
Context
Map
MML
BL
Behaviorspecification
Detailed flight plan synthesis
Robot type
models
Mission
Execution
EngineRL

15. Modeling languages Mission
Context
Map
MML
BL
Behaviorspecification
Detailed flight plan synthesis
Robot type
models
Mission
Execution
EngineRL

16. Operator
in-the-field stakeholder specifying the mission
Robot engineer
models a specific kind of robot
develops the controller that instructs the robot on how to
perform BL basic operations
Platform extender
extends the MML metamodel with new kinds of tasks
develops a synthesizer for transforming each new task to its
corresponding BL operations
MML
RL + software controller
MML + synthesizer
Involved stakeholders

17. Mission layer
sequence of tasks executed by a team of robots
Extension point
Monitoring mission language (MML)

18. Photogrid task
• identifies a virtual grid within an area
• each cell of the grid has a size of n meters
• each drone flies over each cell of the grid at an altitude of
z meters and takes a picture of the ground
Example of extension

19. Context layer
geographical areas that can influence the execution of the
mission
The focus is on spatial context
Monitoring mission language (MML)

20. Hardware and low-level configuration of each type of robot
Robot language (RL)
This makes RL models reusable
and shared across missions,
projects, and organizations

21. Atomic movements
and actions performed
by each robot of the
swarm
Behaviour language (BL)

22. Example (1)
MML model (in the tool)
NF1
NF2
home
RT
PGT

23. Example (2)
Robot model (Parrot)

24. Example (3)
Behavioural model
Drone&
D1&
Drone&
D2&
Drone&
D3&
Start&(ε,&ε)& Start&(ε,&ε)& Start&(ε,&ε)&
TakeOff&(ε,&ε)& TakeOff&(ε,&ε)& TakeOff&(ε,&ε)&
GoTo&(ε,&ε)&GoTo&(ε,&ε)& GoTo&(ε,&ε)&
GoTo&(ε,&{Photo})&GoTo&(ε,&{Photo})& GoTo&(ε,&{Photo})&
GoTo&(ε,{Photo,BroadCast(D3.R1.Done)})&
GoTo&(ε,&ε)&
Land&(ε,&ε)&
Stop&(ε,&ε)&
GoTo&(ε,&ε)&
Land&(ε,&ε)&
Stop&(ε,&ε)&
0GoTo&(ε,&{Photo,&&
BroadCast&(D2.PG1.Done)})&
0
GoTo&(ε,&ε)&
Land&(ε,&ε)&
Stop&(ε,&ε)&
GoTo(ε,&{Photo,&&
BroadCast&(D1.PG1.Done)})&
PG1 PG1
R1

25. Tool support
…
ROS
HTTPS and web
sockets
Web interface
Editor for MML
models
M2M transformation
+ models validation
+ mission workflow
manager
Layer of controllers that interpret BL
models at run-time
HTML5, CSS3,
JavaScript Java + OCL + Rosbridge
Any technology supported by ROS
Drone driver
any
C1
C2
C3
…
Any
protocol

26. Web UI
Mission Design
and
Behaviour
Synthesis
request MML &
drones configurations
1
2 MML concepts & drones
3 mission design
4 submit MML models
6 BL model and analysis results
5
behaviour analysis
and synthesis
7 start mission
Mission Execution
Engine
Mission modeling workflow

27. Mission execution
Mission Execution
Engine
6 send status*
(continuous flow)
start mission
setup drone
controllers
* steps executed in parallel
Monitoring
Engine
tasks execution
commands*
feedback*
feedback*
2
5
1
3
4 C1
D1

28. Mission simulation
Simulation-specific controller
• Software-In-The-Loop (SITL) platform
• Same components of real deployment Mission Execution
Engine
navigation
commands
(MAVLink)
telemetry
(MAVLink)
C1
MAV
Proxy
Drone
Kit
Arducopter
physics
simulator
physics data
Google Earth
drones positions at run-time

29. DEMO

30. Flight plan synthesis

31. OVERVIEW OF THE SYNTHESIS APPROACH
BL model automatic synthesis
BL
Mission entering
Mission leaving
Mission tasks
execution
…
…
dn
…
d2
…
d1
State transition
State
Synchronization and communication message
Overview of the synthesis approach
MML
Each MML task is constrained by:
• geometry
• behavioural strategy
• actions

32. point
line polygon
volume
Geometries supported by MML

33. STRATEGIES AND ACTIONS SUPPORTED BY MML
sweep(d) – full coverage of an area discretized by a grid
whose cells dimension is d
search(target, d) – visit performed on a d-sized grid
towards the discovery of target (e.g., represented by a PNG
image)
track(m, m’, d) – it is like sweep(d) where m and m’are
the restart and stop messages, respectively
Actions:
• taking a picture
• making a video
• detecting the presence of CO2
• etc.
Strategies supported by MML

34. SYNTHESIS METHOD AT WORK ON THE PEM SCENARIO
Let’s go back to the example…
NF1
NF2
home
RT
PGT

35. SYNTHESIS METHOD AT WORK ON THE PEM SCENARIO
Context = ( (NF1,NF2), (OB) )
PGT = ( (p1,p2,p3,p4,p1), sweep(10), (doPhoto(2480 3508), i) )
Swarm ={ (d1, {PGT}), (d2, {PGT}), (d3, {RT}) }
Task dependency graph
( (t0, PGT, RT, t3), ∅ ∪ { (t0, {PGT, RT}) } ∪ { ({PGT, RT}, t3) } )
t0
PGT RT
t3
Synthesis method on the example

36. q4= p2
q1= p1
q3
q14
q5
q6
q15
q16
q2
q13
q12
q11
q7
q8
q9
q10
p3
p4
c1
c2
p1
Synthesis method on the example

37. SYNTHESIS METHOD AT WORK ON THE PEM SCENARIO
s1
0 TakeOff(p1.z) s1
Mission entering
s2s'''2
Goto(c1)
s''2
Goto(c2)
s'2
Goto(p1)Goto(home1)Land
s1
f land1
Mission leaving
non-fluid transition fluid transition Transition label syntax = <OP> |
<ACT> | <OP> / <ACT>
s'1 Goto(c2) s''1 Goto(p1) v1
v1
1
Goto(q1) / DoPhoto(…)
v2
1v16
1
…
NoOp
Mission tasks execution
Goto(q16) / DoPhoto(…)
NoOp
r1
NoOp
Goto(c1)
v17
1
Synthesis method on the example

38. SYNTHESIS AUXILIARY FUNCTIONS
Divide – distributes the geographical area of a task into a set of (sub-)areas
overlapping between no-fly zones, obstacle, and areas
no “cross-cutting” no-fly zones
smallest distance assignment criterion
1 point to 1 drone, 1 line to n drones (l/n)
polygon/volume partitioning algorithm (rif. H. Bast and S. Hert. The area partitioning problem. In Proceedings of the 12th Canadian
Conference on Computational Geometry, 2000)
Appr – generates the obstacle- and collision-free path that a drone d must travel to reach a
geographical area a
path planning problem in 3-dimensional world (rif. S. S. Skiena. The algorithm design manual, 1997. Stony Brook, NY: Telos
Pr, 504)
spherical obstacle enlargement
target points identification (vertexes of the sub-area)
trajectory definition = well-known visibility graph
Cover – takes a starting position s of a drone d, a geographical area a, and a real number r
representing the resolution of the grid that implicitly discretizes a
returns a set of <point,angle>
coverage path planning problem (rif. R. Mannadiar and I. Rekleitis. Optimal coverage of a known arbitrary environment. In Robotics
and Automation (ICRA), 2010 IEEE International Conference on, pages 5525–5530. IEEE, 2010)
minimum visit plan
based on the Boustrophedon cellular decomposition
Auxiliary functions

39. PROPERTIES OF THE SYNTHESIZED QBL MODEL
Safety
• avoidance of collisions (P1), and of traversing no-fly
zones (P2)
• no concurrency issues (P3)
P1&P2
• correct and complete specification
• Appr
• Cover
P3
• Divide
• no message lost
• sound mechanisms for sequencing, join, and fork
• a drone cannot be involved in concurrent tasks
Properties of the synthesized BL model

40. Adaptation at run-time

41. • Drone d1 of the team T must reach a target geographical position p
• Drone d1 identifies an obstacle along its trajectory towards p
• if the obstacle is avoidable (e.g., a tree):
• then d1 adapts its trajectory so that it avoids the obstacle to reach
p
• otherwise (e.g., a large building):
• the behaviour of d1 and other drones in T are adapted so that:
• the position p is still covered by another drone di∈ T
• d1 can cover some other points within the area
Scenario

42. • Safety as first-class element in the design of the system
• Clear separation of concerns between the generic safety-specific
mechanisms and the functional behavior of the robots while
defining the mission
• Decentralized and collective adaptation
• Multiple entities are adapted simultaneously so that
• critical runtime condition are properly addressed
• the working consistency and the collaboration of the ensemble are
preserved
Overview
Sustainable safety via collective adaptationGOAL

43. Software architecture Key features:
• collective adaptation in a
decentralized fashion
• managed at run-time
• new solvers can be
introduced at any time
• separation of concerns

44. • Entities - basic building blocks representing the actors and components of
the system
• Ensembles - set of roles that can be played by participating entities
Ensemble role specification
• Issue - definition of a critical situation that can happen to a role of an
ensemble
• Solver - ability of a role to handle certain types of issues
Issues resolution

46. Support for any kind of robot

47. • Modeling languages
• special care has been put in MML and BL
• Modeling infrastructure
• overall software architecture
• model transformations
What is independent of the used
robots?

48. • Definition of a generic editing environment
• flexible w.r.t. the use of geographical concepts
à we need to relax the constraint of always referencing to a map
• Evaluate alternative architectures
• decentralized
• no assumption of continuous connectivity with ground station
• P2P communication between robots
• Experimentation with other kinds of physical robots
• e.g., underwater autonomous robots
What needs to be done?

49. Step 1 - check the expressiveness of the languages
• we reverse engineered the AUV used in the RALF3 project
Step 2 - extend the modelling languages
• Mission tasks
• Context
• Behaviour
• Robot
Extension for underwater robots
no extension needed Winning features:
• geo and relative coordinates
• behavioural movements driven by
specific conditions (e.g., vision)
• objects as behavioural targets

50. Extended robot language for UAVs

51. Conclusions

52. EXPERIMENTATION
models@runtime to control the
mission execution and manage
adaptation
Support other types of robots
We need you!
Mission Execution
Engine
navigation
commands
(MAVLink)
telemetry
(MAVLink)
C1
MAV
Proxy
Drone
Kit
Arducopter
physics
simulator
physics data
Google Earth
drones positions at run-time

53. Davide Di Ruscio, Ivano Malavolta, Patrizio Pelliccione, and Massimo Tivoli. Automatic Gen-
eration of detailed Flight Plans from High-level Mission Descriptions. In ACM/IEEE 19th
International Conference on Model Driven Engineering Languages and Systems (MODELS),
pages 45–55. ACM/IEEE, Oct 2016.
Federico Ciccozzi, Davide Di Ruscio, Ivano Malavolta, Patrizio Pelliccione (2016) Adopting MDE
for Specifying and Executing Civilian Missions of Mobile Multi-Robot Systems IEEE Access
Journal. http://dx.doi.org/10.1109/ACCESS.2016.2613642
Darko Bozhinoski, Davide Di Ruscio, Ivano Malavolta, Patrizio Pelliccione, Massimo Tivoli (2015).
FLYAQ: Enabling Non-Expert Users to Specify and Generate Missions of Autonomous
Multicopters. In 30th IEEE/ACM International Conference on Automated Software Engineering
(ASE 2015).
Darko Bozhinoski, Ivano Malavolta, Antonio Bucchiarone, Annapaola Marconi (2015). Sustainable
Safety in Mobile Multi-Robot Systems via Collective Adaptation. In Ninth IEEE International
Conference on Self-Adaptive and Self-Organizing Systems, SASO 2015, Cambridge,
Massachusetts, USA, September 21-25, 2015,
Darko Bozhinoski (2015). Managing Safety and Adaptability in Mobile Multi-Robot Systems. In
Proceedings of the 11th International ACM SIGSOFT Conference on Quality of Software
Architectures, pp. 135–140.
References

54. Davide Di Ruscio, Ivano Malavolta, Patrizio Pelliccione (2014). A family of Domain-Specific
Languages for specifying Civilian Missions of Multi-Robot Systems. In Proceedings of the 1st
International Workshop on Model-Driven Robot Software Engineering (MORSE), pp. 13–26.
Davide Di Ruscio and Ivano Malavolta and Patrizio Pelliccione (2014). The Role of Parts in the
System Behaviour. In Software Engineering for Resilient Systems - 6th International Workshop,
SERENE 2014, Budapest, Hungary, October 15-16, 2014. Proceedings, pp. 24–39.
Davide Di Ruscio, Ivano Malavolta, Patrizio Pelliccione (2013). Engineering a Platform for Mission
Planning of Autonomous and Resilient Quadrotors. In Software Engineering for Resilient
Systems - Fifth International Workshop, SERENE 2013, pp. 33–47.
References

55. Contact
Ivano Malavolta |
Assistant professor
Vrije Universiteit Amsterdam
iivanoo
i.malavolta@vu.nl
www.ivanomalavolta.com