Intelligent robotics operating in a dynamic environment require a resilient system. The system should be able to handle known and unknown situations in the world around them and from time to time, be able to assess their own functioning. Today's systems, look at a task and only activate the specific configuration of components for what is needed to complete that task. The intelligent robotic system we envision however, does not have one specific task but should be able to handle a set of tasks, within a set of dynamic environments - likely to increase in number and complexity over time. This complexity might require a system to keep all components, necessary for all possible tasks, running at all times. Certainly not sustainable or efficient. An automatic reconfiguration, adapting to environmental changes and tasks confronting the system is the solution that we set out to prove and test. In this talk, we will take a closer look at how we used Grakn on real-life robotic systems enabling them to be self-aware and to assess and manage themselves at run time.

1. INCREASE SELF-AWARENESS ON
ROBOTIC SYSTEMS
Willeke van Vught

2. ABOUT ME
Willeke van Vught
I live in Rotterdam, Netherlands
Work at TNO
Background in Artificial Intelligence
focused on human-agent/robot teaming – Human factors
2 | test

3. THIS SESSION
Talk about ..
Self awareness
Self assessment
Self management
Of an autonomous vehicle operating in a dynamic environment within a human-robot team.
Project background
Implementation in GRAKN
Usage on real systems
Next steps
3 | test

4. ABOUT THE PROJECT – EXPLORATORY STUDY ON
AUTONOMOUS SYSTEMS
PROJECT GOAL
Building knowledge and expertise on:
Design and engineering of autonomous systems
…
Knowledge exploitation for:
Self-assesment, reconfiguration and optimisation
Threat assessment, task planning, task execution
Shared understanding for human-autonomy teaming (Social AI Layer)
….
4 | test

5. TOOLING
5 | test

6. TOOLING
6 | test

7. AM I COMPETENT..?
7 | test

8. FOCUS OF ON SELF ASSESSMENT AND SELF
MANAGEMENT
The autonomous vehicle receives a task from the commander/mission planner and is, within some
constraints, free to choose how it wants to execute this task.
8 | test
Assignment OK I will execute
the task
Can I perform the
task given my
external and internal
situation
awareness?
In this
configurations,
with these
settings..

9. FOCUS OF ON SELF ASSESSMENT AND SELF
MANAGEMENT
The autonomous vehicle receives a task from the commander/mission planner and is, within some
constraints, free to choose how it wants to execute this task.
9 | test
Assignment I can not execute
the task because ..
Can I perform the
task given my
external and internal
situation
awareness?

10. THE AUTONOMOUS SYSTEM
Effective execution of a
task requires:
▪ Understanding of the
world, and
▪ Controlling your effect
in that world.
Understanding builds upon
an information-knowledge
structure.
10 | test
|---------------autonomy---------------|
Camera
Pilot
AutoPilot
Plan external
actions
External
Sit. Awarns. Information:
External state
INS /
GPS
LiDARCamera
For each autonomous platform
Internal situational
awareness and management
Knowledge:
Ontology
learn
instantiate
|automation|

11. SELF-AWARENESS USING GRAKN
KNOWLEDGE ABOUT THE INTERNAL STATE OF YOURSELF
11 | test

12. MAIN MODEL
12 | test
System is composed of components
that work in configurations.
A component processes an
input to an output.
A component has an health-status
and a performance.
The inputs and outputs of the
component can have a
quality and a quantity.

13. EXAMPLE
13 | test

14. EXAMPLE
14 | test

15. (SUB-)CONFIGURATIONS OF COMPONENTS
Components work in one or
multiple
(sub-)configurations.
System can select the most
appropriate configuration
for the current task in the
current state.
15 | test

16. CHAINING COMPONENTS AND CONFIGURATIONS
16 | test

17. HIERARCHIES
17 | test

18. 18 | test

19. APPLICATION FOR
AUTONOMOUS VEHICLES
19 | test

20. APPLICATION AREA
Reason about :
Task execution
In this
configurations,
with these
settings..
20 | test
Assignment I can not execute
the task because ..
Can I perform the
task given my
external and internal
situation
awareness?

21. REASON ABOUT:
TASK EXECUTION
We know the components of the system
21 | test

22. REASON ABOUT:
TASK EXECUTION
We know the components of the system
We know the input and output of the task and of the actor
22 | test

23. REASON ABOUT: TASK EXECUTION
23 | test

24. DEMO
24 | test

25. APPLICATION AREAS
Reason about :
Task execution
Qualities and quantities
Cause of bad performance
25 | test
Assignment I can not execute
the task because ..
Can I perform the
task given my
external and internal
situation
awareness?

26. 26 | test

27. APPLICATION AREAS
Reason about :
Task execution
Qualities and quantities
Cause of bad performance
Component assessment
27 | test

28. REASON ABOUT: COMPONENT ASSESSMENT
28 | test

29. APPLICATION AREAS
Reason about :
Task execution
Qualities and quantities
Cause of bad performance
Component assessment
Component management
29 | test

30. REASON ABOUT: COMPONENT MANAGEMENT
Manager that is responsible for maintaining a component’s requested output performance
Examples:
Adapt threshold of a (detection) algorithm
when the output is low and preferred
output is high.
For example in an unknown environment
and target detection is critical.
Change the output of a component by activating
another component
For example if it is dark and the camera
output is low, the system could turn on
a small light to increase sight.
30 | test

31. APPLICATION AREAS
Reason about :
Task execution
Qualities and quantities
Cause of bad performance
Component assessment
Component management
Learn about:
New configurations
Unknown tasks
31 | test
Adding:
Uncertainty
More info about outside world (on task
execution)
Teaming (reason about performing actions
together, knowing each others strengths, ..)
Calculate the quality of task execution
…
Suggestions??

32. THANK YOU FOR YOUR
ATTENTION
Take a look:
TNO.NL/TNO-INSIGHTS