1. The document discusses the challenges of developing autonomous robotic systems that can operate in the real, open world as opposed to closed, prepped environments. 2. Key challenges include dealing with uncertain and unknown observations from sensors in the real world and being able to reason about probabilistic information and novel situations. 3. The document proposes using techniques like Markov logic networks and Bayesian networks to help robots reason about uncertainties and handle novelty through probabilistic reasoning.

1. SNOW: AUTONOMY IN THE OPEN WORLD
BY JORIS SIJS

2. Joris Sijs
Electrical Engineering TU Eindhoven
MSc in Systems and Control
PhD in State estimation (Kalman filtering)
Visiting researcher
University of Karlsruhe
TU Delft
Scientist at TNO
Image processing
Sensor networks
Autonomous robotics
SHORT BIO

3. Autonomous, robotic systems
Part of a team
Real world
General tasking
A robot that is able to
conduct a part of the
operation autonomously
Navigation
Exploration
Surveying
WHAT ARE THEY TASKED TO DO
PURPOSE
Automated, robotic systems
Stand-alone
Prepped surrounding
Repetative motion
A robot that is able to repeat
the same task over and over
Production
“Inspection”

4. To decrease manpower and maintain or increase performance
Extend endurance
Extend capabilities
Replace human operation
To keep people safe
Less physical harm and mental stress
Less risky situation entering
Solve ‘unsolvable’ dangers (e.g. nuclear threat)
To operate remotely
Extend or cope with communication limits
Operations in remote environments; expertise at a distance
WHY WOULD A ROBOT BE A SOLUTION
NEEDS

5. FROM AUTOMATED IN A CLOSED WORLD
TO AUTONOMOUS IN AN OPEN WORLD
SNO
from closed & prepped world to open and real world
How do robotic systems know how to respond/behave in the real, open world?

6. FROM AUTOMATED IN A CLOSED WORLD
TO AUTONOMOUS IN AN OPEN WORLD
SNO
from closed & prepped world to open and real world
How do robotic systems know how to respond/behave in the real, open world?

7. WHICH METHODS TO KNOW
SOME CAPABILITIES FOR A ROBOT
Interaction
Cognition
Simulation Engineering
Perception
Navigation
Scene graph &
Scene assessment
Motion Planning
Localisation
Mapping (SLAM)
Verification & Validation
Design & Implementation
Knowledge Eng.
Operational post
Legal directions
Edge computing
Human Machine Language
System-system Collaboration
Human Machine Delegation
15-02-2022
7
Governance
Platform
Obejct Manipulation
Knowledge Discovery
Mission Planning / Task Execution
Active learning / Hypothesis testing
Self-Management
Actuators Sensors
Communications
Detection/ Recognition/ Tracking
Active Perception / Hypothesizing
Model-Based Engineering
Self-assessment
Env. and object
assessment
World (knowledge) modelling
System (knowledge) modelling
Telepresence
Immersive reality (Social) XR
Scene Management
Env. & object
Management
Digital Twin
ROS
Semantic Navigation
Fundamental R
Mature (partially)
Practical R&D
System integration
Transparency

8. HARDWARE FOR THE REAL WORLD

9. HARDWARE FOR THE REAL WORLD

10. Open world means
Being sub-confident
Encountering novelties
… and cope with them
UNCERTAIN AND UNKNOWN OBSERVATIONS
REAL WORLD: CAMERA IMAGES
pole
pole
zebrapad
zebrapad
zebrapad
truck truck
road
road
person
person
person
person
person
person
person
person
person
person
sidewalk
???

11. Open world means
Being sub-confident
Encountering novelties
… and cope with them
UNCERTAIN AND UNKNOWN OBSERVATIONS
REAL WORLD: MICROPHONE RECORDINGS
Ideal versus on Robot
Tough conditions…

12. REAL WORLD USE-CASE
SNOW
2020
SNOW
2021
~ ~
locate & identify
locate & assess
Newly found room (level 5)
change in the problem itself
Door that is blocked (level 4)
change in relations between rooms
Person with particular clothing (level 2)
change in features of an object
New conditions in a room (level 4)
change in the expected performance
Person that needs rescue (level 3)
change in relations between person and ladder

13. Detected New Human
REAL WORLD BOOLEAN?

14. Novelty level 0: update instances
Automatically update attributes, such as Position
EVAL. AUTONOMY: ENVIRONMENTAL COMPLEXITY SNO

15. Novelty level 0: update instances
Automatically update attributes, such as Position
Novelty level 1: re-use classes
Instantiate a new Human (class) in case of a newly detected victim
Instantiate and remove relations as Well-Being and Located (Possibly, NotPossibly, Actually)
EVAL. AUTONOMY: ENVIRONMENTAL COMPLEXITY SNO

16. Novelty level 0: update instances
Automatically update attributes, such as Position
Novelty level 1: re-use classes
Instantiate a new Human (class) in case of a newly detected victim
Instantiate and remove relations as Well-Being and Located (Possibly, NotPossibly, Actually)
Novelty level 2: change in feature not previously relevant
In case Human#i is identified (as George), all information of Human#i is transferred to George
Novelty level 3: change how entities and features are specified
The position of a Door is relative to the Origin of the Room that Snowboy is located
EVAL. AUTONOMY: ENVIRONMENTAL COMPLEXITY SNO

17. Recall the detection of a human in the living
REAL WORLD BOOLEAN?
Robot in
LivingRoom
GoTo next
waypoint
Get
Image
Analyse
Image
Final
waypoint?
no
Terminate
Behavior
yes
When:
Robot in Room X
Human Y in Image
Then
Human Y in Room X

18. The robot is not always succeeding and makes mistakes
REAL WORLD IS PROBABILISTIC

19. PROCESS PROBABILISTIC INFORMATION
Approach & Identify (fail)
Needs rescue
(person_A)?
Question
OR
AtBottom
(person_A, Ladder)?
LayingFaceDown
(person_A)?
AND
person_A
Ladder
OnFloor
(PersonA)
Evidence = NewQuestion
Evidence
Person
Knowledge
(common sense)
Ladder
Floor
Now what?

20. PROCESS PROBABILISTIC INFORMATION
Markov Logic Networks into TypeDB
man phone
Far_a
way
object
object
man
family
father
weight
When:
x isa man
y isa phone
(object: Y, object: X)
Far_away
Then
(x: father) isa family

21. PROCESS PROBABILISTIC INFORMATION
Markov Logic Networks into TypeDB
^
Logic
weight
conclusion
statement
schema
statement
man phone
#objectx
#objecty
Far_a
way
object object
family
father
weight
conclusion
statement
statement
instance
schema

22. PROCESS PROBABILISTIC INFORMATION
Markov Logic Networks into TypeDB
person
#objectx
ladder
#objecty
At_bot
tom
high
low
Well_
being
being
pose
laying
health
Physical
_state
fallen
health
Color:
green ^
weight
statement
SG_la
ying
node
node
#object
statement
statement
statement
statement
SG_fal
len
node
#object
node
conclusion
When
x isa person
y isa ladder
y has color ‘’green’’
(x,y) isa at_bottom
subnetwork(laying)
Then
fallen(x)

23. PROCESS PROBABILISTIC INFORMATION
Markov Logic Networks into TypeDB
prob
prob
prob
prob
prob prob
prob
Pthyon & TypeQL
Extract statement
Transform to MLN
Run PracMLN
Read result
Write to database

24. Autonomous robot that operate in the real world
Have to assume open world
Evidences from the real world are uncertain
Most concepts are unknown
Uncertainties and unknowns cause interventions by operator
We try to reduce such interventions
But we need probabilistic reasoning
Markov Logic Networks
Bayesian Networks
…
SUMMARY:
1
2
4
3
1
2
4
3
5
2020
2021
2022

25. YOUR TIME
THANK YOU FOR