UCAmI Presentation Dec.2013, Guanacaste, Costa Rica
Understanding Movement and Interaction: an Ontology
for Kinect-based 3D Depth Sensors
Natalia Díaz Rodríguez1, Robin Wikström1, Johan Lilius1, Manuel Pegalajar Cuéllar2 and Miguel Delgado
1Turku Centre for Computer Science (TUCS), Dept. of IT, Åbo Akademi University, (Finland)
2Dept. Of Computer Science and Artificial Intelligence, University of Granada (Spain)
7th International Conference on Ubiquitous Computing & Ambient Intelligence (UCAmI 2013) 5.12.13
§ A crucial & challenging task in AmI:
Human behaviour modelling and recognition
§ Video-based monitoring techniques. Applications:
– Technology for detection of in-home activities
– Elderly care (fall detection)
– Exercise monitoring (rehabilitation)
⇒ Can be inaccurate, compromise privacy, become
⇒ No common scheme for skeleton data
⇒ Need for a device independent 3D-Depth Sensors
§ Exercise applications based on 3D depth cameras multimodal
features (gesture + spoken commands)
– Virtual Social Gyms
– Eyes-Free Yoga
– Kinect@Home (crowdsourcing 3D environment datasets)
– Kinect Fusion (real-time 3D reconstruction and interaction)
– Kinect based robots map indoor environments to 3D models
– Ontology-based annotation of images
& semantic maps
§ BML (Behaviour Markup Language)
Proposal: modelling movement and
§ Combine data-driven computer vision with knowledge-driven
semantics to obtain high level & more meaninful info.
=> Annotate semantically physical movement &
interaction to enable automatic knowledge reasoning.
– E.g. Provide feedback when doing exercise to patient +
physiotherapist (quality & frequency).
§ Gathering sensor info, allows semantic queries for further
– E.g. long term evolution of back posture.
Audio (speech recognition engines)
Tracking Modes (Default/Seated, -2 out of 6
§ Gestures (grip, release, push, scroll)
§ Interaction Controls (video, images, text)
§ Object interaction (Kinect Fusion API).
– User-Object Interactions (grab, release,
touch, click etc.)
– Hand –interactive, gripping, pressing- and
Arm state –primary.
§ Body Movement (rotate, bend, extend, elevate):
clockwise, direction or body side.
– E.g. RotateWristClockwise, ElevateFootFront,
§ Skeleton tracking (bone joint rotations + bone orientations)
Excerpt of classes, data & object
Examples of use
Example 1: Defining basic movement (Stand, BendDown,TwistRight, MoveObject, etc).
Example 2: When defining, e.g. SitStandExercise workout, the N of series done in time
as well as the exercise quality can be measured and compared with predefined medical
guidelines, to give feedback.
Examples of use
Example 3: Historic analysis can be provided to monitor posture quality in time. E.g.
having back less straight than 1 year ago can be notified to correct/prevent on
Example 4: An office worker can be notified when he is not having straight back
and neck or when he has been sitting for too long.
§ Protégé, OWL 2
§ Skeleton tracking: Kinect for Windows SDK C#.
– Kinect NUI, Kinect Interaction, Fusion and Audio
§ NeOn Ontology engineering methodology (reuse
ontology resources, requirements specification,
development of required scenarios and dynamic
– Spatial Relations Ontology (contains, disjoint,
§ Validation with physiotherapists exercises (ongoing)
§ Combining computer vision with semantic
models can enhance
– common understanding
– recognition accuracy
– trust and data provenance
§ Tackling feedback
§ Gesture Definition Markup Language (GDML).
§ Large rule dataset scalability + performance
(reasoning, querying/updating/ subscribing)
§ Fuzzy rules to tackle imprecision, vagueness &
– Ease looseness in the model and facilitate user
interaction (linguistic labels for natural language
§ Integration into A) new M3 distributed
architecture -low power distributed processingB) Philips PHL (Personal Health Labs) platform
Thank you for your attention!
Natalia Díaz Rodríguez
Embedded Systems Lab. Department of Information Technologies
Åbo Akademi University, Turku, Finland
TUCS (Turku Centre for Computer Science)
Department of Computer Science and
University of Granada, Spain