Understanding Movement and Interaction: an Ontology
for Kinect-based 3D Depth Sensors

Natalia Díaz Rodríguez1, Robin Wiks...
Introduction
§  A crucial & challenging task in AmI:
Human behaviour modelling and recognition
§  Video-based monitoring...
Introduction
§  WHY Semantic Technologies & Ontologies?
–  Formulate relationships between concepts
–  Independent knowle...
Related Work
§  Exercise applications based on 3D depth cameras multimodal
features (gesture + spoken commands)
–  Virtua...
Proposal: modelling movement and
interaction
Aim:
§  Combine data-driven computer vision with knowledge-driven
semantics ...
Ontology features
Kinect Sensor
3D Volume
Audio (speech recognition engines)
Tracking Modes (Default/Seated, -2 out of 6
u...
Ontology features
§  Object interaction (Kinect Fusion API).
–  User-Object Interactions (grab, release,
touch, click etc...
Ontology features
§  Skeleton tracking (bone joint rotations + bone orientations)

8
Exercises & Workouts

9
10
Excerpt of classes, data & object
properties

11
Examples of use
Example 1: Defining basic movement (Stand, BendDown,TwistRight, MoveObject, etc).

Example 2: When definin...
Examples of use
Example 3: Historic analysis can be provided to monitor posture quality in time. E.g.
having back less str...
Implementation
§  Protégé, OWL 2
§  Skeleton tracking: Kinect for Windows SDK C#.
–  Kinect NUI, Kinect Interaction, Fus...
Conclusions
§  OWL 2 ontology (ALC DL expressivity):
164 classes, 53 object properties, 58 data properties,
93 individual...
Conclusions
§  Validation with physiotherapists exercises (ongoing)
§  Combining computer vision with semantic
models ca...
Future Directions
§  Tackling feedback
§  Gesture Definition Markup Language (GDML).
§  Large rule dataset scalability ...
Future Directions
§  Integration into A) new M3 distributed
architecture -low power distributed processingB) Philips PHL ...
Future Directions
§  On-going: Orthopedic rehabilitation exercises.
§  Hip, Shoulder, Knee post-surgery
§  Cardiac
§  ...
20
Thank you for your attention!
Natalia Díaz Rodríguez
ndiaz@abo.fi
Embedded Systems Lab. Department of Information Technolo...
Upcoming SlideShare
Loading in …5
×

UCAmI Presentation Dec.2013, Guanacaste, Costa Rica

396 views
290 views

Published on

Understanding Movement and Interaction: An Ontology for Kinect-Based 3D Depth Sensors
Natalia Díaz Rodríguez, Robin Wikström, Johan Lilius, Manuel Pegalajar Cuéllar, Miguel Delgado Calvo-Flores

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
396
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

UCAmI Presentation Dec.2013, Guanacaste, Costa Rica

  1. 1. Understanding Movement and Interaction: an Ontology for Kinect-based 3D Depth Sensors Natalia Díaz Rodríguez1, Robin Wikström1, Johan Lilius1, Manuel Pegalajar Cuéllar2 and Miguel Delgado Calvo Flores2 1Turku Centre for Computer Science (TUCS), Dept. of IT, Åbo Akademi University, (Finland) 2Dept. Of Computer Science and Artificial Intelligence, University of Granada (Spain) 7th International Conference on Ubiquitous Computing & Ambient Intelligence (UCAmI 2013) 5.12.13 1
  2. 2. Introduction §  A crucial & challenging task in AmI: Human behaviour modelling and recognition §  Video-based monitoring techniques. Applications: –  Technology for detection of in-home activities (posture, gestures) –  Elderly care (fall detection) –  Exercise monitoring (rehabilitation) ⇒ Can be inaccurate, compromise privacy, become intrusive ⇒ No common scheme for skeleton data ⇒ Need for a device independent 3D-Depth Sensors Ontology 2
  3. 3. Introduction §  WHY Semantic Technologies & Ontologies? –  Formulate relationships between concepts –  Independent knowledge sharing minimizing redundancy –  Extensible, device/platform independent –  Context-awareness: modelling & reasoning for automatic inference, e.g.: §  Other useful context-aware info: stress, heart rate, sleep quality, mood, etc. 3
  4. 4. Related Work §  Exercise applications based on 3D depth cameras multimodal features (gesture + spoken commands) –  Virtual Social Gyms –  Eyes-Free Yoga –  Kinect@Home (crowdsourcing 3D environment datasets) –  Kinect Fusion (real-time 3D reconstruction and interaction) –  Kinect based robots map indoor environments to 3D models –  Ontology-based annotation of images & semantic maps §  BML (Behaviour Markup Language) 4
  5. 5. Proposal: modelling movement and interaction Aim: §  Combine data-driven computer vision with knowledge-driven semantics to obtain high level & more meaninful info. => Annotate semantically physical movement & interaction to enable automatic knowledge reasoning. –  E.g. Provide feedback when doing exercise to patient + physiotherapist (quality & frequency). §  Gathering sensor info, allows semantic queries for further knowledge reasoning –  E.g. long term evolution of back posture. 5
  6. 6. Ontology features Kinect Sensor 3D Volume Audio (speech recognition engines) Tracking Modes (Default/Seated, -2 out of 6 users-) §  Gestures (grip, release, push, scroll) §  §  §  §  §  Interaction Controls (video, images, text) 6
  7. 7. Ontology features §  Object interaction (Kinect Fusion API). –  User-Object Interactions (grab, release, touch, click etc.) –  Hand –interactive, gripping, pressing- and Arm state –primary. §  Body Movement (rotate, bend, extend, elevate): clockwise, direction or body side. –  E.g. RotateWristClockwise, ElevateFootFront, LeftBodyPart 7
  8. 8. Ontology features §  Skeleton tracking (bone joint rotations + bone orientations) 8
  9. 9. Exercises & Workouts 9
  10. 10. 10
  11. 11. Excerpt of classes, data & object properties 11
  12. 12. Examples of use Example 1: Defining basic movement (Stand, BendDown,TwistRight, MoveObject, etc). Example 2: When defining, e.g. SitStandExercise workout, the N of series done in time as well as the exercise quality can be measured and compared with predefined medical guidelines, to give feedback. 12
  13. 13. Examples of use Example 3: Historic analysis can be provided to monitor posture quality in time. E.g. having back less straight than 1 year ago can be notified to correct/prevent on time. Example 4: An office worker can be notified when he is not having straight back and neck or when he has been sitting for too long. 13
  14. 14. Implementation §  Protégé, OWL 2 §  Skeleton tracking: Kinect for Windows SDK C#. –  Kinect NUI, Kinect Interaction, Fusion and Audio modules. §  NeOn Ontology engineering methodology (reuse ontology resources, requirements specification, development of required scenarios and dynamic ontology evolution). –  Spatial Relations Ontology (contains, disjoint, equals, overlaps) 14
  15. 15. Conclusions §  OWL 2 ontology (ALC DL expressivity): 164 classes, 53 object properties, 58 data properties, 93 individuals. §  Exercise & Workout sub-ontology registers performance quality evolution §  Abstract atomic gestures => incremental, fine, and coarse grained activity recognition. 15
  16. 16. Conclusions §  Validation with physiotherapists exercises (ongoing) §  Combining computer vision with semantic models can enhance –  context-awareness –  common understanding –  recognition accuracy –  trust and data provenance Kinect Ontology: http://users.abo.fi/rowikstr/KinectOntology/ 16
  17. 17. Future Directions §  Tackling feedback §  Gesture Definition Markup Language (GDML). §  Large rule dataset scalability + performance (reasoning, querying/updating/ subscribing) §  Fuzzy rules to tackle imprecision, vagueness & uncertainty. –  Ease looseness in the model and facilitate user interaction (linguistic labels for natural language customization). 17
  18. 18. Future Directions §  Integration into A) new M3 distributed architecture -low power distributed processingB) Philips PHL (Personal Health Labs) platform Atom board Future: ARM 18
  19. 19. Future Directions §  On-going: Orthopedic rehabilitation exercises. §  Hip, Shoulder, Knee post-surgery §  Cardiac §  Record + train new users’ patterns §  Hip Extension §  Hip Abduction §  Sit-Stand §  Knee Extension §  Knee Abduction §  Local, distributed, low-power M3 RDF-store 19
  20. 20. 20
  21. 21. Thank you for your attention! Natalia Díaz Rodríguez ndiaz@abo.fi Embedded Systems Lab. Department of Information Technologies Åbo Akademi University, Turku, Finland TUCS (Turku Centre for Computer Science) Department of Computer Science and Artificial Intelligence University of Granada, Spain 21

×