Gesture recognition with Jnect, EMF-IncQuery, Esper and GEF3D

2,682 views

Published on

This presentation is about a tech demo for EclipseCon Europe 2012. We connect Microsoft Kinect with Eclipse technologies using:
- the Jnect plugin for processing Kinect Data
- the Eclipse Modeling Framework as the underlying live model representation format
- EMF-IncQuery for realtime pattern matching to recognize static gestures
- the Esper complex event processor to recognize gesture sequences in the event stream generated by IncQuery.

More details at http://viatra.inf.mit.bme.hu/incquery/demos/jnect

Published in: Technology
  • Be the first to comment

Gesture recognition with Jnect, EMF-IncQuery, Esper and GEF3D

  1. 1. “STANDING  QUERIES”   Real2me  gesture  recogni2on  with  Eclipse  technologies   István  DÁVID,  István  RÁTH hDp://viatra.inf.mit.bme.hu/incquery/demos/jnect   Budapest  University  of  Technology  and  Economics Fault  Tolerant  Systems  Research  GroupBudapest  University  of  Technology  and  EconomicsDepartment  of  Measurement  and  Informa<on  Systems
  2. 2. Overview User  moving  in  front  of  the  sensor PaDern  match   EMF   setX(…) deltas no2fica2on EMF-­‐ Esper   Live  EMF   IncQuery   complex   model paDern   event  Kinect matcher processorsensor data Recognized gesture  sequences
  3. 3. Overview User  moving  in  front  of  the  sensor PaDern  match   EMF   setX(…) deltas no2fica2on EMF-­‐ Esper   Live  EMF   IncQuery   complex   model paDern   event  Kinect matcher processorsensor data Full  model   update  at  ~20   Recognized FPS gesture  sequences
  4. 4. Overview User  moving  in  front  of  the  sensor PaDern  match   EMF   setX(…) deltas no2fica2on EMF-­‐ Esper   Live  EMF   IncQuery   complex   model PaBern  Y()  { paDern   event  Kinect matcher processorsensor data Full  model   update  at  ~20   Recognized FPS gesture  sequences }
  5. 5. Overview User  moving  in  front  of  the  sensor PaDern  match   EMF   setX(…) deltas no2fica2on EMF-­‐ Esper   Live  EMF   IncQuery   complex   model paDern   event  Kinect matcher processorsensor data Full  model   update  at  ~20   Recognized FPS gesture  sequences
  6. 6. Overview Event  PaBern  YMCA User  moving  in  front  of  the  sensor SELECT  *  FROM  paDern[  every((Y)  -­‐>              (M)  -­‐>   EMF   PaDern  match   deltas setX(…) no2fica2on        (C)  -­‐>     EMF-­‐ Esper              (A)   Live  EMF   IncQuery   complex   WHERE  2mer:within(10  sec))] model paDern   event  Kinect matcher processorsensor data Full  model   update  at  ~20   Recognized FPS gesture  sequences
  7. 7. Overview User  moving  in  front  of  the  sensor PaDern  match   EMF   setX(…) deltas no2fica2on EMF-­‐ Esper   Live  EMF   IncQuery   complex   model paDern   event  Kinect matcher processorsensor data Full  model   update  at  ~20   Recognized FPS gesture  sequences
  8. 8. Overview User  moving  in  front  of  the  sensor PaDern  match   EMF   setX(…) deltas no2fica2on EMF-­‐ Esper   Live  EMF   IncQuery   complex   model paDern   event  Kinect matcher processorsensor data Full  model   update  at  ~20   Recognized FPS gesture  sequences
  9. 9. Overview User  moving  in  front  of  the  sensor CEP  output PaDern  match   EMF   setX(…) no2fica2on PaDern  matcher  output deltas EMF-­‐ Esper   Live  EMF   IncQuery   complex   model paDern   event  Kinect matcher processorsensor data GEF3D-­‐based   visualiza2on Full  model   update  at  ~20   Recognized FPS gesture  sequences
  10. 10. EMF-­‐IncQuery§ High  performance  incremental  paDern  matcher o Provides  on-­‐the-­‐fly  model  query  evalua2on  performance  for   complex  queries  over  very  large  EMF  instance  models o Typical  use  cases:  on-­‐the-­‐fly  valida2on,  view  maintenance/ abstract  databinding,  derived  edges/aDributes,  …§ Coming  in  the  following  weeks  to  Eclipse  Modeling§ Version  0.6.7  available  from o The  Eclipse  Marketplace o hDp://viatra.inf.mit.bme.hu/incquery§ Featured  at  EclipseCon  Europe  2012: Fast,  Faster  and  Super-­‐fast  Model  Queries (Ed  Willink  and  István  Ráth)

×