Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Kilix: Heterogeneous Modeling of Gesture-Based 3D Applications

448 views

Published on

Model-driven software engineering is a well-known approach for developing software. It reduces complexity, facilitates maintenance and allows for the simulation, verification, validation and even execution of software models. In this article, we show how model simulation and model execution can be leveraged in the context of human-computer interaction (HCI). We claim that in this application domain, it is beneficial to use heterogeneous models, combining different models of computation for different components of the system. We report on a case study that we have carried out to develop an executable model of a gesture-based application for manipulating 3D objects, using the Kinect sensor as input device, and the OGRE graphical engine as an output device for real-time rendering. This application is fully specified as an heterogeneous model with the ModHel’X modeling environment, allowing the simulation and execution of the model to explore the most appropriate way of implementing the application.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Kilix: Heterogeneous Modeling of Gesture-Based 3D Applications

  1. 1. Kilix: Heterogeneous Modeling of Gesture-Based 3D Applications R. Deshayes and T. Mens Institut COMPLEXYS, University of Mons, Belgium C. Jacquet, C. Hardebolle, F. Boulanger Supelec E3S – Computer Science Dept., France
  2. 2. Goals of our research •  Reduce complexity of developing HCI applicationsMoDELS 2012 – MPM workshop, Innsbruck •  By using visual modeling instead of programming •  Assess the usability of heterogeneous modeling for this purpose •  Evaluate the strengths and shortcomings of ModHel’X, a heterogeneous modeling environment •  Explore and improve its notions of semantic adaptation wwwdi.supelec.fr/software/ModHelX wwwdi.supelec.fr/software/ModHelX/Kilix 2
  3. 3. Case study •  Gestural interaction with a graphical 3DMoDELS 2012 – MPM workshop, Innsbruck application •  Using the Kinect controller to interact with virtual books using hands only 3
  4. 4. Proposed architecture •  Client-server architectureMoDELS 2012 – MPM workshop, Innsbruck •  Low coupling between I/O devices and user interaction models •  Combining different models of computation (MoC) •  choose the most appropriate formalism for the task at hand •  discrete events (DE) •  synchronous data flow (SDF) •  timed finite state machines (TFSM) 4
  5. 5. Proposed architecture •  Client-server architectureMoDELS 2012 – MPM workshop, Innsbruck •  Low coupling between I/O devices and user interaction models •  Combining different models of computation (MoC) •  choose the most appropriate formalism for the task at hand •  discrete events (DE) •  synchronous data flow (SDF) •  timed finite state machines (TFSM) 5
  6. 6. Heterogenous modeling •  Hierarchical architectureMoDELS 2012 – MPM workshop, Innsbruck •  Top-level model contains 4 blocks •  MoC is discrete events (DE) •  communication through timestamped events containing data 6
  7. 7. Semantic adaptation •  Interface blocks adapt the semantics betweenMoDELS 2012 – MPM workshop, Innsbruck outer and inner models using different models of computation •  Adaptation can be made to •  Data (which may be represented differently) •  Time (e.g. different time units, different time scales, continuous vs discrete time) •  Control (trigger observations of the internal model at instants requested by the internal MoC) 7
  8. 8. Sensing block •  Receives data from Kinect and converts it intoMoDELS 2012 – MPM workshop, Innsbruck hand gesture events •  MoC = synchronous data flow (SDF) •  Processes a chain of sampled signals received from Kinect at a fixed rate •  Semantic adapter generates DE events when non- null SDF tokens are produced DE 8
  9. 9. Application Logic block •  Interprets and converts hand gestures intoMoDELS 2012 – MPM workshop, Innsbruck meaningful actions for 3D object manipulation •  MoC = timed finite state machines (TFSM) •  DE/TFSM adapter converts between DE events and symbols for the state machine DE 9
  10. 10. Virtual Scene block •  Represents graphical 3D objects (e.g., book)MoDELS 2012 – MPM workshop, Innsbruck that interpret the actions as object-specific behaviour (e.g. opening or closing the book) •  MoC = TFSM DE 10
  11. 11. Heterogenous modeling •  General overview revisitedMoDELS 2012 – MPM workshop, Innsbruck 11
  12. 12. Discussion •  Heterogeneous modeling is useful for HCIMoDELS 2012 – MPM workshop, Innsbruck applications •  Semantic adaptation can be used •  To adapt between models of computation •  To map application actions (e.g., swipe) to object behaviors (e.g., open or close) •  To use the same component differently in different applications •  Leads to less coupling and higher component reusability •  Dynamic modeling is difficult to achieve •  e.g. variable number of users and books at runtime 12
  13. 13. Future work •  Compare strengths and weaknesses ofMoDELS 2012 – MPM workshop, Innsbruck homogenous and heterogenous modeling •  Based on common case study •  Expressed using statecharts only •  Expressed using high-level Petri nets •  Joint work with Ph. Palanque, Toulouse (PetShop tool) •  Expressed using ModHel’X •  ModHel’X improvements •  Performance issues •  Add support for visual editing of models •  Support domain-specific languages to match the application domain better (work in progress) •  Extend existing MoC (TFSM++) 13
  14. 14. References •  For homogeneous modeling MoDELS 2012 – MPM workshop, Innsbruck •  R. Deshayes and T. Mens. Statechart modelling of interactive gesture-based applications. In ComDeisMoto satellite event of INTERACT 2011. •  D. Navarre, et al. ICOs: A model-based user interface description technique dedicated to interactive systems addressing usability, reliability and scalability. ACM Trans. Comput.-Hum. Interact., 16(4), 2009. •  For heterogeneous modeling •  J. Eker et al. Taming heterogeneity - the Ptolemy approach. Proc. IEEE, 91(1):127–144, 2003. •  C. Hardebolle, F. Boulanger. Exploring multi-paradigm modeling techniques. SIMULATION: Trans. Society for Modeling and Simulation International, 85(11/12):688–708, 2009. •  F. Boulanger et al. Semantic Adaptation for Models of Computation. ACSD 2011: 153-162 •  B. Baudry et al. Bridging the Chasm between Executable Metamodeling and Models of Computation. SLE 2012 14

×