Statechart modeling of interactive gesture-based applications

2,713 views

Published on

Developing intuitive interactive applications that are easy to maintain by developers is quite challenging, due to the complexity and the many technical aspects involved in such applications. In this article, we tackle the problem in two complementary ways. First, we propose a gestural interface to improve the user experience when interacting with applications that require the manipulation of 3D graphical scenes. Second, we reduce the complexity of developing such applications by modeling their executable behaviour using statecharts. We validate our approach by creating a modular and extensible Java framework for the development of in- teractive gesture-based applications. We developed a proof- of-concept application using this framework, that allows the user to construct and manipulate 3D scenes in OpenGL by using hand gestures only. These hand gestures are captured by the Kinect sensor, and translated into events and actions that are interpreted and executed by communicating statecharts that model the main behaviour of the interactive application.

Published in: Design, Technology, Education
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,713
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
8
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Statechart modeling of interactive gesture-based applications

  1. 1. Statechart modeling of interactive gesture-based applications Statechart modeling of interactive gesture-based applications Romuald Deshayes Tom Mens Software Engineering Lab 2010-2011 Deshayes Romuald – UMONS 1 / 20
  2. 2. Statechart modeling of interactive gesture-based applicationsTable des matières 1 Problem Statement 2 Proof-of-concept application 3 Modeling interactive behaviour 4 Statechart models 5 The application framework 6 Conclusion and Future Work Deshayes Romuald – UMONS 2 / 20
  3. 3. Statechart modeling of interactive gesture-based applications Problem StatementOutline : Problem Statement Problem Statement Deshayes Romuald – UMONS 3 / 20
  4. 4. Statechart modeling of interactive gesture-based applications Problem StatementProblem Statement Challenge Develop intuitive interactive applications that are easy to maintain Why is it a challenge ? Complex behaviour Nondeterministic user actions Same events lead to different actions Contributions Gestural user interface for 3D objects manipulation Modeling executable behaviour using a visual formalism Validation with an application framework Deshayes Romuald – UMONS 4 / 20
  5. 5. Statechart modeling of interactive gesture-based applications Proof-of-concept applicationOutline : Proof-of-concept application Proof-of-concept application Deshayes Romuald – UMONS 5 / 20
  6. 6. Statechart modeling of interactive gesture-based applications Proof-of-concept applicationProof-of-concept application 3D visual drawing tool Uses gestures to create and manipulate 3D objects OpenGL graphical library Microsoft Kinect + NITE Deshayes Romuald – UMONS 6 / 20
  7. 7. Statechart modeling of interactive gesture-based applications Proof-of-concept applicationSmall video : www.youtube.com/watch?v=lVcqzWTnpMY Deshayes Romuald – UMONS 7 / 20
  8. 8. Statechart modeling of interactive gesture-based applications Modeling interactive behaviourOutline : Modeling interactive behaviour Modeling interactive behaviour Deshayes Romuald – UMONS 8 / 20
  9. 9. Statechart modeling of interactive gesture-based applications Modeling interactive behaviourModeling interactive behaviour Context Highly reactive event-driven systems Gesture-based interaction Proposed solutions Visual modeling language Statecharts Petri nets Amenable to formal analysis Easier to evolve Reduced complexity Deshayes Romuald – UMONS 9 / 20
  10. 10. Statechart modeling of interactive gesture-based applications Statechart modelsOutline : Statechart models Statechart models Deshayes Romuald – UMONS 10 / 20
  11. 11. Statechart modeling of interactive gesture-based applications Statechart modelsStatechart models - Hand Deshayes Romuald – UMONS 11 / 20
  12. 12. Statechart modeling of interactive gesture-based applications Statechart modelsStatechart models - Component Deshayes Romuald – UMONS 12 / 20
  13. 13. Statechart modeling of interactive gesture-based applications Statechart modelsStatechart models - Scene Deshayes Romuald – UMONS 13 / 20
  14. 14. Statechart modeling of interactive gesture-based applications The application frameworkOutline : The application framework The application framework Deshayes Romuald – UMONS 14 / 20
  15. 15. Statechart modeling of interactive gesture-based applications The application frameworkThe application framework Framework for developing interactive applications Executable behaviour specified using statecharts Client-server architecture Deshayes Romuald – UMONS 15 / 20
  16. 16. Statechart modeling of interactive gesture-based applications The application frameworkThe application framework II Features Generic Abstract classes Observer design pattern Communication between statecharts Uses Java SwingStates library Deshayes Romuald – UMONS 16 / 20
  17. 17. Statechart modeling of interactive gesture-based applications The application frameworkThe application framework III <<library>> Framework SwingStates Server extends Thread -serverSocket : DatagramSocket StateMachine Controller left 0..1 1..* -machine : StateMachine Pointer <<Interface>> right -open : boolean +addComponent(c : Component) 0..1 BodyListener extends -machine : StateMachine +addPointer(p : Pointer) EventListener +addScene(s : Scene) Scene Main Application +handMovedX(e : BodyEvent) 1..* +handMovedY(e : BodyEvent) -machine : StateMachine MainScene +handMovedZ(e : BodyEvent) 1 +handClosed(e : BodyEvent) +handOpened(e : BodyEvent) selectedModel 1..* * 0..1 Component Model BodyAdapter 3DModel -boundingBox : Rectangle -selected : boolean +addAsListenerOf(s : Server) -machine : StateMachine 1..* models BodyAdapters OpenGLWidgets Hand GLButton ExtendableMenu -machine : StateMachine +changeActiveTexture() : void +extend() : void +drawButton(gl : GL) : void +retract() : void +startCounter() : void +drawMenu(gl : GL) : void SpawningMenu +addButton(button : GLButton) : void ImageMenu +addText(text : String) : void ButtonMenu +drawMenu(gl : GL) : void +drawMenu(gl : GL) : void +addButton(button : GLBUtton) : void +onModel() : void +spawn() : void +drawMenu(gl : GL) : void +reduce() : void Deshayes Romuald – UMONS 17 / 20
  18. 18. Statechart modeling of interactive gesture-based applications Conclusion and Future WorkOutline : Conclusion and Future Work Conclusion and Future Work Deshayes Romuald – UMONS 18 / 20
  19. 19. Statechart modeling of interactive gesture-based applications Conclusion and Future WorkConclusion and Future Work Statecharts Appropriate and scalable formalism for modeling the interactive gesture-based behaviour ? Improvements Externalise/decouple statechart representation from framework Compare statecharts with Petri nets Deshayes Romuald – UMONS 19 / 20
  20. 20. Statechart modeling of interactive gesture-based applications Conclusion and Future WorkFuture Work II Apply our approach to other HCI applications Other types of natural interfaces Compare/combine with other UIML Controlled experiments, carry out user studies, empirical studies Deshayes Romuald – UMONS 20 / 20

×