The Augmented Drag-and-Drop


Published on

The development of Augmented Reality (AR) applications is still driven by development at source-code level. Although recent approaches focus to standardize AR functionality, to our knowledge a declarative and model-driven design (MDD) has not been applied for AR development so far. MDD approaches have been successfully applied to model user interfaces for a wide spectrum of modes (such as speech-command interfaces, or remote controls) and media such as HTML, 3D, and smart phone interfaces for instance. In this paper we propose a MDD approach for modeling seamless interaction between Web and AR interfaces. Therefore we implemented a prototype of a web furniture shop that supports arranging furniture in an augmented reality to prove our approach. We then present our approach of modeling a reality spanning Drag-and-Drop interaction between a 2D browser and an AR scene. Finally, we discuss the issues that we were confronted with to support reality spanning interaction like switching interaction modes and coordinate systems and present limitations that we have experienced with using other control modes, like a Wii-Controller or glove-based gesture detection.

Published in: Technology, Sports
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

The Augmented Drag-and-Drop

  1. 1. 1. Juni 2011<br />Model-based Design of Interactions that can bridge RealitiesThe Augmented “Drag-and-Drop”Sebastian Feuerstack, Allan C. M. de Oliveira, Regina B. Araujo<br />Universidade Federal de São CarlosDepartamento de Computacão<br />
  2. 2. The Augmented “Drag-and-Drop”<br />2<br />1. Juni 2011<br />Basic Question<br />How to model Interactions that span Modes and Media ?<br />
  3. 3. The Augmented “Drag-and-Drop<br />3<br />1. Juni 2011<br />Which Media ?<br /><br />
  4. 4. The Augmented “Drag-and-Drop”<br />4<br />1. Juni 2011<br />Which Modes?<br /><br /><br />
  5. 5. The Augmented “Drag-and-Drop<br />5<br />1. Juni 2011<br />How ?<br />Device ; Modality<br />
  6. 6. The Augmented “Drag-and-Drop<br />6<br />1. Juni 2011<br />Why modeling?<br />One specification that<br />is declarative and exact to discuss & execute<br />Is located between Tool and Code<br />can be easily adapted or enhanced to different forms of interaction <br />
  7. 7. The Augmented “Drag-and-Drop<br />7<br />1. Juni 2011<br />Use Case: <br />Augmented Drag and Drop<br />Buying Furniture <br />Shopping Cart<br />Drag and Drop<br />Into my environment<br />
  8. 8. Video: Connecting the Web with AR<br />The Augmented “Drag-and-Drop<br />8<br />1. Juni 2011<br />VIDEO<br />Mouse-based Drag-and-Drop<br />
  9. 9. The Augmented “Drag-and-Drop<br />9<br />1. Juni 2011<br />How to model?<br />Models<br />Interactors(Abstract & Concrete Media, Mode) <br />Static: class diagramm, Behavior: state-chart<br />Mappings Mode-To-Media Synchronization<br />Custom Notation <br />
  10. 10. The Augmented “Drag-and-Drop<br />10<br />1. Juni 2011<br />Abstract Media Model<br />
  11. 11. The Augmented “Drag-and-Drop<br />11<br />1. Juni 2011<br />Abstract Media Model<br />Single Choice<br />Aggregates a set of Entities from that only one can be chosen at a time<br />Examples: Direction (leftorright), Shopping Cart<br />Further Properties: <br />Output tothe User<br />Container -> Aggregation,<br />Discrete,<br />thatare Inputs<br />Contains Single Choice Elements<br />
  12. 12. The Augmented “Drag-and-Drop<br />12<br />1. Juni 2011<br />Abstract Behavior Model<br />
  13. 13. Design of Multimodal Interaction<br />13<br />1. Juni 2011<br />Mode Model<br />(Example: Mouse)<br />
  14. 14. The Augmented “Drag-and-Drop<br />14<br />1. Juni 2011<br />Multimodal Mapping<br />The Drag-and-Drop<br />
  15. 15. The Augmented “Drag-and-Drop<br />15<br />1. Juni 2011<br />Conclusions & Future Work<br />What’s the advantage?<br />Detailed, declarative Modeling of Interactions (Media, Mode)<br />We can combine Modes & Media and bridge Realities<br />What`s still missing is a tool<br />What`s the advantage???<br />Lets see a demo with different Modes<br />
  16. 16. Gesture-based Drag and Drop Control <br />The Augmented “Drag-and-Drop<br />16<br />1. Juni 2011<br />Cam<br />Jessica <br />Pointing<br />Cam<br />Command<br />Monitor<br />
  17. 17. Gesture-based Drag and Drop Control <br />Design of Multimodal Interaction<br />17<br />1. Juni 2011<br />VIDEO<br />Gesture-based Drag-and-Drop<br />
  18. 18. The Augmented “Drag-and-Drop<br />18<br />1. Juni 2011<br />The End …<br />Thank you for your attention!<br />Your questions please …<br />
  19. 19. The Augmented “Drag-and-Drop<br />19<br />1. Juni 2011<br />Abstract Model<br />Reference – Object or Property of an Object to unambigously identify it<br />Examples: Name, Color, Number, Shape, Coordinate<br />Further Properties: <br />Discrete -> Distinguishable<br />Input -> bythe User <br />
  20. 20. The Augmented “Drag-and-Drop<br />20<br />1. Juni 2011<br />Multimodal Mapping<br />(Combining Mode with Media)<br />