2 December 2005
Mobile Multimodal Interaction:
An Investigation and Implementation of
Context-dependent Adaptation
Promote...
Beat Signer - Department of Computer Science - bsigner@vub.ac.be 2February 17, 2011
Outline
 Introduction
 Background st...
Beat Signer - Department of Computer Science - bsigner@vub.ac.be 3February 17, 2011
Introduction
 Mobile Interaction deal...
Beat Signer - Department of Computer Science - bsigner@vub.ac.be 4February 17, 2011
 A systematic study
 Context-aware a...
Beat Signer - Department of Computer Science - bsigner@vub.ac.be 5February 17, 2011
Background Studies
M-G1 MU-G1M-G1 MU-G...
Beat Signer - Department of Computer Science - bsigner@vub.ac.be 6February 17, 2011
Systematic Study
“Analysis of multimod...
Beat Signer - Department of Computer Science - bsigner@vub.ac.be 7February 17, 2011
Systematic Study
Selection Criteria
Sy...
Beat Signer - Department of Computer Science - bsigner@vub.ac.be 8February 17, 2011
Systematic Study
Multimodal Interactio...
Beat Signer - Department of Computer Science - bsigner@vub.ac.be 9February 17, 2011
Systematic Study
Analysis
Combination ...
Beat Signer - Department of Computer Science - bsigner@vub.ac.be 10February 17, 2011
Systematic Study
Combination of Modal...
Beat Signer - Department of Computer Science - bsigner@vub.ac.be 11February 17, 2011
Systematic Study
System-induced Adapt...
Beat Signer - Department of Computer Science - bsigner@vub.ac.be 12February 17, 2011
Systematic Study
Guidelines for Autom...
Beat Signer - Department of Computer Science - bsigner@vub.ac.be 13February 17, 2011
Multimodal Adaptive Agenda
Objectives...
Beat Signer - Department of Computer Science - bsigner@vub.ac.be 14February 17, 2011
Easiness
Of
Interaction
Multimodal Ad...
Beat Signer - Department of Computer Science - bsigner@vub.ac.be 15February 17, 2011
Multimodal Adaptive Agenda
Guidelines...
Beat Signer - Department of Computer Science - bsigner@vub.ac.be 16February 17, 2011
Multimodal Adaptive Agenda
Guidelines...
Beat Signer - Department of Computer Science - bsigner@vub.ac.be 17February 17, 2011
Multimodal Adaptive Agenda
Architectu...
Beat Signer - Department of Computer Science - bsigner@vub.ac.be 18February 17, 2011
Multimodal Adaptive Agenda
User Inter...
Beat Signer - Department of Computer Science - bsigner@vub.ac.be 19February 17, 2011
Multimodal Adaptive Agenda
Demo
Beat Signer - Department of Computer Science - bsigner@vub.ac.be 20February 17, 2011
Conclusions
Future Work
 Extend the ...
Beat Signer - Department of Computer Science - bsigner@vub.ac.be 21February 17, 2011
Conclusions
Summary
 Review of the s...
Upcoming SlideShare
Loading in …5
×

Mobile Multimodal Interaction: An Investigation and Implementation of Context-dependent Adaptation

325 views
274 views

Published on

Masters Thesis Presentation

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
325
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • The mai objective ot the investigation we conducted is:
  • Therefore, we reviewed research articles that met a specific selection critiera..

    However to cover the whole spectrum of adaptation , we reviewed also
    areticles related to user induced adaptation..
  • Aims to provide an insight about the influ ence of the context

    System induced adaptation of input modalities, to analyse to what and how these systems adapts....
  • We summarise all the observed combinations of modalities in this recapitulative table..

    For each individual modality the combination with other modalities is indicated either with an E for equivalent
    combinations or a C if the combination is complementary.

    The most frequently used combination of modalities are highlighted with yellow.
  • In regards to the system-induced adaptation analysis
    We reviewed to what and how exactly these systems adapt, analysing their monitoring entity, adaptation policies and mechanism component

    In regards to the monitoring entity component, we analyse which specific environment variable was used
    to call for an adaptation. Mostly the change of semantic location and noise levels variatons were used to trigger an adaptation call

    Furthermore, we analysed the observed type of adaptation policy and adaptive mechanism techniques
    Mostly in the reviewed articles the approach that was followed is a simple rule based approach..

    This table provides a summary of the articles corresponding to each core component.
  • based on the investigation we conducted, During the analysis phase we noticed that the field of system-induced adaptation has barely been addressed.
    Therefore we proposed a set of guidelines targeted
    to support the design process of automatic adaptation of input channels.

    We organised the guidelines into three main phases:

    The first one comprise the Context and modality suitability analysis,
    In this phase the influence of environment factors
    should be evaluated. Therefore, two activities are important: conduct a context
    analysis and define suitable modalities for each semantic location
    --------------------------------------------------------------------------------------------
    Multimodal tasks definition: After defining the modalities that the application will support, this phase encompasses the design of the multimodal input channels. It includes two guidelines, namely the selection of multimodal tasks and the definition of equivalent input modalities.
    --------------------------------------------------------------------------------------------
    The last phase refers to the Adaptation Design.
    Based on the information obtained from the previous stages, it should be
    specified the design of the adaptation process.

    Taken into account important aspects as:

    the definition of adaptation triggers and monitoring entities,
    the definition of the adaptation policy and mechanism
    context and modalities status feedback.
  • Our propo
  • The resulting interface is depticted in this images and the multimodal tasks are highligted with red...
    Moreover, the applications provides constant feedback about the current environment status and
    The available modalities...
  • Mobile Multimodal Interaction: An Investigation and Implementation of Context-dependent Adaptation

    1. 1. 2 December 2005 Mobile Multimodal Interaction: An Investigation and Implementation of Context-dependent Adaptation Promoter: Prof. Dr. Beat Signer Advisor: Dr. Bruno Dumas María Solorzano
    2. 2. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 2February 17, 2011 Outline  Introduction  Background studies  Systematic study and analysis  Multimodal Adaptive Agenda  Conclusions
    3. 3. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 3February 17, 2011 Introduction  Mobile Interaction deals with multiple limitations  Device hardware and size  Context  Multimodal Interaction is one way to address these limitations  Adaptation is paramount in mobile and multimodal systems
    4. 4. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 4February 17, 2011  A systematic study  Context-aware adaptive multimodal interfaces  A proof-of-concept application based on the study findings Objectives Problem Definition  Current studies do not provide an unified overview of the field.  No systematic description of the design stages Introduction
    5. 5. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 5February 17, 2011 Background Studies M-G1 MU-G1M-G1 MU-G1 MG-2.6 MG-2.7 MU-G2 MG-2.5 MG-2.9 MU-G6 M-G2.2 MU-G5 CA-G4 CA-G5CA-G5 CA-G1 A-G1 A-G2 A-G3  Mobile Interaction  Multimodal Interaction  Context-awareness  Adaptive Interfaces
    6. 6. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 6February 17, 2011 Systematic Study “Analysis of multimodal mobile input channels with a special focus on adaptation triggered or influenced by environmental factors”
    7. 7. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 7February 17, 2011 Systematic Study Selection Criteria System Induced Adaptation User Induced Adaptation (6 articles) (8 articles)
    8. 8. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 8February 17, 2011 Systematic Study Multimodal Interaction  Modalities  Interaction techniques  Interaction sensors  Output Influence  CARE properties Mobile Interaction and Context-awareness  Device (Taxonomy for mobile Terminals - Schiefer and Decker’s )  Environmental properties (Cameleon Reference Framework) Adaptive Interfaces  Monitoring entity  Adaptation policy  Adaptation mechanism Classification Schema
    9. 9. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 9February 17, 2011 Systematic Study Analysis Combination of Modalities Context Influence System-Induced Adaptation * *
    10. 10. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 10February 17, 2011 Systematic Study Combination of Modalities E =equivalent and C= complementary * The numbers refer to the bibliographic references
    11. 11. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 11February 17, 2011 Systematic Study System-induced Adaptation * The numbers refer to the bibliographic references
    12. 12. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 12February 17, 2011 Systematic Study Guidelines for Automatic Adaptation of Input Channels Context and Modality Suitability Analysis 1. Conduct a context analysis 2. Define suitable modalities for each semantic location Multimodal Tasks Definition 1. Select tasks that will support a multimodal behaviour 2. Define equivalent modalities Adaptation Design 1. Define adaptation triggers and monitoring entities 2. Define adaptation policy 3. Define adaption mechanism 4. Provide feedback
    13. 13. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 13February 17, 2011 Multimodal Adaptive Agenda Objectives Mobile Calendar Application + multimodal capabilities + adaptive capabilities  Test proposed design guidelines  Explore less frequently used modalities  Explore fusion engine and adaptation mechanism integration
    14. 14. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 14February 17, 2011 Easiness Of Interaction Multimodal Adaptive Agenda Guidelines Category 1: Context and Modality Suitability Analysis Context Analysis
    15. 15. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 15February 17, 2011 Multimodal Adaptive Agenda Guidelines Category 2: Multimodal Tasks Definition Equivalent modalities Multimodal Tasks
    16. 16. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 16February 17, 2011 Multimodal Adaptive Agenda Guidelines Category 3: Adaptation Design  Adaptation triggers and Monitoring entities  semantic location (indoors, outdoors) GPS  noise level (low, medium, high) Microphone  user activity (walking, still) Accelerometer
    17. 17. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 17February 17, 2011 Multimodal Adaptive Agenda Architecture
    18. 18. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 18February 17, 2011 Multimodal Adaptive Agenda User Interface Context and available modalities status
    19. 19. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 19February 17, 2011 Multimodal Adaptive Agenda Demo
    20. 20. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 20February 17, 2011 Conclusions Future Work  Extend the systematic study  User context  Device context  Develop a rapid prototyping tool  Conduct a field study with users  But, the application’s functionality should be enhanced before
    21. 21. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 21February 17, 2011 Conclusions Summary  Review of the state-of-the-art  Selection, classification and analysis of relevant research projects  Proposal of a set of design guidelines  Design and development of a proof of concept based on the guidelines

    ×