SlideShare a Scribd company logo
1 of 17
Download to read offline
Model-based Measurement of Interaction in Mobile
               Multimodal Environments

                                Pedro Mateo – Stefan Hillmann


                                    aMMI - October 2012




Pedro Mateo – Stefan Hillmann       Model-based Interaction Measurement   aMMI - October 2012   1 / 17
Agenda



Model-based Measurement of Interaction in Mobile Multimodal
Environments:
 1 Research Questions

             motivation, questions and proposed solutions
 2   Research and Development
             MMI metrics, MMI model and Android implementation
 3   Conclusion
             advantages, disadvantages and research directions




 Pedro Mateo – Stefan Hillmann   Model-based Interaction Measurement   aMMI - October 2012   2 / 17
I
Research Questions
Motivation


We analyze interaction in different
systems...




...but independently for each system

 Pedro Mateo – Stefan Hillmann   Model-based Interaction Measurement   aMMI - October 2012   4 / 17
Motivation



And now, for each system...
     different quantification/evaluation metrics
     different representation formats
     different interaction analysis tools
     multimodal communication is not analyzed as a whole




 Pedro Mateo – Stefan Hillmann   Model-based Interaction Measurement   aMMI - October 2012   5 / 17
Research questions and proposed solutions



Question 1:
  MMI assessed on a common level for different modalities?
          ⇒ Use common and generic metrics for all modalities
Question 2:
  Compare interactions observed in different MMI contexts?
          ⇒ Structure metrics in a common representation (model)
Question 3:
  Analyze MMI dinamically, step-by-step, and using real-time data?
          ⇒ Use a run-time model




 Pedro Mateo – Stefan Hillmann   Model-based Interaction Measurement   aMMI - October 2012   6 / 17
II
Research and Development
Defining multimodal metrics




 Pedro Mateo – Stefan Hillmann   Model-based Interaction Measurement   aMMI - October 2012   8 / 17
Modeling multimodal interaction in EMF

                                             Metacommunication




                          User Metacommunication            System Metacommunication


                                                                                        Speech Output
          Speech Input
                                                                                        GUI Output
              GUI Input          User Turn                        System Turn
                                                                                        Gesture Output
         Gesture Input
                                                                                        Cooperativity

                           Modality                  Turn                Dialog



               Synchrony              Modality Change                     Task


                                      Appropriateness




 Pedro Mateo – Stefan Hillmann          Model-based Interaction Measurement            aMMI - October 2012   9 / 17
Modeling multimodal interaction in EMF




 Pedro Mateo – Stefan Hillmann   Model-based Interaction Measurement   aMMI - October 2012   10 / 17
Android implementation




 Pedro Mateo – Stefan Hillmann   Model-based Interaction Measurement   aMMI - October 2012   11 / 17
Android implementation


how does it work?




 Pedro Mateo – Stefan Hillmann   Model-based Interaction Measurement   aMMI - October 2012   12 / 17
III
Conclusion
Good stuff...

     common metrics + uniform representation
     ⇒ unified criteria to quantify MMI in different scenarios
     comparison is allowed, model transformation is possible
     different modalities are on a level ⇒ single thread
     dynamic representation of MMI ⇒ run-time applications
     reuse of metrics and tools is improved




 Pedro Mateo – Stefan Hillmann   Model-based Interaction Measurement   aMMI - October 2012   14 / 17
Not so good stuff...




     some data can’t be automatically collected
     criteria to record parameters in different modalities?
     lack of plasticity
     ⇒ user and context information to enrich the analysis process
     the HOW of interaction is not described




 Pedro Mateo – Stefan Hillmann   Model-based Interaction Measurement   aMMI - October 2012   15 / 17
And now what?



     Integrate this solution in real projects
             Use the framework for real analysis

     Make the model context-aware, more “mobile”
     Explore new possibilities of the model
             E.g. behavioral biometric authentication



Project info: http://www.catedrasaes.org/wiki/MIM




 Pedro Mateo – Stefan Hillmann   Model-based Interaction Measurement   aMMI - October 2012   16 / 17
Acknowledgements




 Pedro Mateo – Stefan Hillmann   Model-based Interaction Measurement   aMMI - October 2012   17 / 17

More Related Content

Similar to Model-based Measurement of Interaction in Mobile Multimodal Environments

A Context-aware Model for the Analysis of User Interaction and QoE in Mobile ...
A Context-aware Model for the Analysis of User Interaction and QoE in Mobile ...A Context-aware Model for the Analysis of User Interaction and QoE in Mobile ...
A Context-aware Model for the Analysis of User Interaction and QoE in Mobile ...Pedro Luis Mateo Navarro
 
Effort Estimation Development Model for Web-based Mobile Application Using Fu...
Effort Estimation Development Model for Web-based Mobile Application Using Fu...Effort Estimation Development Model for Web-based Mobile Application Using Fu...
Effort Estimation Development Model for Web-based Mobile Application Using Fu...TELKOMNIKA JOURNAL
 
From eGov 2.0 to eGov 3.0: The Research Agenda
From eGov 2.0 to eGov 3.0: The Research AgendaFrom eGov 2.0 to eGov 3.0: The Research Agenda
From eGov 2.0 to eGov 3.0: The Research Agendasamossummit
 
Towards Method Engineering of Model-Driven User Interface Development
Towards Method Engineering ofModel-Driven User Interface Development Towards Method Engineering ofModel-Driven User Interface Development
Towards Method Engineering of Model-Driven User Interface Development Jean Vanderdonckt
 
BPMN and Design Patterns for Engineering Social BPM Solutions
BPMN and Design Patterns for Engineering Social BPM SolutionsBPMN and Design Patterns for Engineering Social BPM Solutions
BPMN and Design Patterns for Engineering Social BPM SolutionsMarco Brambilla
 
Understanding and maintaining your market to maximise revenue generation opp...
Understanding and maintaining your market to maximise revenue generation opp...Understanding and maintaining your market to maximise revenue generation opp...
Understanding and maintaining your market to maximise revenue generation opp...Miguel Simões
 
Social Media Technologies, part A of 2
Social Media Technologies, part A of 2Social Media Technologies, part A of 2
Social Media Technologies, part A of 2Paolo Nesi
 
Multimedia Semantics: Metadata, Analysis and Interaction
Multimedia Semantics:Metadata, Analysis and InteractionMultimedia Semantics:Metadata, Analysis and Interaction
Multimedia Semantics: Metadata, Analysis and InteractionRaphael Troncy
 
Personalized Retweet Prediction in Twitter
Personalized Retweet Prediction in TwitterPersonalized Retweet Prediction in Twitter
Personalized Retweet Prediction in TwitterLiangjie Hong
 
Modelling the Media Logic of Software Systems
Modelling the Media Logic of Software SystemsModelling the Media Logic of Software Systems
Modelling the Media Logic of Software SystemsJan Schmidt
 
The Future of Ai How Multimodal Models Are Leading the Way.pdf
The Future of Ai How Multimodal Models Are Leading the Way.pdfThe Future of Ai How Multimodal Models Are Leading the Way.pdf
The Future of Ai How Multimodal Models Are Leading the Way.pdfStephenAmell4
 
AI for UI: How AI technology may support human-technology interaction by Roop...
AI for UI: How AI technology may support human-technology interaction by Roop...AI for UI: How AI technology may support human-technology interaction by Roop...
AI for UI: How AI technology may support human-technology interaction by Roop...Mindtrek
 
Using Model-Driven Engineering for Decision Support Systems Modelling, Implem...
Using Model-Driven Engineering for Decision Support Systems Modelling, Implem...Using Model-Driven Engineering for Decision Support Systems Modelling, Implem...
Using Model-Driven Engineering for Decision Support Systems Modelling, Implem...CSCJournals
 

Similar to Model-based Measurement of Interaction in Mobile Multimodal Environments (20)

A Context-aware Model for the Analysis of User Interaction and QoE in Mobile ...
A Context-aware Model for the Analysis of User Interaction and QoE in Mobile ...A Context-aware Model for the Analysis of User Interaction and QoE in Mobile ...
A Context-aware Model for the Analysis of User Interaction and QoE in Mobile ...
 
Effort Estimation Development Model for Web-based Mobile Application Using Fu...
Effort Estimation Development Model for Web-based Mobile Application Using Fu...Effort Estimation Development Model for Web-based Mobile Application Using Fu...
Effort Estimation Development Model for Web-based Mobile Application Using Fu...
 
Mnemonic Modeling
Mnemonic ModelingMnemonic Modeling
Mnemonic Modeling
 
From eGov 2.0 to eGov 3.0: The Research Agenda
From eGov 2.0 to eGov 3.0: The Research AgendaFrom eGov 2.0 to eGov 3.0: The Research Agenda
From eGov 2.0 to eGov 3.0: The Research Agenda
 
Smef2009
Smef2009Smef2009
Smef2009
 
Towards Method Engineering of Model-Driven User Interface Development
Towards Method Engineering ofModel-Driven User Interface Development Towards Method Engineering ofModel-Driven User Interface Development
Towards Method Engineering of Model-Driven User Interface Development
 
BPMN and Design Patterns for Engineering Social BPM Solutions
BPMN and Design Patterns for Engineering Social BPM SolutionsBPMN and Design Patterns for Engineering Social BPM Solutions
BPMN and Design Patterns for Engineering Social BPM Solutions
 
Social BPM
Social BPMSocial BPM
Social BPM
 
Understanding and maintaining your market to maximise revenue generation opp...
Understanding and maintaining your market to maximise revenue generation opp...Understanding and maintaining your market to maximise revenue generation opp...
Understanding and maintaining your market to maximise revenue generation opp...
 
Social Media Technologies, part A of 2
Social Media Technologies, part A of 2Social Media Technologies, part A of 2
Social Media Technologies, part A of 2
 
WP1 1st Review
WP1 1st ReviewWP1 1st Review
WP1 1st Review
 
Multimedia Semantics: Metadata, Analysis and Interaction
Multimedia Semantics:Metadata, Analysis and InteractionMultimedia Semantics:Metadata, Analysis and Interaction
Multimedia Semantics: Metadata, Analysis and Interaction
 
Handson 1 (5/6)
Handson 1 (5/6)Handson 1 (5/6)
Handson 1 (5/6)
 
Personalized Retweet Prediction in Twitter
Personalized Retweet Prediction in TwitterPersonalized Retweet Prediction in Twitter
Personalized Retweet Prediction in Twitter
 
Modelling the Media Logic of Software Systems
Modelling the Media Logic of Software SystemsModelling the Media Logic of Software Systems
Modelling the Media Logic of Software Systems
 
The Future of Ai How Multimodal Models Are Leading the Way.pdf
The Future of Ai How Multimodal Models Are Leading the Way.pdfThe Future of Ai How Multimodal Models Are Leading the Way.pdf
The Future of Ai How Multimodal Models Are Leading the Way.pdf
 
AI for UI: How AI technology may support human-technology interaction by Roop...
AI for UI: How AI technology may support human-technology interaction by Roop...AI for UI: How AI technology may support human-technology interaction by Roop...
AI for UI: How AI technology may support human-technology interaction by Roop...
 
Hse mda bpmn_210410
Hse mda bpmn_210410Hse mda bpmn_210410
Hse mda bpmn_210410
 
Thesis
ThesisThesis
Thesis
 
Using Model-Driven Engineering for Decision Support Systems Modelling, Implem...
Using Model-Driven Engineering for Decision Support Systems Modelling, Implem...Using Model-Driven Engineering for Decision Support Systems Modelling, Implem...
Using Model-Driven Engineering for Decision Support Systems Modelling, Implem...
 

Recently uploaded

Deutsche EuroShop | Company Presentation | 03/24
Deutsche EuroShop | Company Presentation | 03/24Deutsche EuroShop | Company Presentation | 03/24
Deutsche EuroShop | Company Presentation | 03/24Deutsche EuroShop AG
 
Osisko Gold Royalties Ltd - Corporate Presentation, March 2024
Osisko Gold Royalties Ltd - Corporate Presentation, March 2024Osisko Gold Royalties Ltd - Corporate Presentation, March 2024
Osisko Gold Royalties Ltd - Corporate Presentation, March 2024Osisko Gold Royalties Ltd
 
Advancing Discovery New Craigmont Project’s High-grade Copper Potential
Advancing Discovery New Craigmont Project’s High-grade Copper PotentialAdvancing Discovery New Craigmont Project’s High-grade Copper Potential
Advancing Discovery New Craigmont Project’s High-grade Copper Potentialnicola_mining
 
Terna Capital Markets Day 2024 Presentation
Terna Capital Markets Day 2024 PresentationTerna Capital Markets Day 2024 Presentation
Terna Capital Markets Day 2024 PresentationTerna SpA
 
Nicola Mining Inc. Corporate Presentation March 2024
Nicola Mining Inc. Corporate Presentation March 2024Nicola Mining Inc. Corporate Presentation March 2024
Nicola Mining Inc. Corporate Presentation March 2024nicola_mining
 
WWW.SOFTSOP.COM PITCH DECK PRESENTATION.PDF
WWW.SOFTSOP.COM PITCH DECK PRESENTATION.PDFWWW.SOFTSOP.COM PITCH DECK PRESENTATION.PDF
WWW.SOFTSOP.COM PITCH DECK PRESENTATION.PDFMichael Claudio
 
Deutsche EuroShop | Preliminary Results FY 2023
Deutsche EuroShop | Preliminary Results FY 2023Deutsche EuroShop | Preliminary Results FY 2023
Deutsche EuroShop | Preliminary Results FY 2023Deutsche EuroShop AG
 

Recently uploaded (7)

Deutsche EuroShop | Company Presentation | 03/24
Deutsche EuroShop | Company Presentation | 03/24Deutsche EuroShop | Company Presentation | 03/24
Deutsche EuroShop | Company Presentation | 03/24
 
Osisko Gold Royalties Ltd - Corporate Presentation, March 2024
Osisko Gold Royalties Ltd - Corporate Presentation, March 2024Osisko Gold Royalties Ltd - Corporate Presentation, March 2024
Osisko Gold Royalties Ltd - Corporate Presentation, March 2024
 
Advancing Discovery New Craigmont Project’s High-grade Copper Potential
Advancing Discovery New Craigmont Project’s High-grade Copper PotentialAdvancing Discovery New Craigmont Project’s High-grade Copper Potential
Advancing Discovery New Craigmont Project’s High-grade Copper Potential
 
Terna Capital Markets Day 2024 Presentation
Terna Capital Markets Day 2024 PresentationTerna Capital Markets Day 2024 Presentation
Terna Capital Markets Day 2024 Presentation
 
Nicola Mining Inc. Corporate Presentation March 2024
Nicola Mining Inc. Corporate Presentation March 2024Nicola Mining Inc. Corporate Presentation March 2024
Nicola Mining Inc. Corporate Presentation March 2024
 
WWW.SOFTSOP.COM PITCH DECK PRESENTATION.PDF
WWW.SOFTSOP.COM PITCH DECK PRESENTATION.PDFWWW.SOFTSOP.COM PITCH DECK PRESENTATION.PDF
WWW.SOFTSOP.COM PITCH DECK PRESENTATION.PDF
 
Deutsche EuroShop | Preliminary Results FY 2023
Deutsche EuroShop | Preliminary Results FY 2023Deutsche EuroShop | Preliminary Results FY 2023
Deutsche EuroShop | Preliminary Results FY 2023
 

Model-based Measurement of Interaction in Mobile Multimodal Environments

  • 1. Model-based Measurement of Interaction in Mobile Multimodal Environments Pedro Mateo – Stefan Hillmann aMMI - October 2012 Pedro Mateo – Stefan Hillmann Model-based Interaction Measurement aMMI - October 2012 1 / 17
  • 2. Agenda Model-based Measurement of Interaction in Mobile Multimodal Environments: 1 Research Questions motivation, questions and proposed solutions 2 Research and Development MMI metrics, MMI model and Android implementation 3 Conclusion advantages, disadvantages and research directions Pedro Mateo – Stefan Hillmann Model-based Interaction Measurement aMMI - October 2012 2 / 17
  • 4. Motivation We analyze interaction in different systems... ...but independently for each system Pedro Mateo – Stefan Hillmann Model-based Interaction Measurement aMMI - October 2012 4 / 17
  • 5. Motivation And now, for each system... different quantification/evaluation metrics different representation formats different interaction analysis tools multimodal communication is not analyzed as a whole Pedro Mateo – Stefan Hillmann Model-based Interaction Measurement aMMI - October 2012 5 / 17
  • 6. Research questions and proposed solutions Question 1: MMI assessed on a common level for different modalities? ⇒ Use common and generic metrics for all modalities Question 2: Compare interactions observed in different MMI contexts? ⇒ Structure metrics in a common representation (model) Question 3: Analyze MMI dinamically, step-by-step, and using real-time data? ⇒ Use a run-time model Pedro Mateo – Stefan Hillmann Model-based Interaction Measurement aMMI - October 2012 6 / 17
  • 8. Defining multimodal metrics Pedro Mateo – Stefan Hillmann Model-based Interaction Measurement aMMI - October 2012 8 / 17
  • 9. Modeling multimodal interaction in EMF Metacommunication User Metacommunication System Metacommunication Speech Output Speech Input GUI Output GUI Input User Turn System Turn Gesture Output Gesture Input Cooperativity Modality Turn Dialog Synchrony Modality Change Task Appropriateness Pedro Mateo – Stefan Hillmann Model-based Interaction Measurement aMMI - October 2012 9 / 17
  • 10. Modeling multimodal interaction in EMF Pedro Mateo – Stefan Hillmann Model-based Interaction Measurement aMMI - October 2012 10 / 17
  • 11. Android implementation Pedro Mateo – Stefan Hillmann Model-based Interaction Measurement aMMI - October 2012 11 / 17
  • 12. Android implementation how does it work? Pedro Mateo – Stefan Hillmann Model-based Interaction Measurement aMMI - October 2012 12 / 17
  • 14. Good stuff... common metrics + uniform representation ⇒ unified criteria to quantify MMI in different scenarios comparison is allowed, model transformation is possible different modalities are on a level ⇒ single thread dynamic representation of MMI ⇒ run-time applications reuse of metrics and tools is improved Pedro Mateo – Stefan Hillmann Model-based Interaction Measurement aMMI - October 2012 14 / 17
  • 15. Not so good stuff... some data can’t be automatically collected criteria to record parameters in different modalities? lack of plasticity ⇒ user and context information to enrich the analysis process the HOW of interaction is not described Pedro Mateo – Stefan Hillmann Model-based Interaction Measurement aMMI - October 2012 15 / 17
  • 16. And now what? Integrate this solution in real projects Use the framework for real analysis Make the model context-aware, more “mobile” Explore new possibilities of the model E.g. behavioral biometric authentication Project info: http://www.catedrasaes.org/wiki/MIM Pedro Mateo – Stefan Hillmann Model-based Interaction Measurement aMMI - October 2012 16 / 17
  • 17. Acknowledgements Pedro Mateo – Stefan Hillmann Model-based Interaction Measurement aMMI - October 2012 17 / 17