SlideShare a Scribd company logo
Engineering Device-spanning, Multimodal
   Web Applications using a Model-based
              Design Approach
                     Sebastian Feuerstack and Ednaldo Pizzolato


Dr.-Ing. Sebastian Feuerstack
Universidade Federal de São Carlos
Departamento de Computacão

  16. Oktober 2012                               Sebastian Feuerstack I Sebastian@Feuerstack.org
Outline

• Introduction / Example
• Related Work
• Models (Interactors, Mappings)
• Evaluation
16. Oktober 2012                   2
Basic Question


     How to model and run
  multimodal web applications ?



16. Oktober 2012                    3
Multimodal Interaction Design
             Augmented Drag-and-Drop


                           • Furniture shop
                             – Shopping cart
                             – Drag and Drop
                             – Into my environment



16. Oktober 2012                                4
Drag and Drop using gestures




 16. Oktober 2012              5
Why modeling?
   • Visual language that
         – is declarative and precise to discuss &
           execute
         – Is located between a tool and source code
         – Supports different forms of interaction and
           flexible prototyping of multimodal interfaces

16. Oktober 2012                                      6
What is State of the Art ?
1.      Task        AUI             CUI             FUI

                                    CUI             FUI


2.       Widget
                          W1              W2
        Alg.

                               A1              D1
         Driver

 16. Oktober 2012                                         7
How to model?
  AUI
              Widget              Mapping             CUI
                                                     CUI
                                                           W   W

                                  Modality
                                          D1
• Models
    – Interactors (Abstract & Concrete Media, Mode)
          • Static: class diagram, Behavior: state-chart
    – Mappings
          • Custom Notation

 16. Oktober 2012                                              8
Abstract UI Model
                   Abstract Interactor Model (AIM)                 Interactor
                                                                   - name, states

                                                          previous
                                                                                             next          AIContext
                                                                                                           - text

                                                                                                    context
                    refers                                              AIO
                                    AIIN                       1
                                                                                                              AIOUT
                                                                                      0..n
                                                                      children

                     AIINDiscrete      AIINContinuous parent                AIOUTDiscrete             AIOUTContinuous
                                       - data, min, max                                               - data, min, max




                     AIReference           AIINRange                     AIContainer
                    - text             - min_data, max_data


                                                                     AISinglePresence                 AISingleChoice
                     AICommand          AIChoiceElement              - active_child




                      AISingleChoiceElement               AIMultiChoiceElement                         AIMultiChoice



16. Oktober 2012                                                                                                         9
Abstract Behavior Model
          AUI:AIO:AIChoiceElement:
          AISingleChoiceElement

                             Presenting

                                      drop             listed
           initialized                                             next||prev||parent
                                                   focus
                                dragging                  defocus /aio=find(act);
            organize                                              aio.focus
                                          drag        focused
           organized

                   present    Selection
                                                                       H
          organize                    unchosen
                                                     [in(focused)] choose
               suspend         unchoose              / aios=find(parent.childs.chosen);
                                                                aios.all.unchoose
                                          chosen
           suspended



16. Oktober 2012                                                                          10
Mode Model
                                         Gesture Interactor
                   IR:IN:HandGesture:PointingSelect

                                                              BothHands
                                          NoLeftHand
                                                                                   NoRightHand
                         left_hand_appeared                        right_hand_appeared
                                          left_hand_disappeared                          right_hand_disappeared

                                         Left Hand
                                                                                   Right Hand

                                          Pointing                                 Command

                                            stopped
                                                                                    initialized
                                                       stop                                            select
                                     move
                                                                                         confirm
                                             moving                        confirmed               selected
                                move
                                                                                         select


                                              left_hand_appeared            right_hand_appeared

                                                              NoHands
                         right_hand_disappeared                                            left_hand_disappeared




16. Oktober 2012                                                                                                11
Mappings
         Application-level Mapping
                h = Gesture.LeftHand.Pointing.stopped                             cio.highlight
                                                             S           R
     cio = CUIControl.find_from_coordinates(h.x,h.y)                               Sound.click




         Synchronisation Mapping
              cio=Interactor.CIO.highlighted
                                                        S           aio.focus
     NOT aio = cio.name:Interactor.AIO.focused




         Paradigm-level Mapping
RightHand.Command.confirmed      S             RightHand.Command.selected         C      dst.drop(item)
item=AIChoiceElement.focused                   dst=AISingleChoice.focused
            src=item.parent       item.drag
                                                                 src.drop(item)   fail

   16. Oktober 2012                                                                               12
Evaluation
1. W3C Multimodal Framework Requirements
    –      Structuration Mechanism
    –      Explicit Control Structures
    –      Extensible Event Definition
    –      Data Modeling
    –      Reusable Components


2. Framework Classification Survey (Dumas 2009)
    –      8 frameworks, two available, one under active
           development
    –      criteria:
         •      finite-state machine processing
         •      no frame-based or symbolical-statistical fusion
         •      high-level programming features
 16. Oktober 2012                                                 13
Evaluation 2
3. Processing Performance Evaluation
   –      Most critical part:
          server-sided processing of continuous user actions
   –      Pointing Movements (by Hand or Mouse)
        •      Automated test, 24 interactors, Core2Duo P8600,
               250 repetitions




16. Oktober 2012                                               14
Conclusions & Future Work


                What’s the advantage?
• Detailed Modeling of
  Multimodal
  Interactions
  including behavior
• Existing tools
• No gap between
  design- and runtime
 16. Oktober 2012                       15
Whats next?

                                               MINT-Composer - A Toolchain for
                                               the Model-based Specification of
                                               Post-WIMP Interactors.
Focus on:
      –   Fusion
      –   Paradigm Design
      –   Formalization
      –   Tools
      –   W3C Standardization


                    http://www.multi-access.de
      All papers, Videos, MINT Framework as Open Source

 16. Oktober 2012           Design of Multimodal Interaction               16
16. Oktober 2012   17

More Related Content

Viewers also liked

Auténticos Relevantes y Diferentes
Auténticos Relevantes y DiferentesAuténticos Relevantes y Diferentes
Auténticos Relevantes y Diferentes
Ruben Gimenez Castro
 
Ong Direct Pac4
Ong Direct Pac4Ong Direct Pac4
Ong Direct Pac4
ONGDirect
 
Rm tarea 5º
Rm tarea   5ºRm tarea   5º
Rm tarea 5º
brisagaela29
 
Flipped elt classroom
Flipped elt classroom Flipped elt classroom
Flipped elt classroom
Ahmet Başal
 
Salud | Manual Del Fabricante - La Justicia de Cristo
Salud | Manual Del Fabricante - La Justicia de CristoSalud | Manual Del Fabricante - La Justicia de Cristo
Salud | Manual Del Fabricante - La Justicia de Cristo
Ministerio Palmoni
 
Why electronic instrument
Why electronic instrument Why electronic instrument
Why electronic instrument
ruikarsachin
 
Global Supply Simulation Presentation - Team 14
Global Supply Simulation Presentation - Team 14Global Supply Simulation Presentation - Team 14
Global Supply Simulation Presentation - Team 14
oenterprises
 
المحاضرة الأولى في العروض التقديمية
المحاضرة الأولى في العروض التقديميةالمحاضرة الأولى في العروض التقديمية
المحاضرة الأولى في العروض التقديمية
uqu
 
Green supply chain management
Green supply chain managementGreen supply chain management
Green supply chain management
Surabhi Kaushal
 
Distribución de frecuencias y representaciones graficas
Distribución de frecuencias y representaciones graficasDistribución de frecuencias y representaciones graficas
Distribución de frecuencias y representaciones graficas
innovalabcun
 
Apple life cycle
Apple life cycleApple life cycle
Apple life cyclealexa_goff
 
Measurements and Instrumentation - Electrical and electronics instruments
Measurements and Instrumentation - Electrical and electronics instrumentsMeasurements and Instrumentation - Electrical and electronics instruments
Measurements and Instrumentation - Electrical and electronics instruments
JCT COLLEGE OF ENGINEERING AND TECHNOLOGY
 
Arte gótico.arquitectura
Arte gótico.arquitecturaArte gótico.arquitectura
Arte gótico.arquitectura
Alfredo Rivero Rodríguez
 
To Not Eat Your Heart Away Final
To Not Eat Your Heart Away FinalTo Not Eat Your Heart Away Final
To Not Eat Your Heart Away Final
Justin Price
 

Viewers also liked (14)

Auténticos Relevantes y Diferentes
Auténticos Relevantes y DiferentesAuténticos Relevantes y Diferentes
Auténticos Relevantes y Diferentes
 
Ong Direct Pac4
Ong Direct Pac4Ong Direct Pac4
Ong Direct Pac4
 
Rm tarea 5º
Rm tarea   5ºRm tarea   5º
Rm tarea 5º
 
Flipped elt classroom
Flipped elt classroom Flipped elt classroom
Flipped elt classroom
 
Salud | Manual Del Fabricante - La Justicia de Cristo
Salud | Manual Del Fabricante - La Justicia de CristoSalud | Manual Del Fabricante - La Justicia de Cristo
Salud | Manual Del Fabricante - La Justicia de Cristo
 
Why electronic instrument
Why electronic instrument Why electronic instrument
Why electronic instrument
 
Global Supply Simulation Presentation - Team 14
Global Supply Simulation Presentation - Team 14Global Supply Simulation Presentation - Team 14
Global Supply Simulation Presentation - Team 14
 
المحاضرة الأولى في العروض التقديمية
المحاضرة الأولى في العروض التقديميةالمحاضرة الأولى في العروض التقديمية
المحاضرة الأولى في العروض التقديمية
 
Green supply chain management
Green supply chain managementGreen supply chain management
Green supply chain management
 
Distribución de frecuencias y representaciones graficas
Distribución de frecuencias y representaciones graficasDistribución de frecuencias y representaciones graficas
Distribución de frecuencias y representaciones graficas
 
Apple life cycle
Apple life cycleApple life cycle
Apple life cycle
 
Measurements and Instrumentation - Electrical and electronics instruments
Measurements and Instrumentation - Electrical and electronics instrumentsMeasurements and Instrumentation - Electrical and electronics instruments
Measurements and Instrumentation - Electrical and electronics instruments
 
Arte gótico.arquitectura
Arte gótico.arquitecturaArte gótico.arquitectura
Arte gótico.arquitectura
 
To Not Eat Your Heart Away Final
To Not Eat Your Heart Away FinalTo Not Eat Your Heart Away Final
To Not Eat Your Heart Away Final
 

More from Sebastian Feuerstack

Revealing Differences in Designer‘s and Users‘Perspectives
Revealing Differences in Designer‘s and Users‘PerspectivesRevealing Differences in Designer‘s and Users‘Perspectives
Revealing Differences in Designer‘s and Users‘Perspectives
Sebastian Feuerstack
 
Comparing the Multimodal Interaction Technique Design of MINT with NiMMiT
Comparing the Multimodal Interaction Technique Design of MINT with NiMMiTComparing the Multimodal Interaction Technique Design of MINT with NiMMiT
Comparing the Multimodal Interaction Technique Design of MINT with NiMMiT
Sebastian Feuerstack
 
MINT-Composer – A Toolchain for the Model-based Specification of Post-WIMP In...
MINT-Composer – A Toolchain for the Model-based Specification of Post-WIMP In...MINT-Composer – A Toolchain for the Model-based Specification of Post-WIMP In...
MINT-Composer – A Toolchain for the Model-based Specification of Post-WIMP In...
Sebastian Feuerstack
 
Model-based Design and Generation of a Gesture-based User Interface Navigatio...
Model-based Design and Generation of a Gesture-based User Interface Navigatio...Model-based Design and Generation of a Gesture-based User Interface Navigatio...
Model-based Design and Generation of a Gesture-based User Interface Navigatio...
Sebastian Feuerstack
 
Modelling of User Interfaces based on State- Charts to Ease the Test and Eval...
Modelling of User Interfaces based on State- Charts to Ease the Test and Eval...Modelling of User Interfaces based on State- Charts to Ease the Test and Eval...
Modelling of User Interfaces based on State- Charts to Ease the Test and Eval...
Sebastian Feuerstack
 
Model-based Design of Interactions that can bridge Realities – The Augmented...
 Model-based Design of Interactions that can bridge Realities – The Augmented... Model-based Design of Interactions that can bridge Realities – The Augmented...
Model-based Design of Interactions that can bridge Realities – The Augmented...
Sebastian Feuerstack
 
Model Based Layouting
Model Based LayoutingModel Based Layouting
Model Based Layouting
Sebastian Feuerstack
 

More from Sebastian Feuerstack (7)

Revealing Differences in Designer‘s and Users‘Perspectives
Revealing Differences in Designer‘s and Users‘PerspectivesRevealing Differences in Designer‘s and Users‘Perspectives
Revealing Differences in Designer‘s and Users‘Perspectives
 
Comparing the Multimodal Interaction Technique Design of MINT with NiMMiT
Comparing the Multimodal Interaction Technique Design of MINT with NiMMiTComparing the Multimodal Interaction Technique Design of MINT with NiMMiT
Comparing the Multimodal Interaction Technique Design of MINT with NiMMiT
 
MINT-Composer – A Toolchain for the Model-based Specification of Post-WIMP In...
MINT-Composer – A Toolchain for the Model-based Specification of Post-WIMP In...MINT-Composer – A Toolchain for the Model-based Specification of Post-WIMP In...
MINT-Composer – A Toolchain for the Model-based Specification of Post-WIMP In...
 
Model-based Design and Generation of a Gesture-based User Interface Navigatio...
Model-based Design and Generation of a Gesture-based User Interface Navigatio...Model-based Design and Generation of a Gesture-based User Interface Navigatio...
Model-based Design and Generation of a Gesture-based User Interface Navigatio...
 
Modelling of User Interfaces based on State- Charts to Ease the Test and Eval...
Modelling of User Interfaces based on State- Charts to Ease the Test and Eval...Modelling of User Interfaces based on State- Charts to Ease the Test and Eval...
Modelling of User Interfaces based on State- Charts to Ease the Test and Eval...
 
Model-based Design of Interactions that can bridge Realities – The Augmented...
 Model-based Design of Interactions that can bridge Realities – The Augmented... Model-based Design of Interactions that can bridge Realities – The Augmented...
Model-based Design of Interactions that can bridge Realities – The Augmented...
 
Model Based Layouting
Model Based LayoutingModel Based Layouting
Model Based Layouting
 

Recently uploaded

Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |
AstuteBusiness
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
Hiroshi SHIBATA
 
GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)
Javier Junquera
 
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge GraphGraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
Neo4j
 
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdfMonitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Tosin Akinosho
 
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and BioinformaticiansBiomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Neo4j
 
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success StoryDriving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Safe Software
 
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development ProvidersYour One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
akankshawande
 
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
Edge AI and Vision Alliance
 
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup SlidesProgramming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
Zilliz
 
Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)
Jakub Marek
 
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing InstancesEnergy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Alpen-Adria-Universität
 
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid ResearchHarnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Neo4j
 
What is an RPA CoE? Session 1 – CoE Vision
What is an RPA CoE?  Session 1 – CoE VisionWhat is an RPA CoE?  Session 1 – CoE Vision
What is an RPA CoE? Session 1 – CoE Vision
DianaGray10
 
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
Jason Yip
 
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
Alex Pruden
 
Northern Engraving | Nameplate Manufacturing Process - 2024
Northern Engraving | Nameplate Manufacturing Process - 2024Northern Engraving | Nameplate Manufacturing Process - 2024
Northern Engraving | Nameplate Manufacturing Process - 2024
Northern Engraving
 
9 CEO's who hit $100m ARR Share Their Top Growth Tactics Nathan Latka, Founde...
9 CEO's who hit $100m ARR Share Their Top Growth Tactics Nathan Latka, Founde...9 CEO's who hit $100m ARR Share Their Top Growth Tactics Nathan Latka, Founde...
9 CEO's who hit $100m ARR Share Their Top Growth Tactics Nathan Latka, Founde...
saastr
 
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green MasterplanJavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
Miro Wengner
 
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity serverDandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
Antonios Katsarakis
 

Recently uploaded (20)

Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
 
GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)
 
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge GraphGraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
 
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdfMonitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
 
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and BioinformaticiansBiomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
 
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success StoryDriving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
 
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development ProvidersYour One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
 
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
 
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup SlidesProgramming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
 
Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)
 
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing InstancesEnergy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
 
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid ResearchHarnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
 
What is an RPA CoE? Session 1 – CoE Vision
What is an RPA CoE?  Session 1 – CoE VisionWhat is an RPA CoE?  Session 1 – CoE Vision
What is an RPA CoE? Session 1 – CoE Vision
 
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
 
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
 
Northern Engraving | Nameplate Manufacturing Process - 2024
Northern Engraving | Nameplate Manufacturing Process - 2024Northern Engraving | Nameplate Manufacturing Process - 2024
Northern Engraving | Nameplate Manufacturing Process - 2024
 
9 CEO's who hit $100m ARR Share Their Top Growth Tactics Nathan Latka, Founde...
9 CEO's who hit $100m ARR Share Their Top Growth Tactics Nathan Latka, Founde...9 CEO's who hit $100m ARR Share Their Top Growth Tactics Nathan Latka, Founde...
9 CEO's who hit $100m ARR Share Their Top Growth Tactics Nathan Latka, Founde...
 
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green MasterplanJavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
 
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity serverDandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
 

Engineering Device-spanning, Multimodal Web Applications using a Model-based Design Approach

  • 1. Engineering Device-spanning, Multimodal Web Applications using a Model-based Design Approach Sebastian Feuerstack and Ednaldo Pizzolato Dr.-Ing. Sebastian Feuerstack Universidade Federal de São Carlos Departamento de Computacão 16. Oktober 2012 Sebastian Feuerstack I Sebastian@Feuerstack.org
  • 2. Outline • Introduction / Example • Related Work • Models (Interactors, Mappings) • Evaluation 16. Oktober 2012 2
  • 3. Basic Question How to model and run multimodal web applications ? 16. Oktober 2012 3
  • 4. Multimodal Interaction Design Augmented Drag-and-Drop • Furniture shop – Shopping cart – Drag and Drop – Into my environment 16. Oktober 2012 4
  • 5. Drag and Drop using gestures 16. Oktober 2012 5
  • 6. Why modeling? • Visual language that – is declarative and precise to discuss & execute – Is located between a tool and source code – Supports different forms of interaction and flexible prototyping of multimodal interfaces 16. Oktober 2012 6
  • 7. What is State of the Art ? 1. Task AUI CUI FUI CUI FUI 2. Widget W1 W2 Alg. A1 D1 Driver 16. Oktober 2012 7
  • 8. How to model? AUI Widget Mapping CUI CUI W W Modality D1 • Models – Interactors (Abstract & Concrete Media, Mode) • Static: class diagram, Behavior: state-chart – Mappings • Custom Notation 16. Oktober 2012 8
  • 9. Abstract UI Model Abstract Interactor Model (AIM) Interactor - name, states previous next AIContext - text context refers AIO AIIN 1 AIOUT 0..n children AIINDiscrete AIINContinuous parent AIOUTDiscrete AIOUTContinuous - data, min, max - data, min, max AIReference AIINRange AIContainer - text - min_data, max_data AISinglePresence AISingleChoice AICommand AIChoiceElement - active_child AISingleChoiceElement AIMultiChoiceElement AIMultiChoice 16. Oktober 2012 9
  • 10. Abstract Behavior Model AUI:AIO:AIChoiceElement: AISingleChoiceElement Presenting drop listed initialized next||prev||parent focus dragging defocus /aio=find(act); organize aio.focus drag focused organized present Selection H organize unchosen [in(focused)] choose suspend unchoose / aios=find(parent.childs.chosen); aios.all.unchoose chosen suspended 16. Oktober 2012 10
  • 11. Mode Model Gesture Interactor IR:IN:HandGesture:PointingSelect BothHands NoLeftHand NoRightHand left_hand_appeared right_hand_appeared left_hand_disappeared right_hand_disappeared Left Hand Right Hand Pointing Command stopped initialized stop select move confirm moving confirmed selected move select left_hand_appeared right_hand_appeared NoHands right_hand_disappeared left_hand_disappeared 16. Oktober 2012 11
  • 12. Mappings Application-level Mapping h = Gesture.LeftHand.Pointing.stopped cio.highlight S R cio = CUIControl.find_from_coordinates(h.x,h.y) Sound.click Synchronisation Mapping cio=Interactor.CIO.highlighted S aio.focus NOT aio = cio.name:Interactor.AIO.focused Paradigm-level Mapping RightHand.Command.confirmed S RightHand.Command.selected C dst.drop(item) item=AIChoiceElement.focused dst=AISingleChoice.focused src=item.parent item.drag src.drop(item) fail 16. Oktober 2012 12
  • 13. Evaluation 1. W3C Multimodal Framework Requirements – Structuration Mechanism – Explicit Control Structures – Extensible Event Definition – Data Modeling – Reusable Components 2. Framework Classification Survey (Dumas 2009) – 8 frameworks, two available, one under active development – criteria: • finite-state machine processing • no frame-based or symbolical-statistical fusion • high-level programming features 16. Oktober 2012 13
  • 14. Evaluation 2 3. Processing Performance Evaluation – Most critical part: server-sided processing of continuous user actions – Pointing Movements (by Hand or Mouse) • Automated test, 24 interactors, Core2Duo P8600, 250 repetitions 16. Oktober 2012 14
  • 15. Conclusions & Future Work What’s the advantage? • Detailed Modeling of Multimodal Interactions including behavior • Existing tools • No gap between design- and runtime 16. Oktober 2012 15
  • 16. Whats next? MINT-Composer - A Toolchain for the Model-based Specification of Post-WIMP Interactors. Focus on: – Fusion – Paradigm Design – Formalization – Tools – W3C Standardization http://www.multi-access.de All papers, Videos, MINT Framework as Open Source 16. Oktober 2012 Design of Multimodal Interaction 16

Editor's Notes

  1. Transformationalapproach Not targetedtomultmodalinterfaces, Models areinspectable but transformationsarecomplexAssembly Approach: Black-boxedcomponents, extensibilityproblem