Fusion in Multimodal Interactive Systems: An HMM-Based Algorithm for User-Induced Adaptation

on

  • 474 views

Presented at the fourth ACM SIGCHI Symposium on Engineering Interactive Computing Systems, IT University of Copenhagen, Denmark, June 25–28, 2012.

Presented at the fourth ACM SIGCHI Symposium on Engineering Interactive Computing Systems, IT University of Copenhagen, Denmark, June 25–28, 2012.

Statistics

Views

Total Views
474
Views on SlideShare
474
Embed Views
0

Actions

Likes
0
Downloads
7
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Fusion in Multimodal Interactive Systems: An HMM-Based Algorithm for User-Induced Adaptation Fusion in Multimodal Interactive Systems: An HMM-Based Algorithm for User-Induced Adaptation Presentation Transcript

  • Fusion in Multimodal Interactive Systems:An HMM-Based Algorithm for User-InducedAdaptationBruno Dumas1, Beat Signer1 and Denis Lalanne21 WISE ResearchLab, VrijeUniversiteit Brussel, Belgium2 DIVA Research Group, University of Fribourg, Switzerland 2 December 2005
  • Outline • Fusion of multimodal input • Multimodal interaction: from design to implementation • Algorithms for fusion of multimodal input • A Hidden Markov Model-based fusion algorithm • Instantiation of the algorithm • Evaluation of the algorithm • Qualitative test • Quantitative evaluation • Performance assessment • ConclusionJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 2
  • Fusion of Multimodal Input  Challenges in multimodal interaction engineering: continous and probabilistic inputs, real-time…June 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 3
  • Challenging/AmbiguousFusion Cases • “Put-that-there”  Complementarity case • “Play next track”  Different meanings based on order and contextJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 4
  • CARE Propertiesat the Design Level  Describe how modalitiescanbecombined  Complementarity  All modalities are necessary  Assignment  Absence of choice  Redundancy  Same expressive power betweenmodalities  Equivalence  AnymodalityissufficientJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 5
  • HephaisTK Multimodal FrameworkJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 6
  • Description of Multimodal Dialogues • Syntactic-level description • For more details: check SMUIML languageJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 7
  • Fusion Algorithms • Frame-basedalgorithms&Unification- basedalgorithms • Symbolicapproaches • Both are a de facto standard • Symbolic-statisticalalgorithmsJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 8
  • Symbolic-StatisticalApproaches  Symbolic data  Interpretation of low-level data  Statisticalprocessing  Machine learningalgorithmsJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 9
  • Our HMM-Based Fusion Algorithm  Whyhidden Markov models?  Very good modelling of time-relatedevents  Takesadvantage of input data comingfromprobabilistic input modalities  On-the-fly adaptation of the model - e.g. based on user feedbackJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 10
  • Put-That-ThereExampleJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 11
  • Put-That-ThereExampleJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 12
  • Put-That-ThereExampleJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 13
  • Put-That-ThereExampleJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 14
  • Put-That-ThereExampleJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 15
  • Put-That-ThereExampleJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 16
  • Instantiation of the AlgorithmJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 17
  • Evaluation of the Algorithm  Evaluation on three levels  Qualitative “gut check” test  Benchmarks on real-world examples  Performance assessmentJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 18
  • Evaluation: Qualitative Test • Three expert userswereasked to assess the behaviour of the algorithm  frames-based and HMM-based fusion  5 minutes per condition + interview • Coherentbehaviourbetween the algorithms • No noticeabledifference in responsivenessJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 19
  • Evaluation: BenchmarkJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 20
  • Evaluation: BenchmarkJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 21
  • Benchmark Results (1) • CARE properties: equivalence and redundancy testsJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 22
  • Benchmark Results (2) • Sequential and non-sequentialcomplementarity testsJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 23
  • Benchmark Results (3) • “Play next track” example Meaning frames HMM-basedJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 24
  • Evaluation: Performance • Per fusion algorithm: • 5 runs of 40 input pieces • 5 x 20 expected fusion results • Frame-based: 18.2 ms • Standard deviation: 12.7 ms • HMM-based: 16.6 ms • Standard deviation: 11.6 msJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 25
  • Conclusion • A new symbolic-statistical multimodal fusion algorithmbased on Hidden Markov Models • Integratedinto the HephaisTK framework • Superior recognition ratescompared to existingalgorithms • Handling of ambiguous cases • Processing of real-time user feedback http://wise.vub.ac.be/bruno-dumasJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 26