Fusion in Multimodal Interactive Systems: An HMM-Based Algorithm for User-Induced Adaptation

686 views

Published on

Presented at the fourth ACM SIGCHI Symposium on Engineering Interactive Computing Systems, IT University of Copenhagen, Denmark, June 25–28, 2012.

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
686
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
14
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Fusion in Multimodal Interactive Systems: An HMM-Based Algorithm for User-Induced Adaptation

  1. 1. Fusion in Multimodal Interactive Systems:An HMM-Based Algorithm for User-InducedAdaptationBruno Dumas1, Beat Signer1 and Denis Lalanne21 WISE ResearchLab, VrijeUniversiteit Brussel, Belgium2 DIVA Research Group, University of Fribourg, Switzerland 2 December 2005
  2. 2. Outline • Fusion of multimodal input • Multimodal interaction: from design to implementation • Algorithms for fusion of multimodal input • A Hidden Markov Model-based fusion algorithm • Instantiation of the algorithm • Evaluation of the algorithm • Qualitative test • Quantitative evaluation • Performance assessment • ConclusionJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 2
  3. 3. Fusion of Multimodal Input  Challenges in multimodal interaction engineering: continous and probabilistic inputs, real-time…June 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 3
  4. 4. Challenging/AmbiguousFusion Cases • “Put-that-there”  Complementarity case • “Play next track”  Different meanings based on order and contextJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 4
  5. 5. CARE Propertiesat the Design Level  Describe how modalitiescanbecombined  Complementarity  All modalities are necessary  Assignment  Absence of choice  Redundancy  Same expressive power betweenmodalities  Equivalence  AnymodalityissufficientJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 5
  6. 6. HephaisTK Multimodal FrameworkJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 6
  7. 7. Description of Multimodal Dialogues • Syntactic-level description • For more details: check SMUIML languageJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 7
  8. 8. Fusion Algorithms • Frame-basedalgorithms&Unification- basedalgorithms • Symbolicapproaches • Both are a de facto standard • Symbolic-statisticalalgorithmsJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 8
  9. 9. Symbolic-StatisticalApproaches  Symbolic data  Interpretation of low-level data  Statisticalprocessing  Machine learningalgorithmsJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 9
  10. 10. Our HMM-Based Fusion Algorithm  Whyhidden Markov models?  Very good modelling of time-relatedevents  Takesadvantage of input data comingfromprobabilistic input modalities  On-the-fly adaptation of the model - e.g. based on user feedbackJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 10
  11. 11. Put-That-ThereExampleJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 11
  12. 12. Put-That-ThereExampleJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 12
  13. 13. Put-That-ThereExampleJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 13
  14. 14. Put-That-ThereExampleJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 14
  15. 15. Put-That-ThereExampleJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 15
  16. 16. Put-That-ThereExampleJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 16
  17. 17. Instantiation of the AlgorithmJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 17
  18. 18. Evaluation of the Algorithm  Evaluation on three levels  Qualitative “gut check” test  Benchmarks on real-world examples  Performance assessmentJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 18
  19. 19. Evaluation: Qualitative Test • Three expert userswereasked to assess the behaviour of the algorithm  frames-based and HMM-based fusion  5 minutes per condition + interview • Coherentbehaviourbetween the algorithms • No noticeabledifference in responsivenessJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 19
  20. 20. Evaluation: BenchmarkJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 20
  21. 21. Evaluation: BenchmarkJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 21
  22. 22. Benchmark Results (1) • CARE properties: equivalence and redundancy testsJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 22
  23. 23. Benchmark Results (2) • Sequential and non-sequentialcomplementarity testsJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 23
  24. 24. Benchmark Results (3) • “Play next track” example Meaning frames HMM-basedJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 24
  25. 25. Evaluation: Performance • Per fusion algorithm: • 5 runs of 40 input pieces • 5 x 20 expected fusion results • Frame-based: 18.2 ms • Standard deviation: 12.7 ms • HMM-based: 16.6 ms • Standard deviation: 11.6 msJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 25
  26. 26. Conclusion • A new symbolic-statistical multimodal fusion algorithmbased on Hidden Markov Models • Integratedinto the HephaisTK framework • Superior recognition ratescompared to existingalgorithms • Handling of ambiguous cases • Processing of real-time user feedback http://wise.vub.ac.be/bruno-dumasJune 26, 2012 Bruno Dumas – WISE resarch lab – Department of Computer Science - bdumas@vub.ac.be 26

×