SlideShare a Scribd company logo
1 of 30
Download to read offline
ICS3211 - Intelligent
Interfaces II
Combining design with technology for effective human-
computer interaction
Week 8 Department of AI,
University of Malta
Design for Multimodal
Interfaces
Week 7 overview:
• Multimodal interactions
• Human interaction in multimodal systems
• Design guidelines
• Real world systems
Learning Outcomes
At the end of this session you should be able to:
• Describe the characteristics of multimodal interfaces;
• Draw inferences about the design of multimodal interfaces;
• Compare and contrast the multiple modalities which interfaces
would require depending on the context;
• List the best practices of the design principles for multimodal
interfaces;
Interaction Forms
• Bimanual
• Gesture recognition
• Speech recognition
• Free space gestures
• Background sensing
techniques
• AR - projector based
• BCI
Multimedia vs. Multimodal
• Multimedia – more than one mode of
communication is output to the user; e.g. a sound
clip attached to a presentation.
• Media channels: text, graphics, animation, video:
all visual media
• Multimodal – computer processes more than one
mode of communication; e.g. the combined input of
speech and touch in smart phones
• Sensory modalities: Visual, auditory, tactile,
Multimodal Interactions
• Traditional WIMP offers limited input/output
possibilities;
• Mix of audio/visual interactions important for
communication;
• All senses (including touch) are relevant;
• Combination of multiple modalities (including
speech, gestures, etc.) offer new functionalities.
Multimodal Interactions
• Modality is the mode or path of communication
according to human senses, using different types of
information and different interface devices;
• Some definitions:
Multimodal HCI system is simply one that responds to inputs in
more than one modality or communication channel (e.g. speech,
gesture, writing and others) [James/Sebe]
Multimodal interfaces process two or more combined user input
modes (such as speech, pen, touch, manual gesture, gaze and
head and body movements) in a coordinated manner with
multimedia system output. [Oviatt]
Multimodal Interactions
Case Scenarios for Discussion
Healthcare Companion Robot
• Scenario: A healthcare companion robot designed for elderly care, equipped with voice interaction, facial
recognition, touch-sensitive screens, and gesture control.
• Discussion Points: How do these multimodal interactions benefit the elderly, especially considering potential
impairments (vision, hearing, mobility)? What are the challenges in integrating these technologies seamlessly?
Interactive Learning System for Children
• Scenario: An educational platform for children that uses a combination of interactive storytelling (voice and
visuals), touch-based games, and AR experiences.
• Discussion Points: Discuss the benefits of using multiple modes of interaction in children's learning. How can
the system ensure engagement and effectiveness in education? What considerations should be made for
children at different developmental stages?
Smart Home Control Panel
• Scenario: A smart home interface that integrates voice commands, touchscreen inputs, and gesture controls
to manage home appliances, security, and lighting.
• Discussion Points: Evaluate the practicality and user experience of combining these modes of interaction.
What challenges might arise in terms of user adaptability and interface design?
Why go for Multimodal?
• Exploit the senses
• New uses
• Increased flexibility
• Helps with impairments
• More robust
• More engaging
What happens?
• When humans interact
amongst themselves …
• When humans interact
with a machine …
Input Modalities
• Speech or other sounds
• Head movements
(facial expression,
gaze)
• Pointing, pen, touch
• Body movement/
gestures
• Motion controller
(accelerometer)
• Tangibles
• Positioning
• Brain-computer
interface
• Biomodalities (sweat,
pulse, respiration)
Output Modalities
• Visual:
• Visualization
• 3D GUIs
• Virtual/Augmented Reality
• Auditory:
• Speech – Embodied
Conversational
• Sound
• Haptics
• Force feedback
• Low freq. bass
• Pain
• Taste
• Scent
Speech vs. Gestures
• Information that can be accessed from speech:
• Word recognition
• Language recognition
• Speaker recognition
• Emotion recognition
• Accent recognition
Speech vs. Gestures
• Humans use their body as communication modality:
• Gestures (explicit & implicit)
• Body language
• Focus of attention
• Activity
• Perception by computers:
• Computer vision
• Body mounted sensors
Haptics
• Manipulation tasks require feeling of objects;
• Computers can perceive this by:
• Haptic interfaces
• Tangible objects
• Force sensors
Biophysiological Modalities
• Body information through:
• Brain activity
• Skin conductance
• Temperature
• Heart rate
• Reveal information on:
• Workload
• Emotional state
• Mood
• Fatigue
Types of Multimodal
Interfaces
• Perceptual
• highly interactive
• rich, natural interaction
• Attentive
• context aware
• implicit
• Enactive
• relies on active manipulation through the use of hands or body,
such as TUI
Challenges of Multimodal
Interfaces
• Development of cognitive theories to guide
multimodal system design
• Development of effective natural language
processing
• Dialogue processing
• Error-handling techniques
• Function robustly and adaptively
• Support for collaborative multi-person use
Design of Multimodal
Interfaces
• Multimodal interfaces are designed for:
• compatibility with users’ work practices;
• flexibility;
• Design criteria;
• robustness increases as the number and heterogeneity
of modalities increase;
• performance improves with adaptivity of interface;
• persistence of operation despite physical damage, loss
of power, etc.
A representation of the multimodal man-machine interaction loop (taken from http://
humancarinteraction.com/multimodal-interaction.html )
Guidelines for the Design of
Multimodal Interfaces
• To achieve more natural interaction, like human-
human interaction
• To increase robustness by providing redundant and
complementary information
Guidelines for the Design of
Multimodal Interfaces
1. Requirements specifications
• design for broad range of users (experience, abilities, etc.) and contexts
(home, office, changing environments like car)
• address privacy and security issues
• don’t remember users by default
• use non-speech input for private information, like passwords
2. Designing multimodal input and output
• guidelines stem from cognitive science:
• maximize human cognitive and physical abilities e.g., don’t require
paying attention to two things at once
• reduce memory load
• multiple modes should complement each other, enhance each other
• integrate modalities to be compatible with user preferences, context
and system functionality e.g., match input and output styles
• use multimodal cues, e.g., look at speaker
• synchronize modalities (timing)
• synchronize system state across modalities
3. Adaptivity
• adapt to needs/experiences/skill levels of different users and contexts
• examples: gestures replace sounds in noisy settings, accommodate
for slow bandwidth, adapt quantity and stye of information display
based on user’s perceived skill level
4. Consistency
• use same language/keywords for all modalities
• use same interaction shortcuts for all modalities
• support both user and system switching between modalities
5. Feedback
• users should know what the current modality is and what other modalities
are available
• avoid lengthy instructions
• use common icons, simple instructions and labels
• confirm system interpretation of user’s commands, after fusion of all input
modalities has completed
6. Error preventing and handling
• clearly mark “exits” from: task, modality & system
• support “undo” & include help
• integrate complementary modalities to improve robustness:
strengths of one modality should overcome weaknesses of others
• let users control modality selection
• use rich modalities that can convey semantic information beyond
simple
• point-and-click
• fuse information from multiple sources
• Users do like to interact multimodally with artificial systems
• Multimodal interaction is preferred in spatial domains; we should
offer & expect more than point & speak
• Multimodal interaction changes the way we talk; we need to
adapt our speech processing components
• Speech and gesture are synchronized but not simultaneous;
• We cannot assume redundancy of content; we need to process
modalities in an integrated manner
• Use multimodality for better system error characteristics, through
expecting simplified speech, fusion of modalities to clear out
uncertainty & offering the right modality for the right task
Myths of Multimodal
Interfaces
1. If you build a multimodal system, users will interact multimodally
2. Speech and pointing is the dominant multimodal integration pattern
3. Multimodal input involves simultaneous signals
4. Speech is the primary input mode in any multimodal system that includes it
5. Multimodal language does not differ linguistically from unimodal language
6. Multimodal integration involves redundancy of content between modes
7. Individual error-prone recognition technologies combine multimodally to
produce even greater unreliability
8. All users’ multimodal commands are integrated in a uniform way
9. Different input modes are capable of transmitting comparable content
10.Enhanced efficiency is the main advantage of multimodal systems
MultiModal Research Areas
1. [Input] Applied ML: Speech Recognition/Synthesis,
Gesture Recognition and Motion Tracking, Head, Gait &
Pose Estimation, Multimodal Fusion
2. [Input] HCI: Usability Issues, Context Aware & Ubiquitous
Computing, Design of Multimodal Interfaces
3. [Output] Immersive Environments: Embodied Conversation
Agents, Haptics (force feedback), Audio (feedback)
4. [Output] HCI: Multimodal Feedback (in-vehicle navigation,
surgery, ergonomics, etc.), multimodal Interface design
Case Example
Philippe, S., Souchet, A. D., Lameras, P., Petridis, P., Caporal,
J., Coldeboeuf, G., & Duzan, H. (2020). Multimodal teaching,
learning and training in virtual reality: a review and case
study. Virtual Reality & Intelligent Hardware, 2(5), 421-442.
Key Concepts in Multimodal
Interfaces
• We cannot always believe our intuition on how
interaction will function; we need to find out by
performing well designed user studies
• We need to take a close look at how human – human
communication and interaction works before we can
build systems that resemble this behaviour; costly
collection and annotation of real world data is
necessary
• We need to generate semantic representations from
our sub-symbolic feature space

More Related Content

Similar to ICS3211_lecture 08_2023.pdf

ICS3211 lntelligent Interfaces
ICS3211 lntelligent InterfacesICS3211 lntelligent Interfaces
ICS3211 lntelligent InterfacesVanessa Camilleri
 
Cognitive frameworks
Cognitive frameworksCognitive frameworks
Cognitive frameworksSaqib Shehzad
 
Usability Workshop at Lillebaelt Academy
Usability Workshop at Lillebaelt AcademyUsability Workshop at Lillebaelt Academy
Usability Workshop at Lillebaelt AcademyDániel Góré
 
Usability and User Experience Training Seminar
Usability and User Experience Training SeminarUsability and User Experience Training Seminar
Usability and User Experience Training Seminarlabecvar
 
ICS3211_lecture_week52023.pdf
ICS3211_lecture_week52023.pdfICS3211_lecture_week52023.pdf
ICS3211_lecture_week52023.pdfVanessa Camilleri
 
CSE868 - Week 01 & 2 - Introduction and HCI Principles.pdf
CSE868 - Week 01 & 2 - Introduction and HCI Principles.pdfCSE868 - Week 01 & 2 - Introduction and HCI Principles.pdf
CSE868 - Week 01 & 2 - Introduction and HCI Principles.pdfsofiamuhammad2
 
Human Computer Interaction Notes 176.pdf
Human Computer Interaction Notes 176.pdfHuman Computer Interaction Notes 176.pdf
Human Computer Interaction Notes 176.pdfvijaykumarK44
 
Multi Platform User Exerience
Multi Platform User ExerienceMulti Platform User Exerience
Multi Platform User ExerienceTanya Zavialova
 
Introduction to HCI (UCC)
Introduction to HCI (UCC)Introduction to HCI (UCC)
Introduction to HCI (UCC)apppsych
 
Dialogue management system
Dialogue management systemDialogue management system
Dialogue management systemMayank Agarwal
 
Universal Usability
Universal UsabilityUniversal Usability
Universal UsabilityNaviyaMariya
 
Cognition, information processing, perception and attention
Cognition, information processing, perception and attentionCognition, information processing, perception and attention
Cognition, information processing, perception and attentionSaqib Shehzad
 

Similar to ICS3211_lecture 08_2023.pdf (20)

ICS3211 lntelligent Interfaces
ICS3211 lntelligent InterfacesICS3211 lntelligent Interfaces
ICS3211 lntelligent Interfaces
 
Cognitive frameworks
Cognitive frameworksCognitive frameworks
Cognitive frameworks
 
ICS3211 lecture 02
ICS3211 lecture 02ICS3211 lecture 02
ICS3211 lecture 02
 
Usability Workshop at Lillebaelt Academy
Usability Workshop at Lillebaelt AcademyUsability Workshop at Lillebaelt Academy
Usability Workshop at Lillebaelt Academy
 
Lecture1.ppt
Lecture1.pptLecture1.ppt
Lecture1.ppt
 
Conceptual Model
Conceptual ModelConceptual Model
Conceptual Model
 
ICT L4.pptx
ICT L4.pptxICT L4.pptx
ICT L4.pptx
 
Chapter seven hci
Chapter seven hciChapter seven hci
Chapter seven hci
 
HCI
HCIHCI
HCI
 
Usability and User Experience Training Seminar
Usability and User Experience Training SeminarUsability and User Experience Training Seminar
Usability and User Experience Training Seminar
 
ICS3211_lecture_week52023.pdf
ICS3211_lecture_week52023.pdfICS3211_lecture_week52023.pdf
ICS3211_lecture_week52023.pdf
 
CSE868 - Week 01 & 2 - Introduction and HCI Principles.pdf
CSE868 - Week 01 & 2 - Introduction and HCI Principles.pdfCSE868 - Week 01 & 2 - Introduction and HCI Principles.pdf
CSE868 - Week 01 & 2 - Introduction and HCI Principles.pdf
 
Multimodal Interaction
Multimodal InteractionMultimodal Interaction
Multimodal Interaction
 
Human Computer Interaction Notes 176.pdf
Human Computer Interaction Notes 176.pdfHuman Computer Interaction Notes 176.pdf
Human Computer Interaction Notes 176.pdf
 
Multi Platform User Exerience
Multi Platform User ExerienceMulti Platform User Exerience
Multi Platform User Exerience
 
ICS2208 Lecture2
ICS2208 Lecture2ICS2208 Lecture2
ICS2208 Lecture2
 
Introduction to HCI (UCC)
Introduction to HCI (UCC)Introduction to HCI (UCC)
Introduction to HCI (UCC)
 
Dialogue management system
Dialogue management systemDialogue management system
Dialogue management system
 
Universal Usability
Universal UsabilityUniversal Usability
Universal Usability
 
Cognition, information processing, perception and attention
Cognition, information processing, perception and attentionCognition, information processing, perception and attention
Cognition, information processing, perception and attention
 

More from Vanessa Camilleri

ICS 2208 Lecture 8 Slides AI and VR_.pdf
ICS 2208 Lecture 8 Slides AI and VR_.pdfICS 2208 Lecture 8 Slides AI and VR_.pdf
ICS 2208 Lecture 8 Slides AI and VR_.pdfVanessa Camilleri
 
ICS2208 Lecture6 Notes for SL spaces.pdf
ICS2208 Lecture6 Notes for SL spaces.pdfICS2208 Lecture6 Notes for SL spaces.pdf
ICS2208 Lecture6 Notes for SL spaces.pdfVanessa Camilleri
 
ICS 2208 Lecture Slide Notes for Topic 6
ICS 2208 Lecture Slide Notes for Topic 6ICS 2208 Lecture Slide Notes for Topic 6
ICS 2208 Lecture Slide Notes for Topic 6Vanessa Camilleri
 
ICS2208 Lecture4 Intelligent Interface Agents.pdf
ICS2208 Lecture4 Intelligent Interface Agents.pdfICS2208 Lecture4 Intelligent Interface Agents.pdf
ICS2208 Lecture4 Intelligent Interface Agents.pdfVanessa Camilleri
 
ICS2208 Lecture3 2023-2024 - Model Based User Interfaces
ICS2208 Lecture3 2023-2024 - Model Based User InterfacesICS2208 Lecture3 2023-2024 - Model Based User Interfaces
ICS2208 Lecture3 2023-2024 - Model Based User InterfacesVanessa Camilleri
 
ICS2208 Lecture 2 Slides Interfaces_.pdf
ICS2208 Lecture 2 Slides Interfaces_.pdfICS2208 Lecture 2 Slides Interfaces_.pdf
ICS2208 Lecture 2 Slides Interfaces_.pdfVanessa Camilleri
 
ICS Lecture 11 - Intelligent Interfaces 2023
ICS Lecture 11 - Intelligent Interfaces 2023ICS Lecture 11 - Intelligent Interfaces 2023
ICS Lecture 11 - Intelligent Interfaces 2023Vanessa Camilleri
 
ICS3211_lecture_week72023.pdf
ICS3211_lecture_week72023.pdfICS3211_lecture_week72023.pdf
ICS3211_lecture_week72023.pdfVanessa Camilleri
 
ICS3211_lecture_week62023.pdf
ICS3211_lecture_week62023.pdfICS3211_lecture_week62023.pdf
ICS3211_lecture_week62023.pdfVanessa Camilleri
 

More from Vanessa Camilleri (20)

ICS 2208 Lecture 8 Slides AI and VR_.pdf
ICS 2208 Lecture 8 Slides AI and VR_.pdfICS 2208 Lecture 8 Slides AI and VR_.pdf
ICS 2208 Lecture 8 Slides AI and VR_.pdf
 
ICS2208 Lecture6 Notes for SL spaces.pdf
ICS2208 Lecture6 Notes for SL spaces.pdfICS2208 Lecture6 Notes for SL spaces.pdf
ICS2208 Lecture6 Notes for SL spaces.pdf
 
ICS 2208 Lecture Slide Notes for Topic 6
ICS 2208 Lecture Slide Notes for Topic 6ICS 2208 Lecture Slide Notes for Topic 6
ICS 2208 Lecture Slide Notes for Topic 6
 
ICS2208 Lecture4 Intelligent Interface Agents.pdf
ICS2208 Lecture4 Intelligent Interface Agents.pdfICS2208 Lecture4 Intelligent Interface Agents.pdf
ICS2208 Lecture4 Intelligent Interface Agents.pdf
 
ICS2208 Lecture3 2023-2024 - Model Based User Interfaces
ICS2208 Lecture3 2023-2024 - Model Based User InterfacesICS2208 Lecture3 2023-2024 - Model Based User Interfaces
ICS2208 Lecture3 2023-2024 - Model Based User Interfaces
 
ICS2208 Lecture 2 Slides Interfaces_.pdf
ICS2208 Lecture 2 Slides Interfaces_.pdfICS2208 Lecture 2 Slides Interfaces_.pdf
ICS2208 Lecture 2 Slides Interfaces_.pdf
 
ICS Lecture 11 - Intelligent Interfaces 2023
ICS Lecture 11 - Intelligent Interfaces 2023ICS Lecture 11 - Intelligent Interfaces 2023
ICS Lecture 11 - Intelligent Interfaces 2023
 
ICS3211_lecture 09_2023.pdf
ICS3211_lecture 09_2023.pdfICS3211_lecture 09_2023.pdf
ICS3211_lecture 09_2023.pdf
 
ICS3211_lecture_week72023.pdf
ICS3211_lecture_week72023.pdfICS3211_lecture_week72023.pdf
ICS3211_lecture_week72023.pdf
 
ICS3211_lecture_week62023.pdf
ICS3211_lecture_week62023.pdfICS3211_lecture_week62023.pdf
ICS3211_lecture_week62023.pdf
 
ICS3211_lecture 04 2023.pdf
ICS3211_lecture 04 2023.pdfICS3211_lecture 04 2023.pdf
ICS3211_lecture 04 2023.pdf
 
ICS3211_lecture 03 2023.pdf
ICS3211_lecture 03 2023.pdfICS3211_lecture 03 2023.pdf
ICS3211_lecture 03 2023.pdf
 
ICS3211_lecture 11.pdf
ICS3211_lecture 11.pdfICS3211_lecture 11.pdf
ICS3211_lecture 11.pdf
 
FoundationsAIEthics2023.pdf
FoundationsAIEthics2023.pdfFoundationsAIEthics2023.pdf
FoundationsAIEthics2023.pdf
 
ICS3211_lecture 9_2022.pdf
ICS3211_lecture 9_2022.pdfICS3211_lecture 9_2022.pdf
ICS3211_lecture 9_2022.pdf
 
ICS1020CV_2022.pdf
ICS1020CV_2022.pdfICS1020CV_2022.pdf
ICS1020CV_2022.pdf
 
ARI5902_2022.pdf
ARI5902_2022.pdfARI5902_2022.pdf
ARI5902_2022.pdf
 
ICS2208 Lecture10
ICS2208 Lecture10ICS2208 Lecture10
ICS2208 Lecture10
 
ICS2208 lecture9
ICS2208 lecture9ICS2208 lecture9
ICS2208 lecture9
 
ICS2208 lecture7
ICS2208 lecture7ICS2208 lecture7
ICS2208 lecture7
 

Recently uploaded

Exploring Gemini AI and Integration with MuleSoft | MuleSoft Mysore Meetup #45
Exploring Gemini AI and Integration with MuleSoft | MuleSoft Mysore Meetup #45Exploring Gemini AI and Integration with MuleSoft | MuleSoft Mysore Meetup #45
Exploring Gemini AI and Integration with MuleSoft | MuleSoft Mysore Meetup #45MysoreMuleSoftMeetup
 
Graduate Outcomes Presentation Slides - English (v3).pptx
Graduate Outcomes Presentation Slides - English (v3).pptxGraduate Outcomes Presentation Slides - English (v3).pptx
Graduate Outcomes Presentation Slides - English (v3).pptxneillewis46
 
TỔNG HỢP HƠN 100 ĐỀ THI THỬ TỐT NGHIỆP THPT VẬT LÝ 2024 - TỪ CÁC TRƯỜNG, TRƯ...
TỔNG HỢP HƠN 100 ĐỀ THI THỬ TỐT NGHIỆP THPT VẬT LÝ 2024 - TỪ CÁC TRƯỜNG, TRƯ...TỔNG HỢP HƠN 100 ĐỀ THI THỬ TỐT NGHIỆP THPT VẬT LÝ 2024 - TỪ CÁC TRƯỜNG, TRƯ...
TỔNG HỢP HƠN 100 ĐỀ THI THỬ TỐT NGHIỆP THPT VẬT LÝ 2024 - TỪ CÁC TRƯỜNG, TRƯ...Nguyen Thanh Tu Collection
 
Dementia (Alzheimer & vasular dementia).
Dementia (Alzheimer & vasular dementia).Dementia (Alzheimer & vasular dementia).
Dementia (Alzheimer & vasular dementia).Mohamed Rizk Khodair
 
Basic Civil Engineering notes on Transportation Engineering, Modes of Transpo...
Basic Civil Engineering notes on Transportation Engineering, Modes of Transpo...Basic Civil Engineering notes on Transportation Engineering, Modes of Transpo...
Basic Civil Engineering notes on Transportation Engineering, Modes of Transpo...Denish Jangid
 
Capitol Tech Univ Doctoral Presentation -May 2024
Capitol Tech Univ Doctoral Presentation -May 2024Capitol Tech Univ Doctoral Presentation -May 2024
Capitol Tech Univ Doctoral Presentation -May 2024CapitolTechU
 
Features of Video Calls in the Discuss Module in Odoo 17
Features of Video Calls in the Discuss Module in Odoo 17Features of Video Calls in the Discuss Module in Odoo 17
Features of Video Calls in the Discuss Module in Odoo 17Celine George
 
An Overview of the Odoo 17 Knowledge App
An Overview of the Odoo 17 Knowledge AppAn Overview of the Odoo 17 Knowledge App
An Overview of the Odoo 17 Knowledge AppCeline George
 
Navigating the Misinformation Minefield: The Role of Higher Education in the ...
Navigating the Misinformation Minefield: The Role of Higher Education in the ...Navigating the Misinformation Minefield: The Role of Higher Education in the ...
Navigating the Misinformation Minefield: The Role of Higher Education in the ...Mark Carrigan
 
....................Muslim-Law notes.pdf
....................Muslim-Law notes.pdf....................Muslim-Law notes.pdf
....................Muslim-Law notes.pdfVikramadityaRaj
 
Championnat de France de Tennis de table/
Championnat de France de Tennis de table/Championnat de France de Tennis de table/
Championnat de France de Tennis de table/siemaillard
 
II BIOSENSOR PRINCIPLE APPLICATIONS AND WORKING II
II BIOSENSOR PRINCIPLE APPLICATIONS AND WORKING IIII BIOSENSOR PRINCIPLE APPLICATIONS AND WORKING II
II BIOSENSOR PRINCIPLE APPLICATIONS AND WORKING IIagpharmacy11
 
MichaelStarkes_UncutGemsProjectSummary.pdf
MichaelStarkes_UncutGemsProjectSummary.pdfMichaelStarkes_UncutGemsProjectSummary.pdf
MichaelStarkes_UncutGemsProjectSummary.pdfmstarkes24
 
An Overview of the Odoo 17 Discuss App.pptx
An Overview of the Odoo 17 Discuss App.pptxAn Overview of the Odoo 17 Discuss App.pptx
An Overview of the Odoo 17 Discuss App.pptxCeline George
 
philosophy and it's principles based on the life
philosophy and it's principles based on the lifephilosophy and it's principles based on the life
philosophy and it's principles based on the lifeNitinDeodare
 
Spring gala 2024 photo slideshow - Celebrating School-Community Partnerships
Spring gala 2024 photo slideshow - Celebrating School-Community PartnershipsSpring gala 2024 photo slideshow - Celebrating School-Community Partnerships
Spring gala 2024 photo slideshow - Celebrating School-Community Partnershipsexpandedwebsite
 
ĐỀ THAM KHẢO KÌ THI TUYỂN SINH VÀO LỚP 10 MÔN TIẾNG ANH FORM 50 CÂU TRẮC NGHI...
ĐỀ THAM KHẢO KÌ THI TUYỂN SINH VÀO LỚP 10 MÔN TIẾNG ANH FORM 50 CÂU TRẮC NGHI...ĐỀ THAM KHẢO KÌ THI TUYỂN SINH VÀO LỚP 10 MÔN TIẾNG ANH FORM 50 CÂU TRẮC NGHI...
ĐỀ THAM KHẢO KÌ THI TUYỂN SINH VÀO LỚP 10 MÔN TIẾNG ANH FORM 50 CÂU TRẮC NGHI...Nguyen Thanh Tu Collection
 

Recently uploaded (20)

Exploring Gemini AI and Integration with MuleSoft | MuleSoft Mysore Meetup #45
Exploring Gemini AI and Integration with MuleSoft | MuleSoft Mysore Meetup #45Exploring Gemini AI and Integration with MuleSoft | MuleSoft Mysore Meetup #45
Exploring Gemini AI and Integration with MuleSoft | MuleSoft Mysore Meetup #45
 
Graduate Outcomes Presentation Slides - English (v3).pptx
Graduate Outcomes Presentation Slides - English (v3).pptxGraduate Outcomes Presentation Slides - English (v3).pptx
Graduate Outcomes Presentation Slides - English (v3).pptx
 
TỔNG HỢP HƠN 100 ĐỀ THI THỬ TỐT NGHIỆP THPT VẬT LÝ 2024 - TỪ CÁC TRƯỜNG, TRƯ...
TỔNG HỢP HƠN 100 ĐỀ THI THỬ TỐT NGHIỆP THPT VẬT LÝ 2024 - TỪ CÁC TRƯỜNG, TRƯ...TỔNG HỢP HƠN 100 ĐỀ THI THỬ TỐT NGHIỆP THPT VẬT LÝ 2024 - TỪ CÁC TRƯỜNG, TRƯ...
TỔNG HỢP HƠN 100 ĐỀ THI THỬ TỐT NGHIỆP THPT VẬT LÝ 2024 - TỪ CÁC TRƯỜNG, TRƯ...
 
Dementia (Alzheimer & vasular dementia).
Dementia (Alzheimer & vasular dementia).Dementia (Alzheimer & vasular dementia).
Dementia (Alzheimer & vasular dementia).
 
Basic Civil Engineering notes on Transportation Engineering, Modes of Transpo...
Basic Civil Engineering notes on Transportation Engineering, Modes of Transpo...Basic Civil Engineering notes on Transportation Engineering, Modes of Transpo...
Basic Civil Engineering notes on Transportation Engineering, Modes of Transpo...
 
Mattingly "AI and Prompt Design: LLMs with Text Classification and Open Source"
Mattingly "AI and Prompt Design: LLMs with Text Classification and Open Source"Mattingly "AI and Prompt Design: LLMs with Text Classification and Open Source"
Mattingly "AI and Prompt Design: LLMs with Text Classification and Open Source"
 
Capitol Tech Univ Doctoral Presentation -May 2024
Capitol Tech Univ Doctoral Presentation -May 2024Capitol Tech Univ Doctoral Presentation -May 2024
Capitol Tech Univ Doctoral Presentation -May 2024
 
Features of Video Calls in the Discuss Module in Odoo 17
Features of Video Calls in the Discuss Module in Odoo 17Features of Video Calls in the Discuss Module in Odoo 17
Features of Video Calls in the Discuss Module in Odoo 17
 
An Overview of the Odoo 17 Knowledge App
An Overview of the Odoo 17 Knowledge AppAn Overview of the Odoo 17 Knowledge App
An Overview of the Odoo 17 Knowledge App
 
Word Stress rules esl .pptx
Word Stress rules esl               .pptxWord Stress rules esl               .pptx
Word Stress rules esl .pptx
 
Navigating the Misinformation Minefield: The Role of Higher Education in the ...
Navigating the Misinformation Minefield: The Role of Higher Education in the ...Navigating the Misinformation Minefield: The Role of Higher Education in the ...
Navigating the Misinformation Minefield: The Role of Higher Education in the ...
 
....................Muslim-Law notes.pdf
....................Muslim-Law notes.pdf....................Muslim-Law notes.pdf
....................Muslim-Law notes.pdf
 
Championnat de France de Tennis de table/
Championnat de France de Tennis de table/Championnat de France de Tennis de table/
Championnat de France de Tennis de table/
 
II BIOSENSOR PRINCIPLE APPLICATIONS AND WORKING II
II BIOSENSOR PRINCIPLE APPLICATIONS AND WORKING IIII BIOSENSOR PRINCIPLE APPLICATIONS AND WORKING II
II BIOSENSOR PRINCIPLE APPLICATIONS AND WORKING II
 
MichaelStarkes_UncutGemsProjectSummary.pdf
MichaelStarkes_UncutGemsProjectSummary.pdfMichaelStarkes_UncutGemsProjectSummary.pdf
MichaelStarkes_UncutGemsProjectSummary.pdf
 
IPL Online Quiz by Pragya; Question Set.
IPL Online Quiz by Pragya; Question Set.IPL Online Quiz by Pragya; Question Set.
IPL Online Quiz by Pragya; Question Set.
 
An Overview of the Odoo 17 Discuss App.pptx
An Overview of the Odoo 17 Discuss App.pptxAn Overview of the Odoo 17 Discuss App.pptx
An Overview of the Odoo 17 Discuss App.pptx
 
philosophy and it's principles based on the life
philosophy and it's principles based on the lifephilosophy and it's principles based on the life
philosophy and it's principles based on the life
 
Spring gala 2024 photo slideshow - Celebrating School-Community Partnerships
Spring gala 2024 photo slideshow - Celebrating School-Community PartnershipsSpring gala 2024 photo slideshow - Celebrating School-Community Partnerships
Spring gala 2024 photo slideshow - Celebrating School-Community Partnerships
 
ĐỀ THAM KHẢO KÌ THI TUYỂN SINH VÀO LỚP 10 MÔN TIẾNG ANH FORM 50 CÂU TRẮC NGHI...
ĐỀ THAM KHẢO KÌ THI TUYỂN SINH VÀO LỚP 10 MÔN TIẾNG ANH FORM 50 CÂU TRẮC NGHI...ĐỀ THAM KHẢO KÌ THI TUYỂN SINH VÀO LỚP 10 MÔN TIẾNG ANH FORM 50 CÂU TRẮC NGHI...
ĐỀ THAM KHẢO KÌ THI TUYỂN SINH VÀO LỚP 10 MÔN TIẾNG ANH FORM 50 CÂU TRẮC NGHI...
 

ICS3211_lecture 08_2023.pdf

  • 1. ICS3211 - Intelligent Interfaces II Combining design with technology for effective human- computer interaction Week 8 Department of AI, University of Malta
  • 2. Design for Multimodal Interfaces Week 7 overview: • Multimodal interactions • Human interaction in multimodal systems • Design guidelines • Real world systems
  • 3. Learning Outcomes At the end of this session you should be able to: • Describe the characteristics of multimodal interfaces; • Draw inferences about the design of multimodal interfaces; • Compare and contrast the multiple modalities which interfaces would require depending on the context; • List the best practices of the design principles for multimodal interfaces;
  • 4. Interaction Forms • Bimanual • Gesture recognition • Speech recognition • Free space gestures • Background sensing techniques • AR - projector based • BCI
  • 5. Multimedia vs. Multimodal • Multimedia – more than one mode of communication is output to the user; e.g. a sound clip attached to a presentation. • Media channels: text, graphics, animation, video: all visual media • Multimodal – computer processes more than one mode of communication; e.g. the combined input of speech and touch in smart phones • Sensory modalities: Visual, auditory, tactile,
  • 6. Multimodal Interactions • Traditional WIMP offers limited input/output possibilities; • Mix of audio/visual interactions important for communication; • All senses (including touch) are relevant; • Combination of multiple modalities (including speech, gestures, etc.) offer new functionalities.
  • 7. Multimodal Interactions • Modality is the mode or path of communication according to human senses, using different types of information and different interface devices; • Some definitions: Multimodal HCI system is simply one that responds to inputs in more than one modality or communication channel (e.g. speech, gesture, writing and others) [James/Sebe] Multimodal interfaces process two or more combined user input modes (such as speech, pen, touch, manual gesture, gaze and head and body movements) in a coordinated manner with multimedia system output. [Oviatt]
  • 8. Multimodal Interactions Case Scenarios for Discussion Healthcare Companion Robot • Scenario: A healthcare companion robot designed for elderly care, equipped with voice interaction, facial recognition, touch-sensitive screens, and gesture control. • Discussion Points: How do these multimodal interactions benefit the elderly, especially considering potential impairments (vision, hearing, mobility)? What are the challenges in integrating these technologies seamlessly? Interactive Learning System for Children • Scenario: An educational platform for children that uses a combination of interactive storytelling (voice and visuals), touch-based games, and AR experiences. • Discussion Points: Discuss the benefits of using multiple modes of interaction in children's learning. How can the system ensure engagement and effectiveness in education? What considerations should be made for children at different developmental stages? Smart Home Control Panel • Scenario: A smart home interface that integrates voice commands, touchscreen inputs, and gesture controls to manage home appliances, security, and lighting. • Discussion Points: Evaluate the practicality and user experience of combining these modes of interaction. What challenges might arise in terms of user adaptability and interface design?
  • 9. Why go for Multimodal? • Exploit the senses • New uses • Increased flexibility • Helps with impairments • More robust • More engaging
  • 10. What happens? • When humans interact amongst themselves … • When humans interact with a machine …
  • 11. Input Modalities • Speech or other sounds • Head movements (facial expression, gaze) • Pointing, pen, touch • Body movement/ gestures • Motion controller (accelerometer) • Tangibles • Positioning • Brain-computer interface • Biomodalities (sweat, pulse, respiration)
  • 12. Output Modalities • Visual: • Visualization • 3D GUIs • Virtual/Augmented Reality • Auditory: • Speech – Embodied Conversational • Sound • Haptics • Force feedback • Low freq. bass • Pain • Taste • Scent
  • 13. Speech vs. Gestures • Information that can be accessed from speech: • Word recognition • Language recognition • Speaker recognition • Emotion recognition • Accent recognition
  • 14. Speech vs. Gestures • Humans use their body as communication modality: • Gestures (explicit & implicit) • Body language • Focus of attention • Activity • Perception by computers: • Computer vision • Body mounted sensors
  • 15. Haptics • Manipulation tasks require feeling of objects; • Computers can perceive this by: • Haptic interfaces • Tangible objects • Force sensors
  • 16. Biophysiological Modalities • Body information through: • Brain activity • Skin conductance • Temperature • Heart rate • Reveal information on: • Workload • Emotional state • Mood • Fatigue
  • 17. Types of Multimodal Interfaces • Perceptual • highly interactive • rich, natural interaction • Attentive • context aware • implicit • Enactive • relies on active manipulation through the use of hands or body, such as TUI
  • 18. Challenges of Multimodal Interfaces • Development of cognitive theories to guide multimodal system design • Development of effective natural language processing • Dialogue processing • Error-handling techniques • Function robustly and adaptively • Support for collaborative multi-person use
  • 19. Design of Multimodal Interfaces • Multimodal interfaces are designed for: • compatibility with users’ work practices; • flexibility; • Design criteria; • robustness increases as the number and heterogeneity of modalities increase; • performance improves with adaptivity of interface; • persistence of operation despite physical damage, loss of power, etc.
  • 20. A representation of the multimodal man-machine interaction loop (taken from http:// humancarinteraction.com/multimodal-interaction.html )
  • 21. Guidelines for the Design of Multimodal Interfaces • To achieve more natural interaction, like human- human interaction • To increase robustness by providing redundant and complementary information
  • 22. Guidelines for the Design of Multimodal Interfaces 1. Requirements specifications • design for broad range of users (experience, abilities, etc.) and contexts (home, office, changing environments like car) • address privacy and security issues • don’t remember users by default • use non-speech input for private information, like passwords 2. Designing multimodal input and output • guidelines stem from cognitive science: • maximize human cognitive and physical abilities e.g., don’t require paying attention to two things at once • reduce memory load
  • 23. • multiple modes should complement each other, enhance each other • integrate modalities to be compatible with user preferences, context and system functionality e.g., match input and output styles • use multimodal cues, e.g., look at speaker • synchronize modalities (timing) • synchronize system state across modalities 3. Adaptivity • adapt to needs/experiences/skill levels of different users and contexts • examples: gestures replace sounds in noisy settings, accommodate for slow bandwidth, adapt quantity and stye of information display based on user’s perceived skill level
  • 24. 4. Consistency • use same language/keywords for all modalities • use same interaction shortcuts for all modalities • support both user and system switching between modalities 5. Feedback • users should know what the current modality is and what other modalities are available • avoid lengthy instructions • use common icons, simple instructions and labels • confirm system interpretation of user’s commands, after fusion of all input modalities has completed
  • 25. 6. Error preventing and handling • clearly mark “exits” from: task, modality & system • support “undo” & include help • integrate complementary modalities to improve robustness: strengths of one modality should overcome weaknesses of others • let users control modality selection • use rich modalities that can convey semantic information beyond simple • point-and-click • fuse information from multiple sources
  • 26. • Users do like to interact multimodally with artificial systems • Multimodal interaction is preferred in spatial domains; we should offer & expect more than point & speak • Multimodal interaction changes the way we talk; we need to adapt our speech processing components • Speech and gesture are synchronized but not simultaneous; • We cannot assume redundancy of content; we need to process modalities in an integrated manner • Use multimodality for better system error characteristics, through expecting simplified speech, fusion of modalities to clear out uncertainty & offering the right modality for the right task
  • 27. Myths of Multimodal Interfaces 1. If you build a multimodal system, users will interact multimodally 2. Speech and pointing is the dominant multimodal integration pattern 3. Multimodal input involves simultaneous signals 4. Speech is the primary input mode in any multimodal system that includes it 5. Multimodal language does not differ linguistically from unimodal language 6. Multimodal integration involves redundancy of content between modes 7. Individual error-prone recognition technologies combine multimodally to produce even greater unreliability 8. All users’ multimodal commands are integrated in a uniform way 9. Different input modes are capable of transmitting comparable content 10.Enhanced efficiency is the main advantage of multimodal systems
  • 28. MultiModal Research Areas 1. [Input] Applied ML: Speech Recognition/Synthesis, Gesture Recognition and Motion Tracking, Head, Gait & Pose Estimation, Multimodal Fusion 2. [Input] HCI: Usability Issues, Context Aware & Ubiquitous Computing, Design of Multimodal Interfaces 3. [Output] Immersive Environments: Embodied Conversation Agents, Haptics (force feedback), Audio (feedback) 4. [Output] HCI: Multimodal Feedback (in-vehicle navigation, surgery, ergonomics, etc.), multimodal Interface design
  • 29. Case Example Philippe, S., Souchet, A. D., Lameras, P., Petridis, P., Caporal, J., Coldeboeuf, G., & Duzan, H. (2020). Multimodal teaching, learning and training in virtual reality: a review and case study. Virtual Reality & Intelligent Hardware, 2(5), 421-442.
  • 30. Key Concepts in Multimodal Interfaces • We cannot always believe our intuition on how interaction will function; we need to find out by performing well designed user studies • We need to take a close look at how human – human communication and interaction works before we can build systems that resemble this behaviour; costly collection and annotation of real world data is necessary • We need to generate semantic representations from our sub-symbolic feature space