Matching Attentional Draw with Utility in Interruption Jennifer Gluck, Andrea Bunt, Joanna McGrenere University of British Columbia CHI 2007 April 30, 2007

Matching Attentional Draw with Utility in Interruption? Attentional Draw (AD): How quickly attention is drawn to a notification Utility: Usefulness of the interruption content Matching: Low AD ↔ Low Utility … High AD ↔ High Utility

Attentional Draw (AD):

How quickly attention is drawn to a notification

Utility:

Usefulness of the interruption content

Matching:

Low AD ↔ Low Utility

…

High AD ↔ High Utility

Interruption When something causes a break in our actions, activities, or concentration.

When something causes a break in our actions, activities, or concentration.

Negative Effects Lowered primary task performance Cutrell, Czerwinski, & Horvitz 2001 Gillie & Broadbent 1989 Increased anxiety, annoyance, perceived difficulty in completing primary task Bailey, Konstan, & Carlis, 2001

Lowered primary task performance

Cutrell, Czerwinski, & Horvitz 2001

Gillie & Broadbent 1989

Increased anxiety, annoyance, perceived

difficulty in completing primary task

Bailey, Konstan, & Carlis, 2001

Positive Aspects Technologies that interrupt are popular suggests there is value to interruption Other systems also have the potential to promote beneficial interruption… 

Technologies that interrupt are popular

suggests there is value to interruption

Other systems also have the potential to promote beneficial interruption…

Assist users by making suggestions that can help users complete a task FlexExcel (Thomas & Krogseoter, 1993) Adaptive Bar (Debevc, Meyer, Donlagic & Svecko 1996) Mixed-Initiative Systems How can we design interruption to emphasize positive aspects?

Assist users by making suggestions that can help users complete a task

FlexExcel (Thomas & Krogseoter, 1993)

Adaptive Bar (Debevc, Meyer, Donlagic & Svecko 1996)

Design Guideline Make the amount of attention attracted by a notification relative to the usefulness of the interruption content Obermayer & Nugent, 2000 McFarlane & Latorella, 2002

Make the amount of attention attracted by a notification relative to the usefulness of the interruption content

Obermayer & Nugent, 2000

McFarlane & Latorella, 2002

What did we do? Empirical investigation of the effects of matching attentional draw (AD) of notification and interruption utility

Empirical investigation of the effects

of matching attentional draw (AD) of

notification and interruption utility

Attentional Draw (AD) Empirical investigation of the effects of matching attentional draw (AD) of notification and interruption utility How quickly your attention is drawn to the notification signal attentional draw

Empirical investigation of the effects

of matching attentional draw (AD) of

notification and interruption utility

How quickly your attention is drawn to the notification signal

Utility Empirical investigation of the effects of matching attentional draw (AD) of notification and interruption utility How useful, important, or urgent the interruption content is to the recipient “ How helpful the interruption is in terms of performing a primary task” utility

Empirical investigation of the effects

of matching attentional draw (AD) of

notification and interruption utility

How useful, important, or urgent the interruption content is to the recipient

“ How helpful the interruption is in terms of performing a primary task”

Findings Reduces negative effects Facilitates positive perception of interruption Empirical investigation of the effects of matching attentional draw (AD) of notification and interruption utility

Reduces negative effects

Facilitates positive perception of interruption

Research Overview Study 1: Find some signals Established a set of significantly different notification signals in terms of their attentional draw (AD) Study 2: Match the signals to utility Investigated matching AD and utility to see if this strategy can promote positive interruption

Study 1: Find some signals

Established a set of significantly different notification signals in terms of their attentional draw (AD)

Study 2: Match the signals to utility

Investigated matching AD and utility to see if this strategy can promote positive interruption

Primary Task

Goal: Establish a set of visual notification signals with significantly different detection times Approach: Concurrent detection task Play Memory game Detect notification signals Study 1

Goal:

Establish a set of visual notification signals with significantly different detection times

Approach: Concurrent detection task

Play Memory game

Detect notification signals

Notification Signals 10 signals : Transformations applied to a base icon Properties explored Colour Motion Location

10 signals :

Transformations applied to a base icon

Properties explored

Colour

Motion

Location

Signals Time (seconds)

Study 1 Design Within-subjects 12 subjects 10 signals 120 trials per subject

Within-subjects

12 subjects

10 signals

120 trials per subject

Study 1 Results: Detection Times High AD signal: Follow Med AD signal: Slow Zoom Low AD signal: Flag

Goal: Investigate the effects of matching attentional draw and utility In terms of: Annoyance Perceived benefit Performance Study 2

Goal:

Investigate the effects of matching attentional draw and utility

In terms of:

Annoyance

Perceived benefit

Performance

Interrupting Task Context-sensitive hints 3 hint utility levels: very helpful somewhat helpful not helpful Emulated a mixed-initiative system

Context-sensitive hints

3 hint utility levels:

very helpful

somewhat helpful

not helpful

Emulated a mixed-initiative system

3 Game Conditions Match Match AD to utility Static Same AD for all utilities Control No interruption Low utility hint ↔ Low AD signal Med utility hint ↔ Med AD signal High utility hint ↔ High AD signal   Low utility hint Med utility hint ↔ Med AD signal High utility hint

Match

Match AD to utility

Static

Same AD for all

utilities

Control

No interruption

Study 2 Design Conditions: Match, Static, Control Within-subjects design 24 subjects 15 interruptions per condition

Conditions: Match, Static, Control

Within-subjects design

24 subjects

15 interruptions per condition

Study 2 Measures & Procedure Procedure: Training, {Condition, Survey} x 3, Interview Measures: Annoyance Perceived benefit Fatigue Performance (# of matches) Preference

Procedure:

Training, {Condition, Survey} x 3, Interview

Measures:

Annoyance

Perceived benefit

Fatigue

Performance (# of matches)

Preference

Study 2 Hypotheses H1 : Interruption annoyance is lower in the Match condition than in the Static condition H2 : Perceived benefit is higher in the Match condition than in the Static condition H3 : Performance is higher in the Match condition than in all other conditions

H1 : Interruption annoyance is lower in the Match

condition than in the Static condition

H2 : Perceived benefit is higher in the Match

condition than in the Static condition

H3 : Performance is higher in the Match condition

than in all other conditions

Results: Annoyance & Benefit Annoyance lower in Match ( p = .034) Benefit higher in Match ( p = .037)

Annoyance lower in Match ( p = .034)

Benefit higher in Match ( p = .037)

Condition Preference

Performance Results (# of matches) No main effect of condition ( p = .454) Other effects present Condition x Presentation, Fatigue, Learning, Performance results unclear 

No main effect of condition ( p = .454)

Other effects present

Condition x Presentation, Fatigue, Learning,

Performance results unclear

Study 2: Summary of Results H1 : Interruption annoyance is lower in the Match condition than in the Static condition H2 : Perceived benefit is higher in the Match condition than in the Static condition H3 : Performance is higher in the Match condition than in all other conditions   

H1 : Interruption annoyance is lower in

the Match condition than in the Static condition

H2 : Perceived benefit is higher in the

Match condition than in the Static condition

H3 : Performance is higher in the Match condition

than in all other conditions

Future Research Computational appraisal of utility Notification signals Scope of utility Primary task relevance: Re-examine performance results Personal relevance: Confirm annoyance and benefit results

Computational appraisal of utility

Notification signals

Scope of utility

Primary task relevance:

Re-examine performance results

Personal relevance:

Confirm annoyance and benefit results

Take Home Message Matching AD and utility Reduces annoyance and Increases perception of benefit UI design implications?

Matching AD and utility

Reduces annoyance and

Increases perception of benefit

UI design implications?

Acknowledgements Many thanks to: Natural Science and Engineering Research Council of Canada (NSERC) Lyn Bartram

Many thanks to:

Natural Science and Engineering Research Council of Canada (NSERC)

Lyn Bartram

Matching AD and utility Reduces annoyance and Increases perception of benefit Take Home Message Questions?

Matching AD and utility

Reduces annoyance and

Increases perception of benefit

Additional Information

Interview Results Comprehension of AD-utility relationship Low 70% Med 40% High 45% All 40% None 25% 40% utilized this knowledge to ignore low-utility hints

Comprehension of AD-utility relationship

Low 70%

Med 40%

High 45%

All 40%

None 25%

40% utilized this knowledge to ignore low-utility hints

Related Work Coordination McFarlane 2002 Timing of Onset Fogarty et al. 2004 Bailey et al. 2001 Peripheral Awareness Maglio & Campbell 2000 McCrickard et al. 2003 Notification Bartram, Ware, Calvert 2003 Horvitz, Kadie, Paek, Hovel 2003 Relevance Czerwinski, Cutrell, Horvitz 2000 Utility Horvitz, Koch, Apacible 2004 Horvitz, Jacobs, Hovel 1999 Avrahami & Hudson 2005 Gievska & Sibert 2005

Coordination

McFarlane 2002

Timing of Onset

Fogarty et al. 2004

Bailey et al. 2001

Peripheral Awareness

Maglio & Campbell 2000

McCrickard et al. 2003

Notification

Bartram, Ware, Calvert 2003

Horvitz, Kadie, Paek, Hovel 2003

Relevance

Czerwinski, Cutrell, Horvitz 2000

Utility

Horvitz, Koch, Apacible 2004

Horvitz, Jacobs, Hovel 1999

Avrahami & Hudson 2005

Gievska & Sibert 2005

Study 1: Simple Editor Task

Study 1 Design Within-subjects: 2 (task) x 10 (signal) x 3 (block) Block Design 2 replications per block 3 blocks per primary task (6 total) 120 trials per subject Measures Detection time 12 subjects, 90 minute experiment

Within-subjects: 2 (task) x 10 (signal) x 3 (block)

Block Design

2 replications per block

3 blocks per primary task (6 total)

120 trials per subject

Measures

Detection time

12 subjects, 90 minute experiment

Study 1: Structure of a Trial

Study 1 Results Significant main effect of signal ( p < .001) 3-way significant comparison between Flag, Slow Zoom, and Follow

Significant main effect of signal ( p < .001)

3-way significant comparison between Flag, Slow Zoom, and Follow

Study 1: Pairwise Comparisons

Study 1: Timeouts Main effect of signal (F(2.542,27.961) = 3.630, p = .031, η 2 = .248)

Study 2 Design Conditions: Match, Static, Control Within-subjects design 5 replications in 17-min block (i.e. 15 interruptions per condition) 65-second average interruption frequency 24 subjects 1 hour experiment

Conditions: Match, Static, Control

Within-subjects design

5 replications in 17-min block (i.e. 15 interruptions per condition)

65-second average interruption frequency

24 subjects

1 hour experiment

Study 2: Interruption Structure

Missed-Hints Dialog

Study 2 General Annoyance No main effect of condition ( F (2,24) = 2.788, p = .079, η 2 = .166, power = .429)

Study 2: Presentation Order Effect F (5,14) = 2.720, p = .064, η 2 = .493

Study 2 Interaction Effect F (10,28) = 2..35, p = .068, η 2 = .421

Study 2 Hint Duration Main effect of utility ( F (2,36) = 6.839, p = .003, η 2 = .275)

Study 2 Workload .143 .115 2.335 Physical Demand .027 .687 .381 Frustration .088 .276 1.347 Perceived Performance .008 .889 .118 Effort .071 .357 1.069 Temporal Demand .004 .945 .057 Mental Demand η 2 p F(2,28) NASA-TLX Factor

NASA Task Load Index Mental Demand How much mental and perceptual activity was required to play the game and attend to the hints? (e.g., thinking, remembering, looking, searching, deciding, etc.)? Physical Demand How much physical activity was required to play the game? (e.g. moving the mouse, clicking the mouse button, etc.) Temporal Demand How much time pressure did you feel due to the rate or pace at which the tasks or task elements occurred?

Mental Demand

How much mental and perceptual activity was required to play the game and attend to the hints? (e.g., thinking, remembering, looking, searching, deciding, etc.)?

Physical Demand

How much physical activity was required to play the game? (e.g. moving the mouse, clicking the mouse button, etc.)

Temporal Demand

How much time pressure did you feel due to the rate or pace at which the tasks or task elements occurred?

NASA Task Load Index (2) Effort How hard did you have to work (mentally and physically) to accomplish your level of performance? Perceived Performance How successful do you think you were in accomplishing the goals of the task set by the experimenter (or yourself)? Frustration How insecure, discouraged, irritated, stressed and annoyed versus secure, gratified, content, relaxed and complacent did you feel during the task?

Effort

How hard did you have to work (mentally and physically) to accomplish your level of performance?

Perceived Performance

How successful do you think you were in accomplishing the goals of the task set by the experimenter (or yourself)?

Frustration

How insecure, discouraged, irritated, stressed and annoyed versus secure, gratified, content, relaxed and complacent did you feel during the task?

Performance Results No main effect of condition ( p = .454) Main effect: presentation order ( p = .064, η 2 = .493) Interaction effect: condition & presentation order ( p = .068, η 2 = .421) Learning effect ( p = .053) Fatigue effect ( p = .009) Impossible to interpret performance results

No main effect of condition ( p = .454)

Main effect: presentation order ( p = .064, η 2 = .493)

Interaction effect: condition & presentation order ( p = .068, η 2 = .421)

Learning effect ( p = .053)

Fatigue effect ( p = .009)

Impossible to interpret performance results