Biases in Human Estimation of Interruptibility: Effects and Implications for Practice Daniel Avrahami, James Fogarty & Scott Hudson

Introduction Estimating someone else’s interruptibility is something we do every day At home, at work, at the store …but it’s also something we’re not always good at This becomes much harder when done over a distance Let’s try this together:

Estimating someone else’s interruptibility is something we do every day

At home, at work, at the store

…but it’s also something we’re not always good at

This becomes much harder when done over a distance

Let’s try this together:

Estimating Interruptibility 1 2 3 4 5 Highly Interruptible Highly Non-Interruptible 30 seconds

1 2 3 4 5

Highly Interruptible Highly Non-Interruptible

Estimating Interruptibility 1 2 3 4 5 Highly Interruptible Highly Non-Interruptible

1 2 3 4 5

Highly Interruptible Highly Non-Interruptible

Prompting for Self-Report 1 2 3 4 5 Highly Interruptible Highly Non-Interruptible

1 2 3 4 5

Highly Interruptible Highly Non-Interruptible

Prompting for Self-Report 1 2 3 4 5 Highly Interruptible Highly Non-Interruptible

1 2 3 4 5

Highly Interruptible Highly Non-Interruptible

And the answer is… 1 2 3 4 5 Highly Interruptible Highly Non-Interruptible

1 2 3 4 5

Highly Interruptible Highly Non-Interruptible

And the answer is… 1 2 3 4 5 Highly Interruptible Highly Non-Interruptible 3

1 2 3 4 5

Highly Interruptible Highly Non-Interruptible

Goals A better understanding on the biases in estimating others’ interruptibility can inform the design of CMC and awareness systems Provide an insight into how people are likely to use different pieces of contextual information For example, the state of an office door was significantly correlated with errors in estimation A lot of previous work on CMC and awareness systems [cf. Fish’90, Mantei’91,Dourish’92, Bly’93, Adler’94, Tang’01]

A better understanding on the biases in estimating others’ interruptibility can inform the design of CMC and awareness systems

Provide an insight into how people are likely to use different pieces of contextual information

For example, the state of an office door was significantly correlated with errors in estimation

A lot of previous work on CMC and awareness systems [cf. Fish’90, Mantei’91,Dourish’92, Bly’93, Adler’94, Tang’01]

Research Questions Which contextual cues (e.g., working on the computer) affect the error in human estimation of another person’s interruptibility? What is the source of a contextual cue’s effect on the bias in estimation? e.g., overrating the strength of a cue

Which contextual cues (e.g., working on the computer) affect the error in human estimation of another person’s interruptibility?

What is the source of a contextual cue’s effect on the bias in estimation?

e.g., overrating the strength of a cue

Talk Outline Study Design Measures and Analysis Results Conclusions and Future work Details (f-measures, etc.)

Study Design

Measures and Analysis

Results

Conclusions and Future work

Study Design

Study Design Compare Self-Reports of Interruptibility ( Reported ) with Estimations of Interruptibility ( Estimated ) Two groups of participants: Reporters (in their natural work environment) Estimators (viewing video and audio recordings)

Compare Self-Reports of Interruptibility ( Reported ) with Estimations of Interruptibility ( Estimated )

Two groups of participants:

Reporters (in their natural work environment)

Estimators (viewing video and audio recordings)

Reporters Four high-level staff members (3 females, 1 male) Audio and video recordings in their offices during a month-long period Experience-sampling method used to collect self-reports of interruptibility at random intervals (30-minutes average) 672 self-reports and over 600 hours of data [Used in Hudson’03, Fogarty’05 for the creation of predictive models]

Four high-level staff members (3 females, 1 male)

Audio and video recordings in their offices during a month-long period

Experience-sampling method used to collect self-reports of interruptibility at random intervals

(30-minutes average)

672 self-reports and over 600 hours of data

[Used in Hudson’03, Fogarty’05 for the creation of predictive models]

Estimators 40 subjects (Online recruitment, majority were students) Watched 15- or 30-second clips of reporters Ensured that did not know the Reporters Provided estimates of interruptibility 60 clips each Allowed to watch each clip as many times as wanted Task lasted about 1 hour [Used in Fogarty’05 to compare performance of estimators vs. ML]

40 subjects (Online recruitment, majority were students)

Watched 15- or 30-second clips of reporters

Ensured that did not know the Reporters

Provided estimates of interruptibility

60 clips each

Allowed to watch each clip as many times as wanted

Task lasted about 1 hour

[Used in Fogarty’05 to compare performance of estimators vs. ML]

Estimators 40 subjects (Online recruitment, majority were students) Watched 15- or 30-second clips of reporters Ensured that did not know the Reporters Provided estimates of interruptibility 60 clips each Allowed to watch each clip as many times as wanted Task lasted about 1 hour [Used in Fogarty’05 to compare performance of estimators vs. ML]

40 subjects (Online recruitment, majority were students)

Watched 15- or 30-second clips of reporters

Ensured that did not know the Reporters

Provided estimates of interruptibility

60 clips each

Allowed to watch each clip as many times as wanted

Task lasted about 1 hour

[Used in Fogarty’05 to compare performance of estimators vs. ML]

Estimations vs. Self-Reports Tested the relationship between Estimations and Reports: Estimated Interruptibility was significantly correlated with Reported Interruptibility (p<.001) (This is good). Means that Estimators examined the situation presented to them Estimated Interruptibility was significantly different from Reported Interruptibility (p<.001) Estimators, on average, estimated Reporters to be more interruptible than reported (Two outliers’ data excluded from the remaining analyses)

Tested the relationship between Estimations and Reports:

Estimated Interruptibility was significantly correlated with Reported Interruptibility (p<.001)

(This is good). Means that Estimators examined the situation presented to them

Estimated Interruptibility was significantly different from Reported Interruptibility (p<.001)

Estimators, on average, estimated Reporters to be more interruptible than reported

(Two outliers’ data excluded from the remaining analyses)

Measures and Analysis

Contextual Cues Coded by six paid coders For each 15 seconds segment, coded for a large set of contextual cues that could be coded reliably Reporters activities Guest activities Environmental cues Inter-coder agreement was 93.4% (evaluated on 5% of the data)

Coded by six paid coders

For each 15 seconds segment, coded for a large set of contextual cues that could be coded reliably

Reporters activities

Guest activities

Environmental cues

Inter-coder agreement was 93.4% (evaluated on 5% of the data)

Contextual Cues (cont.) Phone Social Engagement Computer Desk Papers File Cabinet Food Writing Door is Closed Drink Standing Present

“ Estimation Error” Estimation Error = Reported – Estimated Reported 4 3 1 2 5 2 Estimated 1 3 4 5

Estimation Error = Reported – Estimated

“ Estimation Error” Estimation Error = Reported – Estimated 4 3 3 2 2 1 1 1 1 2 -1 -1 -1 -1 -2 -2 -2 -3 -3 -4 0 0 0 0 0 Reported 4 3 1 2 5 2 Estimated 1 3 4 5

Estimation Error = Reported – Estimated

“ Estimation Error” Estimation Error = Reported – Estimated Under-estimation Over-estimation Reported 4 3 1 2 5 -1 -1 -1 -1 -2 -2 -2 -3 -3 -4 0 0 0 0 0 4 3 3 2 2 1 1 1 1 2 0 0 0 0 0 2 Estimated 1 3 4 5

Estimation Error = Reported – Estimated

Analysis Approach Step 1: Find which cues have an effect on Estimation Error Effect on Under-Estimation errors? Effect on Over-Estimation errors? Step 2: Investigate the cause for a cue’s effect on Estimation Error Effect on Reported Interruptibility? Effect on Estimated Interruptibility? „ ‚  

Step 1: Find which cues have an effect on Estimation Error

Effect on Under-Estimation errors?

Effect on Over-Estimation errors?

Step 2: Investigate the cause for a cue’s effect on Estimation Error

Effect on Reported Interruptibility?

Effect on Estimated Interruptibility?

Analysis Approach Step 1: Find which cues have an effect on Estimation Error Effect on Under-Estimation errors? Effect on Over-Estimation errors? Step 2: Investigate the cause for a cue’s effect on Estimation Error Effect on Reported Interruptibility? Effect on Estimated Interruptibility? „ ‚   For example: Found that the Reporter using the had a Significant effect on Over-Estimation (no significant effect on Under-Estimation) Significant effect on Estimated Interruptibility, but… no significant effect on Reported Interruptibility “ Considering a cue that is not significant” ‚   

Step 1: Find which cues have an effect on Estimation Error

Effect on Under-Estimation errors?

Effect on Over-Estimation errors?

Step 2: Investigate the cause for a cue’s effect on Estimation Error

Effect on Reported Interruptibility?

Effect on Estimated Interruptibility?

For example:

Found that the Reporter using the had a

Significant effect on Over-Estimation (no significant effect on Under-Estimation)

Significant effect on Estimated Interruptibility, but… no significant effect on Reported Interruptibility

“ Considering a cue that is not significant”

A couple of notes on the analysis A self-report determines the possible range of Estimation Errors Reported = 1, Error can be -4 … 0 Reported = 5, Error can be 0 … 4 => Need to include the Reported Interruptibility as a control measure in the analysis of error All done using Mixed Model analysis

A self-report determines the possible range of Estimation Errors

Reported = 1, Error can be -4 … 0

Reported = 5, Error can be 0 … 4

=> Need to include the Reported Interruptibility as a control measure in the analysis of error

All done using Mixed Model analysis

Results

Under-estimated when the reporter was socially engaged Over-estimated when the reporter wasn’t socially engaged Social Engagement „ ‚   “ Overrating the strength of a cue ”

Under-estimated when the reporter was socially engaged

Over-estimated when the reporter wasn’t socially engaged

Over-estimated reporter’s interruptibility when wasn’t using the phone Phone   ‚  “ Overrating the strength of a cue ”

Over-estimated reporter’s interruptibility when wasn’t using the phone

Greater over-estimation error when the reporter was standing Reporter Standing significantly correlated with situation more interruptible (both R,E) Link between physical transitions and interruptions [Ho’05] Reporter is Standing  ‚   “ Overrating the strength of a cue ”

Greater over-estimation error when the reporter was standing

Reporter Standing significantly correlated with situation more interruptible (both R,E)

Link between physical transitions and interruptions [Ho’05]

Computer Estimators more likely to interpret a situation as more interruptible than reported when the Reporter was using the computer Link to issues of online presence and availability  ‚   “ Considering a cue that is not significant ”

Estimators more likely to interpret a situation as more interruptible than reported when the Reporter was using the computer

Link to issues of online presence and availability

Under-estimated when the door was closed Correlation between the state of the door and Reported interruptibility was not significant Door is Closed „ ‚   “ Considering a cue that is not significant ”

Under-estimated when the door was closed

Correlation between the state of the door and Reported interruptibility was not significant

Estimators assessing Reporters as more interruptible when they were drinking Correlation between drinking and Reported interruptibility was not significant Drink  ‚   “ Considering a cue that is not significant ”

Estimators assessing Reporters as more interruptible when they were drinking

Correlation between drinking and Reported interruptibility was not significant

Conclusions and Future Work

Conclusions Presented results from an in-depth analysis of causes for biases in human estimation of interruptibility Compared self-reports, collected in the field, and estimations based on audio and video recordings

Presented results from an in-depth analysis of causes for biases in human estimation of interruptibility

Compared self-reports, collected in the field, and estimations based on audio and video recordings

Conclusions (cont.) Findings suggest that providing too much information may not only be a concern for privacy, but may lead to errors in estimations Sharing certain contextual cues will likely result in misinterpretations of a person’s interruptibility A new system, informed by our results, could Avoid exposing certain cues (or specific levels of cues) Enhance (or moderate) others

Findings suggest that providing too much information may not only be a concern for privacy, but may lead to errors in estimations

Sharing certain contextual cues will likely result in misinterpretations of a person’s interruptibility

A new system, informed by our results, could

Avoid exposing certain cues (or specific levels of cues)

Enhance (or moderate) others

Future Work Examine the effect of other clip-durations Examine the effect of degree of familiarity between reporter and estimator on estimation errors Observe reporters in other settings and jobs Investigate the use of these findings for effective creation of computer-supported communication and awareness systems

Examine the effect of other clip-durations

Examine the effect of degree of familiarity between reporter and estimator on estimation errors

Observe reporters in other settings and jobs

Investigate the use of these findings for effective creation of computer-supported communication and awareness systems

Acknowledgements Yaakov Kareev Darren Gergle Laura Dabbish Joonhwan Lee

Yaakov Kareev

Darren Gergle

Laura Dabbish

Joonhwan Lee

Thank you This work was funded in part by NSF Grants IIS-0121560, IIS-0325351, and by DARPA Contract No. NBCHD030010 for more info visit: www.cs.cmu.edu/~nx6 or email: [email_address] [email_address] [email_address]

FAQ 2PT Y-SPRTE LEN1530

2PT

Y-SPRTE

LEN1530

Why separate over and under? Shouldn’t just use absolute or squared error because over and under estimation will make it seem like there is no effect Shouldn’t just put all together because errors will cancel each other out back

Shouldn’t just use absolute or squared error because over and under estimation will make it seem like there is no effect

Shouldn’t just put all together because errors will cancel each other out

Is the length of the clips reasonable? The information available to Estimators in this study (15/30 second video+audio clips) was similar to information available to users of media space systems, and far richer than information available in most awareness systems back

The information available to Estimators in this study (15/30 second video+audio clips) was similar to information available to users of media space systems, and far richer than information available in most awareness systems

Why not use a 2-point scale? With 2 levels, Estimation is either 100% correct, or 100% incorrect With 5-levels, we get degrees of error Doesn’t make sense asking Reporters and Estimators to make a binary decision when Don’t know what interruption is about Don’t know whom the interruption is from Couldn’t analyze 5-scale data as 2-points If Reported =1 and Estimated =4, should we count as correct?? back

With 2 levels, Estimation is either 100% correct, or 100% incorrect

With 5-levels, we get degrees of error

Doesn’t make sense asking Reporters and Estimators to make a binary decision when

Don’t know what interruption is about

Don’t know whom the interruption is from

Couldn’t analyze 5-scale data as 2-points

If Reported =1 and Estimated =4, should we count as correct??