These slides depict the work carried out on the COPE project from 2011 to 2013. We propose a method to identify problems and coping tactics in populations that are constrained. Then we generalise the problems encountered and the strategies employed to broader audiences. We conclude that there is an overlap between the problems and strategies of blind and sighted users. This provides more support to the hypothesis of the accessibility continuum.

1. Detecting Problematic
Interactions on the Web
{
The COPE Project: Coping strategies on
the Web
markel.vigo@{ehu.es | manchester.ac.uk}
@markelvigo
Donostia, 18/11/2013

2. Motivation
The Web is a complex
environment

3. Motivation
Ceaparu,I., Lazar,J., Bessiere,K., Robinson,J., and Shneiderman,B. (2004) Determining causes and severity
of end-user frustration. International Journal of Human-Computer Interaction 17(3), 333–356.

4. Problems
Design
 Unfamiliar conventions
 Lack of skills


5. Detecting problems

Post-hoc analysis: 87% detection
accuracy (Feild, 2010)
- Self-reported frustration
- Usage metrics: query length, task
time.
Feild, H.A., Allan, J. and Jones, R. (2010) Predicting searcher frustration. ACM SIGIR ’10, 34–41.

6. Detecting problems
Task-driven vs. emergent tasks
 Top-down vs. bottom up
 Constrained vs. ecological
 SERP vs. navigation and more


7. Reactions to problems
Reactions are cognitive markers
that indicate problems
 Ask for help (Novick, 2007)
 Other means (Ceaparu, 2004)

-
"I found an alternative solution"
"I figure out how to fix myself"
Novick, D. G., Elizalde, E., and Bean, N. (2007) Toward a more accurate view of when and how people seek help
with computer applications. ACM SIGDOC '07, 95–102.
Ceaparu, I., Lazar, J., Bessiere, K., Robinson, J., and Shneiderman, B. (2004) Determining causes and severity of
end-user frustration. International Journal of Human-Computer Interaction 17(3), 333–356.

8. Identifying strategies
Reactions are cognitive markers
that indicate problems
 If we are able to automatically
detect coping we can provide the
means to overcome the situation


9. Identifying strategies

Challenges
-

Low frequency
Non-overtly manifested
Ecological setting (get out of the lab!)
Consequences
-
-
Significant amount of observations into
the wild
Time and personnel++

10. Hypothesis

We can transfer strategies from
populations who cope more
frequent and overtly to general
audiences

11. Evidence

Overlap of problems between:
-
-

The constrains imposed by reduced
screens and keyboards (situational
impairments)
Problems encountered by motor
impaired desktop users
Same scope, different magnitude
Yesilada,Y., Harper,S., Chen,T., Trewin,S. (2010) Small-device users situationally impaired by input.
Computers in Human Behavior 26 (3), 427–435.
Simon Harper, Yeliz Yesilada, Tianyi Chen (2011) Mobile device impairment...similar problems, similar
solutions? Behaviour & IT 30(5),673-690

12. Hypothesis

There is an accessibility
continuum with overlapping
-
Problems
Strategies
Solutions
SCOPE
lower access
end
people with
disabilities
MAGNITUDE
situational
impairment
able bodied
users
higher access
end

13. Hypothesis
MAGNITUDE
Yesilada et al. (2010)
Harper et al. (2011)
SCOPE
visuallyphysically mobile
disabledimpaired users
able
bodied
users
lower access
end

higher access
end
Our goal is more ambitious: explore the
commonalities of distant populations.
-
Visually disabled users who cope
Able bodied users encountering difficulties

14. Our proposal
4 step method:
1.
2.
3.
4.
Observation and identification of
strategies
Implementation of algorithms to
detect strategies
Deployment into the wild
Run user studies
Go to step 2 until obtaining reliable results
Vigo, M. and Harper, S. (2013) Considering people with disabilities as überusers for eliciting generalisable
coping strategies on the web. ACM Web Science, 441-444.

15. Step 1. Observation
and identification of strategies
2 independent ethnographic
studies and user tests
 24 screen reader and screen
magnifier users
 8 coping strategies (17
implementations) were identified

Vigo, M and Harper, S. (2013) Coping tactics employed by visually disabled users on the web. International
Journal of Human-Computer Studies 71(11), 1013–1025.

16. Step 1. Observation
and identification of strategies
T1. Asking for assistance
T2. Impulsive clicking
-
-
I21. Deliberately clicking on lowscented links
I22. Clicking on any link
I23. Clicking on any link as long as it is
accessible
T3. Exploration tactics
-
-
-
I31. Persevering
I32. Escaping from content by moving
down
I33. Move down without listening to
content
I34. Move around
T5. Gaining orientation
-
I51. Gaining local orientation
I52. Gaining global orientation
T6. Redoing
-
I61. Re-checking
I62. Re-typing
I63. Re-tracing
I64. Re-starting
T7. Not operating
-
-
I71. Delegating on assistive
technology
I72. Swapping assistive technology
I73. Waiting
T8. Giving up
T4. Narrowing down search
Vigo, M and Harper, S. (2013) Coping tactics employed by visually disabled users on the web. International
Journal of Human-Computer Studies 71(11), 1013–1025.

17. Step 1. Further analysis I



We analyse the skill acquisition process of
visually disabled users over time
A decrease in last resort tactics is observed
across sessions
In parallel, users exhibit more exploration tactics
and explore more websites
1.0
2.5
0.8
2.0
tactics
tactics
tactics
ASK
GUP
NOP
0.4
0.2
1.5
EXP
tactics
0.6
NDO
ORI
1.0
RED
0.5
0.0
0.0
1
2
3
sessions
4
5
1
2
3
4
5
sessions
Vigo, M and Harper, S. (under review) A snapshot of the first encounters of visually disabled users with the Web.

18. Step 1. Further analysis I


While effectiveness increases, coping success
does not
Misuse of tactics may be indicative of a larger
repertoire or increased confidence.
1.0
performance
0.8
0.6
Performance
effectiveness
coping success
0.4
0.2
0.0
1
2
3
4
5
sessions
Vigo, M and Harper, S. (under review) A snapshot of the first encounters of visually disabled users with the Web.

19. Step 1. Further analysis II
Users are not always driven by information
scent
 Navigation is driven by the need to
overcome an interaction problem
 Users escape from the current web patch

-
-
Users click on misleading links if by doing this
problems are averted.
Low satisfacing levels: in extraordinary
situations any web patch is “good enough”.
Vigo, M and Harper, S. (2013) Challenging information foraging theory: screen reader users are not always driven
by information scent. ACM Conference on Hypertext and Social Media, 60-68

20. Our proposal
4 step method:
1.
2.
3.
4.
Observation and identification of
strategies ✔
Implementation of algorithms to
detect strategies
Deployment into the wild
Run user studies
Go to step 2 until obtaining reliable results
Vigo, M. and Harper, S. (2013) Considering people with disabilities as überusers for eliciting generalisable
coping strategies on the web. ACM Web Science, 441-444.

21. Step 2. Implementation of
algorithms to detect strategies
I61. Re-checking: quick revisitations
webpagei→wpj→wpiwpj
I63. Re-tracing: users retrace the
steps in a sequence of pages.
wpj→wpk→wpl→wpm→wpl→wpk→wpj→wpk→wpl→wpm

22. Our proposal
4 step method:
1.
2.
3.
4.
Observation and identification of
strategies ✔
Implementation of algorithms to
detect strategies ✔
Deployment into the wild
Run user studies
Go to step 2 until obtaining reliable results
Vigo, M. and Harper, S. (2013) Considering people with disabilities as überusers for eliciting generalisable
coping strategies on the web. ACM Web Science, 441-444.

23. Step 3. Deployment into
the wild
WebTactics
1.1 Detection
t1(){
algorithms are
..
injected onto web }
pages.
ti(){
..
}
tn(){
..
}
2. When the use of a
tactic is detected a
notification is triggered.
3. This event is
asynchronously sent
to a remote location.
4. Reports from users are
viewed by interested parties.
DB
User
Website Manager
/Researcher
1.2 Each algorithm keeps track of determined
sequences of events and actions.
id
timestamp
URL
tactic
ccgu1331569030153
1333922552190
http://www.bbc.co.uk/sport/
t5
ccgu1331569030153
1333922556391
http://www.bbc.co.uk/sport/
t1
Vigo, M. and Harper, S. (2013) Evaluating Accessibility-in-use. W4A '13, article 7.

24. Step 3. Deployment into
the wild
Vigo, M. and Harper, S. (2013) Evaluating Accessibility-in-use. W4A '13, article 7.

25. Our proposal
4 step method:
1.
2.
3.
4.
Observation and identification of
strategies ✔
Implementation of algorithms to
detect strategies ✔
Deployment into the wild ✔
Run user studies
Go to step 2 until obtaining reliable results
Vigo, M. and Harper, S. (2013) Considering people with disabilities as überusers for eliciting generalisable
coping strategies on the web. ACM Web Science, 441-444.

26. Step 4. Run user studies
18 participants, 10 days
 126 retraces and 67 rechecks
 Tabbed browsing was interfering
 Feedback on false positives:

-
“I’m browsing across tabs”
“I’m comparing different web pages”
“I’m navigating through different tabs”
Vigo, M. and Harper, S. (in progress) Detecting navigation tactics to cope on the World 'Wild' Web.
Vigo, M. and Harper, S. (2013) Considering people with disabilities as überusers for eliciting generalisable
coping strategies on the web. ACM Web Science, 441-444.

27. Our proposal
4 step method:
1.
2.
3.
4.
Observation and identification of
strategies ✔
Implementation of algorithms to
detect strategies ✔
Deployment into the wild ✔
Run user studies ✔
Go to step 2 until obtaining reliable results
Vigo, M. and Harper, S. (2013) Considering people with disabilities as überusers for eliciting generalisable
coping strategies on the web. ACM Web Science, 441-444.

28. Step 2. Implementation of
algorithms to detect strategies
I61. Re-checking: quick revisitations
webpagei→wpj→wpiwpj
NONTABBED
NONTABBED
NONTABBED
I63. Re-tracing: users retrace the
steps in a sequence of pages.
wpj→wpk→wpl→wpm→wpl→wpk→wpj→wpk→wpl→wpm
NONTABBED
NONTABBED
NONTABBED
NONNONNONTABBED TABBED TABBED
NONTABBED
NONTABBED
NONTABBED

29. Our proposal
4 step method:
1.
2.
3.
4.
Observation and identification of
strategies ✔
Implementation of algorithms to
detect strategies ✔✔
Deployment into the wild ✔✔
Run user studies ✔
Go to step 2 until obtaining reliable results
Vigo, M. and Harper, S. (2013) Considering people with disabilities as überusers for eliciting generalisable
coping strategies on the web. ACM Web Science, 441-444.

30. Step 4. Run user studies
20 participants, 10 days
 24 retraces and 16 rechecks

1.0
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
study 1
study 2
I63. Retracing
0.0
study 1
study 2
I61. Rechecking
Vigo, M. and Harper, S. (in progress) Detecting navigation tactics to cope on the World 'Wild' Web.
Vigo, M. and Harper, S. (2013) Considering people with disabilities as überusers for eliciting generalisable
coping strategies on the web. ACM Web Science, 441-444.

31. Conclusions
There is an overlap on problems
and strategies
 More evidence supporting the
accessibility continuum
 Implications for accessibility-inuse


32. Future work

Ongoing work
4th iteration with users
- Testing with visually disabled users
 2nd stage of the COPE project
-
-
-
Intervening in the interface
Long term objective
Triangulation

33. Detecting Problematic
Interactions on the Web
{ questions?
markel.vigo@{ehu.es | manchester.ac.uk}
@markelvigo
Donostia, 18/11/2013