Web site evaluation methodologies and validation engines take the view that all accessibility guidelines must be met to gain compliance. Problems exist in this regard as contradictions within the rule set may arise, and the type of impairment or its severity is not isolated. The Barrier Walkthrough (BW) method goes someway to addressing these issues by enabling barrier types derived from guidelines to be applied to different user categories such as motor or hearing impairment, etc. In this paper, we use set theory to create a validation scheme for older users by combining barrier types specific to motor impaired and low vision users, thereby creating a new ``older users'' category from the results of this set addition. To evaluate this approach, we have conducted a BW study with four pages, 19 expert and 48 non-expert judges. This study shows that the BW generates reliable data for the proposed aggregated user category and shows how experts and non-experts evaluate pages differently. The study also highlights a limitation of the BW by showing that a better aggregated user category would have been created by having a severity level of disability for different impairment types. By extending the BW with these impairment levels, we argue that the BW would become more useful for validating Web pages when dealing with users which multiple disabilities and thus we would be able to create a ``Personalised Validation and Repair'' method.

1. The problem Barrier walkthrough Experimental plan Results Discussion and conclusions
Guideline Aggregation: Web Accessibility
Evaluation for Older Users
Giorgio Brajnik (1), Yeliz Yesilada (2), Simon Harper (2)
(1) Dip. di Matematica e Informatica
University of Udine, Italy
www.dimi.uniud.it/giorgio
(2)School of Computer Science
University of Manchester
Manchester, UK
W4A 2009
c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users

2. The problem Barrier walkthrough Experimental plan Results Discussion and conclusions
The problem with analytic evaluation methods
conformance reviews (eg. wrt WCAG20) are
non-contextualized, not specific
evaluators are not guided into assessing consequences of
violations
there’s no reliable way to rate severity of violations
Our approach
1. Provide context to evaluators: focus on specific barriers
and user categories (eg. blind, motor impaired, cognitively
impaired, low vision, ...)
2. Provide more formalized ways to rate severity
c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users

3. The problem Barrier walkthrough Experimental plan Results Discussion and conclusions
Multiple impairments
How to cope with multiple impairments and combinatorial
explosion?
eg. older people
Dynamic Aggregation:
1. do the evaluation for primitive categories
2. and then aggregate
3. eg. barriers for older people = barriers for low vision
∪ those for motor impaired ∪ ...
c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users

4. The problem Barrier walkthrough Experimental plan Results Discussion and conclusions
Barrier Walkthrough
1. Analytic method; similar to quot;heuristic walkthroughquot;
2. Based on barriers (ako quot;vulnerability pointsquot;)
3. Failure modes are contextualized within usage scenarios
4. This helps evaluators in rating severity = F(impact,
persistence) in {1,2,3}
5. See http://www.dimi.uniud.it/giorgio/
projects/bw/bw.html
(Brajnik, ICCHP 2006; ASSETS 2007)
c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users

5. The problem Barrier walkthrough Experimental plan Results Discussion and conclusions
Example of a barrier
Rich images lacking equivalent text
Users: Blind persons using a screen reader
Cause: The page contains some image that provides
information (e.g. a diagram, histogram, picture, drawing,
graph) but only in a graphical format; no equivalent textual
description appears in the page.
Failure mode: The user, even if s/he perceives that there
is an important image, has no way to get the information it
contains. In addition s/he spends time and effort trying to
find out where in the page or site that information is buried.
c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users

6. The problem Barrier walkthrough Experimental plan Results Discussion and conclusions
Experiment
Goal
To explore which conclusions are invariant wrt aggregation.
Do certain differences among sites disappear?
How does reliability change?
How does correctness of evaluations change?
How does the difference b/w expert/non-expert change?
c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users

7. The problem Barrier walkthrough Experimental plan Results Discussion and conclusions
Plan
Mixed design experiment
19 experts + 51 non-experts applying BW; 61 barrier types
(within-subj)
2 primitive user categories: low vision, motor impaired
(within-subj)
1 aggregated category: older adults = union of individual
barriers found for primitive categories
4 pages (1 page/subject, between-subj): IMDB.com,
Facebook.com, novascotiaquilts.com, Sam’s Chop House
c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users

8. The problem Barrier walkthrough Experimental plan Results Discussion and conclusions
Spreadsheet
c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users

9. The problem Barrier walkthrough Experimental plan Results Discussion and conclusions
True Barriers Types
Correct ratings
those where the majority of experts agreed on their severity
Results:
Experts: 27 out of 61 barrier types (quot;ambiguous linksquot;,
quot;functional images w/o textquot;, quot;inflexible layoutquot;, quot;missing
internal linksquot;, ...)
Non-experts: 24 out of those 27 (missed: quot;forms w/o
labelsquot;, quot;moving contentquot;, quot;no css supportquot;)
Certain barriers are specific for specific user categories
c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users

10. The problem Barrier walkthrough Experimental plan Results Discussion and conclusions
Reliability
Reproducibility
given (barrier type, user group, page)
rep = 1 − sd if positive; 1 if M = 0; 0 otherwise
M
where M, sd are mean/std.dev of weighted severity
Agreement
given (user group, page)
on all barrier types compute the ICC (Intraclass Correlation
Coefficient – relative and absolute consistency)
c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users

11. The problem Barrier walkthrough Experimental plan Results Discussion and conclusions
Reproducibility
c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users

12. The problem Barrier walkthrough Experimental plan Results Discussion and conclusions
Reproducibility
c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users

13. The problem Barrier walkthrough Experimental plan Results Discussion and conclusions
Mean weighted severities
c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users

14. The problem Barrier walkthrough Experimental plan Results Discussion and conclusions
Correctness
Ratings:
I
Error rate E = C+I
Accuracy = % of
reported barriers that
are correct
Sensitivity = % of
correct barriers that are
reported
2A·S
F.measure = A+S
c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users

15. The problem Barrier walkthrough Experimental plan Results Discussion and conclusions
Error rates
c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users

16. The problem Barrier walkthrough Experimental plan Results Discussion and conclusions
F-measure
c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users

17. The problem Barrier walkthrough Experimental plan Results Discussion and conclusions
Invariant properties
1. Aggregation does not worsen the problem of missed
barriers
2. Reliability: experts are consistently more reliable; same
pattern across pages
3. Severities: experts are more judgmental; ranks of pages
do not change
4. Quality: error rates maintain a similar difference (expert vs
non-experts)
5. Quality: F-measure conf. intervals shrink; they keep same
relationship
c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users

18. The problem Barrier walkthrough Experimental plan Results Discussion and conclusions
Conclusions
1. Aggregation seems to work: it enables contextualized
evaluations and leads to results that are potentially valid
2. It could be extended to cope with degrees of impairment
Limitations
1. We did not validate our conclusions against an
independent assessment
2. We don’t know if the same conclusions would hold for any
set of primitive user categories
Questions?
c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users

19. The problem Barrier walkthrough Experimental plan Results Discussion and conclusions
Evaluation framework
based on reliability (reproducibility + agreement),
correctness (error rate, accuracy, sensitivity and
F-measure)
is viable
is discriminatory
It can be used to assess pros and cons of an evaluation
method.
c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users