Guideline Aggregation: Web Accessibility

The problem Barrier walkthrough Experimental plan Results Discussion and conclusions Guideline Aggregation: Web Accessibility Evaluation for Older Users Giorgio Brajnik (1), Yeliz Yesilada (2), Simon Harper (2) (1) Dip. di Matematica e Informatica University of Udine, Italy www.dimi.uniud.it/giorgio (2)School of Computer Science University of Manchester Manchester, UK W4A 2009 c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users

The problem Barrier walkthrough Experimental plan Results Discussion and conclusions The problem with analytic evaluation methods conformance reviews (eg. wrt WCAG20) are non-contextualized, not speciﬁc evaluators are not guided into assessing consequences of violations there’s no reliable way to rate severity of violations Our approach 1. Provide context to evaluators: focus on speciﬁc barriers and user categories (eg. blind, motor impaired, cognitively impaired, low vision, ...) 2. Provide more formalized ways to rate severity c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users

The problem Barrier walkthrough Experimental plan Results Discussion and conclusions Multiple impairments How to cope with multiple impairments and combinatorial explosion? eg. older people Dynamic Aggregation: 1. do the evaluation for primitive categories 2. and then aggregate 3. eg. barriers for older people = barriers for low vision ∪ those for motor impaired ∪ ... c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users

The problem Barrier walkthrough Experimental plan Results Discussion and conclusions Barrier Walkthrough 1. Analytic method; similar to \"heuristic walkthrough\" 2. Based on barriers (ako \"vulnerability points\") 3. Failure modes are contextualized within usage scenarios 4. This helps evaluators in rating severity = F(impact, persistence) in {1,2,3} 5. See http://www.dimi.uniud.it/giorgio/ projects/bw/bw.html (Brajnik, ICCHP 2006; ASSETS 2007) c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users

The problem Barrier walkthrough Experimental plan Results Discussion and conclusions Example of a barrier Rich images lacking equivalent text Users: Blind persons using a screen reader Cause: The page contains some image that provides information (e.g. a diagram, histogram, picture, drawing, graph) but only in a graphical format; no equivalent textual description appears in the page. Failure mode: The user, even if s/he perceives that there is an important image, has no way to get the information it contains. In addition s/he spends time and effort trying to ﬁnd out where in the page or site that information is buried. c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users

The problem Barrier walkthrough Experimental plan Results Discussion and conclusions Experiment Goal To explore which conclusions are invariant wrt aggregation. Do certain differences among sites disappear? How does reliability change? How does correctness of evaluations change? How does the difference b/w expert/non-expert change? c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users

The problem Barrier walkthrough Experimental plan Results Discussion and conclusions Plan Mixed design experiment 19 experts + 51 non-experts applying BW; 61 barrier types (within-subj) 2 primitive user categories: low vision, motor impaired (within-subj) 1 aggregated category: older adults = union of individual barriers found for primitive categories 4 pages (1 page/subject, between-subj): IMDB.com, Facebook.com, novascotiaquilts.com, Sam’s Chop House c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users

The problem Barrier walkthrough Experimental plan Results Discussion and conclusions Spreadsheet c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users

The problem Barrier walkthrough Experimental plan Results Discussion and conclusions True Barriers Types Correct ratings those where the majority of experts agreed on their severity Results: Experts: 27 out of 61 barrier types (\"ambiguous links\", \"functional images w/o text\", \"inflexible layout\", \"missing internal links\", ...) Non-experts: 24 out of those 27 (missed: \"forms w/o labels\", \"moving content\", \"no css support\") Certain barriers are specific for specific user categories c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users

The problem Barrier walkthrough Experimental plan Results Discussion and conclusions Reliability Reproducibility given (barrier type, user group, page) rep = 1 − sd if positive; 1 if M = 0; 0 otherwise M where M, sd are mean/std.dev of weighted severity Agreement given (user group, page) on all barrier types compute the ICC (Intraclass Correlation Coefﬁcient – relative and absolute consistency) c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users

The problem Barrier walkthrough Experimental plan Results Discussion and conclusions Reproducibility c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users

The problem Barrier walkthrough Experimental plan Results Discussion and conclusions Mean weighted severities c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users

The problem Barrier walkthrough Experimental plan Results Discussion and conclusions Correctness Ratings: I Error rate E = C+I Accuracy = % of reported barriers that are correct Sensitivity = % of correct barriers that are reported 2A·S F.measure = A+S c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users

The problem Barrier walkthrough Experimental plan Results Discussion and conclusions Error rates c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users

The problem Barrier walkthrough Experimental plan Results Discussion and conclusions F-measure c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users

The problem Barrier walkthrough Experimental plan Results Discussion and conclusions Invariant properties 1. Aggregation does not worsen the problem of missed barriers 2. Reliability: experts are consistently more reliable; same pattern across pages 3. Severities: experts are more judgmental; ranks of pages do not change 4. Quality: error rates maintain a similar difference (expert vs non-experts) 5. Quality: F-measure conf. intervals shrink; they keep same relationship c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users

The problem Barrier walkthrough Experimental plan Results Discussion and conclusions Conclusions 1. Aggregation seems to work: it enables contextualized evaluations and leads to results that are potentially valid 2. It could be extended to cope with degrees of impairment Limitations 1. We did not validate our conclusions against an independent assessment 2. We don’t know if the same conclusions would hold for any set of primitive user categories Questions? c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users

The problem Barrier walkthrough Experimental plan Results Discussion and conclusions Evaluation framework based on reliability (reproducibility + agreement), correctness (error rate, accuracy, sensitivity and F-measure) is viable is discriminatory It can be used to assess pros and cons of an evaluation method. c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users

Guideline Aggregation: Web Accessibility

0 comments

Notes on slide 1

1 Group & 1 Event