Guideline Aggregation: Web Accessibility

1,154 views
1,088 views

Published on

Web site evaluation methodologies and validation engines take the
view that all accessibility guidelines must be met to gain
compliance. Problems exist in this regard as contradictions within
the rule set may arise, and the type of impairment or its severity
is not isolated. The Barrier Walkthrough (BW) method goes someway to
addressing these issues by enabling barrier types derived from
guidelines to be applied to different user categories such as motor
or hearing impairment, etc. In this paper, we use set theory to
create a validation scheme for older users by combining barrier
types specific to motor impaired and low vision users,
thereby creating a new ``older users'' category from the results of
this set addition. To evaluate this approach, we have conducted a BW
study with four pages, 19 expert and 48 non-expert judges. This
study shows that the BW generates reliable data for the proposed
aggregated user category and shows how experts and non-experts
evaluate pages differently. The study also highlights a limitation
of the BW by showing that a better aggregated user category would
have been created by having a severity level of disability for
different impairment types. By extending the BW with these
impairment levels, we argue that the BW would become more useful for
validating Web pages when dealing with users which multiple
disabilities and thus we would be able to create a ``Personalised
Validation and Repair'' method.

Published in: Technology, Design
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,154
On SlideShare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
9
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Guideline Aggregation: Web Accessibility

  1. 1. The problem Barrier walkthrough Experimental plan Results Discussion and conclusions Guideline Aggregation: Web Accessibility Evaluation for Older Users Giorgio Brajnik (1), Yeliz Yesilada (2), Simon Harper (2) (1) Dip. di Matematica e Informatica University of Udine, Italy www.dimi.uniud.it/giorgio (2)School of Computer Science University of Manchester Manchester, UK W4A 2009 c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users
  2. 2. The problem Barrier walkthrough Experimental plan Results Discussion and conclusions The problem with analytic evaluation methods conformance reviews (eg. wrt WCAG20) are non-contextualized, not specific evaluators are not guided into assessing consequences of violations there’s no reliable way to rate severity of violations Our approach 1. Provide context to evaluators: focus on specific barriers and user categories (eg. blind, motor impaired, cognitively impaired, low vision, ...) 2. Provide more formalized ways to rate severity c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users
  3. 3. The problem Barrier walkthrough Experimental plan Results Discussion and conclusions Multiple impairments How to cope with multiple impairments and combinatorial explosion? eg. older people Dynamic Aggregation: 1. do the evaluation for primitive categories 2. and then aggregate 3. eg. barriers for older people = barriers for low vision ∪ those for motor impaired ∪ ... c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users
  4. 4. The problem Barrier walkthrough Experimental plan Results Discussion and conclusions Barrier Walkthrough 1. Analytic method; similar to quot;heuristic walkthroughquot; 2. Based on barriers (ako quot;vulnerability pointsquot;) 3. Failure modes are contextualized within usage scenarios 4. This helps evaluators in rating severity = F(impact, persistence) in {1,2,3} 5. See http://www.dimi.uniud.it/giorgio/ projects/bw/bw.html (Brajnik, ICCHP 2006; ASSETS 2007) c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users
  5. 5. The problem Barrier walkthrough Experimental plan Results Discussion and conclusions Example of a barrier Rich images lacking equivalent text Users: Blind persons using a screen reader Cause: The page contains some image that provides information (e.g. a diagram, histogram, picture, drawing, graph) but only in a graphical format; no equivalent textual description appears in the page. Failure mode: The user, even if s/he perceives that there is an important image, has no way to get the information it contains. In addition s/he spends time and effort trying to find out where in the page or site that information is buried. c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users
  6. 6. The problem Barrier walkthrough Experimental plan Results Discussion and conclusions Experiment Goal To explore which conclusions are invariant wrt aggregation. Do certain differences among sites disappear? How does reliability change? How does correctness of evaluations change? How does the difference b/w expert/non-expert change? c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users
  7. 7. The problem Barrier walkthrough Experimental plan Results Discussion and conclusions Plan Mixed design experiment 19 experts + 51 non-experts applying BW; 61 barrier types (within-subj) 2 primitive user categories: low vision, motor impaired (within-subj) 1 aggregated category: older adults = union of individual barriers found for primitive categories 4 pages (1 page/subject, between-subj): IMDB.com, Facebook.com, novascotiaquilts.com, Sam’s Chop House c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users
  8. 8. The problem Barrier walkthrough Experimental plan Results Discussion and conclusions Spreadsheet c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users
  9. 9. The problem Barrier walkthrough Experimental plan Results Discussion and conclusions True Barriers Types Correct ratings those where the majority of experts agreed on their severity Results: Experts: 27 out of 61 barrier types (quot;ambiguous linksquot;, quot;functional images w/o textquot;, quot;inflexible layoutquot;, quot;missing internal linksquot;, ...) Non-experts: 24 out of those 27 (missed: quot;forms w/o labelsquot;, quot;moving contentquot;, quot;no css supportquot;) Certain barriers are specific for specific user categories c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users
  10. 10. The problem Barrier walkthrough Experimental plan Results Discussion and conclusions Reliability Reproducibility given (barrier type, user group, page) rep = 1 − sd if positive; 1 if M = 0; 0 otherwise M where M, sd are mean/std.dev of weighted severity Agreement given (user group, page) on all barrier types compute the ICC (Intraclass Correlation Coefficient – relative and absolute consistency) c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users
  11. 11. The problem Barrier walkthrough Experimental plan Results Discussion and conclusions Reproducibility c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users
  12. 12. The problem Barrier walkthrough Experimental plan Results Discussion and conclusions Reproducibility c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users
  13. 13. The problem Barrier walkthrough Experimental plan Results Discussion and conclusions Mean weighted severities c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users
  14. 14. The problem Barrier walkthrough Experimental plan Results Discussion and conclusions Correctness Ratings: I Error rate E = C+I Accuracy = % of reported barriers that are correct Sensitivity = % of correct barriers that are reported 2A·S F.measure = A+S c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users
  15. 15. The problem Barrier walkthrough Experimental plan Results Discussion and conclusions Error rates c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users
  16. 16. The problem Barrier walkthrough Experimental plan Results Discussion and conclusions F-measure c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users
  17. 17. The problem Barrier walkthrough Experimental plan Results Discussion and conclusions Invariant properties 1. Aggregation does not worsen the problem of missed barriers 2. Reliability: experts are consistently more reliable; same pattern across pages 3. Severities: experts are more judgmental; ranks of pages do not change 4. Quality: error rates maintain a similar difference (expert vs non-experts) 5. Quality: F-measure conf. intervals shrink; they keep same relationship c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users
  18. 18. The problem Barrier walkthrough Experimental plan Results Discussion and conclusions Conclusions 1. Aggregation seems to work: it enables contextualized evaluations and leads to results that are potentially valid 2. It could be extended to cope with degrees of impairment Limitations 1. We did not validate our conclusions against an independent assessment 2. We don’t know if the same conclusions would hold for any set of primitive user categories Questions? c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users
  19. 19. The problem Barrier walkthrough Experimental plan Results Discussion and conclusions Evaluation framework based on reliability (reproducibility + agreement), correctness (error rate, accuracy, sensitivity and F-measure) is viable is discriminatory It can be used to assess pros and cons of an evaluation method. c Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users

×