1.
Evaluation of the Effectiveness of a Toolto Support Novice Auditors Christopher Bailey Dr. Elaine Pearson Teesside University c.p.bailey@tees.ac.uk

2.
Practitioner Context• Computing students need support with accessibility as: – Lack general awareness of accessibility. – Minimal inclusion in UG and/or PG curriculum. – Evaluation is only one element of accessibility. – Projects include developing live websites. – Limited face-to-face student/tutor instruction. – Limited time to dedicate to accessibility. – Limited access to expertise/end users for testing. – Need skills to enhance employability.

3.
Research Context• Evaluation report (audit) has motivational and educational value (Sloan, 2006).• Manual evaluation is important (WCAG 2.0, UWEM, BW).• Evaluation support tools aimed at experienced evaluators.• The expertise level of the evaluator is particularly significant (WCAG 2.0, Brajnik, 2010; BW, Yesilada et al, 2009).• Fewer false positives and false negatives, faster, and more confident in judgements.• Comprehension, Knowledge and Effort (Alonso et al, 2010).

4.
Accessibility Evaluation Assistant• Educational evaluation support tool for novices• Support functions: – User Group – Site Features – Check Categories• Structured Walkthrough Method – Translation of expert process – Title and Summary of Accessibility Principle (Heuristic) – The User Group(s) affected – The nature of problem caused and barrier presented – A step-by-step checking procedure (manual and/or tool support – Guidance for verification (interpreting results of tool) – An example video tutorial

5.
How Effective is AEA?• Initial trial showed promising validity and reliability (Bailey & Pearson, 2011)• Compare relative effectiveness to WCAG 2.0• Reliability (reach same decision)• Validity – Correctness (in matching experienced evaluator) – Sensitivity (identifying true barriers)• Usefulness• Usability• Efficiency

6.
Trial Methodology• 38 Undergraduate Computing Students, 12 week elective Accessibility and Adaptive Technology Module.• Conducted within constraints of curriculum as an assessment• 4 Tasks: – 2 Evaluation Exercises: Sunsail and Harley Davidson Home Pages – 2 Reflective Pieces: Personas/User Group, Experience of Evaluation (Compare Methods, Describe Problems).• Evaluate 15 AEA Heuristics, equivalent 17 WCAG 2.0 Success Criteria, relevant to both webpages.• Check criteria is Met, Not Met or Partly Met and explain/justify their decision.

7.
Results: Reliability Week 1 Week 2Grp. Method/Website Rel. Method/Website Rel.1 AEA - Harley Davidson 73% WCAG 2.0 - Sunsail 63%2 WCAG 2.0 – 67% AEA - Sunsail 78% Harley Davidson3 WCAG 2.0 – AEA - Sunsail 71% 59% Harley Davidson4 WCAG 2.0 - Sunsail 70% AEA - Harley Davidson 63% Week 1 Week 2 Overall Method WCAG WCAG WCAG AEA AEA AEA 2.0 2.0 2.0 Reliability 72% 68.5% 70.5% 61% 71.25% 64.75%

8.
Results: Validity (Correctness) Week 1 Week 2Grp. Method/Website Validity Method/Website Validity1 AEA - Harley Davidson 73% WCAG 2.0 - Sunsail 49%2 WCAG 2.0 – 59% AEA - Sunsail 73% Harley Davidson3 AEA - Sunsail 66% WCAG 2.0 – 49% Harley Davidson4 WCAG 2.0 - Sunsail 50% AEA - Harley Davidson 62% Week 1 Week 2 Overall Method WCAG WCAG WCAG AEA AEA AEA 2.0 2.0 2.0 Validity 69.5% 54.5% 67.5% 49% 68.5% 51.75%

9.
Results: Validity (Sensitivity - AEA) Sunsail Home Page Harley Davidson Home Page Heuristic Failure Detection Heuristic Failure DetectionImages of Text 72% Images of Text 88%Text Size 89% Colour Contrast 59%Keyboard Navigation 94% Text Size 88%Skip Navigation 89% Keyboard Navigation 76%Image Text Skip Navigation 94% 94%Alternatives Image Text Alternatives 82%Headings 83% Headings 76%Form Labels 39% Form Labels 76%Valid (X)HTML Code 94% Identify Language of 76%Site Map 100% PageAverage 85% Valid (X)HTML Code 100% Search Function 94% Average 83%

10.
Results: Validity (Sensitivity)• Overall Comparison Sunsail Home Page Harley Davidson Home Page Method Average Method Average Detection Rate Detection RateAEA 85% AEA 83%WCAG 2.0 80% WCAG 2.0 74%

11.
Qualitative Feeback (AEA)• Usable, Useful, Efficient? Positive Negative Easy/Simple to Use/Concise Check explanation too brief Easy to Understand/Clear Poor UI Usability/Videos too Terminology small Explanation Guides User Checks Require Individual Judgement Categorisation/Grouping of Hard to Judge Met/Not Met Checks Speed of Check Process No Advice on Solving Problems Helpful Videos Bugs and Errors

12.
Qualitative Feeback (WCAG)• Usable, Useful, Efficient? Positive Negative Detailed Explanations Confusing/Difficult to Understand Linked to Regulation and Complex/Hard to Use Industry Real Examples in More Documentation Knowledge/Experience Required Fewer Individual Hard to Judge Relevance Judgements Required of Check Easy to Navigate No Explanation for Documentation Performing Check

13.
Findings and Implications• Use of AEA leads to higher levels of reliability and correctness, and novices are able to identify true barriers.• Subjective judgements influence results (evaluator effect).• We can provide a method, but we still can’t control: – Thoroughness – Effort – Competence• Novices can relate heuristics to WCAG 2.0 Success Criteria• Structured approach makes it easier to apply, repeat and learn.• Complement existing methods for use in industry.

14.
Future Work• Enhance Structured Walkthrough approach to further limit errors: – Remove ambiguities to reduce misinterpretation – Enhance guidance for decision making – Develop a simple metric for determining severity• Redevelop AEA to increase flexibility of delivery• Trial in another institution• Add functionality to formally support WCAG 2.0 evaluation• Longitudinal studies: – Further demonstrate effectiveness of AEA – Barriers which novices consistently identify/miss – Identify best strategies to teach novices (evaluation, simulation, demonstration, personas)

15.
Student Experience• “….using the AEA are that you are told precisely what you are checking clearly and concisely, making the process faster and simpler. With WCAG 2.0 the sentences are extremely long…. I had to repeatedly read the check.”• “The AEA is very easy to understand, follow and implement the checks. The step-by-step instructions are not only helpful, but informative and made my understand why I should be performing the check”.

16.
Student Experience• “The advantages of the AEA tool are speed, simplicity and ease-of-use. Using the AEA definitely felt more effective as it follows the WCAG 2.0 guideline but streamlines the majority of checks allowing you bypass the technical jargon and bureaucracy in WCAG.”• “I found the AEA tool significantly easier and more intuitive that the WCAG 2.0 approach….it guides the tester to a relevant section, explains what to look for in an easy to understand manner as well as explaining the purpose of the test.”

17.
Evaluation of the Effectiveness of a Toolto Support Novice Auditorshttp://arc.tees.ac.uk/aea Christopher Bailey Dr. Elaine Pearson Teesside University c.p.bailey@tees.ac.uk