Successfully reported this slideshow.

W4A12 Bailey

0

Share

1 of 17
1 of 17

More Related Content

Related Books

Free with a 14 day trial from Scribd

See all

Related Audiobooks

Free with a 14 day trial from Scribd

See all

W4A12 Bailey

  1. 1. Evaluation of the Effectiveness of a Tool to Support Novice Auditors Christopher Bailey Dr. Elaine Pearson Teesside University c.p.bailey@tees.ac.uk
  2. 2. Practitioner Context • Computing students need support with accessibility as: – Lack general awareness of accessibility. – Minimal inclusion in UG and/or PG curriculum. – Evaluation is only one element of accessibility. – Projects include developing live websites. – Limited face-to-face student/tutor instruction. – Limited time to dedicate to accessibility. – Limited access to expertise/end users for testing. – Need skills to enhance employability.
  3. 3. Research Context • Evaluation report (audit) has motivational and educational value (Sloan, 2006). • Manual evaluation is important (WCAG 2.0, UWEM, BW). • Evaluation support tools aimed at experienced evaluators. • The expertise level of the evaluator is particularly significant (WCAG 2.0, Brajnik, 2010; BW, Yesilada et al, 2009). • Fewer false positives and false negatives, faster, and more confident in judgements. • Comprehension, Knowledge and Effort (Alonso et al, 2010).
  4. 4. Accessibility Evaluation Assistant • Educational evaluation support tool for novices • Support functions: – User Group – Site Features – Check Categories • Structured Walkthrough Method – Translation of expert process – Title and Summary of Accessibility Principle (Heuristic) – The User Group(s) affected – The nature of problem caused and barrier presented – A step-by-step checking procedure (manual and/or tool support – Guidance for verification (interpreting results of tool) – An example video tutorial
  5. 5. How Effective is AEA? • Initial trial showed promising validity and reliability (Bailey & Pearson, 2011) • Compare relative effectiveness to WCAG 2.0 • Reliability (reach same decision) • Validity – Correctness (in matching experienced evaluator) – Sensitivity (identifying true barriers) • Usefulness • Usability • Efficiency
  6. 6. Trial Methodology • 38 Undergraduate Computing Students, 12 week elective Accessibility and Adaptive Technology Module. • Conducted within constraints of curriculum as an assessment • 4 Tasks: – 2 Evaluation Exercises: Sunsail and Harley Davidson Home Pages – 2 Reflective Pieces: Personas/User Group, Experience of Evaluation (Compare Methods, Describe Problems). • Evaluate 15 AEA Heuristics, equivalent 17 WCAG 2.0 Success Criteria, relevant to both webpages. • Check criteria is Met, Not Met or Partly Met and explain/justify their decision.
  7. 7. Results: Reliability Week 1 Week 2 Grp. Method/Website Rel. Method/Website Rel. 1 AEA - Harley Davidson 73% WCAG 2.0 - Sunsail 63% 2 WCAG 2.0 – 67% AEA - Sunsail 78% Harley Davidson 3 WCAG 2.0 – AEA - Sunsail 71% 59% Harley Davidson 4 WCAG 2.0 - Sunsail 70% AEA - Harley Davidson 63% Week 1 Week 2 Overall Method WCAG WCAG WCAG AEA AEA AEA 2.0 2.0 2.0 Reliability 72% 68.5% 70.5% 61% 71.25% 64.75%
  8. 8. Results: Validity (Correctness) Week 1 Week 2 Grp. Method/Website Validity Method/Website Validity 1 AEA - Harley Davidson 73% WCAG 2.0 - Sunsail 49% 2 WCAG 2.0 – 59% AEA - Sunsail 73% Harley Davidson 3 AEA - Sunsail 66% WCAG 2.0 – 49% Harley Davidson 4 WCAG 2.0 - Sunsail 50% AEA - Harley Davidson 62% Week 1 Week 2 Overall Method WCAG WCAG WCAG AEA AEA AEA 2.0 2.0 2.0 Validity 69.5% 54.5% 67.5% 49% 68.5% 51.75%
  9. 9. Results: Validity (Sensitivity - AEA) Sunsail Home Page Harley Davidson Home Page Heuristic Failure Detection Heuristic Failure Detection Images of Text 72% Images of Text 88% Text Size 89% Colour Contrast 59% Keyboard Navigation 94% Text Size 88% Skip Navigation 89% Keyboard Navigation 76% Image Text Skip Navigation 94% 94% Alternatives Image Text Alternatives 82% Headings 83% Headings 76% Form Labels 39% Form Labels 76% Valid (X)HTML Code 94% Identify Language of 76% Site Map 100% Page Average 85% Valid (X)HTML Code 100% Search Function 94% Average 83%
  10. 10. Results: Validity (Sensitivity) • Overall Comparison Sunsail Home Page Harley Davidson Home Page Method Average Method Average Detection Rate Detection Rate AEA 85% AEA 83% WCAG 2.0 80% WCAG 2.0 74%
  11. 11. Qualitative Feeback (AEA) • Usable, Useful, Efficient? Positive Negative Easy/Simple to Use/Concise Check explanation too brief Easy to Understand/Clear Poor UI Usability/Videos too Terminology small Explanation Guides User Checks Require Individual Judgement Categorisation/Grouping of Hard to Judge Met/Not Met Checks Speed of Check Process No Advice on Solving Problems Helpful Videos Bugs and Errors
  12. 12. Qualitative Feeback (WCAG) • Usable, Useful, Efficient? Positive Negative Detailed Explanations Confusing/Difficult to Understand Linked to Regulation and Complex/Hard to Use Industry Real Examples in More Documentation Knowledge/Experience Required Fewer Individual Hard to Judge Relevance Judgements Required of Check Easy to Navigate No Explanation for Documentation Performing Check
  13. 13. Findings and Implications • Use of AEA leads to higher levels of reliability and correctness, and novices are able to identify true barriers. • Subjective judgements influence results (evaluator effect). • We can provide a method, but we still can’t control: – Thoroughness – Effort – Competence • Novices can relate heuristics to WCAG 2.0 Success Criteria • Structured approach makes it easier to apply, repeat and learn. • Complement existing methods for use in industry.
  14. 14. Future Work • Enhance Structured Walkthrough approach to further limit errors: – Remove ambiguities to reduce misinterpretation – Enhance guidance for decision making – Develop a simple metric for determining severity • Redevelop AEA to increase flexibility of delivery • Trial in another institution • Add functionality to formally support WCAG 2.0 evaluation • Longitudinal studies: – Further demonstrate effectiveness of AEA – Barriers which novices consistently identify/miss – Identify best strategies to teach novices (evaluation, simulation, demonstration, personas)
  15. 15. Student Experience • “….using the AEA are that you are told precisely what you are checking clearly and concisely, making the process faster and simpler. With WCAG 2.0 the sentences are extremely long…. I had to repeatedly read the check.” • “The AEA is very easy to understand, follow and implement the checks. The step-by-step instructions are not only helpful, but informative and made my understand why I should be performing the check”.
  16. 16. Student Experience • “The advantages of the AEA tool are speed, simplicity and ease-of-use. Using the AEA definitely felt more effective as it follows the WCAG 2.0 guideline but streamlines the majority of checks allowing you bypass the technical jargon and bureaucracy in WCAG.” • “I found the AEA tool significantly easier and more intuitive that the WCAG 2.0 approach….it guides the tester to a relevant section, explains what to look for in an easy to understand manner as well as explaining the purpose of the test.”
  17. 17. Evaluation of the Effectiveness of a Tool to Support Novice Auditors http://arc.tees.ac.uk/aea Christopher Bailey Dr. Elaine Pearson Teesside University c.p.bailey@tees.ac.uk

×