1.
Evaluation of the Effectiveness of a Tool
to Support Novice Auditors
Christopher Bailey
Dr. Elaine Pearson
Teesside University
c.p.bailey@tees.ac.uk

2.
Practitioner Context
• Computing students need support with accessibility as:
– Lack general awareness of accessibility.
– Minimal inclusion in UG and/or PG curriculum.
– Evaluation is only one element of accessibility.
– Projects include developing live websites.
– Limited face-to-face student/tutor instruction.
– Limited time to dedicate to accessibility.
– Limited access to expertise/end users for testing.
– Need skills to enhance employability.

3.
Research Context
• Evaluation report (audit) has motivational and educational value
(Sloan, 2006).
• Manual evaluation is important (WCAG 2.0, UWEM, BW).
• Evaluation support tools aimed at experienced evaluators.
• The expertise level of the evaluator is particularly significant (WCAG
2.0, Brajnik, 2010; BW, Yesilada et al, 2009).
• Fewer false positives and false negatives, faster, and more
confident in judgements.
• Comprehension, Knowledge and Effort (Alonso et al, 2010).

4.
Accessibility Evaluation Assistant
• Educational evaluation support tool for novices
• Support functions:
– User Group
– Site Features
– Check Categories
• Structured Walkthrough Method – Translation of expert process
– Title and Summary of Accessibility Principle (Heuristic)
– The User Group(s) affected
– The nature of problem caused and barrier presented
– A step-by-step checking procedure (manual and/or tool support
– Guidance for verification (interpreting results of tool)
– An example video tutorial

5.
How Effective is AEA?
• Initial trial showed promising validity and reliability (Bailey &
Pearson, 2011)
• Compare relative effectiveness to WCAG 2.0
• Reliability (reach same decision)
• Validity
– Correctness (in matching experienced evaluator)
– Sensitivity (identifying true barriers)
• Usefulness
• Usability
• Efficiency

6.
Trial Methodology
• 38 Undergraduate Computing Students, 12 week elective
Accessibility and Adaptive Technology Module.
• Conducted within constraints of curriculum as an assessment
• 4 Tasks:
– 2 Evaluation Exercises: Sunsail and Harley Davidson Home
Pages
– 2 Reflective Pieces: Personas/User Group, Experience of
Evaluation (Compare Methods, Describe Problems).
• Evaluate 15 AEA Heuristics, equivalent 17 WCAG 2.0 Success
Criteria, relevant to both webpages.
• Check criteria is Met, Not Met or Partly Met and explain/justify their
decision.

7.
Results: Reliability
Week 1 Week 2
Grp.
Method/Website Rel. Method/Website Rel.
1 AEA - Harley Davidson 73% WCAG 2.0 - Sunsail 63%
2 WCAG 2.0 –
67% AEA - Sunsail 78%
Harley Davidson
3 WCAG 2.0 –
AEA - Sunsail 71% 59%
Harley Davidson
4 WCAG 2.0 - Sunsail 70% AEA - Harley Davidson 63%
Week 1 Week 2 Overall
Method WCAG WCAG WCAG
AEA AEA AEA
2.0 2.0 2.0
Reliability 72% 68.5% 70.5% 61% 71.25% 64.75%

8.
Results: Validity (Correctness)
Week 1 Week 2
Grp.
Method/Website Validity Method/Website Validity
1 AEA - Harley Davidson 73% WCAG 2.0 - Sunsail 49%
2 WCAG 2.0 – 59% AEA - Sunsail 73%
Harley Davidson
3 AEA - Sunsail 66% WCAG 2.0 – 49%
Harley Davidson
4 WCAG 2.0 - Sunsail 50% AEA - Harley Davidson 62%
Week 1 Week 2 Overall
Method WCAG WCAG WCAG
AEA AEA AEA
2.0 2.0 2.0
Validity 69.5% 54.5% 67.5% 49% 68.5% 51.75%

9.
Results: Validity (Sensitivity - AEA)
Sunsail Home Page Harley Davidson Home Page
Heuristic Failure Detection Heuristic Failure Detection
Images of Text 72% Images of Text 88%
Text Size 89% Colour Contrast 59%
Keyboard Navigation 94% Text Size 88%
Skip Navigation 89% Keyboard Navigation 76%
Image Text Skip Navigation 94%
94%
Alternatives Image Text Alternatives 82%
Headings 83% Headings 76%
Form Labels 39% Form Labels 76%
Valid (X)HTML Code 94% Identify Language of 76%
Site Map 100% Page
Average 85% Valid (X)HTML Code 100%
Search Function 94%
Average 83%

10.
Results: Validity (Sensitivity)
• Overall Comparison
Sunsail Home Page Harley Davidson Home Page
Method Average Method Average
Detection Rate Detection Rate
AEA 85% AEA 83%
WCAG 2.0 80% WCAG 2.0 74%

11.
Qualitative Feeback (AEA)
• Usable, Useful, Efficient?
Positive Negative
Easy/Simple to Use/Concise Check explanation too brief
Easy to Understand/Clear Poor UI Usability/Videos too
Terminology small
Explanation Guides User Checks Require Individual
Judgement
Categorisation/Grouping of Hard to Judge Met/Not Met
Checks
Speed of Check Process No Advice on Solving
Problems
Helpful Videos Bugs and Errors

12.
Qualitative Feeback (WCAG)
• Usable, Useful, Efficient?
Positive Negative
Detailed Explanations Confusing/Difficult to
Understand
Linked to Regulation and Complex/Hard to Use
Industry
Real Examples in More
Documentation Knowledge/Experience
Required
Fewer Individual Hard to Judge Relevance
Judgements Required of Check
Easy to Navigate No Explanation for
Documentation Performing Check

13.
Findings and Implications
• Use of AEA leads to higher levels of reliability and correctness, and
novices are able to identify true barriers.
• Subjective judgements influence results (evaluator effect).
• We can provide a method, but we still can’t control:
– Thoroughness
– Effort
– Competence
• Novices can relate heuristics to WCAG 2.0 Success Criteria
• Structured approach makes it easier to apply, repeat and learn.
• Complement existing methods for use in industry.

14.
Future Work
• Enhance Structured Walkthrough approach to further limit errors:
– Remove ambiguities to reduce misinterpretation
– Enhance guidance for decision making
– Develop a simple metric for determining severity
• Redevelop AEA to increase flexibility of delivery
• Trial in another institution
• Add functionality to formally support WCAG 2.0 evaluation
• Longitudinal studies:
– Further demonstrate effectiveness of AEA
– Barriers which novices consistently identify/miss
– Identify best strategies to teach novices (evaluation, simulation,
demonstration, personas)

15.
Student Experience
• “….using the AEA are that you are told precisely
what you are checking clearly and concisely, making
the process faster and simpler. With WCAG 2.0 the
sentences are extremely long…. I had to repeatedly
read the check.”
• “The AEA is very easy to understand, follow and
implement the checks. The step-by-step instructions
are not only helpful, but informative and made my
understand why I should be performing the check”.

16.
Student Experience
• “The advantages of the AEA tool are speed, simplicity
and ease-of-use. Using the AEA definitely felt more
effective as it follows the WCAG 2.0 guideline but
streamlines the majority of checks allowing you bypass
the technical jargon and bureaucracy in WCAG.”
• “I found the AEA tool significantly easier and more
intuitive that the WCAG 2.0 approach….it guides the
tester to a relevant section, explains what to look for in
an easy to understand manner as well as explaining the
purpose of the test.”

17.
Evaluation of the Effectiveness of a Tool
to Support Novice Auditors
http://arc.tees.ac.uk/aea
Christopher Bailey
Dr. Elaine Pearson
Teesside University
c.p.bailey@tees.ac.uk