SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.
SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.
Successfully reported this slideshow.
Activate your 14 day free trial to unlock unlimited reading.
1.
Evaluation of the Effectiveness of a Tool
to Support Novice Auditors
Christopher Bailey
Dr. Elaine Pearson
Teesside University
c.p.bailey@tees.ac.uk
2.
Practitioner Context
• Computing students need support with accessibility as:
– Lack general awareness of accessibility.
– Minimal inclusion in UG and/or PG curriculum.
– Evaluation is only one element of accessibility.
– Projects include developing live websites.
– Limited face-to-face student/tutor instruction.
– Limited time to dedicate to accessibility.
– Limited access to expertise/end users for testing.
– Need skills to enhance employability.
3.
Research Context
• Evaluation report (audit) has motivational and educational value
(Sloan, 2006).
• Manual evaluation is important (WCAG 2.0, UWEM, BW).
• Evaluation support tools aimed at experienced evaluators.
• The expertise level of the evaluator is particularly significant (WCAG
2.0, Brajnik, 2010; BW, Yesilada et al, 2009).
• Fewer false positives and false negatives, faster, and more
confident in judgements.
• Comprehension, Knowledge and Effort (Alonso et al, 2010).
4.
Accessibility Evaluation Assistant
• Educational evaluation support tool for novices
• Support functions:
– User Group
– Site Features
– Check Categories
• Structured Walkthrough Method – Translation of expert process
– Title and Summary of Accessibility Principle (Heuristic)
– The User Group(s) affected
– The nature of problem caused and barrier presented
– A step-by-step checking procedure (manual and/or tool support
– Guidance for verification (interpreting results of tool)
– An example video tutorial
5.
How Effective is AEA?
• Initial trial showed promising validity and reliability (Bailey &
Pearson, 2011)
• Compare relative effectiveness to WCAG 2.0
• Reliability (reach same decision)
• Validity
– Correctness (in matching experienced evaluator)
– Sensitivity (identifying true barriers)
• Usefulness
• Usability
• Efficiency
6.
Trial Methodology
• 38 Undergraduate Computing Students, 12 week elective
Accessibility and Adaptive Technology Module.
• Conducted within constraints of curriculum as an assessment
• 4 Tasks:
– 2 Evaluation Exercises: Sunsail and Harley Davidson Home
Pages
– 2 Reflective Pieces: Personas/User Group, Experience of
Evaluation (Compare Methods, Describe Problems).
• Evaluate 15 AEA Heuristics, equivalent 17 WCAG 2.0 Success
Criteria, relevant to both webpages.
• Check criteria is Met, Not Met or Partly Met and explain/justify their
decision.
9.
Results: Validity (Sensitivity - AEA)
Sunsail Home Page Harley Davidson Home Page
Heuristic Failure Detection Heuristic Failure Detection
Images of Text 72% Images of Text 88%
Text Size 89% Colour Contrast 59%
Keyboard Navigation 94% Text Size 88%
Skip Navigation 89% Keyboard Navigation 76%
Image Text Skip Navigation 94%
94%
Alternatives Image Text Alternatives 82%
Headings 83% Headings 76%
Form Labels 39% Form Labels 76%
Valid (X)HTML Code 94% Identify Language of 76%
Site Map 100% Page
Average 85% Valid (X)HTML Code 100%
Search Function 94%
Average 83%
10.
Results: Validity (Sensitivity)
• Overall Comparison
Sunsail Home Page Harley Davidson Home Page
Method Average Method Average
Detection Rate Detection Rate
AEA 85% AEA 83%
WCAG 2.0 80% WCAG 2.0 74%
11.
Qualitative Feeback (AEA)
• Usable, Useful, Efficient?
Positive Negative
Easy/Simple to Use/Concise Check explanation too brief
Easy to Understand/Clear Poor UI Usability/Videos too
Terminology small
Explanation Guides User Checks Require Individual
Judgement
Categorisation/Grouping of Hard to Judge Met/Not Met
Checks
Speed of Check Process No Advice on Solving
Problems
Helpful Videos Bugs and Errors
12.
Qualitative Feeback (WCAG)
• Usable, Useful, Efficient?
Positive Negative
Detailed Explanations Confusing/Difficult to
Understand
Linked to Regulation and Complex/Hard to Use
Industry
Real Examples in More
Documentation Knowledge/Experience
Required
Fewer Individual Hard to Judge Relevance
Judgements Required of Check
Easy to Navigate No Explanation for
Documentation Performing Check
13.
Findings and Implications
• Use of AEA leads to higher levels of reliability and correctness, and
novices are able to identify true barriers.
• Subjective judgements influence results (evaluator effect).
• We can provide a method, but we still can’t control:
– Thoroughness
– Effort
– Competence
• Novices can relate heuristics to WCAG 2.0 Success Criteria
• Structured approach makes it easier to apply, repeat and learn.
• Complement existing methods for use in industry.
14.
Future Work
• Enhance Structured Walkthrough approach to further limit errors:
– Remove ambiguities to reduce misinterpretation
– Enhance guidance for decision making
– Develop a simple metric for determining severity
• Redevelop AEA to increase flexibility of delivery
• Trial in another institution
• Add functionality to formally support WCAG 2.0 evaluation
• Longitudinal studies:
– Further demonstrate effectiveness of AEA
– Barriers which novices consistently identify/miss
– Identify best strategies to teach novices (evaluation, simulation,
demonstration, personas)
15.
Student Experience
• “….using the AEA are that you are told precisely
what you are checking clearly and concisely, making
the process faster and simpler. With WCAG 2.0 the
sentences are extremely long…. I had to repeatedly
read the check.”
• “The AEA is very easy to understand, follow and
implement the checks. The step-by-step instructions
are not only helpful, but informative and made my
understand why I should be performing the check”.
16.
Student Experience
• “The advantages of the AEA tool are speed, simplicity
and ease-of-use. Using the AEA definitely felt more
effective as it follows the WCAG 2.0 guideline but
streamlines the majority of checks allowing you bypass
the technical jargon and bureaucracy in WCAG.”
• “I found the AEA tool significantly easier and more
intuitive that the WCAG 2.0 approach….it guides the
tester to a relevant section, explains what to look for in
an easy to understand manner as well as explaining the
purpose of the test.”
17.
Evaluation of the Effectiveness of a Tool
to Support Novice Auditors
http://arc.tees.ac.uk/aea
Christopher Bailey
Dr. Elaine Pearson
Teesside University
c.p.bailey@tees.ac.uk
0 likes
Be the first to like this
Views
Total views
3,229
On SlideShare
0
From Embeds
0
Number of Embeds
3
You have now unlocked unlimited access to 20M+ documents!
Unlimited Reading
Learn faster and smarter from top experts
Unlimited Downloading
Download to take your learnings offline and on the go
You also get free access to Scribd!
Instant access to millions of ebooks, audiobooks, magazines, podcasts and more.
Read and listen offline with any device.
Free access to premium services like Tuneln, Mubi and more.