1. The document describes the development and testing of an Accessibility Evaluation Assistant (AEA) tool to help novice evaluators learn how to manually assess websites for accessibility. 2. A trial of the AEA tool involved 38 undergraduate students evaluating the accessibility of pages on 4 different websites. 3. The results showed that when guided by the AEA, novice evaluators were able to reliably identify accessibility barriers, though some check categories like subjective user checks produced lower agreement with experts. The AEA shows promise as an educational tool for developing evaluation skills.

1. Christopher Bailey Dr. Elaine Pearson Teesside University [email_address] Development and Trial of an Educational Tool to Support the Accessibility Evaluation Process

2. Related Work Manual evaluation plays an important role in accessibility evaluation (WCAG 2.0, UWEM, BW). The expertise level of the evaluator is significant. Experts take less time, are more productive, more confident in their judgements and can rate pages again different disability types (Yesilada et al, 2009 – Barrier Walkthrogh). Experts produce fewer false positive and false negatives than novices (Brajnik, 2010 – WCAG 2.0). Only 8 of 25 level-A success criteria reliably testable by novices (Alonso et al, 2010). Important factors are Comprehension, Knowledge and Effort.

3. Context How can novices develop manual evaluation skills? Evaluation reports have positive educational and motivational aspects. Novices conducting evaluation has the potential for strong pedagogical value. Produce a method to guide novices through an evaluation, mimicking an expert process. Students in our institution need support with accessibility. Developed the Accessibility Evaluation Assistant (AEA) as a solution.

4. AEA Development The tool lists accessibility checks based on three evaluation contexts: User Group Site Features Check Categories Evaluation focussed on Check Categories Design Checks: concerned with aspects of visual presentation. User Checks: Auditor must interact with the website, results are often subjective. Structural Checks: Structural and Semantic information Technical Checks: Metadata and Valid Code, Core Checks: Checks which refer to specific content, functionality or apply to the entire website.

5. Structured Walkthrough Method Based on Barrier Walkthrough ( Brajnik) A heuristic evaluation technique based on checking for potential accessibility barriers. Information for each check: The Accessibility Principle they are checking for. The User Group(s) affected. The nature of the barrier or problem caused. A procedure for checking and verifying the issue. An example video tutorial . Procedure could be manual, automatic, or both.

6. Trial Methodology Aimed to test the reliability (different evaluators reach same decision) and validity (true accessibility barriers are identified) of the AEA. Trialled with 38 Undergraduate Computing Students. Participant assigned 1 website (from 4) and had to evaluate 3 page, the Home Page being compulsory. For each check the participant had to decide whether the check was either Met; Not Met; Partly Met or Not Applicable. Participant had to explain and justify their decision.

7. Consensus Analysis W3C state a check is reliably human testable if 80% of knowledgeable evaluators reach same conclusion. 53% 68% 82% Combined Results 56% (27/48) 67% (32/48) 79% (38/48) WalMart 45% (22/48) 58% (32/48) 71% (34/48) Premier League 50% (24/48) 66% (32/48) 83% (40/48) CNN 60% (29/48) 79% (38/48) 96% (46/48) Vancouver Olympics >70% >60% >50% Home Page

8. Expert Agreement by Website If we take the expert decision as ‘correct’ we can make some initial conclusions about reliability of AEA method. 124/192 60% Combined Results 29/48 58% WalMart 25/48 56% Premier League 31/48 60% CNN 36/48 65% Vancouver Olympics Checks with majority expert agreement Decisions Matching Expert (%) Home Page

9. Expert Agreement by Category The example refers to the Vancouver website. We examined which checks were most likely to be correctly evaluated by the novices, and be ‘valid’. 85% Global 76% Technical 68% Structural 49% User 59% Design Novice agreement with expert Check Category

10. Implications Given that the auditors conducting their first evaluation, we consider these figures promising. Validity would have been higher without the option of ‘Partly Met’. Novices had 4 choices compared to 3. Novices using the AEA tool are able to identify accessibility barriers so the method is partially effective. Structured makes it usable so it can be applied, repeated, learned and remembered. Checks that require subjective judgement still have lowest level of accuracy. We can provide a method, but we can’t control how well the novices use it, or how much effort they put into using it.

11. Student Experience As an evaluation methodology there are issues, but potential for AEA as an educational tool. Even if students made incorrect decisions, they were made aware of the range of issues and implications. “ I did not know there were so many considerations when developing a website – the AEA made me realise there is more than just producing valid code.” “ Carrying out an accessibility evaluation really helped me understand the importance of accessibility. Before I carried out the evaluation I didn’t know what checks to carry out”.

12. Future Work Improve the clarity and accuracy of checks by removing ambiguities. Provide clearer guidance on what conditions are required for each decision. Improve interface, group related checks and integrate automated tools. Further analysis of results to identify overall figure of false positive, false negatives. Identify the extent to which the novices identified the accessibility barriers present on the site.

13. Christopher Bailey Dr. Elaine Pearson Teesside University [email_address] Accessibility Evaluation Assistant http://arc.tees.ac.uk/aea Questions?