Accessibility metrics Accessibility Data Metrics and Reporting – Industry Best Practices

1. Accessibility Metrics Ted Drake and Sagar Barbhaya Intuit Accessibility CSUN 2017 Accessible version: http://www.last- child.com/a11y-data-metrics/

2. What Do You Need? Write it on a post it note

3. Three Stages 1. Accessibility Management Goals 2. State of the Industry 3. Solutions to ﬁll gaps

4. Current State • Manual Testing and Evaluations • Continual improvements • Documentation • Community Outreach • Stakeholders • Centralized Accessibility Leadership

5. What we have • Product database with evaluations, scores, stakeholders, and documents • Bug Tracking • Scattered Employee Resources • Centralized documentation • Non-integrated automation

6. What we need • Dashboards to surface data and trends • Connections between platforms • Automation and unit testing for developers • Shared test results • Connected resources • Support channel integration

7. Future Goal • Measurable lifecycle from tests/evaluation to production • Deep consumer feedback integration • Diverse workforce with support channels • Developer ownership of testing and solutions • Product ownership of process and success

8. State of the Industry

9. How are we doing? - Product Manager

10. What to Measure?

11. Product Quality Metrics • Issues by Priority – 77% • Issues by WCAG category – 54% • last time product had a manual evaluation – 46% • Closed Captioning support – 38% • Number of accessibility bugs opened per month/ year – 31%

12. Priority Levels • Impact on Customer • Template / Module / Unique • Density of Issues • High Trafﬁc, Key Flows

13. Test to Ticket ﬂow

14. Company Metrics •AODA compliance – 55% •Number of media inquiries about product/company accessibility – 45% •Conference talks by employees on accessibility – 45% •Product VPAT Coverage – 45% •Compliance towards 21st Century Communications Act (CVAA) – 36%

15. Compliance and Legal

16. Human Resources • Section 503: Self-Identiﬁcation • Accommodations Requests • Ergonomic Evaluations • Service/Perk Requests

17. Customer Metrics •Usability participants with a disability – 63% •Feedback via customer support – 50% •Usability activities, including user personas, with disabilities – 38% •Customer support calls - 38% •Tickets ﬁled by customer support – 38%

18. Net Promoter Score

19. Storage and Sharing

20. QuickBase

21. Spreadsheets

22. Pa11y Testing + Dashboard

23. Google Docs

24. Cloud Storage • DropBox • Box • Amazon Web Services • Google Drive • Slack

25. WorldSpace

26. Solutions

27. Bug Tracking

28. JIRA

29. JIRA Queries • labels = accessibility • labels=accessibility and status=Open • labels=accessibility and status = blocked • labels=accessibility and status=Closed and resolved > “-4w”

30. DOMO

31. Integrated Testing • Developer testing: Tenon, aXe, Nemo • Unit tests include accessibility • Export test results to JIRA • Tracking tool usage

32. Finding Outliers • Extremely critical to prioritize • Outliers need to be   identified and fixed first • Example - Turbo Tax in   season production bugs

33. Customer Support Logs • Search for Deaf and HOH customer requests:   [ LEVEL:SENTENCE TYPE:KEYWORD {deaf, hearing}]   [ LEVEL:SENTENCE TYPE:AND_KEYWORD {capability, impaired, challenged, hard}] • “Accessibility” topic within Salesforce for reports • Microsoft: Dedicated Support Channel

34. What Did You Need?

35. Questions?