Ted Gies, profile picture
Uploaded byTed Gies
PPTX, PDF180 views

CSUN 2023 Analytics.pptx

The document discusses accessibility analytics and the assessment of various Elsevier products against WCAG 2.1 standards, focusing on the importance of accessibility metrics and the use of VPAT evaluations. It highlights progress in meeting accessibility goals, such as improving VPAT scores and addressing shortcomings in products' compliance with audio descriptions, alt text, and other criteria. The document also emphasizes the use of data-driven insights for informed decision-making to enhance web accessibility across their offerings.

Embed presentation

Downloaded 26 times
Accessibility Analytics – Being Data Driven With VPATs CSUN Thursday March 16 2023 Platinum 7-8 9:20 am Jay Nemchik Ted Gies
Elsevier Mission Elsevier helps researchers and healthcare professionals advance science and improve health outcomes for the benefit of society. 2
Back in 2016 – Based Upon a True Story “As our leadership team is planning our objectives and key results for this year, could you help identify some accessibility metrics.”
VPAT/ACR Criteria Conformance Level Remarks and Explanations 1.1.1 Non-text Content (Level A) 1.2.1 Audio-only and Video-only (Prerecorded) (Level A) 1.2.2 Captions (Prerecorded) (Level A) 1.2.3 Audio Description or Media Alternative (Prerecorded) (Level A) 1.3.1 Info and Relationships (Level A) 1.3.2 Meaningful Sequence (Level A) 1.3.3 Sensory Characteristics (Level A) 1.4.1 Use of Color (Level A) 1.4.2 Audio Control (Level A) 2.1.1 Keyboard (Level A) 2.1.2 No Keyboard Trap (Level A) 2.1.4 Character Key Shortcuts (Level A 2.1 only) 2.2.1 Timing Adjustable (Level A) 2.2.2 Pause, Stop, Hide (Level A) 2.3.1 Three Flashes or Below Threshold (Level A) 2.4.1 Bypass Blocks (Level A) 2.4.2 Page Titled (Level A) 2.4.3 Focus Order (Level A) 2.4.4 Link Purpose (In Context) (Level A) 2.5.1 Pointer Gestures (Level A 2.1 only) 2.5.2 Pointer Cancellation (Level A 2.1 only) 2.5.3 Label in Name (Level A 2.1 only) 2.5.4 Motion Actuation (Level A 2.1 only) 3.1.1 Language of Page (Level A) 3.2.1 On Focus (Level A) 3.2.2 On Input (Level A) 3.3.1 Error Identification (Level A) 3.3.2 Labels or Instructions (Level A) 4.1.1 Parsing (Level A)
Example of 1 VPAT/ACR Checkpoint WCAG 2.1 Checkpoint Supporting Features Remarks 1.1.1: Non-Text Content (A) Provide text alternatives for non- text content (e.g. images) Partially Supports The product presents non-text content with a text alternative that serves the equivalent purpose. Supporting Remarks Linked icons are labelled such as Help and Institution. Exceptions A rating of Partially Supports has been given for the following reasons: • There are images within the pages that do not have an alt attribute or aria- label set. • There are buttons using icons only that are missing accessible name/labels. • There are values within the alt attribute that are not descriptive. • Invalid use of the alt attribute.
| 6 ACR Evaluation of 1 Product WCAG 2.1 Success Criterion Evaluation 1.1.1 Non-text Content A Partially Supports 1.2.1 Audio-only and Video-only (Prerecorded) A Supports (N/A) 1.2.2 Captions (Prerecorded) A Supports (N/A) 1.2.3 Audio Description /Full Text Alternative A Supports (N/A) 1.2.4 Captions (Live) AA Supports (N/A) 1.2.5 Audio Description AA Supports (N/A) 1.3.1 Info and Relationships A Supports 1.3.2 Meaningful Sequence A Partially Supports 1.3.3 Sensory Characteristics A Supports 1.3.4 Orientation AA Supports 1.3.5 Identify Input Purpose AA Does not Support 1.4.1 Use of Color A Partially Supports 1.4.2 Audio Control A Supports (N/A) 1.4.3 Contrast (Minimum) AA Partially Supports 1.4.4 Resize text AA Supports 1.4.5 Images of Text AA Supports 1.4.10 Reflow AA Supports 1.4.11 Non-text Contrast AA Partially Supports 1.4.12 Text Spacing AA Supports 1.4.13 Content on Hover or Focus AA Does not Support 2.1.1 Keyboard A Partially Supports 2.1.2 No Keyboard Trap A Supports 2.1.4 Character Key Shortcuts A Supports (N/A) 2.2.1 Timing Adjustable A Supports 2.2.2 Pause, Stop, Hide A Supports (N/A) WCAG 2.1 Success Criterion Evaluation 2.3.1 Three Flashes or Below Threshold A Supports (N/A) 2.4.1 Bypass Blocks A Supports 2.4.2 Page Titled A Supports 2.4.3 Focus Order A Supports 2.4.4 Link Purpose (In Context) A Partially Supports 2.4.5 Multiple Ways AA Supports 2.4.6 Headings and Labels AA Supports 2.4.7 Focus Visible AA Supports 2.5.1 Pointer Gestures A Supports (N/A) 2.5.2 Pointer Cancellation A Supports 2.5.3 Label in Name A Supports 2.5.4 Motion Actuation A Supports (N/A) 3.1.1 Language of Page A Supports 3.1.2 Language of Parts AA Supports (N/A) 3.2.1 On Focus A Supports 3.2.2 On Input A Supports 3.2.3 Consistent Navigation AA Partially Supports 3.2.4 Consistent Identification AA Supports 3.3.1 Error Identification A Supports 3.3.2 Labels or Instructions A Partially Supports 3.3.3 Error Suggestion AA Supports 3.3.4 Error Prevention (Legal, Financial, Data) AA Supports 4.1.1 Parsing A Supports 4.1.2 Name, Role, Value A Partially Supports 4.1.3 Status Messages AA Supports
Problem Statements • Managers and employees use numbers to measure things and set goals. • Web accessibility data generated from an ACR is mostly qualitative. • There is a mismatch between the nature of ACR accessibility data and providing simple quantitative reports. Goal: We want to inform decision makers with analytics to help improve and maintain high levels of accessibility
Our Approach: Create an Excel Database out of Mined ACR Data • Originally input 38 websites ACR Ratings into Excel • Translated the conformance grading to percentages in Excel • Used Excel Pivot Tables/Pivot Charts to generate graphs Total Supports Total Partially supports Total Does not Support Percentage Supports Percentage Partially Supports Percentage Does not Support Product 1 40 8 2 80% 16% 4% Product 2 38 3 9 76% 6% 18% Product 3 45 3 2 90% 6% 4%
Data Around Each WCAG 2.1 Criterion 1.1.1: Non-text Content 1.2.2: Captions (Prerecorded) 1.2.1: Audio-only and Video-only (Prerecorded) 1.2.2: Captions (Prerecorded) 1.2.3: Audio Description or Full Text Alternative 1.2.4: Captions (Live) 1.2.5: Audio Description Product 1 Partially supports Supports Supports Supports Supports Supports Partially supports Product 2 Partially supports Supports Supports Supports Supports Supports Partially supports Product 3 Partially supports Supports Supports Supports Supports Supports Partially supports 1.1.1: Non-text Content 1.2.1: Audio-only and Video-only (Prerecorded) 1.2.2: Captions (Prerecorded) 1.2.3: Audio Description or Full Text Alternative 1.2.4: Captions (Live) 1.2.5: Audio Description Total Supporting Products 6 59 51 52 68 43 Total Partially Supporting Products 57 3 4 10 0 8 Total Not Supporting Products 5 6 13 6 0 17
Other Than Self-Torture, Why Did We Do This? • New Capability to make comparisons between products • Rankings became part of our topline service package summary • Allowed mining of data to surface top problem areas (alt text or closed captions?) • Model was influenced by customers for a 80% fully supports threshold
Percentage Compliant/Distribution Schema • 80 % fully compliant = 40 of the 50 WCAG 2.1 are fully supported • Not good enough, so we provide a distribution of fully supports, partially supports does not support. [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] Elsevier Products February 2023 Support of the 50 A and AA WCAG 2.1 Requirements Fully Supported Partially Supported Does not Support
Comparisons in ACR Scores 34 42 15 8 1 0 2020 2022 Two-year Comparison of VPAT Scores 71% 68% 67% 62% 24% 26% 25% 31% 5% 6% 8% 7% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% BU1 n=19 BU2 n=20 BU3 n=17 BU4 n=6 Business Unit Average Compliance Distribution Across 62 Products Fully Supports Partially Supports Does Not Support
Product Case Study (Evolve vs. Sherpath) • Setting a division target of 80% minimum • Evolve achieves 1st place • Sherpath edges out Evolve • How Evolve reacted to Sherpath 1st place − Indignant response to being overtaken − Wanting a separate ranking for larger vs. smaller products − Led to stressing that compliance rankings are not the only way to determine accessibility • Evolve strikes back the following year to take back 1st place. “The ability to see how our site compares to other sites in the company is great motivation for the squad, they pride themselves on having a #1 position” – Evolve Product Manager
Accessibility Goals for ScienceDirect 2023 Improve VPAT score from 78% to >80% with a dedicated accessibility quarter (Q2) [WCAG 2.1 AA] 1. Teams to dedicate Q2 activities to tackling accessibility concerns left from 2022 VPAT 2. Teams to address accessibility regressions identified in draft 2023 VPAT (due in April) 3. Official (public) VPAT 2023 to be finalized in Q3 “Seeing where our site excels in accessibility and where it falls short makes it easy for us to prioritize and plan improvements” – Product Manager
Deeper Analytics • Most frequently failed checkpoints • Checkpoints which need the most work (failed + partially supports) • Identifying low hanging fruit amongst the weeds 2 4 4 4 5 8 9 17 24 23 24 24 25 30 33 39 44 44 48 48 48 51 54 56 56 56 57 57 57 58 60 60 61 62 63 64 64 64 58 51 55 55 59 53 55 44 24 34 40 40 39 32 13 8 3 20 2 9 16 6 10 2 8 8 4 5 7 6 4 4 3 2 1 0 0 0 4 9 5 5 0 3 0 3 16 7 0 0 0 2 18 17 17 0 14 7 0 7 0 6 0 0 3 2 0 0 0 0 0 0 0 0 0 0 1.3.1: Info and Relationships 3.3.2: Labels or Instructions 4.1.2: Name, Role, Value 1.1.1: Non-text Content 1.4.3: Contrast (Minimum) 2.1.1: Keyboard 2.4.3: Focus Order 2.4.7: Focus Visible 2.4.2: Page Titled 2.4.1: Bypass Blocks 1.4.1: Use of Color 1.3.2: Meaningful Sequence 2.4.4: Link Purpose (In Context) 3.3.1: Error Identification 3.1.1: Language of Page 1.2.5: Audio Description 2.2.1: Timing Adjustable 1.4.5: Images of Text 1.2.2: Captions (Prerecorded) 1.2.3: Audio Description or Full Text Alternative 1.4.4: Resize text 3.1.2: Language of Parts 4.1.1: Parsing 1.2.1: Audio-only and Video-only (Prerecorded) 3.2.3: Consistent Navigation 2.4.5: Multiple Ways 2.2.2: Pause, Stop, Hide 2.1.2: No Keyboard Trap 3.2.2: On Input 1.3.3: Sensory Characteristics 3.2.1: On Focus 2.4.6: Headings and Labels 3.3.3: Error Suggestion 3.2.4: Consistent Identification 1.4.2: Audio Control 3.3.4: Error Prevention (Legal, Financial, Data) 2.3.1: Three Flashes or Below Threshold 1.2.4: Captions (Live) Support for WCAG 2.0 Standard Across 64 Reviewed Elsevier Products
Most Frequently Failed Checkpoints • Page Language, Page Titled are top 4 but 2 of easiest to fix/pass. • Compared to 2019, Audio Description crept from 5th most failed to 2nd most failed. 2 2 3 3 3 4 5 5 6 7 7 7 9 14 16 17 17 18 3.3.1: Error Identification 2.1.2: No Keyboard Trap 2.4.7: Focus Visible 2.2.2: Pause, Stop, Hide 2.1.1: Keyboard 1.3.1: Info and Relationships 4.1.2: Name, Role, Value 1.1.1: Non-text Content 1.2.1: Audio-only and Video-only… 3.1.2: Language of Parts 2.4.1: Bypass Blocks 1.2.3: Audio Description or Full Text… 3.3.2: Labels or Instructions 1.2.2: Captions (Prerecorded) 2.4.2: Page Titled 2.2.1: Timing Adjustable 1.2.5: Audio Description 3.1.1: Language of Page Counts of WCAG 2.0 AA Checkpoints Graded as 'Does Not Support' 64 Elsevier Products
Top 10 Least Supported Checkpoints 24 34 44 55 53 59 55 55 51 58 16 7 3 0 3 0 5 5 9 4 2.4.2: Page Titled 2.4.1: Bypass Blocks 2.4.7: Focus Visible 2.4.3: Focus Order 2.1.1: Keyboard 1.4.3: Contrast (Minimum) 4.1.2: Name, Role, Value 1.1.1: Non-text Content 3.3.2: Labels or Instructions 1.3.1: Info and Relationships WCAG 2.0 AA Checkpoints in Most Need of Remediation
Other Helpful Areas • Helps keep track of total complete VPATs • Approach allows comparing product VPAT scores between versions 34 42 15 8 1 0 2020 2022 Two-year Comparison of VPAT Scores
Drawbacks/Limitations to this Approach • Possible confusion, understanding what this really means − “92% of checkpoints in full compliance!” vs. “92% of site is fully compliant” • Consistency of reviewer ratings • Weights Partially Supports the same as Does not Supports. • Gaming the system is possible by only targeting the easiest checkpoints.
Drawbacks/Limitations to this Approach (Continued) • Example of high impact requirements − e.g., Captions have same weight as Page Language. • Takes deliberate work to maintain data (you better love Excel) • Higher compliance percentage does not equal better accessibility • Some sites are very simple with no videos. Video adds at least 5 checkpoints. − According to W3C on conformance, "If there is no content to which a success criterion applies, the success criterion is satisfied." • Treats all applications on the same scale.
Other Challenges • WCAG updates require rethinking our data structure − We already had to adjust when WCAG 2.1 came out. − Another update is coming with WCAG 2.2. A new strategy is required for this. − WCAG 3 could introduce a whole new way of scoring applications that could render our work here unnecessary. • Excel is a bit cumbersome way to manage an ever-increasing amount of data. • Versioning a document where all team members have access can be difficult. • Ensuring team members know how to update data and graphs.
Conclusions on our Approach • We inform decision makers with analytics to help improve and maintain high levels of accessibility. • Creates a new layer of intelligence around ACRs • Digestible benchmark and ranking system around ACRs proved useful to help answer stakeholders' questions
After Analytics – Based Upon a True Story Jay (Accessibility Team Lead) gets another call from the CTO (played by Ted)
Thank You/ Q&A
Links and References ScienceDirect VPAT/ACR Example ScienceDirect Accessibility Statement ITI VPAT Templates Accessibility Checklist Tool Elsevier Accessibility Statement Contact us at accessibility@elsevier.com
EXTRAS
VPAT/ACR is Just 1 Lens to Measure Accessibility Automated Scanning VPATs User Testing Company Maturity Model “the quick and dirty” “the full compliance picture” “the truth” “the how” P R O S Customers use it too Easy to measure, Easy to improve Industry standard, Can rank products using analytics Is it accessible by a real person? Is it usable? Quick to measure Provides higher assessment C O N S Only finds 30% of issues Amplifies common issues (100 instances of a color violation) All 50 WCAG 2.1 AA Slow to measure, slow to improve, Too many freebies to allow gaming scores. Slow to improve Can be TRANSFORMATIVE Is the company doing it right? Slow to measure, slow to improve. Not just UX/Dev. T I M E <1 hour 1.5 weeks 1.5 weeks 1 year
Why Care About Accessibility Data Analytics? • Accessibility reviews/VPATs generate complex data • People want to set benchmarks and goals around initiatives • What is the difference between a good and bad score for VPATs? • Organizational drive to make data driven decisions • "How is my product doing compared to other Elsevier products?" • Hidden insights may lie within complex data. • What are the main trends? • Identify global issues across many products.
Short History of VPAT Evolution In 2008 we only had 3 VPATs for all Elsevier products Today we have 80+ in our database TBD Standards major changes (508 > WCAG 2 > WCAG 2.1) More internal questions about the overall state of accessibility of products
Why Did We Do This? • Quantitative vs. Qualitative data • Most of the review results are qualitative. However, a lot of business decisions are based on quantitative data. • This effort is a way to provide that quantitative data to those stakeholders.

More Related Content

PPTX
Big Data Fundamentals
byCloudera, Inc.
 
PDF
Accessibility for Digital Products along with WCAG .pdf
bymuditadubey2
 
PPTX
Delivering Accessible Content with WCAG 2.0
byMike Paciello
 
PPTX
Accessibility for Content Developer, Designer, Code Developer and Tester
byJatin Kochhar
 
PDF
Implementing WCAG 2.2 into Your Digital Strategy
by3Play Media
 
PPSX
A Developers Role in Web Accessibility
byLyn Sawyer
 
PPTX
Wcag 2.0 level_a_all_ejames
byelianna james
 
DOCX
Section 508-checklist
byLarry Wilson
 
Big Data Fundamentals
byCloudera, Inc.
 
Accessibility for Digital Products along with WCAG .pdf
bymuditadubey2
 
Delivering Accessible Content with WCAG 2.0
byMike Paciello
 
Accessibility for Content Developer, Designer, Code Developer and Tester
byJatin Kochhar
 
Implementing WCAG 2.2 into Your Digital Strategy
by3Play Media
 
A Developers Role in Web Accessibility
byLyn Sawyer
 
Wcag 2.0 level_a_all_ejames
byelianna james
 
Section 508-checklist
byLarry Wilson
 

Similar to CSUN 2023 Analytics.pptx

PPTX
What's New in WCAG 2.1
byJennifer Smith
 
PDF
A Toolkit for Digital Accessibility
by3Play Media
 
PPTX
WCAG 2.2 - An Overview of the New Accessibility Guidelines.pptx
byAbilityNet
 
PPTX
Toolkit for the Digital Accessibility Space
by3Play Media
 
PPT
Accessibility Workshop
byLar Veale
 
PDF
Intro to Web Accessibility
byAshley Dzick
 
PPTX
Accessibility Update: Section 508 and WCAG in a Library 2.0 World
byNina McHale
 
DOCX
Microsoft Word - Sample Report
byMicrosoft Accessibility
 
PDF
Enhancing the User Experience for People with Disabilities
byUXPA International
 
PPTX
Understanding wcag 2.0
bySrinivasu Chakravarthula
 
PPTX
Accessibility (WCAG) Draft 1
byJaime Brown
 
PPTX
Accessibility Overview - 508 and WCAG Compliance
byFrank Walsh
 
PPTX
Accessibility And 508 Compliance In 2009
byEmagination ®
 
PDF
Do you have a website? Do you want to get sued?
byDevin Olson
 
PDF
Usability ≠ Accessibility. An intro to web accessibility for agencies.
byKate Horowitz
 
PDF
Don't Panic! How to perform an accessibility evaluation with limited resources
byMichael Ryan
 
PDF
Digital Accessibility
byADA Site Compliance
 
KEY
Build Accessibly - Community Day 2012
byKaren Mardahl
 
PPTX
My talk at Riga 2016 Web Accessibility Meetup
byVladimir Tomberg
 
PDF
Website Accessibility: It’s the Right Thing to do
byDesignHammer
 
What's New in WCAG 2.1
byJennifer Smith
 
A Toolkit for Digital Accessibility
by3Play Media
 
WCAG 2.2 - An Overview of the New Accessibility Guidelines.pptx
byAbilityNet
 
Toolkit for the Digital Accessibility Space
by3Play Media
 
Accessibility Workshop
byLar Veale
 
Intro to Web Accessibility
byAshley Dzick
 
Accessibility Update: Section 508 and WCAG in a Library 2.0 World
byNina McHale
 
Microsoft Word - Sample Report
byMicrosoft Accessibility
 
Enhancing the User Experience for People with Disabilities
byUXPA International
 
Understanding wcag 2.0
bySrinivasu Chakravarthula
 
Accessibility (WCAG) Draft 1
byJaime Brown
 
Accessibility Overview - 508 and WCAG Compliance
byFrank Walsh
 
Accessibility And 508 Compliance In 2009
byEmagination ®
 
Do you have a website? Do you want to get sued?
byDevin Olson
 
Usability ≠ Accessibility. An intro to web accessibility for agencies.
byKate Horowitz
 
Don't Panic! How to perform an accessibility evaluation with limited resources
byMichael Ryan
 
Digital Accessibility
byADA Site Compliance
 
Build Accessibly - Community Day 2012
byKaren Mardahl
 
My talk at Riga 2016 Web Accessibility Meetup
byVladimir Tomberg
 
Website Accessibility: It’s the Right Thing to do
byDesignHammer
 

More from Ted Gies

PPTX
Accessible Next Level Visualizations
byTed Gies
 
PPTX
CSUN 2024 Simplifying Accessible Data Visualizations - 5 April 2024.pptx
byTed Gies
 
PPTX
Accessible svg charts using aria 2016
byTed Gies
 
PPTX
CSUN 2020 VPATs: For Business or Measure
byTed Gies
 
PPTX
CSUN 2017 VPATs For Business or Measure
byTed Gies
 
PPTX
Accessibility in the Engineering Village CSUN 2019
byTed Gies
 
PPTX
CSUN 2012: ScienceDirect Article Of The Future Collaboration
byTed Gies
 
PPTX
Highcharts- The Next Chapter CSUN 2019
byTed Gies
 
PPTX
CSUN 2020 Accessible Visualizations: Maps, Annotations, and Spark lines
byTed Gies
 
PPTX
CSUN 2018 Dont Play Me - 2 Games in Web Accessibility
byTed Gies
 
PPTX
CSUN 2023 Automated Descriptions 3 March 2023 TG.pptx
byTed Gies
 
PPT
World Usability Day 2012 ARIA
byTed Gies
 
Accessible Next Level Visualizations
byTed Gies
 
CSUN 2024 Simplifying Accessible Data Visualizations - 5 April 2024.pptx
byTed Gies
 
Accessible svg charts using aria 2016
byTed Gies
 
CSUN 2020 VPATs: For Business or Measure
byTed Gies
 
CSUN 2017 VPATs For Business or Measure
byTed Gies
 
Accessibility in the Engineering Village CSUN 2019
byTed Gies
 
CSUN 2012: ScienceDirect Article Of The Future Collaboration
byTed Gies
 
Highcharts- The Next Chapter CSUN 2019
byTed Gies
 
CSUN 2020 Accessible Visualizations: Maps, Annotations, and Spark lines
byTed Gies
 
CSUN 2018 Dont Play Me - 2 Games in Web Accessibility
byTed Gies
 
CSUN 2023 Automated Descriptions 3 March 2023 TG.pptx
byTed Gies
 
World Usability Day 2012 ARIA
byTed Gies
 

Recently uploaded

PDF
Hart, Inc. M&A Data Transition Checklist
byHart, Inc.
 
PDF
Adding Copilot+ PCs with Snapdragon to your HP fleet won’t require IT deploym...
byPrincipled Technologies
 
PDF
Just Eat Data Scraping For Restaurant Reviews And Pricing.pdf
byfusiondata2026
 
PDF
Special Reeling Cable Specification _ Anhui Feichun Special Cable Co., Ltd_.pdf
byAnhui Feichun Special Cable Co., Ltd.
 
PDF
2006 IEEE International Solid-State Circuits Conference
bySlide_N
 
PDF
Best 10 Dedicated Servers in Amsterdam, Netherlands (2026 Enterprise Edition)...
byAtal Networks
 
PDF
6 Insights from the Patron Saint of AI Failure
byScott M. Graffius
 
PDF
Top 5 Best Dedicated Server Providers in Los Angeles (Updated Edition)
byAtal Networks
 
PDF
A paper in the leading academic journal Telecommunication Policy cited Scott ...
byScott M. Graffius
 
PPTX
Microsoft Azure News - January 2026 - BAUG
byDaniel Toomey
 
PPTX
Robot Vacuums are Cleaning Up. The global robot vacuum segment has high marke...
byRobert March
 
PDF
Top 10 Dedicated Server Providers in Stockholm (Updated Edition).pdf
byAtal Networks
 
PDF
Why Enterprises Are Moving to Dedicated Servers in the Netherlands
byAtal Networks
 
PPTX
Apache Kafka 101 for Techies in IT dep.pptx
byamulyareddyk97
 
PPTX
computer science class 11 ( phishing and fraud mails) .pptx
bybalajichess314
 
PDF
Zniber Partners. Audience Foresight for Confident Decisions
bySamuel Zniber
 
PPTX
WIFI Enabled 8 by 32 Matrix.pptx for final year project
bymdsabbirhossain524856
 
PPTX
The Abomination of AI – keynote at ICoSCI 2026
byAlan Dix
 
PDF
Salesforce Sales Cloud and Generative AI Present a Fresh Approach to Personal...
byMinuscule Technologies
 
PPTX
Fortify Your Digital Defenses with ServiceNow Security Operations by Suma Soft
byGavin Ellis
 
Hart, Inc. M&A Data Transition Checklist
byHart, Inc.
 
Adding Copilot+ PCs with Snapdragon to your HP fleet won’t require IT deploym...
byPrincipled Technologies
 
Just Eat Data Scraping For Restaurant Reviews And Pricing.pdf
byfusiondata2026
 
Special Reeling Cable Specification _ Anhui Feichun Special Cable Co., Ltd_.pdf
byAnhui Feichun Special Cable Co., Ltd.
 
2006 IEEE International Solid-State Circuits Conference
bySlide_N
 
Best 10 Dedicated Servers in Amsterdam, Netherlands (2026 Enterprise Edition)...
byAtal Networks
 
6 Insights from the Patron Saint of AI Failure
byScott M. Graffius
 
Top 5 Best Dedicated Server Providers in Los Angeles (Updated Edition)
byAtal Networks
 
A paper in the leading academic journal Telecommunication Policy cited Scott ...
byScott M. Graffius
 
Microsoft Azure News - January 2026 - BAUG
byDaniel Toomey
 
Robot Vacuums are Cleaning Up. The global robot vacuum segment has high marke...
byRobert March
 
Top 10 Dedicated Server Providers in Stockholm (Updated Edition).pdf
byAtal Networks
 
Why Enterprises Are Moving to Dedicated Servers in the Netherlands
byAtal Networks
 
Apache Kafka 101 for Techies in IT dep.pptx
byamulyareddyk97
 
computer science class 11 ( phishing and fraud mails) .pptx
bybalajichess314
 
Zniber Partners. Audience Foresight for Confident Decisions
bySamuel Zniber
 
WIFI Enabled 8 by 32 Matrix.pptx for final year project
bymdsabbirhossain524856
 
The Abomination of AI – keynote at ICoSCI 2026
byAlan Dix
 
Salesforce Sales Cloud and Generative AI Present a Fresh Approach to Personal...
byMinuscule Technologies
 
Fortify Your Digital Defenses with ServiceNow Security Operations by Suma Soft
byGavin Ellis
 

CSUN 2023 Analytics.pptx

  • 1.
    Accessibility Analytics – BeingData Driven With VPATs CSUN Thursday March 16 2023 Platinum 7-8 9:20 am Jay Nemchik Ted Gies
  • 2.
    Elsevier Mission Elsevier helpsresearchers and healthcare professionals advance science and improve health outcomes for the benefit of society. 2
  • 3.
    Back in 2016– Based Upon a True Story “As our leadership team is planning our objectives and key results for this year, could you help identify some accessibility metrics.”
  • 4.
    VPAT/ACR Criteria Conformance Level Remarks and Explanations 1.1.1Non-text Content (Level A) 1.2.1 Audio-only and Video-only (Prerecorded) (Level A) 1.2.2 Captions (Prerecorded) (Level A) 1.2.3 Audio Description or Media Alternative (Prerecorded) (Level A) 1.3.1 Info and Relationships (Level A) 1.3.2 Meaningful Sequence (Level A) 1.3.3 Sensory Characteristics (Level A) 1.4.1 Use of Color (Level A) 1.4.2 Audio Control (Level A) 2.1.1 Keyboard (Level A) 2.1.2 No Keyboard Trap (Level A) 2.1.4 Character Key Shortcuts (Level A 2.1 only) 2.2.1 Timing Adjustable (Level A) 2.2.2 Pause, Stop, Hide (Level A) 2.3.1 Three Flashes or Below Threshold (Level A) 2.4.1 Bypass Blocks (Level A) 2.4.2 Page Titled (Level A) 2.4.3 Focus Order (Level A) 2.4.4 Link Purpose (In Context) (Level A) 2.5.1 Pointer Gestures (Level A 2.1 only) 2.5.2 Pointer Cancellation (Level A 2.1 only) 2.5.3 Label in Name (Level A 2.1 only) 2.5.4 Motion Actuation (Level A 2.1 only) 3.1.1 Language of Page (Level A) 3.2.1 On Focus (Level A) 3.2.2 On Input (Level A) 3.3.1 Error Identification (Level A) 3.3.2 Labels or Instructions (Level A) 4.1.1 Parsing (Level A)
  • 5.
    Example of 1VPAT/ACR Checkpoint WCAG 2.1 Checkpoint Supporting Features Remarks 1.1.1: Non-Text Content (A) Provide text alternatives for non- text content (e.g. images) Partially Supports The product presents non-text content with a text alternative that serves the equivalent purpose. Supporting Remarks Linked icons are labelled such as Help and Institution. Exceptions A rating of Partially Supports has been given for the following reasons: • There are images within the pages that do not have an alt attribute or aria- label set. • There are buttons using icons only that are missing accessible name/labels. • There are values within the alt attribute that are not descriptive. • Invalid use of the alt attribute.
  • 6.
    | 6 ACR Evaluationof 1 Product WCAG 2.1 Success Criterion Evaluation 1.1.1 Non-text Content A Partially Supports 1.2.1 Audio-only and Video-only (Prerecorded) A Supports (N/A) 1.2.2 Captions (Prerecorded) A Supports (N/A) 1.2.3 Audio Description /Full Text Alternative A Supports (N/A) 1.2.4 Captions (Live) AA Supports (N/A) 1.2.5 Audio Description AA Supports (N/A) 1.3.1 Info and Relationships A Supports 1.3.2 Meaningful Sequence A Partially Supports 1.3.3 Sensory Characteristics A Supports 1.3.4 Orientation AA Supports 1.3.5 Identify Input Purpose AA Does not Support 1.4.1 Use of Color A Partially Supports 1.4.2 Audio Control A Supports (N/A) 1.4.3 Contrast (Minimum) AA Partially Supports 1.4.4 Resize text AA Supports 1.4.5 Images of Text AA Supports 1.4.10 Reflow AA Supports 1.4.11 Non-text Contrast AA Partially Supports 1.4.12 Text Spacing AA Supports 1.4.13 Content on Hover or Focus AA Does not Support 2.1.1 Keyboard A Partially Supports 2.1.2 No Keyboard Trap A Supports 2.1.4 Character Key Shortcuts A Supports (N/A) 2.2.1 Timing Adjustable A Supports 2.2.2 Pause, Stop, Hide A Supports (N/A) WCAG 2.1 Success Criterion Evaluation 2.3.1 Three Flashes or Below Threshold A Supports (N/A) 2.4.1 Bypass Blocks A Supports 2.4.2 Page Titled A Supports 2.4.3 Focus Order A Supports 2.4.4 Link Purpose (In Context) A Partially Supports 2.4.5 Multiple Ways AA Supports 2.4.6 Headings and Labels AA Supports 2.4.7 Focus Visible AA Supports 2.5.1 Pointer Gestures A Supports (N/A) 2.5.2 Pointer Cancellation A Supports 2.5.3 Label in Name A Supports 2.5.4 Motion Actuation A Supports (N/A) 3.1.1 Language of Page A Supports 3.1.2 Language of Parts AA Supports (N/A) 3.2.1 On Focus A Supports 3.2.2 On Input A Supports 3.2.3 Consistent Navigation AA Partially Supports 3.2.4 Consistent Identification AA Supports 3.3.1 Error Identification A Supports 3.3.2 Labels or Instructions A Partially Supports 3.3.3 Error Suggestion AA Supports 3.3.4 Error Prevention (Legal, Financial, Data) AA Supports 4.1.1 Parsing A Supports 4.1.2 Name, Role, Value A Partially Supports 4.1.3 Status Messages AA Supports
  • 7.
    Problem Statements • Managersand employees use numbers to measure things and set goals. • Web accessibility data generated from an ACR is mostly qualitative. • There is a mismatch between the nature of ACR accessibility data and providing simple quantitative reports. Goal: We want to inform decision makers with analytics to help improve and maintain high levels of accessibility
  • 8.
    Our Approach: Createan Excel Database out of Mined ACR Data • Originally input 38 websites ACR Ratings into Excel • Translated the conformance grading to percentages in Excel • Used Excel Pivot Tables/Pivot Charts to generate graphs Total Supports Total Partially supports Total Does not Support Percentage Supports Percentage Partially Supports Percentage Does not Support Product 1 40 8 2 80% 16% 4% Product 2 38 3 9 76% 6% 18% Product 3 45 3 2 90% 6% 4%
  • 9.
    Data Around EachWCAG 2.1 Criterion 1.1.1: Non-text Content 1.2.2: Captions (Prerecorded) 1.2.1: Audio-only and Video-only (Prerecorded) 1.2.2: Captions (Prerecorded) 1.2.3: Audio Description or Full Text Alternative 1.2.4: Captions (Live) 1.2.5: Audio Description Product 1 Partially supports Supports Supports Supports Supports Supports Partially supports Product 2 Partially supports Supports Supports Supports Supports Supports Partially supports Product 3 Partially supports Supports Supports Supports Supports Supports Partially supports 1.1.1: Non-text Content 1.2.1: Audio-only and Video-only (Prerecorded) 1.2.2: Captions (Prerecorded) 1.2.3: Audio Description or Full Text Alternative 1.2.4: Captions (Live) 1.2.5: Audio Description Total Supporting Products 6 59 51 52 68 43 Total Partially Supporting Products 57 3 4 10 0 8 Total Not Supporting Products 5 6 13 6 0 17
  • 10.
    Other Than Self-Torture,Why Did We Do This? • New Capability to make comparisons between products • Rankings became part of our topline service package summary • Allowed mining of data to surface top problem areas (alt text or closed captions?) • Model was influenced by customers for a 80% fully supports threshold
  • 11.
    Percentage Compliant/Distribution Schema •80 % fully compliant = 40 of the 50 WCAG 2.1 are fully supported • Not good enough, so we provide a distribution of fully supports, partially supports does not support. [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] Elsevier Products February 2023 Support of the 50 A and AA WCAG 2.1 Requirements Fully Supported Partially Supported Does not Support
  • 12.
    Comparisons in ACRScores 34 42 15 8 1 0 2020 2022 Two-year Comparison of VPAT Scores 71% 68% 67% 62% 24% 26% 25% 31% 5% 6% 8% 7% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% BU1 n=19 BU2 n=20 BU3 n=17 BU4 n=6 Business Unit Average Compliance Distribution Across 62 Products Fully Supports Partially Supports Does Not Support
  • 13.
    Product Case Study(Evolve vs. Sherpath) • Setting a division target of 80% minimum • Evolve achieves 1st place • Sherpath edges out Evolve • How Evolve reacted to Sherpath 1st place − Indignant response to being overtaken − Wanting a separate ranking for larger vs. smaller products − Led to stressing that compliance rankings are not the only way to determine accessibility • Evolve strikes back the following year to take back 1st place. “The ability to see how our site compares to other sites in the company is great motivation for the squad, they pride themselves on having a #1 position” – Evolve Product Manager
  • 14.
    Accessibility Goals forScienceDirect 2023 Improve VPAT score from 78% to >80% with a dedicated accessibility quarter (Q2) [WCAG 2.1 AA] 1. Teams to dedicate Q2 activities to tackling accessibility concerns left from 2022 VPAT 2. Teams to address accessibility regressions identified in draft 2023 VPAT (due in April) 3. Official (public) VPAT 2023 to be finalized in Q3 “Seeing where our site excels in accessibility and where it falls short makes it easy for us to prioritize and plan improvements” – Product Manager
  • 15.
    Deeper Analytics • Mostfrequently failed checkpoints • Checkpoints which need the most work (failed + partially supports) • Identifying low hanging fruit amongst the weeds 2 4 4 4 5 8 9 17 24 23 24 24 25 30 33 39 44 44 48 48 48 51 54 56 56 56 57 57 57 58 60 60 61 62 63 64 64 64 58 51 55 55 59 53 55 44 24 34 40 40 39 32 13 8 3 20 2 9 16 6 10 2 8 8 4 5 7 6 4 4 3 2 1 0 0 0 4 9 5 5 0 3 0 3 16 7 0 0 0 2 18 17 17 0 14 7 0 7 0 6 0 0 3 2 0 0 0 0 0 0 0 0 0 0 1.3.1: Info and Relationships 3.3.2: Labels or Instructions 4.1.2: Name, Role, Value 1.1.1: Non-text Content 1.4.3: Contrast (Minimum) 2.1.1: Keyboard 2.4.3: Focus Order 2.4.7: Focus Visible 2.4.2: Page Titled 2.4.1: Bypass Blocks 1.4.1: Use of Color 1.3.2: Meaningful Sequence 2.4.4: Link Purpose (In Context) 3.3.1: Error Identification 3.1.1: Language of Page 1.2.5: Audio Description 2.2.1: Timing Adjustable 1.4.5: Images of Text 1.2.2: Captions (Prerecorded) 1.2.3: Audio Description or Full Text Alternative 1.4.4: Resize text 3.1.2: Language of Parts 4.1.1: Parsing 1.2.1: Audio-only and Video-only (Prerecorded) 3.2.3: Consistent Navigation 2.4.5: Multiple Ways 2.2.2: Pause, Stop, Hide 2.1.2: No Keyboard Trap 3.2.2: On Input 1.3.3: Sensory Characteristics 3.2.1: On Focus 2.4.6: Headings and Labels 3.3.3: Error Suggestion 3.2.4: Consistent Identification 1.4.2: Audio Control 3.3.4: Error Prevention (Legal, Financial, Data) 2.3.1: Three Flashes or Below Threshold 1.2.4: Captions (Live) Support for WCAG 2.0 Standard Across 64 Reviewed Elsevier Products
  • 16.
    Most Frequently FailedCheckpoints • Page Language, Page Titled are top 4 but 2 of easiest to fix/pass. • Compared to 2019, Audio Description crept from 5th most failed to 2nd most failed. 2 2 3 3 3 4 5 5 6 7 7 7 9 14 16 17 17 18 3.3.1: Error Identification 2.1.2: No Keyboard Trap 2.4.7: Focus Visible 2.2.2: Pause, Stop, Hide 2.1.1: Keyboard 1.3.1: Info and Relationships 4.1.2: Name, Role, Value 1.1.1: Non-text Content 1.2.1: Audio-only and Video-only… 3.1.2: Language of Parts 2.4.1: Bypass Blocks 1.2.3: Audio Description or Full Text… 3.3.2: Labels or Instructions 1.2.2: Captions (Prerecorded) 2.4.2: Page Titled 2.2.1: Timing Adjustable 1.2.5: Audio Description 3.1.1: Language of Page Counts of WCAG 2.0 AA Checkpoints Graded as 'Does Not Support' 64 Elsevier Products
  • 17.
    Top 10 LeastSupported Checkpoints 24 34 44 55 53 59 55 55 51 58 16 7 3 0 3 0 5 5 9 4 2.4.2: Page Titled 2.4.1: Bypass Blocks 2.4.7: Focus Visible 2.4.3: Focus Order 2.1.1: Keyboard 1.4.3: Contrast (Minimum) 4.1.2: Name, Role, Value 1.1.1: Non-text Content 3.3.2: Labels or Instructions 1.3.1: Info and Relationships WCAG 2.0 AA Checkpoints in Most Need of Remediation
  • 18.
    Other Helpful Areas •Helps keep track of total complete VPATs • Approach allows comparing product VPAT scores between versions 34 42 15 8 1 0 2020 2022 Two-year Comparison of VPAT Scores
  • 19.
    Drawbacks/Limitations to thisApproach • Possible confusion, understanding what this really means − “92% of checkpoints in full compliance!” vs. “92% of site is fully compliant” • Consistency of reviewer ratings • Weights Partially Supports the same as Does not Supports. • Gaming the system is possible by only targeting the easiest checkpoints.
  • 20.
    Drawbacks/Limitations to thisApproach (Continued) • Example of high impact requirements − e.g., Captions have same weight as Page Language. • Takes deliberate work to maintain data (you better love Excel) • Higher compliance percentage does not equal better accessibility • Some sites are very simple with no videos. Video adds at least 5 checkpoints. − According to W3C on conformance, "If there is no content to which a success criterion applies, the success criterion is satisfied." • Treats all applications on the same scale.
  • 21.
    Other Challenges • WCAGupdates require rethinking our data structure − We already had to adjust when WCAG 2.1 came out. − Another update is coming with WCAG 2.2. A new strategy is required for this. − WCAG 3 could introduce a whole new way of scoring applications that could render our work here unnecessary. • Excel is a bit cumbersome way to manage an ever-increasing amount of data. • Versioning a document where all team members have access can be difficult. • Ensuring team members know how to update data and graphs.
  • 22.
    Conclusions on ourApproach • We inform decision makers with analytics to help improve and maintain high levels of accessibility. • Creates a new layer of intelligence around ACRs • Digestible benchmark and ranking system around ACRs proved useful to help answer stakeholders' questions
  • 23.
    After Analytics –Based Upon a True Story Jay (Accessibility Team Lead) gets another call from the CTO (played by Ted)
  • 24.
  • 25.
    Links and References ScienceDirectVPAT/ACR Example ScienceDirect Accessibility Statement ITI VPAT Templates Accessibility Checklist Tool Elsevier Accessibility Statement Contact us at accessibility@elsevier.com
  • 26.
  • 27.
    VPAT/ACR is Just1 Lens to Measure Accessibility Automated Scanning VPATs User Testing Company Maturity Model “the quick and dirty” “the full compliance picture” “the truth” “the how” P R O S Customers use it too Easy to measure, Easy to improve Industry standard, Can rank products using analytics Is it accessible by a real person? Is it usable? Quick to measure Provides higher assessment C O N S Only finds 30% of issues Amplifies common issues (100 instances of a color violation) All 50 WCAG 2.1 AA Slow to measure, slow to improve, Too many freebies to allow gaming scores. Slow to improve Can be TRANSFORMATIVE Is the company doing it right? Slow to measure, slow to improve. Not just UX/Dev. T I M E <1 hour 1.5 weeks 1.5 weeks 1 year
  • 28.
    Why Care AboutAccessibility Data Analytics? • Accessibility reviews/VPATs generate complex data • People want to set benchmarks and goals around initiatives • What is the difference between a good and bad score for VPATs? • Organizational drive to make data driven decisions • "How is my product doing compared to other Elsevier products?" • Hidden insights may lie within complex data. • What are the main trends? • Identify global issues across many products.
  • 29.
    Short History ofVPAT Evolution In 2008 we only had 3 VPATs for all Elsevier products Today we have 80+ in our database TBD Standards major changes (508 > WCAG 2 > WCAG 2.1) More internal questions about the overall state of accessibility of products
  • 30.
    Why Did WeDo This? • Quantitative vs. Qualitative data • Most of the review results are qualitative. However, a lot of business decisions are based on quantitative data. • This effort is a way to provide that quantitative data to those stakeholders.

Editor's Notes

  • #3 Elsevier helps researchers and healthcare professionals advance science and improve health outcomes for the benefit of society. We are the largest publisher of books and journals in Science Prolific publisher in disability research and accessibility Lancet, Cell, ScienceDirect, Scopus, Clinical Key Books by Jonathan Lazar
  • #4 Ted: could play managing director of research products business unit. Hearing great things about the accessibility team. Thank you. As our leadership team is planning our objectives and key results for this year, could you help identify some accessibility metrics. Let’s start by establishing a baseline accessibility metric and then set an achievable target for teams to meet. Try to come up with something in lay person’s terms. See you in a month. So that was a skit based upon a recurring question our a11y team receives, typically from PMs to help establish meaningful metrics around accessibility. So how do we identify some accessibility metrics, how go about that?
  • #5 Before we help the managing director and need for simple metrics, first let’s define the nature of accessibility conformance data. And to do that let’s also go through some terms. Has anyone ever heard of a VPAT? Voluntary Product Accessibility Template, ACR = Accessibility Conformance Report? An accessibility expert uses the VPAT to evaluate the accessibility conformance of a digital product. This is a blank VPAT, it shows a cover page of the WCAG edition and the success criteria overview for WCAG 2.1.
  • #6 Taking an example checkpoint from the VPAT, here’s 1.1.1 with a grading of partially supports. The Remarks provides a narrative of supporting remarks and exceptions. Some icons are labeled with alt text, but there are some exceptions.
  • #7 When completed, the ACR represents 50 checkpoints and serves as like the nutrition facts of accessibility for a digital product. Ours are about 12 pages in length. This particular ACR shows a website that is mostly compliant. Of the 50 WCAG 38 fully supported, 10 partially supports, and 2 does not support. Now with all of this detail, is it easy to explain if 1 website has improved since last VPAT? How about a program tracking 50 or 100 websites? Keep in mind there isn’t an official scoring system provided with VPATs
  • #8 So what is the problem then we are trying to solve? The problem is the mismatch between the nature of mostly qualitiatve ACR accessibility data and the need for simple metrics. As in our skit, managers want to know simple easy to use scores and answers questions like “did my products improve since last VPAT evaluation”? Important decision making. What if a website owner has a false sense of security that they have good accessibility? We want to inform decisions makers with analytics to help improve and maintain high levels of accessibility How then? With some metrics.
  • #9 So what did we do to help solve the problem of needing to create simple quantifiable metrics around accessibility? Created our own excel database using the gradings for each WCAG checkpoints excerpted ACRs. Back in 2019 we started with 38 rows of 38 WCAG 2.0, 1444 data points. As shown in this example table, each product has a row in the Excel sheet reflecting the evaluation from the ACR. Each product shows the total number of checkpoints (fully supports, partially supports, does not support) We provide a count and a percentage of the total number of WCAG checkpoints. This db is the basis of our analytics work
  • #10 For each product we also added the individual ratings of each of the 50 WCAG criteria. This then allowed us to total the numbers of fully supported, partially supported, and not supported checkpoints across products. So for this table it shows that for non text content, 6 products fully support non-text content, 57 partially support, and 5 do not support
  • #11 What did creating the database allow us to do? There was some warming up to Excel. This data provided a new capability to compare VPAT ratings between products and rank them. And ranking became a new part of our topline VPAT service for the company. Also allowed teams the ability to set a benchmark, let’s say to improve a score from 70% to 80%. Guess what we have been asked by customers to meet a certain fully supported score Jay next slides
  • #15 TO JAY – I added this slide from Astrid to illustrate how our work translates into a real annual objective. If you like it we can keep it PM quote related to the overall analytics…  “The ability to see how our site compares to other sites in the company is great motivation for the squad, they pride themselves on having a #1 position”   PM quote Related to Evolve analytics…  “Seeing where our site excels in accessibility and where it falls short makes it easy for us to prioritize and plan improvements”
  • #23 TG In Summary: Our problem statement is now more of a solution statement instead of an aim We inform decisions makers with analytics to help improve and maintain high levels of accessibility Our approach created a new layer of intelligence around our ACRs new comparison ability that didn’t exist before Driving competition between teams is good Motivates New way to view most commons problem WCAG areas
  • #24 Ring, Hi Jay, this is Ted Gies, Managing Director of Research Products. How’s everything Very curious to know what you’ve been working on Ted looking very interested in Jay’s report Great Objective Setting Call you again next month?
  • #28 *MAYBE LOSE THIS SLIDE OR PUT POINTS INTO THE DRAWBACKS SLIDE* At Elsevier we use VPATs for business and measure Eventhough there are other ways to measure A11y and usasbility, user testing the truth. The VPAT is our preferred benchmark because 1) it’s what our customers ask for (US Govt, Colleges around the world). 2) it’s comprehensive and in line with international standards. We’ve been creating VPATs since 2007 to the delight of 100s of customers. We like VPATs because they provide the full compliance picture allowing us to use measure accessibility using manual and automated tools to give a well rounded view against WCAG 2.1. VPAT is one lens of 4 we use around a11y, we want to use all of these lenses, each will ensure accessibility. There are data analytics available in tools (ARC, PopeTech) however all based upon machine testable criterion (20%-30% can be automated of the story) 37 of WCAG 2.1 cannot be automated (Sensory characteristics, Pointer Gestures, Error Prevention).