SlideShare a Scribd company logo
Getting one voice:
tuning up experts’ assessment in
     measuring accessibility
                                      Silvia Mirri
                              Ludovico A. Muratori
                                  Paola Salomoni
                                 Matteo Battistelli

                    Department of Computer Science
                               University of Bologna
Summary


   Introduction
   Automatic and manual accessibility evaluations
   Our proposed metric
   Conclusions and future works




W4A 2012 – April 16th&17th, 2012 - Lyon, France      2
Introduction

                    Web accessibility evaluations

             automatic tools + human assessment


   Metrics quantify accessibility level or barriers, providing
   numerical synthesis
   •    automatic tools return binary values
   •    human assessments are subjective and can get values from a
        continuous range


W4A 2012 – April 16th&17th, 2012 - Lyon, France                      3
Our main goal

   Providing a metric to measure how far a Web
   page is from its accessibility version, taking into
   account

   • integration of human assessments with automatic
     evaluations on the same target
   • many humans assessments




W4A 2012 – April 16th&17th, 2012 - Lyon, France          4
Steps

   1. Mixing up the manual evaluation together with the
      automatic ones


   2. Combining the assessments coming from different
      human evaluations
        •    Values distributed into a given range
        •    The more experts' assessments contribute to compute a
             value, the more this value is stable and reliable




W4A 2012 – April 16th&17th, 2012 - Lyon, France                      5
Automatic and manual evaluations: an example

   Combination between the IMG element and its ALT
   attribute:
   1. If the ALT attribute is omitted the automatic check outputs 1
   2. If the ALT attribute is present the automatic check outputs 0

   Manual evaluation might state that:
   • there is no lack of information once the images are hidden (this
     can happen in case 1, if the image is a pure decorative one)
   • there is a lack of information once the image is hidden




W4A 2012 – April 16th&17th, 2012 - Lyon, France                         6
Our metric
  • A first version of our metric (Barriers Impact Factor) is
    computed on the basis of a barrier-error association
    table
  • This table reports the list of assistive
    technologies/disabilities affected by any error
       •    screen reader/blindness
       •    screen magnifier/low vision
       •    color blindness
       •    input device independence/movement impairments
       •    deafness
       •    cognitive disabilities
       •    photosensitive epilepsy


W4A 2012 – April 16th&17th, 2012 - Lyon, France                 7
Our metric

 • Comparing automatic checks with WCAG 2.0 success
   criteria and identified relationships

                                               a certain error occurs or a
          A check fails
                                              manual control is necessary

 • Each barrier is related to one success criterion and to
   one level of conformity (A, AA or AAA)
 • Manual evaluations take values on the [0, 1] real
   numbers interval:
      •     1 means that an accessibility error occurs
      •     0 means the absence of that accessibility error

W4A 2012 – April 16th&17th, 2012 - Lyon, France                              8
Our metric




W4A 2012 – April 16th&17th, 2012 - Lyon, France   9
Weighting automatic and manual checks

     1. m(i)=a(i): the formula is a mere average among automatically
     and manually detected errors
     2. m(i)>a(i): the failure in manual assessment is considered more
     significant than the automatic one
     3. m(i)<a(i): the failure in automatic assessment is considered
        more significant than the manual one

                         AUTOMATIC                                AUTOMATIC
                         0        1                              0       1

                   [0,   I       III                       [0,   I       II
          MANUAL




                                                  MANUAL
                   ,1]   II      IV                        ,1]   III     IV




W4A 2012 – April 16th&17th, 2012 - Lyon, France                               10
Some considerations

  • The more human operators provide evaluations about
    an accessibility barrier and the more the value of
    accessibility level is reliable
  • Behavior similar to online rating systems ones
  • New users rating can be influenced by already
    expressed evaluations from other users
  • Variance must be considered so as to reinforce the
    computed accessibility level



W4A 2012 – April 16th&17th, 2012 - Lyon, France          11
A first assessment
                 PAGE CONTENT                       MANUAL EVALUATIONS

                                                        0,7        Expert A

                                                         1         Expert B

                                                        0,8        Expert C

                                                         1         Expert D

                  ALT=“Image”                           0,5        Expert E



              NO LINK, NO TITLE
                                                      CBIF



             AUTOMATIC EVALUATION
                                                        m=2
                                                        a=1
             0 (no known errors,                    Average=0,8               CBIF=0,53
             1 alert: placeholder                 Variance=0,036
                  detected)


W4A 2012 – April 16th&17th, 2012 - Lyon, France                                           12
Conclusions

  • We have defined an accessibility metric with the aim to
    evaluate barriers as a whole, combining results
    provided by using automatic tools and manual
    evaluations done by experts
  • The metric has been preliminary tested by measuring
    accessibility barriers in several local public
    administration Web sites
  • Five experts are manually evaluating barriers related to
    WCAG 2.0 1.1.1 (using an automatic monitoring system
    to verify the page content and to collect data from
    manual evaluations)

W4A 2012 – April 16th&17th, 2012 - Lyon, France               13
Future Work


  • Propose and discuss weights for the whole WCAG 2.0
    set of barriers

  • Investigate how the number of experts involved in the
    evaluation, together with their rating variance, could
    influence the reliability of the computed values




W4A 2012 – April 16th&17th, 2012 - Lyon, France              14
Contacts


       Thank you for your attention!

       For further information:
           silvia.mirri@unibo.it




W4A 2012 – April 16th&17th, 2012 - Lyon, France   15

More Related Content

Similar to Mirri w4a2012

Empirical evaluation in 2020: how big, how beautiful?
Empirical evaluation in 2020: how big, how beautiful?Empirical evaluation in 2020: how big, how beautiful?
Empirical evaluation in 2020: how big, how beautiful?
Massimiliano Di Penta
 
Agile Tour Brussels 2015 : Lean UX workshop
Agile Tour Brussels 2015 : Lean UX workshopAgile Tour Brussels 2015 : Lean UX workshop
Agile Tour Brussels 2015 : Lean UX workshop
Frederik Vannieuwenhuyse
 
A macroscopic web accessibility evaluation at different processing phases
A macroscopic web accessibility evaluation at different processing phasesA macroscopic web accessibility evaluation at different processing phases
A macroscopic web accessibility evaluation at different processing phases
Nádia Fernandes
 
EdenRW18-OpenVM
EdenRW18-OpenVMEdenRW18-OpenVM
EdenRW18-OpenVM
Gemma Tur
 
Upa Conference Loic Nunez 18 June2008
Upa Conference Loic Nunez 18 June2008Upa Conference Loic Nunez 18 June2008
Upa Conference Loic Nunez 18 June2008
Loic Nunez
 
Project "Visual Lean: Audit Tracking System" by LeanSoft company
Project "Visual Lean: Audit Tracking System" by LeanSoft companyProject "Visual Lean: Audit Tracking System" by LeanSoft company
Project "Visual Lean: Audit Tracking System" by LeanSoft company
LeanSoft_Rus
 
Fehlmann and Kranich - Measuring tests using cosmic
Fehlmann and Kranich - Measuring tests using cosmicFehlmann and Kranich - Measuring tests using cosmic
Fehlmann and Kranich - Measuring tests using cosmic
International Software Benchmarking Standards Group (ISBSG)
 
Mirri At W4a2009
Mirri At W4a2009Mirri At W4a2009
Mirri At W4a2009
guest950315
 
Answers siebel-set-i
Answers siebel-set-iAnswers siebel-set-i
Answers siebel-set-i
Ucome
 
Main Presentantion of the EightProject Meeting
Main Presentantion of the EightProject MeetingMain Presentantion of the EightProject Meeting
Main Presentantion of the EightProject Meeting
Imede
 
Presentacion Dcai 2010
Presentacion Dcai 2010Presentacion Dcai 2010
Presentacion Dcai 2010
Victor Codina
 
E Quality Portfolio
E Quality Portfolio E Quality Portfolio
E Quality Portfolio
Imede
 
Seeking value by Michael Ballé at the European Lean IT Summit 2012
Seeking value by Michael Ballé at the European Lean IT Summit 2012Seeking value by Michael Ballé at the European Lean IT Summit 2012
Seeking value by Michael Ballé at the European Lean IT Summit 2012
Institut Lean France
 
5 Steps to Data-driven Training
5 Steps to Data-driven Training5 Steps to Data-driven Training
5 Steps to Data-driven Training
Lambda Solutions
 
Introduction to OSLC
Introduction to OSLCIntroduction to OSLC
Introduction to OSLC
openservices
 
Evaluation in hci
Evaluation in hciEvaluation in hci
Evaluation in hci
sajid rao
 
INDUSTRIAL TRAINING EVALUATION MANAGEMENT SYSTEM (IT-EMS) PRESENTATION
INDUSTRIAL TRAINING EVALUATION MANAGEMENT SYSTEM (IT-EMS) PRESENTATIONINDUSTRIAL TRAINING EVALUATION MANAGEMENT SYSTEM (IT-EMS) PRESENTATION
INDUSTRIAL TRAINING EVALUATION MANAGEMENT SYSTEM (IT-EMS) PRESENTATION
PRIYA DARSHINI A/P VEJAN
 
Value stream mapping for complex processes (innovation, Lean, service design)
Value stream mapping for complex processes (innovation, Lean, service design) Value stream mapping for complex processes (innovation, Lean, service design)
Value stream mapping for complex processes (innovation, Lean, service design)
Teemu Toivonen
 
Enhancing the assessment experience through closer integration between the SR...
Enhancing the assessment experience through closer integration between the SR...Enhancing the assessment experience through closer integration between the SR...
Enhancing the assessment experience through closer integration between the SR...
RichardM_Walker
 
UI Integrations Test
UI Integrations TestUI Integrations Test
UI Integrations Test
Herwidodo kusumobroto
 

Similar to Mirri w4a2012 (20)

Empirical evaluation in 2020: how big, how beautiful?
Empirical evaluation in 2020: how big, how beautiful?Empirical evaluation in 2020: how big, how beautiful?
Empirical evaluation in 2020: how big, how beautiful?
 
Agile Tour Brussels 2015 : Lean UX workshop
Agile Tour Brussels 2015 : Lean UX workshopAgile Tour Brussels 2015 : Lean UX workshop
Agile Tour Brussels 2015 : Lean UX workshop
 
A macroscopic web accessibility evaluation at different processing phases
A macroscopic web accessibility evaluation at different processing phasesA macroscopic web accessibility evaluation at different processing phases
A macroscopic web accessibility evaluation at different processing phases
 
EdenRW18-OpenVM
EdenRW18-OpenVMEdenRW18-OpenVM
EdenRW18-OpenVM
 
Upa Conference Loic Nunez 18 June2008
Upa Conference Loic Nunez 18 June2008Upa Conference Loic Nunez 18 June2008
Upa Conference Loic Nunez 18 June2008
 
Project "Visual Lean: Audit Tracking System" by LeanSoft company
Project "Visual Lean: Audit Tracking System" by LeanSoft companyProject "Visual Lean: Audit Tracking System" by LeanSoft company
Project "Visual Lean: Audit Tracking System" by LeanSoft company
 
Fehlmann and Kranich - Measuring tests using cosmic
Fehlmann and Kranich - Measuring tests using cosmicFehlmann and Kranich - Measuring tests using cosmic
Fehlmann and Kranich - Measuring tests using cosmic
 
Mirri At W4a2009
Mirri At W4a2009Mirri At W4a2009
Mirri At W4a2009
 
Answers siebel-set-i
Answers siebel-set-iAnswers siebel-set-i
Answers siebel-set-i
 
Main Presentantion of the EightProject Meeting
Main Presentantion of the EightProject MeetingMain Presentantion of the EightProject Meeting
Main Presentantion of the EightProject Meeting
 
Presentacion Dcai 2010
Presentacion Dcai 2010Presentacion Dcai 2010
Presentacion Dcai 2010
 
E Quality Portfolio
E Quality Portfolio E Quality Portfolio
E Quality Portfolio
 
Seeking value by Michael Ballé at the European Lean IT Summit 2012
Seeking value by Michael Ballé at the European Lean IT Summit 2012Seeking value by Michael Ballé at the European Lean IT Summit 2012
Seeking value by Michael Ballé at the European Lean IT Summit 2012
 
5 Steps to Data-driven Training
5 Steps to Data-driven Training5 Steps to Data-driven Training
5 Steps to Data-driven Training
 
Introduction to OSLC
Introduction to OSLCIntroduction to OSLC
Introduction to OSLC
 
Evaluation in hci
Evaluation in hciEvaluation in hci
Evaluation in hci
 
INDUSTRIAL TRAINING EVALUATION MANAGEMENT SYSTEM (IT-EMS) PRESENTATION
INDUSTRIAL TRAINING EVALUATION MANAGEMENT SYSTEM (IT-EMS) PRESENTATIONINDUSTRIAL TRAINING EVALUATION MANAGEMENT SYSTEM (IT-EMS) PRESENTATION
INDUSTRIAL TRAINING EVALUATION MANAGEMENT SYSTEM (IT-EMS) PRESENTATION
 
Value stream mapping for complex processes (innovation, Lean, service design)
Value stream mapping for complex processes (innovation, Lean, service design) Value stream mapping for complex processes (innovation, Lean, service design)
Value stream mapping for complex processes (innovation, Lean, service design)
 
Enhancing the assessment experience through closer integration between the SR...
Enhancing the assessment experience through closer integration between the SR...Enhancing the assessment experience through closer integration between the SR...
Enhancing the assessment experience through closer integration between the SR...
 
UI Integrations Test
UI Integrations TestUI Integrations Test
UI Integrations Test
 

Recently uploaded

Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptxUse Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
SynapseIndia
 
Zaitechno Handheld Raman Spectrometer.pdf
Zaitechno Handheld Raman Spectrometer.pdfZaitechno Handheld Raman Spectrometer.pdf
Zaitechno Handheld Raman Spectrometer.pdf
AmandaCheung15
 
(CISOPlatform Summit & SACON 2024) Regulation & Response In Banks.pdf
(CISOPlatform Summit & SACON 2024) Regulation & Response In Banks.pdf(CISOPlatform Summit & SACON 2024) Regulation & Response In Banks.pdf
(CISOPlatform Summit & SACON 2024) Regulation & Response In Banks.pdf
Priyanka Aash
 
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
bellared2
 
(CISOPlatform Summit & SACON 2024) Orientation by CISO Platform_ Using CISO P...
(CISOPlatform Summit & SACON 2024) Orientation by CISO Platform_ Using CISO P...(CISOPlatform Summit & SACON 2024) Orientation by CISO Platform_ Using CISO P...
(CISOPlatform Summit & SACON 2024) Orientation by CISO Platform_ Using CISO P...
Priyanka Aash
 
Feature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptxFeature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptx
ssuser1915fe1
 
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
aslasdfmkhan4750
 
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptxIntroduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
313mohammedarshad
 
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
shanihomely
 
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision MakingConnector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
DianaGray10
 
Computer HARDWARE presenattion by CWD students class 10
Computer HARDWARE presenattion by CWD students class 10Computer HARDWARE presenattion by CWD students class 10
Computer HARDWARE presenattion by CWD students class 10
ankush9927
 
Dublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptx
Dublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptxDublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptx
Dublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptx
Kunal Gupta
 
Data Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining DataData Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining Data
Safe Software
 
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Muhammad Ali
 
Tailored CRM Software Development for Enhanced Customer Insights
Tailored CRM Software Development for Enhanced Customer InsightsTailored CRM Software Development for Enhanced Customer Insights
Tailored CRM Software Development for Enhanced Customer Insights
SynapseIndia
 
The Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF GuideThe Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF Guide
Shiv Technolabs
 
Acumatica vs. Sage Intacct _Construction_July (1).pptx
Acumatica vs. Sage Intacct _Construction_July (1).pptxAcumatica vs. Sage Intacct _Construction_July (1).pptx
Acumatica vs. Sage Intacct _Construction_July (1).pptx
BrainSell Technologies
 
Using LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and MilvusUsing LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and Milvus
Zilliz
 
Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024
Nicolás Lopéz
 
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
alexjohnson7307
 

Recently uploaded (20)

Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptxUse Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
 
Zaitechno Handheld Raman Spectrometer.pdf
Zaitechno Handheld Raman Spectrometer.pdfZaitechno Handheld Raman Spectrometer.pdf
Zaitechno Handheld Raman Spectrometer.pdf
 
(CISOPlatform Summit & SACON 2024) Regulation & Response In Banks.pdf
(CISOPlatform Summit & SACON 2024) Regulation & Response In Banks.pdf(CISOPlatform Summit & SACON 2024) Regulation & Response In Banks.pdf
(CISOPlatform Summit & SACON 2024) Regulation & Response In Banks.pdf
 
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
 
(CISOPlatform Summit & SACON 2024) Orientation by CISO Platform_ Using CISO P...
(CISOPlatform Summit & SACON 2024) Orientation by CISO Platform_ Using CISO P...(CISOPlatform Summit & SACON 2024) Orientation by CISO Platform_ Using CISO P...
(CISOPlatform Summit & SACON 2024) Orientation by CISO Platform_ Using CISO P...
 
Feature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptxFeature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptx
 
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
 
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptxIntroduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
 
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
 
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision MakingConnector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
 
Computer HARDWARE presenattion by CWD students class 10
Computer HARDWARE presenattion by CWD students class 10Computer HARDWARE presenattion by CWD students class 10
Computer HARDWARE presenattion by CWD students class 10
 
Dublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptx
Dublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptxDublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptx
Dublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptx
 
Data Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining DataData Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining Data
 
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
 
Tailored CRM Software Development for Enhanced Customer Insights
Tailored CRM Software Development for Enhanced Customer InsightsTailored CRM Software Development for Enhanced Customer Insights
Tailored CRM Software Development for Enhanced Customer Insights
 
The Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF GuideThe Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF Guide
 
Acumatica vs. Sage Intacct _Construction_July (1).pptx
Acumatica vs. Sage Intacct _Construction_July (1).pptxAcumatica vs. Sage Intacct _Construction_July (1).pptx
Acumatica vs. Sage Intacct _Construction_July (1).pptx
 
Using LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and MilvusUsing LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and Milvus
 
Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024
 
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
 

Mirri w4a2012

  • 1. Getting one voice: tuning up experts’ assessment in measuring accessibility Silvia Mirri Ludovico A. Muratori Paola Salomoni Matteo Battistelli Department of Computer Science University of Bologna
  • 2. Summary Introduction Automatic and manual accessibility evaluations Our proposed metric Conclusions and future works W4A 2012 – April 16th&17th, 2012 - Lyon, France 2
  • 3. Introduction Web accessibility evaluations automatic tools + human assessment Metrics quantify accessibility level or barriers, providing numerical synthesis • automatic tools return binary values • human assessments are subjective and can get values from a continuous range W4A 2012 – April 16th&17th, 2012 - Lyon, France 3
  • 4. Our main goal Providing a metric to measure how far a Web page is from its accessibility version, taking into account • integration of human assessments with automatic evaluations on the same target • many humans assessments W4A 2012 – April 16th&17th, 2012 - Lyon, France 4
  • 5. Steps 1. Mixing up the manual evaluation together with the automatic ones 2. Combining the assessments coming from different human evaluations • Values distributed into a given range • The more experts' assessments contribute to compute a value, the more this value is stable and reliable W4A 2012 – April 16th&17th, 2012 - Lyon, France 5
  • 6. Automatic and manual evaluations: an example Combination between the IMG element and its ALT attribute: 1. If the ALT attribute is omitted the automatic check outputs 1 2. If the ALT attribute is present the automatic check outputs 0 Manual evaluation might state that: • there is no lack of information once the images are hidden (this can happen in case 1, if the image is a pure decorative one) • there is a lack of information once the image is hidden W4A 2012 – April 16th&17th, 2012 - Lyon, France 6
  • 7. Our metric • A first version of our metric (Barriers Impact Factor) is computed on the basis of a barrier-error association table • This table reports the list of assistive technologies/disabilities affected by any error • screen reader/blindness • screen magnifier/low vision • color blindness • input device independence/movement impairments • deafness • cognitive disabilities • photosensitive epilepsy W4A 2012 – April 16th&17th, 2012 - Lyon, France 7
  • 8. Our metric • Comparing automatic checks with WCAG 2.0 success criteria and identified relationships a certain error occurs or a A check fails manual control is necessary • Each barrier is related to one success criterion and to one level of conformity (A, AA or AAA) • Manual evaluations take values on the [0, 1] real numbers interval: • 1 means that an accessibility error occurs • 0 means the absence of that accessibility error W4A 2012 – April 16th&17th, 2012 - Lyon, France 8
  • 9. Our metric W4A 2012 – April 16th&17th, 2012 - Lyon, France 9
  • 10. Weighting automatic and manual checks 1. m(i)=a(i): the formula is a mere average among automatically and manually detected errors 2. m(i)>a(i): the failure in manual assessment is considered more significant than the automatic one 3. m(i)<a(i): the failure in automatic assessment is considered more significant than the manual one AUTOMATIC AUTOMATIC 0 1 0 1 [0, I III [0, I II MANUAL MANUAL ,1] II IV ,1] III IV W4A 2012 – April 16th&17th, 2012 - Lyon, France 10
  • 11. Some considerations • The more human operators provide evaluations about an accessibility barrier and the more the value of accessibility level is reliable • Behavior similar to online rating systems ones • New users rating can be influenced by already expressed evaluations from other users • Variance must be considered so as to reinforce the computed accessibility level W4A 2012 – April 16th&17th, 2012 - Lyon, France 11
  • 12. A first assessment PAGE CONTENT MANUAL EVALUATIONS 0,7 Expert A 1 Expert B 0,8 Expert C 1 Expert D ALT=“Image” 0,5 Expert E NO LINK, NO TITLE CBIF AUTOMATIC EVALUATION m=2 a=1 0 (no known errors, Average=0,8 CBIF=0,53 1 alert: placeholder Variance=0,036 detected) W4A 2012 – April 16th&17th, 2012 - Lyon, France 12
  • 13. Conclusions • We have defined an accessibility metric with the aim to evaluate barriers as a whole, combining results provided by using automatic tools and manual evaluations done by experts • The metric has been preliminary tested by measuring accessibility barriers in several local public administration Web sites • Five experts are manually evaluating barriers related to WCAG 2.0 1.1.1 (using an automatic monitoring system to verify the page content and to collect data from manual evaluations) W4A 2012 – April 16th&17th, 2012 - Lyon, France 13
  • 14. Future Work • Propose and discuss weights for the whole WCAG 2.0 set of barriers • Investigate how the number of experts involved in the evaluation, together with their rating variance, could influence the reliability of the computed values W4A 2012 – April 16th&17th, 2012 - Lyon, France 14
  • 15. Contacts  Thank you for your attention!  For further information:  silvia.mirri@unibo.it W4A 2012 – April 16th&17th, 2012 - Lyon, France 15