PhD Dissertation

1. Assisted Interaction for Improving Web Accessibility: An Approach Driven and Tested by Users with Disabilities PhD Dissertation Candidate: J. Eduardo Pérez Supervisors: Myriam Arrue Julio Abascal Donostia/San Sebastian – Fri. 6 Nov., 2020 Informatika Fakultatea, Dept. of Computer Architecture and Technology Laboratory of Human-Computer Interaction for Special Needs

2. Presentation Overview •  Part I: Introduction –  Motivation –  Approach –  Methodology –  General objectives –  Phases of the research •  Part II: A platform to assist user testing –  “Assisted Interaction Data Analysis of Web-Based User Studies” (INTERACT 2015) •  Part III: An accessible platform for inclusive in situ and remote experiments –  “Inclusive Web Empirical Studies in Remote and In-Situ Settings: A User Evaluation of the RemoTest Platform” (Int. J. of Human-Computer Interaction, Vol. 35, Issue 7, 2019) •  Part IV: Adaptations for touch screen tablets –  “Adapting the Web for People With Upper Body Motor Impairments Using Touch Screen Tablets” (Interacting with Computers, Vol. 29, Issue 6, 2017) •  Part V: Adaptations for people with low vision –  “An exploratory study of web adaptation techniques for people with low vision” (Universal Access in the Information Society, 2020) •  Part VI: Virtual cursors for people with motor impairments –  “Evaluation of two virtual cursors for assisting web access to people with motor impairments” (Int. J. of Human-Computer Studies, Vol. 132, 2019) •  Part VII: Longitudinal study of both virtual cursors –  “Longitudinal Study of Two Virtual Cursors for People With Motor Impairments: A Performance and Satisfaction Analysis on Web Navigation” (IEEE Access, Vol. 8, 2020) •  Part VIII: –  Overall conclusions –  Main contributions –  Future work 2/43

3. •  Part I: Introduction –  Motivation –  Approach –  Methodology –  General objectives –  Phases of the research •  Part II: A platform to assist user testing –  “Assisted Interaction Data Analysis of Web-Based User Studies” (INTERACT 2015) •  Part III: An accessible platform for inclusive in situ and remote experiments –  “Inclusive Web Empirical Studies in Remote and In-Situ Settings: A User Evaluation of the RemoTest Platform” (Int. J. of Human-Computer Interaction, Vol. 35, Issue 7, 2019) •  Part IV: Adaptations for touch screen tablets –  “Adapting the Web for People With Upper Body Motor Impairments Using Touch Screen Tablets” (Interacting with Computers, Vol. 29, Issue 6, 2017) •  Part V: Adaptations for people with low vision –  “An exploratory study of web adaptation techniques for people with low vision” (Universal Access in the Information Society, 2020) •  Part VI: Virtual cursors for people with motor impairments –  “Evaluation of two virtual cursors for assisting web access to people with motor impairments” (Int. J. of Human-Computer Studies, Vol. 132, 2019) •  Part VII: Longitudinal study of both virtual cursors –  “Longitudinal Study of Two Virtual Cursors for People With Motor Impairments: A Performance and Satisfaction Analysis on Web Navigation” (IEEE Access, Vol. 8, 2020) •  Part VIII: Final conclusions & future work Presentation Overview 2/43

4. Motivation •  The Web has become essential in our lives •  Potential to promote social inclusion and participation of people with disabilities •  Many initiatives (guidelines, policies,…) to build accessible web sites [1, 2, 3] •  Need for research on web accessibility: -  Majority of websites include a single design to fit all users -  Vast diversity within users with similar impairments -  Decline in productivity & satisfaction of users on web navigation [1] “Web Content Accessibility Guidelines (WCAG 1.0, 2.0, 2.1),” by W3C (1999, 2008, 2018). [2] “Accessibility of the websites and mobile applications of public sector bodies,” by European Union (2016). [3] “Section 508 of the Rehabilitation Act of 1973,” by US Congress (1998). Part I – Introduction 3/43

5. Approach •  Transcoding systems for web accessibility: -  Invented to help people with disabilities access inaccessible Web pages [1] -  Intermediary server to convert the content automatically without asking the author -  Need of an annotation process to achieve more complex adaptations -  Several examples of transcoding systems [2] mainly for people with visual impairments •  Virtual enhancements for assisting target selection: -  Assist the use of pointing devices to select, click, drag, etc., but also for tactile interaction -  Mainly aimed at users with motor impairments [3] •  Explore two disabilities: -  People with motor impairments (lack of dexterity in upper limbs) -  People with low vision Part I – Introduction [1] Asakawa et al. (2019) “Transcoding,” In Web Accessibility: A Foundation for Research. [2] Asakawa & Itoh (1998) “User interface of a home page reader,” In ASSETS ’98. [3] Trewin et al. (2006) “Developing steady clicks: a method of cursor assistance for people with motor impairments,” In ASSETS ’06. 4/43

6. Methodology ①  Observational studies of human behaviour and personal interviews -  To identify web accessibility issues of target user populations ②  Literature review -  To select appropriate guidelines, standards and techniques to support web accessibility ③  Follow usability and accessibility principles, as well as UCD techniques -  To foster acceptance and compatibility of developed adaptation techniques and interaction aids ④  Conduct user tests -  To evaluate proposed advancements: performance and satisfaction metrics Part I – Introduction 5/43

7. General Objectives §  To analyse the needs of users with disabilities on the Web §  To search adaptation techniques and alternative interaction methods §  To develop and test an evaluation tool to conduct user tests and record user interactions §  To develop technical aids (interface adaptations and other virtual enhancements for target selection) §  To test and evaluate the technical aids developed §  To propose criteria to enhance web accessibility Part I – Introduction 6/43

8. Phases of the research Remotest tool conception, development and test Point & click assistance through virtual cursors UI adaptations through transcoding II. Remotest functionality evaluation III. Remotest accessibility evaluation IV. Touch screen for motor impairment V. Desktop for low vision VII. Longitudinal study VI. In situ study Part I – Introduction 7/43

9. Overview •  Part I: Introduction –  Motivation –  Background –  Methodology –  Objectives •  Part II: A platform to assist web-based user testing –  “Assisted Interaction Data Analysis of Web-Based User Studies” (INTERACT 2015) •  Part III: An accessible platform for inclusive in situ and remote experiments –  “Inclusive Web Empirical Studies in Remote and In-Situ Settings: A User Evaluation of the RemoTest Platform” (Int. J. of Human-Computer Interaction, Vol. 35, Issue 7, 2019) •  Part IV: Adaptations for touch screen tablets –  “Adapting the Web for People With Upper Body Motor Impairments Using Touch Screen Tablets” (Interacting with Computers, Vol. 29, Issue 6, 2017) •  Part V: Adaptations for people with low vision –  “An exploratory study of web adaptation techniques for people with low vision” (Universal Access in the Information Society, 2020) •  Part VI: Virtual cursors for people with motor impairments –  “Evaluation of two virtual cursors for assisting web access to people with motor impairments” (Int. J. of Human-Computer Studies, Vol. 132, 2019) •  Part VII: Longitudinal study of both virtual cursors –  “Longitudinal Study of Two Virtual Cursors for People With Motor Impairments: A Performance and Satisfaction Analysis on Web Navigation” (IEEE Access, Vol. 8, 2020) •  Part VIII: Final conclusions & future work 8/43

10. Rationale Part II – “Assisted Interaction Data Analysis of Web-Based User Studies” (INTERACT 2015) •  Study user behaviour on the Web requires conducting experimental sessions •  We built the RemoTest platform to assist experimenters with remote and in situ experimental sessions •  In this paper we presented: –  A proof of concept developed to evaluate Remotest validity •  To define and conduct experiments with users •  To analyse the interaction data collected 9/43

11. RemoTest platform •  Experimenter Module define all components of experiments •  Coordinator Module stores & manages experiments defined with the Experimenter Module Part II – “Assisted Interaction Data Analysis of Web-Based User Studies” (INTERACT 2015) •  Result Viewer Module organizes and presents the abundant interaction data gathered in an experiment, provided by the Coordinator Module •  Participant Module add-on for FireFox browser 10/43

12. Objective & Methodology •  Objective: –  Compare results from manual video recording analysis about navigation behaviours of participants with automatically calculated measures by RemoTest •  Methodology: –  16 participants •  11 users with motor impairments and 5 without impairments –  3 tasks: •  Filling a questionnaire about demographic data •  Free navigation task •  Searching task Part II – “Assisted Interaction Data Analysis of Web-Based User Studies” (INTERACT 2015) 11/43

13. Results & Conclusions •  RemoTest was successfully used –  To define and conduct the experimental sessions, –  To gather interaction data from participants –  To analyse interaction data •  Results automatically obtained were useful –  to characterize cursor movement and –  To identify profiles Part II – “Assisted Interaction Data Analysis of Web-Based User Studies” (INTERACT 2015) 12/43

14. Overview •  Part I: Introduction –  Motivation –  Background –  Methodology –  Objectives •  Part II: A platform to assist user testing –  “Assisted Interaction Data Analysis of Web-Based User Studies” (INTERACT 2015) •  Part III: An accessible platform for inclusive in situ and remote experiments –  “Inclusive Web Empirical Studies in Remote and In-Situ Settings: A User Evaluation of the RemoTest Platform” (Int. J. of Human-Computer Interaction, Vol. 35, Issue 7, 2019) •  Part IV: Adaptations for touch screen tablets –  “Adapting the Web for People With Upper Body Motor Impairments Using Touch Screen Tablets” (Interacting with Computers, Vol. 29, Issue 6, 2017) •  Part V: Adaptations for people with low vision –  “An exploratory study of web adaptation techniques for people with low vision” (Universal Access in the Information Society, 2020) •  Part VI: Virtual cursors for people with motor impairments –  “Evaluation of two virtual cursors for assisting web access to people with motor impairments” (Int. J. of Human-Computer Studies, Vol. 132, 2019) •  Part VII: Longitudinal study of both virtual cursors –  “Longitudinal Study of Two Virtual Cursors for People With Motor Impairments: A Performance and Satisfaction Analysis on Web Navigation” (IEEE Access, Vol. 8, 2020) •  Part VIII: Final conclusions & future work 13/43

15. Rationale Part III – “Inclusive Web Empirical Studies in Remote and In-Situ Settings: A User Evaluation of the RemoTest Platform” (IJHCI) •  To ensure RemoTest can be used remotely by a wide range of users, needs to meet accessibility requirements •  In this paper we presented: –  Evaluation of environments created by RemoTest by mean of user tests –  Installation, questionnaires and general satisfaction with the tool 14/43

16. Methodology Part III – “Inclusive Web Empirical Studies in Remote and In-Situ Settings: A User Evaluation of the RemoTest Platform” (IJHCI) In situ study: •  36 participants with different disabilities: –  13 motor impairment, 10 blind, 8 low vision, and 5 without disabilities –  All had experience with computers and the Web •  5 tasks: –  Installation, questionnaires, and web navigation tasks •  Data collection: –  Interaction data, video recordings, observations, and interviews 15/43

17. Results & Conclusions Part III – “Inclusive Web Empirical Studies in Remote and In-Situ Settings: A User Evaluation of the RemoTest Platform” (IJHCI) •  The RemoTest platform proved to be usable and accessible for participants with different disabilities •  Stimuli generated by RemoTest proved to be accessible both in remote and in situ contexts •  Improvements based on comments by participants –  Provide shortcuts –  Numbering the questions –  Clear instructions –  Larger text and controls 16/43

19. Rationale •  People with disabilities face aggravated accessibility barriers when accessing the Web from mobile devices •  In this paper we presented: –  A transcoding system that implements adaptations techniques for assisting people with motor impairments on mobile devices –  Four alternative tactile interaction methods implemented to assist them –  Evaluation of the enhancements proposed with eight users with motor impairments Part IV – “Adapting the Web for People With Upper Body Motor Impairments Using Touch Screen Tablets” (IwC) 18/43

20. Transcoding tool architecture •  A transcoding system composed of four modules –  Presentation –  Coordinator –  Knowledge base –  Adaptation Part IV – “Adapting the Web for People With Upper Body Motor Impairments Using Touch Screen Tablets” (IwC) 19/43

21. Methodology •  A formal evaluation with three main objectives: –  To collect general characteristics of the users –  To test different interaction techniques –  To measure the results of applying various adaptation techniques •  Participants: –  Eight users with motor impairments in upper limbs –  Two of them use regularly their own tablets or smartphones •  Experimental setting –  A Firefox Web browser with the RemoTest add-on presenting stimuli and gathering interaction data from participants •  Tasks –  Phase 1: navigation tasks –  Phase 2: target acquisition tasks Part IV – “Adapting the Web for People With Upper Body Motor Impairments Using Touch Screen Tablets” (IwC) 20/43

22. Results & Conclusions •  Transcoded pages were revealed to be the preferred option for most of the participants, as they require less physical effort. •  Alternative interaction methods tested by motor impaired users: ‘end tap’ and ‘steady tap’ proved to be helpful for people with less finger movement control. Part IV – “Adapting the Web for People With Upper Body Motor Impairments Using Touch Screen Tablets” (IwC) 21/43

24. Rationale •  People with low vision rely on assistive technologies (magnification, zooming) to access the Web •  However, zooming and magnification assistances entail other accessibility issues for web browsing •  In this paper we presented: –  Literature review and an observational study to identify adaptations techniques to assist people with low vision –  Results of an experimental session with 12 users with low vision to evaluate these adaptations techniques Part V – “An exploratory study of web adaptation techniques for people with low vision” (UAIS) 23/43

25. Methodology Part V – “An exploratory study of web adaptation techniques for people with low vision” (UAIS) •  First phase –  Literature review –  Observational study •  Second phase –  Exploratory Study •  Twelve participants with low vision having a high level of web navigation expertise •  Conducted experimental sessions with adapted versions of the web interfaces & gathered user interaction data 24/43

26. Results & Conclusions Part V – “An exploratory study of web adaptation techniques for people with low vision” (UAIS) •  The advantages of the adaptation techniques in relation to the type of assistive technology used. •  Some adaptation techniques were helpful only for screen magnifier users, but not for browser zoom users. •  New research hypotheses for a future experimental study obtained. 25/43

28. Rationale Part VI – “Evaluation of two virtual cursors for assisting web access to people with motor impairments” (IJHCS) •  Link selection fundamental action for web browsing but challenging for people with motor impairments •  Despite the many mouse alternatives, target selection is still challenging for them •  Based on the literature and on previous studies we developed and evaluated 2 virtual cursors to assist link selection: –  The cross cursor (keyboard based assistance) –  The area cursor (larger activation area) 27/43

29. Virtual cursors Part VI – “Evaluation of two virtual cursors for assisting web access to people with motor impairments” (IJHCS) •  Both virtual cursors developed as browser add-ons using JavaScript (handle interactions and page content) and SVG (visual representation) •  The cross cursor: –  Combine cursor movement and one- letter shortcuts to reduce trajectories –  Links at reach are those “crossed” by the cross cursor •  The area cursor: –  Reduce accuracy required for clicking links by providing a larger activation area –  If more than one link within fixed size activation area: selectable closest to cursor 28/43

30. Study Part VI – “Evaluation of two virtual cursors for assisting web access to people with motor impairments” (IJHCS) •  3 cursor variants (within-subjects design): –  cross cursor, area cursor, and original unassisted •  15 participants: –  9 with motor impairments –  Expert users with their pointing device –  Grouped based on pointing device: •  2 websites used as stimuli: –  informational [discapnet.com] and institutional [gipuzkoa.eus] •  2 kind of tasks: –  4 searching tasks and 48 target acquisition tasks with each cursor variant Area cursor Cross cursor Original cursor JU group KU group MU group Keyboard group – KU Joystick & trackball group – JU Mouse group – MU 4 participants 5 participants 6 participants 29/43

31. Measuring performance & satisfaction Part VI – “Evaluation of two virtual cursors for assisting web access to people with motor impairments” (IJHCS) •  Six metrics to study performance on target acquisition tasks based on cursor path evaluation: –  Movement time, pointing time, clicking time, distance travelled, curvature index, and number of pauses •  Eight categories to ask about participants satisfaction with each cursor variant: –  Learnable, easy to remember, accurate, easy to use, effortless, natural, fun, and not frustrating •  Participants were also asked to rank all 3 variants from most to least preferred choice for browsing the Web 30/43

32. Results Part VI – “Evaluation of two virtual cursors for assisting web access to people with motor impairments” (IJHCS) Results of performance: •  Keyboard users group achieved the best results with the cross cursor 31/43

33. Results Part VI – “Evaluation of two virtual cursors for assisting web access to people with motor impairments” (IJHCS) Results of performance: •  Keyboard users group achieved the best results with the cross cursor •  Joystick and trackball users group achieved better results with the area cursor 31/43

34. Results Part VI – “Evaluation of two virtual cursors for assisting web access to people with motor impairments” (IJHCS) Results of performance: •  Keyboard users group achieved the best results with the cross cursor •  Joystick and trackball users group achieved better results with the area cursor •  Mouse users group achieved similar results with both cursors 31/43

35. Results Part VI – “Evaluation of two virtual cursors for assisting web access to people with motor impairments” (IJHCS) Results of performance: •  Keyboard users group achieved the best results with the cross cursor •  Joystick and trackball users group achieved better results with the area cursor •  Mouse users group achieved similar results with both cursors Results of satisfaction •  Overall ranking – “preferred choice for web browsing”: •  Most participants with motor impairments (7 / 9) preferred one of the two virtual cursor proposed 31/43

36. Conclusions Part VI – “Evaluation of two virtual cursors for assisting web access to people with motor impairments” (IJHCS) •  Virtual cursors tested proved to be beneficial for participants with motor impairments •  Results showed improvements in their performance •  Keyboard users achieved best results with the cross cursor, whereas joystick & trackball users with the area cursor. •  Promising results: –  need of personalized adaptations to assist point & click interactions –  they suggested a performance improvement with prolonged use of virtual cursors 32/43

38. Rationale Part VII – “Longitudinal Study of Two Virtual Cursors for People With Motor Impairments: A Performance and Satisfaction Analysis on Web Navigation” (IEEE Access) •  Does continuous use … … enhance the performance? … affect satisfaction? •  We performed a longitudinal study to analyse: –  Learning effect –  Influence of distractors (other links) on target selection –  Satisfaction evolution over time –  Significance of results 34/43

39. Study •  2 pairs of cursors (between-subjects design): –  cross vs original cursor –  area vs original cursor •  3 target configurations to study influence of distractors •  2 kind of tasks: –  Free navigation & Target acquisition task •  8 participants with motor impairments: –  Expert users with their pointing device –  Grouped based on pointing device: •  6 weeks long unsupervised study, from home with their personal computers Keyboard group – KU Joystick & trackball group – JU 4 participants 4 participants: 1 trackball + 3 joystick Part VII – “Longitudinal Study of Two Virtual Cursors for People With Motor Impairments: A Performance and Satisfaction Analysis on Web Navigation” (IEEE Access) Cross cursor Area cursor Original cursor Original cursor KU group JU group 3 target configurations to study influence of distractors 35/43

40. Measurements Part VII – “Longitudinal Study of Two Virtual Cursors for People With Motor Impairments: A Performance and Satisfaction Analysis on Web Navigation” (IEEE Access) •  Virtual cursors acceptance based on measures calculated from free navigation •  Effective use of the virtual cursors based on target selection without clicking over targets •  Seven metrics to study performance on target acquisition task –  Same 6 metrics from prev. study, plus target re-entry –  Also the throughput (TP) progression on target acquisition task over the course of the 15 sessions •  10 items (SUS questionnaire) to rate (5-point Likert scale) participants satisfaction with each cursor tested 36/43

41. Results on performance Part VII – “Longitudinal Study of Two Virtual Cursors for People With Motor Impairments: A Performance and Satisfaction Analysis on Web Navigation” (IEEE Access) Users Movement Time Pointing Time Clicking Time Distance travelled Curvature Index No. of pauses Target re-entry Keyboard 3 Joystick & 1 trackball Users Movement Time Pointing Time Clicking Time Distance travelled Curvature Index No. of pauses Target re-entry Keyboard 3 Joystick & 1 trackball Users Movement Time Pointing Time Clicking Time Distance travelled Curvature Index No. of pauses Target re-entry Keyboard 3 Joystick & 1 trackball Target configuration 1 – close distractors Target configuration 2 – distant distractors Target configuration 3 – without distractors 37/43

42. Conclusions Part VII – “Longitudinal Study of Two Virtual Cursors for People With Motor Impairments: A Performance and Satisfaction Analysis on Web Navigation” (IEEE Access) •  Both virtual cursors improved the performance of each group: –  The cross cursor to keyboard users –  The area cursor to joystick and trackball users •  Positive scores on satisfaction after a prolonged use –  Keyboard users improved SUS score (90/100 è 93.1/100) •  Improvements suggested: –  To reduce the clicking time –  To improve performance of the standard area cursor with closely spaced links 38/43

44. Overall conclusions •  Methodology applied was valid for improving web accessibility to different groups of users with special needs •  Technical aids tested improved web access to specific groups of users. –  Assistive technology used (screen magnifiers or alternative pointing devices) was a determining factor to select the most appropriate. Part VIII – Conclusions & future work 40/43

45. Overall conclusions •  In addition to performance measures, valuable information from interviews with participants to improve techniques proposed •  Longitudinal study (remote and unsupervised) provided insight that could not be obtained from the other studies –  More valuable and natural interaction data from unsupervised navigation –  Technology acceptance based on habits of use –  Greater amount of data to analyse significance of results –  Continuous monitoring allow to analyse the learning effect Part VIII – Conclusions & future work 41/43

46. Main contributions •  A platform for conducting formal web-based user studies has been developed •  A new heuristic method for delimiting point and click cursor trajectories for link selection has been defined •  Additional knowledge about what difficulties are encountered in the Web by different groups of users has been provided •  Keyboard-only users with the novel cross cursor significantly improved their performance has been proved Part VIII – Conclusions & future work 42/43

47. Future work Part VIII – Conclusions & future work •  To develop downloadable versions of technical aids. •  To apply same methodology to other groups of users (e.g., people with intellectual impairments), and to other contexts (e.g., smartphones) •  To investigate alternatives, such as crowdsourcing or gamification techniques, to speed up the annotation process •  To investigate point & click over other interaction components (e.g., opening dropdown menus, filling forms, etc.) 43/43

48. Assisted Interaction for Improving Web Accessibility: An Approach Driven and Tested by Users with Disabilities Thank you for your attention Donostia/San Sebastian – Fri. 6 Nov., 2020 Informatika Fakultatea, Dept. of Computer Architecture and Technology Laboratory of Human-Computer Interaction for Special Needs