Vikas Ashok from Old Dominion University presented on using visual saliency models to improve web accessibility for people with visual impairments. He described two projects: SVIM, a saliency-driven video magnifier that tracks regions of interest in videos to aid low vision users, and SAIL, which automatically injects ARIA landmarks into webpages to streamline navigation for blind screen reader users. Both projects use deep learning models to detect salient regions and objects. SVIM clusters salient pixels to adjust the video viewport for magnification, while SAIL identifies salient areas to tag with landmarks. Evaluations found SVIM enhanced the video experience and SAIL reduced task completion times for screen reader users compared to manually landmarked pages.

1. Improving Web
Accessibility Using Visual
Saliency Models
VIKAS ASHOK
DEPARTMENT OF COMPUTER SCIENCE
OLD DOMINION UNIVERSITY

2. Scope of the Talk
 Web Accessibility
 People with Visual Impairments
 Visual Saliency
 Accessing Online Videos
 For Low Vision Screen-Magnifier Users
 SVIM – Saliency Driven Video Magnifier
 Webpage Navigation
 For Blind Screen-Reader Users
 SAIL – Saliency Driven Injection of ARIA Landmarks

3. Web
Accessibility
 “Web accessibility means that websites, tools, and technologies
are designed and developed so that people with disabilities can
use them. More specifically, people can:
 perceive, understand, navigate, and interact with the Web
 contribute to the Web”
- W3C

4. Improving Usability
Webpage Transcoding
Web Automation
Auxiliary Input Interfaces
Speech Assistants
• Audio-Haptic
• Dial
• Modify Webpages
Crowdsourcing
• Ask Others
• Voice Commands• Hand-Coded
Macros

5. Visual Saliency
Image source: Jiang, Lai, et al. "Deepvs: A deep learning based video saliency prediction approach." Proceedings of the
european conference on computer vision (eccv). 2018.

6. Saliency Driven
Low-Vision
Screen Magnifier

7. Video
Magnification
 Windows Magnifier
 Zoom + Pan with mouse
 Directly over content
VLC Magnifier
 Zoom + Pan with mouse
 Indirectly via thumbnail

8. SVIM ARCHITECTURE
Image source: Aydin, Ali Selman, et al. "Towards making videos accessible for low vision screen
magnifier users." Proceedings of the 25th International Conference on Intelligent User Interfaces.
2020.

9. Saliency Detection
 DeepVS [1]
 Trained on a video saliency dataset (LEDOV)
 Object-to-motion CNN for incorporating objects and motion
 ConvLSTM layers for incorporating temporal information
Image source: [1] Jiang, Lai, et al. "Deepvs: A deep learning based video saliency prediction
approach." Proceedings of the european conference on computer vision (eccv). 2018.

10. Region of Interest (ROI) Detection
Clustering Salient Pixels
Source: Aydin, Ali Selman, et al. "Towards making videos accessible for low vision screen
magnifier users." Proceedings of the 25th International Conference on Intelligent User Interfaces.
2020.

11. Temporal
Tracking
 Kalman Filter Smoothing to
Handle Jitter
 Viewport adjusted only for
significant movements
Source: Aydin, Ali Selman, et al. "Towards making videos accessible for low vision screen
magnifier users." Proceedings of the 25th International Conference on Intelligent User Interfaces.
2020.

12. SVIM Interface Features
Source: Aydin, Ali Selman, et al. "Towards making videos accessible for low vision screen
magnifier users." Proceedings of the 25th International Conference on Intelligent User Interfaces.
2020.

13. SVIM
DEMO

14. Evaluation
Source: Aydin, Ali Selman, et al. "Towards making videos accessible for low vision screen
magnifier users." Proceedings of the 25th International Conference on Intelligent User Interfaces.
2020.

15. Evaluation
Source: Aydin, Ali Selman, et al. "Towards making videos accessible for low vision screen
magnifier users." Proceedings of the 25th International Conference on Intelligent User Interfaces.
2020.

16. Saliency Based
Automatic
Injection of ARIA
Landmarks

17. Webpage
Navigation with
Screen readers
Linear Navigation
 Tedious and Tiring:
 70 'p' or 9 'h' shortcuts to reach
the red box!
 Solution: ARIA landmarks
 Caveat: Not widely utilized
Image Source: Aydin, Ali Selman, et al. "SaIL: saliency-
driven injection of ARIA landmarks." Proceedings of the
25th International Conference on Intelligent User
Interfaces. 2020.

18. Saliency Based
Landmark
Injection
 Detect Salient Regions
 Filter Candidates
 Inject ARIA Landmarks
Image Source: Aydin, Ali Selman, et al. "SaIL: saliency-driven injection of ARIA landmarks." Proceedings of the 25th International Conference on
Intelligent User Interfaces. 2020.

19. Webpage
Saliency with
GANs
 SalGAN [1] originally trained
for image saliency, re-trained
to predict web saliency
[1] Pan, Junting, et al. "Salgan: Visual saliency prediction with adversarial networks." CVPR
Scene Understanding Workshop (SUNw). 2017.

20. Automatic Landmark Injection

21. Examples
Image Source: Aydin, Ali Selman, et al. "SaIL: saliency-driven injection of ARIA landmarks." Proceedings of the 25th International Conference on
Intelligent User Interfaces. 2020.

22. More Examples
Image Source: Aydin, Ali Selman, et al. "SaIL: saliency-driven injection of ARIA landmarks." Proceedings of the 25th International Conference on
Intelligent User Interfaces. 2020.

23. Evaluation
 C1 – Without SAIL
 C2 – With SAIL
Task Completion Time (Seconds)
Source: Aydin, Ali Selman, et al. "SaIL: saliency-driven injection of ARIA landmarks." Proceedings of the 25th International Conference on
Intelligent User Interfaces. 2020.

24. Concluding Remark
Visual Saliency Models Have Tremendous Potential For Improving
Usability For People With Visual Impairments

25. Thank You!