1. A Usable User Interface
for
Accessible Web Automation
Yury Puzis
Yevgen Borodin
Faisal Ahmed
I.V. Ramakrishnan
Stony Brook University
Stony Brook, USA
2012

2. Introduction
• Non-visual browsing is hard
• Not enough and too much information
• Not enough and too much control
• Browsing can be made easier
• We will try
• Interface > algorithms
• Our contribution
• Accessible and usable web automation interface 2
• Validation of the interface with a user study

3. Making a Purchase on Amazon.com
Minimum of 6 – 8 steps just to complete a transaction
Amazon is one of the most usable and one of the
most popular shopping websites with visually impaired 3
Can this be automated?

4. Web Automation for Sighted People
• Approaches:
• Handcrafting scripts
• Programming-by-example
• Tools:
• Form-filling in browsers
• CoScripter
• CoCo
• Creo
• Robofox
• SmartBookmarks
• …
• Not designed for blind people 6

5. Web Automation for Blind People
• Screen-readers
• Handcrafting: impractically hard
• Hearsay
• Scientific prototype
• programming-by-example (record / replay)
• Trailblazer
• Adaptation of CoScripter to JAWS
• No deep integration with JAWS user interface 7

6. Usable and Accessible
Web Automation Interface
Very few actions are applicable in a given browsing state /
context, and even fewer can lead to a meaningful and
desirable result.
A very simple automation interface
8

7. Usable and Accessible
Web Automation Interface
A very simple automation interface:
1. Ask for a suggestion of a single action
2. Confirm execution, skip, or ignore
… and continue browsing …
Atari’s Star Trek video game instructions:
1. Insert coins
2. Avoid Klingons 9

8. Advantages
• Focus on what’s important
• No information overload
• No need to manage macros
• Automation without taking away control
• The user knows what’s going on at all times
• The user can deviate away from the scenario
• The user can steer into a scenario
10

9. User Study
• 17 participants
• Comfortable / very comfortable with computers
• JAWS is the primary screen-reader
• 2 scenarios
• Buying a book (audiobooks.com)
• Reserving a hotel room (hilton.com)
• 3 trials for each scenario
• 2 without automation, 1 with
• No computational model
• Model’s deficiencies may affect the UI evaluation 11

10. Task Completion Time
600 Manual (trial 1)
Manual (trial 2)
500
Average time (seconds)
Automated Assistant
400
300
200
100
0
Scenario 1 Scenario 2
Average completion time 12
(1) buying a book (2) reserving a hotel room

11. Perceived Task Difficulty
5.00 Manual (trial 1)
4.50
(1=very easy, 5=very hard)
Manual (trial 2)
4.00 Automated Assistant
Average Difficulty
3.50
3.00
2.50
2.00
1.50
1.00
Scenario 1 Scenario 2
Average difficulty
13
(1) buying a book (2) reserving a hotel room

12. Questionnaire
I wish I could do online transactions faster than I can 3.88 (1.11)
with a regular screen reader
I often experience difficulties while doing online 2.62 (1.02)
transactions with a screen reader
Doing the same online transaction for the second time 4.59 (0.71)
was easier than the first time
Doing a transaction with the Automated Assistant was 4.12 (1.27)
the easiest
I want to use Automated Assistant in the future 4.29 (1.10)
14
Average 5-Point Likert scale values (St. Dev.)
Scale 1=Strongly Disagree to 5=Strongly Agree

13. Conclusion
• Web automation can help make the web more
accessible
• Automation is good in moderation
• Encouraging user study results
Testimonials
• “Assistant mode is really, really good”
• “Automated Assistant makes the site easier to use”
• “Looks like Automated Assistant can save a lot of
15
time, especially if it works with AJAX”

14. What Is Missing?
• Computational model
• Submitted to UIST 2012
• Deployment!
• Crowd sourcing / Social Accessibility
• User feedback on a large scale
16

15. Thank You!
17
This material is based upon work supported by the National Science
Foundation - Awards: IIS-0808678 and CNS-0751083