A Usable User Interface for Accessible Web Automation

Yury Puzis
Yury PuzisResearch Scientist at Stony Brook University
A Usable User Interface
            for
Accessible Web Automation
              Yury Puzis
           Yevgen Borodin
            Faisal Ahmed
         I.V. Ramakrishnan

       Stony Brook University
          Stony Brook, USA

               2012
Introduction
• Non-visual browsing is hard
  • Not enough and too much information
  • Not enough and too much control

• Browsing can be made easier
  • We will try
  • Interface > algorithms

• Our contribution
  • Accessible and usable web automation interface   2
  • Validation of the interface with a user study
Making a Purchase on Amazon.com




 Minimum of 6 – 8 steps just to complete a transaction

   Amazon is one of the most usable and one of the
 most popular shopping websites with visually impaired   3


               Can this be automated?
Web Automation for Sighted People
• Approaches:
  • Handcrafting scripts
  • Programming-by-example
• Tools:
  •   Form-filling in browsers
  •   CoScripter
  •   CoCo
  •   Creo
  •   Robofox
  •   SmartBookmarks
  •   …
• Not designed for blind people     6
Web Automation for Blind People
• Screen-readers
  • Handcrafting: impractically hard

• Hearsay
  • Scientific prototype
  • programming-by-example (record / replay)

• Trailblazer
   • Adaptation of CoScripter to JAWS
   • No deep integration with JAWS user interface   7
Usable and Accessible
Web Automation Interface
Very few actions are applicable in a given browsing state /
context, and even fewer can lead to a meaningful and
desirable result.


         A very simple automation interface




                                                              8
Usable and Accessible
Web Automation Interface
 A very simple automation interface:
 1. Ask for a suggestion of a single action
 2. Confirm execution, skip, or ignore
                … and continue browsing …



 Atari’s Star Trek video game instructions:
 1. Insert coins
 2. Avoid Klingons                            9
Advantages
• Focus on what’s important
  • No information overload
  • No need to manage macros

• Automation without taking away control
  • The user knows what’s going on at all times
  • The user can deviate away from the scenario
  • The user can steer into a scenario
                                                  10
User Study
• 17 participants
• Comfortable / very comfortable with computers
• JAWS is the primary screen-reader
• 2 scenarios
  • Buying a book (audiobooks.com)
  • Reserving a hotel room (hilton.com)
• 3 trials for each scenario
  • 2 without automation, 1 with
• No computational model
  • Model’s deficiencies may affect the UI evaluation   11
Task Completion Time
                          600                                Manual (trial 1)
                                                             Manual (trial 2)
                          500
 Average time (seconds)




                                                             Automated Assistant
                          400

                          300

                          200

                          100

                            0
                                   Scenario 1        Scenario 2
                                   Average completion time                         12
                          (1) buying a book (2) reserving a hotel room
Perceived Task Difficulty
                               5.00                              Manual (trial 1)
                               4.50
(1=very easy, 5=very hard)



                                                                 Manual (trial 2)
                               4.00                              Automated Assistant
    Average Difficulty




                               3.50
                               3.00
                               2.50
                               2.00
                               1.50
                               1.00
                                         Scenario 1       Scenario 2
                                          Average difficulty
                                                                                       13
                             (1) buying a book (2) reserving a hotel room
Questionnaire
I wish I could do online transactions faster than I can   3.88 (1.11)
with a regular screen reader
I often experience difficulties while doing online        2.62 (1.02)
transactions with a screen reader
Doing the same online transaction for the second time 4.59 (0.71)
was easier than the first time
Doing a transaction with the Automated Assistant was      4.12 (1.27)
the easiest
I want to use Automated Assistant in the future           4.29 (1.10)



                                                                        14
              Average 5-Point Likert scale values (St. Dev.)
              Scale 1=Strongly Disagree to 5=Strongly Agree
Conclusion
• Web automation can help make the web more
  accessible
• Automation is good in moderation
• Encouraging user study results

Testimonials
• “Assistant mode is really, really good”
• “Automated Assistant makes the site easier to use”
• “Looks like Automated Assistant can save a lot of
                                                       15
  time, especially if it works with AJAX”
What Is Missing?
• Computational model
  • Submitted to UIST 2012
• Deployment!
• Crowd sourcing / Social Accessibility
• User feedback on a large scale




                                          16
Thank You!




                                                                     17
This material is based upon work supported by the National Science
Foundation - Awards: IIS-0808678 and CNS-0751083
1 of 15

More Related Content

Similar to A Usable User Interface for Accessible Web Automation

IITK WorkshopIITK Workshop
IITK WorkshopArun Ranganathan
362 views85 slides
Automated ui testingAutomated ui testing
Automated ui testingDavidReidy
2.6K views83 slides
Justin IsonJustin Ison
Justin IsonCodeFest
198 views38 slides

Similar to A Usable User Interface for Accessible Web Automation(20)

Reliable mobile test automationReliable mobile test automation
Reliable mobile test automation
Vishal Banthia4.2K views
IITK WorkshopIITK Workshop
IITK Workshop
Arun Ranganathan362 views
Automated ui testingAutomated ui testing
Automated ui testing
DavidReidy2.6K views
Malware Analysis on a Shoestring BudgetMalware Analysis on a Shoestring Budget
Malware Analysis on a Shoestring Budget
Michael Boman1.3K views
Justin IsonJustin Ison
Justin Ison
CodeFest198 views
Ui design-day1Ui design-day1
Ui design-day1
Tim Proffitt171 views
Automated testing web applicationAutomated testing web application
Automated testing web application
Kiattikhun Prathumma3.5K views
State Of Ajax   Zend Con 08State Of Ajax   Zend Con 08
State Of Ajax Zend Con 08
bgalbs4.3K views
Automated Exploratory TestingAutomated Exploratory Testing
Automated Exploratory Testing
Justin Ison1.7K views
Virtual mouseVirtual mouse
Virtual mouse
Nikhil Mane14.2K views
The Limits of Unit Testing by Craig StuntzThe Limits of Unit Testing by Craig Stuntz
The Limits of Unit Testing by Craig Stuntz
QA or the Highway493 views
The limits of unit testing by Craig StuntzThe limits of unit testing by Craig Stuntz
The limits of unit testing by Craig Stuntz
QA or the Highway434 views
Infrastructure as Code, Theory Crash CourseInfrastructure as Code, Theory Crash Course
Infrastructure as Code, Theory Crash Course
Dr. Sven Balnojan198 views

Recently uploaded(20)

A Usable User Interface for Accessible Web Automation

  • 1. A Usable User Interface for Accessible Web Automation Yury Puzis Yevgen Borodin Faisal Ahmed I.V. Ramakrishnan Stony Brook University Stony Brook, USA 2012
  • 2. Introduction • Non-visual browsing is hard • Not enough and too much information • Not enough and too much control • Browsing can be made easier • We will try • Interface > algorithms • Our contribution • Accessible and usable web automation interface 2 • Validation of the interface with a user study
  • 3. Making a Purchase on Amazon.com Minimum of 6 – 8 steps just to complete a transaction Amazon is one of the most usable and one of the most popular shopping websites with visually impaired 3 Can this be automated?
  • 4. Web Automation for Sighted People • Approaches: • Handcrafting scripts • Programming-by-example • Tools: • Form-filling in browsers • CoScripter • CoCo • Creo • Robofox • SmartBookmarks • … • Not designed for blind people 6
  • 5. Web Automation for Blind People • Screen-readers • Handcrafting: impractically hard • Hearsay • Scientific prototype • programming-by-example (record / replay) • Trailblazer • Adaptation of CoScripter to JAWS • No deep integration with JAWS user interface 7
  • 6. Usable and Accessible Web Automation Interface Very few actions are applicable in a given browsing state / context, and even fewer can lead to a meaningful and desirable result. A very simple automation interface 8
  • 7. Usable and Accessible Web Automation Interface A very simple automation interface: 1. Ask for a suggestion of a single action 2. Confirm execution, skip, or ignore … and continue browsing … Atari’s Star Trek video game instructions: 1. Insert coins 2. Avoid Klingons 9
  • 8. Advantages • Focus on what’s important • No information overload • No need to manage macros • Automation without taking away control • The user knows what’s going on at all times • The user can deviate away from the scenario • The user can steer into a scenario 10
  • 9. User Study • 17 participants • Comfortable / very comfortable with computers • JAWS is the primary screen-reader • 2 scenarios • Buying a book (audiobooks.com) • Reserving a hotel room (hilton.com) • 3 trials for each scenario • 2 without automation, 1 with • No computational model • Model’s deficiencies may affect the UI evaluation 11
  • 10. Task Completion Time 600 Manual (trial 1) Manual (trial 2) 500 Average time (seconds) Automated Assistant 400 300 200 100 0 Scenario 1 Scenario 2 Average completion time 12 (1) buying a book (2) reserving a hotel room
  • 11. Perceived Task Difficulty 5.00 Manual (trial 1) 4.50 (1=very easy, 5=very hard) Manual (trial 2) 4.00 Automated Assistant Average Difficulty 3.50 3.00 2.50 2.00 1.50 1.00 Scenario 1 Scenario 2 Average difficulty 13 (1) buying a book (2) reserving a hotel room
  • 12. Questionnaire I wish I could do online transactions faster than I can 3.88 (1.11) with a regular screen reader I often experience difficulties while doing online 2.62 (1.02) transactions with a screen reader Doing the same online transaction for the second time 4.59 (0.71) was easier than the first time Doing a transaction with the Automated Assistant was 4.12 (1.27) the easiest I want to use Automated Assistant in the future 4.29 (1.10) 14 Average 5-Point Likert scale values (St. Dev.) Scale 1=Strongly Disagree to 5=Strongly Agree
  • 13. Conclusion • Web automation can help make the web more accessible • Automation is good in moderation • Encouraging user study results Testimonials • “Assistant mode is really, really good” • “Automated Assistant makes the site easier to use” • “Looks like Automated Assistant can save a lot of 15 time, especially if it works with AJAX”
  • 14. What Is Missing? • Computational model • Submitted to UIST 2012 • Deployment! • Crowd sourcing / Social Accessibility • User feedback on a large scale 16
  • 15. Thank You! 17 This material is based upon work supported by the National Science Foundation - Awards: IIS-0808678 and CNS-0751083

Editor's Notes

  1. Speak fasterDon’t say “you can see” too much
  2. Both interface and algorithms that support it are important, but priorities make a huge difference.
  3. And blind users don’t shop – shopping is too hard
  4. A few words about existing web automation solutions
  5. assuming that there is a model thatcan make this possibleThe model will need toRecord all actionsRefresh the suggestions at every change of state
  6. We are focusing on the information and actions that (we think) is / are important for doing things that are usually done on this website.We are removing the need to manage sequences of actions, and sets of sequences of actionsWe do not leave the screen-reader interface and keep control in user’s handsThe user can always deviate from what s/he is doing, into / from a scenario, and the system should pick up from there
  7. Not using a computational model because we do not want the deficiencies of a computational model to affect the results.
  8. Limited at 10 min. A few FAILSThe users were reasonably proficient
  9. Add numbersMore results