Trailblazer: Enabling Blind Web Users to Blaze Trails Through the Web

Jeffrey P. Bigham University of Washington CSE [email_address] Enabling Blind Users to Blaze Trails Through the Web Tessa Lau and Jeffrey Nichols IBM Almaden Research {tessalau, jwnichols}@us.ibm.com start

Scale of Access

37 Million Blind People in the World

(125 million with low vision)

Increasing

Aging Population

Situational Impairments

Access technology benefits 25% of web users 1

[1] “The Market for Accessible Technology. “ Forrester Research, 2003.

Web Potential for Blind People

Web Advantages

Convertible to Accessible Form

Shop From Home

Blind People Read 7x as Many Books

The Law

U.S. Government Requires Accessibility 1,2

NFB vs. Target Corporation 2

Making Access Possible is Not Enough

Should be Intuitive and Fun

[1] Section 508 of the U.S. Rehabilitation Act. [2] Americans with Disabilities Act.

Keyboard Shortcuts X X X X h h h e b b START

Keyboard Shortcuts X X X X <span class=“heading”> e b b START <span class=“heading”> <span class=“heading”> .heading { font-weight: bold; }

Task: Placing Book on Hold 1. Enter “Book You Want” 2. Click “Find” Button 3. Find Correct Result 4. Click “Place Hold” Button 5. Click “Request” Button 6. Verify Queue Position

How can blind people more effectively complete web tasks?

Avoid Linear Searches

Direct Users to Important Content

Users Maintain Control

Interface should be understandable

Guide users not direct them

Flexible and Adaptable

Work in full web domain

Compatibility with Tools Users Know

Supplement to existing tools (screen readers)

CoScripter 1 : Recording and Replaying Tasks

Pseudo-Natural Language Commands

[1] Little et al. “Koala: capture, share, automate, personalize business processes on the web.” CHI 2007.

go to “http://www.united.com”

enter your “flight number” (e.g., 0168) into the “Flight number” textbox

enter your “airport” (e.g., SJC) into the “From:” textbox

click the “Check” button

CoScript Repository

Example TrailBlazer Placing a book on hold at the Seattle Public Library.

Set Focus & Provide Contextual Help Step description. Inline Playback Controls

Formative Evaluation

Generalizing with Suggestions Choose from one of the following suggestions: Click the “Check” button. Enter into the “From:” textbox. Enter into the “To:” textbox. Select from the “Date:” listbox. Select from the “Time:” listbox.

Providing Suggestions

User enters a short task description (query)

“ flight status on united”

“ place hold on book at Seattle library”

(etc.)

Machine learning model ranks all actions.

Features

Task Description

Related Scripts from the Script Repository

User’s History of Actions

Apply existing scripts to new tasks or new sites

No need to explicitly find script first

Click the “profile” link. Click the “Home” link. Click the “Books” link. … Enter into the “Name” textbox. Enter into the “Zip” textbox. … Click the “Submit” button. Click the “Check Now” button. … Turn on the “Yes” Radio Button. … . Example Action List x 1 x 2 x 6 … y

Generalizing Scripts to New Sites 0.08 0.27 0.10 0.76 0.20 “ flight status on united” Task Description 1. go to “www.aa.com” 2. enter your “Flight Number” (e.g. 144) into the “Flight Number:” textbox 3. click the fifth “GO” button Flight Status on American Airlines 1. go to “http://www.southwest.com/” 2. select “SJC” from the “Select the Departure City:” listbox 3. enter your “flight number” into the first “Enter the Flight Number” textbox 4. click the first “View Flight Status Information” button Flight Status on Southwest Airlines Click “my profile” link Click “Flight Information” link Enter into “Mileage Plus” textbox Enter into “flight number” textbox Click “Check” button Possible Actions similarity ( , ) Repository

Using History to Match Sub-Tasks Click “my profile” link Enter into ‘password’ password box Click “Login” button Enter into “flight number” textbox Click “Check” button 0.08 0.88 0.10 0.36 0.20 Last Action Update Facebook Status Reserve UW Meeting Room Possible Actions 1. go to “www.facebook.com” 2. enter your “facebook username” into the “username” textbox 3. enter your “facebook password” into the password box 4. Click “Login” button 5. … 7. … 8. enter your “CSE login” into the “username” textbox 9. enter your “CSE password” into the password box 10. … 0.00 0.87 0.46 0.14 0.04 0.00 0.75 0.42 0.13 “ enter into ‘username’ textbox” similarity ( , ) Repository

Evaluation

Tested on 102 Actions

15 Most Popular CoScripts

User-provided title used as task description

6.8 steps per script

42 possible actions

Leave-One-Out Cross-Validation

Removed from repository too

Top 5 in 76% of cases

(5.1 words on avg.) Suggestions Provided CoScript Repository

Ongoing and Future Work

User Studies

What happens when TrailBlazer is wrong?

Incentives vs. Cost

Better Model

Continuous Learning

Popularity, Past Behavior, etc.

Improved Web Access

Voice Over Mobile Phones

Small Screens

Conclusion

TrailBlazer makes web access more efficient

Record, replay and share trails

Targets non-visual use

Uses existing repository of trails

Suggestion Component

Finds appropriate trails

Generalizes existing scripts to new tasks and sites

The End Thanks to : Allen Cypher, Clemens Drews, Jimmy Lin, Yevgen Borodin, Charles Chen and T.V. Raman. Want to hire me? [email_address] www.jeffreybigham.com

Directing Users to Content

Direct to Content

Avoid inefficient linear searches

5 Participants/20 Regions

Mechanical Turk

3 Commonalities

Semantic Desciption (78%)

Heading of Region (53%)

Words Used in Region (37%)

Clip the “search results” region

Directing Users to Content

Direct to Content

Avoid linear searches

5 Descriptions of 20 Regions

Mechanical Turk

27 Users

Clip the “search results” region Clip the region containing “batting average”

Generalizing Scripts to New Sites 0.08 0.27 0.10 0.76 0.20 “ flight status on united” Task Description 1. go to “www.aa.com” 2. enter your “Flight Number” (e.g. 144) into the “Flight Number:” textbox 3. click the fifth “GO” button Flight Status on American Airlines 1. go to “http://www.southwest.com/” 2. select “SJC” from the “Select the Departure City:” listbox 3. enter your “flight number” into the first “Enter the Flight Number” textbox 4. click the first “View Flight Status Information” button Flight Status on Southwest Airlines Click “my profile” link Click “Flight Information” link Enter into “Mileage Plus” textbox Enter into “flight number” textbox Click “Check” button Possible Actions Actions a Scripts s 1: enter 1: flight 1: number 1: textbox 2: button 1: city 2: click 1: departure 3: enter 4: flight 3: go 1: information 1: listbox 4: number 1: select 1: SJC 1: status 2: textbox 1: view 2: your Similarity = cos( a , s ) Repository

Matching Partial Scripts Click “my profile” link Enter into ‘password’ password box Click “Login” button Enter into “flight number” textbox Click “Check” button 0.08 0.88 0.10 0.76 0.20 Last Action Update Facebook Status Reserve UW Meeting Room Possible Actions Action a Scripts s 1: enter 2: password 1: box box: 0.9 button: 0.1 click: 0.0 CSE: 1.2 enter: 2.5 facebook: 1.2 go: 0.0 login: 0.1 password: 1.8 textbox: 1.6 W(s i ) = d * W(s i-1 ) + cos( t , s i ) 1. go to “www.facebook.com” 2. enter your “facebook username” into the “username” textbox 3. enter your “facebook password” into the password box 4. Click “Login” button 5. … 7. … 8. enter your “CSE login” into the “username” textbox 9. enter your “CSE password” into the password box 10. … 0.30 * 0.00 0.30 * 0.00 + 0.87 0.30 * 0.87 + 0.20 0.30 * 0.46 + 0.00 0.30 * 0.14 + 0.00 0.30 * 0.00 + 0.00 0.30 * 0.00 + 0.75 0.30 * 0.75 + 0.20 0.30 * 0.43 + … 0.00 0.87 0.46 0.14 0.04 0.00 0.75 0.42 0.13 d = 0.30 “ enter into ‘username’ textbox” Repository