TurKit: A Toolkit for Human Computation Algorithms

1
TurKit: A Toolkit for Human Computation Algorithms
Rob Miller & Greg Little
User Interface Design Group
MIT CSAIL
Joint work with Lydia Chilton, Max Goldman, Jeff Bigham, Aubrey Tatarowicz, Rajeev Najak, Michael Bernstein
June 10, 2010

Outline
Iterative human computation
TurKit: a toolkit for human computation algorithms
Systems with human computation inside
VizWiz: vision for blind users with camera phones
Soylent: putting a crowd inside Microsoft Word
June 10, 2010
2

What is TurKit?

2 Ways
What is TurKit?

Way 1
ADD 1, 2

Way 1
ADD 1, 2
JMP somewhere

Way 1
ADD 1, 2
JMP somewhere
…

Way 1
ADD 1, 2
JMP somewhere
…
TURK “How do you feel?”

Demo
TurKit

What can you do with TurKit?

Improve

Vote
Improve

Handwriting Recognition

Image Description

Outline to Prose
Progression of Other Offer Description:
$10,000 more
better package
far more competitive offer
more competitive offer
competing offer

Picture Sorting

Subjective Ratings
x

Subjective Ratings

Workflow Comparisons

Iterative

Parallel
Iterative

Which is better?

Which is better?
30 images

Which is better?
30 images
6 iterations

Which is better?

Larger Algorithms

Outline
June 10, 2010
58

VizWiz: Helping the Blind See
59
joint work with Jeff Bigham (University of Rochester)
Which door is the women’s restroom?
June 10, 2010

Helping the Blind See
60
MechanicalTurk
the left one
LEFT
on the left
June 10, 2010

Field Study
Field deployment with 11 blind iPhone users
Answers received within a minute for ~5 cents a question
Latency can be reduced to less than 30 seconds by keeping workers warmed up (at $4 per hour)
June 10, 2010
61

All Mobile Users Are Situationally Disabled
62
joint work with Rajeev Najak
MechanicalTurk
When is Barack Obama speaking at MIT?
June 10, 2010

Soylent: Putting a Crowd inside MS Word
June 10, 2010
joint work with Michael Bernstein
63

Crowd-Driven Proofreading
June 10, 2010
64

Crowd-Driven Shortening
June 10, 2010
65

Bigger Idea: REAL Wizard of Oz
Wizard of Oz is a tried-and-true prototyping technique in AI and HCI
Putting a human behind the curtain until we figure out how to put software there
Crowd computing enables Wizard of Oz systems that are useful and deployable
So we can start collecting data about how the system is really used in practice
Adding the AI backend becomes a performance or cost optimization
66
June 10, 2010

Outline
Funded in part by
June 10, 2010
67

TurKit: A Toolkit for Human Computation Algorithms

by Greg Little

Accessibility

Categories

Upload Details

Usage Rights

1 Embed 1

Statistics