The document describes a system called AutoMan that integrates human computation via Amazon Mechanical Turk (MTurk) into programming. AutoMan allows programmers to write functions that are implemented by having MTurk workers complete small tasks. It addresses challenges like ensuring quality work from workers and preventing gaming of the system. AutoMan manages pricing, timing of tasks, and number of workers to balance cost, speed and accuracy of results for user-defined functions.
54. How
many
giraffes
are
in
this
picture?
k = 3 choices!
AutoMan
handles
“radio
button”
questions
55. How
many
giraffes
are
in
this
picture?
k = 3 choices!Risk: Homer &
Bender always guess
56. How
many
giraffes
are
in
this
picture?
k = 3 choices!E.g., always choose
first option.
57. How
many
giraffes
are
in
this
picture?
k = 3 choices!
To combat this,
AutoMan randomizes
answers.
58. 25 choices!
Which
are
from
Sesame
Street?
Kermit
the
Frog
Spongebob
Squarepants
Cookie
Monster
The
Count
Oscar
the
Grouch
☐
☐
☐
☐
☐
“Checkbox” questions
59. Which
are
from
Sesame
Street?
Kermit
the
Frog
Spongebob
Squarepants
Cookie
Monster
The
Count
Oscar
the
Grouch
þ
þ
þ
þ
þ
25 choices!
Same risk: random respondents
60. Which
are
from
Sesame
Street?
Kermit
the
Frog
Spongebob
Squarepants
Cookie
Monster
The
Count
Oscar
the
Grouch
þ
þ
☐
þ
☐
25 choices!
AutoMan checks each randomly
61. What
does
this
license
plate
say?
36d choices!
XXXXXX
366 choices = !2176782336[A-Z0-9]{6}!
Last question category:
constrained free-text
62. Which
one
of
these
doesn’t
belong?
[95%
conf.]
AUTOMAN:
spawns
3
tasks
@
$0.06;
30s
work
t1
t2
t3
Example real execution
63. Which
one
of
these
doesn’t
belong?
[95%
conf.]
AUTOMAN:
spawns
3
tasks
@
$0.06;
30s
work
t1
t2
t3
1m
50s
64. Which
one
of
these
doesn’t
belong?
[95%
conf.]
AUTOMAN:
spawns
3
tasks
@
$0.06;
30s
work
t1
t2
t3
1m
50s
2m
30s
65. Which
one
of
these
doesn’t
belong?
[95%
conf.]
AUTOMAN:
spawns
3
tasks
@
$0.06;
30s
work
t1
t2
t3
1m
50s
2m
30s
2m
50s
66. Which
one
of
these
doesn’t
belong?
[95%
conf.]
AUTOMAN:
spawns
3
tasks
@
$0.06;
30s
work
t1
t2
t3
1m
50s
2m
30s
2m
50s
Inconclusive!
67. Which
one
of
these
doesn’t
belong?
[95%
conf.]
AUTOMAN:
spawns
3
more
tasks
t1
t2
t3
t4
t5
t6
68. Which
one
of
these
doesn’t
belong?
[95%
conf.]
AUTOMAN:
spawns
3
more
tasks
t1
t2
t3
t4
t5
t6
7m
69. Which
one
of
these
doesn’t
belong?
[95%
conf.]
AUTOMAN:
spawns
3
more
tasks
t1
t2
t3
t4
t5
t6
18m
50s
7m
70. Which
one
of
these
doesn’t
belong?
[95%
conf.]
AUTOMAN:
spawns
3
more
tasks
t1
t2
t3
t4
t5
t6
7m
18m
50s
51m
Timeout: double pay and time
71. Which
one
of
these
doesn’t
belong?
[95%
conf.]
AUTOMAN:
spawn
1
more
task
@
$0.12,
60s
work
t1
t2
t3
t4
t5
t6
t7
72. Which
one
of
these
doesn’t
belong?
[95%
conf.]
AUTOMAN:
spawn
1
more
task
@
$0.12,
60s
work
t1
t2
t3
t4
t5
t6
t7
1h
9m
50s;
cost
=
$0.36
AUTOMAN:
5
out
of
6
⇒
95%
confidence;
return