The document discusses using crowdsourcing and machine learning techniques to analyze data. It describes a hybrid crowd-machine learning platform that has participants complete microtasks to nominate predictive features for machine learning models. These models can then be used to make predictions, with the goal of improving accuracy by combining human and machine intelligence. Charts and graphs are used in some microtasks to help workers analyze trends in the data and provide explanations for their assessments.

3. 3/41
●
●
●
-
-

4. 4/41
●
-
mammal placental carnivore canine dog working dog husky
vehicle craft watercraft sailing vessel sailboat trimaran
Figure 1: A snapshot of two root-to-leaf branches of ImageNet: the top row is from the mammal subtree; the bottom row is from the
vehicle subtree. For each synset, 9 randomly sampled images are presented.
Summary of selected subtrees
Imagenet Cat SubtreeESP Cat Subtree

5. 5/41
●

6. 6/41
Ranjay Krishna et al.
8 Ranjay Kris
Fig. 6: From all of the region descriptions, we extract
all objects mentioned. For example, from the region de-
scription “man jumping over a fire hydrant,” we extract
man and fire hydrant.
Fig. 8: Our dataset also captures the relations
interactions between objects in our images. In
ample, we show the relationship jumping o
tween the objects man and fire hydrant.
over on the object fire hydrant. Each
Ranjay Krishna et al.

7. 7/41
●
-
●
(e.g., how much to reward workers, whether to reject their
work, or impose a reputation penalty) their power is
attenuated due to factors such as lack of direct supervision
and visibility into their work behavior, lack of nuanced and
individualized rewards, and the difficulty of imposing
stringent and lasting sanctions (since workers can leave
Th
w
va
m
be
pa
to
pr
an
st
m
A
w
ea
B
te
cr
sp
ne
fo
th
un
fo
pr
cu
ex
th
ef
cl
in
[6
Figure 2: Proposed framework for future crowd work
processes to support complex and interdependent work.

8. 8/41
●
o
o
●
●

9. 9/41
●
-
●
-
-
Microtasks and Crowdsourcing #chi4good, CHI 2016, San Jose, CA, USA

11. 11/41
●
●
●
FALSE
TRUE
TRUE
TRUE?
FALSE?

12. 12/41
●
TRUE TRUE TRUE
FALSE TRUE TRUE
TRUE FALSE TRUE
?
?
?

13. 13/41
● 𝑗
●
TRUE FALSE
TRUE
FALSE
↵j
1
↵j
0
↵j
1
↵j
0

14. 14/41
●
TRUE TRUE TRUE
FALSE TRUE TRUE
TRUE FALSE TRUE
?
?
?

15. 15/41
●
-
-
●
wj ( , + )
xi [0, + )
Pr [yij = ti] =
1
1 + exp ( wjxi)

16. 16/41
●
●
Pr [yij = 1] =
w>
j xi ⌧j
j
!
xi
wj

17. 17/41
●
-
-
●
(d) ARJ
(b) SRJ
(c) CF
Accurate (28%) Mosty accurate (12%) Outer class biased (29%) Middle class biased (29%)
uth Predicted Truth Predicted Truth Predicted Truth Predicted
Truth PredictedTruth PredictedTruth Predicteduth Predicted
Truth PredictedTruth PredictedTruth Predicted
0 0 0
11
0.5
0
11
0.5
0
11
0
0.5
-1
01
-1
0
1
-1
01 1 0 -1-1
01
1
0 -1
1
0
-1
1 0 -1
-1
01
-1
01
-1
0 1 0 -1-1
01 0 -10 -1
1 0 -1
11 1
0 1 234
0
12 3
34
0 1 234
012 3
34
0 1 234
012 3
34
0.8
0.4
0
0.8
0.4
0
0.8
0.4
0
0.8
Middle class biased (34%)Accurate (28%) Mosty accurate (12%) Outer class biased (29%)
Decisive (5%)Conservative (4%)Calibrated (91%)

18. 18/41
●
●
TRUE FALSE
TRUE
FALSE
：
j
10
j
10
j
11
j
01
j
00

19. 19/41
●
●

20. 20/41
●
●
-
860 43 …
43 215 215 860 860
860 860 215 215 215 215

21. 21/41
“A silver tabby
cat is howling
with his mouth
wide open’’
“A sleeping
cat’’
“Dreaming of
becoming a
lion’’
4.7
1.2
2.6

22. 22/41
●
●
●
2.6
“Dreaming
of becoming
a lion’’

23. 23/41
●
-
-
●
-
-
µa
a
r
r

25. 25/41
●
●
-
-

26. 26/41
●
-
●
-

27. 27/41
●
●
-
-
●
-
-

28. 28/41
●
-
-
healthline
Figure 3: Example ad to attract users
time a user clicks on the a
we record a conversion ev
the advertising system. T
the system to optimize th
mizing the number of con
contribution yield, instea
the number of clicks.
Although optimizing fo
Figure 1: Screenshot of the Quizz system.

29. 29/41
●
●
-
-

30. 30/41

31. 31/41
qmw qz qua qb
qM
QqN
TI
K
e
UKu
u
E D P 2
V
qt qJ
'2
V'I
'
E
qmyM
K
Figure 1: Fine Grained Truth Discovery Model for Crowd-
sourced Data Aggregation.
• For the q-th question (q = 1, 2, · · · , Q)
– Draw a topic zq ∼ Multi(θ)
to question q. Le
topical words in qu
ground words, θk
φkw = p(w|k) be
topic k. Then the
times in question q
ity of background
independent, and t
der topic k and wo
p(wq|k, y
where V is the num
V
w=1(nw
q,y=1 + n
dent, and the proba
p(w|θ, φ, φ′
) =
𝑞
𝑞
𝑚
𝑞 𝑢
𝑘 𝑞 𝑢
𝜎 𝑥 = 1/ 1 + exp −𝑥
(⇢kueku bq)
𝑘 𝑢
𝑘
𝑢
𝑞

32. 32/41
●
-
●
●
Q1. “The accident triggered the establishment of the Law for the …”
☐ ☑☑ ☑

34. ● 𝑥0, 𝑦0
● 𝑥0, 𝑦03, 𝑦04, ⋯
●
-
-
34/41
Pr [yi = 1 | xi] =
1
1 + exp ( w>xi)
Pr [yij | yi = 1] =
⇣
↵j
1
⌘yij
⇣
1 ↵j
1
⌘(1 yij )

35. 35/41
●
●
●
Visual Recognition with Humans in the Loop 3
er vision is helpful Computer vision is not helpfuler vision is helpful Computer vision is not helpful
The bird is a
Black‐footed
Albatross
Is the belly
white? yes
Are the eyes
white? yes
Th bi d i
Is the beak cone‐shaped? yes
Is the upper‐tail brown? yes
Is the breast solid colored? no
Is the breast striped? yes
I h h hi ?The bird is a
Parakeet Auklet
Is the throat white? yes
The bird is a Henslow’s
Sparrow
amples of the visual 20 questions game on the 200 class Bird dataset.
ponses (shown in red) to questions posed by the computer (shown in blue)
drive up recognition accuracy. In the left image, computer vision algorithms
he bird species correctly without any user interaction. In the middle image,
vision reduces the number of questions to 2. In the right image, computer
ides little help.
YES
YES
NO

36. 36/41
●
-
Crowd-Machine Learning Classifiers
Justin Cheng and Michael S. Bernstein
Stanford University
{jcccf, msb}@cs.stanford.edu
ing classifiers: clas-
en description of a
predictive features
tures using machine
rate and use human-
lassifiers enable fast
that can improve on
judgment, and ac-
extraction is not yet
arning platform, in-
nformative features,
d examples, an ap-
g. The crowd’s ef-
s of the input space
Figure 1. Flock is a hybrid crowd-machine learning platform that capi-
talizes on analogical encoding to guide crowds to nominate effective fea-

37. 37/41
●
-
●
-
-
…

38. 38/41
0.4
0.6
0.8
1.0
Aug 19 Aug 26 Sep 02 Sep 09 Sep 16
提出日
AUC
参加者1
参加者2
参加者3
参加者4
参加者5
参加者6
参加者7
参加者8
参加者9
参加者10
参加者11
参加者12
参加者13
参加者14
参加者15
参加者16
NLDR
LM
統合
AUC=0.982
AUC=0.946
AUC=0.717

39. 39/41
●
-
●
●

40. 40/41
Figure 3. An example analysis microtask shows a single chart (a)
along with chart-reading subtasks (b) an annotation subtask (c) and
a feature-oriented explanation prompt designed to encourage workers
Figure 3. An example analysis microtask shows a single chart (a)
along with chart-reading subtasks (b) an annotation subtask (c) and
a feature-oriented explanation prompt designed to encourage workers

41. 41/41
●
-
●
-
●
-
-
-