11. Gold Standards
Gold Standard Worker 1 Worker 2
A A A
D B D
B C
A A
2 4 2
2 8
Callison-Burch, 2009
Downs et al. 2010
12. Worker Agreement
Callison-Burch, 2009 Ipeirotis et al. 2010
Dekel & Shamir, 2009 Snow et al. 2008
WorkerID Video Transcript
AF34DB9 Coloring comics in Photoshop is easy. First you…
QR40Q2 <blank>
TK421S9S Coloring comcs in photoshop is easy first u…
WOPR21 QOJEORJOFJDSKGNDKSNGSDSJGKDSJGLKSDJGL
APC3C55 Coloring comics in Photoshop is easy. First you…
IDKFAD1 Coloring comics in photoshop is easy. First you…
13. Disadvantages
Task Design
- Task-dependent
- Time consuming iteration
Gold Standard
- Not always available
- Not always applicable
Worker Agreement
- Limits response range
- Subject to majority effects
- Not always applicable
33. Experimental MTurk Tasks
Noun Classification
“Identify which words are nouns.”
Image Tagging
“Generate 3-5 tags for each of 4 images.”
Reading Comprehension
“Read and then answer the following questions.”
37. Model Features
# Unique Letters
Total Time Elap.
Image Tagging
17 cheats, 97 regular
decision tree, 10-fold CV
93.0% accuracy
38. Image Tagging
Regressing on ‘quality’
SMOreg, 10-fold CV
r=0.5874 (generally ±0.5 on Likert)
Model Features
# Unique Letters
Total Time Elapsed
# of Clicks
Fields Accessed
62. Series and series of works starting with “Godzilla"
monster movie Godzilla was released in special
effects is the name of the fictional monster of
their work.
The film named “Godzilla" is a special effects
monster movie, and Godzilla is the monster.
ゴジラはに公開した特撮怪獣映画『ゴジラ』に始まる一連のシリーズ
作品及び、それらの作品に登場する架空の怪獣の名称である。
63.
64.
65.
66. I like to visit Alappuzha in Kerala, the God's own country.
This place is blessed with heavenly lakes. The view of lake
shores from the boats is very beautiful. We can enjoy cool
wind standing in the boats. We can see a complete
picture of a beautiful village by a trip through the lakes of
Alappuzha.
My favorite place in the world to be is right at home!
Ideally my daughter and husband and I just spending
time together watching television, sharing a meal, or
discussing our day. I know it might sound boring to some
but I find joy and security in the familiar surroundings. Sure,
there are other places I enjoy but given the choice my
home will always be the place I am the happiest.
67.
68.
69.
70. Have you ever wondered what causes swing sets to move
with no one on them? Or why bridges twist and tear
themselves apart? The answer is aeroelastic tension. But the
science behind it is the most interesting part. Watch this and
you'll find out how it works in less than two minutes.
collapse bridge physics tension aeroelastic
Tacoma Narrows Bridge Collapse
71. Limitations
Logging is not perfect
Information overload
Worker traces
Parallel coordinates
Interpreting behavior
Feature set, ML algorithms
![Jeff Rzeszotarski
[rez-oh-tar’-ski]
4th Year Ph.D. Student
HCII, CMU
www.jeffrz.com
jeffrz@cs.cmu.edu
Instrumenting the Crowd](https://image.slidesharecdn.com/taskfuwv3-130919130329-phpapp01/85/Instrumenting-the-Crowd-Task-Fingerprinting-Overview-1-320.jpg)
