Jeff Rzeszotarski
[rez-oh-tar’-ski]
4th Year Ph.D. Student
HCII, CMU
www.jeffrz.com
jeffrz@cs.cmu.edu
Instrumenting the Cr...
James Cridland @ Flickr
James Cridland @ Flickr
James Cridland @ Flickr
Amazon Mechanical Turk
oDesk
CrowdFlower
Bernstein et al. 2010
Task Design
Gold Standard
Worker Agreement
Task Design
Cheating Resistant Design
Kittur, Chi, Suh, 2008
Von Ahn & Dabbish, 2004
Gold Standards
Gold Standard Worker 1 Worker 2
A A A
D B D
B C
A A
2 4 2
2 8
Callison-Burch, 2009
Downs et al. 2010
Worker Agreement
Callison-Burch, 2009 Ipeirotis et al. 2010
Dekel & Shamir, 2009 Snow et al. 2008
WorkerID Video Transcrip...
Disadvantages
Task Design
- Task-dependent
- Time consuming iteration
Gold Standard
- Not always available
- Not always ap...
Worker A Worker B
Worker A
Worker A
Worker A
Worker A
Worker A
Worker B
Worker B
Worker B
Worker B
Worker B
Worker B
Worker B
Logging
ABC
SCROLL
MOUSE
MOVE
MOUSE
MOVE CLICK CLICKDELAY DELAYTYPING
120px 300px,
20px
Field1 1.1s ‘Hello’ 0.9s 100px,
22...
Features
Experimental MTurk Tasks
Noun Classification
“Identify which words are nouns.”
Image Tagging
“Generate 3-5 tags for each o...
Image Tagging
Provide at least 3 tags for
the following image…
ocean mountains
pretty
x4
Image Tagging
Image Tagging
52 participants, 114 submissions
Human Labeled Cheating
17 cheats, 97 regular
Human Labeled Quality
AVG 3.5 ...
Model Features
# Unique Letters
Total Time Elap.
Image Tagging
17 cheats, 97 regular
decision tree, 10-fold CV
93.0% accur...
Image Tagging
Regressing on ‘quality’
SMOreg, 10-fold CV
r=0.5874 (generally ±0.5 on Likert)
Model Features
# Unique Lette...
Generalizing Models
Training
Reading
Comprehension
63 points
Testing
Noun
Identification
185 Points
Model predicts for a s...
Other Findings
Clustering workers by
behavioral patterns
De-anonymizing
worker submissions
Hanging Questions
Complex & creative work?
Human interpretability
Organizer cognitive models
CrowdScape
ABC
SCROLL
MOUSE
MOVE
MOUSE
MOVE CLICK CLICKDELAY DELAYTYPING
120px 300px,
20px
Field1 1.1s ‘Hello’ 0.9s 100px,
220px
Subm...
Interac(vely	
  visualizing	
  user	
  
behavior	
  and	
  output
Thanks, d3 toolkit
Series and series of works starting with “Godzilla"
monster movie Godzilla was released in special
effects is the name of ...
I like to visit Alappuzha in Kerala, the God's own country.
This place is blessed with heavenly lakes. The view of lake
sh...
Have you ever wondered what causes swing sets to move
with no one on them? Or why bridges twist and tear
themselves apart?...
Limitations
Logging is not perfect
Information overload
Worker traces
Parallel coordinates
Interpreting behavior
Feature s...
Instrumenting the Crowd + Task Fingerprinting Overview
Instrumenting the Crowd + Task Fingerprinting Overview
Instrumenting the Crowd + Task Fingerprinting Overview
Instrumenting the Crowd + Task Fingerprinting Overview
Instrumenting the Crowd + Task Fingerprinting Overview
Instrumenting the Crowd + Task Fingerprinting Overview
Instrumenting the Crowd + Task Fingerprinting Overview
Instrumenting the Crowd + Task Fingerprinting Overview
Instrumenting the Crowd + Task Fingerprinting Overview
Instrumenting the Crowd + Task Fingerprinting Overview
Instrumenting the Crowd + Task Fingerprinting Overview
Instrumenting the Crowd + Task Fingerprinting Overview
Instrumenting the Crowd + Task Fingerprinting Overview
Instrumenting the Crowd + Task Fingerprinting Overview
Instrumenting the Crowd + Task Fingerprinting Overview
Instrumenting the Crowd + Task Fingerprinting Overview
Instrumenting the Crowd + Task Fingerprinting Overview
Instrumenting the Crowd + Task Fingerprinting Overview
Instrumenting the Crowd + Task Fingerprinting Overview
Instrumenting the Crowd + Task Fingerprinting Overview
Instrumenting the Crowd + Task Fingerprinting Overview
Instrumenting the Crowd + Task Fingerprinting Overview
Instrumenting the Crowd + Task Fingerprinting Overview
Instrumenting the Crowd + Task Fingerprinting Overview
Instrumenting the Crowd + Task Fingerprinting Overview
Instrumenting the Crowd + Task Fingerprinting Overview
Instrumenting the Crowd + Task Fingerprinting Overview
Instrumenting the Crowd + Task Fingerprinting Overview
Instrumenting the Crowd + Task Fingerprinting Overview
Upcoming SlideShare
Loading in...5
×

Instrumenting the Crowd + Task Fingerprinting Overview

58

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
58
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Instrumenting the Crowd + Task Fingerprinting Overview"

  1. 1. Jeff Rzeszotarski [rez-oh-tar’-ski] 4th Year Ph.D. Student HCII, CMU www.jeffrz.com jeffrz@cs.cmu.edu Instrumenting the Crowd
  2. 2. James Cridland @ Flickr
  3. 3. James Cridland @ Flickr
  4. 4. James Cridland @ Flickr Amazon Mechanical Turk oDesk CrowdFlower
  5. 5. Bernstein et al. 2010
  6. 6. Task Design Gold Standard Worker Agreement
  7. 7. Task Design Cheating Resistant Design Kittur, Chi, Suh, 2008 Von Ahn & Dabbish, 2004
  8. 8. Gold Standards Gold Standard Worker 1 Worker 2 A A A D B D B C A A 2 4 2 2 8 Callison-Burch, 2009 Downs et al. 2010
  9. 9. Worker Agreement Callison-Burch, 2009 Ipeirotis et al. 2010 Dekel & Shamir, 2009 Snow et al. 2008 WorkerID Video Transcript AF34DB9 Coloring comics in Photoshop is easy. First you… QR40Q2 <blank> TK421S9S Coloring comcs in photoshop is easy first u… WOPR21 QOJEORJOFJDSKGNDKSNGSDSJGKDSJGLKSDJGL APC3C55 Coloring comics in Photoshop is easy. First you… IDKFAD1 Coloring comics in photoshop is easy. First you…
  10. 10. Disadvantages Task Design - Task-dependent - Time consuming iteration Gold Standard - Not always available - Not always applicable Worker Agreement - Limits response range - Subject to majority effects - Not always applicable
  11. 11. Worker A Worker B
  12. 12. Worker A
  13. 13. Worker A
  14. 14. Worker A
  15. 15. Worker A
  16. 16. Worker A
  17. 17. Worker B
  18. 18. Worker B
  19. 19. Worker B
  20. 20. Worker B
  21. 21. Worker B
  22. 22. Worker B
  23. 23. Worker B
  24. 24. Logging ABC SCROLL MOUSE MOVE MOUSE MOVE CLICK CLICKDELAY DELAYTYPING 120px 300px, 20px Field1 1.1s ‘Hello’ 0.9s 100px, 220px Submit Example Log: Typing and submitting “Hello”
  25. 25. Features
  26. 26. Experimental MTurk Tasks Noun Classification “Identify which words are nouns.” Image Tagging “Generate 3-5 tags for each of 4 images.” Reading Comprehension “Read and then answer the following questions.”
  27. 27. Image Tagging Provide at least 3 tags for the following image… ocean mountains pretty x4
  28. 28. Image Tagging
  29. 29. Image Tagging 52 participants, 114 submissions Human Labeled Cheating 17 cheats, 97 regular Human Labeled Quality AVG 3.5 / 5, SD 1.13
  30. 30. Model Features # Unique Letters Total Time Elap. Image Tagging 17 cheats, 97 regular decision tree, 10-fold CV 93.0% accuracy
  31. 31. Image Tagging Regressing on ‘quality’ SMOreg, 10-fold CV r=0.5874 (generally ±0.5 on Likert) Model Features # Unique Letters Total Time Elapsed # of Clicks Fields Accessed
  32. 32. Generalizing Models Training Reading Comprehension 63 points Testing Noun Identification 185 Points Model predicts for a similar task r=0.4948
  33. 33. Other Findings Clustering workers by behavioral patterns De-anonymizing worker submissions
  34. 34. Hanging Questions Complex & creative work? Human interpretability Organizer cognitive models
  35. 35. CrowdScape
  36. 36. ABC SCROLL MOUSE MOVE MOUSE MOVE CLICK CLICKDELAY DELAYTYPING 120px 300px, 20px Field1 1.1s ‘Hello’ 0.9s 100px, 220px Submit
  37. 37. Interac(vely  visualizing  user   behavior  and  output
  38. 38. Thanks, d3 toolkit
  39. 39. Series and series of works starting with “Godzilla" monster movie Godzilla was released in special effects is the name of the fictional monster of their work. The film named “Godzilla" is a special effects monster movie, and Godzilla is the monster. ゴジラはに公開した特撮怪獣映画『ゴジラ』に始まる一連のシリーズ 作品及び、それらの作品に登場する架空の怪獣の名称である。  
  40. 40. I like to visit Alappuzha in Kerala, the God's own country. This place is blessed with heavenly lakes. The view of lake shores from the boats is very beautiful. We can enjoy cool wind standing in the boats. We can see a complete picture of a beautiful village by a trip through the lakes of Alappuzha. My favorite place in the world to be is right at home! Ideally my daughter and husband and I just spending time together watching television, sharing a meal, or discussing our day. I know it might sound boring to some but I find joy and security in the familiar surroundings. Sure, there are other places I enjoy but given the choice my home will always be the place I am the happiest.
  41. 41. Have you ever wondered what causes swing sets to move with no one on them? Or why bridges twist and tear themselves apart? The answer is aeroelastic tension. But the science behind it is the most interesting part. Watch this and you'll find out how it works in less than two minutes. collapse bridge physics tension aeroelastic Tacoma Narrows Bridge Collapse
  42. 42. Limitations Logging is not perfect Information overload Worker traces Parallel coordinates Interpreting behavior Feature set, ML algorithms

×