Instrumenting the Crowd + Task Fingerprinting Overview

  • 46 views
Uploaded on

 

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
46
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
0
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Jeff Rzeszotarski [rez-oh-tar’-ski] 4th Year Ph.D. Student HCII, CMU www.jeffrz.com jeffrz@cs.cmu.edu Instrumenting the Crowd
  • 2. James Cridland @ Flickr
  • 3. James Cridland @ Flickr
  • 4. James Cridland @ Flickr Amazon Mechanical Turk oDesk CrowdFlower
  • 5. Bernstein et al. 2010
  • 6. Task Design Gold Standard Worker Agreement
  • 7. Task Design Cheating Resistant Design Kittur, Chi, Suh, 2008 Von Ahn & Dabbish, 2004
  • 8. Gold Standards Gold Standard Worker 1 Worker 2 A A A D B D B C A A 2 4 2 2 8 Callison-Burch, 2009 Downs et al. 2010
  • 9. Worker Agreement Callison-Burch, 2009 Ipeirotis et al. 2010 Dekel & Shamir, 2009 Snow et al. 2008 WorkerID Video Transcript AF34DB9 Coloring comics in Photoshop is easy. First you… QR40Q2 <blank> TK421S9S Coloring comcs in photoshop is easy first u… WOPR21 QOJEORJOFJDSKGNDKSNGSDSJGKDSJGLKSDJGL APC3C55 Coloring comics in Photoshop is easy. First you… IDKFAD1 Coloring comics in photoshop is easy. First you…
  • 10. Disadvantages Task Design - Task-dependent - Time consuming iteration Gold Standard - Not always available - Not always applicable Worker Agreement - Limits response range - Subject to majority effects - Not always applicable
  • 11. Worker A Worker B
  • 12. Worker A
  • 13. Worker A
  • 14. Worker A
  • 15. Worker A
  • 16. Worker A
  • 17. Worker B
  • 18. Worker B
  • 19. Worker B
  • 20. Worker B
  • 21. Worker B
  • 22. Worker B
  • 23. Worker B
  • 24. Logging ABC SCROLL MOUSE MOVE MOUSE MOVE CLICK CLICKDELAY DELAYTYPING 120px 300px, 20px Field1 1.1s ‘Hello’ 0.9s 100px, 220px Submit Example Log: Typing and submitting “Hello”
  • 25. Features
  • 26. Experimental MTurk Tasks Noun Classification “Identify which words are nouns.” Image Tagging “Generate 3-5 tags for each of 4 images.” Reading Comprehension “Read and then answer the following questions.”
  • 27. Image Tagging Provide at least 3 tags for the following image… ocean mountains pretty x4
  • 28. Image Tagging
  • 29. Image Tagging 52 participants, 114 submissions Human Labeled Cheating 17 cheats, 97 regular Human Labeled Quality AVG 3.5 / 5, SD 1.13
  • 30. Model Features # Unique Letters Total Time Elap. Image Tagging 17 cheats, 97 regular decision tree, 10-fold CV 93.0% accuracy
  • 31. Image Tagging Regressing on ‘quality’ SMOreg, 10-fold CV r=0.5874 (generally ±0.5 on Likert) Model Features # Unique Letters Total Time Elapsed # of Clicks Fields Accessed
  • 32. Generalizing Models Training Reading Comprehension 63 points Testing Noun Identification 185 Points Model predicts for a similar task r=0.4948
  • 33. Other Findings Clustering workers by behavioral patterns De-anonymizing worker submissions
  • 34. Hanging Questions Complex & creative work? Human interpretability Organizer cognitive models
  • 35. CrowdScape
  • 36. ABC SCROLL MOUSE MOVE MOUSE MOVE CLICK CLICKDELAY DELAYTYPING 120px 300px, 20px Field1 1.1s ‘Hello’ 0.9s 100px, 220px Submit
  • 37. Interac(vely  visualizing  user   behavior  and  output
  • 38. Thanks, d3 toolkit
  • 39. Series and series of works starting with “Godzilla" monster movie Godzilla was released in special effects is the name of the fictional monster of their work. The film named “Godzilla" is a special effects monster movie, and Godzilla is the monster. ゴジラはに公開した特撮怪獣映画『ゴジラ』に始まる一連のシリーズ 作品及び、それらの作品に登場する架空の怪獣の名称である。  
  • 40. I like to visit Alappuzha in Kerala, the God's own country. This place is blessed with heavenly lakes. The view of lake shores from the boats is very beautiful. We can enjoy cool wind standing in the boats. We can see a complete picture of a beautiful village by a trip through the lakes of Alappuzha. My favorite place in the world to be is right at home! Ideally my daughter and husband and I just spending time together watching television, sharing a meal, or discussing our day. I know it might sound boring to some but I find joy and security in the familiar surroundings. Sure, there are other places I enjoy but given the choice my home will always be the place I am the happiest.
  • 41. Have you ever wondered what causes swing sets to move with no one on them? Or why bridges twist and tear themselves apart? The answer is aeroelastic tension. But the science behind it is the most interesting part. Watch this and you'll find out how it works in less than two minutes. collapse bridge physics tension aeroelastic Tacoma Narrows Bridge Collapse
  • 42. Limitations Logging is not perfect Information overload Worker traces Parallel coordinates Interpreting behavior Feature set, ML algorithms