Instrumenting the Crowd + Task Fingerprinting Overview


Published on

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Instrumenting the Crowd + Task Fingerprinting Overview

  1. 1. Jeff Rzeszotarski [rez-oh-tar’-ski] 4th Year Ph.D. Student HCII, CMU Instrumenting the Crowd
  2. 2. James Cridland @ Flickr
  3. 3. James Cridland @ Flickr
  4. 4. James Cridland @ Flickr Amazon Mechanical Turk oDesk CrowdFlower
  5. 5. Bernstein et al. 2010
  6. 6. Task Design Gold Standard Worker Agreement
  7. 7. Task Design Cheating Resistant Design Kittur, Chi, Suh, 2008 Von Ahn & Dabbish, 2004
  8. 8. Gold Standards Gold Standard Worker 1 Worker 2 A A A D B D B C A A 2 4 2 2 8 Callison-Burch, 2009 Downs et al. 2010
  9. 9. Worker Agreement Callison-Burch, 2009 Ipeirotis et al. 2010 Dekel & Shamir, 2009 Snow et al. 2008 WorkerID Video Transcript AF34DB9 Coloring comics in Photoshop is easy. First you… QR40Q2 <blank> TK421S9S Coloring comcs in photoshop is easy first u… WOPR21 QOJEORJOFJDSKGNDKSNGSDSJGKDSJGLKSDJGL APC3C55 Coloring comics in Photoshop is easy. First you… IDKFAD1 Coloring comics in photoshop is easy. First you…
  10. 10. Disadvantages Task Design - Task-dependent - Time consuming iteration Gold Standard - Not always available - Not always applicable Worker Agreement - Limits response range - Subject to majority effects - Not always applicable
  11. 11. Worker A Worker B
  12. 12. Worker A
  13. 13. Worker A
  14. 14. Worker A
  15. 15. Worker A
  16. 16. Worker A
  17. 17. Worker B
  18. 18. Worker B
  19. 19. Worker B
  20. 20. Worker B
  21. 21. Worker B
  22. 22. Worker B
  23. 23. Worker B
  24. 24. Logging ABC SCROLL MOUSE MOVE MOUSE MOVE CLICK CLICKDELAY DELAYTYPING 120px 300px, 20px Field1 1.1s ‘Hello’ 0.9s 100px, 220px Submit Example Log: Typing and submitting “Hello”
  25. 25. Features
  26. 26. Experimental MTurk Tasks Noun Classification “Identify which words are nouns.” Image Tagging “Generate 3-5 tags for each of 4 images.” Reading Comprehension “Read and then answer the following questions.”
  27. 27. Image Tagging Provide at least 3 tags for the following image… ocean mountains pretty x4
  28. 28. Image Tagging
  29. 29. Image Tagging 52 participants, 114 submissions Human Labeled Cheating 17 cheats, 97 regular Human Labeled Quality AVG 3.5 / 5, SD 1.13
  30. 30. Model Features # Unique Letters Total Time Elap. Image Tagging 17 cheats, 97 regular decision tree, 10-fold CV 93.0% accuracy
  31. 31. Image Tagging Regressing on ‘quality’ SMOreg, 10-fold CV r=0.5874 (generally ±0.5 on Likert) Model Features # Unique Letters Total Time Elapsed # of Clicks Fields Accessed
  32. 32. Generalizing Models Training Reading Comprehension 63 points Testing Noun Identification 185 Points Model predicts for a similar task r=0.4948
  33. 33. Other Findings Clustering workers by behavioral patterns De-anonymizing worker submissions
  34. 34. Hanging Questions Complex & creative work? Human interpretability Organizer cognitive models
  35. 35. CrowdScape
  36. 36. ABC SCROLL MOUSE MOVE MOUSE MOVE CLICK CLICKDELAY DELAYTYPING 120px 300px, 20px Field1 1.1s ‘Hello’ 0.9s 100px, 220px Submit
  37. 37. Interac(vely  visualizing  user   behavior  and  output
  38. 38. Thanks, d3 toolkit
  39. 39. Series and series of works starting with “Godzilla" monster movie Godzilla was released in special effects is the name of the fictional monster of their work. The film named “Godzilla" is a special effects monster movie, and Godzilla is the monster. ゴジラはに公開した特撮怪獣映画『ゴジラ』に始まる一連のシリーズ 作品及び、それらの作品に登場する架空の怪獣の名称である。  
  40. 40. I like to visit Alappuzha in Kerala, the God's own country. This place is blessed with heavenly lakes. The view of lake shores from the boats is very beautiful. We can enjoy cool wind standing in the boats. We can see a complete picture of a beautiful village by a trip through the lakes of Alappuzha. My favorite place in the world to be is right at home! Ideally my daughter and husband and I just spending time together watching television, sharing a meal, or discussing our day. I know it might sound boring to some but I find joy and security in the familiar surroundings. Sure, there are other places I enjoy but given the choice my home will always be the place I am the happiest.
  41. 41. Have you ever wondered what causes swing sets to move with no one on them? Or why bridges twist and tear themselves apart? The answer is aeroelastic tension. But the science behind it is the most interesting part. Watch this and you'll find out how it works in less than two minutes. collapse bridge physics tension aeroelastic Tacoma Narrows Bridge Collapse
  42. 42. Limitations Logging is not perfect Information overload Worker traces Parallel coordinates Interpreting behavior Feature set, ML algorithms