Social Participation:       How Collective Activity Can Make Change       Benjamin B. Bederson       Computer Science Depa...
Approach       Let people collaborate with computers       in a way that can be aggregated to provide value to a greater g...
Approach       Let people collaborate with computers       in a way that can be aggregated to provide value to a greater g...
Human Computation Taxonomy                     Replace computers                       with humans                        ...
Trade-off Space                              Computers       Speed, Affordability                                          ...
Online Labor MarketsWednesday, October 3, 12
Active Area                           ESP Game - www.gwap.comWednesday, October 3, 12
Active Area                           Fold It - fold.itWednesday, October 3, 12
Active Area                           VizWiz - www.vizwiz.orgWednesday, October 3, 12
Active Area     Color Name Models - vis.stanford.edu/color-names/Wednesday, October 3, 12
A Real-World Problem: ICDL      	 Now:                                    	 Goal:            – 4,386 books                ...
Machine                                Transla,on        Speed / Affordability                                            ...
The key ideaWednesday, October 3, 12
Translation with the Crowd      Translate with the Monolingual Crowd            Wikipedia: 900 translators vs. 1,200,000 c...
Machine                                Translation        Speed / Affordability                                           ...
The Protocol                            Stopping Condition                            Machine Translation                 ...
La0sonda0...Wednesday, October 3, 12
Wednesday, October 3, 12
Accuracy                                         Sentences"per"Accuracy"Category"                                 160"    ...
MonoTrans2 UI Complexity                                   Hard to get casual users                                   Too ...
MonoTrans Widgets    Too many sentences     => One sentence at a time    Too many tasks     => One task at a timeWednesday...
27Wednesday, October 3, 12
28Wednesday, October 3, 12
Julio0say0that0the0     Julio0said0that0the0                           mouse0needed0the0       mouse0needed0the0          ...
Deployment                           Aug 2011 – May 2012                           155,000 visits                         ...
Widgets Evaluation                           2 books      739 users                           Spanish      14 days        ...
Sentences"per"Accuracy"Category"                                                      Accuracy                            ...
How0could0a0crowd   Alex Quinn                                               CS Ph.D. student                           he...
Wednesday, October 3, 12
AskSheet Example Problems                 Pick a grad                   school                           Where to buy     ...
Decision modeling process                             Define the Problem                              Develop a Model      ...
Decision modeling process                           Frugal input acquisition  Research focus                   Acquire Inp...
Create Model                           Create a blank decision model with ASK formulas in cells                           ...
Configure Task           Set up the task by specifying who will receive the HITs, instructions, and other           details...
Preview Task         At the bottom of the setup screen is a dynamically updated preview of what workers         will see. ...
Data is Collected                    The system prioritizes the inputs. Obtaining the dark green cells first               ...
Task Completed                  The model is now complete. Store C wins because its maximum possible                  pric...
“ASK” formula                               A1: =ASK(1,     1,     5)                                       cost,   min,  ...
“ASK” formula                                     A1: =ASK(1,          1,       5)                                 B1: =AS...
AskSheet Trial                         Task: Find pediatrician that is rated well,                       accepted by insur...
Open Problems       •Evaluation of student work in MOOCs       •Aggregation of comments on blogs / class forums       •Ass...
Design Lessons          Assuring quality at a viable cost remains hard           •Assume adversarial participants.        ...
Ethics of Human Computation                           $$$   AnonymityWednesday, October 3, 12
Money Problems       • Bulk of problems are money/quality         related       • Workers complain about             – Low...
Current Situation       • Workers and Requesters alike build their own         reputation/quality mgmt systems            ...
Current Situation       • Workers and Requesters alike build their own         reputation/quality mgmt systems            ...
Current Situation       • Workers and Requesters alike build their own         reputation/quality mgmt systems            ...
Questions?       MonoTrans0supported0by0Na,onal0Science0Founda,on       MonoTrans0&0SearchParty0supported0by0Google       ...
Upcoming SlideShare
Loading in …5
×

Social Participation: How Collective Activity Can Make Change

1,406 views

Published on

Keynote talk at South African Institute for Computer Scientists and Information Technologists (SAICSIT) Oct 2012

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,406
On SlideShare
0
From Embeds
0
Number of Embeds
48
Actions
Shares
0
Downloads
5
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Social Participation: How Collective Activity Can Make Change

  1. 1. Social Participation: How Collective Activity Can Make Change Benjamin B. Bederson Computer Science Department Human-Computer Interaction Lab Institute for Advanced Computer Studies University of Maryland www.cs.umd.edu/~bederson @bedersonWednesday, October 3, 12
  2. 2. Approach Let people collaborate with computers in a way that can be aggregated to provide value to a greater goal => sometimes called Human Computation Blog commenting Q&A Sites Twitter & FacebookWednesday, October 3, 12
  3. 3. Approach Let people collaborate with computers in a way that can be aggregated to provide value to a greater goal Translation Photo tagging Things Face recognition Things COMPUTERS Human detection HUMANS can do Speech recognition can do Text analysis Planning Human ComputationWednesday, October 3, 12
  4. 4. Human Computation Taxonomy Replace computers with humans Human Computation Replace humans with humans Social Crowdsourcing Computing Collective Intelligence Data MiningWednesday, October 3, 12
  5. 5. Trade-off Space Computers Speed, Affordability Human Computation Human Workers (traditional) QualityWednesday, October 3, 12
  6. 6. Online Labor MarketsWednesday, October 3, 12
  7. 7. Active Area ESP Game - www.gwap.comWednesday, October 3, 12
  8. 8. Active Area Fold It - fold.itWednesday, October 3, 12
  9. 9. Active Area VizWiz - www.vizwiz.orgWednesday, October 3, 12
  10. 10. Active Area Color Name Models - vis.stanford.edu/color-names/Wednesday, October 3, 12
  11. 11. A Real-World Problem: ICDL Now: Goal: – 4,386 books 10,000 books – 54 languages 100 languages – Translations in a few languages Every book in every – 100K unique visitors/month language! www.childrenslibrary.orgWednesday, October 3, 12
  12. 12. Machine Transla,on Speed / Affordability Amateur0Bilingual0Human0 Par,cipa,on Professional0Bilingual0Human0 Par,cipa,on QualityWednesday, October 3, 12
  13. 13. The key ideaWednesday, October 3, 12
  14. 14. Translation with the Crowd Translate with the Monolingual Crowd Wikipedia: 900 translators vs. 1,200,000 contributorsWednesday, October 3, 12
  15. 15. Machine Translation Speed / Affordability Philip Resnik Linguistics Monolingual Chang Hu, Ph.D. Human => Microsoft Par,cipa,on Amateur Bilingual Human Participation Professional Bilingual Human Participation QualityWednesday, October 3, 12
  16. 16. The Protocol Stopping Condition Machine Translation Annotation Projection Source Best Sentence Translation Machine Back-Translation Annotation ProjectionWednesday, October 3, 12
  17. 17. La0sonda0...Wednesday, October 3, 12
  18. 18. Wednesday, October 3, 12
  19. 19. Accuracy Sentences"per"Accuracy"Category" 160" 140" Google" 120" MonoTrans2" #"of"Sentences" 100" 80" 60" 40" 20" 0" 1=None" 2" 3" 4" 5=All" Google: 10% of sentences MonoTrans2: 68% of sentences 24Wednesday, October 3, 12
  20. 20. MonoTrans2 UI Complexity Hard to get casual users Too many sentences Too many tasksWednesday, October 3, 12
  21. 21. MonoTrans Widgets Too many sentences => One sentence at a time Too many tasks => One task at a timeWednesday, October 3, 12
  22. 22. 27Wednesday, October 3, 12
  23. 23. 28Wednesday, October 3, 12
  24. 24. Julio0say0that0the0 Julio0said0that0the0 mouse0needed0the0 mouse0needed0the0 tooth0for0his0secret0 tooth0for0his0secret0 formula. formula. 29Wednesday, October 3, 12
  25. 25. Deployment Aug 2011 – May 2012 155,000 visits 11 books (6 languages) 1,282 sentences 31,000 tasksWednesday, October 3, 12
  26. 26. Widgets Evaluation 2 books 739 users Spanish 14 days English 54 sentencesWednesday, October 3, 12
  27. 27. Sentences"per"Accuracy"Category" Accuracy 50" Google" 40" Widgets" #"of"Sentences" 30" 20" 10" 0" 1=None" 2" 3" 4" 5=All" Google: 31% of sentences Widgets: 52% of sentencesWednesday, October 3, 12
  28. 28. How0could0a0crowd Alex Quinn CS Ph.D. student help0you0make0an important decision?Wednesday, October 3, 12
  29. 29. Wednesday, October 3, 12
  30. 30. AskSheet Example Problems Pick a grad school Where to buy groceries? Where/when go on vacation? Which papers should be accepted?Wednesday, October 3, 12
  31. 31. Decision modeling process Define the Problem Develop a Model Acquire Input Data Develop a Solution Test the SolutionWednesday, October 3, 12
  32. 32. Decision modeling process Frugal input acquisition Research focus Acquire Input DataWednesday, October 3, 12
  33. 33. Create Model Create a blank decision model with ASK formulas in cells C2:E6. AskSheet highlights priority cells in dark green.Wednesday, October 3, 12
  34. 34. Configure Task Set up the task by specifying who will receive the HITs, instructions, and other details. The "Prioritization" slider controls how many inputs to include in each batch.Wednesday, October 3, 12
  35. 35. Preview Task At the bottom of the setup screen is a dynamically updated preview of what workers will see. The input form is automatically generated from the spreadsheet model.Wednesday, October 3, 12
  36. 36. Data is Collected The system prioritizes the inputs. Obtaining the dark green cells first would provide the greatest opportunity to eliminate other cells. Braces expressions show the possible range of output values for each cell.Wednesday, October 3, 12
  37. 37. Task Completed The model is now complete. Store C wins because its maximum possible price is $68, is lower than the minimum possible price of either Store A ($85) or Store B. ($71). The system avoided requesting 11 of the 27 inputs.Wednesday, October 3, 12
  38. 38. “ASK” formula A1: =ASK(1, 1, 5) cost, min, max B1: =ASK(1, 1, 10) C1: =IF(A1 < B1, "A", "B")Wednesday, October 3, 12
  39. 39. “ASK” formula A1: =ASK(1, 1, 5) B1: =ASK(1, 1, 10) C1: =IF(A1 < B1, "A", "B") Task Prioritization A1 B1 1 2 3 4 5 6 7 8 9 10Wednesday, October 3, 12
  40. 40. AskSheet Trial Task: Find pediatrician that is rated well, accepted by insurance, and sufficiently close Answer rcvd in < 9 hrs Eliminated 74% of workWednesday, October 3, 12
  41. 41. Open Problems •Evaluation of student work in MOOCs •Aggregation of comments on blogs / class forums •Assuring quality of work (i.e., product reviews)Wednesday, October 3, 12
  42. 42. Design Lessons Assuring quality at a viable cost remains hard •Assume adversarial participants. •Best practices: redundancy, gold standards, inform you are watching, treat workers as employees The "motivation" problem remains hard •Don’t forget "whats in it for me". Make it fundamental to primary task EthicsWednesday, October 3, 12
  43. 43. Ethics of Human Computation $$$ AnonymityWednesday, October 3, 12
  44. 44. Money Problems • Bulk of problems are money/quality related • Workers complain about – Low wages – Not getting paid – Slow payment • Requesters (who are less anonymous) complain about – Low quality work • Also, significant other issues – Decontextualization – Tasks that are illegal or unacceptable – Privacy / anonymityWednesday, October 3, 12
  45. 45. Current Situation • Workers and Requesters alike build their own reputation/quality mgmt systems – Turker NationWednesday, October 3, 12
  46. 46. Current Situation • Workers and Requesters alike build their own reputation/quality mgmt systems – Turker Nation – TurkopticonWednesday, October 3, 12
  47. 47. Current Situation • Workers and Requesters alike build their own reputation/quality mgmt systems – Turker Nation – Turkopticon – CrowdFlowerWednesday, October 3, 12
  48. 48. Questions? MonoTrans0supported0by0Na,onal0Science0Founda,on MonoTrans0&0SearchParty0supported0by0Google www.cs.umd.edu/~bederson @bedersonWednesday, October 3, 12

×