Your SlideShare is downloading. ×
  • Like
Social Participation: How Collective Activity Can Make Change
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Now you can save presentations on your phone or tablet

Available for both IPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Social Participation: How Collective Activity Can Make Change

  • 1,115 views
Published

Keynote talk at South African Institute for Computer Scientists and Information Technologists (SAICSIT) Oct 2012

Keynote talk at South African Institute for Computer Scientists and Information Technologists (SAICSIT) Oct 2012

Published in Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
1,115
On SlideShare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
3
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Social Participation: How Collective Activity Can Make Change Benjamin B. Bederson Computer Science Department Human-Computer Interaction Lab Institute for Advanced Computer Studies University of Maryland www.cs.umd.edu/~bederson @bedersonWednesday, October 3, 12
  • 2. Approach Let people collaborate with computers in a way that can be aggregated to provide value to a greater goal => sometimes called Human Computation Blog commenting Q&A Sites Twitter & FacebookWednesday, October 3, 12
  • 3. Approach Let people collaborate with computers in a way that can be aggregated to provide value to a greater goal Translation Photo tagging Things Face recognition Things COMPUTERS Human detection HUMANS can do Speech recognition can do Text analysis Planning Human ComputationWednesday, October 3, 12
  • 4. Human Computation Taxonomy Replace computers with humans Human Computation Replace humans with humans Social Crowdsourcing Computing Collective Intelligence Data MiningWednesday, October 3, 12
  • 5. Trade-off Space Computers Speed, Affordability Human Computation Human Workers (traditional) QualityWednesday, October 3, 12
  • 6. Online Labor MarketsWednesday, October 3, 12
  • 7. Active Area ESP Game - www.gwap.comWednesday, October 3, 12
  • 8. Active Area Fold It - fold.itWednesday, October 3, 12
  • 9. Active Area VizWiz - www.vizwiz.orgWednesday, October 3, 12
  • 10. Active Area Color Name Models - vis.stanford.edu/color-names/Wednesday, October 3, 12
  • 11. A Real-World Problem: ICDL Now: Goal: – 4,386 books 10,000 books – 54 languages 100 languages – Translations in a few languages Every book in every – 100K unique visitors/month language! www.childrenslibrary.orgWednesday, October 3, 12
  • 12. Machine Transla,on Speed / Affordability Amateur0Bilingual0Human0 Par,cipa,on Professional0Bilingual0Human0 Par,cipa,on QualityWednesday, October 3, 12
  • 13. The key ideaWednesday, October 3, 12
  • 14. Translation with the Crowd Translate with the Monolingual Crowd Wikipedia: 900 translators vs. 1,200,000 contributorsWednesday, October 3, 12
  • 15. Machine Translation Speed / Affordability Philip Resnik Linguistics Monolingual Chang Hu, Ph.D. Human => Microsoft Par,cipa,on Amateur Bilingual Human Participation Professional Bilingual Human Participation QualityWednesday, October 3, 12
  • 16. The Protocol Stopping Condition Machine Translation Annotation Projection Source Best Sentence Translation Machine Back-Translation Annotation ProjectionWednesday, October 3, 12
  • 17. La0sonda0...Wednesday, October 3, 12
  • 18. Wednesday, October 3, 12
  • 19. Accuracy Sentences"per"Accuracy"Category" 160" 140" Google" 120" MonoTrans2" #"of"Sentences" 100" 80" 60" 40" 20" 0" 1=None" 2" 3" 4" 5=All" Google: 10% of sentences MonoTrans2: 68% of sentences 24Wednesday, October 3, 12
  • 20. MonoTrans2 UI Complexity Hard to get casual users Too many sentences Too many tasksWednesday, October 3, 12
  • 21. MonoTrans Widgets Too many sentences => One sentence at a time Too many tasks => One task at a timeWednesday, October 3, 12
  • 22. 27Wednesday, October 3, 12
  • 23. 28Wednesday, October 3, 12
  • 24. Julio0say0that0the0 Julio0said0that0the0 mouse0needed0the0 mouse0needed0the0 tooth0for0his0secret0 tooth0for0his0secret0 formula. formula. 29Wednesday, October 3, 12
  • 25. Deployment Aug 2011 – May 2012 155,000 visits 11 books (6 languages) 1,282 sentences 31,000 tasksWednesday, October 3, 12
  • 26. Widgets Evaluation 2 books 739 users Spanish 14 days English 54 sentencesWednesday, October 3, 12
  • 27. Sentences"per"Accuracy"Category" Accuracy 50" Google" 40" Widgets" #"of"Sentences" 30" 20" 10" 0" 1=None" 2" 3" 4" 5=All" Google: 31% of sentences Widgets: 52% of sentencesWednesday, October 3, 12
  • 28. How0could0a0crowd Alex Quinn CS Ph.D. student help0you0make0an important decision?Wednesday, October 3, 12
  • 29. Wednesday, October 3, 12
  • 30. AskSheet Example Problems Pick a grad school Where to buy groceries? Where/when go on vacation? Which papers should be accepted?Wednesday, October 3, 12
  • 31. Decision modeling process Define the Problem Develop a Model Acquire Input Data Develop a Solution Test the SolutionWednesday, October 3, 12
  • 32. Decision modeling process Frugal input acquisition Research focus Acquire Input DataWednesday, October 3, 12
  • 33. Create Model Create a blank decision model with ASK formulas in cells C2:E6. AskSheet highlights priority cells in dark green.Wednesday, October 3, 12
  • 34. Configure Task Set up the task by specifying who will receive the HITs, instructions, and other details. The "Prioritization" slider controls how many inputs to include in each batch.Wednesday, October 3, 12
  • 35. Preview Task At the bottom of the setup screen is a dynamically updated preview of what workers will see. The input form is automatically generated from the spreadsheet model.Wednesday, October 3, 12
  • 36. Data is Collected The system prioritizes the inputs. Obtaining the dark green cells first would provide the greatest opportunity to eliminate other cells. Braces expressions show the possible range of output values for each cell.Wednesday, October 3, 12
  • 37. Task Completed The model is now complete. Store C wins because its maximum possible price is $68, is lower than the minimum possible price of either Store A ($85) or Store B. ($71). The system avoided requesting 11 of the 27 inputs.Wednesday, October 3, 12
  • 38. “ASK” formula A1: =ASK(1, 1, 5) cost, min, max B1: =ASK(1, 1, 10) C1: =IF(A1 < B1, "A", "B")Wednesday, October 3, 12
  • 39. “ASK” formula A1: =ASK(1, 1, 5) B1: =ASK(1, 1, 10) C1: =IF(A1 < B1, "A", "B") Task Prioritization A1 B1 1 2 3 4 5 6 7 8 9 10Wednesday, October 3, 12
  • 40. AskSheet Trial Task: Find pediatrician that is rated well, accepted by insurance, and sufficiently close Answer rcvd in < 9 hrs Eliminated 74% of workWednesday, October 3, 12
  • 41. Open Problems •Evaluation of student work in MOOCs •Aggregation of comments on blogs / class forums •Assuring quality of work (i.e., product reviews)Wednesday, October 3, 12
  • 42. Design Lessons Assuring quality at a viable cost remains hard •Assume adversarial participants. •Best practices: redundancy, gold standards, inform you are watching, treat workers as employees The "motivation" problem remains hard •Don’t forget "whats in it for me". Make it fundamental to primary task EthicsWednesday, October 3, 12
  • 43. Ethics of Human Computation $$$ AnonymityWednesday, October 3, 12
  • 44. Money Problems • Bulk of problems are money/quality related • Workers complain about – Low wages – Not getting paid – Slow payment • Requesters (who are less anonymous) complain about – Low quality work • Also, significant other issues – Decontextualization – Tasks that are illegal or unacceptable – Privacy / anonymityWednesday, October 3, 12
  • 45. Current Situation • Workers and Requesters alike build their own reputation/quality mgmt systems – Turker NationWednesday, October 3, 12
  • 46. Current Situation • Workers and Requesters alike build their own reputation/quality mgmt systems – Turker Nation – TurkopticonWednesday, October 3, 12
  • 47. Current Situation • Workers and Requesters alike build their own reputation/quality mgmt systems – Turker Nation – Turkopticon – CrowdFlowerWednesday, October 3, 12
  • 48. Questions? MonoTrans0supported0by0Na,onal0Science0Founda,on MonoTrans0&0SearchParty0supported0by0Google www.cs.umd.edu/~bederson @bedersonWednesday, October 3, 12