Community-based Crowdsourcing

2,376 views

Published on

Slides presented at the 2nd International Workshop on the Theory and Practice of Social Machines (at WWW2014 conference in Seoul)

Published in: Software, Technology
1 Comment
0 Likes
Statistics
Notes
  • Really interesting Andrea, thanks​!​

    I think that you would be really interested in some of the most cutting-edge research that I have come across explaining crowds, open innovation, and citizen science.​

    http://papers.ssrn.com/sol3/cf_dev/AbsByAuth.cfm?per_id=1919614

    And you may also enjoy this blog about the same too:
    https://thecrowdsociety.jux.com/

    Powerful stuff, no?
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

No Downloads
Views
Total views
2,376
On SlideShare
0
From Embeds
0
Number of Embeds
1,630
Actions
Shares
0
Downloads
9
Comments
1
Likes
0
Embeds 0
No embeds

No notes for slide

Community-based Crowdsourcing

  1. 1. Dipartimento di Elettronica, Informazione e Bioingegneria COMMUNITY-BASED CROWDSOURCING Marco Brambilla, Stefano Ceri, Andrea Mauri, Riccardo Volonterio
  2. 2. Dipartimento di Elettronica, Informazione e Bioingegneria Crowd-based Applications  Emerging crowd-based applications: – opinion mining – localized information gathering – marketing campaigns – expert response gathering  General structure: – the requestor poses some questions – a wide set of responders are in charge of providing answers (typically unknown to the requestor) – the system organizes a response collection campaign SOCM’14, Monday, April 7 Community-basedCrowdsourcing 3
  3. 3. Dipartimento di Elettronica, Informazione e Bioingegneria Our approach  Combines a conceptual framework, a specification paradigm and a reactive execution control environment  Supports designing, deploying, and monitoring applications on top of crowd-based systems – Design is top-down, platform-independent – Deployment turns declarative specifications into platform- specific implementations which include social networks and crowdsourcing platforms – Monitoring provides reactive control, which guarantees applications’ adaptation and interoperability SOCM’14, Monday, April 7 Community-basedCrowdsourcing 4
  4. 4. Dipartimento di Elettronica, Informazione e Bioingegneria Control  Controlling crowdsourcing tasks is a fundamental issue – Cost – Time – Quality  In our approach – Implemented trough active rules • Control is easily expressible  Simple control data structures  Familiar formalism • Support to arbitrarily complex controls  Extensibility mechanisms • Active rules can be system-generated  Well-defined semantic SOCM’14, Monday, April 7 Community-basedCrowdsourcing 5
  5. 5. Dipartimento di Elettronica, Informazione e Bioingegneria Community Set of people that share  Interests  Feature ..or belong to  common entity  social network SOCM’14, Monday, April 7 Community-basedCrowdsourcing 6
  6. 6. Dipartimento di Elettronica, Informazione e Bioingegneria Leveraging communities  Why? – Experts – More engaged  How? – Determine the communities of performers – Monitor them taking into the account the behavior of their member • Community Control SOCM’14, Monday, April 7 Community-basedCrowdsourcing 7
  7. 7. Dipartimento di Elettronica, Informazione e Bioingegneria Community Control Community control allow the adaptation of the crowdsourcing campaign  Task / Object allocation  Static / Dynamic Implemented with the reactive environment present in our approach SOCM’14, Monday, April 7 Community-basedCrowdsourcing 8
  8. 8. Dipartimento di Elettronica, Informazione e Bioingegneria Example (dynamic control) µTObjExecution Performer TaskImage StatusStartTsEndTs µTaskID Object Control Performer Control Task Control C om pO bjs TaskIDCom pExecs Name PerformerID Score ProfPhoto M aterials NonRelevant PeoplePlace TaskID ObjectID Correct ImgUrl Category ProfessorID ObjectID Answer Eval ObjectID PerformerID TaskID PerformerID CommunityCommunityID Name Community Control CommunityID Score CommunityID Enabled Status Status LastExec Tests Execs SOCM’14, Monday, April 7 Community-basedCrowdsourcing 10 e: AFTER UPDATE FOR μTObjExecution c: CommunityControl[CommunityID== NEW.CommunityID].score<=0.5 CommunityControl[CommunityID== NEW.CommunityID].eval=10 a: SET CommunityControl[CommunityID == DB-Group].Enabled = true ?
  9. 9. Dipartimento di Elettronica, Informazione e Bioingegneria Crowdsearcher A prototype that allows the definition, execution and control of a crowdsourcing campaign according to our approach SOCM’14, Monday, April 7 Community-basedCrowdsourcing 11 http://crowdsearcher.search-computing.org/
  10. 10. Dipartimento di Elettronica, Informazione e Bioingegneria Experiment  16 professors within two research groups in our department (DB and AI groups)  The top 50 images returned by the Google Image API for each query  Each microtask consisted of evaluating 5 images regarding a professor.  Results are accepted (and thus the corresponding object is closed) when enough agreement on the class of the image is reached  Closed objects are removed from new executions. SOCM’14, Monday, April 7 Community-basedCrowdsourcing 12
  11. 11. Dipartimento di Elettronica, Informazione e Bioingegneria Communities The communities:  the research group of the professor,  the research area containing the group (e.g. Computer Science)  and the whole department (which accounts for more than 600 people in different areas) Invitations are sent:  inside-out: we started with invitations to experts, e.g. people the same groups as the professor (DB and AI), and then expanded invitations to Computer Science, then to the whole Department, and finally to open social networks (Alumni and PhDs communities on Facebook and Linkedin);  outside-in: we proceeded in the opposite way, starting with the Department members, then restricting to Computer Scientists, and finally to the group's members. SOCM’14, Monday, April 7 Community-basedCrowdsourcing 13
  12. 12. Dipartimento di Elettronica, Informazione e Bioingegneria Number of performers per community SOCM’14, Monday, April 7 Community-basedCrowdsourcing 14 0" 10" 20" 30" 40" 50" 60" 70" 7/18/2013" 7/19/2013" 7/20/2013" 7/21/2013" 7/22/2013" 7/23/2013" 7/24/2013" 7/25/2013" 7/26/2013" 7/27/2013" 7/28/2013" #"Performers" Time" research"group" research"area" department" social"network" total" 46% 24% 16% 9 / “a lot”
  13. 13. Dipartimento di Elettronica, Informazione e Bioingegneria Precision of performers per community SOCM’14, Monday, April 7 Community-basedCrowdsourcing 15 0" 0.1" 0.2" 0.3" 0.4" 0.5" 0.6" 0.7" 0.8" 0.9" 1" 0" 500" 1000" 1500" 2000" 2500" 3000" Precision) #Evalua0ons) research"group" research"area" department" social"network" total"
  14. 14. Dipartimento di Elettronica, Informazione e Bioingegneria Precision of the evaluated objects  Precision decreases with less expert communities  Inside-out strategy (from expert to generic users) outperforms Outside-in strategy (from generic to expert users) SOCM’14, Monday, April 7 Community-basedCrowdsourcing 16 0.6$ 0.65$ 0.7$ 0.75$ 0.8$ 0.85$ 0.9$ 0.95$ 1$ 0$ 100$ 200$ 300$ 400$ 500$ 600$ 700$ 800$ Precision) #Closed)Objects) precision$(main$experiment)$ precision$(reverse$invita<ons)$
  15. 15. Dipartimento di Elettronica, Informazione e Bioingegneria General observations Experts from community feel more engaged with the task  They are more demanding with respect to the quality of the application UI and the evaluated objects  Provide feedbacks on the application, question and the objects evaluated – “How is it possible that this image is related to me?!” SOCM’14, Monday, April 7 Community-basedCrowdsourcing 17
  16. 16. Dipartimento di Elettronica, Informazione e Bioingegneria Thanks for you attention Any question? SOCM’14, Monday, April 7 Community-basedCrowdsourcing 19

×