Community-based Crowdsourcing

  • 1,667 views
Uploaded on

Slides presented at the 2nd International Workshop on the Theory and Practice of Social Machines (at WWW2014 conference in Seoul)

Slides presented at the 2nd International Workshop on the Theory and Practice of Social Machines (at WWW2014 conference in Seoul)

More in: Software , Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • Really interesting Andrea, thanks​!​

    I think that you would be really interested in some of the most cutting-edge research that I have come across explaining crowds, open innovation, and citizen science.​

    http://papers.ssrn.com/sol3/cf_dev/AbsByAuth.cfm?per_id=1919614

    And you may also enjoy this blog about the same too:
    https://thecrowdsociety.jux.com/

    Powerful stuff, no?
    Are you sure you want to
    Your message goes here
    Be the first to like this
No Downloads

Views

Total Views
1,667
On Slideshare
0
From Embeds
0
Number of Embeds
10

Actions

Shares
Downloads
6
Comments
1
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Dipartimento di Elettronica, Informazione e Bioingegneria COMMUNITY-BASED CROWDSOURCING Marco Brambilla, Stefano Ceri, Andrea Mauri, Riccardo Volonterio
  • 2. Dipartimento di Elettronica, Informazione e Bioingegneria Crowd-based Applications  Emerging crowd-based applications: – opinion mining – localized information gathering – marketing campaigns – expert response gathering  General structure: – the requestor poses some questions – a wide set of responders are in charge of providing answers (typically unknown to the requestor) – the system organizes a response collection campaign SOCM’14, Monday, April 7 Community-basedCrowdsourcing 3
  • 3. Dipartimento di Elettronica, Informazione e Bioingegneria Our approach  Combines a conceptual framework, a specification paradigm and a reactive execution control environment  Supports designing, deploying, and monitoring applications on top of crowd-based systems – Design is top-down, platform-independent – Deployment turns declarative specifications into platform- specific implementations which include social networks and crowdsourcing platforms – Monitoring provides reactive control, which guarantees applications’ adaptation and interoperability SOCM’14, Monday, April 7 Community-basedCrowdsourcing 4
  • 4. Dipartimento di Elettronica, Informazione e Bioingegneria Control  Controlling crowdsourcing tasks is a fundamental issue – Cost – Time – Quality  In our approach – Implemented trough active rules • Control is easily expressible  Simple control data structures  Familiar formalism • Support to arbitrarily complex controls  Extensibility mechanisms • Active rules can be system-generated  Well-defined semantic SOCM’14, Monday, April 7 Community-basedCrowdsourcing 5
  • 5. Dipartimento di Elettronica, Informazione e Bioingegneria Community Set of people that share  Interests  Feature ..or belong to  common entity  social network SOCM’14, Monday, April 7 Community-basedCrowdsourcing 6
  • 6. Dipartimento di Elettronica, Informazione e Bioingegneria Leveraging communities  Why? – Experts – More engaged  How? – Determine the communities of performers – Monitor them taking into the account the behavior of their member • Community Control SOCM’14, Monday, April 7 Community-basedCrowdsourcing 7
  • 7. Dipartimento di Elettronica, Informazione e Bioingegneria Community Control Community control allow the adaptation of the crowdsourcing campaign  Task / Object allocation  Static / Dynamic Implemented with the reactive environment present in our approach SOCM’14, Monday, April 7 Community-basedCrowdsourcing 8
  • 8. Dipartimento di Elettronica, Informazione e Bioingegneria Example (dynamic control) µTObjExecution Performer TaskImage StatusStartTsEndTs µTaskID Object Control Performer Control Task Control C om pO bjs TaskIDCom pExecs Name PerformerID Score ProfPhoto M aterials NonRelevant PeoplePlace TaskID ObjectID Correct ImgUrl Category ProfessorID ObjectID Answer Eval ObjectID PerformerID TaskID PerformerID CommunityCommunityID Name Community Control CommunityID Score CommunityID Enabled Status Status LastExec Tests Execs SOCM’14, Monday, April 7 Community-basedCrowdsourcing 10 e: AFTER UPDATE FOR μTObjExecution c: CommunityControl[CommunityID== NEW.CommunityID].score<=0.5 CommunityControl[CommunityID== NEW.CommunityID].eval=10 a: SET CommunityControl[CommunityID == DB-Group].Enabled = true ?
  • 9. Dipartimento di Elettronica, Informazione e Bioingegneria Crowdsearcher A prototype that allows the definition, execution and control of a crowdsourcing campaign according to our approach SOCM’14, Monday, April 7 Community-basedCrowdsourcing 11 http://crowdsearcher.search-computing.org/
  • 10. Dipartimento di Elettronica, Informazione e Bioingegneria Experiment  16 professors within two research groups in our department (DB and AI groups)  The top 50 images returned by the Google Image API for each query  Each microtask consisted of evaluating 5 images regarding a professor.  Results are accepted (and thus the corresponding object is closed) when enough agreement on the class of the image is reached  Closed objects are removed from new executions. SOCM’14, Monday, April 7 Community-basedCrowdsourcing 12
  • 11. Dipartimento di Elettronica, Informazione e Bioingegneria Communities The communities:  the research group of the professor,  the research area containing the group (e.g. Computer Science)  and the whole department (which accounts for more than 600 people in different areas) Invitations are sent:  inside-out: we started with invitations to experts, e.g. people the same groups as the professor (DB and AI), and then expanded invitations to Computer Science, then to the whole Department, and finally to open social networks (Alumni and PhDs communities on Facebook and Linkedin);  outside-in: we proceeded in the opposite way, starting with the Department members, then restricting to Computer Scientists, and finally to the group's members. SOCM’14, Monday, April 7 Community-basedCrowdsourcing 13
  • 12. Dipartimento di Elettronica, Informazione e Bioingegneria Number of performers per community SOCM’14, Monday, April 7 Community-basedCrowdsourcing 14 0" 10" 20" 30" 40" 50" 60" 70" 7/18/2013" 7/19/2013" 7/20/2013" 7/21/2013" 7/22/2013" 7/23/2013" 7/24/2013" 7/25/2013" 7/26/2013" 7/27/2013" 7/28/2013" #"Performers" Time" research"group" research"area" department" social"network" total" 46% 24% 16% 9 / “a lot”
  • 13. Dipartimento di Elettronica, Informazione e Bioingegneria Precision of performers per community SOCM’14, Monday, April 7 Community-basedCrowdsourcing 15 0" 0.1" 0.2" 0.3" 0.4" 0.5" 0.6" 0.7" 0.8" 0.9" 1" 0" 500" 1000" 1500" 2000" 2500" 3000" Precision) #Evalua0ons) research"group" research"area" department" social"network" total"
  • 14. Dipartimento di Elettronica, Informazione e Bioingegneria Precision of the evaluated objects  Precision decreases with less expert communities  Inside-out strategy (from expert to generic users) outperforms Outside-in strategy (from generic to expert users) SOCM’14, Monday, April 7 Community-basedCrowdsourcing 16 0.6$ 0.65$ 0.7$ 0.75$ 0.8$ 0.85$ 0.9$ 0.95$ 1$ 0$ 100$ 200$ 300$ 400$ 500$ 600$ 700$ 800$ Precision) #Closed)Objects) precision$(main$experiment)$ precision$(reverse$invita<ons)$
  • 15. Dipartimento di Elettronica, Informazione e Bioingegneria General observations Experts from community feel more engaged with the task  They are more demanding with respect to the quality of the application UI and the evaluated objects  Provide feedbacks on the application, question and the objects evaluated – “How is it possible that this image is related to me?!” SOCM’14, Monday, April 7 Community-basedCrowdsourcing 17
  • 16. Dipartimento di Elettronica, Informazione e Bioingegneria Thanks for you attention Any question? SOCM’14, Monday, April 7 Community-basedCrowdsourcing 19