Community-based Crowdsourcing

Dipartimento di
Elettronica, Informazione e
Bioingegneria
COMMUNITY-BASED
CROWDSOURCING
Marco Brambilla, Stefano Ceri, Andrea Mauri, Riccardo Volonterio

Dipartimento di Elettronica,
Informazione e Bioingegneria
Crowd-based Applications
 Emerging crowd-based applications:
– opinion mining
– localized information gathering
– marketing campaigns
– expert response gathering
 General structure:
– the requestor poses some questions
– a wide set of responders are in charge of providing answers
(typically unknown to the requestor)
– the system organizes a response collection campaign
SOCM’14, Monday, April 7
Community-basedCrowdsourcing
3

Our approach
 Combines a conceptual framework, a specification
paradigm and a reactive execution control environment
 Supports designing, deploying, and monitoring
applications on top of crowd-based systems
– Design is top-down, platform-independent
– Deployment turns declarative specifications into platform-
specific implementations which include social networks and
crowdsourcing platforms
– Monitoring provides reactive control, which guarantees
applications’ adaptation and interoperability
4

Control
 Controlling crowdsourcing tasks is a fundamental issue
– Cost
– Time
– Quality
 In our approach
– Implemented trough active rules
• Control is easily expressible
 Simple control data structures
 Familiar formalism
• Support to arbitrarily complex controls
 Extensibility mechanisms
• Active rules can be system-generated
 Well-defined semantic
5

Community
Set of people that share
 Interests
 Feature
..or belong to
 common entity
 social network
6

Leveraging communities
 Why?
– Experts
– More engaged
 How?
– Determine the communities of performers
– Monitor them taking into the account the behavior of their
member
• Community Control
7

Community Control
Community control allow the adaptation of the
crowdsourcing campaign
 Task / Object allocation
 Static / Dynamic
Implemented with the reactive environment present in our
approach
8

Example (dynamic control)
µTObjExecution
Performer
TaskImage
StatusStartTsEndTs
µTaskID
Object
Control
Performer
Control
Task
Control
C
om
pO
bjs
TaskIDCom
pExecs
Name
PerformerID
Score
ProfPhoto
M
aterials
NonRelevant
PeoplePlace
TaskID
ObjectID
Correct
ImgUrl
Category
ProfessorID
ObjectID
Answer
Eval
ObjectID
PerformerID
TaskID
PerformerID
CommunityCommunityID
Name
Community
Control
CommunityID
Score
CommunityID
Enabled
Status
Status
LastExec
Tests
Execs
10
e: AFTER UPDATE FOR μTObjExecution
c: CommunityControl[CommunityID== NEW.CommunityID].score<=0.5
CommunityControl[CommunityID== NEW.CommunityID].eval=10
a: SET CommunityControl[CommunityID == DB-Group].Enabled = true
?

Crowdsearcher
A prototype that allows
the definition,
execution and control
of a crowdsourcing
campaign according to
our approach
11
http://crowdsearcher.search-computing.org/

Experiment
 16 professors within two
research groups in our
department (DB and AI
groups)
 The top 50 images returned
by the Google Image API for
each query
 Each microtask consisted of
evaluating 5 images
regarding a professor.
 Results are accepted (and
thus the corresponding
object is closed) when
enough agreement on the
class of the image is reached
 Closed objects are removed
from new executions.
12

Communities
The communities:
 the research group of the professor,
 the research area containing the group (e.g. Computer Science)
 and the whole department (which accounts for more than 600
people in different areas)
Invitations are sent:
 inside-out: we started with invitations to experts, e.g. people the
same groups as the professor (DB and AI), and then expanded
invitations to Computer Science, then to the whole Department, and
finally to open social networks (Alumni and PhDs communities on
Facebook and Linkedin);
 outside-in: we proceeded in the opposite way, starting with the
Department members, then restricting to Computer Scientists, and
finally to the group's members.
13

Number of performers per community
14
0"
10"
20"
30"
40"
50"
60"
70"
7/18/2013" 7/19/2013" 7/20/2013" 7/21/2013" 7/22/2013" 7/23/2013" 7/24/2013" 7/25/2013" 7/26/2013" 7/27/2013" 7/28/2013"
#"Performers"
Time"
research"group"
research"area"
department"
social"network"
total"
46%
24%
16%
9 / “a lot”

Precision of performers per community
15
0"
0.1"
0.2"
0.3"
0.4"
0.5"
0.6"
0.7"
0.8"
0.9"
1"
0" 500" 1000" 1500" 2000" 2500" 3000"
Precision)
#Evalua0ons)
research"group"
research"area"
department"
social"network"
total"

Precision of the evaluated objects
 Precision decreases with less expert communities
 Inside-out strategy (from expert to generic users)
outperforms Outside-in strategy (from generic to expert
users)
16
0.6$
0.65$
0.7$
0.75$
0.8$
0.85$
0.9$
0.95$
1$
0$ 100$ 200$ 300$ 400$ 500$ 600$ 700$ 800$
Precision)
#Closed)Objects)
precision$(main$experiment)$
precision$(reverse$invita<ons)$

General observations
Experts from community feel more engaged with the task
 They are more demanding with respect to the quality of
the application UI and the evaluated objects
 Provide feedbacks on the application, question and the
objects evaluated
– “How is it possible that this image is related to me?!”
17

Thanks for you attention
Any question?
19

Community-based Crowdsourcing

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Community-based Crowdsourcing

Similar to Community-based Crowdsourcing (20)

More from Andrea Mauri

More from Andrea Mauri (6)

Recently uploaded

Recently uploaded (20)

Community-based Crowdsourcing