1.
Dipartimento di
Elettronica, Informazione e
Bioingegneria
COMMUNITY-BASED
CROWDSOURCING
Marco Brambilla, Stefano Ceri, Andrea Mauri, Riccardo Volonterio

2.
Dipartimento di Elettronica,
Informazione e Bioingegneria
Crowd-based Applications
 Emerging crowd-based applications:
– opinion mining
– localized information gathering
– marketing campaigns
– expert response gathering
 General structure:
– the requestor poses some questions
– a wide set of responders are in charge of providing answers
(typically unknown to the requestor)
– the system organizes a response collection campaign
SOCM’14, Monday, April 7
Community-basedCrowdsourcing
3

3.
Dipartimento di Elettronica,
Informazione e Bioingegneria
Our approach
 Combines a conceptual framework, a specification
paradigm and a reactive execution control environment
 Supports designing, deploying, and monitoring
applications on top of crowd-based systems
– Design is top-down, platform-independent
– Deployment turns declarative specifications into platform-
specific implementations which include social networks and
crowdsourcing platforms
– Monitoring provides reactive control, which guarantees
applications’ adaptation and interoperability
SOCM’14, Monday, April 7
Community-basedCrowdsourcing
4

4.
Dipartimento di Elettronica,
Informazione e Bioingegneria
Control
 Controlling crowdsourcing tasks is a fundamental issue
– Cost
– Time
– Quality
 In our approach
– Implemented trough active rules
• Control is easily expressible
 Simple control data structures
 Familiar formalism
• Support to arbitrarily complex controls
 Extensibility mechanisms
• Active rules can be system-generated
 Well-defined semantic
SOCM’14, Monday, April 7
Community-basedCrowdsourcing
5

5.
Dipartimento di Elettronica,
Informazione e Bioingegneria
Community
Set of people that share
 Interests
 Feature
..or belong to
 common entity
 social network
SOCM’14, Monday, April 7
Community-basedCrowdsourcing
6

6.
Dipartimento di Elettronica,
Informazione e Bioingegneria
Leveraging communities
 Why?
– Experts
– More engaged
 How?
– Determine the communities of performers
– Monitor them taking into the account the behavior of their
member
• Community Control
SOCM’14, Monday, April 7
Community-basedCrowdsourcing
7

7.
Dipartimento di Elettronica,
Informazione e Bioingegneria
Community Control
Community control allow the adaptation of the
crowdsourcing campaign
 Task / Object allocation
 Static / Dynamic
Implemented with the reactive environment present in our
approach
SOCM’14, Monday, April 7
Community-basedCrowdsourcing
8

8.
Dipartimento di Elettronica,
Informazione e Bioingegneria
Example (dynamic control)
µTObjExecution
Performer
TaskImage
StatusStartTsEndTs
µTaskID
Object
Control
Performer
Control
Task
Control
C
om
pO
bjs
TaskIDCom
pExecs
Name
PerformerID
Score
ProfPhoto
M
aterials
NonRelevant
PeoplePlace
TaskID
ObjectID
Correct
ImgUrl
Category
ProfessorID
ObjectID
Answer
Eval
ObjectID
PerformerID
TaskID
PerformerID
CommunityCommunityID
Name
Community
Control
CommunityID
Score
CommunityID
Enabled
Status
Status
LastExec
Tests
Execs
SOCM’14, Monday, April 7
Community-basedCrowdsourcing
10
e: AFTER UPDATE FOR μTObjExecution
c: CommunityControl[CommunityID== NEW.CommunityID].score<=0.5
CommunityControl[CommunityID== NEW.CommunityID].eval=10
a: SET CommunityControl[CommunityID == DB-Group].Enabled = true
?

9.
Dipartimento di Elettronica,
Informazione e Bioingegneria
Crowdsearcher
A prototype that allows
the definition,
execution and control
of a crowdsourcing
campaign according to
our approach
SOCM’14, Monday, April 7
Community-basedCrowdsourcing
11
http://crowdsearcher.search-computing.org/

10.
Dipartimento di Elettronica,
Informazione e Bioingegneria
Experiment
 16 professors within two
research groups in our
department (DB and AI
groups)
 The top 50 images returned
by the Google Image API for
each query
 Each microtask consisted of
evaluating 5 images
regarding a professor.
 Results are accepted (and
thus the corresponding
object is closed) when
enough agreement on the
class of the image is reached
 Closed objects are removed
from new executions.
SOCM’14, Monday, April 7
Community-basedCrowdsourcing
12

11.
Dipartimento di Elettronica,
Informazione e Bioingegneria
Communities
The communities:
 the research group of the professor,
 the research area containing the group (e.g. Computer Science)
 and the whole department (which accounts for more than 600
people in different areas)
Invitations are sent:
 inside-out: we started with invitations to experts, e.g. people the
same groups as the professor (DB and AI), and then expanded
invitations to Computer Science, then to the whole Department, and
finally to open social networks (Alumni and PhDs communities on
Facebook and Linkedin);
 outside-in: we proceeded in the opposite way, starting with the
Department members, then restricting to Computer Scientists, and
finally to the group's members.
SOCM’14, Monday, April 7
Community-basedCrowdsourcing
13

12.
Dipartimento di Elettronica,
Informazione e Bioingegneria
Number of performers per community
SOCM’14, Monday, April 7
Community-basedCrowdsourcing
14
0"
10"
20"
30"
40"
50"
60"
70"
7/18/2013" 7/19/2013" 7/20/2013" 7/21/2013" 7/22/2013" 7/23/2013" 7/24/2013" 7/25/2013" 7/26/2013" 7/27/2013" 7/28/2013"
#"Performers"
Time"
research"group"
research"area"
department"
social"network"
total"
46%
24%
16%
9 / “a lot”

13.
Dipartimento di Elettronica,
Informazione e Bioingegneria
Precision of performers per community
SOCM’14, Monday, April 7
Community-basedCrowdsourcing
15
0"
0.1"
0.2"
0.3"
0.4"
0.5"
0.6"
0.7"
0.8"
0.9"
1"
0" 500" 1000" 1500" 2000" 2500" 3000"
Precision)
#Evalua0ons)
research"group"
research"area"
department"
social"network"
total"

14.
Dipartimento di Elettronica,
Informazione e Bioingegneria
Precision of the evaluated objects
 Precision decreases with less expert communities
 Inside-out strategy (from expert to generic users)
outperforms Outside-in strategy (from generic to expert
users)
SOCM’14, Monday, April 7
Community-basedCrowdsourcing
16
0.6$
0.65$
0.7$
0.75$
0.8$
0.85$
0.9$
0.95$
1$
0$ 100$ 200$ 300$ 400$ 500$ 600$ 700$ 800$
Precision)
#Closed)Objects)
precision$(main$experiment)$
precision$(reverse$invita<ons)$

15.
Dipartimento di Elettronica,
Informazione e Bioingegneria
General observations
Experts from community feel more engaged with the task
 They are more demanding with respect to the quality of
the application UI and the evaluated objects
 Provide feedbacks on the application, question and the
objects evaluated
– “How is it possible that this image is related to me?!”
SOCM’14, Monday, April 7
Community-basedCrowdsourcing
17

16.
Dipartimento di Elettronica,
Informazione e Bioingegneria
Thanks for you attention
Any question?
SOCM’14, Monday, April 7
Community-basedCrowdsourcing
19