SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.
SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.
Successfully reported this slideshow.
Activate your 14 day free trial to unlock unlimited reading.
1.
Dipartimento di
Elettronica, Informazione e
Bioingegneria
COMMUNITY-BASED
CROWDSOURCING
Marco Brambilla, Stefano Ceri, Andrea Mauri, Riccardo Volonterio
2.
Dipartimento di Elettronica,
Informazione e Bioingegneria
Crowd-based Applications
Emerging crowd-based applications:
– opinion mining
– localized information gathering
– marketing campaigns
– expert response gathering
General structure:
– the requestor poses some questions
– a wide set of responders are in charge of providing answers
(typically unknown to the requestor)
– the system organizes a response collection campaign
SOCM’14, Monday, April 7
Community-basedCrowdsourcing
3
3.
Dipartimento di Elettronica,
Informazione e Bioingegneria
Our approach
Combines a conceptual framework, a specification
paradigm and a reactive execution control environment
Supports designing, deploying, and monitoring
applications on top of crowd-based systems
– Design is top-down, platform-independent
– Deployment turns declarative specifications into platform-
specific implementations which include social networks and
crowdsourcing platforms
– Monitoring provides reactive control, which guarantees
applications’ adaptation and interoperability
SOCM’14, Monday, April 7
Community-basedCrowdsourcing
4
4.
Dipartimento di Elettronica,
Informazione e Bioingegneria
Control
Controlling crowdsourcing tasks is a fundamental issue
– Cost
– Time
– Quality
In our approach
– Implemented trough active rules
• Control is easily expressible
Simple control data structures
Familiar formalism
• Support to arbitrarily complex controls
Extensibility mechanisms
• Active rules can be system-generated
Well-defined semantic
SOCM’14, Monday, April 7
Community-basedCrowdsourcing
5
5.
Dipartimento di Elettronica,
Informazione e Bioingegneria
Community
Set of people that share
Interests
Feature
..or belong to
common entity
social network
SOCM’14, Monday, April 7
Community-basedCrowdsourcing
6
6.
Dipartimento di Elettronica,
Informazione e Bioingegneria
Leveraging communities
Why?
– Experts
– More engaged
How?
– Determine the communities of performers
– Monitor them taking into the account the behavior of their
member
• Community Control
SOCM’14, Monday, April 7
Community-basedCrowdsourcing
7
7.
Dipartimento di Elettronica,
Informazione e Bioingegneria
Community Control
Community control allow the adaptation of the
crowdsourcing campaign
Task / Object allocation
Static / Dynamic
Implemented with the reactive environment present in our
approach
SOCM’14, Monday, April 7
Community-basedCrowdsourcing
8
8.
Dipartimento di Elettronica,
Informazione e Bioingegneria
Example (dynamic control)
µTObjExecution
Performer
TaskImage
StatusStartTsEndTs
µTaskID
Object
Control
Performer
Control
Task
Control
C
om
pO
bjs
TaskIDCom
pExecs
Name
PerformerID
Score
ProfPhoto
M
aterials
NonRelevant
PeoplePlace
TaskID
ObjectID
Correct
ImgUrl
Category
ProfessorID
ObjectID
Answer
Eval
ObjectID
PerformerID
TaskID
PerformerID
CommunityCommunityID
Name
Community
Control
CommunityID
Score
CommunityID
Enabled
Status
Status
LastExec
Tests
Execs
SOCM’14, Monday, April 7
Community-basedCrowdsourcing
10
e: AFTER UPDATE FOR μTObjExecution
c: CommunityControl[CommunityID== NEW.CommunityID].score<=0.5
CommunityControl[CommunityID== NEW.CommunityID].eval=10
a: SET CommunityControl[CommunityID == DB-Group].Enabled = true
?
9.
Dipartimento di Elettronica,
Informazione e Bioingegneria
Crowdsearcher
A prototype that allows
the definition,
execution and control
of a crowdsourcing
campaign according to
our approach
SOCM’14, Monday, April 7
Community-basedCrowdsourcing
11
http://crowdsearcher.search-computing.org/
10.
Dipartimento di Elettronica,
Informazione e Bioingegneria
Experiment
16 professors within two
research groups in our
department (DB and AI
groups)
The top 50 images returned
by the Google Image API for
each query
Each microtask consisted of
evaluating 5 images
regarding a professor.
Results are accepted (and
thus the corresponding
object is closed) when
enough agreement on the
class of the image is reached
Closed objects are removed
from new executions.
SOCM’14, Monday, April 7
Community-basedCrowdsourcing
12
11.
Dipartimento di Elettronica,
Informazione e Bioingegneria
Communities
The communities:
the research group of the professor,
the research area containing the group (e.g. Computer Science)
and the whole department (which accounts for more than 600
people in different areas)
Invitations are sent:
inside-out: we started with invitations to experts, e.g. people the
same groups as the professor (DB and AI), and then expanded
invitations to Computer Science, then to the whole Department, and
finally to open social networks (Alumni and PhDs communities on
Facebook and Linkedin);
outside-in: we proceeded in the opposite way, starting with the
Department members, then restricting to Computer Scientists, and
finally to the group's members.
SOCM’14, Monday, April 7
Community-basedCrowdsourcing
13
12.
Dipartimento di Elettronica,
Informazione e Bioingegneria
Number of performers per community
SOCM’14, Monday, April 7
Community-basedCrowdsourcing
14
0"
10"
20"
30"
40"
50"
60"
70"
7/18/2013" 7/19/2013" 7/20/2013" 7/21/2013" 7/22/2013" 7/23/2013" 7/24/2013" 7/25/2013" 7/26/2013" 7/27/2013" 7/28/2013"
#"Performers"
Time"
research"group"
research"area"
department"
social"network"
total"
46%
24%
16%
9 / “a lot”
13.
Dipartimento di Elettronica,
Informazione e Bioingegneria
Precision of performers per community
SOCM’14, Monday, April 7
Community-basedCrowdsourcing
15
0"
0.1"
0.2"
0.3"
0.4"
0.5"
0.6"
0.7"
0.8"
0.9"
1"
0" 500" 1000" 1500" 2000" 2500" 3000"
Precision)
#Evalua0ons)
research"group"
research"area"
department"
social"network"
total"
14.
Dipartimento di Elettronica,
Informazione e Bioingegneria
Precision of the evaluated objects
Precision decreases with less expert communities
Inside-out strategy (from expert to generic users)
outperforms Outside-in strategy (from generic to expert
users)
SOCM’14, Monday, April 7
Community-basedCrowdsourcing
16
0.6$
0.65$
0.7$
0.75$
0.8$
0.85$
0.9$
0.95$
1$
0$ 100$ 200$ 300$ 400$ 500$ 600$ 700$ 800$
Precision)
#Closed)Objects)
precision$(main$experiment)$
precision$(reverse$invita<ons)$
15.
Dipartimento di Elettronica,
Informazione e Bioingegneria
General observations
Experts from community feel more engaged with the task
They are more demanding with respect to the quality of
the application UI and the evaluated objects
Provide feedbacks on the application, question and the
objects evaluated
– “How is it possible that this image is related to me?!”
SOCM’14, Monday, April 7
Community-basedCrowdsourcing
17
16.
Dipartimento di Elettronica,
Informazione e Bioingegneria
Thanks for you attention
Any question?
SOCM’14, Monday, April 7
Community-basedCrowdsourcing
19