Micro-task crowdsourcing has become a successful mean to obtain high-quality data from a large crowd of diverse people. In this context, trust between all the involved actors (i.e. requesters, workers, and platform owners) is a critical factor for acceptance and long-term success. In this talk, I will discuss some problematic aspects of existing micro-task crowdsourcing platforms, where trust is built on fragmented, opaque, and often incomplete knowledge. I will provide examples in which the adoption of open, transparent, and socially-aware trust-building strategies have led to better crowdsourcing performance. I will then conclude with several proposals on how to increase the amount of trust cues available in crowdsourcing platforms, possibly with methods drawn from related disciplines such as user modelling and HCI.

1. TRUSTWORTHY MICRO-TASK CROWDSOURCING:
CHALLENGES AND OPPORTUNITIES
ALESSANDRO BOZZON - DELFT UNIVERSITY OF TECHNOLOGY

2. ABOUT ME

3. http://blog.electricbricks.com/?attachment_id=16621
About Me
Assistant Professor @TU Delft
Web Information Systems, Social Data Science
Faculty Fellow @ IBM Benelux
Inclusive Enterprise
Research Fellow @ AMS
Social Sensing, Smart Citizens
Web Engineering
Web Science
Information Retrieval
User Modelling
Crowdsourcing
Human Computation

4. Create,Interpret
Engage,Retain
Machines
10/04/15 09:55
Page 1 of 1http://uxrepo.com/static/icon-sets/linecons/svg/database.svg
Data
People
How can humans and machines better collaborate in solving
(computational) problems?
Process
DescribeReality
troughPeople
Interact
Fundamental and Experimental Research
Asocio-technicalsystem
How can human-generated Web
data be transformed into a source
that informs Web system design?
How to enhance Web-based
systems with automated, large-
scale human computation?

5. Application Domains
3 use-cases, spanning different contexts and societal/industrial needs
Workforce
Well-being
enterprise crowdsourcing
Knowledge Creation and
Acceleration
online content creation
Intelligent Cities
urban sensing / crowd sensing

6. OUTLINE
▸ What is trust?
▸ Challenges
▸ Opportunities

7. WHAT IS TRUST?

8. WHAT IS TRUST TO
YOU?

9. THE ECONOMICS OF TRUST
Trust = Speed Cost
Trust = Speed Cost

10. TRUST: I KNOW IT WHEN I SEE IT
▸ Trust is a complex social phenomenon
▸ There is no universally accepted scholarly definition
▸ What is meant by “trust” differs from discipline to discipline
▸ Has varying importance at different stages of relationship development
▸ Confidence that [one] will find what is desired [from
another] rather than what is feared.
Morton Deutsch (psychologist)
considered the founder of modem theory and
research on trust

11. TRUST: ACCEPTED DEFINITIONS
▸ “Expectations of benign behaviour from
someone in a socially uncertain situation due
to beliefs about the person’s dispositions
(including his feelings towards you)”
▸ “Psychological state comprising the intention
to accept vulnerability based upon positive
expectations of the intentions or behaviour
of another”
Rousseau, D. M., Sitkin, S. B., Burt, R. S. and Camerer, C. (1998) ‘Not So
Different After All: A Cross-discipline View of Trust’, Academy of
Management Review 23(3): 393–404.
Yamagishi, T.. Trust: The evolutionary game of mind and society. New
York, NY: Springer.

12. TRUSTWORTHINESS
▸ “The character trait of a trustee,
that is, his or her disposition to
act in an altruistic or ethical
manner even when the action is
not backed up by self-interest”
▸ Trustworthiness is a trait of a
trustee, trust is a trait of a truster
Yamagishi, T.. Trust: The evolutionary game of mind and society. New
York, NY: Springer.

13. ▸ A speaker's ethos (being
trustworthy) is based on:
▸ Professional competence,
spirited personal integrity (aretê)
▸ Intelligent good sense, practical
wisdom (phronêsis)
▸ Good will and respect (eúnoiâ)
MODELS OF TRUST /1 - ARISTOTLE’S RHETORIC

14. MODELS OF TRUST /2
▸ Three characteristics determine the perceived trustworthiness:
▸ Ability: skills, competencies, and characteristics that enable a party to
have influence within some specific domain
▸ Benevolence: the extent to which a trustee is believed to want to do good
to the trustor, aside from an egocentric profit motive
▸ Integrity: involves the trustor's perception that the trustee adheres to a set
of principles that the trustor finds acceptable
▸ Each of the three factors can vary along a continuum
An Integrative Model of Organizational Trust
Roger C. Mayer, James H. Davis and F. David
Schoorman
The Academy of Management Review Vol. 20,
No. 3 (Jul., 1995), pp. 709-734
14K Citations
FIGURE 1
Proposed Model of Trust
Factors of
Perceived
Trustworthiness
| | Ability 1 ~~PerceivedRisk
Benevolence Trust Relationship Outcomes
| Propensity |
measure focuses on a generalized trust of others-something akin to a
personality trait that a person would presumably carry from one situation
to another. For example, typical items in his scale are "In dealing with
strangers one is better off to be cautious until they have provided evi-
dence that they are trustworthy" and "Parents usually can be relied upon
to keep their promises."
Several other authors have discussed trust in similar ways. For ex-
ample, Dasgupta's treatment of trust includes generalized expectations of
others; for example, "Can I trust people to come to my rescue if I am about
to drown?" (1988: 53; emphasis added). Similarly, Farris, Senner, and But-
terfield (1973: 145) defined trust as "a personality trait of people interact-
ing with peripheral environment of an organization." In this approach
trust is viewed as a trait that leads to a generalized expectation about the
trustworthiness of others. In the proposed model this trait is referred to as
the propensity to trust.
Propensity to trust is proposed to be a stable within-party factor that
will affect the likelihood the party will trust. People differ in their inherent
propensity to trust. Propensity might be thought of as the general will-
ingness to trust others. Propensity will influence how much trust one has

15. MODELS OF TRUST /2
▸ Risk is inherent in the behavioural manifestation of the
willingness to be vulnerable
▸ To have trust, no need to risk anything; but trusting action implies taking a risk
▸ Trust is not involved in all risk-taking behaviour
▸ Context matters!
▸ Stakes involved
▸ Balance of power in the relationship
▸ The alternatives available to the trustor
An Integrative Model of Organizational Trust
Roger C. Mayer, James H. Davis and F. David
Schoorman
The Academy of Management Review Vol. 20,
No. 3 (Jul., 1995), pp. 709-734
14K Citations
FIGURE 1
Proposed Model of Trust
Factors of
Perceived
Trustworthiness
| | Ability 1 ~~PerceivedRisk
Benevolence Trust Relationship Outcomes
| Propensity |
measure focuses on a generalized trust of others-something akin to a
personality trait that a person would presumably carry from one situation
to another. For example, typical items in his scale are "In dealing with
strangers one is better off to be cautious until they have provided evi-
dence that they are trustworthy" and "Parents usually can be relied upon
to keep their promises."
Several other authors have discussed trust in similar ways. For ex-
ample, Dasgupta's treatment of trust includes generalized expectations of
others; for example, "Can I trust people to come to my rescue if I am about
to drown?" (1988: 53; emphasis added). Similarly, Farris, Senner, and But-
terfield (1973: 145) defined trust as "a personality trait of people interact-
ing with peripheral environment of an organization." In this approach
trust is viewed as a trait that leads to a generalized expectation about the
trustworthiness of others. In the proposed model this trait is referred to as
the propensity to trust.
Propensity to trust is proposed to be a stable within-party factor that
will affect the likelihood the party will trust. People differ in their inherent
propensity to trust. Propensity might be thought of as the general will-
ingness to trust others. Propensity will influence how much trust one has

16. MODELS OF TRUST /3
▸ Trust can be split into three different components
▸ Calculative: based on a rational choice (risk - return calculation)
▸ As in Mayer et. al
▸ Institutional: confidence in regulatory factors (e.g. a legal system
that protects individuals’ rights and property)
▸ which promote the creation of trust
▸ Relational: derived over time from repeated interactions
between trustor and trustee
▸ covers such factors as familiarity and experience with each other
Rousseau, D. M., Sitkin, S. B., Burt, R. S. and Camerer, C. (1998) ‘Not So
Different After All: A Cross-discipline View of Trust’, Academy of
Management Review 23(3): 393–404.
Early Middle
Developmental time
Later

17. TRUST-WARRANTING PROPERTIES
▸ “Trust is produced through complex information processing rather than
by simplification of information”
▸ Only systems that support the exchange of reliable trust cues - and thus
allow for correct trust attribution - will be viable in the long run
▸ The goal is to encourage trustworthy action
▸ And – subsequently – trust
▸ Technology can help
▸ Transmit signals of trust prior to trusting action (e.g. a reputation score)
▸ Be the channel for trusting action (e.g. performed work)
▸ Be used for fulfillments (e.g. compensate work)
Yamagishi, T.. Trust: The evolutionary game of mind and society. New
York, NY: Springer.

18. LACK OF TRUST IN MICRO-
TASK CROWDSOURCING

19. Vs.
5¢
THE COMMON NARRATIVE

20. THIEVES VS. DICTATORS
THE COMMON NARRATIVE
▸ Workers are malicious
▸ Bad work = malicious
workers
▸ Workers are people that
fill in time, to make some
extra money
▸ No need to pay too
much, quality != reward
▸ Spammer requesters
▸ Requesters are unfair
▸ Requesters are forgetful
▸ AMT does nothing, so
we are invisible, and
without power

21. FROM THE REQUESTER’S POINT OF VIEW
trust. Uncertainty, and thus the need for trust, stems from our lack of detailed knowledge
about the other actor’s abilities and motivation (Deutsch, 1958). If we had accurate
insight into their reasoning, trust would not be an issue (Giddens, 1990). We develop our
framework from the sequential interaction between two actors (not always people):
trustor (the trusting actor) and trustee (the trusted actor). Figure 1 shows a model of a
prototypical trust-requiring situation.
Figure 1: The basic interaction between trustor and trustee.
Outside
Option 1 Signals
TRUSTEETRUSTOR
Separation in
Time
Separation in
Space
+ UNCERTAINTY
+ UNCERTAINTY
2a Trusting Action
2b Withdrawal
3a Fulfilment 3b Defection
RISK
Picture adapted from: The Mechanics of Trust: A Framework for Research and Design (2005)
1. REPUTATION / QUALIFICATION
CALCULATIVE TRUST
2A. OFFER PRECIOUS DATA TO WORK WITH
3A. EXECUTE TASK 3B. ABANDON
2B. NOPE, NOT HERE
REQUESTER WORKER
THE TRUST CREATION PROCESS

22. AMT MARKETPLACE IS A HIGH-
NOISE ENVIRONMENT WHERE LOW-
QUALITY WORKERS LIKE SPAMMERS
ARE PREVALENT
Wang, Ipeirotis, Provost
Managing Crowdsourcing Workers.
2011

23. http://turkrequesters.blogspot.nl/2013/01/the-reasons-why-amazon-mechanical-turk.html

24. FROM THE WORKER’S POINT OF VIEW
trust. Uncertainty, and thus the need for trust, stems from our lack of detailed knowledge
about the other actor’s abilities and motivation (Deutsch, 1958). If we had accurate
insight into their reasoning, trust would not be an issue (Giddens, 1990). We develop our
framework from the sequential interaction between two actors (not always people):
trustor (the trusting actor) and trustee (the trusted actor). Figure 1 shows a model of a
prototypical trust-requiring situation.
Figure 1: The basic interaction between trustor and trustee.
Outside
Option 1 Signals
TRUSTEETRUSTOR
Separation in
Time
Separation in
Space
+ UNCERTAINTY
+ UNCERTAINTY
2a Trusting Action
2b Withdrawal
3a Fulfilment 3b Defection
RISK
Picture adapted from: The Mechanics of Trust: A Framework for Research and Design (2005)
1. ????????
2A. PERFORM TASK
3A. PAY / BONUS 3B. DON’T PAY
2B. IGNORE
WORKER REQUESTER
INFORMATION ASYMMETRY / LACK OF
TRANSPARENCY / POWER DIFFERENTIAL
THE TRUST CREATION PROCESS

25. https://www.reddit.com/r/mturk/
comments/1gkb1l/
how_it_feels_sometimes_as_a_turke
r/?ref=share&ref_source=link
https://www.reddit.com/r/mturk/comments/1bni10/
yesterday_when_i_was_trying_to_explain_turking_to/
?ref=share&ref_source=link

26. WE CAN BE REJECTED YET THE
REQUESTERS STILL HAVE OUR
ARTICLES AND SENTENCES. NOT FAIR
A worker
Martin, D., Hanrahan, B.V., O’Neill,
J., and Gupta, N. Being a turker. In
Proc. CSCW 2014. 2014

27. I WOULD LIKE TO SEE THE ABILITY TO RETURN A HIT AS
DEFECTIVE SO IT DINGS THE REQUESTER’S REPUTATION AND
NOT MINE. LET’S FACE IT, IF I’M SUPPOSED TO FIND AN ITEM
FOR SALE ON AMAZON BUT THEY SHOW ME A CHILD’S
CRAYON DRAWING...THERE REALLY NEEDS TO BE A WAY TO
HANDLE THAT WITHOUT IT ALTERING MY NUMBERS
A workerM. Six Silberman, Lilly Irani, and
Joel Ross. 2010. Ethics and tactics
of professional crowdwork. XRDS
17, 2 (December 2010)

28. WHAT ABOUT INSTITUTIONAL TRUST?

29. “I DON’T CARE ABOUT THE PENNY I DIDN’T EARN FOR
KNOWING THE DIFFERENCE BE- TWEEN AN APPLE AND A
GIRAFFE. I’M ANGRY THAT AMT WILL TAKE REQUESTERS’
MONEY BUT NOT MANAGE, OVERSEE, OR MEDIATE THE
PROBLEMS AND INJUSTICES ON THEIR SITE. “
A workerM. Six Silberman, Lilly Irani, and
Joel Ross. 2010. Ethics and tactics
of professional crowdwork. XRDS
17, 2 (December 2010)

30. IS THIS A RELEVANT DISCUSSION? YEAH! - TO SOME, IT’S A JOB
TO OTHERS, A 2ND SOURCE OF INCOME

31. IS THIS A RELEVANT DISCUSSION? YEAH! - YOU PAY TAXES

32. IS THIS A RELEVANT DISCUSSION? YEAH! - GOOD REQUESTERS NEED GOOD WORKERS

33. PLANSOURCING: GENERATING BEHAVIOR CHANGE PLANS WITH FRIENDS AND CROWDS
CSCW2016 - E. AGAPIE, L. COLUSSO, S. A. MUNSON, G. HSIEH
▸ PlanSourcing: Generating Behavior Change Plans with Friends and Crowds
▸ IDEA: Seek help from strangers in online task markets (upwork and Amazon
Mechanical Turk), to suggest personalised behaviour change plan
▸ There is a social costs in asking for help (e.g. worrying about being judged)
▸ Participants primarily expressed concerns with regard to friends
▸ More comfortable sharing information with strangers
▸ More diverse recommendations from strangers than friends
Each worker created one plan for one participant. Planners
were given the participant’s description, goal, and activity
log and asked to create a one week plan to help the person
exercise more, eat healthier, or save more money. The
planners had three days to create the plan. Workers on
Mechanical Turk and oDesk were limited to working at most
two hours on the plan. The planners were provided with a
similar structure as the activity logs (Figure 2) in which they
could create their plan. Other than that, we provided no other
constraints so planners may flexibility structure and present
their plans.
A sample of an activity log and instructions provided to the
planner are available at the following github link:
https://github.com/eagapie/PlanSourcing-Generating-
Behavior-Change-Plans-with-Friends-and-Crowds.
Recruitment and Participants
Through Craigslist and a university mailing list, we recruited
participants interested in increasing their physical activity,
Planners
For these 22 participants, we recruited 66 planners, including
friends and workers on Mechanical Turk and oDesk. 60 of
these planners completed a follow up survey (discussed in
the next sections). Out of those, 19 were male, 40 female,
and one identified as other. There were more females than
males among oDesk planners: two male and 19 females.
ODesk planners were recruited from the Personal Assistants
role on oDesk. We chose this category because it included
some oDesk workers with expertise in exercise, finance, or
nutrition. However, these categories were not clearly
delimited, so we recruited Personal Assistants more broadly.
The friend planners were recruited by the participants, and
had known the participants for an average of 13 years, with
a range of 1 to 35 years. Most friend planners were close to
the participants: family members, spouses, siblings, other
close friends. Only one friend planner was recruited from the
broader network of Facebook friends of the participant. Each
planner reported talking with the participants at least a
couple of times per week. Friends were compensated by
Figure 1. Study Structure
IS THIS A RELEVANT DISCUSSION? YEAH! - GOOD REQUESTERS NEED GOOD WORKERS

34. ISSUES ARE WELL-KNOWN
1. Uncertainty about payment
2. Unaccountable and seemingly arbitrary rejections
3.The apparent prevalence of fraudulent requesters
4. Prohibitive time limits
5. Long pay delays
6. Uncommunicative requesters and administrators
7. Technologically defective HITs
8. HITs with unclear or inadequate instructions
9. Low pay, arbitrariness of bonuses
10. Data privacy
M. Six Silberman, Joel Ross, Lilly Irani, and Bill Tomlinson. 2010.
Sellers' problems in human computation markets. ACM SIGKDD
Workshop on Human Computation (HCOMP '10)
2010!

35. ISSUES ARE WELL-KNOWN / 2
▸ “MTurk is not a game or a social network, it is an unregulated labor
marketplace: a system which deliberately does not pay fair wages,
does not pay due taxes, and provides no protections for workers.“
▸ Much of the challenges posed by this kind of work can be
attributed to the anonymity of all parties, unchecked authority of
the requester to decide payment terms, and the general imbalance
of information
Karën Fort, Gilles Adda, and K. Bretonnel Cohen.
2011. Amazon mechanical turk: Gold mine or coal
mine?. Comput. Linguist. 37, 2 (June 2011), 413-420.
Benjamin B. Bederson and Alexander J. Quinn. 2011.
Web workers unite! addressing challenges of online
laborers. In CHI '11 Extended Abstracts on Human
Factors in Computing Systems (CHI EA '11)
2011!

36. ▸ Workers are not happy with the wage (and treatment) they receive
▸ Information, opportunity, and choice are all rather limited
▸ On AMT choice and opportunity are largely determined by:
▸ experience
▸ ratings
▸ skills and qualifications
▸ information
Being A Turker
David Martin, Benjamin V. Hanrahan, Jacki O’Neill
Xerox Research Centre Europe
6 chemin de Maupertuis, Grenoble France
{david.martin, ben.hanrahan, jacki.oneill}@xrce.xerox.com
Neha Gupta
University of Nottingham
University Park NG7 2TD Nottingham
neha.gupta@xrce.xerox.com
ABSTRACT
We conducted an ethnomethodological analysis of publicly
available content on Turker Nation, a general forum for
Amazon Mechanical Turk (AMT) users. Using forum data
we provide novel depth and detail on how the Turker Nation
members operate as economic actors, working out which
Requesters and jobs are worthwhile to them. We show some
of the key ways Turker Nation functions as a community and
also look further into Turker-Requester relationships from
the Turker perspective – considering practical, emotional and
moral aspects. Finally, following Star and Strauss [25] we
analyse Turking as a form of invisible work. We do this to
illustrate practical and ethical issues relating to working with
Turkers and AMT, and to promote design directions to
support Turkers and their relationships with Requesters.
Author Keywords
Ethnomethodology; content analysis; crowdsourcing;
microtasking; Amazon Mechanical Turk; Turker Nation.
ACM Classification Keywords
believe that this will be beneficial for researchers and
businesses working within the crowdsourcing space.
Crowdsourcing encompasses multiple types of activity:
invention, project work, creative activities, and microtasking.
This latter is our focus here. The most well-known microtask
platform is Amazon Mechanical Turk (AMT)2
, and the
Turker Nation forum that we studied is dedicated to users of
this platform. The basic philosophy of microtasking and
AMT is to delegate tasks that are difficult for computers to
do to a human workforce. This has been termed ‘artificial
artificial intelligence’. Tasks like image tagging, duplicate
recognition, translation, transcription, object classification,
and content generation are common. ‘Requesters’ (the AMT
term for people who have work to be completed) post
multiple, similar jobs as Human Intelligence Tasks (HITs),
which can then be taken up by registered ‘Turkers’. Turkers
(termed ‘Providers’ by AMT) are the users completing the
HITs, which typically take seconds or minutes paid at a few
cents at a time.
CSCW 2014 • Performing Crowd Work February 15-19, 2014, Baltimore, MD, USA

37. WORKERS, UNITE!
▸ Workers organised in communities, outside AMT
▸ TurkerNation, mTurkForum, etc
▸ Workers maintain and repair AMT, to help it work as intended
▸ They help resolve breakdowns
▸ They advise employers about flawed task designs or bugs
▸ They teach each other how to use tools
▸ They also help each other
▸ Suggesting good HITS
▸ Instructing newcomers
▸ Discussing requesters
Web workers unite! addressing challenges of online laborers
BB Bederson, AJ Quinn
CHI'11 Extended Abstracts on Human Factors in Computing
Systems, 97-106

38. CHI 2015: N. SALEHI, L.C. IRANI, M,S. BERNSTEIN, A. ALKHATIB, E. OGBE, K. MILLAND
WE ARE DYNAMO: OVERCOMING STALLING AND FRICTION IN COLLECTIVE ACTION FOR CROWD WORKERS
▸ A platform to support the Mechanical Turk community in
forming publics around issues and then mobilizing
▸ GOALS: reduce stalling and friction

39. THE PROBLEM WITH ONLINE COMMUNITIES
▸ Efforts that require sustained effort and critical mass are less likely to succeed
▸ Divided loyalties
▸ Time pressures to earn money
▸ Risks that agitation poses to their reputations
▸ disputes over members who were suspected of operating multiple accounts
▸ making a statement taken as insult
▸ The forums archive these interactions and reconciliation can be difficult online
▸ Members of online-only communities may struggle to achieve trust
Dahlberg, L. Computer mediated communication and the public sphere: A critical
analysis. Journal of Computer Mediated Communication, 2001.
Niloufar Salehi, Lilly C. Irani, Michael S. Bernstein, Ali Alkhatib,
Eva Ogbe, Kristy Milland, and Clickhappier. 2015. We Are Dynamo:
Overcoming Stalling and Friction in Collective Action for Crowd
Workers. In Proceedings of the 33rd Annual ACM Conference on
Human Factors in Computing Systems (CHI '15)

40. A SUCCESS STORY: TURKOPTICON
▸ Measuring benevolence and integrity of requesters
▸ communicativity: How responsive has this requester been to communications or
concerns you have raised?
▸ generosity: How well has this requester paid for the amount of time their HITs take?
▸ fairness: How fair has this requester been in approving or rejecting your work?
▸ promptness: How promptly has this requester approved your work and paid?
Lilly Irani and M.SixSilberman.2013. Turkopticon: Interrupting
Worker Invisibility on Amazon Mechanical Turk. In Proceedings of
the SIGCHI Conference on Human Factors in Computing Systems
(CHI 2013), 611-620.

41. IMPROVING THE EXCHANGE
OF TRUST CUES

42. FROM VISION PAPER
Publish
Execute
Participate
Assess, discuss
Requester Worker community
Worker
Platform
Task
Task modeling Worker modeling Interaction cycle Security & privacy

43. THE ROLE OF SOFTWARE
▸ Workers rely on number of Turking tools
▸ A list of all requesters
▸ Script to record worker history
▸ Client-side scripts to hide HITs posted by particular requesters
▸ Scripts to monitor the market status
▸ Shouldn’t these tool be made “official”?
▸ To cope with lack of institutional trust?

44. MINIMISE REQUESTERS’ ERRORS
▸ (semi) automated task analysis, to check and assess tasks before
submission
▸ To check for clarity of instruction
▸ To assess complexity
▸ reward according
▸ establish fair completion time
▸ To improve verifiability and transparency — Kittur et al. (2008)
▸ By making the submission easy to verify, quality
management will be facilitated

45. MEASURING CROWDSOURCING EFFORT WITH ERROR-TIME CURVES
▸ Crowdsourcing systems lack effective measures of the effort required to
complete each task
▸ Objective measures could help task selection, and reward estimation
▸ Error-time area model the effort required for a worker to accurately complete
a task
1.Calculated by recruiting workers to complete the task under different time
limits
2.Fit curve
3.Calculate ETA (area under the curve)
CHI 2015: JUSTIN CHENG, JAIME TEEVAN, AND MICHAEL S. BERNSTEIN
rcing CHI 2015, Crossings, Seoul, Korea

46. UIST ’15: STANFORD CROWD RESEARCH COLLECTIVE
DAEMO: A SELF-GOVERNED CROWDSOURCING MARKETPLACE
▸ A self-governed crowdsourcing marketplace
▸ open-governance model to achieve equitable representation
▸ Prototype task to improve the work quality
▸ milestones used to build a common ground and adjust the task description.
▸ It also facilitates discussing cost and time to do a job
erned Crowdsourcing Marketplace
ord Crowd Research Collective ⇤
Stanford HCI Group
daemo@cs.stanford.edu
ities for au-
s well as so-
ces, workers
and low pay-
ults they re-
on of power
ous concerns
address the

47. ASIST 2011: JÖRN KLINGER, MATTHEW LEASE
ENABLING TRUST IN CROWD LABOR RELATIONS THROUGH IDENTITY SHARING
▸ Link Crowdsourcing IDs to
social network profiles
▸ No disclosing of sensitive data
▸ Reduced the anonymity in the
Crowdmarket
▸ Accountability for malicious
work
Figure 1. Authorizing the application.
After authorizing with the application, we see the main
menu. Since no ID linking has taken place so far, the
options available are to link the Facebook profile to a
worker or a requester ID (it is possible to link a single
profile to both a worker and a requester ID).
Figure 2. Linking the Facebook profile to the worker ID.
The next step in this process is to click on "Link your
Facebook profile to a worker ID." On the next screen, we
see the information that the application reads from our
Facebook profile. The current prototype accesses gender,
birthday, current location, hometown and languages
spoken. To give an idea of upcoming features, the prototype
allows one to specify their skills in these languages.
Analogously, workers see a list of jobs offered to them
based on their skills and background.
Figure 7. The Worker HQ allows workers to access and
manage the jobs offered to them.
For each job, workers can see the specified task link,
qualification code, description, and a link to the requester’s
(corporate) Facebook profile, giving workers the

48. ▸ IDEA: Increasing social transparency to enhance
accountability
▸ Sharing of demographic information
▸ Peer-dependent reward schemes (team and
competition - pairs )
▸ Social transparency lead to better results when
information shared with their colleagues
▸ No difference in peer-dependent reward schemes
▸ Increasing social transparency = concern for privacy
issues
CHI 2013: SHIH-WEN HUANG AND WAI-TAT FU
DON’T HIDE IN THE CROWD! INCREASING SOCIAL TRANSPARENCY
BETWEEN PEER WORKERS IMPROVES CROWDSOURCING OUTCOMES
Don’t Hide in the Crowd! Increasing Social Transparency
Between Peer Workers Improves Crowdsourcing Outcomes
Shih-Wen Huang and Wai-Tat Fu
University of Illinois at Urbana-Champaign
Department of Computer Science
Urbana, IL 61801
{shuang51,wfu}@illinois.edu
ABSTRACT
This paper studied how social transparency and different
peer-dependent reward schemes (i.e., individual, teamwork,
and competition) affect the outcomes of crowdsourcing. The
results showed that when social transparency was increased
by asking otherwise anonymous workers to share their de-
mographic information (e.g., name, nationality) to the paired
worker, they performed significantly better. A more detailed
analysis showed that in a teamwork reward scheme, in which
the reward of the paired workers depended only on the col-
lective outcomes, increasing social transparency could off-
set effects of social loafing by making them more account-
able to their teammates. In a competition reward scheme, in
which workers competed against each other and the reward
depended on how much they outperformed their opponent,
increasing social transparency could augment effects of so-
cial facilitation by providing more incentives for them to out-
perform their opponent. The results suggested that a careful
combination of methods that increase social transparency and
different reward schemes can significantly improve crowd-
sourcing outcomes.
Author Keywords
Crowdsourcing; human computation; social transparency;
social facilitation; social loafing
ACM Classification Keywords
H.5.m. Information Interfaces and Presentation (e.g. HCI):
Miscellaneous
INTRODUCTION
Crowdsourcing has been proven as an effective way to solve
various kinds of problems [8]. One of the most notable ex-
amples is the ESP game [26], which recruits people to gen-
erate image labels while playing an online game. By 2008,
this game had recruited 200,000 players and collected more
Figure 1. Sharing demographic information between paired workers
allows the crowdsourcing system to collect outcomes with higher quality.
easily recruit online workers from Amazon Mechanical Turk
(AMT)1
to solve problems that are difficult for digital com-
puters (e.g., collecting labeled data for natural language pro-
cessing [23] and relevance evaluation [1]) at a very low cost.
These examples have merely begun to demonstrate the po-
tential of crowdsourcing as a social computing technique that
can be applied in a wide range of situations.
However, quality control is still one of the biggest issues for
crowdsourcing2
[11]. For example, although the ESP game
can successfully recruit players to create a large amount of
image labels, a study [22] showed that many of these are low-
quality labels that can be generated by robots with little real
world knowledge. Moreover, the reliability of workers re-
cruited from AMT is also questionable. People have found
that there are many spammers or even robots in AMT, which
greatly reduce the quality of the outputs collected from this
platform [8]. As a result, designing mechanisms that can en-
sure the quality of crowdsourcing outcomes is crucial to its
success.

49. CSCW 2013: P. KINNAIRD, L, DABBISH, S. KIESLER, H. FASTE
CO-WORKER TRANSPARENCY IN A MICROTASK MARKETPLACE
▸ IDEA: show an image
displaying whether or not
there are coworkers
▸ Co-worker information can
significantly influence worker
motivation and work quality.
▸ When 1 out of many, reduced
feelings of task significance
and work importance.
shown an image displaying whether or not they had co-
workers (see Figure 1).
The effect of co-worker information on worker motivation
must be carefully tested. On the one hand, if workers learn
they have some co-workers, they might become more
motivated because they feel like part of a social entity.
Common identity engenders loyalty and social feelings
among members [10,12]. A worker learning he has co-
workers might feel good that he has compatriots, and is
sharing their fate. A recent study in which MTurk workers
were shown where they stood in a workflow suggests this
is a possibility [14]. On the other hand, information about
co-workers could backfire if workers receiving this
information feel their work is redundant and they are a
small part of a large, impersonal process.
How many co-workers?
group of bystanders, suggesting diffusion of responsibility
in larger crowds [19]. Larger numbers of co-workers could
also suggest to workers that they are redundant, meaning
their work is less valuable to the final product. Studies
show that perceived redundancy reduces worker effort and
motivation [7,8]
TWO EXPERIMENTS ON CO-WORKER VISIBILITY
We conducted two between-subjects experiments on co-
worker visibility in Mechanical Turk. We drew on the
literature above to pose the following hypotheses:
H1: Workers informed they have a small number of
coworkers will perform better quality work than those not
informed about the existence of co-workers.
H2: Workers informed they have a small number of
coworkers will perform better quality work than those
informed they have more coworkers.
Figure 1. Co-worker information in MTurk Experiment 1.

50. PICK-A-CROWD: TELL ME WHAT YOU LIKE, AND I'LL TELL YOU WHAT TO DO
▸ Workers profile (ability?) are built from social networking profiles
▸ Workers are assigned tasks based on their interests
▸ Assumption: workers will perform these tasks given their existing interests
▸ Average a 29% relative improvement over the best accuracy obtained by the AMT model5.2. System Architecture
WWW ’13: D.E. DIFALLAH, G. DEMARTINI, P. CUDRÉ-MAUROUX.

51. BEYOND SIMPLE ABILITY MEASURES
▸ Trust clues are currently built on a “result-centred”
interpretation of reliability
▸ Modelling abilities (expertise, skills) can help, but now not
fully exploited
▸ What about benevolence, integrity, and intent of workers?

52. BEYOND MONETARY REWARDS
▸ Non-cash rewards are known to trigger emotional
response
https://www.reddit.com/r/mturk/
comments/4dogxk/
been_turking_for_2_weeks_recently_h
it_1000
https://www.reddit.com/r/
mturk/comments/2sy6vc/
woke_up_checked_my_email_a
nd_holy_f***
The relative relativity of material and experiential purchases.
Carter, Travis J.; Gilovich, Thomas
Journal of Personality and Social Psychology, Vol 98(1), Jan 2010, 146-159.
WHY COULDN’T AMT DO THE SAME?

53. THE ROLE OF COMMUNITIES
▸ On-line worker communities are now “isolated” from
crowdsourcing platforms
▸ Integrate knowledge from communities inside the
platform
▸ Beyond requester assessment
▸ To express, in more details, relational trust

54. THE BEST TIME TO PLANT A
SEED IS 20 YEARS AGO. THE
SECOND BEST TIME IS TODAY
CHINESE PROVERB

55. Deadline approaching (June 5, 2016)
Keynotes:
Maarten de Rijke
(Naturally Intelligent Search)
Philippe Cudre-Mauroux
(Entity-Centric Data Management)
Special issue!