With an increasing micro-labor supply and a larger available workforce, new microtask platforms have emerged providing an extensive list of marketplaces where microtasks are offered by requesters and completed by crowd workers. The current microtask crowdsourcing infrastructure does not offer the possibility to be recognised for already accomplished and offered work in different microtask platforms. This lack of information leads to uninformed decisions in selection processes, which have been acknowledged as a promising way to improve the quality of crowd work. To overcome this limitation, we propose Crowd Work CV, an RDF-based data model that, similarly to traditional Curriculum Vitae, captures crowd workers’ interests, qualifications and work history, as well as requesters’ information. Crowd Work CV enables the representation of crowdsourcing agents’ identities and promotes their work experience across the different microtask marketplaces.

1. Crowd Work CV:
Recognition for Micro Work
Cristina Sarasua and Matthias Thimm
SoHuman 2014, Barcelona
Institute for Web Science and Technologies · University of Koblenz-Landau, Germany

2. The Problem
Crowd Work CV: Recognition Cristina Sarasua for Micro Work 2

3. “Better theoretical models, markets or automatic
computational matching processes could
drastically reduce development costs and
address search friction, an important issue in
labor economics“
The Future of Crowd Work (Kittur et al., 2013)
Task assignment Career ladders Transparency
Crowd Work CV: Recognition Cristina Sarasua for Micro Work 3

4. Motivational Scenario
Is Alice a good candidate for this micotask?
Alice´s previous work
Spanish qualification
Same job
Tweet sentiment analysis
Alice
Crowd Work CV: Recognition Cristina Sarasua for Micro Work 4

5. Motivational Scenario
Is Alice a good candidate for this micotask?
Alice´s previous work
Spanish qualification
Same job
Tweet sentiment analysis
Alice
Crowd Work CV: Recognition Cristina Sarasua for Micro Work 5

6. Proposed Approach
Crowd Work CV: Recognition Cristina Sarasua for Micro Work 6

7. Curriculum Vitae in Microtask Crowdsourcing
• RDF data model to describe a microtask
crowdsourcing agent's
• interests
• obtained qualifications
• work history
• It enables the representation of crowdsourcing
agents’ identities and promotes their work
experience across different microtask marketplaces
Crowd Work CV: Recognition Cristina Sarasua for Micro Work 7

8. Data Model
Crowd Work CV: Recognition Cristina Sarasua for Micro Work 8

9. Requirements
1. Domain independence
2. Marketplace independence
3. Semantic and syntactic interoperability
4. Extensibility
5. Compatibility with traditional CV information
Crowd Work CV: Recognition Cristina Sarasua for Micro Work 9

10. Design
Crowd Work CV: Recognition Cristina Sarasua for Micro Work 10

11. Design (II)
Crowd Work CV: Recognition Cristina Sarasua for Micro Work 11

12. Design (III)
Crowd Work CV: Recognition Cristina Sarasua for Micro Work 12

13. Why Semantic Web technologies?
agreed upon vocabulary
integration of Crowd Work
CV data (e.g. with marketplaces,
other applications, Web data sets)
graph of requesters,
microtasks and contributors
extensible data model that
enables knowledge
inference
Picture: Linking Open Data cloud diagram 2014, by
Max Schmachtenberg, Christian Bizer, Anja
Jentzsch and Richard Cyganiak. http://lod-cloud.
net/
Crowd Work CV ontology (OWL):https://github.com/criscod/CrowdWorkCV
Crowd Work CV: Recognition Cristina Sarasua for Micro Work 13

14. Example
ex:rex a cwcv:RequesterExperience;
cwcv:byUserAccount ex:acc1;
cwcv:basedOn ex:mm1;
Data described using the
Crowd Work CV ontology
ex:cv1 cwcv:hasRequesterExperience ex:rex;
e:mm1 a cwcv:MasterMicrotask;
cwcv:publishedAt ex:ClixSense;
cwcv:publishedAt ex:GetPaid;
cwcv:purpose microtaskpurposes:SentimentAnalysis;
ex:mm2 a cwcv:MasterMicrotask;
cwcv:hasQualification ex:q1;
cwcv:publishedAt ex:Neobux;
Crowd Work CV: Recognition Cristina Sarasua for Micro Work 14

15. Example (cont.)
ex:mex1 a cwcv:WorkerExperience;
cwcv:basedOn ex:m1;
cwcv:basedOn ex:mi;
cwcv:byUserAccount ex:acc2;
cwcv:accuracy 0.9;
ex:m1 a cwcv:Microtask;
cwcv:hasMasterMicrotask ex:mm1;
ex:q1 a cwcv:Qualification;
cwcv:name Spanish A1;
cwcv:isRelatedToComptenece disco:Capability1;
ex:cv2 cwcv:hasWorkerExperience ex:mex1;
cwcv:obtainedQualification ex:q1;
Crowd Work CV: Recognition Cristina Sarasua for Micro Work 15

16. Verification
Verified correctness with OOPS! Pitfall Scanner
http://oeg-lia3.dia.fi.upm.es/oops/index-content.jsp
Requirements fulfilment
1. Domain attached separately (e.g. SKOS vocabularies) Domain
independence
2. Abstract concepts (e.g. evaluation) Marketplace independence
3. Ontology defined separately with OWL Semantic and syntactic
interoperability
4. Subclasses, subproperties Extensibility
5. Work experience, educational achievements, related skills and
competences Compatibility with traditional CV information
Crowd Work CV: Recognition Cristina Sarasua for Micro Work 16

17. Differences from Related Work
Khazankin and colleagues, Goel
# matching tasks and workers
and colleagues, Difallah and
colleagues
No shareable and reusable description of worker expertise that
can be used across-platforms
# vocabularies in crowdsourcing
Describes only tasks, users, rewards and capabilities primarily
for routing. We describe work experience, microtasks,
marketplaces, qualifications, requesters’ info; CV summaries
# CV / HCOMP-PROV vocabularies
Lacking crowd work-specific nuances
Ul Hassan and colleagues
Bojars, Celino
Crowd Work CV: Recognition Cristina Sarasua for Micro Work 17

18. Crowd Work CV: FAQs
• Is a Crowd Work CV a user
profile? It is more than that! Work
history plays a key role
• Who should own and share the
Crowd Work CV data? The agent
represented in the CV (e.g.
crowd worker)
• How to avoid spam? With
certified information
Crowd Work CV: Recognition Cristina Sarasua for Micro Work 18

19. Crowd Work CV
Data Management
Crowd Work CV: Recognition Cristina Sarasua for Micro Work 19

20. Overview
Crowd Work CV: Recognition Cristina Sarasua for Micro Work 20

21. Closing
Crowd Work CV: Recognition Cristina Sarasua for Micro Work 21

22. Conclusions
Crowd Work CV is an approach for modelling and
sharing knowledge about crowd work experience,
which enriches the way reputation and credentials are
managed in the current crowd workplace.
Crowd Work CV could facilitate a fruitful requester-crowd
worker interaction in microtask marketplaces and weave
relations of trust.
Crowd Work CV: Recognition Cristina Sarasua for Micro Work 22

23. Future research directions
To what extent do crowd workers (and requesters)
work (and publish work) across different marketplaces?
How to extract summarized information out of many
Crowd Work CV RDF triples?
To what extent does considering cross-platform CV
information while selecting candidates, improve the
accuracy of results?
Crowd Work CV: Recognition Cristina Sarasua for Micro Work 23

24. Possible topics for the discussion
What kind of information would you expect from a Crowd Work CV if you
were a …
… requester?
… crowd worker?
How would you like Crowd Work CVs to be used if you were a …
… requester?
… crowd worker?
Which arguments would you need for adopting the Crowd Work CV
approach in your marketplace, if you were a marketplace owner (e.g.
MTurk)?
What pros and cons do you identify?
Any other comment or question is welcome!
Crowd Work CV: Recognition Cristina Sarasua for Micro Work 24

25. Thank you for your attention!
Cristina Sarasua
Institute for Web Science and Technologies
Universität Koblenz-Landau
csarasua@uni-koblenz.de
Institute for Web Science and Technologies · University of Koblenz-Landau, Germany

26. References
Kittur, A., Nickerson, J.V., Bernstein, M.S., Gerber, E.M., Aaron, S.,
Zimmerman, J., Lease, M., Horton, J.J.: The future of crowd work. In: 16th
ACM Conference on Computer Supported Cooperative Work (CSCW 2013)
(2013)
Difallah, D.E., Demartini, G., Cudré-Mauroux, P.: Pick-a-crowd: tell me what
you like, and I’ll tell you what to do. In: Proceedings of the 22nd international
conference on World Wide Web (WWW2013) (2013)
Gagan Goel, A.K., Singla, A.: Matching workers expertise with tasks:
Incentives in heterogeneous crowdsourcing markets. In: NIPS13 Workshop
on Crowdsourcing: Theory, Algorithms and Applications (2013)
Ul Hassan, U., O’Riain, S., Curry, E.: Slua: Towards semantic linking of users
with actions in crowdsourcing. In: CrowdSem (2013)
Khazankin, R., Psaier, H., Schall, D., Dustdar, S.: QoS-based task
scheduling in crowdsourcing environments. In: Proceedings of the 9th
international conference on Service-Oriented Computing (2011)
Crowd Work CV: Recognition Cristina Sarasua for Micro Work 26

27. References (II)
Bojars, U., Breslin, J. ResumeRDF: Expressing Skill Information on the
Semantic Web . In: Proceedings of the 1st International ExpertFinder
Workshop,Knowledge Web General Assembly 2007 (2007)
Celino, I. Human Computation VGI Provenance: Semantic Web-Based
Representation and Publishing. In: IEEE Transactions on Geoscience and
Remote Sensing. (2013)
Sarasua, C., Thimm, M.: Microtask available, send us your cv! In:
International Workshop on Crowd Work and Human Computation
(CrowdWork2013) (2013)
Crowd Work CV: Recognition Cristina Sarasua for Micro Work 27