Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Human Computation

66 views

Published on

Seminar about Human Computation and Games with a Purpose in the context of the Data Semantics course (Data Science Master course) at the University of Milano Bicocca

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Human Computation

  1. 1. HUMAN COMPUTATION Irene Celino – irene.celino@cefriel.com Cefriel, Viale Sarca 226, 20126 Milano Seminar @ Data Semantics course – April 11th, 2018
  2. 2. 1. Introduction 2. Linked Data and Knowledge Graph Refinement 3. Human Computation and Games with a Purpose 4. Examples of GWAP for Data Linking 5. Truth Inference and Open Science 6. Guidelines 7. Indirect People Involvement 2copyright © 2018 Cefriel – All rights reserved
  3. 3. from ideation to business value 3 1. INTRODUCTION Is the Web a pure technological artefact? What role can people play on the Web? copyright © 2018 Cefriel – All rights reserved
  4. 4. WEB AS A SOCIAL ARTEFACT “The Web isn’t about what you can do with computers. It’s people and, yes, they are connected by computers. But computer science, as the study of what happens in a computer, doesn’t tell you about what happens on the Web” – sir Tim Berners-Lee 4copyright © 2018 Cefriel – All rights reserved
  5. 5. Open Source Software “Given enough eyeballs, all bugs are shallow.” Eric S. Raymond (The Cathedral and the Bazaar) OPEN EVERYTHING Open Content “It is easy when you skip the intermediaries” original motto of Creative Commons (EN video) (IT video) Open Data 5copyright © 2018 Cefriel – All rights reserved “Raw. Data. Now.” Tim Berners-Lee (The year open data went worldwide – TED Talk)
  6. 6. COOPERATION ON THE WEB TO PRODUCE OPEN KNOWLEDGE 6copyright © 2018 Cefriel – All rights reserved
  7. 7. WISDOM OF CROWDS • “Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations” • Criteria for a wise crowd • Diversity of opinion (importance of interpretation) • Independence (not a “single mind”) • Decentralization (importance of local knowledge) • Aggregation (aim to get a collective decision) • The are also failures/risks in crowd decisions: • Homogeneity, centralization, division, imitation, emotionality 7copyright © 2018 Cefriel – All rights reserved James Surowiecki The wisdom of crowds Anchor, 2005
  8. 8. from ideation to business value 8 2. LINKED DATA & KNOWLEDGE GRAPH REFINEMENT Do we need to involve people in Semantic Web systems? What semantic data management tasks can we effectively “outsource” to humans? copyright © 2018 Cefriel – All rights reserved
  9. 9. HUMANS IN THE SEMANTIC WEB • Knowledge-intensive and/or context-specific character of Semantic Web tasks: • e.g., conceptual modelling, multi-language resource labelling, content annotation with ontologies, concept/entity similarity recognition, … • Need to engage users and involve them in executing tasks: • e.g., wikis for semantic content authoring, folksonomies to bootstrap formal ontologies, instance creation by data entry, … 9copyright © 2018 Cefriel – All rights reserved
  10. 10. SEMANTIC WEB TASKS (ALSO) FOR HUMANS 10copyright © 2018 Cefriel – All rights reserved Fact level Schema level Collection Creation CorrectionValidation Filtering Ranking Linking Conceptual modelling Ontology population Quality assessment Ontology re- engineering Ontology pruning Ontology elicitation Knowledge acquisition Ontology repair Knowledge base update Data search/ selection Link generation Ontology alignment Ontology matching
  11. 11. AUTOMATIC METHODS IN THE SEMANTIC WEB? • Knowledge Graph Refinement (and, in general, linked dataset refinement) is an emerging and hot topic to (1) identify and correct errors and (2) add missing knowledge • e.g., completing type assertions via classification, predicting relations from textual sources, finding erroneous type assertions, identifying erroneous literal values through anomaly/outlier detection, … • Statistical and machine learning approaches require some partial gold standard, i.e. a “ground truth” dataset to train automatic models • Ground truth is usually put together manually by expert • Sourcing gold standard from humans is expensive! 11copyright © 2018 Cefriel – All rights reserved Heiko Paulheim. Knowledge graph refinement: A survey of approaches and evaluation methods. Semantic Web Journal, 2017
  12. 12. DATA LINKING • Creation of links in the form of RDF triples (subject, predicate, object) • Within the same dataset (i.e. generating new connections between resources of the same dataset or knowledge graph) • Across different datasets (i.e. creating RDF links, as named in the Linked Data world) • Note: • In literature, data linking often means finding equivalent resources (similarly to record linkage in database research), i.e. triples with correspondence/match predicate (e.g. owl:sameAs)  in the following, data linking is intended in its broader meaning (i.e. links with any predicate) 12copyright © 2018 Cefriel – All rights reserved
  13. 13. DATA LINKING: SOME DEFINITIONS • Resources R is the set of all resources (and literals), whenever possible also described by the respective types. More specifically: R = Rs ∪ Ro, where Rs is the set of resources that can take the role of subject in a triple and Ro is the set of resources that can take the role of object in a triple; as said above the two sets are not necessarily disjoint, i.e. it can happen that Rs ∩ Ro ≠ ∅. • Predicates P is the set of all predicates, whenever possible also described by the respective domain and range. • Links L is the set of all links; since links are triples created between resources and predicates it is: L ⊂ Rs × P × Ro; each link is defined as l = (rs,p,ro) ∈ L with rs ∈ Rs, p ∈ P, ro ∈ Ro. L is usually smaller than the full Cartesian product of Rs, P, Ro, because in each link (rs,p,ro) it must be true that rs ∈ domain(p) and ro ∈ range(p). • Link scores σ is the score of a link, i.e. a value indicating the confidence on the truth value of the link; usually σ ∈ [0,1]; each link l ∈ L can have an associated score. 13copyright © 2018 Cefriel – All rights reserved
  14. 14. CASES OF DATA LINKING • Link creation: a link l is created: given R = Rs ∪ Ro and P, the link l = (rs,p,ro), with rs ∈ Rs, p ∈ P, ro ∈ Ro is created and added to L • e.g., music classification: assign one or more music styles to audio tracks by creating the link (track,genre,style) • Link ranking: given the set of links L, a score σ ∈ [0,1] is assigned to each link l. The score represents the probability of the link to be recognized as true. Links can be ordered on the basis of their score σ, thus obtaining a ranking • e.g., ranking photos depicting a specific person (an actor, a singer, a politician) to identify the pictures in which the person is more recognizable or more clearly depicted • Link validation: given the set of links L, a score σ ∈ [0,1] is assigned to each link l. The score represents the actual truth value of the link. A threshold t ∈ [0,1] is set so that all links with score σ ≥ t are considered true • e.g., assessing the correct music style identification in audio tracks (music classification) 14copyright © 2018 Cefriel – All rights reserved
  15. 15. from ideation to business value 15 3. HUMAN COMPUTATION & GAMES WITH A PURPOSE What goals can humans help machines to achieve? How to involve a crowd of persons? What extrinsic rewards (money, prizes, etc.) or intrinsic incentives can we adopt to motivate people? copyright © 2018 Cefriel – All rights reserved
  16. 16. HUMAN COMPUTATION • Human Computation is a computer science technique in which a computational process is performed by outsourcing certain steps to humans. Unlike traditional computation, in which a human delegates a task to a computer, in Human Computation the computer asks a person or a large group of people to solve a problem; then it collects, interprets and integrates their solutions • The original concept of Human Computation by its inventor Luis von Ahn derived from the common sense observation that people are intrinsically very good at solving some kinds of tasks which are, on the other hand, very hard to address for a computer; this is the case of a number of targets of Artificial Intelligence (like image recognition or natural language understanding) for which research is still open 16copyright © 2018 Cefriel – All rights reserved Edith Law and Luis von Ahn. Human computation. Synthesis Lectures on Artificial Intelligence and Machine Learning, 2011
  17. 17. HUMAN COMPUTATION 17copyright © 2018 Cefriel – All rights reserved Problem: an Artificial Intelligence algorithm is unable to achieve an adequate result with a satisfactory level of confidence Solution: ask people to intervene when the AI system fails, “masking” the task within another human process Example: https://www.google.com/recaptcha/
  18. 18. CROWDSOURCING • Crowdsourcing is the process to outsource tasks to a “crowd” of distributed people. The possibility to exploit the Internet as vehicle to recruit contributors and to assign tasks led to the rise of micro-work platforms, thus often (but not always) implying a monetary reward. The term Crowdsourcing, although quite recent, is used to indicate a wide range of practices; however, the most common meaning of Crowdsourcing implies that the “crowd” of workers involved in the solution of tasks is different from the traditional or intended groups of task solvers 18copyright © 2018 Cefriel – All rights reserved Jeff Howe. Crowdsourcing: How the power of the crowd is driving the future of business. Random House, 2008
  19. 19. CROWDSOURCING 19copyright © 2018 Cefriel – All rights reserved Problem: a company needs to execute a lot of simple tasks, but cannot afford hiring a person to do that job Solution: pack tasks in bunches (human intelligence tasks or HITs) and outsource them to a very cheap workforce through an online platform Example: https://www.mturk.com/
  20. 20. CITIZEN SCIENCE • Citizen Science is the involvement of volunteers to collect or process data as part of a scientific or research experiment; those volunteers can be the scientists and researchers themselves, but more often the name of this discipline “implies a form of science developed and enacted by citizens” including those “outside of formal scientific institutions”, thus representing a form of public participation to science. Formally, Citizen Science has been defined as “the systematic collection and analysis of data; development of technology; testing of natural phenomena; and the dissemination of these activities by researchers on a primarily avocational basis”. 20copyright © 2018 Cefriel – All rights reserved Alan Irwin. Citizen science: A study of people, expertise and sustainable development. Psychology Press, 1995
  21. 21. CITIZEN SCIENCE 21copyright © 2018 Cefriel – All rights reserved Example: https://www.zooniverse.org/ Problem: a scientific experiment requires the execution of a lot of simple tasks, but researchers are busy Solution: engage the general audience in solving those tasks, explaining that they are contributing to science, research and the public good
  22. 22. SPOT THE DIFFERENCE… • Similarities: • Involvement of people • No automatic replacement • Variations: • Motivation • Reward (glory, money, passion/need) • Hybrids or parallel! 22copyright © 2018 Cefriel – All rights reserved Citizen Science Crowdsourcing Human Computation
  23. 23. GAMES WITH A PURPOSE • A GWAP lets to outsource to humans some steps of a computational process in an entertaining way • The application has a “collateral effect”, because players’ actions are exploited to solve a hidden task • The application *IS* a fully-fledged game (opposed to gamification, which is the use of game-like features in non-gaming environments) • The players are (usually) unaware of the hidden purpose, they simply meet game challenges 23copyright © 2018 Cefriel – All rights reserved Luis Von Ahn. Games with a purpose. Computer, 39(6):92–94, 2006 Luis Von Ahn and Laura Dabbish. Designing games with a purpose. Communications of the ACM, 51(8):58–67, 2008
  24. 24. GAMES WITH A PURPOSE (GWAP) 24copyright © 2018 Cefriel – All rights reserved Problem: it’s the same of Human Computation (ask humans when AI fails) Solution: Solution: hide the task within a game, so that users are motivated by game challenges, often remaining unaware of the hidden purpose, task solution comes from agreement between players
  25. 25. from ideation to business value 25 4. GWAPS FOR DATA LINKING Can we embed data linking tasks within Games with a Purpose? copyright © 2018 Cefriel – All rights reserved
  26. 26. 26 • Input: set of all links <asset> foaf:depiction <photo> • Goal: assign score 𝜎 to rank links on their recognisability/representa- tiveness • The score 𝜎 is a function of 𝑋 𝑁 where 𝑋 is the no. of successes (=recognitions) and 𝑁 the no. of trials of the Bernoulli process (guess or not guess) realized by the game • Cultural heritage assets in Milano and their pictures LINK RANKING copyright © 2018 Cefriel – All rights reserved http://bit.ly/indomilando Pure GWAP with hidden purpose Points, badges, leaderboard as intrinsic reward Link ranking is a result of the “agreement” between players But also an educational “collateral effect” Irene Celino, Andrea Fiano, Riccardo Fino. Analysis of a Cultural Heritage Game with a Purpose with an Educational Incentive. 16th International Conference on Web Engineering, 2016
  27. 27. 27 • Input: set of links <land-area> clc:hasLandCover <land-cover> • Goal: assign score 𝜎 to each link to discover the “right” land cover class • Score 𝜎 of each link is updated on the basis of players’ choices (incremented if link selected, decremented if link not selected) • When the score of a link overcomes the threshold 𝜎 ≥ 𝑡 , the link is considered “true” (and removed from the game) • Two automatic classifications in disagreement: <land-cover-assigned-by-DUSAF> ≠ <land-cover-assigned-by-GL30> LINK VALIDATION copyright © 2018 Cefriel – All rights reserved https://youtu.be/Q0ru1hhDM9Q http://bit.ly/foss4game Pure GWAP with not-so-hidden purpose (played by “experts”) Points, badges, leaderboard as intrinsic reward A player scores if he/she guess one of the two disagreeing classifications Link validation is a result of the “agreement” between players Maria Antonia Brovelli, Irene Celino, Andrea Fiano, Monia Elisa Molinari, Vijaycharan Venkatachalam. A crowdsourcing-based game for land cover validation. Applied Geomatics, 2017
  28. 28. 28 • Input: set of subject resources (pictures) and object resources (classification categories) • Goal: create links <picture> hasCategory <category> and assign score 𝜎 to each link • Score 𝜎 of each link is updated on the basis of players’ choices (incremented if link selected) • When the score of a link overcomes the threshold 𝜎 ≥ 𝑡 , the link is considered “true” (and the picture is removed from the game) • Identify pictures of cities from above between those taken on board of the ISS (the pictures are used then in a scientific process in light pollution research) LINK COLLECTION & VALIDATION copyright © 2018 Cefriel – All rights reserved http://nightknights.eu Pure GWAP with not-so-hidden purpose (but played by anybody) Points, badges, leaderboard as intrinsic reward A player scores if he/she agrees with another player “Bonus” intrinsic reward with NASA pictures! Gloria Re Calegari, Gioele Nasi, Irene Celino. Human Computation vs. Machine Learning: an Experimental Comparison for Image Classification. Human Computation Journal, 2018.
  29. 29. from ideation to business value 29 5. TRUTH INFERENCE & OPEN SCIENCE How do we aggregate the contributions from the crowd? Are individual contribution of any value? copyright © 2018 Cefriel – All rights reserved
  30. 30. AGGREGATION OF CONTRIBUTIONS • The same task is usually given to multiple human contributors (named workers in crowdsourcing) • Results on the same task are then aggregated across different contributors (“wisdom of crowds”) • How to perform the truth inference process? • Simplistic solution: majority voting across all contributors • But… are all contributors “created equal”? No! Less simplistic solutions: • Majority voting across “quality” contributors (filtering out “spammers”) • Weighted majority voting with estimation of contributors “reliability” • Expectation maximization • Message passing… and a lot more! • How to compute contributor reliability? • Assessment tasks (gold standard) with known solution to measure reliability • History of contributions/past behaviours to compute a “reputation” value 30copyright © 2018 Cefriel – All rights reserved
  31. 31. TRUTH INFERENCE GENERIC ALGORITHM 31copyright © 2018 Cefriel – All rights reserved Yudian Zheng, Guoliang Li, Yuanbing Li, Caihua Shan, Reynold Cheng. Truth Inference in Crowdsourcing: Is the Problem Solved? VLDB 2017 Input: contributions Output: truth and reliability Step 2: compute an estimation of contributor reliability (e.g. precision on truth estimation) Step 1: compute an estimation of the truth (e.g. majority voting) Iterate until convergence (e.g. until some difference w.r.t. previous step is really small)
  32. 32. OPEN SCIENCE: ENABLING COMPARE & CONTRAST • Open Science has the aim to make scientific research and data accessible to all levels of society • Repeatability and reproducibility are among the foundational principles of open science • Human Computation aims at involving people in some step of the scientific process • Human contributors generate data to solve assigned tasks • Algorithms aggregate contributions in the truth inference process • Can we compare different truth inference algorithms? • Yes, if we make available the data of the Human Computation process! • What can we share, e.g. in the case of data linking tasks? • “True” and “false” links • Confidence scores of the links • Individual contributions and aggregation process 32copyright © 2018 Cefriel – All rights reserved
  33. 33. PROV-O AND HUMAN COMPUTATION ONTOLOGY • Provenance is information about entities, activities, and people involved in producing a piece of data or thing (used to assess its quality, reliability or trustworthiness) • W3C defined the PROV-O ontology to capture provenance information https://www.w3.org/TR/prov-o/ • The Human Computation ontology extends PROV-O to describe the data shared within a Human Computation Process http://swa.cefriel.it/ontologies/hc • Data linking process information can be published according to linked data principles described with the HC ontology (e.g. data from the Urbanopoly GWAP at http://swa.cefriel.it/linkeddata/) 33copyright © 2018 Cefriel – All rights reserved aggregatedFrom Contributor Contribution Human Computation Task provo:Agent provo:Entity provo:Activity Consolidated Information solvedBy enabledBy contributionFrom solutionTo aggregatedBy Human Computation Algorithm Irene Celino. Human Computation VGI Provenance: Semantic Web-based Representation and Publishing. IEEE Transactions on Geoscience and Remote Sensing, 2013
  34. 34. from ideation to business value 34 6. GUIDELINES Is it that easy to involve people on the Web? What should we care of when designing a human computation system? copyright © 2018 Cefriel – All rights reserved
  35. 35. MICE AND MEN (OR: KEEP IT SIMPLE) • Crowdsourcing workers behave like mice • Mice prefer to use their motor skills (biologically cheap, e.g. pressing a lever to get food) rather than their cognitive skills (biologically expensive, e.g. going through a labyrinth to get food) • Workers prefer/are better at simple tasks (e.g. those that can be solved at first sight) and discard/are worse at more complex tasks (e.g. those that require logics) • Crowdsourcing tasks should be carefully designed • Tasks as simple as possible for the workers to solve • Complex tasks together with other incentives (e.g. variety/novelty) 35copyright © 2018 Cefriel – All rights reserved Panos Ipeirotis. On Mice and Men: The Role of Biology in Crowdsourcing, Keynote talk at Collective Intelligence, 2012.
  36. 36. DIVIDE ET IMPERA (OR: FIND-FIX-VERIFY) • Find-Fix-Verify crowd programming pattern • A long and “expensive” task… • Summarize a text to shorten its total length • …is decomposed in more atomic tasks… 1. find sentences that need to be shortened 2. fix a sentence by shortening it 3. verify which summarized sentence maintains original meaning • …and the complex task is turned into a workflow of simple tasks, and each step is outsourced to a crowd 36copyright © 2018 Cefriel – All rights reserved M. Bernstein, G. Little, R. Miller, B. Hartmann, M. Ackerman, D. Karger, D. Crowell, K. Panovich. Soylent: A Word Processor with a Crowd Inside, UIST Proceedings, 2010.
  37. 37. COMPARE AND CONTRAST • A sort of “wisdom of the crowd(sourcing methods)”: (1) apply different approaches to solve the same problem and (2) compare results • Which is the best approach for a specific use case? • Which is the most suitable crowd? • Is human computation better/faster/cheaper than machine computation? • Knowledge Graph Refinement: use Human Computation to “crowdsource” a gold standard and then use it to train some statistical/machine learning algorithm 37copyright © 2018 Cefriel – All rights reserved input task output solution Human Computation Machine Computation input task output solution Human Computation Machine Computation input task output solution Machine Computation Human Computation input task output solution Machine Computation Human Computation Human Computation Gloria Re Calegari, Gioele Nasi, Irene Celino. Human Computation vs. Machine Learning: an Experimental Comparison for Image Classification. Human Computation Journal, 2018.
  38. 38. FINAL NOTE ON DISAGREEMENT • Is there always a “right answer”? Or is there a “crowd truth”? • Not always true/false, because of human subjectivity, ambiguity and uncertainty • Disagreement across contributors is not necessarily bad, but a sign of: different opinions, interpretations, contexts, perspectives, … • Remember the long tail theory… • …and ask yourself who are your users and who you want to involve 38copyright © 2018 Cefriel – All rights reserved Lora Aroyo, Chris Welty. Truth is a Lie: 7 Myths about Human Annotation. AI Magazine 2014.
  39. 39. from ideation to business value 39 7. INDIRECT PEOPLE INVOLVEMENT Are there indirect ways to involve humans in data processing? copyright © 2018 Cefriel – All rights reserved
  40. 40. HUMANS AS A SOURCE OF INFORMATION • People are not only task executors, they are also information providers! • Opportunistic sensing • Voluntary or involuntary digital traces of human-related activities • e.g., phone call logs, GPS traces, social media activities • Open content and cooperative knowledge • Data explicitly provided by people can “hide” further information • e.g., logs of wiki editing, statistical distribution of contributes 40copyright © 2018 Cefriel – All rights reserved
  41. 41. FROM POI INFORMATION AND PHONE CALL LOGS TO LAND USE • General topic: exploit “low-cost” information about a geographic area as features to train a predictive model that outputs “expensive” information about the same area • “Inexpensive” input information: • Geo-information about points of interests • Mobile traffic data processed using different time series techniques – smoothing, decomposition, filtering, time-windowing • “Expensive” output information: • Land use characterization (usually collected through long and expensive workflows that mix machine processing and costly human labour) 41copyright © 2018 Cefriel – All rights reserved Gloria Re Calegari, Emanuela Carlino, Diego Peroni, Irene Celino. Extracting Urban Land Use from Linked Open Geospatial Data. IJGI, 2015 Gloria Re Calegari, Emanuela Carlino, Diego Peroni, Irene Celino. Filtering and Windowing Mobile Traffic Time Series for Territorial Land Use Classification. COMCOM, 2016
  42. 42. FROM SPATIAL ANALYTICS TO GEO-ONTOLOGY ENGINEERING • OpenStreetMap collects information about points of interest (POI) • Spatial distribution and conglomeration of specific POIs can give hints about the geographical space • Re-engineering of spatial features through comparison between areas: same POI type shows different distribution  evidence for different semantics (e.g. what is a pub in Milano vs. London) • Semantic specification of spatial neighbourhoods: • Emerging neighbourhoods from spatial clustering of POIs (opposed to administrative divisions) • Spatial version of tf-idf to compare between different areas (e.g. central or peripheral areas in different cities) and to characterise neighbourhoods (e.g. shopping district) 42copyright © 2018 Cefriel – All rights reserved Gloria Re Calegari, Emanuela Carlino, Irene Celino, Diego Peroni. Supporting Geo-Ontology Engineering through Spatial Data Analytics. 13th Extended Semantic Web Conference, 2016
  43. 43. MILANO viale Sarca 226, 20126, Milano - Italy LONDON 4th floor 57 Rathbone Place London W1T 1JU – UK NEW YORK One Liberty Plaza, 165 Broadway, 23rd Floor, New York City, New York, 10006 USA Cefriel.com Thanks for your attention! Any question? Irene Celino Knowledge Technologies Digital Interaction Division irene.celino@cefriel.com

×