Tutorial by Omar Alonso and Matthew Lease, presented July 24, 2011 at the 34th Annual ACM SIGIR Conference in Beijing, China. See http://www.sigir2011.org/crowdsourcing-for-information-retrieval.htm.
For beginners in Computer Networking field, know about basics about what is Computer Networks, what are ways through which we can communicate, what is meant by protocols, he famous OSI its layers and TCP/IP and its layers, What makes a Networking Device, basic topologies of Computer Networks, special purpose network devices like Switch, routers, repeter, bridges and hub, and basics of networking methods, basics of ethernet, TCP its characteristics, UDP, TCP vs UDP, One would wonder what happens when information particular to each layer is read by the corresponding protocols at target machine or why is it required?
INTRODUCTION : Server Centric IT Architecture and its Limitations; Storage – Centric IT Architecture and its advantages; Case study: Replacing a server with Storage Networks; The Data Storage and Data Access problem; The Battle for size and access.
INTELLIGENT DISK SUBSYSTEMS – 1
Architecture of Intelligent Disk Subsystems; Hard disks and Internal I/O Channels, JBOD, Storage virtualization using RAID and different RAID levels;
Distributed database system is collection of loosely coupled sites that are independeant of each other.
Distributed transaction model
Concurrency control
2 phase commit protocol
For beginners in Computer Networking field, know about basics about what is Computer Networks, what are ways through which we can communicate, what is meant by protocols, he famous OSI its layers and TCP/IP and its layers, What makes a Networking Device, basic topologies of Computer Networks, special purpose network devices like Switch, routers, repeter, bridges and hub, and basics of networking methods, basics of ethernet, TCP its characteristics, UDP, TCP vs UDP, One would wonder what happens when information particular to each layer is read by the corresponding protocols at target machine or why is it required?
INTRODUCTION : Server Centric IT Architecture and its Limitations; Storage – Centric IT Architecture and its advantages; Case study: Replacing a server with Storage Networks; The Data Storage and Data Access problem; The Battle for size and access.
INTELLIGENT DISK SUBSYSTEMS – 1
Architecture of Intelligent Disk Subsystems; Hard disks and Internal I/O Channels, JBOD, Storage virtualization using RAID and different RAID levels;
Distributed database system is collection of loosely coupled sites that are independeant of each other.
Distributed transaction model
Concurrency control
2 phase commit protocol
Take a look at what's changed in the deployment world, and how you can start getting your deployments ready for migration to the next OS your organization adopts. We'll talk about project planning, technical planning, and user adoption. Learn more: http://dell.to/1GDYpr8
The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using a simple programming model. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-avaiability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-availabile service on top of a cluster of computers, each of which may be prone to failures.
Useful documents for engineering students of CSE, and specially for students of aryabhatta knowledge university, Bihar (A.K.U. Bihar). It covers following topics: Background, logical vs. physical address space, swapping, contiguous memory
allocation, paging, segmentation
Information retrieval 13 alternative set theoretic modelsVaibhav Khanna
Alternative Set Theoretic Models
Fuzzy Set Model :a set theoretic model of document retrieval based on fuzzy theory.
Extended Boolean Model:a set theoretic model of document retrieval based on an extension of the classic Boolean model. The idea is to interpret partial matches as Euclidean distances represented in a vectorial space of index terms.
Install and Configure NGINX Unit, the Universal Application, Web, and Proxy S...NGINX, Inc.
In this hands-on demo and lab, we take you step-by-step through installing NGINX Unit on a Linux system, then configuring it as an app server, web server, and reverse proxy. Following a short review of production features and demo of the lab environment, we let you loose in a disposable lab environment to try NGINX Unit for yourself. During the lab, we’re available online to answer questions or demo anything you might be stuck on.
INTRODUCTION TO INFORMATION RETRIEVAL
This lecture will introduce the information retrieval problem, introduce the terminology related to IR, and provide a history of IR. In particular, the history of the web and its impact on IR will be discussed. Special attention and emphasis will be given to the concept of relevance in IR and the critical role it has played in the development of the subject. The lecture will end with a conceptual explanation of the IR process, and its relationships with other domains as well as current research developments.
INFORMATION RETRIEVAL MODELS
This lecture will present the models that have been used to rank documents according to their estimated relevance to user given queries, where the most relevant documents are shown ahead to those less relevant. Many of these models form the basis for many of the ranking algorithms used in many of past and today’s search applications. The lecture will describe models of IR such as Boolean retrieval, vector space, probabilistic retrieval, language models, and logical models. Relevance feedback, a technique that either implicitly or explicitly modifies user queries in light of their interaction with retrieval results, will also be discussed, as this is particularly relevant to web search and personalization.
Take a look at what's changed in the deployment world, and how you can start getting your deployments ready for migration to the next OS your organization adopts. We'll talk about project planning, technical planning, and user adoption. Learn more: http://dell.to/1GDYpr8
The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using a simple programming model. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-avaiability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-availabile service on top of a cluster of computers, each of which may be prone to failures.
Useful documents for engineering students of CSE, and specially for students of aryabhatta knowledge university, Bihar (A.K.U. Bihar). It covers following topics: Background, logical vs. physical address space, swapping, contiguous memory
allocation, paging, segmentation
Information retrieval 13 alternative set theoretic modelsVaibhav Khanna
Alternative Set Theoretic Models
Fuzzy Set Model :a set theoretic model of document retrieval based on fuzzy theory.
Extended Boolean Model:a set theoretic model of document retrieval based on an extension of the classic Boolean model. The idea is to interpret partial matches as Euclidean distances represented in a vectorial space of index terms.
Install and Configure NGINX Unit, the Universal Application, Web, and Proxy S...NGINX, Inc.
In this hands-on demo and lab, we take you step-by-step through installing NGINX Unit on a Linux system, then configuring it as an app server, web server, and reverse proxy. Following a short review of production features and demo of the lab environment, we let you loose in a disposable lab environment to try NGINX Unit for yourself. During the lab, we’re available online to answer questions or demo anything you might be stuck on.
INTRODUCTION TO INFORMATION RETRIEVAL
This lecture will introduce the information retrieval problem, introduce the terminology related to IR, and provide a history of IR. In particular, the history of the web and its impact on IR will be discussed. Special attention and emphasis will be given to the concept of relevance in IR and the critical role it has played in the development of the subject. The lecture will end with a conceptual explanation of the IR process, and its relationships with other domains as well as current research developments.
INFORMATION RETRIEVAL MODELS
This lecture will present the models that have been used to rank documents according to their estimated relevance to user given queries, where the most relevant documents are shown ahead to those less relevant. Many of these models form the basis for many of the ranking algorithms used in many of past and today’s search applications. The lecture will describe models of IR such as Boolean retrieval, vector space, probabilistic retrieval, language models, and logical models. Relevance feedback, a technique that either implicitly or explicitly modifies user queries in light of their interaction with retrieval results, will also be discussed, as this is particularly relevant to web search and personalization.
Talk presented at the ID360 Conference (http://identity.utexas.edu/id360), May 1, 2013. Paper: http://ssrn.com/abstract=2228728. Joint work with Jessica Hullman, Jeffrey P. Bigham, Michael S. Bernstein, Juho Kim, Walter S. Lasecki, Saeideh Bakhshi, Tanushree Mitra, and Robert C. Miller.
Discovering and Navigating Memes in Social MediaMatthew Lease
Invited talk at SBP 2012: Intl. Conf. on Social Computing, Behavioral-Cultural Modeling, & Prediction (April 3, 2012). Based on paper by Ryu, Lease, and Woodward, to appear at ACM HyperText 2012. Joint work with Hohyon Ryu and Nicholas Woodward.
Keynote on Crowd Computing presented at The 5th International Joint Conference on Natural Language Processing (IJCNLP2011) on November 10th in Chiang Mai, Thailand
Tutorial on foundations of Information Retrieval Models by Thomas Roelleke and Ingo Frommholz presented at the Information Retrieval and Foraging Autumn School at Schloß Dagstuhl, Germany, September 2014.
Dynamic Information Retrieval Tutorial - SIGIR 2015Marc Sloan
Dynamic aspects of Information Retrieval (IR), including changes found in data, users and systems, are increasingly being utilized in search engines and information filtering systems. Examples include large datasets containing sequential data capturing document dynamics and modern IR systems observing user dynamics through interactivity. Existing IR techniques are limited in their ability to optimize over changes, learn with minimal computational footprint and be responsive and adaptive.
The objective of this tutorial is to provide a comprehensive and up-to-date introduction to Dynamic Information Retrieval Modeling. Dynamic IR Modeling is the statistical modeling of IR systems that can adapt to change. It is a natural follow-up to previous statistical IR modeling tutorials with a fresh look on state-of-the-art dynamic retrieval models and their applications including session search and online advertising. The tutorial covers techniques ranging from classic relevance feedback to the latest applications of partially observable Markov decision processes (POMDPs) and presents to fellow researchers and practitioners a handful of useful algorithms and tools for solving IR problems incorporating dynamics.
http://www.dynamic-ir-modeling.org/
A newer version of this tutorial presented at WSDM 2015 can be found here http://www.slideshare.net/marcCsloan/dynamic-information-retrieval-tutorial-wsdm-2015
This version has a greater emphasis on the underlying theory and a guest lecture on evaluation by Dr Emine Yilmaz. The newer version presents a wider range of applications of DIR in state of the art research and includes a guest lecture on evaluation by Prof Charles Clarke.
@inproceedings{Yang:2014:DIR:2600428.2602297,
author = {Yang, Hui and Sloan, Marc and Wang, Jun},
title = {Dynamic Information Retrieval Modeling},
booktitle = {Proceedings of the 37th International ACM SIGIR Conference on Research \&\#38; Development in Information Retrieval},
series = {SIGIR '14},
year = {2014},
isbn = {978-1-4503-2257-7},
location = {Gold Coast, Queensland, Australia},
pages = {1290--1290},
numpages = {1},
url = {http://doi.acm.org/10.1145/2600428.2602297},
doi = {10.1145/2600428.2602297},
acmid = {2602297},
publisher = {ACM},
address = {New York, NY, USA},
keywords = {dynamic information retrieval modeling, probabilistic relevance model, reinforcement learning},
}
Some tips for successful crowdsourcing that I learned during some crowdsourcing projects including the making of a movie,
Useful for everyone who considers crowdsourcing in his/her organisation.
Participant driven professional development that is collaborative rather than competitive,
promotes sharing of best practices: grass-roots innovation, teacher to teacher, educator to educator, student to student, parent to parent, one to another, one to each other, inclusive rather than exclusive, and with everyone joined in a common endeavour. All of this can be enhanced through the use of technology, leveraged for maximum engagement across time, distance and space (but not the most critical element)..
There are those who say that education will only get better if teachers get better. The way to do that, across the globe, is for those of us whose minds, hearts, and souls are thusly engaged, is to share, and to care... The rest will take care of itself if we do this one thing: share & care... There will be no obstacle that is capable of resisting the force of millions, no billions of people, who have determined that in the end, changing the status quo is within our will to effect it, one by one and two by two, relentlessly.
That is the grass-roots power of EdCamps all over the world... Be the change you want to see in education, and it will be so, sooner rather than later...
A talk on my experiences building crowdsourcing applications, both at the Guardian newspaper and for my own personal projects. Presented at Web Directions @media 2010 on June 9th.
Crowdsourcing & Human Computation Labeling Data & Building Hybrid SystemsMatthew Lease
Tutorial given at SIAM Data Mining Conference (http://www.siam.org/meetings/sdm13/), May 3, 2013. Based on earlier tutorials given jointly with Omar Alonso from Microsoft Bing.
Once you’ve made the decision to leverage AI and/or machine learning, now you need to figure out how you will source the training data that is necessary for a fully functioning algorithm. Depending on your use case, you might need a significant amount of training data, and you’ll want to consider how that is labeled and annotated too.
View Applause's webinar with Cognilytica principal analysts Ronald Schmelzer and Kathleen Walch, alongside Kristin Simonini, Applause’s Vice President of Product, as they tackle the modern challenges that today’s companies face with sourcing training data.
The second of the BDVe series of webinars related to Big Data technologies presents the QROWD project. Elena Simperl (University of Southampton) will provide an overview and technical details on how human interaction and crowdsourcing could help in different steps of the data value chain, from data acquisition to data curation and completion, etc. Examples of how to add human in the loop in the domains of Smart Cities and Smart Transportation will be provided.
The second of the BDVe series of webinars related to Big Data technologies presents the QROWD project. Elena Simperl (University of Southampton) will provide an overview and technical details on how human interaction and crowdsourcing could help in different steps of the data value chain, from data acquisition to data curation and completion, etc. Examples of how to add human in the loop in the domains of Smart Cities and Smart Transportation will be provided.
Reflections from a realist evaluation in progress: Scaling ladders and stitch...Debbie_at_IDS
In this session, Isabel Vogel, Melanie Punton and Rob Lloyd will reflect on the first year of a three-year realist impact evaluation, examining the Building Capacity to Use Research Evidence (BCURE) programme funded by the UK Department for International Development.
Crowd sourced intelligence built into search over hadooplucenerevolution
Presented by Ted Dunning, Chief Application Architect, MapR
& Grant Ingersoll, Chief Technology Officer, LucidWorks
Search has quickly evolved from being an extension of the data warehouse to being run as a real time decision processing system. Search is increasingly being used to gather intelligence on multi-structured data leveraging distributed platforms such as Hadoop in the background. This session will provide details on how search engines can be abused to use not text, but mathematically derived tokens to build models that implement reflected intelligence. In such a system, intelligent or trend-setting behavior of some users is reflected back at other users. More importantly, the mathematics of evaluating these models can be hidden in a conventional search engine like SolR, making the system easy to build and deploy. The session will describe how to integrate Apache Solr/Lucene with Hadoop. Then we will show how crowd-sourced search behavior can be looped back into analysis and how constantly self-correcting models can be created and deployed. Finally, we will show how these models can respond with intelligent behavior in realtime.
The slide has details on below points:
1. Introduction to Machine Learning
2. What are the challenges in acceptance of Machine Learning in Banks
3. How to overcome the challenges in adoption of Machine Learning in Banks
4. How to find new use cases of Machine Learning
5. Few current interesting use cases of Machine Learning
Please contact me (shekup@gmail.com) or connect with me on LinkedIn (https://www.linkedin.com/in/shekup/) for more explanation on ML and how it may help your business.
The slides are inspired by:
Survey & interviews done by me with Bankers & Technology Professionals
Presentation from Google NEXT 2017
Presentation by DATUM on Youtube
Royal Society Machine Learning
Big Data & Social Analytics Course from MIT & GetSmarter
Peter Levesque explores the critical areas of measuring, interpreting, and analyzing results to ensure continual improvement of KT activities to produce intended results.
Similar to Crowdsourcing for Information Retrieval: Principles, Methods, and Applications (20)
Automated Models for Quantifying Centrality of Survey ResponsesMatthew Lease
Research talk presented at "Innovations in Online Research" (October 1, 2021)
Event URL: https://web.cvent.com/event/d063e447-1f16-4f70-a375-5d6978b3feea/websitePage:b8d4ce12-3d02-4d24-897d-fd469ca4808a.
Explainable Fact Checking with Humans in-the-loopMatthew Lease
Invited Keynote at KDD 2021 TrueFact Workshop: Making a Credible Web for Tomorrow, August 15, 2021.
https://www.microsoft.com/en-us/research/event/kdd-2021-truefact-workshop-making-a-credible-web-for-tomorrow/#!program-schedule
Talk given at Delft University speaker series on "Crowd Computing & Human-Centered AI" (https://www.academicfringe.org/). November 23, 2020. Covers two 2020 works:
(1) Anubrata Das, Brandon Dang, and Matthew Lease. Fast, Accurate, and Healthier: Interactive Blurring Helps Moderators Reduce Exposure to Harmful Content. In Proceedings of the 8th AAAI Conference on Human Computation and Crowdsourcing (HCOMP), 2020.
Alexander Braylan and Matthew Lease. Modeling and Aggregation of Complex Annotations via Annotation Distances. In Proceedings of the Web Conference, pages 1807--1818, 2020.
AI & Work, with Transparency & the Crowd Matthew Lease
Invited talk at the 2019 AAAI Fall Symposium (https://aaai.org/Symposia/Fall/fss19.php) on Artificial Intelligence and Work (https://waim.network/fs19).
Presentation given at the Linguistic Data Consortium (LDC), University of Pennsylvania, April 2019. Based on presentations at the 6th ACM Collective Intelligence Conference, 2018 and the 6th AAAI Conference on Human Computation & Crowdsourcing (HCOMP), 2018. Blog post: https://blog.humancomputation.com/?p=9932.
Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact...Matthew Lease
Presented at the 31st ACM User Interface Software and Technology Symposium (UIST), 2018. Paper: https://www.ischool.utexas.edu/~ml/papers/nguyen-uist18.pdf
Mix and Match: Collaborative Expert-Crowd Judging for Building Test Collectio...Matthew Lease
Presentation at the 1st Biannual Conference on Design of Experimental Search & Information Retrieval Systems (DESIRES 2018). August 30, 2018. Paper: https://www.ischool.utexas.edu/~ml/papers/kutlu-desires18.pdf
Talk given August 29, 2018 at the 1st Biannual Conference on Design of Experimental Search & Information Retrieval Systems (DESIRES 2018). Paper: https://www.ischool.utexas.edu/~ml/papers/lease-desires18.pdf
Your Behavior Signals Your Reliability: Modeling Crowd Behavioral Traces to E...Matthew Lease
Presentation at the 6th AAAI Conference on Human Computation and Crowdsourcing (HCOMP), July 7, 2018. Work by Tanya Goyal, Tyler McDonnell, Mucahid Kutlu, Tamer Elsayed, and Matthew Lease. Pages 41-49 in conference proceedings. Online version of paper includes corrections to official version in proceedings: https://www.ischool.utexas.edu/~ml/papers/goyal-hcomp18
What Can Machine Learning & Crowdsourcing Do for You? Exploring New Tools for...Matthew Lease
Invited Talk at the ACM JCDL 2018 WORKSHOP ON CYBERINFRASTRUCTURE AND MACHINE LEARNING FOR DIGITAL LIBRARIES AND ARCHIVES. https://www.tacc.utexas.edu/conference/jcdl18
Deep Learning for Information Retrieval: Models, Progress, & OpportunitiesMatthew Lease
Talk given at the 8th Forum for Information Retrieval Evaluation (FIRE, http://fire.irsi.res.in/fire/2016/), December 10, 2016, and at the Qatar Computing Research Institute (QCRI), December 15, 2016.
Systematic Review is e-Discovery in Doctor’s ClothingMatthew Lease
Presentation at ACM SIGIR 2016 Medical Information Retrieval (MedIR) Workshop (http://medir2016.imag.fr), July 21, 2016. Joint work with Gordon V. Comack, An T. Nguyen, Thomas A. Trikalinos, Byron C. Wallace.
Talk at South-by-southwest (SXSW) on crowdsourcing + human computation = crowd computing., March 11, 2016. See http://www.humancomputation.com.
Global growth in Internet connectivity and participation is driving a renaissance in human computation: use of people rather than machines to perform certain computations for which human competency continues to exceed that of state-of-the-art algorithms (e.g. “AI-hard” tasks such as interpreting text or images). While current AI limitations will certainly improve with time, using human computation lets us bulid applications which deliver superior results today. Just as cloud computing now enables us to harness vast Internet computing resources on demand, new crowdsourcing APIsenable us to build computing systems which integrate human computation at run-time, invoking crowd labor on-demand and at-scale. Moreover, we can achieve the best of both worlds by integrating automated AI with human computation, creating hybrid systems with capabilities greater than the sum of their parts. When AI falls short, not only can human computation meet the immediate end-user need, but the results can be fed back into the system to further improve the AI. As a consequence, AI limitations are no longer a bottleneck to delivering innovative, new applications. Such enhanced capabilities have begun to change how we design and implement intelligent systems. While early work in crowd computing focused only on collecting more data from crowds to better train AI, we are increasingly seeing hybrid, socio-computational system emerge which creatively blend human computation and AI at run-time to solve hard computing problems. As such, we find ourselves today in an exhilarating new design space in which intelligent system capabilities are seemingly limited only by our imagination and creativity in designing new algorithms to compute effectively using crowds as well as silicon.
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
GridMate - End to end testing is a critical piece to ensure quality and avoid...ThomasParaiso2
End to end testing is a critical piece to ensure quality and avoid regressions. In this session, we share our journey building an E2E testing pipeline for GridMate components (LWC and Aura) using Cypress, JSForce, FakerJS…
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...Neo4j
Leonard Jayamohan, Partner & Generative AI Lead, Deloitte
This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Communications Mining Series - Zero to Hero - Session 1DianaGray10
This session provides introduction to UiPath Communication Mining, importance and platform overview. You will acquire a good understand of the phases in Communication Mining as we go over the platform with you. Topics covered:
• Communication Mining Overview
• Why is it important?
• How can it help today’s business and the benefits
• Phases in Communication Mining
• Demo on Platform overview
• Q/A
Unlocking Productivity: Leveraging the Potential of Copilot in Microsoft 365, a presentation by Christoforos Vlachos, Senior Solutions Manager – Modern Workplace, Uni Systems
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
A tale of scale & speed: How the US Navy is enabling software delivery from l...sonjaschweigert1
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
Threats to mobile devices are more prevalent and increasing in scope and complexity. Users of mobile devices desire to take full advantage of the features
available on those devices, but many of the features provide convenience and capability but sacrifice security. This best practices guide outlines steps the users can take to better protect personal devices and information.
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...SOFTTECHHUB
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Crowdsourcing for Information Retrieval: Principles, Methods, and Applications
1. Crowdsourcing for Information Retrieval:
Principles, Methods, and Applications
Omar Alonso
Microsoft
Matthew Lease
University of Texas at Austin
July 28, 2011
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 1
2. Tutorial Objectives
• What is crowdsourcing? (!= MTurk)
• How and when to use crowdsourcing?
• How to use Mechanical Turk
• Experimental setup and design guidelines for
working with the crowd
• Quality control: issues, measuring, and improving
• IR + Crowdsourcing
– research landscape and open challenges
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 2
3. Tutorial Outline
I. Introduction and Motivating Examples
II. Amazon Mechanical Turk (and CrowdFlower)
III. Relevance Judging and Crowdsourcing
IV. Design of experiments (the good stuff)
V. From Labels to Human Computation
VI. Worker Incentives (money isn’t everything)
VII.The Road Ahead (+ refs at end)
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 3
4. Terminology We’ll Cover
• Crowdsourcing: more than a buzzword?
– What is and isn’t crowdsourcing?
– Subset we discuss: micro-tasks (diagram coming)
• Human Computation = having people do stuff
– Functional view of human work, both helpful & harmful
• AMT / MTurk
– HIT, Requester, Assignment, Turking & Turkers
• Quality Control (QC)
– spam & spammers
– label aggregation, consensus, plurality, multi-labeling
– “gold” data, honey pots, verifiable answers, trap questions
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 4
6. From Outsourcing to Crowdsourcing
• Take a job traditionally
performed by a known agent
(often an employee)
• Outsource it to an undefined,
generally large group of
people via an open call
• New application of principles
from open source movement
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 6
7. Community Q&A / Social Search /
Public Polling
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 7
8. July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 8
9. Mechanical What?
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 9
10. Mechanical Turk (MTurk)
Chess machine constructed and
unveiled in 1770 by Wolfgang
von Kempelen (1734–1804)
J. Pontin. Artificial Intelligence, With Help From
the Humans. NY Times (March 25, 2007)
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 10
11. • “Micro-task” crowdsourcing marketplace
• On-demand, scalable, real-time workforce
• Online since 2005 (and still in “beta”)
• Programmer’s API & “Dashboard” GUI
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 11
12. This isn’t just a lab toy…
http://www.mturk-tracker.com (P. Ipeirotis’10)
From 1/09 – 4/10, 7M HITs from 10K requestors
worth $500,000 USD (significant under-estimate)
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 12
13. Why Crowdsourcing for IR?
• Easy, cheap and fast labeling
• Ready-to use infrastructure
– MTurk payments, workforce, interface widgets
– CrowdFlower quality control mechanisms, etc.
• Allows early, iterative, frequent experiments
– Iteratively prototype and test new ideas
– Try new tasks, test when you want & as you go
• Proven in major IR shared task evaluations
– CLEF image, TREC, INEX, WWW/Yahoo SemSearch
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 13
14. Legal Disclaimer:
Caution Tape and Silver Bullets
• Often still involves more art than science
• Not a magic panacea, but another alternative
– one more data point for analysis, complements other methods
• Quality may be sacrificed for time/cost/effort
• Hard work & experimental design still required!
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 14
15. Hello World Demo
• We’ll show a simple, short demo of MTurk
• This is a teaser highlighting things we’ll discuss
– Don’t worry about details; we’ll revisit them
• Specific task unimportant
• Big idea: easy, fast, cheap to label with MTurk!
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 15
16. Jane saw the man with the binoculars
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 16
17. DEMO
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 17
18. Traditional Annotation / Data Collection
• Setup data collection software / harness
• Recruit volunteers (often undergrads)
• Pay a flat fee for experiment or hourly wage
• Characteristics
– Slow
– Expensive
– Tedious
– Sample Bias
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 18
19. How about some real examples?
• Let’s see examples of MTurk’s use in prior
studies (many areas!)
– e.g. IR, NLP, computer vision, user studies, usability
testing, psychological studies, surveys, …
• Check bibliography at end for more references
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 19
20. NLP Example – Dialect Identification
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 20
21. NLP Example – Spelling correction
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 21
22. NLP Example – Machine Translation
• Manual evaluation on translation quality is
slow and expensive
• High agreement between non-experts and
experts
• $0.10 to translate a sentence
C. Callison-Burch. “Fast, Cheap, and Creative: Evaluating Translation Quality Using Amazon’s Mechanical Turk”, EMNLP 2009.
B. Bederson et al. Translation by Iteractive Collaboration between Monolingual Users, GI 2010
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 22
23. Snow et al. (2008). EMNLP
• 5 Tasks
– Affect recognition
– Word similarity
– Recognizing textual entailment
– Event temporal ordering
– Word sense disambiguation
• high agreement between crowd
labels and expert “gold” labels
– assumes training data for worker bias correction
• 22K labels for $26 !
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 23
24. CV Example – Painting Similarity
Kovashka & Lease, CrowdConf’10
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 24
25. IR Example – Relevance and ads
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 25
26. Okay, okay! I’m a believer!
How can I get started with MTurk?
• You have an idea (e.g. novel IR technique)
• Hiring editors too difficult / expensive / slow
• You don’t have a large traffic query log
Can you test your idea via crowdsourcing?
• Is my idea crowdsourcable?
• How do I start?
• What do I need?
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 26
27. II
AMAZON MECHANICAL TURK
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 27
28. The Requester
• Sign up with your Amazon account
• Amazon payments
• Purchase prepaid HITs
• There is no minimum or up-front fee
• MTurk collects a 10% commission
• The minimum commission charge is $0.005 per HIT
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 28
29. Mturk Dashboard
• Three tabs
– Design
– Publish
– Manage
• Design
– HIT Template
• Publish
– Make work available
• Manage
– Monitor progress
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 29
30. Dashboard - II
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 30
31. API
• Amazon Web Services API
• Rich set of services
• Command line tools
• More flexibility than dashboard
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 31
32. Dashboard vs. API
• Dashboard
– Easy to prototype
– Setup and launch an experiment in a few minutes
• API
– Ability to integrate AMT as part of a system
– Ideal if you want to run experiments regularly
– Schedule tasks
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 32
33. But where do my labels come from?
• An all powerful black box?
• A magical, faraway land?
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 33
34. Nope, MTurk has actual workers too!
• Sign up with your Amazon account
• Tabs
– Account: work approved/rejected
– HIT: browse and search for work
– Qualifications: browse & search qualifications
• Start turking!
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 34
35. Doing some work
• Strongly recommended
• Do some work before you create work
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 35
36. But who are
my workers?
• A. Baio, November 2008. The Faces of Mechanical Turk.
• P. Ipeitorotis. March 2010. The New Demographics of
Mechanical Turk
• J. Ross, et al. Who are the Crowdworkers?... CHI 2010.
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 36
37. Worker Demographics
• 2008-2009 studies found
less global and diverse
than previously thought
– US
– Female
– Educated
– Bored
– Money is secondary
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 37
38. 2010 shows increasing diversity
47% US, 34% India, 19% other (P. Ipeitorotis. March 2010)
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 38
39. Is MTurk my only choice? No, see below.
• Crowdflower (since 2007, www.crowdflower.com)
• CloudCrowd
• DoMyStuff
• Livework
• Clickworker
• SmartSheet
• uTest
• Elance
• oDesk
• vWorker (was rent-a-coder)
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 39
40. (since 2007)
• Labor on-demand
• Channels
• Quality control features
• Sponsor: CSE’10, CSDM’11, CIR’11, TREC’11 Crowd Track
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 40
41. High-level Issues in Crowdsourcing
• Process
– Experimental design, annotation guidelines, iteration
• Choose crowdsourcing platform (or roll your own)
• Human factors
– Payment / incentives, interface and interaction design,
communication, reputation, recruitment, retention
• Quality Control / Data Quality
– Trust, reliability, spam detection, consensus labeling
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 41
42. III
RELEVANCE JUDGING & CROWDSOURING
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 42
43. July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 43
44. Relevance and IR
• What is relevance?
– Multidimensional
– Dynamic
– Complex but systematic and measurable
• Relevance in Information Retrieval
• Frameworks
• Types
– System or algorithmic
– Topical
– Pertinence
– Situational
– Motivational
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 44
45. Evaluation
• Relevance is hard to evaluate
– Highly subjective
– Expensive to measure
• Click data
• Professional editorial work
• Verticals
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 45
46. Crowdsourcing and Relevance Evaluation
• For relevance, it combines two main
approaches
– Explicit judgments
– Automated metrics
• Other features
– Large scale
– Inexpensive
– Diversity
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 46
47. User Studies
• Investigate attitudes about saving, sharing, publishing,
and removing online photos
• Survey
– A scenario-based probe of respondent attitudes, designed
to yield quantitative data
– A set of questions (close and open-ended)
– Importance of recent activity
– 41 question
– 7 point scale
• 250 respondents
C. Marshall and F. Shipman. “The Ownership and Reuse of Visual Media”, JCDL 2011.
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 47
48. Elicitation Criteria
• Relevance in a vertical like e-commerce
• Is classical criteria right for e-commerce?
• Classical criteria (Barry and Schamber)
– Accuracy & validity, consensus within the field, content
novelty, depth & scope, presentation, recency, reliability,
verifiability
• E-commerce criteria
– Brand name, product name, price/value (cheap,
affordable, expensive, not suspiciously cheap),
availability, ratings & user reviews, latest model/version,
personal aspects, perceived value, genre & age
• Experiment
– Select e-C and non e-C queries
– Each workerr 1 query/need (e-C or non e-C)
– 7 workers per HIT
O. Alonso and S. Mizzaro. “Relevance criteria for e-commerce: a crowdsourcing-based experimental analysis”, SIGIR 2009.
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 48
49. IR Example – Product Search
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 49
50. IR Example – Snippet Evaluation
• Study on summary lengths
• Determine preferred result length
• Asked workers to categorize web queries
• Asked workers to evaluate snippet quality
• Payment between $0.01 and $0.05 per HIT
M. Kaisser, M. Hearst, and L. Lowe. “Improving Search Results Quality by Customizing Summary Lengths”, ACL/HLT, 2008.
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 50
51. IR Example – Relevance Assessment
• Replace TREC-like relevance assessors with MTurk?
• Selected topic “space program” (011)
• Modified original 4-page instructions from TREC
• Workers more accurate than original assessors!
• 40% provided justification for each answer
O. Alonso and S. Mizzaro. “Can we get rid of TREC assessors? Using Mechanical Turk for relevance assessment”, SIGIR Workshop
on the Future of IR Evaluation, 2009.
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 51
52. IR Example – Timeline Annotation
• Workers annotate timeline on politics, sports, culture
• Given a timex (1970s, 1982, etc.) suggest something
• Given an event (Vietnam, World cup, etc.) suggest a timex
K. Berberich, S. Bedathur, O. Alonso, G. Weikum “A Language Modeling Approach for Temporal Information Needs”. ECIR 2010
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 52
53. IR Example – Is Tweet Interesting?
• Detecting uninteresting content text streams
– Alonso et al. SIGIR 2010 CSE Workshop.
• Is this tweet interesting to the author and
friends only?
• Workers classify tweets
• 5 tweets per HIT, 5 workers, $0.02
• 57% is categorically not interesting
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 53
54. July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 54
55. Started with a joke …
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 55
56. Results for {idiot} at WSDM
February 2011: 5/7 (R), 2/7 (NR)
– Most of the time those TV reality stars have absolutely no talent. They do whatever
they can to make a quick dollar. Most of the time the reality tv stars don not have
a mind of their own. R
– Most are just celebrity wannabees. Many have little or no talent, they just want
fame. R
– I can see this one going both ways. A particular sort of reality star comes to
mind, though, one who was voted off Survivor because he chose not to use his
immunity necklace. Sometimes the label fits, but sometimes it might be unfair. R
– Just because someone else thinks they are an "idiot", doesn't mean that is what the
word means. I don't like to think that any one person's photo would be used to
describe a certain term. NR
– While some reality-television stars are genuinely stupid (or cultivate an image of
stupidity), that does not mean they can or should be classified as "idiots." Some
simply act that way to increase their TV exposure and potential earnings. Other
reality-television stars are really intelligent people, and may be considered as
idiots by people who don't like them or agree with them. It is too subjective an
issue to be a good result for a search engine. NR
– Have you seen the knuckledraggers on reality television? They should be required to
change their names to idiot after appearing on the show. You could put numbers
after the word idiot so we can tell them apart. R
– Although I have not followed too many of these shows, those that I have encountered
have for a great part a very common property. That property is that most of the
participants involved exhibit a shallow self-serving personality that borders on
social pathological behavior. To perform or act in such an abysmal way could only
be an act of an idiot. R
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 56
57. Two Simple Examples of MTurk
1. Ask workers to classify a query
2. Ask workers to judge document relevance
Steps
• Define high-level task
• Design & implement interface & backend
• Launch, monitor progress, and assess work
• Iterate design
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 57
58. Query Classification Task
• Ask the user to classify a query
• Show a form that contains a few categories
• Upload a few queries (~20)
• Use 3 workers
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 58
59. DEMO
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 59
60. July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 60
61. Relevance Evaluation Task
• Relevance assessment task
• Use a few documents from TREC
• Ask user to perform binary evaluation
• Modification: graded evaluation
• Use 5 workers
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 61
62. DEMO
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 62
63. Typical Workflow
• Define and design what to test
• Sample data
• Design the experiment
• Run experiment
• Collect data and analyze results
• Quality control
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 63
64. Crowdsourcing in Major IR Evaluations
• CLEF image
• Nowak and Ruger, MIR’10
• TREC blog
• McCreadie et al., CSE’10, CSDM’11
• INEX book
• Kazai et al., SIGIR’11
• SemSearch
• Blanco et al., SIGIR’11
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 64
65. BREAK
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 65
66. IV
DESIGN OF EXPERIMENTS
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 66
67. July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 67
68. Survey Design
• One of the most important parts
• Part art, part science
• Instructions are key
• Prepare to iterate
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 68
69. Questionnaire Design
• Ask the right questions
• Workers may not be IR experts so don’t
assume the same understanding in terms of
terminology
• Show examples
• Hire a technical writer
– Engineer writes the specification
– Writer communicates
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 69
70. UX Design
• Time to apply all those usability concepts
• Generic tips
– Experiment should be self-contained.
– Keep it short and simple. Brief and concise.
– Be very clear with the relevance task.
– Engage with the worker. Avoid boring stuff.
– Always ask for feedback (open-ended question) in
an input box.
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 70
71. UX Design - II
• Presentation
• Document design
• Highlight important concepts
• Colors and fonts
• Need to grab attention
• Localization
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 71
72. Examples - I
• Asking too much, task not clear, “do NOT/reject”
• Worker has to do a lot of stuff
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 72
73. Example - II
• Lot of work for a few cents
• Go here, go there, copy, enter, count …
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 73
74. A Better Example
• All information is available
– What to do
– Search result
– Question to answer
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 74
75. July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 75
76. Form and Metadata
• Form with a close question (binary relevance) and
open-ended question (user feedback)
• Clear title, useful keywords
• Workers need to find your task
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 76
77. TREC Assessment – Example I
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 77
78. TREC Assessment – Example II
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 78
79. How Much to Pay?
• Price commensurate with task effort
– Ex: $0.02 for yes/no answer + $0.02 bonus for optional feedback
• Ethics & market-factors: W. Mason and S. Suri, 2010.
– e.g. non-profit SamaSource contracts workers refugee camps
– Predict right price given market & task: Wang et al. CSDM’11
• Uptake & time-to-completion vs. Cost & Quality
– Too little $$, no interest or slow – too much $$, attract spammers
– Real problem is lack of reliable QA substrate
• Accuracy & quantity
– More pay = more work, not better (W. Mason and D. Watts, 2009)
• Heuristics: start small, watch uptake and bargaining feedback
• Worker retention (“anchoring”)
See also: L.B. Chilton et al. KDD-HCOMP 2010.
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 79
80. Development Framework
• Incremental approach
• Measure, evaluate, and adjust as you go
• Suitable for repeatable tasks
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 80
81. Implementation
• Similar to a UX
• Build a mock up and test it with your team
– Yes, you need to judge some tasks
• Incorporate feedback and run a test on MTurk
with a very small data set
– Time the experiment
– Do people understand the task?
• Analyze results
– Look for spammers
– Check completion times
• Iterate and modify accordingly
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 81
82. Implementation – II
• Introduce quality control
– Qualification test
– Gold answers (honey pots)
• Adjust passing grade and worker approval rate
• Run experiment with new settings & same data
• Scale on data
• Scale on workers
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 82
83. Experiment in Production
• Lots of tasks on MTurk at any moment
• Need to grab attention
• Importance of experiment metadata
• When to schedule
– Split a large task into batches and have 1 single
batch in the system
– Always review feedback from batch n before
uploading n+1
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 83
84. July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 84
85. Quality Control
• Extremely important part of the experiment
• Approach as “overall” quality; not just for workers
• Bi-directional channel
– You may think the worker is doing a bad job.
– The same worker may think you are a lousy requester.
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 85
86. Quality Control - II
• Approval rate: easy to use, & just as easily defeated
– P. Ipeirotis. Be a Top Mechanical Turk Worker: You Need
$5 and 5 Minutes. Oct. 2010
• Mechanical Turk Masters (June 23, 2011)
– Very recent addition, amount of benefit uncertain
• Qualification test
– Pre-screen workers’ ability to do the task (accurately)
– Example and pros/cons in next slides
• Assess worker quality as you go
– Trap questions with known answers (“honey pots”)
– Measure inner-annotator agreement between workers
• No guarantees
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 86
87. A qualification test snippet
<Question>
<QuestionIdentifier>question1</QuestionIdentifier>
<QuestionContent>
<Text>Carbon monoxide poisoning is</Text>
</QuestionContent>
<AnswerSpecification>
<SelectionAnswer>
<StyleSuggestion>radiobutton</StyleSuggestion>
<Selections>
<Selection>
<SelectionIdentifier>1</SelectionIdentifier>
<Text>A chemical technique</Text>
</Selection>
<Selection>
<SelectionIdentifier>2</SelectionIdentifier>
<Text>A green energy treatment</Text>
</Selection>
<Selection>
<SelectionIdentifier>3</SelectionIdentifier>
<Text>A phenomena associated with sports</Text>
</Selection>
<Selection>
<SelectionIdentifier>4</SelectionIdentifier>
<Text>None of the above</Text>
</Selection>
</Selections>
</SelectionAnswer>
</AnswerSpecification>
July 24, 2011
</Question> Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 87
88. Qualification tests: pros and cons
• Advantages
– Great tool for controlling quality
– Adjust passing grade
• Disadvantages
– Extra cost to design and implement the test
– May turn off workers, hurt completion time
– Refresh the test on a regular basis
– Hard to verify subjective tasks like judging relevance
• Try creating task-related questions to get worker
familiar with task before starting task in earnest
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 88
89. Methods for measuring agreement
• What to look for
– Agreement, reliability, validity
• Inter-agreement level
– Agreement between judges
– Agreement between judges and the gold set
• Some statistics
– Percentage agreement
– Cohen’s kappa (2 raters)
– Fleiss’ kappa (any number of raters)
– Krippendorff’s alpha
• With majority vote, what if 2 say relevant, 3 say not?
– Use expert to break ties (Kochhar et al, HCOMP’10; GQR)
– Collect more judgments as needed to reduce uncertainty
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 89
90. Inter-rater reliability
• Lots of research
• Statistics books cover most of the material
• Three categories based on the goals
– Consensus estimates
– Consistency estimates
– Measurement estimates
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 90
91. Quality control on relevance assessments
• INEX 2008 Book track
• Home grown system (no MTurk)
• Propose a game for collecting assessments
• CRA Method
G. Kazai, N. Milic-Frayling, and J. Costello. “Towards Methods for the Collective Gathering and Quality Control of Relevance
Assessments”, SIGIR 2009.
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 91
92. July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 92
93. Quality Control & Assurance
• Filtering
– Approval rate (built-in but defeatable)
– Geographic restrictions (e.g. US only, built-in)
– Worker blocking
– Qualification test
• Con: slows down experiment, difficult to “test” relevance
• Solution: create questions to let user get familiar before the assessment
– Does not guarantee success
• Assessing quality
– Interject verifiable/gold answers (trap questions, honey pots)
• P. Ipeitotis. Worker Evaluation in Crowdsourcing: Gold Data or Multiple Workers? Sept. 2010.
– 2-tier approach: Group 1 does task, Group 2 verifies
• Quinn and B. Bederson’09, Bernstein et al.’10
• Identify workers that always disagree with the majority
– Risk: masking cases of ambiguity or diversity, “tail” behaviors
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 93
94. More on quality control & assurance
• HR issues: recruiting, selection, & retention
– e.g., post/tweet, design a better qualification test,
bonuses, …
• Collect more redundant judgments…
– at some point defeats cost savings of crowdsourcing
– 5 workers is often sufficient
• Use better aggregation method
– Voting
– Consensus
– Averaging
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 94
95. Data quality
• Data quality via repeated labeling
• Repeated labeling can improve label quality
and model quality
• When labels are noisy, repeated labeling can
preferable to a single labeling
• Cost issues with labeling
V. Sheng, F. Provost, P. Ipeirotis. “Get Another Label? Improving Data Quality and Data Mining Using Multiple, Noisy Labelers” KDD 2008.
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 95
96. Scales and labels
• Binary
– Yes, No
• 5-point Likert
– Strongly disagree, disagree, neutral, agree, strongly agree
• Graded relevance:
– DCG: Irrelevant, marginally, fairly, highly (Jarvelin, 2000)
– TREC: Highly relevant, relevant, (related), not relevant
– Yahoo/MS: Perfect, excellent, good, fair, bad (PEGFB)
– The Google Quality Raters Handbook (March 2008)
– 0 to 10 (0 = totally irrelevant, 10 = most relevant)
• Usability factors
– Provide clear, concise labels that use plain language
– Avoid unfamiliar jargon and terminology
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 96
97. Was the task difficult?
• Ask workers to rate the difficulty of a topic
• 50 topics, TREC; 5 workers, $0.01 per task
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 97
98. Other quality heuristics
• Justification/feedback as quasi-captcha
– Successfully used at TREC and INEX experiments
– Should be optional
– Automatically verifying feedback was written by a
person may be difficult (classic spam detection task)
• Broken URL/incorrect object
– Leave an outlier in the data set
– Workers will tell you
– If somebody answers “excellent” on a graded
relevance test for a broken URL => probably spammer
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 98
99. MTurk QA: Tools and Packages
• QA infrastructure layers atop MTurk promote
useful separation-of-concerns from task
– TurkIt
• Quik Turkit provides nearly realtime services
– Turkit-online (??)
– Get Another Label (& qmturk)
– Turk Surveyor
– cv-web-annotation-toolkit (image labeling)
– Soylent
– Boto (python library)
• Turkpipe: submit batches of jobs using the command line.
• More needed…
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 99
100. Dealing with bad workers
• Pay for “bad” work instead of rejecting it?
– Pro: preserve reputation, admit if poor design at fault
– Con: promote fraud, undermine approval rating system
• Use bonus as incentive
– Pay the minimum $0.01 and $0.01 for bonus
– Better than rejecting a $0.02 task
• If spammer “caught”, block from future tasks
– May be easier to always pay, then block as needed
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 100
101. Worker feedback
• Real feedback received via email after rejection
• Worker XXX
I did. If you read these articles most of them have
nothing to do with space programs. I’m not an idiot.
• Worker XXX
As far as I remember there wasn't an explanation about
what to do when there is no name in the text. I believe I
did write a few comments on that, too. So I think you're
being unfair rejecting my HITs.
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 101
102. Real email exchange with worker after rejection
WORKER: this is not fair , you made me work for 10 cents and i lost my 30 minutes
of time ,power and lot more and gave me 2 rejections atleast you may keep it
pending. please show some respect to turkers
REQUESTER: I'm sorry about the rejection. However, in the directions given in the
hit, we have the following instructions: IN ORDER TO GET PAID, you must judge all 5
webpages below *AND* complete a minimum of three HITs.
Unfortunately, because you only completed two hits, we had to reject those hits.
We do this because we need a certain amount of data on which to make decisions
about judgment quality. I'm sorry if this caused any distress. Feel free to contact me
if you have any additional questions or concerns.
WORKER: I understood the problems. At that time my kid was crying and i went to
look after. that's why i responded like that. I was very much worried about a hit
being rejected. The real fact is that i haven't seen that instructions of 5 web page
and started doing as i do the dolores labs hit, then someone called me and i went
to attend that call. sorry for that and thanks for your kind concern.
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 102
103. Exchange with worker
• Worker XXX
Thank you. I will post positive feedback for you at
Turker Nation.
Me: was this a sarcastic comment?
• I took a chance by accepting some of your HITs to see if
you were a trustworthy author. My experience with you
has been favorable so I will put in a good word for you
on that website. This will help you get higher quality
applicants in the future, which will provide higher
quality work, which might be worth more to you, which
hopefully means higher HIT amounts in the future.
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 103
104. Build Your Reputation as a Requestor
• Word of mouth effect
– Workers trust the requester (pay on time, clear
explanation if there is a rejection)
– Experiments tend to go faster
– Announce forthcoming tasks (e.g. tweet)
• Disclose your real identity?
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 104
105. Other practical tips
• Sign up as worker and do some HITs
• “Eat your own dog food”
• Monitor discussion forums
• Address feedback (e.g., poor guidelines,
payments, passing grade, etc.)
• Everything counts!
– Overall design only as strong as weakest link
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 105
106. Content quality
• People like to work on things that they like
• TREC ad-hoc vs. INEX
– TREC experiments took twice to complete
– INEX (Wikipedia), TREC (LA Times, FBIS)
• Topics
– INEX: Olympic games, movies, salad recipes, etc.
– TREC: cosmic events, Schengen agreement, etc.
• Content and judgments according to modern times
– Airport security docs are pre 9/11
– Antarctic exploration (global warming )
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 106
107. Content quality - II
• Document length
• Randomize content
• Avoid worker fatigue
– Judging 100 documents on the same subject can
be tiring, leading to decreasing quality
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 107
108. Presentation
• People scan documents for relevance cues
• Document design
• Highlighting no more than 10%
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 108
109. Presentation - II
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 109
110. Relevance justification
• Why settle for a label?
• Let workers justify answers
• INEX
– 22% of assignments with comments
• Must be optional
• Let’s see how people justify
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 110
111. “Relevant” answers
[Salad Recipes]
Doesn't mention the word 'salad', but the recipe is one that could be considered a
salad, or a salad topping, or a sandwich spread.
Egg salad recipe
Egg salad recipe is discussed.
History of salad cream is discussed.
Includes salad recipe
It has information about salad recipes.
Potato Salad
Potato salad recipes are listed.
Recipe for a salad dressing.
Salad Recipes are discussed.
Salad cream is discussed.
Salad info and recipe
The article contains a salad recipe.
The article discusses methods of making potato salad.
The recipe is for a dressing for a salad, so the information is somewhat narrow for
the topic but is still potentially relevant for a researcher.
This article describes a specific salad. Although it does not list a specific recipe,
it does contain information relevant to the search topic.
gives a recipe for tuna salad
relevant for tuna salad recipes
relevant to salad recipes
this is on-topic for salad recipes
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 111
112. “Not relevant” answers
[Salad Recipes]
About gaming not salad recipes.
Article is about Norway.
Article is about Region Codes.
Article is about forests.
Article is about geography.
Document is about forest and trees.
Has nothing to do with salad or recipes.
Not a salad recipe
Not about recipes
Not about salad recipes
There is no recipe, just a comment on how salads fit into meal formats.
There is nothing mentioned about salads.
While dressings should be mentioned with salads, this is an article on one specific
type of dressing, no recipe for salads.
article about a swiss tv show
completely off-topic for salad recipes
not a salad recipe
not about salad recipes
totally off base
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 112
113. July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 113
114. Feedback length
• Workers will justify answers
• Has to be optional for good
feedback
• In E51, mandatory comments
– Length dropped
– “Relevant” or “Not Relevant
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 114
115. Other design principles
• Text alignment
• Legibility
• Reading level: complexity of words and sentences
• Attractiveness (worker’s attention & enjoyment)
• Multi-cultural / multi-lingual
• Who is the audience (e.g. target worker community)
– Special needs communities (e.g. simple color blindness)
• Parsimony
• Cognitive load: mental rigor needed to perform task
• Exposure effect
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 115
116. Platform alternatives
• Why MTurk
– Amazon brand, lots of research papers
– Speed, price, diversity, payments
• Why not
– Crowdsourcing != Mturk
– Spam, no analytics, must build tools for worker & task quality
• How to build your own crowdsourcing platform
– Back-end
– Template language for creating experiments
– Scheduler
– Payments?
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 116
117. The human side
• As a worker
– I hate when instructions are not clear
– I’m not a spammer – I just don’t get what you want
– Boring task
– A good pay is ideal but not the only condition for engagement
• As a requester
– Attrition
– Balancing act: a task that would produce the right results and
is appealing to workers
– I want your honest answer for the task
– I want qualified workers; system should do some of that for me
• Managing crowds and tasks is a daily activity
– more difficult than managing computers
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 117
118. Things that work
• Qualification tests
• Honey-pots
• Good content and good presentation
• Economy of attention
• Things to improve
– Manage workers in different levels of expertise
including spammers and potential cases.
– Mix different pools of workers based on different
profile and expertise levels.
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 118
119. Things that need work
• UX and guidelines
– Help the worker
– Cost of interaction
• Scheduling and refresh rate
• Exposure effect
• Sometimes we just don’t agree
• How crowdsourcable is your task
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 119
120. V
FROM LABELING TO HUMAN COMPUTATION
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 120
121. The Turing Test (Alan Turing, 1950)
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 121
122. July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 122
123. The Turing Test (Alan Turing, 1950)
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 123
124. What is a Computer?
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 124
125. • What was old becomes new
• “Crowdsourcing: A New
Branch of Computer Science”
(March 29, 2011)
Princeton University Press, 2005
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 125
126. Davis et al. (2010) The HPU.
HPU
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 126
127. Human Computation
Rebirth of people as ‘computists’; people do tasks computers cannot (do well)
Stage 1: Detecting robots
– CAPTCHA: Completely Automated Public Turing test to tell Computers and Humans Apart
– No useful work produced; people just answer questions with known answers
Stage 2: Labeling data (at scale)
– E.g. ESP game, typical use of MTurk
– Game changer for AI: starving for data
Stage 3: General “human computation” (HPU)
– people do arbitrarily sophisticated tasks (i.e. compute arbitrary functions)
– HPU as core component in system architecture, many “HPC” invocations
– blend HPU with automation for a new class of hybrid applications
– New tradeoffs possible in latency/cost vs. functionality/accuracy
L. von Ahn has pioneered the field. See bibliography for examples of his work.
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 127
128. Mobile Phone App: “Amazon Remembers”
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 128
129. ReCaptcha
L. von Ahn et al. (2008). In Science.
Harnesses human work as invisible by product.
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 129
130. CrowdSearch and mCrowd
• T. Yan, MobiSys 2010
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 130
131. Soylent: A Word Processor with a Crowd Inside
• Bernstein et al., UIST 2010
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 131
132. Translation by monolingual speakers
• C. Hu, CHI 2009
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 132
133. fold.it
• S. Cooper et al. (2010).
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 133
134. VI
WORKER INCENTIVES
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 134
135. Worker Incentives
• Pay ($$$)
• Fun (or avoid boredom)
• Socialize
• Earn acclaim/prestige
• Altruism
• Learn something new (e.g. English)
• Unintended by-product (e.g. re-Captcha)
• Create self-serving resource (e.g. Wikipedia)
Multiple incentives are typically at work in parallel
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 135
136. Pay ($$$)
P. Ipeirotis March 2010
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 136
137. Pro
• Ready marketplaces (e.g. MTurk, CrowdFlower, …)
• Less need for creativity
• Simple motivation knob
Con: Quality and Quality control required
• Can diminish intrinsic rewards that promote quality:
– Fun/altruistic value of task
– Taking pride in doing quality work
Pay ($$$)
– Self-assessment
• Can attract workers only interested in the pay, fraud
• $$$ (though other schemes cost indirectly)
How much to pay?
• Mason & Watts 2009: more $ = more work, not better work
• Wang et al. 2011: predict from market?
• More later…
Zittrain 2010: if Encarta had paid for contributions, would we have Wikipedia?
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 137
138. Fun (or avoid boredom)
• Games with a Purpose (von Ahn)
– Data is by-product
– IR: Law et al. SearchWar. HCOMP 2009.
• distinct from Serious Gaming / Edutainment
– Player learning / training / education is by-product
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 138
139. • Learning to map from web pages to queries
• Human computation game to elicit data
• Home grown system (no AMT)
• Try it!
pagehunt.msrlivelabs.com
See also:
• H. Ma. et al. “Improving Search Engines Using Human Computation Games”, CIKM 2009.
• Law et al. SearchWar. HCOMP 2009.
• Bennett et al. Picture This. HCOMP 2009.
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 139
140. Fun (or avoid boredom)
• Pro:
– Enjoyable “work” people want to do (or at least better
than anything else they have to do)
– Scalability potential from involving non-workers
• Con:
– Need for design creativity
• some would say this is a plus
• better performance in game should produce better/more work
– Some tasks more amenable than others
• Annotating syntactic parse trees for fun?
• Inferring syntax implicitly from a different activity?
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 140
141. Socialization & Prestige
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 141
142. Socialization & Prestige
• Pro:
– “free”
– enjoyable for connecting with one another
– can share infrastructure across tasks
• Con:
– need infrastructure beyond simple micro-task
– need critical mass (for uptake and reward)
– social engineering knob more complex than $$
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 142
143. Altruism
• Contribute knowledge
• Help others (who need knowledge)
• Help workers (e.g. SamaSource)
• Charity (e.g. http://www.freerice.com)
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 143
144. Altruism
• Pro
– “free”
– can motivate quality work for a cause
• Con
– Seemingly small workforce for pure altruism
What if Mechanical Turk let you donate $$ per HIT?
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 144
145. Unintended by-product
• Pro
– effortless (unnoticed) work
– Scalability from involving non-workers
• Con
– Design challenge
• Given existing activity, find useful work to harness from it
• Given target work, find or create another activity for
which target work is by-product?
– Maybe too invisible (disclosure, manipulation)
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 145
146. Multiple Incentives
• Ideally maximize all
• Wikipedia, cQA, Gwap
– fun, socialization, prestige, altruism
• Fun vs. Pay
– gwap gives Amazon certificates
– Workers maybe paid in game currency
– Pay tasks can also be fun themselves
• Pay-based
– Other rewards: e.g. learn something, socialization
– altruism: worker (e.g. SamaSource) or task itself
– social network integration could help everyone
(currently separate and lacking structure)
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 146
147. VII
THE ROAD AHEAD
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 147
148. Wisdom of Crowds (WoC)
Requires
• Diversity
• Independence
• Decentralization
• Aggregation
Input: large, diverse sample
(to increase likelihood of overall pool quality)
Output: consensus or selection (aggregation)
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 148
149. WoC vs. Ensemble Learning
• Combine multiple models to improve performance
over any constituent model
– Can use many weak learners to make a strong one
– Compensate for poor models with extra computation
• Works better with diverse, independent learners
• cf. NIPS’10 Workshop
– Computational Social Science & the Wisdom of Crowds
• More investigation needed of traditional feature-
based machine learning & ensemble methods for
consensus labeling with crowdsourcing
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 149
150. Unreasonable Effectiveness of Data
• Massive free Web data
changed how we train
learning systems
– Banko and Brill (2001).
Human Language Tech.
– Halevy et al. (2009). IEEE
Intelligent Systems.
• How might access to cheap & plentiful labeled
data change the balance again?
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 150
151. MapReduce with human computation
• Commonalities
– Large task divided into smaller sub-problems
– Work distributed among worker nodes (workers)
– Collect all answers and combine them
– Varying performance of heterogeneous
CPUs/HPUs
• Variations
– Human response latency / size of “cluster”
– Some tasks are not suitable
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 151
152. CrowdForge: MapReduce for
Automation + Human Computation
Kittur et al., CHI 2011
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 152
153. Research problems – operational
• Methodology
– Budget, people, document, queries, presentation,
incentives, etc.
– Scheduling
– Quality
• What’s the best “mix” of HC for a task?
• What are the tasks suitable for HC?
• Can I crowdsource my task?
– Eickhoff and de Vries, WSDM 2011 CSDM Workshop
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 153
154. More problems
• Human factors vs. outcomes
• Editors vs. workers
• Pricing tasks
• Predicting worker quality from observable
properties (e.g. task completion time)
• HIT / Requestor ranking or recommendation
• Expert search : who are the right workers given
task nature and constraints
• Ensemble methods for Crowd Wisdom consensus
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 154
155. Problems: crowds, clouds and algorithms
• Infrastructure
– Current platforms are very rudimentary
– No tools for data analysis
• Dealing with uncertainty (propagate rather than mask)
– Temporal and labeling uncertainty
– Learning algorithms
– Search evaluation
– Active learning (which example is likely to be labeled correctly)
• Combining CPU + HPU
– Human Remote Call?
– Procedural vs. declarative?
– Integration points with enterprise systems
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 155
156. Conclusions
• Crowdsourcing for relevance evaluation works
• Fast turnaround, easy to experiment, cheap
• Still have to design the experiments carefully!
• Usability considerations
• Worker quality
• User feedback extremely useful
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 156
157. Conclusions - II
• Crowdsourcing is here to stay
• Lots of opportunities to improve current platforms
• Integration with current systems
• MTurk is a popular platform and others are emerging
• Open research problems
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 157
158. VIII
RESOURCES AND REFERENCES
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 158
159. Little written for the general public
• July 2010, kindle-only
• “This book introduces you
to the top crowdsourcing
sites and outlines step by
step with photos the exact
process to get started as a
requester on Amazon
Mechanical Turk.“
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 159
160. Crowdsourcing @ SIGIR’11
Workshop on Crowdsourcing for Information Retrieval
Roi Blanco, Harry Halpin, Daniel Herzig, Peter Mika, Jeffrey Pound, Henry Thompson, Thanh D. Tran. “Repeatable and
Reliable Search System Evaluation using Crowd-Sourcing”.
Yen-Ta Huang, An-Jung Cheng, Liang-Chi Hsieh, Winston H. Hsu, Kuo-Wei Chang. “Region-Based Landmark Discovery by
Crowdsourcing Geo-Referenced Photos.” Poster.
Gabriella Kazai, Jaap Kamps, Marijn Koolen, Natasa Milic-Frayling. “Crowdsourcing for Book Search Evaluation: Impact
of Quality on Comparative System Ranking.”
Abhimanu Kumar, Matthew Lease . “Learning to Rank From a Noisy Crowd”. Poster.
Edith Law, Paul N. Bennett, and Eric Horvitz. “The Effects of Choice in Routing Relevance Judgments”. Poster.
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 160
161. 2011 Workshops & Conferences
SIGIR-CIR: Workshop on Crowdsourcing for Information Retrieval (July 28)
• WSDM-CSDM: Crowdsourcing for Search and Data Mining (Feb. 9)
• CHI-CHC: Crowdsourcing and Human Computation (May 8)
• Crowdsourcing: Improving … Scientific Data Through Social Networking (June 13)
• Crowdsourcing Technologies for Language and Cognition Studies (July 27)
• 2011 AAAI-HCOMP: 3rd Human Computation Workshop (Aug. 8)
• UbiComp: 2nd Workshop on Ubiquitous Crowdsourcing (Sep. 18)
• CIKM: BooksOnline (Oct. 24, “crowdsourcing … online books”)
• CrowdConf 2011 -- 2nd Conf. on the Future of Distributed Work (Nov. 1-2)
• TREC-Crowd: Crowdsourcing Track at TREC (Nov. 16-18)
• ACIS: Crowdsourcing, Value Co-Creation, & Digital Economy Innovation (Nov. 30 – Dec. 2)
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 161
162. 2011 Tutorials
• SIGIR (yep, this is it!)
• WSDM: Crowdsourcing 101: Putting the WSDM of Crowds to Work for You
– Omar Alonso and Matthew Lease (Feb. 9)
• WWW: Managing Crowdsourced Human Computation
– Panos Ipeirotis and Praveen Paritosh (March 29)
• HCIC: Quality Crowdsourcing for Human Computer Interaction Research
– Ed Chi (June 14-18)
– Also see Chi’s Crowdsourcing for HCI Research with Amazon Mechanical Turk
• AAAI: Human Computation: Core Research Questions and State of the Art
– Edith Law and Luis von Ahn (Aug. 7)
• VLDB: Crowdsourcing Applications and Platforms
– AnHai Doan, Michael Franklin, Donald Kossmann, and Tim Kraska (Aug. 29)
• CrowdConf: Crowdsourcing for Fun and Profit
– Omar Alonso and Matthew Lease (Nov. 1)
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 162
163. Other Events & Resources
ir.ischool.utexas.edu/crowd
2011 book: Omar Alonso, Gabriella Kazai, and
Stefano Mizzaro. Crowdsourcing for Search Engine
Evaluation: Why and How.
Forthcoming special issue of Springer’s Information
Retrieval journal on Crowdsourcing
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 163
164. Thank You!
For questions about tutorial or crowdsourcing, email:
omar.alonso@microsoft.com
ml@ischool.utexas.edu
Cartoons by Mateo Burtch (buta@sonic.net)
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 164
165. Crowdsourcing in IR: 2008-2010
2008
O. Alonso, D. Rose, and B. Stewart. “Crowdsourcing for relevance evaluation”, SIGIR Forum, Vol. 42, No. 2.
2009
O. Alonso and S. Mizzaro. “Can we get rid of TREC Assessors? Using Mechanical Turk for … Assessment”. SIGIR Workshop on the Future of IR Evaluation.
P.N. Bennett, D.M. Chickering, A. Mityagin. Learning Consensus Opinion: Mining Data from a Labeling Game. WWW.
G. Kazai, N. Milic-Frayling, and J. Costello. “Towards Methods for the Collective Gathering and Quality Control of Relevance Assessments”, SIGIR.
G. Kazai and N. Milic-Frayling. “… Quality of Relevance Assessments Collected through Crowdsourcing”. SIGIR Workshop on the Future of IR Evaluation.
Law et al. “SearchWar”. HCOMP.
H. Ma, R. Chandrasekar, C. Quirk, and A. Gupta. “Improving Search Engines Using Human Computation Games”, CIKM 2009.
2010
SIGIR Workshop on Crowdsourcing for Search Evaluation.
O. Alonso, R. Schenkel, and M. Theobald. “Crowdsourcing Assessments for XML Ranked Retrieval”, ECIR.
K. Berberich, S. Bedathur, O. Alonso, G. Weikum “A Language Modeling Approach for Temporal Information Needs”, ECIR.
C. Grady and M. Lease. “Crowdsourcing Document Relevance Assessment with Mechanical Turk”. NAACL HLT Workshop on … Amazon's Mechanical Turk.
Grace Hui Yang, Anton Mityagin, Krysta M. Svore, and Sergey Markov . “Collecting High Quality Overlapping Labels at Low Cost”. SIGIR.
G. Kazai. “An Exploration of the Influence that Task Parameters Have on the Performance of Crowds”. CrowdConf.
G. Kazai. “… Crowdsourcing in Building an Evaluation Platform for Searching Collections of Digitized Books”., Workshop on Very Large Digital Libraries (VLDL)
Stephanie Nowak and Stefan Ruger. How Reliable are Annotations via Crowdsourcing? MIR.
Jean-François Paiement, Dr. James G. Shanahan, and Remi Zajac. “Crowdsourcing Local Search Relevance”. CrowdConf.
Maria Stone and Omar Alonso. “A Comparison of On-Demand Workforce with Trained Judges for Web Search Relevance Evaluation”. CrowdConf.
T. Yan, V. Kumar, and D. Ganesan. CrowdSearch: exploiting crowds for accurate real-time image search on mobile phones. MobiSys pp. 77--90, 2010.
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 165
166. Crowdsourcing in IR: 2011
WSDM Workshop on Crowdsourcing for Search and Data Mining.
SIGIR Workshop on Crowdsourcing for Information Retrieval.
SIGIR papers/posters mentioned earlier
O. Alonso and R. Baeza-Yates. “Design and Implementation of Relevance Assessments using Crowdsourcing, ECIR.
G. Kasneci, J. Van Gael, D. Stern, and T. Graepel, CoBayes: Bayesian Knowledge Corroboration with Assessors of
Unknown Areas of Expertise, WSDM.
Hyun Joon Jung, Matthew Lease . “Improving Consensus Accuracy via Z-score and Weighted Voting”. HCOMP. Poster.
Gabriella Kazai,. “In Search of Quality in Crowdsourcing for Search Engine Evaluation”, ECIR.
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 166
167. Bibliography: General IR
M. Hearst. “Search User Interfaces”, Cambridge University Press, 2009
K. Jarvelin, and J. Kekalainen. IR evaluation methods for retrieving highly relevant documents. Proceedings of the 23rd annual
international ACM SIGIR conference . pp.41—48, 2000.
M. Kaisser, M. Hearst, and L. Lowe. “Improving Search Results Quality by Customizing Summary Lengths”, ACL/HLT, 2008.
D. Kelly. “Methods for evaluating interactive information retrieval systems with users”. Foundations and Trends in Information
Retrieval, 3(1-2), 1-224, 2009.
S. Mizzaro. Measuring the agreement among relevance judges, MIRA 1999
J. Tang and M. Sanderson. “Evaluation and User Preference Study on Spatial Diversity”, ECIR 2010
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 167
168. Bibliography: Other
J. Barr and L. Cabrera. “AI gets a Brain”, ACM Queue, May 2006.
Bernstein, M. et al. Soylent: A Word Processor with a Crowd Inside. UIST 2010. Best Student Paper award.
Bederson, B.B., Hu, C., & Resnik, P. Translation by Iteractive Collaboration between Monolingual Users, Proceedings of Graphics
Interface (GI 2010), 39-46.
N. Bradburn, S. Sudman, and B. Wansink. Asking Questions: The Definitive Guide to Questionnaire Design, Jossey-Bass, 2004.
C. Callison-Burch. “Fast, Cheap, and Creative: Evaluating Translation Quality Using Amazon’s Mechanical Turk”, EMNLP 2009.
P. Dai, Mausam, and D. Weld. “Decision-Theoretic of Crowd-Sourced Workflows”, AAAI, 2010.
J. Davis et al. “The HPU”, IEEE Computer Vision and Pattern Recognition Workshop on Advancing Computer Vision with Human
in the Loop (ACVHL), June 2010.
M. Gashler, C. Giraud-Carrier, T. Martinez. Decision Tree Ensemble: Small Heterogeneous Is Better Than Large Homogeneous, ICMLA 2008.
D. A. Grief. When Computers Were Human. Princeton University Press, 2005. ISBN 0691091579
JS. Hacker and L. von Ahn. “Matchin: Eliciting User Preferences with an Online Game”, CHI 2009.
J. Heer, M. Bobstock. “Crowdsourcing Graphical Perception: Using Mechanical Turk to Assess Visualization Design”, CHI 2010.
P. Heymann and H. Garcia-Molina. “Human Processing”, Technical Report, Stanford Info Lab, 2010.
J. Howe. “Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business”. Crown Business, New York, 2008.
P. Hsueh, P. Melville, V. Sindhwami. “Data Quality from Crowdsourcing: A Study of Annotation Selection Criteria”. NAACL HLT
Workshop on Active Learning and NLP, 2009.
B. Huberman, D. Romero, and F. Wu. “Crowdsouring, attention and productivity”. Journal of Information Science, 2009.
P.G. Ipeirotis. The New Demographics of Mechanical Turk. March 9, 2010. PDF and Spreadsheet.
P.G. Ipeirotis, R. Chandrasekar and P. Bennett. Report on the human computation workshop. SIGKDD Explorations v11 no 2 pp. 80-83, 2010.
P.G. Ipeirotis. Analyzing the Amazon Mechanical Turk Marketplace. CeDER-10-04 (Sept. 11, 2010)
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 168
169. Bibliography: Other (2)
A. Kittur, E. Chi, and B. Suh. “Crowdsourcing user studies with Mechanical Turk”, SIGCHI 2008.
Aniket Kittur, Boris Smus, Robert E. Kraut. CrowdForge: Crowdsourcing Complex Work. CHI 2011
Adriana Kovashka and Matthew Lease. “Human and Machine Detection of … Similarity in Art”. CrowdConf 2010.
K. Krippendorff. "Content Analysis", Sage Publications, 2003
G. Little, L. Chilton, M. Goldman, and R. Miller. “TurKit: Tools for Iterative Tasks on Mechanical Turk”, HCOMP 2009.
T. Malone, R. Laubacher, and C. Dellarocas. Harnessing Crowds: Mapping the Genome of Collective Intelligence.
2009.
W. Mason and D. Watts. “Financial Incentives and the ’Performance of Crowds’”, HCOMP Workshop at KDD 2009.
J. Nielsen. “Usability Engineering”, Morgan-Kaufman, 1994.
A. Quinn and B. Bederson. “A Taxonomy of Distributed Human Computation”, Technical Report HCIL-2009-23, 2009
J. Ross, L. Irani, M. Six Silberman, A. Zaldivar, and B. Tomlinson. “Who are the Crowdworkers?: Shifting
Demographics in Amazon Mechanical Turk”. CHI 2010.
F. Scheuren. “What is a Survey” (http://www.whatisasurvey.info) 2004.
R. Snow, B. O’Connor, D. Jurafsky, and A. Y. Ng. “Cheap and Fast But is it Good? Evaluating Non-Expert Annotations
for Natural Language Tasks”. EMNLP-2008.
V. Sheng, F. Provost, P. Ipeirotis. “Get Another Label? Improving Data Quality … Using Multiple, Noisy Labelers”
KDD 2008.
S. Weber. “The Success of Open Source”, Harvard University Press, 2004.
L. von Ahn. Games with a purpose. Computer, 39 (6), 92–94, 2006.
L. von Ahn and L. Dabbish. “Designing Games with a purpose”. CACM, Vol. 51, No. 8, 2008.
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 169
170. Bibliography: Other (3)
C. Marshall and F. Shipman “The Ownership and Reuse of Visual Media”, JCDL, 2011.
AnHai Doan, Raghu Ramakrishnan, Alon Y. Halevy: Crowdsourcing systems on the World-Wide Web. CACM, 2011
Paul Heymann, Hector Garcia-Molina: Turkalytics: analytics for human computation. WWW 2011.
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 170
171. Other Resources
Blogs
Behind Enemy Lines (P.G. Ipeirotis, NYU)
Deneme: a Mechanical Turk experiments blog (Gret Little, MIT)
CrowdFlower Blog
http://experimentalturk.wordpress.com
Jeff Howe
Sites
The Crowdsortium
Crowdsourcing.org
CrowdsourceBase (for workers)
Daily Crowdsource
MTurk Forums and Resources
Turker Nation: http://turkers.proboards.com
http://www.turkalert.com (and its blog)
Turkopticon: report/avoid shady requestors
Amazon Forum for MTurk
July 24, 2011 Crowdsourcing for Information Retrieval: Principles, Methods, and Applications 171