Meetup #1 low-code, Pourquoi ? Pour qui ? Comment ? Rencontrons-nous !Simplicité Software
Diaporama de présentation du premier Meetup Low-code Paris.
Vous avez envie de discuter du low-code ? Vous avez des idées ? Des questions sans réponses ? Votre curiosité vous anime ?
Découvrez le panorama des plateformes low-code, les différentes utilisations possibles, des exemples concrets d'applications et le témoignage d'un client DSI.
Automating the Cloud with Terraform, and AnsibleBrian Hogan
Need a web server? So did I. But setting everything up by hand is tedious. In this talk, you'll see how to build a load-balanced web server using Ansible, Terraform, and DigitalOcean, a cloud provider aimed at developers. First, you'll see how to build out the servers and load balancer, and then you'll see how to use Ansible Playbooks to install and upload the web site. When we're done, you'll have scripts you can run to set up your own environment.
Meetup #1 low-code, Pourquoi ? Pour qui ? Comment ? Rencontrons-nous !Simplicité Software
Diaporama de présentation du premier Meetup Low-code Paris.
Vous avez envie de discuter du low-code ? Vous avez des idées ? Des questions sans réponses ? Votre curiosité vous anime ?
Découvrez le panorama des plateformes low-code, les différentes utilisations possibles, des exemples concrets d'applications et le témoignage d'un client DSI.
Automating the Cloud with Terraform, and AnsibleBrian Hogan
Need a web server? So did I. But setting everything up by hand is tedious. In this talk, you'll see how to build a load-balanced web server using Ansible, Terraform, and DigitalOcean, a cloud provider aimed at developers. First, you'll see how to build out the servers and load balancer, and then you'll see how to use Ansible Playbooks to install and upload the web site. When we're done, you'll have scripts you can run to set up your own environment.
History and Basics of containers, LXC, Docker and Kubernetes. This presentation is given to Engineering colleage students at VIT DevFest 2018. Beginner to Intermediate level.
Mistakes - I’ve made a few. Blunders in event-driven architecture | Simon Aub...HostedbyConfluent
Building systems around an event-driven architecture is a powerful pattern for creating awesome data intensive applications. Apache Kafka simplifies scalability and provides an event-driven backbone for service architectures.
But what can go wrong? Let me share some of my own blunders and lessons learnt in building event driven architecture so you don’t have to repeat my mistakes.
Windows Server and Docker - The Internals Behind Bringing Docker and Containe...Docker, Inc.
Docker leverages capabilities in Linux like namespaces and cgroups to enable containers and then builds tooling on top to enable users to build distributed apps. A common question is "What about Docker support for Windows?" In this session the Windows engineering leads will dive deep into the primitives within Windows to enable an awesome Docker experience on Windows. This session will also include a live demo of Docker and Windows Server.
Kubernetes Architecture - beyond a black box - Part 2Hao H. Zhang
This continues the Kubernetes architecture deep dive series. (Part 1 see https://www.slideshare.net/harryzhang735/kubernetes-beyond-a-black-box-part-1)
In Part 2 I'm going to cover the following:
- Kubernetes's 3 most import design choices: Micro-service Choreography, Level-Triggered Control, Generalized Workload and Centralized Controller
- Default scheduler limitation and community's next step
- Interface to production environment
- Workload abstraction: strength and limitations
This concludes my work and knowledge sharing about Kubernetes.
What Is A Docker Container? | Docker Container Tutorial For Beginners| Docker...Simplilearn
This presentation on Docker Container will help you understand what is Docker, the architecture of Docker, what is a Docker Container, how to create a Docker Container, benefits of Docker Container, basic commands of Containers and you will also see a demo on creating Docker Container. Docker is a very lightweight software container and containerization platform. Docker containers provide a way to run software in isolation. It is an open source platform that helps to package an application and its dependencies into a Docker container for the development and deployment of software and a Docker COntainer is a portable executable package which includes applications and their dependencies. With Docker Containers, applications can work efficiently in different computer environments.
Below DevOps tools are explained in this Docker Container presentation:
1. What is Docker?
2. The architecture of Docker?
3. What is a Docker Container?
4. How to create a Docker Container?
5. Benefits of Docker Containers
6. Basic commands of Containers
Simplilearn's DevOps Certification Training Course will prepare you for a career in DevOps, the fast-growing field that bridges the gap between software developers and operations. You’ll become an expert in the principles of continuous development and deployment, automation of configuration management, inter-team collaboration and IT service agility, using modern DevOps tools such as Git, Docker, Jenkins, Puppet and Nagios. DevOps jobs are highly paid and in great demand, so start on your path today.
Why learn DevOps?
Simplilearn’s DevOps training course is designed to help you become a DevOps practitioner and apply the latest in DevOps methodology to automate your software development lifecycle right out of the class. You will master configuration management; continuous integration deployment, delivery and monitoring using DevOps tools such as Git, Docker, Jenkins, Puppet and Nagios in a practical, hands-on and interactive approach. The DevOps training course focuses heavily on the use of Docker containers, a technology that is revolutionizing the way apps are deployed in the cloud today and is a critical skillset to master in the cloud age.
After completing the DevOps training course you will achieve hands-on expertise in various aspects of the DevOps delivery model. The practical learning outcomes of this Devops training course are:
An understanding of DevOps and the modern DevOps toolsets
The ability to automate all aspects of a modern code delivery and deployment pipeline using:
1. Source code management tools
2. Build tools
3. Test automation tools
4. Containerization through Docker
5. Configuration management tools
6. Monitoring tools
DevOps jobs are the third-highest tech role ranked by employer demand on Indeed.com but have the second-highest talent deficit.
Learn more at https://www.simplilearn.com/cloud-computing/devops-practitioner-certification-training
Docker is one of the fastest-growing open source projects ever, and the ecosystem that has grown around it is evolving at a similar pace. For these reasons, we want to introduce developers, system administrators, and other computer users of a mixed skillset to the Docker project and Linux container concepts.
Introduction to Docker presented by MANAOUIL Karim at the Shellmates's Hack.INI event. The teams deployed were assisted to deploy a Python Flask application behind an Nginx load balancer.
This presentation will introduce you to Container, Docker, Kubernetes, and Google Kubernetes Engine (GKE) with a live demo. This also explains Kubernetes basic concepts such as Pod, Deployment, Service, Ingress, and Rolling Update.
See the recorded session on Facebook live here (min 46.49):
https://www.facebook.com/gdgcloudkl/videos/1013942759041907
There's also recorded session on Youtube here (min 46.49):
https://www.youtube.com/watch?v=ht0ynVjkDcI
GDG Cloud KL July Webinar on July 12, 2020
[Harvard CS264] 09 - Machine Learning on Big Data: Lessons Learned from Googl...npinto
Abstract:
Machine learning researchers and practitioners develop computer
algorithms that "improve performance automatically through
experience". At Google, machine learning is applied to solve many
problems, such as prioritizing emails in Gmail, recommending tags for
YouTube videos, and identifying different aspects from online user
reviews. Machine learning on big data, however, is challenging. Some
"simple" machine learning algorithms with quadratic time complexity,
while running fine with hundreds of records, are almost impractical to
use on billions of records.
In this talk, I will describe lessons drawn from various Google
projects on developing large scale machine learning systems. These
systems build on top of Google's computing infrastructure such as GFS
and MapReduce, and attack the scalability problem through massively
parallel algorithms. I will present the design decisions made in
these systems, strategies of scaling and speeding up machine learning
systems on web scale data.
Speaker biography:
Max Lin is a software engineer with Google Research in New York City
office. He is the tech lead of the Google Prediction API, a machine
learning web service in the cloud. Prior to Google, he published
research work on video content analysis, sentiment analysis, machine
learning, and cross-lingual information retrieval. He had a PhD in
Computer Science from Carnegie Mellon University.
History and Basics of containers, LXC, Docker and Kubernetes. This presentation is given to Engineering colleage students at VIT DevFest 2018. Beginner to Intermediate level.
Mistakes - I’ve made a few. Blunders in event-driven architecture | Simon Aub...HostedbyConfluent
Building systems around an event-driven architecture is a powerful pattern for creating awesome data intensive applications. Apache Kafka simplifies scalability and provides an event-driven backbone for service architectures.
But what can go wrong? Let me share some of my own blunders and lessons learnt in building event driven architecture so you don’t have to repeat my mistakes.
Windows Server and Docker - The Internals Behind Bringing Docker and Containe...Docker, Inc.
Docker leverages capabilities in Linux like namespaces and cgroups to enable containers and then builds tooling on top to enable users to build distributed apps. A common question is "What about Docker support for Windows?" In this session the Windows engineering leads will dive deep into the primitives within Windows to enable an awesome Docker experience on Windows. This session will also include a live demo of Docker and Windows Server.
Kubernetes Architecture - beyond a black box - Part 2Hao H. Zhang
This continues the Kubernetes architecture deep dive series. (Part 1 see https://www.slideshare.net/harryzhang735/kubernetes-beyond-a-black-box-part-1)
In Part 2 I'm going to cover the following:
- Kubernetes's 3 most import design choices: Micro-service Choreography, Level-Triggered Control, Generalized Workload and Centralized Controller
- Default scheduler limitation and community's next step
- Interface to production environment
- Workload abstraction: strength and limitations
This concludes my work and knowledge sharing about Kubernetes.
What Is A Docker Container? | Docker Container Tutorial For Beginners| Docker...Simplilearn
This presentation on Docker Container will help you understand what is Docker, the architecture of Docker, what is a Docker Container, how to create a Docker Container, benefits of Docker Container, basic commands of Containers and you will also see a demo on creating Docker Container. Docker is a very lightweight software container and containerization platform. Docker containers provide a way to run software in isolation. It is an open source platform that helps to package an application and its dependencies into a Docker container for the development and deployment of software and a Docker COntainer is a portable executable package which includes applications and their dependencies. With Docker Containers, applications can work efficiently in different computer environments.
Below DevOps tools are explained in this Docker Container presentation:
1. What is Docker?
2. The architecture of Docker?
3. What is a Docker Container?
4. How to create a Docker Container?
5. Benefits of Docker Containers
6. Basic commands of Containers
Simplilearn's DevOps Certification Training Course will prepare you for a career in DevOps, the fast-growing field that bridges the gap between software developers and operations. You’ll become an expert in the principles of continuous development and deployment, automation of configuration management, inter-team collaboration and IT service agility, using modern DevOps tools such as Git, Docker, Jenkins, Puppet and Nagios. DevOps jobs are highly paid and in great demand, so start on your path today.
Why learn DevOps?
Simplilearn’s DevOps training course is designed to help you become a DevOps practitioner and apply the latest in DevOps methodology to automate your software development lifecycle right out of the class. You will master configuration management; continuous integration deployment, delivery and monitoring using DevOps tools such as Git, Docker, Jenkins, Puppet and Nagios in a practical, hands-on and interactive approach. The DevOps training course focuses heavily on the use of Docker containers, a technology that is revolutionizing the way apps are deployed in the cloud today and is a critical skillset to master in the cloud age.
After completing the DevOps training course you will achieve hands-on expertise in various aspects of the DevOps delivery model. The practical learning outcomes of this Devops training course are:
An understanding of DevOps and the modern DevOps toolsets
The ability to automate all aspects of a modern code delivery and deployment pipeline using:
1. Source code management tools
2. Build tools
3. Test automation tools
4. Containerization through Docker
5. Configuration management tools
6. Monitoring tools
DevOps jobs are the third-highest tech role ranked by employer demand on Indeed.com but have the second-highest talent deficit.
Learn more at https://www.simplilearn.com/cloud-computing/devops-practitioner-certification-training
Docker is one of the fastest-growing open source projects ever, and the ecosystem that has grown around it is evolving at a similar pace. For these reasons, we want to introduce developers, system administrators, and other computer users of a mixed skillset to the Docker project and Linux container concepts.
Introduction to Docker presented by MANAOUIL Karim at the Shellmates's Hack.INI event. The teams deployed were assisted to deploy a Python Flask application behind an Nginx load balancer.
This presentation will introduce you to Container, Docker, Kubernetes, and Google Kubernetes Engine (GKE) with a live demo. This also explains Kubernetes basic concepts such as Pod, Deployment, Service, Ingress, and Rolling Update.
See the recorded session on Facebook live here (min 46.49):
https://www.facebook.com/gdgcloudkl/videos/1013942759041907
There's also recorded session on Youtube here (min 46.49):
https://www.youtube.com/watch?v=ht0ynVjkDcI
GDG Cloud KL July Webinar on July 12, 2020
[Harvard CS264] 09 - Machine Learning on Big Data: Lessons Learned from Googl...npinto
Abstract:
Machine learning researchers and practitioners develop computer
algorithms that "improve performance automatically through
experience". At Google, machine learning is applied to solve many
problems, such as prioritizing emails in Gmail, recommending tags for
YouTube videos, and identifying different aspects from online user
reviews. Machine learning on big data, however, is challenging. Some
"simple" machine learning algorithms with quadratic time complexity,
while running fine with hundreds of records, are almost impractical to
use on billions of records.
In this talk, I will describe lessons drawn from various Google
projects on developing large scale machine learning systems. These
systems build on top of Google's computing infrastructure such as GFS
and MapReduce, and attack the scalability problem through massively
parallel algorithms. I will present the design decisions made in
these systems, strategies of scaling and speeding up machine learning
systems on web scale data.
Speaker biography:
Max Lin is a software engineer with Google Research in New York City
office. He is the tech lead of the Google Prediction API, a machine
learning web service in the cloud. Prior to Google, he published
research work on video content analysis, sentiment analysis, machine
learning, and cross-lingual information retrieval. He had a PhD in
Computer Science from Carnegie Mellon University.
Deep learning is a subset of machine learning in which algorithms are used to model high-level abstractions in data. By using a deep learning algorithm, a computer can learn to recognize complex patterns in data, such as images or spoken language. Deep learning is used in a variety of applications, including image classification, natural language processing, and time series prediction.
An "AI Literacy" talk I gave to the Salt Lake Oasis Community (http://www.saltlakeoasis.org) on January 13th, 2019. The aim of the talk was to give the audience literacy with some AI concepts.
The IT discipline of machine learning has become increasingly important in recent years. It promises to solve types of problems for which normal software development is considered unsuitable or too costly.
Big Data, a space adventure - Mario Cartia - Codemotion Rome 2015Codemotion
Codemotion Rome 2015 - I Big Data sono indubbiamente tra i temi più "caldi" del panorama tecnologico attuale. Ad oggi nel mondo sono stati prodotti circa 5 Exabytes di dati che costituiscono una potenziale fonte di "intelligenza" che è possibile sfruttare, grazie alle tecnologie più recenti, in svariati ambiti che spaziano dalla medicina alla sociologia passando per il marketing. Il talk si propone, tramite una gita virtuale nello spazio, di introdurre i concetti, le tecniche e gli strumenti che consentono di iniziare a sfruttare il potenziale dei Big Data nel lavoro quotidiano.
Quizz: Targeted Crowdsourcing with a Billion (Potential) UsersPanos Ipeirotis
We describe Quizz, a gamified crowdsourcing system that simultaneously assesses the knowledge of users and acquires new knowledge from them. Quizz operates by asking users to complete short quizzes on specific topics; as a user answers the quiz questions, Quizz estimates the user’s competence. To acquire new knowledge, Quizz also incorporates questions for which we do not have a known answer; the answers given by competent users provide useful signals for selecting the correct answers for these questions. Quizz actively tries to identify knowledgeable users on the Internet by running advertising campaigns, effectively leveraging “for free” the targeting capabilities of existing, publicly available, ad placement services. Quizz quantifies the contributions of the users using information theory and sends feedback to the advertising system about each user. The feedback allows the ad targeting mechanism to further optimize ad placement.
Our experiments, which involve over ten thousand users, confirm that we can crowdsource knowledge curation for niche and specialized topics, as the advertising network can automatically identify users with the desired expertise and interest in the given topic. We present controlled experiments that examine the effect of various incentive mechanisms, highlighting the need for having short-term rewards as goals, which incentivize the users to contribute. Finally, our cost-quality analysis indicates that the cost of our approach is below that of hiring workers through paid-crowdsourcing platforms, while offering the additional advantage of giving access to billions of potential users all over the planet, and being able to reach users with specialized expertise that is not typically available through existing labor marketplaces.
Slides from the "Humanities and Technology Unite" event at the Museum of the City of New York http://m.mcny.org/programs/Humanities-and-Technology-Unite
A special evening to celebrate the NEH Grant awarded to MCNY and Tagasauris, a NYC Technology start-up, to annotate the Museum's archives using crowdsouring and semantic web technologies.
Strata Jumpstart, New York, September 19 2011
Big Data, Stupid Decisions: The Importance Of Measuring What We Should Be Measuring
Video at http://www.youtube.com/watch?v=LXDFwphuwCs
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Generating a custom Ruby SDK for your web service or Rails API using Smithyg2nightmarescribd
Have you ever wanted a Ruby client API to communicate with your web service? Smithy is a protocol-agnostic language for defining services and SDKs. Smithy Ruby is an implementation of Smithy that generates a Ruby SDK using a Smithy model. In this talk, we will explore Smithy and Smithy Ruby to learn how to generate custom feature-rich SDKs that can communicate with any web service, such as a Rails JSON API.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
2. Outline
• Introduction: Human computation and crowdsourcing
• Managing quality for simple tasks
• Complex tasks using workflows
• Task optimization
• Incentivizing the crowd
Incentivizing the crowd
• Market design
• Behavioral aspects and cognitive biases
Behavioral aspects and cognitive biases
• Game design
• Case studies
d
3. Human Computation, Round 1
• Humans were the first
“computers,” used for
computers, used for
math computations
Grier, When computers were human, 2005
Grier, IEEE Annals 1998
4. Human Computation, Round 1
• Humans were the first
“computers,” used for
math computations
• Organized computation:
– Clairaut, astronomy, 1758:
y
Computed the Halley’s
comet orbit (three‐body
problem) dividing the
problem) dividing the
labor of numeric
computations across 3
astronomers
Grier, When computers were human, 2005
Grier, IEEE Annals 1998
5. Human Computation, Round 1
• Organized computation:
– Maskelyne, astronomical almanac
with moon positions, used for
p ,
navigation, 1760. Quality
assurance by doing calculations
twice and compared by third
verifier.
verifier
– De Prony, 1794, hires hairdressers
(unemployed after French
(unemployed after French
revolution; knew only addition
and subtraction) to create
logarithmic and trigonometric
tables. He managed the process
tables He managed the process
by splitting the work into very
detailed workflows. (Hairdressers better
than mathematicians in arithmetic!)
Grier, When computers were human, 2005
Grier, IEEE Annals 1998
6. Human Computation, Round 1
• Organized computation:
– Clairaut, astronomy, 1758
– Maskelyne, 1760
– De Prony, log/trig tables, 1794
– Galton, biology, 1893
– Pearson, biology, 1899
– …
– Cowles, stock market, 1929
– Math Tables Project, unskilled
Math Tables Project unskilled
labor, 1938
Grier, When computers were human, 2005
Grier, IEEE Annals 1998
7. Human Computation, Round 1
• Patterns emerging
Patterns emerging
– Division of labor
– Mass production
Mass production
– Professional managers
• Then we got the
“automatic computers”
8. Human Computation, Round 2
• Now we need humans
again for the “AI‐complete”
i f th “AI l t ”
tasks
– Tag images [ESP Game: voh Ahn and
Dabbish 2004, ImageNet]
– Determine if page relevant
Determine if page relevant
[Alonso et al., 2011]
– Determine song genre
– Check page for offensive
Check page for offensive
content
–…
ImageNet: http://www.image‐net.org/about‐publication
9. Focus of the tutorial
Focus of the tutorial
Examine cases where humans interact with
Examine cases where humans interact with
computers in order to solve a computational
problem (usually too hard to be solved by
(usually too hard to be solved by
computers alone)
10. Crowdsourcing and human computation
• Crowdsourcing: From macro to micro
g
– Netflix, Innocentive
– Quirky, Threadless
– oDesk, Guru, eLance, vWorker
D k G L W k
– Wikipedia et al.
– ESP Game, FoldIt, Phylo, …
S Ga e, o d t, y o,
– Mechanical Turk, CloudCrowd, …
• Crowdsourcing greatly facilitates human
computation (but they are not equivalent)
11. Micro‐Crowdsourcing Example:
Labeling Images
L b li I
using the ESP Game Luis von Ahn
MacArthur Fellowship
"genius grant"
• Two player online game
• Partners don’t know each other and can’t
communicate
• Object of the game: type the same word
• The only thing in common is an image
The only thing in common is an image
12.
13. PLAYER 1 PLAYER 2
GUESSING: CAR GUESSING: BOY
GUESSING: HAT GUESSING: CAR
GUESSING: KID SUCCESS!
YOU AGREE ON CAR
SUCCESS!
YOU AGREE ON CAR
16. Demographics of MTurk workers
http://bit.ly/mturk‐demographics
http://bit.ly/mturk demographics
Country of residence
Country of residence
• United States: 46.80%
• India: 34.00%
• Miscellaneous: 19.20%
19. Outline
• Introduction: Human computation and crowdsourcing
• Managing quality for simple tasks
• Complex tasks using workflows
• Task optimization
• Incentivizing the crowd
Incentivizing the crowd
• Market design
• Behavioral aspects and cognitive biases
Behavioral aspects and cognitive biases
• Game design
• Case studies
d
20. Managing quality for simple tasks
Managing quality for simple tasks
• Quality through redundancy: Combining votes
Quality through redundancy: Combining votes
– Majority vote
– Quality adjusted vote
Quality‐adjusted vote
– Managing dependencies
• Quality through gold data
Q lit th h ld d t
• Estimating worker quality (Redundancy + Gold)
• Joint estimation of worker quality and difficulty
• Active data collection
Active data collection
25. Bad news: Spammers!
Worker ATAMRO447HWJQ
labeled X (porn) sites as G (general audience)
26. Majority Voting and Label Quality
Ask multiple labelers, keep majority label as “true” label
Quality is probability of being correct
1
p=1.0
0.9
p=0.9
09
0.8 p=0.7
Quality for Majority Vote
p=0.8
0.7 p=0.6
p is probability 0.6
of individual labeler
of individual labeler p=0.5
p=0 5
M
0.5
being correct
0.4 p=0.4
0.3
0.2
1 3 5 7 9 11 13
Number of labelers
Binary classification
Binary classification
26
Kuncheva et al., PA&A, 2003
27. What if qualities of workers are different?
3 workers, qualities: p‐d, p, p+d
Region where majority better
• Majority vote works best when workers have similar quality
j p
• Otherwise better to just pick the vote of the best worker
• …or model worker qualities and combine [coming next]
29. What happens if we have dependencies?
Clemen and Winkler, 1985
Positive dependencies decrease the number of effective labelers
Positive dependencies decrease the number of effective labelers
30. What happens if we have dependencies?
Yule’s Q
Y l ’ Q
measure of correlation
Kuncheva et al., PA&A, 2003
Positive dependencies decrease the number of effective labelers
Positive dependencies decrease the number of effective labelers
Negative dependencies can improve results (unlikely both workers
to be wrong at the same time)
31. Vote combination: Meta studies
Vote combination: Meta‐studies
• Simple averages tend to work well
Simple averages tend to work well
• C
Complex models slightly better but less robust
l d l li h l b b l b
[Clemen and Winkler, 1999, Ariely et al. 2000]
32. From aggregate labels to worker quality
Look at our spammer friend ATAMRO447HWJQ
h ih h 9 k
together with other 9 workers
After aggregation, we compute confusion matrix for each worker
After majority vote, confusion matrix for ATAMRO447HWJQ
P[G → G]=100% P[G → X]=0%
P[X → G]=100% P[X → X]=0%
33. Algorithm of Dawid & Skene, 1979
Iterative process to estimate worker error rates
1. Initialize by aggregating labels for each object (e.g., use majority vote)
2. Estimate confusion matrix for workers (using aggregate
labels)
l b l )
3. Estimate aggregate labels (using confusion matrix)
• Keep labels for “gold data unchanged
gold data”
4. Go to Step 2 and iterate until convergence
Confusion matrix for ATAMRO447HWJQ
Our f i d
O friend ATAMRO447HWJQ
P[G → G]=99.947% P[G → X]=0.053%
marked almost all sites as G.
P[X → G]=99.153% P[X → X]=0.847%
Seems like a spammer…
34. And many variations…
And many variations…
• van der Linden et al, 1997: Item‐Response Theory
a de de et a , 99 : te espo se eo y
• Uebersax, Biostatistics 1993: Ordered categories
• Uebersax, JASA 1993: Ordered categories, with worker
Uebersax, JASA 1993: Ordered categories, with worker
expertise and bias, item difficulty
• Carpenter, 2008: Hierarchical Bayesian versions
p , y
And more recently at NIPS:
y
• Whitehill et al., 2009: Adding item difficulty
• Welinder et al., 2010: Adding worker expertise
et al., 2010: Adding worker expertise
36. Challenge 1:
Spammers are lazy and smart!
Spammers are lazy and smart!
Confusion matrix for spammer Confusion matrix for good worker
P[X → X]=0% P[X → G]=100%
X] 0% G] 100% P[X → X]=80% P[X → G]=20%
P[G → X]=0% P[G → G]=100% P[G → X]=20% P[G → G]=80%
• Spammers figure out how to fly under the radar…
• I
In reality, we have 85% G sites and 15% X sites
lit h 85% G it d 15% X it
• Errors of spammer = 0% * 85% + 100% * 15% = 15%
Errors of spammer 0% 85% + 100% 15% 15%
• Error rate of good worker = 85% * 20% + 85% * 20% = 20%
False negatives: Spam workers pass as legitimate
37. Challenge 2:
Humans are biased!
Humans are biased!
Error rates for CEO of AdSafe
P[G → G]=20.0% P[G → P]=80.0% P[G → R]=0.0% P[G → X]=0.0%
P[P → G]=0.0% P[P → P]=0.0% P[P → R]=100.0% P[P → X]=0.0%
P[R → G]=0.0% P[R → P]=0.0% P[R → R]=100.0% P[R → X]=0.0%
P[X → G] 0 0%
G]=0.0% P[X → P] 0 0%
P]=0.0% P[X → R] 0 0%
R]=0.0% P[X → X] 100 0%
X]=100.0%
In reality, we have 85% G sites, 5% P sites, 5% R sites, 5% X sites
Errors of spammer (all in G) = 0% * 85% + 100% * 15% = 15%
Error rate of biased worker = 80% * 85% + 100% * 5% = 73%
False positives: Legitimate workers appear to be spammers
38. Solution: Reverse errors first, compute
f d
error rate afterwards
Error Rates for CEO of AdSafe
P[G → G]=20 0%
G]=20.0% P[G → P]=80 0%
P]=80.0% P[G → R]=0 0%
R]=0.0% P[G → X]=0 0%
X]=0.0%
P[P → G]=0.0% P[P → P]=0.0% P[P → R]=100.0% P[P → X]=0.0%
P[R → G]=0.0% P[R → P]=0.0% P[R → R]=100.0% P[R → X]=0.0%
P[X → G]=0.0%
[ ] P[X → P]=0.0%
[ ] P[X → R]=0.0%
[ ] P[X → X]=100.0%
[ ]
• When biased worker says G, it is 100% G
• When biased worker says P, it is 100% G
Wh bi d k P it i 100% G
• When biased worker says R, it is 50% P, 50% R
• When biased worker says X, it is 100% X
Small ambiguity for “R‐rated” votes but other than that, fine!
39. Solution: Reverse errors first, compute
f d
error rate afterwards
Error Rates for spammer: ATAMRO447HWJQ
P[G → G]=100.0% P[G → P]=0.0% P[G → R]=0.0% P[G → X]=0.0%
P[P → G]=100.0% P[P → P]=0.0% P[P → R]=0.0% P[P → X]=0.0%
P[R → G]=100.0% P[R → P]=0.0% P[R → R]=0.0% P[R → X]=0.0%
P[X → G]=100 0%
G]=100.0% P[X → P]=0 0%
P]=0.0% P[X → R]=0 0%
R]=0.0% P[X → X]=0 0%
X]=0.0%
• When spammer says G, it is 25% G, 25% P, 25% R, 25% X
• When spammer says P, it is 25% G, 25% P, 25% R, 25% X
• When spammer says R, it is 25% G, 25% P, 25% R, 25% X
• When spammer says X, it is 25% G, 25% P, 25% R, 25% X
[note: assume equal priors]
The results are highly ambiguous. No information provided!
h l h hl b f
41. Quality Score
• A spammer is a worker who always assigns labels randomly,
regardless of what the true class is.
QualityScore = 1 ‐ ExpCost(Worker)/ExpCost(Spammer)
• Q li S
QualityScore i
is useful for the purpose of blocking bad workers and
f lf h f bl ki b d k d
rewarding good ones
• Essentially a multi class cost sensitive AUC metric
Essentially a multi‐class, cost‐sensitive AUC metric
• AUC = area under the ROC curve
42. What about Gold testing?
Naturally integrated into the latent class model
1. (e.g.,
1 Initialize by aggregating labels for each object (e g use majority vote)
2. Estimate error rates for workers (using aggregate labels)
3. Estimate aggregate labels (using error rates, weight worker
votes according to quality)
• Keep labels for “gold data” unchanged
4 Go to Step 2 and iterate until convergence
4.
43. • 3 labels per example
• 2 categories, 50/50
Gold Testing
Gold Testing •
•
Quality range: 0.55:0.05:1.0
200 labelers
l b l
No significant advantage under “good conditions”
(balanced datasets, good worker quality)
http://bit.ly/gold‐or‐repeated
Wang, Ipeirotis, Provost, WCBI 2011
44. • 5 labels per example
• 2 categories, 50/50
Gold Testing
Gold Testing •
•
Quality range: 0.55:1.0
200 labelers
l b l
No significant advantage under “good conditions”
(balanced datasets, good worker quality)
45. • 10 labels per example
• 2 categories, 50/50
Gold Testing
Gold Testing •
•
Quality range: 0.55:1.0
200 labelers
l b l
No significant advantage under “good conditions”
(balanced datasets, good worker quality)
46. • 10 labels per example
• 2 categories, 90/10
Gold Testing
Gold Testing •
•
Quality range: 0.55:0.1.0
l b l
200 labelers
Advantage under imbalanced datasets
47. • 5 labels per example
• 2 categories, 50/50
Gold Testing
Gold Testing •
•
Quality range: 0.55:0.65
200 labelers
l b l
Advantage with bad worker quality
48. • 10 labels per example
• 2 categories, 90/10
Gold Testing?
Gold Testing? •
•
Quality range: 0.55:0.65
200 labelers
l b l
Significant advantage under “bad conditions”
(imbalanced datasets, bad worker quality)
49. Testing workers
Testing workers
• An exploration‐exploitation scheme:
An exploration exploitation
– Explore: Learn about the quality of the workers
– Exploit: Label new examples using the quality
Exploit: Label new examples using the quality
50. Testing workers
Testing workers
• An exploration‐exploitation scheme:
An exploration exploitation
– Assign gold labels when benefit in learning better
quality of worker outweighs the loss for labeling a
quality of worker outweighs the loss for labeling a
gold (known label) example [Wang et al, WCBI 2011]
– Assign an already labeled example (by other
Assign an already labeled example (by other
workers) and see if it agrees with majority [Donmez et
al., KDD 2009]
– If worker quality changes over time, assume
f k l h
accuracy given by HMM and φ(τ) = φ(τ‐1) + Δ
[
[Donmez et al., SDM 2010]
, ]
51. Example: Build an “Adult Web Site” Classifier
Example: Build an Adult Web Site
Get people to look at sites and classify them as:
p p y
G (general audience) PG (parental guidance) R (restricted) X (porn)
But we are not going to label the whole Internet…
Expensive
Slow
52. Integrating with Machine Learning
Integrating with Machine Learning
• Crowdsourcing is cheap but not free
Crowdsourcing is cheap but not free
– Cannot scale to web without help
• Solution: Build automatic classification models
using crowdsourced data
53. Simple solution
• Humans label training data
Humans label training data
• Use training data to build model
Data from existing
crowdsourced answers
N
New C
Case Automatic Model Automatic
(through machine learning) Answer
54. Quality and Classification Performance
Noisy labels lead to degraded task performance
Labeling quality increases classification quality increases
Labeling quality increases classification quality increases
Quality = 100%
100
Quality 80%
Quality = 80%
90
80
AUC
Quality = 60%
70
60
Quality = 50%
50
40
100
120
140
160
180
200
220
240
260
280
300
1
20
40
60
80
Number of examples ("Mushroom" data set) Single‐labeler quality
(p
(probability of assigning
y g 54 g
correctly a binary label)
http://bit.ly/gold‐or‐repeated
Sheng, Provost, Ipeirotis, KDD 2008
55. Tradeoffs for Machine Learning Models
• Get more data Improve model accuracy
• Improve data quality Improve classification
p q y p
Data Quality = 100%
100
Data Quality = 80%
Q li 80%
90
80
uracy
70 Data Quality = 60%
Q y
Accu
60
50 Data Quality = 50%
40
0
0
0
0
0
0
0
0
0
0
0
1
20
40
60
80
10
12
14
16
18
20
22
24
26
28
30
Number of examples (Mushroom)
55
56. Tradeoffs for Machine Learning Models
• Get more data: Active Learning, select which
unlabeled example to label [Settles, http://active‐learning.net/]
unlabeled example to label [S ttl htt // ti l i t/]
• Impro e data q alit
Improve data quality:
Repeated Labeling, label again an already labeled
example [Sheng et al. 2008, Ipeirotis et al, 2010]
example [Sheng et al 2008 Ipeirotis et al 2010]
56
58. Rule of Thumb Results
Rule of Thumb Results
• With high quality labelers (80% and above): One
g q y ( )
worker per case (more data better)
• With low quality labelers (~60%)
Multiple workers per case (to improve quality)
Multiple workers per case (to improve quality)
[Sheng et al KDD 2008; Kumar and Lease CSDM 2011]
et al, KDD 2008; Kumar and Lease, CSDM 2011]
58
59. Dawid & Skene meets a Classifier
& Skene meets a Classifier
• [Raykar et al. JMLR 2010]: Use the
et al. JMLR 2010]: Use the
Dawid&Skene scheme but add a classifier as
an additional worker
an additional worker
• Classifier in each iteration learns from the
consensus labeling
59
60. Selective Repeated Labeling
Selective Repeated‐Labeling
• We do not need to label everything same number of times
• Key observation: we have additional information to guide
selection of data for repeated labeling
p g
the current multiset of labels
• Example: {+ ‐ + ‐ ‐ +} vs {+ + + + + +}
Example: {+,‐,+,‐,‐,+} vs. {+,+,+,+,+,+}
60
62. + + ‐ ‐‐ ‐
+ + + +
+ + ‐ ‐‐ ‐ ‐ + ‐ ‐
‐
Model Uncertainty (MU) + +
+ +
+ +
++
+ +‐ ‐ ‐ ‐‐ ‐ ‐
+ +
+ + ‐‐‐‐ ‐‐‐‐
‐‐
‐‐
• Learning models of the data provides an
alternative source of information about label
certainty
• M d l
Model uncertainty: get more labels for instances
t i t t l b l f i t Examples
that cause model uncertainty
• Intuition?
Models
– for modeling: why improve training data quality if “Self‐healing” process
model already is certain there? [Brodley et al, JAIR 1999]
[Ipeirotis et al NYU 2010]
et al, NYU 2010]
– for data quality, low‐certainty “regions” may be due to
incorrect labeling of corresponding instances
62
64. Too much theory?
Too much theory?
Open source implementation available at:
p p
http://code.google.com/p/get‐another‐label/
• Input:
– Labels from Mechanical Turk
– Cost of incorrect labelings (e.g., XG costlier than GX)
• Output:
Output:
– Corrected labels
– Worker error rates
– Ranking of workers according to their quality
65. Learning from imperfect data
Learning from imperfect data
100
• With inherently noisy
With inherently noisy 90
80
Accuracy
data, good to have 70
60
learning algorithms that
learning algorithms that 50
are robust to noise. 40
0
0
0
0
0
0
0
0
0
0
0
1
20
40
60
80
18
20
22
24
26
28
30
10
12
14
16
Number of examples (Mushroom)
• Or use techniques
designed to handle
d i d h dl
explicitly noisy data
[Lugosi 1992; Smyth, 1995, 1996]
66. Outline
• Introduction: Human computation and crowdsourcing
• Managing quality for simple tasks
• Complex tasks using workflows
• Task optimization
• Incentivizing the crowd
Incentivizing the crowd
• Market design
• Behavioral aspects and cognitive biases
Behavioral aspects and cognitive biases
• Game design
• Case studies
d
67. How to handle free‐form answers?
• Q: “My task does not have discrete answers….”
• A: Break into two HITs:
– “C t ” HIT
“Create”
– “Vote” HIT
Creation HIT Voting HIT:
(e.g. find a URL about a topic) Correct or not?
• Vote HIT controls quality of Creation HIT
Vote HIT controls quality of Creation HIT
• Redundancy controls quality of Voting HIT
• Catch: If “creation” very good, in voting workers just vote “yes”
– Solution: Add some random noise (e.g. add typos)
Example: Collect URLs
68. But my free‐form is Describe this
j g g
not just right or wrong…
• “Create” HIT
• “Improve” HIT
p
• “Compare” HIT
Creation HIT
( g
(e.g. describe the image)
g )
Improve HIT Compare HIT (voting)
(e.g. improve description) Which is better?
TurkIt toolkit [Little et al., UIST 2010]: http://groups.csail.mit.edu/uid/turkit/
69. version 1:
A parial view of a pocket calculator together with
some coins and a pen.
version 2:
version 2:
A view of personal items a calculator, and some gold and
copper coins, and a round tip pen, these are all pocket
and wallet sized item used for business, writting, calculating
prices or solving math problems and purchasing items.
version 3:
A close‐up photograph of the following items: A CASIO
multi‐function calculator. A ball point pen, uncapped.
Various coins, apparently European, both copper and gold.
Seems to be a theme illustration for a brochure or document
b h ll f b h d
cover treating finance, probably personal finance.
version 4:
…Various British coins; two of £1 value, three of 20p value
and one of 1p value. …
and one of 1p value
version 8:
“A close‐up photograph of the following items: A
CASIO multi‐function, solar powered scientific
calculator. A blue ball point pen with a blue rubber
grip and the tip extended. Six British coins; two of £1
value, three of 20p value and one of 1p value. Seems
to be a theme illustration for a brochure or
document cover treating finance ‐ probably personal
finance."
70. Independence or Not?
Independence or Not?
• Building iteratively (lack of independent) allows better
outcomes for image description task…
• In the FoldIt game workers built on each other’s results
In the FoldIt game, workers built on each other s results
[Little et al, HCOMP 2010]
71. Independence or Not?
• But lack of independence
But lack of independence
may cause high
dependence on starting
conditions and create
conditions and create
groupthink
p
• …but also prevents
disasters
[Little et al, HCOMP 2010]
72. Independence or Not?
Collective Problem Solving
• Exploration / exploitation tradeoff
– Can accelerate learning, by sharing good solutions
– But can lead to premature convergence on
suboptimal solution
[Mason and Watts, submitted to Science, 2011]
73.
74.
75.
76.
77. Individual search strategy affects group success
Individual search strategy affects group success
• More players copying
More players copying
each other (i.e., fewer
exploring) in current
round
Lower probability of
finding peak on next
roundd
88. Individual search strategy affects group success
Individual search strategy affects group success
• No significant
No significant
differences in % of
games in which peak
was found
• Network affects
willingness to explore
90. TurKontrol: Decision Theoretic Modeling
TurKontrol: Decision‐Theoretic Modeling
• Optimizing workflow execution using decision
Optimizing workflow execution using decision‐
theoretic approaches [Dai et al, AAAI 2010; Kern et al. 2010]
• Si ifi
Significant work in control theory [Montgomery, 2007]
ki l h
93. Find “Identify at least one area
t at can
that ca be s o te ed
shortened
without changing the
meaning of the
paragraph.”
Independent agreement to identify patches
Fix “Edit the highlighted
g g
section to shorten its
length without changing Soylent, a prototype...
the meaning
of the paragraph.”
Randomize order of suggestions
Verify “Choose at least one
rewrite that has style
errors, and
at least one rewrite that
changes the meaning
of the sentence.”
94. Crowd‐created Workflows: CrowdForge
• Map‐Reduce framework for crowds [Kittur et al, CHI
2011]
– Identify sights worth checking
out (
(one tip per worker))
• Vote and rank
– Brief tips for each monument
(
(one tip p worker)
p per )
• Vote and rank
– Aggregate tips in meaningful
summary
• It t to improve…
Iterate t i
My Boss is a Robot (mybossisarobot.com), Nikki Kittur (CMU) + Jim Giles (New Scientist)
95. Crowd‐created Workflows: TurkoMatic
• Crowd creates workflows
• Turkomatic [Kalkani et al, CHI 2011]:
1. Ask workers to decompose task into steps (Map)
2. Can step be completed within 10 minutes?
1. Yes: solve it.
2.
2 No: decompose further (recursion)
3. Given all partial solutions, solve big problem (Reduce)
96. Crowdsourcing Patterns
Crowdsourcing Patterns
• Generate / Create
i d
• Find Creation
C ti
• Improve / Edit / Fix
• Vote for accept‐reject
Q
Quality
y
• Vote up, vote down, to generate rank
Control
• Vote for best / select top‐k
• Split task
• Aggregate Flow Control
Flow Control
97. Outline
• Introduction: Human computation and crowdsourcing
• Managing quality for simple tasks
• Complex tasks using workflows
• Task optimization
• Incentivizing the crowd
Incentivizing the crowd
• Market design
• Behavioral aspects and cognitive biases
Behavioral aspects and cognitive biases
• Game design
• Case studies
d
98. Defining Task Parameters
Defining Task Parameters
Three main goals:
Three main goals:
• Minimize Cost (cheap)
i i i C ( h )
• Maximize Quality (good)
• Minimize Completion Time (fast)
99. Effect of Payment: Quality
Effect of Payment: Quality
• Cost does not affect quality [Mason and Watts, 2009, AdSafe]
• Similar results for bigger tasks [Ariely et al, 2009]
0.45
0.40
0.35
0.30
Error Rate
0.25 2cents
5cents
0.20 10cents
0.15
0
0.10
0.05
0.00
0 10 20 30
Number of Labelers
100. Effect of Payment: #Tasks
Effect of Payment: #Tasks
• Payment incentives increase speed, though
[Mason and Watts, 2009]
101. Predicting Completion Time
• Model timing of individual task
[Yan, Kumar, Ganesan, 2010]
– Assume rate of task completion λ
– Exponential distribution for
single task
g
– Erlang distribution for sequential
tasks
– On the fly estimation of λ for
On‐the‐fly estimation of λ for
parallel
• Optimize using early
acceptance/termination
– Sequential experiment setting
– Stop early if confident
Stop early if confident
102. Prediction Completion Time
Prediction Completion Time
• For Freebase, workers use log‐normal time to
complete a task [Kochhar et al, HCOMP 2010]
104. Effect of #HITs: Monotonic, but sublinear
h(t) = 0.998^#HITs
• 10 HITs 2% slower than 1 HIT
• 100 HITs 19% slower than 1 HIT
• 1000 HITs 87% slower than 1 HIT
1000 HITs 87% slower than 1 HIT
or, 1 group of 1000 7 times faster than 1000 sequential groups of 1
[Wang et al, CSDM 2011]
105. HIT Topics
topic 1 : cw castingwords podcast transcribe english mp3 edit confirm snippet grade
topic 2: d
i 2 data collection search image entry listings website review survey opinion
ll i h i li i bi i i i
topic 3: categorization product video page smartsheet web comment website opinion
topic 4: easy quick survey money research fast simple form answers link
topic 5: question answer nanonano dinkle article write writing review blog articles
topic 6: writing answer article question opinion short advice editing rewriting paul
topic 7: transcribe transcription improve retranscribe edit answerly voicemail answer
[Wang et al, CSDM 2011]
106. Effect of Topic: The CastingWords Effect
topic 1 : cw castingwords podcast transcribe english mp3 edit confirm snippet grade
topic 2: data collection search image entry listings website review survey opinion
topic 3: categorization product video page smartsheet web comment website opinion
topic 4: easy quick survey money research fast simple form answers link
topic 5: question answer nanonano dinkle article write writing review blog articles
p q g g
topic 6: writing answer article question opinion short advice editing rewriting paul
topic 7: transcribe transcription improve retranscribe edit answerly voicemail query question answer
[Wang et al, CSDM 2011]
107. Effect of Topic: Surveys=fast (even with redundancy!)
topic 1 : cw castingwords podcast transcribe english mp3 edit confirm snippet grade
topic 2: data collection search image entry listings website review survey opinion
topic 3: categorization product video page smartsheet web comment website opinion
topic 4: easy quick survey money research fast simple form answers link
topic 5: question answer nanonano dinkle article write writing review blog articles
p q g g
topic 6: writing answer article question opinion short advice editing rewriting paul
topic 7: transcribe transcription improve retranscribe edit answerly voicemail query question answer
[Wang et al, CSDM 2011]
108. Effect of Topic: Writing takes time
topic 1 : cw castingwords podcast transcribe english mp3 edit confirm snippet grade
topic 2: data collection search image entry listings website review survey opinion
topic 3: categorization product video page smartsheet web comment website opinion
topic 4: easy quick survey money research fast simple form answers link
topic 5: question answer nanonano dinkle article write writing review blog articles
p q g g
topic 6: writing answer article question opinion short advice editing rewriting paul
topic 7: transcribe transcription improve retranscribe edit answerly voicemail query question answer
[Wang et al, CSDM 2011]
109. Optimizing Completion Time
Optimizing Completion Time
• Workers pick tasks that have large number of
Workers pick tasks that have large number of
HITs or are recent [Chilton et al., HCOMP 2010]
• VizWiz optimizations [[Bingham, UIST 2011] :
optimizations i h S 20 ]
– Posts HITs continuously (to be recent)
– Mk bi HIT
Mkes big HIT groups (to be large)
( b l )
– HITs are “external HITs” (i.e., IFRAME hosted)
– HITs populated when the worker accepts them
110. Optimizing Completion Time
Optimizing Completion Time
• Completion rate varies with
Completion rate varies with
time of day, depending on
the audience location (India
the audience location (India
vs US vs Middle East)
• Quality tends to remain the
same, independent of
same independent of
completion time
[Huang et al., HCOMP 2010]
[Huang et al., HCOMP 2010]
111. Other Optimizations
Other Optimizations
• Qurk [Markus et al., CIDR 2011] and CrowdDB [Franklin et al., SIGMOD 2011]:
[ ]
Treat humans as uncertain UDFs + apply relational
optimization, plus the “GoodEnough” and “StopAfter”
operator.
operator
• CrowdFlow [Quinn et al ]: Integrate crowd with machine
[Quinn et al.]: Integrate crowd with machine
learning to reach balance of speed, quality, cost
• Ask humans for directions in a graph: [Parameswaran et
al., VLDB 2011]. See also [Kleinberg, Nature 2000;
Mitzenmacher, XRDS 2010; Deng, ECCV 2010]
Mitzenmacher XRDS 2010; Deng ECCV 2010]
112. Outline
• Introduction: Human computation and crowdsourcing
• Managing quality for simple tasks
• Complex tasks using workflows
• Task optimization
• Incentivizing the crowd
Incentivizing the crowd
• Market design
• Behavioral aspects and cognitive biases
Behavioral aspects and cognitive biases
• Game design
• Case studies
d
114. Incentives: Money
Incentives: Money
• Money does not improve quality but (generally)
Money does not improve quality but (generally)
increase participation [Ariely, 2009; Mason & Watts, 2009]
• But workers may be “target earners” (stop after
reaching their daily goal) [Horton & Chilton, 2010 for MTurk;
hi h i d il l)
Camerer et al. 1997, Farber 2008, for taxi drivers; Fehr and Goette 2007]
115. Incentives: Money and Trouble
Incentives: Money and Trouble
• Careful: Paying a little often worse than paying
y g p y g
nothing!
– “Pay enough or not at all” [Gneezy et al, 2000]
– Small pay now locks future pay
Small pay now locks future pay
– Payment replaces internal motivation (paying kids to collect
donations decreased enthusiasm; spam classification; “thanks for
dinner, here is $100 )
dinner here is $100”)
– Lesson: Be the Tom Sawyer (“how I like painting the
fence”), not the scrooge‐y boss…
• Paying a lot is a counter‐incentive:
– People focus on the reward and not on the task
People focus on the reward and not on the task
– On MTurk spammers routinely attack highly‐paying tasks
117. Incentives: Leaderboards
Incentives: Leaderboards
• Leaderboards (“top participants”) frequent
( top participants ) frequent
motivator
– Should motivate correct behavior not just
Should motivate correct behavior, not just
measurable behavior
– Newcomers should have hope of reaching top
Newcomers should have hope of reaching top
– Whatever is measured, workers will optimize for
this (e.g., Orkut country leaderboard; complaints for quality score drops)
this (e.g., Orkut country leaderboard; complaints for quality score drops)
– Design guideline: Christmas‐tree dashboard (Green / Red lights only)
[Farmer and Glass, 2010]
118. Incentives: Purpose of Work
Incentives: Purpose of Work
• Contrafreeloading: Rats and animals prefer to
Contrafreeloading: Rats and animals prefer to
“earn” their food
• Destroying work after production demotivates
workers. [Ariely et al, 2008]
k [A i l l 2008]
• Showing result of “completed task” improves
satisfaction
119. Incentives: Purpose of Work
Incentives: Purpose of Work
• Workers enjoy learning new skills (oft cited reason for
j y g (
Mturk participation)
• Design tasks to be educational
– DuoLingo: Translate while learning new language [von Ahn et al,
duolingo.com]
– Galaxy Zoo, Clickworkers: Classify astronomical objects
[Raddick et al, 2010; http://en.wikipedia.org/wiki/Clickworkers]
– Citizen Science: Learn about biology gy
[http://www.birds.cornell.edu/citsci/]
– National Geographic “Field Expedition: Mongolia”, tag
potential archeological sites, learn about archeology
potential archeological sites, learn about archeology
120. Incentives: Credit and Participation
Incentives: Credit and Participation
• Public credit contributes to sense of
Public credit contributes to sense of
participation
• Credit also a form of reputation
Credit also a form of reputation
• (Anonymity of MTurk‐like settings discourage this factor)
122. Incentive: Altruism
Incentive: Altruism
• Contributing back (tit for tat): Early reviewers
Contributing back (tit for tat): Early reviewers
writing reviews because read other useful
review
• Effect amplified in social networks: “If all my
Effect amplified in social networks: If all my
friends do it…” or “Since all my friends will see
this…”
• Contributing to shared goal
Contributing to shared goal
123. Incentives: Altruism and Purpose
Incentives: Altruism and Purpose
• On MTurk [Chandler and Kapelner 2010]
On MTurk [Chandler and Kapelner, 2010]
– Americans [older, more leisure‐driven] work
harder for meaningful work
harder for “meaningful work”
– Indians [more income‐driven] were not affected
– Quality unchanged for both groups
Quality unchanged for both groups
124. Incentives: Fair share
Incentives: Fair share
• Anecdote: Same HIT (spam classification)
Anecdote: Same HIT (spam classification)
– Case 1: Requester doing as side‐project, to “clean
the market would be out‐of‐pocket expense no
the market”, would be out of pocket expense, no
pay to workers
– Case 2: Requester researcher at university, spam
Case 2: Requester researcher at university, spam
classification now a university research project,
payment to workers
What setting worked best?
125. Incentives: FUN!
Incentives: FUN!
• Game‐ify the task (design details later)
• Examples
p
– ESP Game: Given an image, type the same
word (generated image descriptions)
– Phylo: aligned color blocks (used for genome
Phylo: aligned color blocks (used for genome
alignment)
– FoldIt: fold structures to optimize energy
(protein folding)
(protein folding)
• Fun factors [Malone 1980, 1982]:
– ti d
timed response
– score keeping
– player skill level
– high‐score lists
– and randomness
126. Outline
• Introduction: Human computation and crowdsourcing
• Managing quality for simple tasks
• Complex tasks using workflows
• Task optimization
• Incentivizing the crowd
Incentivizing the crowd
• Market design
• Behavioral aspects and cognitive biases
Behavioral aspects and cognitive biases
• Game design
• Case studies
d
127. Market Design Organizes the Crowd
Market Design Organizes the Crowd
• Reputation Mechanisms
eputat o ec a s s
– Seller‐side: Ensure worker quality
– Buy‐side: Ensure employee trustworthiness
• Task organization for task discovery (worker finds
employer/task)
/ )
• Worker expertise recording for task assignment
(employer/task finds worker)
128. Lack of Reputation and Market for Lemons
Lack of Reputation and Market for Lemons
• “When quality of sold good is uncertain and hidden before
transaction, prize goes to value of lowest valued good
transaction, prize goes to value of lowest valued good”
[Akerlof, 1970; Nobel prize winner]
Market evolution steps:
1. Employers pays $10 to good worker, $0.1 to bad worker
2. 50% good workers, 50% bad; indistinguishable from each other
2 50% d k 50% b d d h bl f h h
3. Employer offers price in the middle: $5
4. Some good workers leave the market (pay too low)
4 Some good workers leave the market (pay too low)
5. Employer revised prices downwards as % of bad increased
6. More good workers leave the market… death spiral
g p
http://en.wikipedia.org/wiki/The_Market_for_Lemons
129. Lack of Reputation and Market for Lemons
Lack of Reputation and Market for Lemons
• Market for lemons also on the employer side:
– Workers distrust (good) newcomer employers: Charge risk premium,
or work only for little bit. Good newcomers get disappointed
– Bad newcomers have no downside (will not pay), continue to offer
work.
work
– Market floods with bad employers
• TurkOpticon, external reputation system
• “Mechanical Turk: Now with 40.92% spam” http://bit.ly/ew6vg4
• Gresham's Law: the bad drives out the good
• No‐trade equilibrium: no good employer offers work in a
No trade equilibrium: no good employer offers work in a
market with bad workers, no good worker wants to work for
bad employers…
• In reality, we need to take into consideration that this is a
In reality, we need to take into consideration that this is a
repeated game (but participation follows a heavy tail…)
http://en.wikipedia.org/wiki/The_Market_for_Lemons
130. Reputation systems
Reputation systems
• Significant number of reputation mechanisms
Significant number of reputation mechanisms
[Dellarocas et al, 2007]
• Link analysis techniques [TrustRank, EigenTrust,
NodeRanking, NetProbe, Snare] often applicable
f li bl
131. Challenges in the Design of Reputation Systems
• Insufficient participation
p p
• Overwhelmingly positive feedback
• Dishonest reports
• Identity changes
Identity changes
• Value imbalance exploitation (“milking the
reputation”)
132. Insufficient Participation
• Free‐riding: feedback constitutes a public good. Once available,
everyone can costless‐ly benefit from it.
• Disadvantage of early evaluators: provision of feedback
p
presupposes that the rater will assume the risks of transacting with
pp g
the ratee (competitive advantage to others).
• [A
[Avery et al. 1999] propose a mechanism whereby early
l 1999] h i h b l
evaluators are paid to provide information and later evaluators
pay to balance the budget.
133. Overwhelmingly Positive Feedback (I)
More than 99% of all feedbacks posted on eBay are positive.
H I i d f 16% f ll
However, Internet auctions accounted for 16% of all consumer
fraud complaints received by the Federal Trade Commission in
2004. (http://www.consumer .gov/sentinel/)
Reporting Bias
The perils of reciprocity:
• Reciprocity: Seller evaluated buyer, buyer evaluates seller
• Exchange of courtesies
• Positive reciprocity: positive ratings are given in the hope
of getting a positive rating in return
• Negative reciprocity: negative ratings are avoided because
of fear of retaliation from the other party
134. Overwhelmingly Positive Feedback (II)
“The sound of silence”: No news, bad news…
• [Dellarocas and Wood 2008] Explore the frequency of
different feedback patterns and use the non‐reports to
compensate for reporting bias.
• eBay traders are more likely to post feedback when satisfied
than when dissatisfied
than when dissatisfied
• Support presence of positive and negative reciprocation
among eBay traders.
135. Dishonest Reports
• “Ballot stuffing” (unfairly high ratings): a seller colludes with a
group of buyers in order to be given unfairly high ratings by them.
group of buyers in order to be given unfairly high ratings by them
• “Bad‐mouthing” (unfairly low ratings): Sellers can collude with
buyers in order to “bad‐mouth” other sellers that they want to drive
y y
out the market.
• Design incentive‐compatible mechanism to elicit honest feedbacks
[
[Jurca and Faltings 2003: pay rater if report matches next;
g p y p ;
Miller et al. 2005: use a proper scoring rule to price value of report;
Papaioannou and Stamoulis 2005: delay next transaction over time]
• U “l t t l ”
Use “latent class” models described earlier in the tutorial
d l d ib d li i th t t i l
(reputation systems is a form of crowdsourcing after all…)
136. Identity Changes
• “Cheap pseudonyms”: easy to disappear and re‐
pp y y pp
register under a new identity with almost zero cost.
[Friedman and Resnick 2001]
• I
Introduce opportunities to misbehave without paying
d ii ib h ih i
reputational consequences.
• Increase the difficulty of online identity changes
• Impose upfront costs to new entrants: allow new identities
(forget the past) but make it costly to create them
137. Value Imbalance Exploitation
Three men attempt to sell a fake painting on eBay for $US
135,805. The sale was abandoned just prior to purchase when
135 805 Th l b d dj i h h
the buyer became suspicious.(http://news.cnet.com/2100‐
1017‐253848.html)
• Reputation can be seen as an asset not only to
Reputation can be seen as an asset, not only to
promote oneself, but also as something that can be
cashed in through a fraudulent transaction with high
g g
gain.
“The Market for Evaluations”
138. The Market for Positive Feedbacks
A selling strategy that eBay users are actually using the
g gy y y g
feedback market for gains in other markets.
“Riddle for a PENNY! No shipping‐Positive Feedback”
Riddle for a PENNY! No shipping Positive Feedback
• 29‐cent loss even in the event of a successful sale
• Price low, speed feedback accumulation
Possible solutions:
• Make the details of the transaction (besides the feedback itself) visible to other users
• T
Transaction‐weighted reputational statistics
ti i ht d t ti l t ti ti
[Brown 2006]
139. Challenges for Crowdsourcing Markets (I)
• Two‐sided opportunistic behavior
• Reciprocal systems worse than one‐side evaluation. In e‐commerce
markets, only sellers are likely to behave opportunistically. No need for
markets only sellers are likely to behave opportunistically No need for
reciprocal evaluation!
• In crowdsourcing markets, both sides can be fraudulent. Reciprocal
systems are fraught with problems, though!
• I
Imperfect monitoring and heavy‐tailed participation
f t it i dh t il d ti i ti
• In e‐commerce markets, buyers can assess the product quality directly
upon receiving.
• In crowdsourcing markets, verifying the answers is sometimes as costly as
providing them.
• Sampling often does not work due to heavy tailed participation
Sampling often does not work, due to heavy tailed participation
distribution (lognormal, according to self‐reported surveys)
140. Challenges for Crowdsourcing Markets (II)
• Constrained capacity of workers
• In e‐commerce markets sellers usually have unlimited supply of
In e commerce markets, sellers usually have unlimited supply of
products.
• In crowdsourcing, workers have constrained capacity (cannot be
recommended continuously)
• No “price premium” for high quality workers
No “price premium” for high‐quality workers
• In e‐commerce markets, sellers with high reputation can sell their
products at a relatively high price (premium).
• In crowdsourcing, it is the requester who set the prices, which are
generally the same for all the workers.
141. Market Design Organizes the Crowd
Market Design Organizes the Crowd
• Reputation Mechanisms
eputat o ec a s s
– Seller‐side: Ensure worker quality
– Buy‐side: Ensure employee trustworthiness
• Task organization for task discovery (worker finds
employer/task)
/ )
• Worker expertise recording for task assignment
(employer/task finds worker)
143. The Importance and Danger of Priorities
The Importance and Danger of Priorities
• [Barabasi, Nature 2005] showed that human actions
[ , ]
have power‐law completion times
– Mainly result of prioritization
– Wh t k
When tasks ranked by priorities, power‐law results
k db i iti l lt
• [Cobham 1954] If queuing system completes tasks
[Cobham, 1954] If queuing system completes tasks
with two priority queues, and λ=μ, then power‐law
completion times
• [Chilton et al., HCOMP 2010] Workers on Turk pick
tasks from most HITs or most recent queues
tasks from “most HITs” or “most recent” queues
144. The UI hurts the market!
• Practitioners know that HITs in 3rd page and after,
p g ,
are not picked by workers.
• Many such HITs are left to expire after months
Many such HITs are left to expire after months,
never completed.
• Badly designed task discovery interface hurts every
participant in the market! (and the reason for scientific modeling…)
ti i t i th k t!
• Better modeling as a queuing system may
demonstrate other such improvements
145. Market Design Organizes the Crowd
Market Design Organizes the Crowd
• Reputation Mechanisms
eputat o ec a s s
– Seller‐side: Ensure worker quality
– Buy‐side: Ensure employee trustworthiness
• Task organization for task discovery (worker finds
employer/task)
/ )
• Worker expertise recording for task assignment
(employer/task finds worker)
146. Expert Search
Expert Search
• Find the best worker for a task (or within a task)
Find the best worker for a task, (or within a task)
• For a task:
k
– Significant amount of research in the topic of expert
search [TREC track; Macdonald and Ounis, 2006]
h
– Check quality of workers across tasks
http://url‐annotator.appspot.com/Admin/WorkersReport
http://url annotator appspot com/Admin/WorkersReport
• Within a task: [Donmez et al., 2009; Welinder, 2010]
t a tas : [ o e et a , 009; e de , 0 0]
147. Directions for future research
Directions for future research
• Optimize allocation of tasks to worker based on completion
time and expected quality
• Explicitly take into consideration competition in market and
Explicitly take into consideration competition in market and
switch task for worker only when benefit outweighs
switching overhead (task switching in CPU from O/S)
• Recommender system for tasks (“workers like you
performed well in…”)
• Create a market with dynamic pricing for tasks, following
the pricing model of the stock market (prices increase for
task when work supply low, and vice versa)
task when work supply low and vice versa)
148. Outline
• Introduction: Human computation and crowdsourcing
• Managing quality for simple tasks
• Complex tasks using workflows
• Task optimization
• Incentivizing the crowd
Incentivizing the crowd
• Market design
• Behavioral aspects and cognitive biases
Behavioral aspects and cognitive biases
• Game design
• Case studies
d
149. Human Computation
• Humans are not perfect mathematical models
Humans are not perfect mathematical models
• They exhibit noisy, stochastic behavior…
h hibi i h i b h i
• And exhibit common and systematic biases
150. Score the following from 1 to 10
1: not particularly bad or wrong
1 t ti l l b d
10: extremely evil
a) Stealing a towel from a hotel
b) Keeping a dime you find on the ground
p g y g
c) Poisoning a barking dog
[Parducci, 1968]
151. Score the following from 1 to 10
1: not particularly bad or wrong
1 t ti l l b d
10: extremely evil
a) Testifying falsely for pay
gg g
b) Using guns on striking workers
c) Poisoning a barking dog
[Parducci, 1968]
152. Anchoring
Anchoring
• “Humans start with a first approximation (anchor) and
f pp ( )
then make adjustments to that number based on
additional information.” [Tversky & Kahneman, 1974]
• [Paolacci et al, 2010]
– Q1a: More or less than 65 African countries in UN?
Q1a: More or less than 65 African countries in UN?
– Q1b: More or less than 12 African countries in UN?
– Q2: How many countries in Africa?
– Group A mean: 42.6
– Group B mean: 18 5
Group B mean: 18.5
153. Anchoring
Anchoring
• Write down the last digit of their social security
g f y
number before placing bid for wine bottles. Users
with lower SSN numbers bid lower…
• In the Netflix contest, user with high ratings early
on, biased towards higher ratings later in a
on, biased towards higher ratings later in a
session…
• Crowdsourcing tasks can be affected by
anchoring. [Moren et al, NIPS 2010] describe
techniques for removing effects
techniques for removing effects
154. Priming
• Exposure to one stimulus influences another
Exposure to one stimulus influences another
• Stereotypes:
– Asian americans perform better in math
Asian‐americans perform better in math
– Women perform worse in math
• [Shih et al., 1999] asked Asian‐American women:
– Q ti
Questions about race: They did better in math test
b t Th did b tt i th t t
– Questions about gender: They did worse in math test
155. Exposure Effect
Exposure Effect
• Familiarity leads to liking
Familiarity leads to liking...
• [S
[Stone and Alonso, 2010]: Evaluators of Bing
d l 20 0] l f i
search engine increase their ratings of
relevance over time, for the same results
l i f h l
156. Framing
• Presenting the same option in different
Presenting the same option in different
formats leads to different formats. People
avert options that imply loss [Tversky and
avert options that imply loss [Tversky and
Kahneman (1981)]
157. Framing:
600 people affected by deadly disease
600 people affected by deadly disease
Room 1
a) save 200 people's lives
save 200 people s lives
b) 33% chance of saving all 600 people and a 66% chance saving no one
• 72% of participants chose option A
72% of participants chose option A
• 28% of participants chose option B
Room 2
Room 2
c) 400 people die
d) 33% chance that no people will die; a 66% chance that all 600 will die
• 78% of participants chose option D (equivalent to option B)
• 22% of participants chose option C (equivalent to option A)
People avert options that imply loss
158. Very long list of cognitive biases…
Very long list of cognitive biases…
• http://en.wikipedia.org/wiki/List_of_cognitive_biases
p // p g/ / g
• [Mozer et al., 2010] try to learn and remove sequential effects
from human computation data…
159. Outline
• Introduction: Human computation and crowdsourcing
• Managing quality for simple tasks
• Complex tasks using workflows
• Task optimization
• Incentivizing the crowd
Incentivizing the crowd
• Market design
• Behavioral aspects and cognitive biases
Behavioral aspects and cognitive biases
• Game design
• Case studies
d
160. Games with a Purpose
[Luis von Ahn and Laura Dabbish, CACM 2008]
Three generic game structures
• Output agreement:
– Type same output
• I
Input agreement:
– Decide if having same input
• Inversion problem:
Inversion problem:
– P1 generates output from input
– P2 looks at P1‐output and guesses P1‐input
161. Output Agreement: ESP Game
Output Agreement: ESP Game
• Players look at common input
Players look at common input
• Need to agree on output
162. Improvements
• Game‐theoretic analysis indicates that players
Game theoretic analysis indicates that players
will converge to easy words [Jain and Parkes]
• Solution 1: Add “Taboo words” to prevent
Solution 1: Add Taboo words to prevent
guessing easy words
• S l i 2 Ki Ki B
Solution 2: KissKissBan, third player tries to
hi d l i
guess (and block) agreement
163. Input Agreement: TagATune
p g g
• Sometimes difficult to type identical output
(e.g., “describe this song”)
p ,
• Show same of different input, let users
describe, ask players if they have same input
164. Inversion Problem: Peekaboom
Inversion Problem: Peekaboom
• Non symmetric players
Non‐symmetric players
• Input: Image with word
• Player 1 slowly reveals pic
l l l l i
• Player 2 tries to guess word
171. Protein folding
Protein folding
• Protein folding: Proteins fold from long chains into
small balls, each in a very specific shape
• Shape is the lower energy setting which the most
Shape is the lower‐energy setting, which the most
stable
• Fold shape is very important to understand interactions
with out molecules
• Extremely expensive computationally! (too many
degrees of freedom)
172. FoldIt Game
• Humans are very good at reducing the search
Humans are very good at reducing the search
space
• Humans try to fold the protein into a minimal
energy state.
• Can leave protein unfinished and let others try
from there…
173. Outline
• Introduction: Human computation and crowdsourcing
• Managing quality for simple tasks
• Complex tasks using workflows
• Task optimization
• Incentivizing the crowd
Incentivizing the crowd
• Market design
• Behavioral aspects and cognitive biases
Behavioral aspects and cognitive biases
• Game design
• Case studies
d
180. A few of the tasks in the past
• Detect pages that discuss swine flu
– Pharmaceutical firm had drug “treating” (off-label) swine flu
– FDA prohibited pharmaceutical company to display drug ad in pages
about swine flu
– Two days to build and go live
• Big fast-food chain does not want ad to
appear:
– In pages that discuss the brand (99% negative sentiment)
– In pages discussing obesity
– Three days to build and go live
180
181. Need to build models fast
Need to build models fast
• T diti
Traditionally, modeling teams have invested substantial
ll d li t h i t d b t ti l
internal resources in data formulation, information
extraction, cleaning, and other preprocessing
No time for such things…
• However, now, we can outsource preprocessing tasks, such
as labeling, feature extraction, verifying information
l b li f t t ti if i i f ti
extraction, etc.
– using Mechanical Turk, oDesk, etc.
– quality may be lower than expert labeling (much?)
– but low costs can allow massive scale
18
1
182. AdSafe workflow
• Find URLs for a given topic (hate speech, gambling, alcohol
abuse, guns, bombs, celebrity gossip, etc etc)
b b b l bi i )
http://url‐collector.appspot.com/allTopics.jsp
• Classify URLs into appropriate categories
Classify URLs into appropriate categories
http://url‐annotator.appspot.com/AdminFiles/Categories.jsp
• Mesure quality of the labelers and remove spammers
http://qmturk.appspot.com/
htt // t k t /
• Get humans to “beat” the classifier by providing cases where
the classifier fails
http://adsafe‐beatthemachine.appspot.com/
18
2
184. Scaling Crowdsourcing: Use Machine Learning
Need to scale crowdsourcing
Basic idea: Build a machine learning model and use it
instead of humans
New case Automatic Model Automatic
(through machine learning) Answer
Existing data (through
crowdsourcing)
185. Scaling Crowdsourcing: Iterative training
Ti
Triage:
– machine when confident
– humans when not confident
Retrain using the new human input
Automatic
→ improve model
Answer
→ reduce need for human input
New Case Automatic Model
(through machine learning)
Data from existing Get human(s) to
answer
crowdsourced answers
186. Scaling Crowdsourcing: Iterative training, with noise
Machine when confident, humans otherwise
Ask as many humans as necessary to ensure quality
Automatic
Answer
New Case Automatic Model
Not confident
(through machine l
(th h hi learning)
i )
for quality?
Data from existing Get human(s) to
crowdsourced answers answer
Confident for quality?
187. Scaling Crowdsourcing: Iterative training, with noise
Machine when confident, humans otherwise
Ask as many humans as necessary to ensure quality
– Or even get other machines…
machines
Automatic
Answer
New Case Automatic Model
Not confident
(through machine l
(th h hi learning)
i )
about quality?
Data from existing Get human(s) or
crowdsourced answers other machines
Confident about quality? to answer
188. Example: ReCAPTCHA + Google Books
portion distinguished
Fixes errors of Optical Character Recognition (OCR ~ 1% error rate 20%
rate, 20%-
30% for 18th and 19th century books, according to today’s NY Times article)
Improves further the OCR algorithm, reducing error rate
“40 million R CAPTCHA every d ” (2008) Fi i 40 000 b k a day
illi ReCAPTCHAs day” Fixing 40,000 books d
– [Unofficial quote from Luis]: 400M/day (2010)
– All books ever written:100 million books (~12yrs??)