CrowdFlower University Oct. 21 2013
 

CrowdFlower University Oct. 21 2013

on

  • 958 views

On Monday, Oct. 21, CrowdFlower ran its first session of CrowdFlower University. The turn out was incredible. We had more than 30 people representing a range of organizations, from startups to Fortune ...

On Monday, Oct. 21, CrowdFlower ran its first session of CrowdFlower University. The turn out was incredible. We had more than 30 people representing a range of organizations, from startups to Fortune 100 companies. It was an engaging 4 hour session filled with great questions.

Statistics

Views

Total Views
958
Views on SlideShare
657
Embed Views
301

Actions

Likes
0
Downloads
8
Comments
0

6 Embeds 301

http://dashboard.skilljar.com 109
https://crowdflower.com 82
http://www.crowdflower.com 72
http://crowdflower.demo.skilljar.com 36
http://crowdflower.com 1
http://www.feedspot.com 1

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

CrowdFlower University Oct. 21 2013 CrowdFlower University Oct. 21 2013 Presentation Transcript

  • CrowdFlower University October 21, 2013
  • Schedule for the day 1:00 - 1:05 // Arrive, Plug-in 1:05 - 1:30 // Intro to Workforce Management and the CrowdFlower Platform 1:30 - 2:30 // Process Deconstruction & Job Creation 2:30 - 2:40 // Break 2:40 - 3:30 // Job Monitoring and Reports 3:30 - 4:00 // From the Field: Case Studies 4:00 - 4:10 // Break 4:10 - 4:30 // What’s to come: Product Roadmap 4:30 - 5:00 // Q&A 2 CrowdFlower, Inc. – Proprietary and Confidential
  • Purpose CrowdFlower University will teach you the CrowdFlower Platform, empowering you to manage workforces online. 3 CrowdFlower, Inc. – Proprietary and Confidential View slide
  • Success Center 4 CrowdFlower, Inc. – Proprietary and Confidential View slide
  • Success Center success.crowdflower.com 5 CrowdFlower, Inc. – Proprietary and Confidential
  • Workforce Management 6 CrowdFlower, Inc. – Proprietary and Confidential
  • Workforce Management Geography Performance Levels and Skills Workflows Test Questions Channels 7 CrowdFlower, Inc. – Proprietary and Confidential
  • Geography 8 CrowdFlower, Inc. – Proprietary and Confidential
  • Geography 9 CrowdFlower, Inc. – Proprietary and Confidential
  • Performance Levels and Skills 10 CrowdFlower, Inc. – Proprietary and Confidential
  • Skills and Performance Levels The CrowdFlower Platform enables you to add Skill requirements and Performance Levels to your job. Enabling the Skilled Crowds will likely increase the quality of your results and may decrease the throughput of your judgments. 11 CrowdFlower, Inc. – Proprietary and Confidential
  • Workflows 12 CrowdFlower, Inc. – Proprietary and Confidential
  • Workflows 13 CrowdFlower, Inc. – Proprietary and Confidential
  • Test Questions 14 CrowdFlower, Inc. – Proprietary and Confidential
  • Test Questions Test Questions are a powerful method for measuring the quality of the contributors in your job. By providing the answers to a small percentage of units, you can tell how accurate someone is, and automatically remove anyone who doesn't meet your requirements. 15 CrowdFlower, Inc. – Proprietary and Confidential
  • Channels 16 CrowdFlower, Inc. – Proprietary and Confidential
  • Channels 17 CrowdFlower, Inc. – Proprietary and Confidential
  • CrowdFlower Terminology 18 CrowdFlower, Inc. – Proprietary and Confidential
  • CrowdFlower Terminology •  •  Unit - One row on a spreadsheet (One company, one officer) •  Dataset - The entire set of data for a project •  Task - A subset of data from the whole dataset •  Workflow - Multiple jobs that strung together •  Channel - A website in which a contributor can access a task •  Test Questions - Training data created in the platform •  19 Judgment - One (of many) answers given by an individual contributor Contributor - Person completing assignments online CrowdFlower, Inc. – Proprietary and Confidential
  • Process Deconstruction and Job Creation 20 CrowdFlower, Inc. – Proprietary and Confidential
  • Process Deconstruction & Job Creation Let’s learn how to build a job! Today we’re going to build and run a sentiment analysis job about CrowdFlower. 21 CrowdFlower, Inc. – Proprietary and Confidential
  • Best Practices – Job Construction 22 CrowdFlower, Inc. – Proprietary and Confidential
  • Best Practices – Job Construction •  •  Leverage the Template Library •  Use the Instructions Template •  Help contributors find the data (provide links, images, etc) •  Provide examples in the instructions •  Leverage CML logic wherever possible •  Use validations to control text inputs •  23 Break down problem into smallest components Allow an "escape" option for multiple choice questions regarding information in the job or a link: "I don't know", "I couldn't find it", "the link is broken", etc. CrowdFlower, Inc. – Proprietary and Confidential
  • Best Practices – Test Questions 24 CrowdFlower, Inc. – Proprietary and Confidential
  • Best Practices – Test Questions •  •  Enable random data for Test Questions •  Copy units to source •  Be as thorough in your Test Question reasons as possible – these will help train the crowd throughout the task •  Create Test Questions out of edge case (subjective or tricky) units to train the crowd •  You don’t need to answer every question •  25 Try to create an even distribution of answers For subjective questions you can allow for multiple correct answers CrowdFlower, Inc. – Proprietary and Confidential
  • Best Practices – Job Calibration 26 CrowdFlower, Inc. – Proprietary and Confidential
  • Best Practices – Job Calibration •  •  Set variable judgments on trickier CML elements •  27 Set the Max Judgments Per IP at the number of units in an assignment multiplied by the amount of gold in the task •  This ensures that a contributor will not see the same gold unit twice Set threshold for trust of contributors CrowdFlower, Inc. – Proprietary and Confidential
  • Launch! 28 CrowdFlower, Inc. – Proprietary and Confidential
  • Task Monitoring and Reports 29 CrowdFlower, Inc. – Proprietary and Confidential
  • Task Monitoring and Reports Task Monitoring Interpreting Reports Interpreting Results *This session will teach you how to understand your results. 30 CrowdFlower, Inc. – Proprietary and Confidential
  • Task Monitoring 31 CrowdFlower, Inc. – Proprietary and Confidential
  • Task Monitoring • Review Test Questions • Monitor Contributors 32 CrowdFlower, Inc. – Proprietary and Confidential
  • Interpreting Reports 33 CrowdFlower, Inc. – Proprietary and Confidential
  • Interpreting Reports 34 CrowdFlower, Inc. – Proprietary and Confidential
  • Interpreting Results 35 CrowdFlower, Inc. – Proprietary and Confidential
  • Interpreting Results • Confidence Scores • Aggregation Methods • Auditing for Accuracy 36 CrowdFlower, Inc. – Proprietary and Confidential
  • Case Studies 37 CrowdFlower, Inc. – Proprietary and Confidential
  • Case Studies In this session we will cover several practical applications of the CrowdFlower Platform. 38 CrowdFlower, Inc. – Proprietary and Confidential
  • Search Relevance Testing 39 CrowdFlower, Inc. – Proprietary and Confidential
  • Example 1 40 CrowdFlower, Inc. – Proprietary and Confidential
  • Example 2 41 CrowdFlower, Inc. – Proprietary and Confidential
  • Master Data Management 42 CrowdFlower, Inc. – Proprietary and Confidential
  • Find URL 43 CrowdFlower, Inc. – Proprietary and Confidential
  • Find About Us Page 44 CrowdFlower, Inc. – Proprietary and Confidential
  • Categorize Businesses 45 CrowdFlower, Inc. – Proprietary and Confidential
  • De-Duplication 46 CrowdFlower, Inc. – Proprietary and Confidential
  • Content Categorization 47 CrowdFlower, Inc. – Proprietary and Confidential
  • Photo Categorization 48 CrowdFlower, Inc. – Proprietary and Confidential
  • Product Categorization 49 CrowdFlower, Inc. – Proprietary and Confidential
  • Content Moderation 50 CrowdFlower, Inc. – Proprietary and Confidential
  • Photo Moderation 51 CrowdFlower, Inc. – Proprietary and Confidential
  • Text Moderation 52 CrowdFlower, Inc. – Proprietary and Confidential
  • Image and Text Interpretation 53 CrowdFlower, Inc. – Proprietary and Confidential
  • Image Transcription 54 CrowdFlower, Inc. – Proprietary and Confidential
  • OCR Verification 55 CrowdFlower, Inc. – Proprietary and Confidential
  • Product Roadmap 56 CrowdFlower, Inc. – Proprietary and Confidential
  • Q4 2013 Automatic Test Question access for Enterprise customers only A wizard for first-time users so that your team needs less training A reputation score for our contributors and improvements to our performance levels 57 CrowdFlower, Inc. – Proprietary and Confidential
  • Q1 2014 In-platform templates for you to get started quickly Improvements to our GUI Team collaboration on jobs In-platform auditing tool so that your internal team and our strongest contributors can easily evaluate the output of your job 58 CrowdFlower, Inc. – Proprietary and Confidential
  • And beyond… New task analytics for improved slicing and dicing of your data Workflow building tool  Minimize human input (e.g. variable pay where you set the budget, we set the pay for contributors) 59 CrowdFlower, Inc. – Proprietary and Confidential
  • Questions? 60 CrowdFlower, Inc. – Proprietary and Confidential
  • Contact Me Nathan Zukoff Platform Program Manager nathan@crowdflower.com 973-997-1921 61 CrowdFlower, Inc. – Proprietary and Confidential