CrowdFlower University Oct. 21 2013

1,573 views

Published on

On Monday, Oct. 21, CrowdFlower ran its first session of CrowdFlower University. The turn out was incredible. We had more than 30 people representing a range of organizations, from startups to Fortune 100 companies. It was an engaging 4 hour session filled with great questions.

Published in: Education, Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,573
On SlideShare
0
From Embeds
0
Number of Embeds
494
Actions
Shares
0
Downloads
16
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

CrowdFlower University Oct. 21 2013

  1. 1. CrowdFlower University October 21, 2013
  2. 2. Schedule for the day 1:00 - 1:05 // Arrive, Plug-in 1:05 - 1:30 // Intro to Workforce Management and the CrowdFlower Platform 1:30 - 2:30 // Process Deconstruction & Job Creation 2:30 - 2:40 // Break 2:40 - 3:30 // Job Monitoring and Reports 3:30 - 4:00 // From the Field: Case Studies 4:00 - 4:10 // Break 4:10 - 4:30 // What’s to come: Product Roadmap 4:30 - 5:00 // Q&A 2 CrowdFlower, Inc. – Proprietary and Confidential
  3. 3. Purpose CrowdFlower University will teach you the CrowdFlower Platform, empowering you to manage workforces online. 3 CrowdFlower, Inc. – Proprietary and Confidential
  4. 4. Success Center 4 CrowdFlower, Inc. – Proprietary and Confidential
  5. 5. Success Center success.crowdflower.com 5 CrowdFlower, Inc. – Proprietary and Confidential
  6. 6. Workforce Management 6 CrowdFlower, Inc. – Proprietary and Confidential
  7. 7. Workforce Management Geography Performance Levels and Skills Workflows Test Questions Channels 7 CrowdFlower, Inc. – Proprietary and Confidential
  8. 8. Geography 8 CrowdFlower, Inc. – Proprietary and Confidential
  9. 9. Geography 9 CrowdFlower, Inc. – Proprietary and Confidential
  10. 10. Performance Levels and Skills 10 CrowdFlower, Inc. – Proprietary and Confidential
  11. 11. Skills and Performance Levels The CrowdFlower Platform enables you to add Skill requirements and Performance Levels to your job. Enabling the Skilled Crowds will likely increase the quality of your results and may decrease the throughput of your judgments. 11 CrowdFlower, Inc. – Proprietary and Confidential
  12. 12. Workflows 12 CrowdFlower, Inc. – Proprietary and Confidential
  13. 13. Workflows 13 CrowdFlower, Inc. – Proprietary and Confidential
  14. 14. Test Questions 14 CrowdFlower, Inc. – Proprietary and Confidential
  15. 15. Test Questions Test Questions are a powerful method for measuring the quality of the contributors in your job. By providing the answers to a small percentage of units, you can tell how accurate someone is, and automatically remove anyone who doesn't meet your requirements. 15 CrowdFlower, Inc. – Proprietary and Confidential
  16. 16. Channels 16 CrowdFlower, Inc. – Proprietary and Confidential
  17. 17. Channels 17 CrowdFlower, Inc. – Proprietary and Confidential
  18. 18. CrowdFlower Terminology 18 CrowdFlower, Inc. – Proprietary and Confidential
  19. 19. CrowdFlower Terminology •  •  Unit - One row on a spreadsheet (One company, one officer) •  Dataset - The entire set of data for a project •  Task - A subset of data from the whole dataset •  Workflow - Multiple jobs that strung together •  Channel - A website in which a contributor can access a task •  Test Questions - Training data created in the platform •  19 Judgment - One (of many) answers given by an individual contributor Contributor - Person completing assignments online CrowdFlower, Inc. – Proprietary and Confidential
  20. 20. Process Deconstruction and Job Creation 20 CrowdFlower, Inc. – Proprietary and Confidential
  21. 21. Process Deconstruction & Job Creation Let’s learn how to build a job! Today we’re going to build and run a sentiment analysis job about CrowdFlower. 21 CrowdFlower, Inc. – Proprietary and Confidential
  22. 22. Best Practices – Job Construction 22 CrowdFlower, Inc. – Proprietary and Confidential
  23. 23. Best Practices – Job Construction •  •  Leverage the Template Library •  Use the Instructions Template •  Help contributors find the data (provide links, images, etc) •  Provide examples in the instructions •  Leverage CML logic wherever possible •  Use validations to control text inputs •  23 Break down problem into smallest components Allow an "escape" option for multiple choice questions regarding information in the job or a link: "I don't know", "I couldn't find it", "the link is broken", etc. CrowdFlower, Inc. – Proprietary and Confidential
  24. 24. Best Practices – Test Questions 24 CrowdFlower, Inc. – Proprietary and Confidential
  25. 25. Best Practices – Test Questions •  •  Enable random data for Test Questions •  Copy units to source •  Be as thorough in your Test Question reasons as possible – these will help train the crowd throughout the task •  Create Test Questions out of edge case (subjective or tricky) units to train the crowd •  You don’t need to answer every question •  25 Try to create an even distribution of answers For subjective questions you can allow for multiple correct answers CrowdFlower, Inc. – Proprietary and Confidential
  26. 26. Best Practices – Job Calibration 26 CrowdFlower, Inc. – Proprietary and Confidential
  27. 27. Best Practices – Job Calibration •  •  Set variable judgments on trickier CML elements •  27 Set the Max Judgments Per IP at the number of units in an assignment multiplied by the amount of gold in the task •  This ensures that a contributor will not see the same gold unit twice Set threshold for trust of contributors CrowdFlower, Inc. – Proprietary and Confidential
  28. 28. Launch! 28 CrowdFlower, Inc. – Proprietary and Confidential
  29. 29. Task Monitoring and Reports 29 CrowdFlower, Inc. – Proprietary and Confidential
  30. 30. Task Monitoring and Reports Task Monitoring Interpreting Reports Interpreting Results *This session will teach you how to understand your results. 30 CrowdFlower, Inc. – Proprietary and Confidential
  31. 31. Task Monitoring 31 CrowdFlower, Inc. – Proprietary and Confidential
  32. 32. Task Monitoring • Review Test Questions • Monitor Contributors 32 CrowdFlower, Inc. – Proprietary and Confidential
  33. 33. Interpreting Reports 33 CrowdFlower, Inc. – Proprietary and Confidential
  34. 34. Interpreting Reports 34 CrowdFlower, Inc. – Proprietary and Confidential
  35. 35. Interpreting Results 35 CrowdFlower, Inc. – Proprietary and Confidential
  36. 36. Interpreting Results • Confidence Scores • Aggregation Methods • Auditing for Accuracy 36 CrowdFlower, Inc. – Proprietary and Confidential
  37. 37. Case Studies 37 CrowdFlower, Inc. – Proprietary and Confidential
  38. 38. Case Studies In this session we will cover several practical applications of the CrowdFlower Platform. 38 CrowdFlower, Inc. – Proprietary and Confidential
  39. 39. Search Relevance Testing 39 CrowdFlower, Inc. – Proprietary and Confidential
  40. 40. Example 1 40 CrowdFlower, Inc. – Proprietary and Confidential
  41. 41. Example 2 41 CrowdFlower, Inc. – Proprietary and Confidential
  42. 42. Master Data Management 42 CrowdFlower, Inc. – Proprietary and Confidential
  43. 43. Find URL 43 CrowdFlower, Inc. – Proprietary and Confidential
  44. 44. Find About Us Page 44 CrowdFlower, Inc. – Proprietary and Confidential
  45. 45. Categorize Businesses 45 CrowdFlower, Inc. – Proprietary and Confidential
  46. 46. De-Duplication 46 CrowdFlower, Inc. – Proprietary and Confidential
  47. 47. Content Categorization 47 CrowdFlower, Inc. – Proprietary and Confidential
  48. 48. Photo Categorization 48 CrowdFlower, Inc. – Proprietary and Confidential
  49. 49. Product Categorization 49 CrowdFlower, Inc. – Proprietary and Confidential
  50. 50. Content Moderation 50 CrowdFlower, Inc. – Proprietary and Confidential
  51. 51. Photo Moderation 51 CrowdFlower, Inc. – Proprietary and Confidential
  52. 52. Text Moderation 52 CrowdFlower, Inc. – Proprietary and Confidential
  53. 53. Image and Text Interpretation 53 CrowdFlower, Inc. – Proprietary and Confidential
  54. 54. Image Transcription 54 CrowdFlower, Inc. – Proprietary and Confidential
  55. 55. OCR Verification 55 CrowdFlower, Inc. – Proprietary and Confidential
  56. 56. Product Roadmap 56 CrowdFlower, Inc. – Proprietary and Confidential
  57. 57. Q4 2013 Automatic Test Question access for Enterprise customers only A wizard for first-time users so that your team needs less training A reputation score for our contributors and improvements to our performance levels 57 CrowdFlower, Inc. – Proprietary and Confidential
  58. 58. Q1 2014 In-platform templates for you to get started quickly Improvements to our GUI Team collaboration on jobs In-platform auditing tool so that your internal team and our strongest contributors can easily evaluate the output of your job 58 CrowdFlower, Inc. – Proprietary and Confidential
  59. 59. And beyond… New task analytics for improved slicing and dicing of your data Workflow building tool  Minimize human input (e.g. variable pay where you set the budget, we set the pay for contributors) 59 CrowdFlower, Inc. – Proprietary and Confidential
  60. 60. Questions? 60 CrowdFlower, Inc. – Proprietary and Confidential
  61. 61. Contact Me Nathan Zukoff Platform Program Manager nathan@crowdflower.com 973-997-1921 61 CrowdFlower, Inc. – Proprietary and Confidential

×