Recommended
More Related Content
Similar to Practical Crowdsourcing for ML at Scale
Similar to Practical Crowdsourcing for ML at Scale (20)
More from Bill Liu
More from Bill Liu (20)
Recently uploaded
Recently uploaded (20)
Practical Crowdsourcing for ML at Scale
- 1. Daria Baidakova Data labeling on a large scale: the missing pillar of AI
- 2. Labeled data: the missing pillar of AI TensorFlow, PyTorch, CatBoost, etc AWS, MS Azure, Google Cloud, Yandex Cloud, etc ??? Algorithms Hardware Data AI
- 3. 3 Intro Daria Baidakova — Director of Educational Programs at Toloka Responsible for consulting and supporting Toloka requesters in integrating crowdsourcing methodology in AI projects. She also manages crowdsourcing courses at top data analysis schools (Yandex School of Data Analysis, Y- Data, etc) and organizes tutorials and hackathons for crowdsourcing specialists. Co-author of four hands-on tutorials on efficient crowdsourcing (at WSDM'20, CVPR'20, SIGMOD'20, WWW'21) and a co-organizer of the crowd science workshop at NeurIPS'2020.
- 4. 10 years experience in real industry 4 Car Sharing Self Driving Car Health Cloud Search Engine Voice Assistant Browser Weather Ad Tech Taxi Food Delivery E-commerce Personalized Stories Feed Mail Storage Auto.ru Jobs Realty Movies Tickets Music Maps Navi Transport Auto
- 5. Infrastructure for AI worldwide 5 Infrastructure for Search Infrastructure for Yandex Infrastructure for AI industry
- 6. ML production pipeline 6 Sample schematic block Control in production Training Retraining Validating
- 7. 7 Tolóka — an ancient tradition
- 8. Tolokers Requesters Intelligent platform Toloka — an open crowdsourcing platform
- 9. Tolokers distribution 9 * New regions and languages can be quickly allocated upon customer request Argentina Cote d'Ivoire France India Philippines Tunisia Morocco Turkey Ukraine Brazil Russia Top countries with active tolokers* Kenya Pakistan Venezuela Egypt Mexico Nigeria Peru Portugal Spain USA Vietnam Top languages* ∙ English ∙ Spanish ∙ Arabic ∙ Portuguese ∙ Russian ∙ Ukrainian ∙ French ∙ German ∙ Italian ∙ Polish ∙ Latvian ∙ Bulgarian ∙ Czech ∙ Turkish ∙ Hindi ∙ Vietnamese ∙ Japanese ∙ Chinese ∙ Korean ∙ Indonesian 9+ million tolokers available across every time zone, 24 hours a day.
- 10. Use cases
- 14. Say Hey, Alisa! Say «Hey, Alisa!» Was the phrase correct? Final result: audio recording Accept, if yes Reject, if no
- 16. 16 Update info about an organization
- 17. 18
- 18. Pipeline for business verification 19 • Comes to point; • Takes photos GPS ok? Original photos? Business found? • Is there address on the photo? • Was the whole area photographed? Rewrite from photo: Name, tel, website, working hours, etc. Computer vision for company codes and other fields Is the photo ok to be shown on maps? Final result: verified business No, task declined no no no Decline if not possible yes Task accepted; pay
- 19. 21 Manual information collection on the Internet
- 21. Data annotation
- 22. Real-life cases 24 Side-by-side comparison 10 min $2.4 # 1 000 tasks Text Classification 2 hrs $18 # 1 000 tasks Phrase generation for a chatbot 15 min $1 # 500 phrases on the same topic Object classification 15 min $1.2 # 1 000 photos Object segmentation 5 hrs $3.6 # 1 000 objects in 100 photos Audio transcription 20 min $6 # 100 recordings 25 minute long
- 23. 25 Search relevance results evaluation
- 25. 27 Objects segmentation for CV
- 26. 28 Object segmentation for CV
- 31. Moderation of content: human-in-the-loop 33 ML No Needed confidence level? Training and selection Is the content OK? 2nd line Final result: moderation verdict Yes
- 32. Product decision making and market research
- 34. 37 Preferences
- 35. Quality on a large scale
- 36. Smart marketplace Tolokers: rate requesters Requesters: control the quality of Tolokers Platform: general rating, rating of tasks
- 37. Managing process not people 41
- 38. Managing process not people 42 Selecting performers â–º Language & region â–º Age â–º Gender â–º Task specific skills â–º Device filters
- 39. Managing process not people 43 Advanced training & motivation Selecting performers â–º Educational projects â–º Performers examination â–º Trained performers â–º Quality based pricing â–º Language & region â–º Age â–º Gender â–º Task specific skills â–º Device filters
- 40. Managing process not people 44 â–º Behavior control â–º Anti-robotic tools â–º Hidden control tasks â–º Multiple users consensus â–º Verification of assignments Project-level controls Advanced training & motivation Selecting performers â–º Educational projects â–º Performers examination â–º Trained performers â–º Quality based pricing â–º Language & region â–º Age â–º Gender â–º Task specific skills â–º Device filters
- 41. Managing process not people 45 â–º Behavoiur control â–º Anti-robotic tools â–º Hidden control tasks â–º Multiple users consensus â–º Verification of assignments Project-level controls Advanced training & motivation Selecting performers â–º Educational projects â–º Performers examination â–º Trained performers â–º Quality based pricing â–º Language & region â–º Age â–º Gender â–º Task specific skills â–º Device filters â–º System-level ML â–º Multiple aggregation models â–º Result-based performers selection â–º Real-time insights Aggregation of results & analysis
- 42. People management as an engineering task Deal with the Crowd as with yet another computing cluster Require minimal effort from people
- 43. People management as an engineering task Deal with the Crowd as with yet another computing cluster Require minimal effort from people Free, powerful API
- 44. Toloka open source: examples and algorithms https://github.com/Toloka/crowd-kit https://github.com/Toloka/toloka-kit
- 46. https://toloka.ai/academy Our goal is to help the industry overcome the bottleneck that data constitutes today.
- 47. 59 Crowd Science Lecturers are experts in crowdsourcing: — ML developers & Crowd Solution Architects from one of the top IT companies in Europe We share our methodology: — based on years of research and unique industry expertise
- 48. 60 Practice — Lecturers with real experience developing products and working with ML and AI tasks — Tasks with real-life applications — Case studies from Yandex, one of Europe’s top IT companies — The only crowdsourcing course with practice based on real, large- scale industry use-cases
- 49. https://www.coursera.org/learn/practical-crowdsourcing 61 Our success stories Coursera online-course Courses in top Swiss Universities, Tel-Aviv University & Y-Data
- 50. https://toloka.ai/academy 62 What we offer Talks, lectures & hands-on tutorials Research grants Student plans for data labeling projects Ready-to-use materials Other ideas? Let’s discuss!
- 51. Resources 63 Toloka Academy Seminar series on crowdsourcing and beyond Toloka Github Coursera Course: Practical Crowdsourcing for ML Try Toloka https://toloka.ai/
- 53. Daria Baidakova Director of Educational Programs Thank you! https://www.linkedin.com/in/dbaidakova/ https://toloka.ai/