Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Strategies for integrating people and machine learning in online systems

https://www.youtube.com/watch?v=rf2asCOOvfQ

  • Be the first to comment

Strategies for integrating people and machine learning in online systems

  1. 1. Strategies for integrating people and machine learning in online systems Jason Laska Ph.D. Machine Learning @ Clara Labs June 2017 presenting the work of Michael Akilian, Briana Burgess, Joey Carmello, Matthew Ebeweber, David Gouldin, Evan Hadfield, Olga Narvskaia, Maran Nelson, Jodi Nicolli, Emily Pitts, Gavin Schulz, Oliver Song @claralabswww.claralabs.com @chappaquack
  2. 2. Scheduling assistant Email Coordination
  3. 3. Scheduling assistant Customer PreferencesEmail Coordination
  4. 4. Example: Suggesting times M Tu W Th F 8a 5p 12 busy OOO busy busy busy busy Lunch busy busy busy busy Lunch NY CT CTNY NYrecurring location: “Let’s meet in Greenwich for coffee next week.” next week: Lunch Lunch
  5. 5. Example: Suggesting times M Tu W Th F 8a 5p 12 busy OOO busy busy busy busy Lunch busy busy busy busy Lunch NY CT CTNY NYrecurring location: “Let’s meet in Greenwich for coffee next week.” next week: Apply constraints: Lunch Lunch
  6. 6. Example: Suggesting times M Tu W Th F 8a 5p 12 busy OOO busy busy busy busy Lunch busy busy busy busy Lunch NY CT CTNY NYrecurring location: “Let’s meet in Greenwich for coffee next week.” next week: Lunch Lunch Apply constraints: Location: Greenwich, CT
  7. 7. Example: Suggesting times M Tu W Th F 8a 5p 12 busy OOO busy busy busy busy Lunch busy busy busy busy Lunch NY CT CTNY NYrecurring location: “Let’s meet in Greenwich for coffee next week.” next week: Lunch Lunch Apply constraints: Location: Greenwich, CT Coffee: 8am — Noon (preference)
  8. 8. Example: Suggesting times M Tu W Th F 8a 5p 12 busy OOO busy busy busy busy Lunch busy busy busy busy Lunch NY CT CTNY NYrecurring location: “Let’s meet in Greenwich for coffee next week.” next week: Lunch Lunch Apply constraints: Location: Greenwich, CT Max Daily Meetings: 3 Coffee: 8am — Noon (preference) (preference)
  9. 9. Example: Suggesting times M Tu W Th F 8a 5p 12 busy OOO busy busy busy busy Lunch busy busy busy busy Lunch NY CT CTNY NYrecurring location: “Let’s meet in Greenwich for coffee next week.” next week: Lunch Lunch ?Apply constraints: OOO: out of the office, what does that mean in this context? Apply NLP on calendar: Location: Greenwich, CT Max Daily Meetings: 3 Coffee: 8am — Noon (preference) (preference)
  10. 10. Example: Suggesting times M Tu W Th F 8a 5p 12 busy OOO busy busy busy busy Lunch busy busy busy busy Lunch NY CT CTNY NYrecurring location: “Let’s meet in Greenwich for coffee next week.” next week: Lunch Lunch ? ?Apply constraints: OOO: out of the office, what does that mean in this context? Lunch: can we schedule over this or is it important? Apply NLP on calendar: Location: Greenwich, CT Max Daily Meetings: 3 Coffee: 8am — Noon (preference) (preference)
  11. 11. Example: Suggesting times M Tu W Th F 8a 5p 12 busy OOO busy busy busy busy Lunch busy busy busy busy Lunch NY CT CTNY NYrecurring location: “Let’s meet in Greenwich for coffee next week.” Location: Greenwich, CT Max Daily Meetings: 3 next week: OOO: out of the office, what does that mean in this context? Lunch: can we schedule over this or is it important? Apply NLP on calendar: Lunch Lunch ? ?Apply constraints: Relax: can relax constraints if there’s enough travel time? ? Coffee: 8am — Noon (preference) (preference)
  12. 12. Example: Suggesting times M Tu W Th F 8a 5p 12 busy OOO busy busy busy busy Lunch busy busy busy busy Lunch NY CT CTNY NYrecurring location: “Let’s meet in Greenwich for coffee next week.” Location: Greenwich, CT Max Daily Meetings: 3 next week: OOO: out of the office, what does that mean in this context? Lunch: can we schedule over this or is it important? Apply NLP on calendar: Lunch Lunch ? ?Apply constraints: Relax: can relax constraints if there’s enough travel time? ? Coffee: 8am — Noon (preference) (preference) ?
  13. 13. Example: Suggesting times M Tu W Th F 8a 5p 12 busy OOO busy busy busy busy Lunch busy busy busy busy Lunch NY CT CTNY NY Lunch Lunch ? ? ? graceful and intuitive edge-case handling customers really want
  14. 14. How Clara handles this example
  15. 15. How Clara handles this example
  16. 16. How Clara handles this example preference constraints participant availabilities/unavailabilities any accessible party calendars integrated with calendar
  17. 17. Breaking work into tasks “Let’s meet in Greenwich for coffee next week.” location: Greenwich channel: coffee time-pref: next week intent: schedule TASK TYPE: Predict & Annotate
  18. 18. TASK TYPE: Predict & Annotate fix incorrect predictions augment with missing parameters feedback loop to machine learning simple high precision rules (before feedback) after feedback detector only single parameter example Breaking work into tasks
  19. 19. TASK TYPE: Compute & Review location: Greenwich channel: coffee time-pref: next week intent: schedule state: new + action: suggest times Breaking work into tasks
  20. 20. TASK TYPE: Compute & Review check output makes sense check customer needs met feedback loop to product and engineering Breaking work into tasks
  21. 21. manual scheduling highly credentialed worker TASK TYPE: Manual Override Breaking work into tasks
  22. 22. Automation is a spectrum tightly integrate worker-operations, machine learning, and UX-design leverage task differences avg throughput by hour feb mar apr ~1.4xall sessions sessions with worker ~1.8x 1x tasks fully to partially automated match task difficulty with processing skill (person or machine) cost and speed gains without full automation requires
  23. 23. Distributing a knowledge-workforce constraints (requirements) challenges bounded processing time bounded processing cost people are naturally slow at data entry tasks people hours are more expensive than cpu cycles/hr workforce size learning the platform people are naturally noisy at data entry high accuracy bar “staffing on a dime”workforce elasticity
  24. 24. Worker lifecycle self-directed program build commitment to platform through promotion as increase credentials qualification sandbox beginner expert candidate worker sourcing exit platform • english comprehension • other testing • hours practice • exceeds threshold worker support customer support exit platform if accuracy drops below threshold for too long production
  25. 25. task aggregate & compare replicate state production sandbox — “scheduling multiverse” candidate 1 score candidate 2 score candidate n score worker candidates submit Sandbox environment
  26. 26. Task assignment hardeasiesttasks (most automated) (less automated)automatable
  27. 27. escalate escalate expert Task assignment beginner $$$$ hardeasiesttasks (most automated) (less automated)automatable cost worker pool queue
  28. 28. escalate escalate expert Task assignment beginner $$$$ hardeasiesttasks (most automated) (less automated)automatable cost worker pool queue
  29. 29. escalate escalate expert Task assignment beginner $$$$ hardeasiesttasks (most automated) (less automated)automatable cost worker pool queue
  30. 30. 2nd escalate easiest 1st escalate Work recycling recycle the first queue n times beginners may escalate when unsure (equivalent to “skip” in this case) # of recycles = proxy to task difficulty works well with incentive to avoid mistakes
  31. 31. 2nd escalate easiest 1st escalate other good strategies Work recycling “Double or Nothing: Multiplicative Incentive Mechanisms for Crowdsourcing” by N. Shah, D. Zhou recycle the first queue n times beginners may escalate when unsure (equivalent to “skip” in this case) # of recycles = proxy to task difficulty works well with incentive to avoid mistakes allow worker to “skip” tasks reward based on known-examples: no pay for skipping high penalty for incorrect labels high reward for correct labels incentivizes skipping if worker confidence is low
  32. 32. workforce size Distributing a knowledge-workforce constraints (requirements) challenges bounded processing time bounded processing cost workforce elasticity people are naturally slow at data entry tasks “staffing on a dime” people hours are more expensive than cpu cycles/hr learning the platform people are naturally noisy at data entry high accuracy bar
  33. 33. Incentives competing incentives drive both throughput and accuracy worker throughput worker accuracy per-task payment time preference (by ranked accuracy*) *workers must also maintain a minimum acuracy to remain on the platform
  34. 34. via turtlebot in community Slack channel Staffing: Supply and demand (workers) (customer requests) (all potential weekly supply) workers find out about “pick up” work requested hours predicted demand/hour worker reliability worker throughput worker accuracy when supply > demand: when supply << demand: hour assignments Jodi’s 3-pass assignment algorithm
  35. 35. Measuring accuracy: Mistake tracking worker peer review Worker Dashboard workers have access to previous annotations workers have task context function of time/cost constraints
  36. 36. Measuring accuracy: Mistake tracking other good strategies worker peer review Worker Dashboard 1) assign multiple workers to same task 2) estimate underlying annotation values and worker accuracy and biases a) jointly e.g., “Quality Management on Amazon Mechanical Turk” by Ipeirotis, Provost, & Wang b) via Cohen/Fleiss kappa: … many other ways  := 1 1 pobserved agreement 1 pchance agreement workers have access to previous annotations workers have task context function of time/cost constraints
  37. 37. community Slack channel we love the CRAs!! Community is essential community manager runs worker support builds platform documentation community pulse surveys promote the needs of the community “Clara Remote Assistant” (CRA) beta launches CRAs love trying new tools before they roll out we get amazing feedback ask/answer questions, pictures of pets, turtlebot, etc. [realwaystoearnmoneyonline.com] [workfromhomehappines.com]
  38. 38. kinds of people we’re looking for Come work with us! @claralabswww.claralabs.com modeling email requests combining ML & UX for data-entry systems natural language understanding fullstack & frontend engineers machine learning engineers come talk to me! we work in python/flask, react/redux, aws services, sklearn, keras… jason@claralabs.com @chappaquack software problems design problems ML problems continuous model train and integration worker task management defining new Clara capabilities

×