Making outsourcing work for you (or getting other people to do your work)

Chris Conley chris@chrisconley.me

Mechanical Turk Marketplace for Work • HITs - Human Intelligence Tasks • Turkers - Mechanical Turk Workers • Requesters ~ Employers • Example HITs: • Product Categorization ($0.03) • Edit Zappos Product Reviews ($0.05) • Find Company Contact Info ($0.02)

Mechanical Turk Marketplace for Work • HITs - Human Intelligence Tasks • Turkers - Mechanical Turk Workers • Requesters ~ Employers • Example HITs: • Product Categorization ($0.03) • Edit Zappos Product Reviews ($0.05) • Find Company Contact Info ($0.02) Artiﬁcial Artiﬁcial Intelligence

Problem • Time and effort still required for • Qualifying Turkers • Reviewing Work

Problem • Time and effort still required for • Qualifying Turkers • Reviewing Work Automatic Artiﬁcial Artiﬁcial Intelligence

Trustiﬁcation Tactics • Qualiﬁcation Tests • Classifying Turkers • Turker Reviewed Work • Spam Filtering

Qualiﬁcation Tests • Qualiﬁcation AND Training

Qualiﬁcation Tests • Qualiﬁcation AND Training Work Quality 5 Worker Pool 1 Ease of Implementation 4 Automatic-ness 3

Classifying Workers • Track Trusted / Rejected

Classifying Workers • Track Trusted / Rejected Work Quality 5 Worker Pool 5 Ease of Implementation 3 Automatic-ness 3

Turker Reviewed Work • Compare / Majority Vote • Work Then Verify

Turker Reviewed Work • Compare / Majority Vote • Work Then Verify Work Quality 3.5 Worker Pool 5 Ease of Implementation 1 Automatic-ness 5

Spam Filtering • Examples: Avg work duration, # of approved HITs, answer content

Spam Filtering • Examples: Avg work duration, # of approved HITs, answer content Work Quality 3 Worker Pool 5 Ease of Implementation 2 Automatic-ness 4

Trustiﬁcation Tactics • Qualiﬁcation Tests • Classifying Turkers • Turker Reviewed Work • Spam Filtering Questions?

Event data collection with Mechanical Turk - Andy Guy Basecamp Business

 Business Events Calendar − Wide coverage − Input events to sustain coverage  Sultan − Internal web app − Large API use We are the Sultans

 External form  Worker Qualifications − Test, Hand picked  Event split into 2 HITs − Event, Address  Multiple (2) HITs per event Process Highlights

Sultan Event Acquisition Process Source URL

Sultan Event Acquisition Process Source URL HIT HIT HIT HIT Event Event Address Address

Sultan Event Acquisition Process Source URL HIT HIT HIT HIT Event Event Address Address E E E E E E E E A A A A A A A A E E E E E E E E A A A A A A A A

Sultan Event Acquisition Process Source URL HIT HIT HIT HIT Event Event Address Address E E E E E E E E A A A A A A A A E E E E E E E E A A A A A A A A Group Name Start Date Start Time

Sultan Event Acquisition Process Source URL HIT HIT HIT HIT Event Event Address Address E E E E E E E E A A A A A A A A E E E E E E E E A A A A A A A A Group E E Name Start Date E E Start Time E E

Sultan Event Acquisition Process Source URL HIT HIT HIT HIT Event Event Address Address E E E E E E E E A A A A A A A A E E E E E E E E A A A A A A A A Group E E Match Name Name Start Date E E Start Date Start Time URL E E

Sultan Event Acquisition Process Source URL HIT HIT HIT HIT Event Event Address Address E E E E E E E E A A A A A A A A E E E E E E E E A A A A A A A A Group E E Match Name Name Start Date E E Start Date Start Time URL E E Reviewable Event

Sultan Event Acquisition Process Source URL HIT HIT HIT HIT Event Event Address Address E E E E E E E E A A A A A A A A E E E E E E E E A A A A A A A A Group E E Match Name Name Start Date E E Start Date Start Time URL E E Resolve discrepancies Reviewable and Event Approve

 Identify workers  Communicate and motivate workers  Fine tune price Keys

Satyajeet Shahade

6- 25 /hr $400 - $1600 $1500 – $500 – $4500 $3000

Outsourcing on oDesk Creating a Worldwide Team Phil Ives April 20, 2010 @ Philly Startup Leaders

Process for oDesk Projects • Deﬁne value of in-house staff – stick to it • Scope Small • How to hire… or how I learned to understand the NCAA tournament seeds • Trust, but verify • Build a real team, get related • SCR-AGILEo – My day to day process

What would you say ya do here?

Deﬁne Value in the process • Hard • Harder – Deﬁne your and local Stick to what you said staff’s value. When timelines slip… – What can’t be stick to what you said outsourced? When people turnover… – If you don’t have know.. stick to what you said better rethink your career

Bite size is better

Scope Small • Scope early and often • Keep tasks small for accountability • Check in with the freelancer early and often

How to Hire • What’s your time worth? • What would an interview process on email and annoying Skype give you? • Why would you do that to yourself? • Hire 2x - 3x actual need and weed out.

Trust, but Verify • Like anything important to your business – Verify that work is correct and complete – Create tests and validation procedures – Spot check – Use the tools you have: Team Room, Bugzilla, Skype – Have a conversation….

Build a Team • Really, they aren’t actually “mechanical” • Manage them like your local team • Use the technology to have conversations • Get acquainted: Who are they / what goals? • Where possible, let them create from nothing • Inspire them with the goals of your project

SCR-AGILEo • Loose agile process • Daily Check-in • Timelines, coordinate time of days • Don’t work with consultants who can’t work on your schedule - make them stick to it. • Iterate both the product and your process for managing it

http://davidspeers.com	265
http://www.slideshare.net	4
http://www.tumblr.com	1
http://www.techgig.com	1

Making outsourcing work for you (or getting other people to do your work)

by davespeers on Apr 20, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

4 Embeds 271

Statistics

Making outsourcing work for you (or getting other people to do your work) — Presentation Transcript