0
Chris Conley
  chris@chrisconley.me
Mechanical Turk
 Marketplace for Work
• HITs - Human Intelligence Tasks
• Turkers - Mechanical Turk Workers
• Requesters ~...
Mechanical Turk
 Marketplace for Work
• HITs - Human Intelligence Tasks
• Turkers - Mechanical Turk Workers
• Requesters ~...
Problem
• Time and effort still required for
 • Qualifying Turkers
 • Reviewing Work
Problem
• Time and effort still required for
 • Qualifying Turkers
 • Reviewing Work


          Automatic
Artificial Artifi...
Trustification Tactics

• Qualification Tests
• Classifying Turkers
• Turker Reviewed Work
• Spam Filtering
Qualification Tests

• Qualification AND Training
Qualification Tests

• Qualification AND Training



    Work Quality              5
     Worker Pool              1
Ease of...
Classifying Workers

• Track Trusted / Rejected
Classifying Workers

• Track Trusted / Rejected



    Work Quality             5
     Worker Pool             5
Ease of I...
Turker Reviewed Work

• Compare / Majority Vote
• Work Then Verify
Turker Reviewed Work

• Compare / Majority Vote
• Work Then Verify


    Work Quality            3.5
     Worker Pool     ...
Spam Filtering
• Examples: Avg work duration, # of approved
  HITs, answer content
Spam Filtering
• Examples: Avg work duration, # of approved
  HITs, answer content



      Work Quality               3
 ...
Trustification Tactics

• Qualification Tests
• Classifying Turkers
• Turker Reviewed Work
• Spam Filtering


              ...
Event data collection with Mechanical Turk
                - Andy Guy




     Basecamp Business
   Business Events
             Calendar
                −   Wide coverage
                −   Input events to
          ...
   External form
            Worker Qualifications
                −   Test, Hand picked
            Event split into 2...
Sultan Event Acquisition Process
                                   Source
                                    URL
Sultan Event Acquisition Process
                                     Source
                                      URL



...
Sultan Event Acquisition Process
                                           Source
                                       ...
Sultan Event Acquisition Process
                                           Source
                                       ...
Sultan Event Acquisition Process
                                           Source
                                       ...
Sultan Event Acquisition Process
                                           Source
                                       ...
Sultan Event Acquisition Process
                                           Source
                                       ...
Sultan Event Acquisition Process

                                           Source
                                      ...
   Identify workers
   Communicate and motivate workers
   Fine tune price




                      Keys
Satyajeet Shahade
6- 25 /hr   $400 -
            $1600




            $1500 –
$500 –      $4500
$3000
Outsourcing on oDesk
 Creating a Worldwide Team
          Phil Ives

April 20, 2010 @ Philly Startup
            Leaders
Process for oDesk Projects
• Define value of in-house staff – stick to it
• Scope Small
• How to hire… or how I learned to
...
What would you say ya do here?
Define Value in the process
• Hard                        • Harder
 – Define your and local        Stick to what you said
  ...
Bite size is better
Scope Small
• Scope early and often
• Keep tasks small for accountability
• Check in with the freelancer early and
  often
How to Hire
• What’s your time worth?
• What would an interview process on
  email and annoying Skype give you?
• Why woul...
Trust, but Verify
• Like anything important to your business
 – Verify that work is correct and complete
 – Create tests a...
Build a Team
• Really, they aren’t actually “mechanical”
• Manage them like your local team
• Use the technology to have c...
SCR-AGILEo
• Loose agile process
• Daily Check-in
• Timelines, coordinate time of days
• Don’t work with consultants who c...
Making outsourcing work for you (or getting other people to do your work)
Making outsourcing work for you (or getting other people to do your work)
Making outsourcing work for you (or getting other people to do your work)
Upcoming SlideShare
Loading in...5
×

Making outsourcing work for you (or getting other people to do your work)

1,761

Published on

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,761
On Slideshare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
10
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide




































  • Transcript of "Making outsourcing work for you (or getting other people to do your work)"

    1. 1. Chris Conley chris@chrisconley.me
    2. 2. Mechanical Turk Marketplace for Work • HITs - Human Intelligence Tasks • Turkers - Mechanical Turk Workers • Requesters ~ Employers • Example HITs: • Product Categorization ($0.03) • Edit Zappos Product Reviews ($0.05) • Find Company Contact Info ($0.02)
    3. 3. Mechanical Turk Marketplace for Work • HITs - Human Intelligence Tasks • Turkers - Mechanical Turk Workers • Requesters ~ Employers • Example HITs: • Product Categorization ($0.03) • Edit Zappos Product Reviews ($0.05) • Find Company Contact Info ($0.02) Artificial Artificial Intelligence
    4. 4. Problem • Time and effort still required for • Qualifying Turkers • Reviewing Work
    5. 5. Problem • Time and effort still required for • Qualifying Turkers • Reviewing Work Automatic Artificial Artificial Intelligence
    6. 6. Trustification Tactics • Qualification Tests • Classifying Turkers • Turker Reviewed Work • Spam Filtering
    7. 7. Qualification Tests • Qualification AND Training
    8. 8. Qualification Tests • Qualification AND Training Work Quality 5 Worker Pool 1 Ease of Implementation 4 Automatic-ness 3
    9. 9. Classifying Workers • Track Trusted / Rejected
    10. 10. Classifying Workers • Track Trusted / Rejected Work Quality 5 Worker Pool 5 Ease of Implementation 3 Automatic-ness 3
    11. 11. Turker Reviewed Work • Compare / Majority Vote • Work Then Verify
    12. 12. Turker Reviewed Work • Compare / Majority Vote • Work Then Verify Work Quality 3.5 Worker Pool 5 Ease of Implementation 1 Automatic-ness 5
    13. 13. Spam Filtering • Examples: Avg work duration, # of approved HITs, answer content
    14. 14. Spam Filtering • Examples: Avg work duration, # of approved HITs, answer content Work Quality 3 Worker Pool 5 Ease of Implementation 2 Automatic-ness 4
    15. 15. Trustification Tactics • Qualification Tests • Classifying Turkers • Turker Reviewed Work • Spam Filtering Questions?
    16. 16. Event data collection with Mechanical Turk - Andy Guy Basecamp Business
    17. 17.  Business Events Calendar − Wide coverage − Input events to sustain coverage  Sultan − Internal web app − Large API use We are the Sultans
    18. 18.  External form  Worker Qualifications − Test, Hand picked  Event split into 2 HITs − Event, Address  Multiple (2) HITs per event Process Highlights
    19. 19. Sultan Event Acquisition Process Source URL
    20. 20. Sultan Event Acquisition Process Source URL HIT HIT HIT HIT Event Event Address Address
    21. 21. Sultan Event Acquisition Process Source URL HIT HIT HIT HIT Event Event Address Address E E E E E E E E A A A A A A A A E E E E E E E E A A A A A A A A
    22. 22. Sultan Event Acquisition Process Source URL HIT HIT HIT HIT Event Event Address Address E E E E E E E E A A A A A A A A E E E E E E E E A A A A A A A A Group Name Start Date Start Time
    23. 23. Sultan Event Acquisition Process Source URL HIT HIT HIT HIT Event Event Address Address E E E E E E E E A A A A A A A A E E E E E E E E A A A A A A A A Group E E Name Start Date E E Start Time E E
    24. 24. Sultan Event Acquisition Process Source URL HIT HIT HIT HIT Event Event Address Address E E E E E E E E A A A A A A A A E E E E E E E E A A A A A A A A Group E E Match Name Name Start Date E E Start Date Start Time URL E E
    25. 25. Sultan Event Acquisition Process Source URL HIT HIT HIT HIT Event Event Address Address E E E E E E E E A A A A A A A A E E E E E E E E A A A A A A A A Group E E Match Name Name Start Date E E Start Date Start Time URL E E Reviewable Event
    26. 26. Sultan Event Acquisition Process Source URL HIT HIT HIT HIT Event Event Address Address E E E E E E E E A A A A A A A A E E E E E E E E A A A A A A A A Group E E Match Name Name Start Date E E Start Date Start Time URL E E Resolve discrepancies Reviewable and Event Approve
    27. 27.  Identify workers  Communicate and motivate workers  Fine tune price Keys
    28. 28. Satyajeet Shahade
    29. 29. 6- 25 /hr $400 - $1600 $1500 – $500 – $4500 $3000
    30. 30. Outsourcing on oDesk Creating a Worldwide Team Phil Ives April 20, 2010 @ Philly Startup Leaders
    31. 31. Process for oDesk Projects • Define value of in-house staff – stick to it • Scope Small • How to hire… or how I learned to understand the NCAA tournament seeds • Trust, but verify • Build a real team, get related • SCR-AGILEo – My day to day process
    32. 32. What would you say ya do here?
    33. 33. Define Value in the process • Hard • Harder – Define your and local Stick to what you said staff’s value. When timelines slip… – What can’t be stick to what you said outsourced? When people turnover… – If you don’t have know.. stick to what you said better rethink your career
    34. 34. Bite size is better
    35. 35. Scope Small • Scope early and often • Keep tasks small for accountability • Check in with the freelancer early and often
    36. 36. How to Hire • What’s your time worth? • What would an interview process on email and annoying Skype give you? • Why would you do that to yourself? • Hire 2x - 3x actual need and weed out.
    37. 37. Trust, but Verify • Like anything important to your business – Verify that work is correct and complete – Create tests and validation procedures – Spot check – Use the tools you have: Team Room, Bugzilla, Skype – Have a conversation….
    38. 38. Build a Team • Really, they aren’t actually “mechanical” • Manage them like your local team • Use the technology to have conversations • Get acquainted: Who are they / what goals? • Where possible, let them create from nothing • Inspire them with the goals of your project
    39. 39. SCR-AGILEo • Loose agile process • Daily Check-in • Timelines, coordinate time of days • Don’t work with consultants who can’t work on your schedule - make them stick to it. • Iterate both the product and your process for managing it
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×