5
DOLLAR DATA PROJECT
by: Dan Rockwell - dan@lextant.com
a series of experiments using
amazon mechanical turk
last updated: 2/24/2009

WHAT IS THE 5 DOLLAR DATA PROJECT?
How much data can ya get for 5 bucks using
amazon’s mechanical turk service?
what the hell is that?
A web service that allows you to assign
human interface tasks, or HITS to real people
who will then complete these task for real
money.

WHO ARE THESE PEOPLE?
Stay at home moms, people working on lunch
breaks, average joe’s, you really don’t know.
why do they do this?
It’s money. Most of the tasks on mechanical turk
are basic things like:
- look at an image and give keywords to help train
a computer how to look at images for search
engines
- find search results on the web, give a human
perspective to teach a computer

How’s it Work?

How do you create a HIT?
1. put some money into your account
2. click create a hit
3. ask the question, write instructions, etc
4. define the answer type
5. define how many people or answers you want
6. specify a time frame for completion
7 set the price you’ll pay for an answer
.
8. start receiving answers
9. approve/reject/message
10. download results

1
STATE OF PARENTAL CONTROL FOR PC’S AND THE WEB
HOW IT SHOULD IT CHANGE

PROJECT 1 - SIMPLE QUESTION
37 responses in 4 days
Paid 10 cents per answer
Total cost $3.70
example We make computer time \"family time.\" When the kids want to go online, we
allow her to do so, however, one of us has to be with her, monitoring what
she is doing. She is only 9, so when she goes online it's to a gaming site or
to look something up. I would like to allow her to go online without me
looking over her shoulder, but I also want her to have freedom to surf safely.
I would like to see some kind of summary or reporting of the sites she went
to, the people she talked to and the content of any emails she may receive/
send.

PROJECT 1 - LEARNINGS
pretty cool
fast
fun to see the data arrive
how do i get at demographic info
how can i figure out who these folks are?
not supposed to ask who they are bt w

2
CHOOSE 5 WEBSITES THAT DEFINE YOUR STYLE & WHY

PROJECT 2 - DEFINE YOUR STYLE IN URL’S
12 responses in 3 days
Paid 10 cents per answer
Total cost $3.80
example
Taylormade.com - the reason this site
represents my style is that I love to golf
and the only clubs I hit are Taylormade.
Scrapbook.com - I like this page because I TelevisionWithoutPity - I love TV,
love to scrapbook. It gives me ideas for pages but not so much that I am blind to
that I can incorporate into my scrapbooks. the bad parts. I'm smart as well as a
smart-ass. This site isn't too
cheerful, or too happy. just like me.

PROJECT 2 - LEARNINGS
very cool
people seemed to enjoy this task
loads of meaty content!
how can push for something emotional?

3
DETAIL 5 SITUATIONS / EMOTIONAL EVENTS THAT
OCCURRED TO YOU TODAY
FIND A WEB SITE THAT EMBODIES THAT FEELING
TELL ME WHY

PROJECT 3 - EMOTIONAL OUTPUT
14 responses in 3 days
Paid 25 cents per answer
Total cost $2.75
example
Anger - Angry365 site- I was angry at my computer today.
Shock - postsecret.com - I am shocked
and depressed about people today.
Tired - Bedding.com site -
I felt really sleepy today,
worn down.
Frisky - FREDRICKS.com - I felt attractive today, and
what's sexier than lingerie?

PROJECT 3 - LEARNINGS
very cool
people seemed to enjoy this task
a bit tricky with the whole emotional
embodiment thing
what are the limits to what people will do?

4
CREATE A YOUTUBE VIDEO ANSWERING THE QUESTION
WHAT IS DESIGN TO YOU, WHATS GOOD DESIGN FEEL LIKE
GIVE AN EXAMPLE OF GOOD DESIGN

PROJECT 4 - GO MAKE A VIDEO
1 response in 3 days
Paid 2 DOLLARS per answer
Total cost $2
example

PROJECT 4 - LEARNINGS
not so good
$2 wasn’t enough
maybe be hard to do
people may not have web cams
let’s test prime next!

5
CREATE A COLLAGE OF YOU WERE, ARE, AND WILL BE
+
DO THE “FEARS” COLLAGE FROM DELL PROJECT
+
ANSWER SOME BASIC QUESTIONS

PROJECT 5 - PRIME TEST!
25 responses in 2 HOURS!
Paid 1 DOLLAR per completed work
Total cost $25
example

PROJECT 5 - LEARNINGS
GREAT stress test for PRIME!
62 people hit PRIME in 2 HOURS!
swimming in data
people appeared to enjoy the task
people figured out how to group concepts easily
this is very cool

6
FIND AND UPLOAD 10 PHOTOS THAT REPRESENT
AN IDEAL EXPERIENCE AND TELL WHY
( test our photo journal tool )

A tarrot card
PROJECT 6 - PRIME PHOTO JOURNAL reading is fun.
11 responses in 2 days
Paid 50 cents per completed work
Total cost $5.50
I love drinking wine.
example
My family.
I want to be in the spotlight. Outdoor concert.

PROJECT 6 - LEARNINGS
another GREAT stress test for PRIME!
tested our photo journal process
tested our server load capability
we asked people to rate the task itself:
people thought it was easy, simple, fast but boring,
lame and they should of been paid a buck

7
transcribe an audio file
access the web, listen to a 45min file, transcribe

Rich:Yeah. I mean, I don’t have an answer for what the solution
PROJECT 7 - transcription to that is, but I would say that when it comes to the message
boards that I have experienced I can almost classify them into
two groups. There’s the philanthropic kind of message board
where people are just talking about non concrete things and it
1 responses in 1 days is all a view, it is just opinions. Which is great for some people,
but I am not interested in that. If I want someone’s opinion I
will talk to them.
When I have a concrete problem and I need an answer for
something specific and I don’t have anyone I can ask. I have
Paid $8.00 for transcription tapped out everyone that I can ask. Then it is great to be able
to go to that and say, “Hey, how do I do this?” I have answered
quite a few people over the years in that respect too.You
know, I try to pay back where I get that information from.
Total cost $8.00 Female: Correct me if I am wrong. So you see that as a
supplemental resource to what you have offline.
Rich: Right. Right.Yeah.
Female: Ok.
example Rich: I mean if I think about an organization. I don’t know who
you guys are doing this for. But say it is IBM or whoever. First
and foremost I think it is about the people. Then yo have go all
these other supplemental things like job boards, or message
https://www.box.net/shared/static/25l2q8fsws.mov boards, or technical help, or discounts on things or whatever
they have. But those are all kind of like secondary parts of
those organizations and communities
Male 1: So in your ideal community the center would be
around people. And what would you get out of this ideal
community of yours?
Rich: Well like I said, this is going to make me seem like a lush,
but I think one of the best where they just get a bar and they
say if you are a designer show up here and you can shoot the
shit with some other people. I think for me that works out
great because it is after hour and I can turn off the work Rich
for a little while and I can walk in there and still have a pretty
open discussion about…

PROJECT 7 - LEARNINGS
at first it felt hit or miss if someone would do this
1 person took on the task and did an amazing job
reads like a movie script
turks can access audio on the web, listen to it, and
document/transcribe
transcription appears to be one of the bigger uses of
mechanical turk (based on what i’ve seen)

8
analyze some data

PROJECT 8 - analysis DUDE THIS IS TOO MUCH
WORK FOR $5.00!!
2 responses in 3 days
each turk only did one participant of
Paid $5.00 for analysis data
each explained how they did their
analysis
Total cost $10.00
each commented on how much this work
was, and while interesting couldn’t
example complete the work due to time/money

PROJECT 8 - LEARNINGS
too much work for the money involved
analysis many never be viable for turkers
without context turks were unsure what they were
looking for in the data
turks do have excel and can edit office documents

9help me brainstorm
( can turks help you brainstorm while you do something else)

• Contact staffing and temporary work agencies (that
PROJECT 9 - brainstorm specialize in placement for industrial, construction or
warehouse related activities)
27 responses in 4 hours • Contact the unemployment office
• On site / Near site intercepts (try to time it for the
morning or afternoon when the “roach coach” is
Paid $.05 per answer there)
• Contact facilities: Distribution centers, warehouses,
bottling companies, drywall suppliers.
Total cost $1.35 • Contact organizations: Teamsters, Longshoremans,
Warehousemans, United Transportation, Association
of machinists, Aeorspace Workers
our question: • Contact companies: Costco, Sam’s clu, Fed Ex, UPS,
Supermarkets
• Contact other: DMV (to find out who has a reach
how would you go about truck license), local fork lift suppliers (to see who they
sell their trucks to): http://www.chicagolift.com/
recruiting forklift home.html , http://www.yeslifts.com/
drivers in chicago for a • Search / post on job boards like Monster.com: http://
www.forklift.driver.jobs.com/ , http://
research study? www.nationalcontractors.com/
• Print attractive, colorful posters and place them at
service stations, bars, public places
• Drop off and fax flyers to appropriate agencies and
companies

PROJECT 9 - LEARNINGS
yeeehaw people love fun open ended questions
many responses very fast
simple, fun task
turks seemed to enjoy the advisory/helping like
action of this task
time & energy to do worth the 5 cent reward

10
paid ya, can ya give me a bit more?

PROJECT 10 - the drill down
10 responses in 2 hours
Paid $.25 per answer
Total cost $2.50
task:
Complete a collage. After • The goal of this project was to test
initial completion, ask the whether or not participants after having
completed an initial assignment would
participant to come back return (by our request) to give further
detail on the assignment without giving
and drill down further on additional payment.
an aspect of the collage.

PROJECT 10 - LEARNINGS
yes it is possible
for turks to complete a task and then return to give additional information - after
payment
this suggests that turks are honorable
to a degree of the work they do, payment accepted and are willing to go the extra
“mile” to give additional clarification on an assignment
turks are potentially perfect participants
for studies because of the environment they are on, and expected interaction they
often have
turks could be potentially too eager to express an
opinion, but is that a bad thing?

11
brand ransom

PROJECT 11 - brand ransom
23 responses in 2 hours
Paid $1.00 per ransom note
Total cost $25.00
task: create a ransom note
Using a ransom note
generator on the web,
recall a recent frustration
with a brand/experience
and articulate it via the
online tool.

PROJECT 11 - LEARNINGS
awesome task, turks had a lot of fun with this one
unique & fun task for participants, engaging and
expressive
had to lead by example but this assured a good range
of success
compared to typical quant data, this task
essentially asked the same question but gave the
participant a richer language to answer, resulting in
an artifact that is both meaningful + art like

mechanical turk take aways
FAST & easy to use
perfect for testing web apps in development
excellent discovery tool
great for independent projects, pilot homework
test instructions, wild methods, explore ideas
extremely cost effective way to get a taste of data on
topic you desire
excellent to pair up with fast, fun, and unique
engagement apps online

mechanical turk issues
still not believable
despite having some demographical data, its hard for
people to believe that anyone would do these tasks
exploitive
how much credibility can you give the results?
real back to back testing is required
where does it belong in your tool box

NEXT STEPS
continue to experiment and think beyond what you
feel is silly or doesn’t make sense to do
use it to test homework assignments, methods, rapid
protos
conduct more experiments FASTER
do more demographic / personality profiling of turks
integrate it into PRIME and other tool sets
develop a methodology/offer around it? fast trending?