This document provides an overview of Amazon Mechanical Turk (MTurk) and how it can be used for crowdsourcing projects. It discusses key MTurk concepts like requesters, workers, HITs, assignments, and qualifications. It then walks through the steps to create an MTurk project, including defining the HIT properties, previewing templates, creating batches, publishing HITs, and reviewing results. Finally, it discusses best practices like testing HITs in the sandbox environment and monitoring worker forums.
Artificial Artificial Intelligence: Using Amazon Mechanical Turk and .NET to ...goodfriday
Amazon Mechanical Turk is a new Web service that allows .NET software developers to incorporate the power of human decision-making into their automated software systems
I composed this brief introduction to Amazon's crowdsourcing tool, Mechanical Turk (AMT), at the request of a professor who advocates AMT for marketing research.
Amazon Mechanical Turk (https://requester.mturk.com), a crowdsourcing marketplace, adds value by processing work that cannot be computerized.
Typically, these include large-scale [data] projects - think Amazon.com's 500MM product database - which requires some form of human judgment to help digitize, validate, moderate, tag, cleanse, categorize, dedupe, etc...This is accomplished by breaking down the project into micro-tasks, distributing to qualified cloud-based workers, who then deliver the results.
Slides presented at the Mechanical Turk Office Hours on January 20, 2011.
Video of the presentation can be found on the Mechanical Turk blog:
http://mechanicalturk.typepad.com
Artificial Artificial Intelligence: Using Amazon Mechanical Turk and .NET to ...goodfriday
Amazon Mechanical Turk is a new Web service that allows .NET software developers to incorporate the power of human decision-making into their automated software systems
I composed this brief introduction to Amazon's crowdsourcing tool, Mechanical Turk (AMT), at the request of a professor who advocates AMT for marketing research.
Amazon Mechanical Turk (https://requester.mturk.com), a crowdsourcing marketplace, adds value by processing work that cannot be computerized.
Typically, these include large-scale [data] projects - think Amazon.com's 500MM product database - which requires some form of human judgment to help digitize, validate, moderate, tag, cleanse, categorize, dedupe, etc...This is accomplished by breaking down the project into micro-tasks, distributing to qualified cloud-based workers, who then deliver the results.
Slides presented at the Mechanical Turk Office Hours on January 20, 2011.
Video of the presentation can be found on the Mechanical Turk blog:
http://mechanicalturk.typepad.com
Presented at Tech Tuesdays in Edinburg, TX. In this presentation, a sample Hangman game made with MeteorJS. Source code available at: github.com/ibolmo/hangmanapp
Data Platform at Twitter: Enabling Real-time & Batch Analytics at ScaleSriram Krishnan
The Data Platform at Twitter supports engineers and data scientists running batch jobs on Hadoop clusters that are several 1000s of nodes, and real-time jobs on top of systems such as Storm. In this presentation, I discuss the overall Data Platform stack at Twitter. In particular, I talk about enabling real-time and batch analytics at scale with the help of Scalding, which is a Scala DSL for batch jobs using MapReduce, Summingbird, which is a framework for combined real-time and batch processing, and Tsar, which is a framework for real-time time-series aggregations.
IBM Rational Host Access Transformation Services (HATS) can dynamically transform your terminal-based applications into rich web pages. It is highly customizable and built on Java EE technology. We'll discuss some lessons learned from a very (very) complex HATS engagement. We'll discuss proper development strategies, and how to distribute workload across team members. We'll introduce a novel approach to unit testing advanced customizations using JUnit, and will also talk about how to address functional testing.
Busy Developers Guide to AngularJS (Tiberiu Covaci)ITCamp
Since last year, Single Page Applications have grown exponentially in popularity and the framework of choice for many developers is Angular.js. In this session we will go through some of the features that make Angular such a popular framework so you can start using it in your own projects.
Patterns for Scalability in Windows Azure Applications (Alex Mang)ITCamp
So you've learned what elasticity means and why it is important to consider scalability in your cloud application architecture. But how can you easily manage your code in order to implement all the theories around scalability?
During this session, I will talk and demo the most common patterns used when designing a cloud application: the valet key pattern, the sharding pattern, the materialized view pattern, the event sourcing pattern and the CQRS pattern.
[DevDay 2016] Real Unit Testing with mocking framework - Speaker: Phat Vu – S...DevDay.org
Why do programmers hate writing Unit Tests? One big reason is object dependency. An object under testing may have dependencies on other complex objects, which might not have been implemented or been complicated when invoking.
Join the session refresh your thinking about Unit Testing and overview of mocking framework, as well as learn some practice/gotcha to write a real Unit Test, how to isolate the behavior of the object you want to test, how to simulate the behavior of the dependencies.
———
Speaker: Phat Vu – Scrum Master at Axon Active Vietnam
SigOpt at MLconf - Reducing Operational Barriers to Model TrainingSigOpt
In this talk at MLconf NYC, Alexandra Johnson, platform engineering lead at SigOpt, discusses common operational challenges with scaling model training and how solutions are designed to
Feedback on building Production-Ready Microsoft Teams AppsGuillaume Meyer
Building mockups and prototypes is fine, but when it comes to designing real-world, production-ready Microsoft Teams Apps, some new interesting problems arise.
This session is a technical feedback "from the trenches", the story of building a brand new AI-Based product for Microsoft Teams, from start to finish, and will cover:
- Microsoft Graph API (Authentication, Caching, Throttling…)
- Persistence and Analytics
- Live monitoring and performance management
Production debugging is hard, and it’s getting harder. With architectures becoming more distributed and code more asynchronous and reactive, pinpointing and resolving errors that happen in production is no child’s game. This session covers some essential tools and more advanced techniques Scala developers can use to debug live applications and resolve errors quickly. It explores crucial techniques for distributed debugging - and some of the pitfalls that make resolution much harder, and can lead to downtime. The talk also touches on some little-known JVM tools and capabilities that give you super-deep visibility at high scale without making you restart it or attach debuggers.
Towards agile formal methods
The main goal of this work is to overcome the aforementioned limitations by enabling automated decision gates in performance testing of microservices that allow requirements traceability. We seek to achieve this goal by endowing common agile practices used in microservice performance testing, with the ability to automatically learn and then formally verify a performance model of the System Under Test (SUT) to achieve strong assurances of quality. Even if the separation between agile and formal methods increased over the years, we support the claim that formal methods are at a stage where they can be effectively incorporated into agile methods to give them rigorous engineering foundations and make them systematic and effective with strong guarantees.
Presented at Tech Tuesdays in Edinburg, TX. In this presentation, a sample Hangman game made with MeteorJS. Source code available at: github.com/ibolmo/hangmanapp
Data Platform at Twitter: Enabling Real-time & Batch Analytics at ScaleSriram Krishnan
The Data Platform at Twitter supports engineers and data scientists running batch jobs on Hadoop clusters that are several 1000s of nodes, and real-time jobs on top of systems such as Storm. In this presentation, I discuss the overall Data Platform stack at Twitter. In particular, I talk about enabling real-time and batch analytics at scale with the help of Scalding, which is a Scala DSL for batch jobs using MapReduce, Summingbird, which is a framework for combined real-time and batch processing, and Tsar, which is a framework for real-time time-series aggregations.
IBM Rational Host Access Transformation Services (HATS) can dynamically transform your terminal-based applications into rich web pages. It is highly customizable and built on Java EE technology. We'll discuss some lessons learned from a very (very) complex HATS engagement. We'll discuss proper development strategies, and how to distribute workload across team members. We'll introduce a novel approach to unit testing advanced customizations using JUnit, and will also talk about how to address functional testing.
Busy Developers Guide to AngularJS (Tiberiu Covaci)ITCamp
Since last year, Single Page Applications have grown exponentially in popularity and the framework of choice for many developers is Angular.js. In this session we will go through some of the features that make Angular such a popular framework so you can start using it in your own projects.
Patterns for Scalability in Windows Azure Applications (Alex Mang)ITCamp
So you've learned what elasticity means and why it is important to consider scalability in your cloud application architecture. But how can you easily manage your code in order to implement all the theories around scalability?
During this session, I will talk and demo the most common patterns used when designing a cloud application: the valet key pattern, the sharding pattern, the materialized view pattern, the event sourcing pattern and the CQRS pattern.
[DevDay 2016] Real Unit Testing with mocking framework - Speaker: Phat Vu – S...DevDay.org
Why do programmers hate writing Unit Tests? One big reason is object dependency. An object under testing may have dependencies on other complex objects, which might not have been implemented or been complicated when invoking.
Join the session refresh your thinking about Unit Testing and overview of mocking framework, as well as learn some practice/gotcha to write a real Unit Test, how to isolate the behavior of the object you want to test, how to simulate the behavior of the dependencies.
———
Speaker: Phat Vu – Scrum Master at Axon Active Vietnam
SigOpt at MLconf - Reducing Operational Barriers to Model TrainingSigOpt
In this talk at MLconf NYC, Alexandra Johnson, platform engineering lead at SigOpt, discusses common operational challenges with scaling model training and how solutions are designed to
Feedback on building Production-Ready Microsoft Teams AppsGuillaume Meyer
Building mockups and prototypes is fine, but when it comes to designing real-world, production-ready Microsoft Teams Apps, some new interesting problems arise.
This session is a technical feedback "from the trenches", the story of building a brand new AI-Based product for Microsoft Teams, from start to finish, and will cover:
- Microsoft Graph API (Authentication, Caching, Throttling…)
- Persistence and Analytics
- Live monitoring and performance management
Production debugging is hard, and it’s getting harder. With architectures becoming more distributed and code more asynchronous and reactive, pinpointing and resolving errors that happen in production is no child’s game. This session covers some essential tools and more advanced techniques Scala developers can use to debug live applications and resolve errors quickly. It explores crucial techniques for distributed debugging - and some of the pitfalls that make resolution much harder, and can lead to downtime. The talk also touches on some little-known JVM tools and capabilities that give you super-deep visibility at high scale without making you restart it or attach debuggers.
Towards agile formal methods
The main goal of this work is to overcome the aforementioned limitations by enabling automated decision gates in performance testing of microservices that allow requirements traceability. We seek to achieve this goal by endowing common agile practices used in microservice performance testing, with the ability to automatically learn and then formally verify a performance model of the System Under Test (SUT) to achieve strong assurances of quality. Even if the separation between agile and formal methods increased over the years, we support the claim that formal methods are at a stage where they can be effectively incorporated into agile methods to give them rigorous engineering foundations and make them systematic and effective with strong guarantees.
Similar to Hands On: Amazon Mechanical Turk - M. Acosta - ESWC SS 2014 (20)
3. MTurk
Basic
Concepts
(1)
Requester
Worker
Source:
h?ps://requester.mturk.com/tour/how_it_works
• Requester:
creates
and
submits
tasks
to
the
pla9orm.
• Worker:
person
who
solves
the
tasks.
• Human
Intelligence
Task
(HIT):
work
unit.
Amazon
Mechanical
Turk
hands-‐on
session
3
4. MTurk
Basic
Concepts
HIT
(2)
Amazon
Mechanical
Turk
hands-‐on
session
Project
• Project:
HIT
HTML
+
HIT
metadata
• The
elements
that
stay
the
same
in
every
HIT
are
denominated
template
• The
data
that
will
vary
from
HIT
to
HIT
are
specified
via
variables
• NOTE:
If
no
variables
are
specified
in
the
project,
we
will
create
a
single
HIT
• Variables:
allow
creaQng
several
HITs
in
the
project
4
HIT
4
3
HIT
2
HIT
1
5. MTurk
Basic
Concepts
(3)
HIT
4
HIT
3
HIT
2
HIT
1
Batch
Amazon
Mechanical
Turk
hands-‐on
session
Project
• Batch:
Group
of
HITs
created
by
instanQaQng
the
variable(s)
of
a
project
• The
values
of
the
variables
are
specified
in
(CSV,
TSV)
files:
• Each
column
corresponds
to
a
variable
• Each
row
is
an
instance
-‐>
HIT
• Each
file
corresponds
to
a
batch
• We
can
create
several
batches
for
the
same
project
5
6. MTurk
Basic
Concepts
(4)
HIT
4
HIT
3
HIT
2
HIT
1
Batch
HIT
1
HIT
HIT
1
Assig.1
HIT
1
Assig.2
Assignments
Amazon
Mechanical
Turk
hands-‐on
session
Project
Q1
Q2
QuesCons
• HIT:
HIT
1
Assig.3
Work
unit.
The
same
HIT
can
be
solved
by
1
or
more
workers
(assignments)
• Assignment:
How
many
workers
should
solve
one
exact
same
HIT
• QuesCons:
A
single
HIT
may
contain
one
or
several
quesQons
6
7. MTurk
Basic
Concepts
(5)
HIT
4
HIT
3
HIT
2
HIT
1
Batch
HIT
1
HIT
Amazon
Mechanical
Turk
hands-‐on
session
Project
Q1
Q2
QuesCons
HIT
1
Assig.1
HIT
1
Assig.2
HI
T1
Assig.3
Assignments
Total
cost
of
the
project
=
No.
of
HITs
x
No.
of
Assignments
x
(Reward
per
HIT
+
Fee)
7
8. MTurk
Basic
Concepts
(6)
Example
of
Human
Intelligence
Tasks
(HITs)
• Projects
can
be
broken
into
smaller
tasks
called
HITs
• A
HIT
represents
a
single
work
unit
Tagging
(describing)
900
images
Amazon
Mechanical
Turk
hands-‐on
session
Project:
Create
tags
for
image
X1
HIT:
Create
tags
for
image
X2
HIT:
…
No.
of
HITS
=
900
8
9. MTurk
Basic
Concepts
(7)
Example
of
Human
Intelligence
Tasks
(HITs)
• Projects
can
be
broken
into
smaller
tasks
called
HITs
• A
HIT
represents
a
single
work
unit
Tagging
(describing)
900
images
Amazon
Mechanical
Turk
hands-‐on
session
Project:
Create
tags
for
images
X1,
X2,
X3
HIT:
Create
tags
for
image
X4,
X5,
X6
HIT:
…
Several
quesQons
in
a
single
HIT!
No.
of
HITS
=
300
9
10. MTurk
Basic
Concepts
(8)
When
creaQng
a
project
or
individual
HITs,
the
HIT
properCes
must
be
specified:
• General
informaQon:
includes
the
Qtle
and
descripQon
of
the
HIT,
as
well
as
keywords
which
are
used
by
worker
for
searching
HITs
• HIT
duraQon
Qme:
Qme
allo?ed
to
solve
the
HIT
(before
it
is
given
to
another
worker)
• HIT
life
Qme:
how
long
will
the
HIT
be
available
on
the
pla9orm
• #
Assignments:
number
of
different
persons
that
will
perform
the
exact
same
HIT
• Reward:
payment
for
correctly
solving
each
assignment
Amazon
Mechanical
Turk
hands-‐on
session
10
11. MTurk
Workflow
for
Requesters
Amazon
Mechanical
Turk
hands-‐on
session
Project
CreaCon
&
Design
HIT
Test
(Sandbox)
HIT
PublicaQon
Workers
solve
the
HITs
Review
of
the
results
Completed
project
ProducCon
site
11
reject
all
assignments
accepted
12. MTurk
Sandbox
The
Sandbox
is
a
simulated
MTurk
environment
to
test
HITs.
• Log
in
as
requester:
preview
and
test
the
interface
of
your
HITs
– h?ps://requestersandbox.mturk.com
• Log
in
as
worker:
solve
your
own
HITs
to
test
their
funcQonaliQes
and
result
output
–
h?ps://workersandbox.mturk.com
• Best
pracCce:
Always
test
your
HITs
(as
requester
and
worker)
before
publishing
them
in
the
producQon
site
Amazon
Mechanical
Turk
hands-‐on
session
12
13. Managing
HITs
in
MTurk
There
are
three
different
mechanism
to
manage
your
HITs
in
MTurk:
Amazon
Mechanical
Turk
hands-‐on
session
API
Command
Line
Tools
Web
Interface
13
15. Hands
On!
• Project:
Crowdsourcing
DBpedia
triples
to
verify
the
links
to
external
web
pages
prefix
dbpedia-‐ont:<http://dbpedia.org/ontology/>
prefix
foaf:<http://xmlns.com/foaf/0.1/>
SELECT
*
WHERE
{
?s
dbpedia-‐ont:wikiPageExternalLink
?o;
foaf:name
?s_name;
foaf:isPrimaryTopicOf
?s_wikipage
.
Amazon
Mechanical
Turk
hands-‐on
session
15
}
LIMIT
200
Triple
to
crowdsource
Triples
to
build
the
UI
http://dbpedia.org/sparql
MTurkDemo/data/sparql.csv
16. Hands
On!
• Go
to
Mturk
Sandbox
as
a
requester:
– h?ps://requestersandbox.mturk.com/
• Click
on
Sign
In
– Email
address:
own_tp@gmx.li
– Password:
sourcrowd
• Now
we
are
at
“home”
Amazon
Mechanical
Turk
hands-‐on
session
16
17. 1.
Creating
a
Project
Amazon
Mechanical
Turk
hands-‐on
session
Different
predefined
templates:
Select
“other”
17
18. 2.
Setting
up
the
HIT
Properties
(1)
Amazon
Mechanical
Turk
hands-‐on
session
HIT
descripQon
18
19. 2.
Setting
up
the
HIT
Properties
(2)
Very
IMPORTANT:
Set
up
quality
mechanisms
Masters
are
selected
by
default
Amazon
Mechanical
Turk
hands-‐on
session
HIT
properQes
19
20. 3.
Selecting
Qualifications
Worker
requirements
(filters)
Very
IMPORTANT:
Set
up
quality
mechanisms
Masters
are
selected
by
default
HIT
properQes
Amazon
Mechanical
Turk
hands-‐on
session
20
Masters
expect
higher
rewards
MTurk
charges
20%
for
masters
21. 4.
Defining
the
Task
Amazon
Mechanical
Turk
hands-‐on
session
WYSIWYG
HTML
editor
21
22. 4.
Defining
the
Task
(with
Variables)
Template:
Amazon
Mechanical
Turk
hands-‐on
session
elements
that
stay
the
same
in
every
HIT
Variables:
data
that
will
vary
from
HIT
to
HIT.
Are
denoted
as
follows:
${var_name}
22
23. 5.
Previewing
the
Template
This
is
what
the
workers
will
see
Amazon
Mechanical
Turk
hands-‐on
session
23
The
variables
will
be
replaced
by
the
input
data
25. 7.
Previewing
the
HITs
Amazon
Mechanical
Turk
hands-‐on
session
Variables
are
replaced
with
the
data
from
the
input
file
25
26. 8.
Publishing
the
HITs
Amazon
Mechanical
Turk
hands-‐on
session
Summary
of
the
project:
• #
of
HITs
• Rewards
• Total
payment
• Account
balance
26
27. 9.
Retrieving
the
Results
Amazon
Mechanical
Turk
hands-‐on
session
27
29. Project/HIT
Creation
&
Design
(1)
• The
requester
is
able
to
create
projects
or
individual
HITs
• Build
user-‐friendly
interfaces
(using
web
technologies)
• Then,
the
HIT
properCes
must
be
specified:
– General
informaQon:
includes
the
Qtle
and
descripQon
of
the
HIT,
as
well
as
keywords
which
are
used
by
worker
for
searching
HITs.
– HIT
duraQon
Qme:
Qme
allo?ed
to
solve
the
HIT
(before
it
is
given
to
another
worker).
– HIT
life
Qme:
how
long
will
the
HIT
be
available
on
the
pla9orm.
– #
Assignments:
number
of
different
persons
that
will
perform
the
same
HIT.
– Reward:
payment
for
correctly
solving
each
assignment.
Amazon
Mechanical
Turk
hands-‐on
session
29
30. Project/HIT
Creation
&
Design
(2)
• SelecQon
of
MTurk
quality
control
mechanisms:
• These
• High
quality
workers
• Masters
• Photo
moderaQon
masters
• CategorizaQon
masters
• System
qualificaCons
• LocaQon
by
country
• HIT
submission
rate
(%)
• HIT
approval/rejecQon
rate
(%)
• (Absolute)
Number
of
HITs
approved
• QualificaCon
types
• Simply
granted
or
a?ributed
via
customized
tests
filters
are
automaQcally
performed
by
the
pla9orm
Amazon
Mechanical
Turk
hands-‐on
session
Worker
requirements
Masters
expect
higher
rewards
MTurk
charges
20%
for
masters
30
31. HIT
Test
• Best
pracCce:
Always
test
your
HITs
before
publishing
them
1. Perform
technical
tests
(both
as
requester
and
worker)
in
the
MTurk
Sandbox
environment.
Source:
h?ps://requester.mturk.com/developer/sandbox
2. Publish
a
small
subset
of
tasks
in
the
producQon
site
to
test
usability
and
responsiveness.
Amazon
Mechanical
Turk
hands-‐on
session
31
32. Run
live
HITs
• HIT
publicaCon:
– Make
the
HITs
available
to
the
workers
• Review
the
results:
– Monitor
the
submi?ed
assignments
constantly
– Download
the
results
– Accept/reject
assignments,
provide
feedback
when
rejecQng
– Block
spammers
(opQonal)
• Update
HIT/Project:
– Extend/expire
HITs
or
modify
other
HIT
properQes
– Add
addiQonal
assignments
Amazon
Mechanical
Turk
hands-‐on
session
32
33. Lessons
Learned
• Introduce
yourself
on
Worker
forums
(regular
requester)
• Be
responsive
to
workers
– Reply
to
emails
with
quesQons
about
tasks
• Use
monitoring
tools:
– Forums
– TurkopQcon
(Source:h?p://turkopQcon.differenceengines.com/)
34. Choosing
the
Right
Tool
Source:
h?ps://requestersandbox.mturk.com/tour/choose_the_right_tool
Amazon
Mechanical
Turk
hands-‐on
session
34
35. IS
THERE
MORE
THAN
MTURK?
Amazon
Mechanical
Turk
hands-‐on
session
35
37. CrowdFlower
Platform
CrowdFlower
• Client:
CrowdFlower
contributors
Creates
and
submits
jobs
(MTurk
=
requester)
• Contributor:
person
who
solves
the
jobs
(MTurk
=
worker)
• Job:
unit
work
(MTurk
=
task)
Amazon
Mechanical
Turk
hands-‐on
session
37
client
39. Why
CrowdFlower?
(2)
Quality
control
mechanisms
Allows
for
easily
creaQng
a
“gold
standard”,
which
is
further
used
to
detect
low
quality
workers
Amazon
Mechanical
Turk
hands-‐on
session
39
40. Why
CrowdFlower?
(3)
Report
generaCon
and
analyCcs
Amazon
Mechanical
Turk
hands-‐on
session
40
41. Why
NOT
CrowdFlower?
• At
the
beginning,
clients
must
wait
unQl
their
projects
are
approved
by
the
CrowdFlower
staff
before
publishing
them
– Wait
Qme:
From
a
couple
of
hours
up
to
(5)*
days
• Jobs
must
be
specified
in
a
non-‐standard
language:
– CML:
CrowdFlower
Markup
Language
• There
are
certain
configuraCons
that
cannot
be
executed
in
the
pla9orm
*Personal
experience
of
the
presenter
Amazon
Mechanical
Turk
hands-‐on
session
41
42. References
• AMT.
Geyng
Started
Guide.
API
Version
2012-‐03-‐25
h?p://s3.amazonaws.com/awsdocs/MechTurk/latest/amt-‐gsg.pdf
• The
Mechanical
Turk
Blog
h?p://mechanicalturk.typepad.com/
• MTurk
Java
API
h?p://people.csail.mit.edu/gli?le/MTurkJavaAPI/
• CrowdFlower
Pla9orm
h?p://crowdflower.com
Amazon
Mechanical
Turk
hands-‐on
session
42