SlideShare a Scribd company logo
1 of 29
Download to read offline
Who We Are
Jim Jones ( jim.jones1@gmail.com )
Creator of the Turkee gem, a Rails form library that allows a
developer to easily integrate with Mechanical Turk using the
standard Rails form helpers. The gem also makes it easy to
retrieve the data submitted to Mechanical Turk and map those
values back to your existing models.

Mark Percival ( m@mdp.im )
Creator of the RTurk gem, a simple wrapper and library for
Amazon’s Mechanical Turk API.
What is Mechanical Turk
The Amazon Mechanical Turk (MTurk) is one of the suites of
Amazon Web Services, a crowd-sourcing Internet marketplace that
enables computer programmers (known as Requesters) to co-
ordinate the use of human intelligence to perform tasks which
computers are unable to do. The people that perform these tasks
are informally referred to as “Turkers”.
Mechanical Turk Demographics
 http://waxy.org/2008/11/the_faces_of_mechanical_turk/
Ruby, Turkee and Mechanical Turk
Ruby, Turkee and Mechanical Turk
More Demographics
 http://behind-the-enemy-lines.blogspot.com/2010/03/new-
 demographics-of-mechanical-turk.html
 United States: 46.80%
 India: 34.00%
 Miscellaneous: 19.20%
Ruby, Turkee and Mechanical Turk
The Mechanical Turk Interface
 Worker Interface https://www.mturk.com/mturk/findhits?
 match=false
 Requester Interface https://requester.mturk.com/bulk/batches
Common Uses
 Tagging of images
 Collection of survey data
 Sentiment analysis
Creative Uses of Mechanical Turk
   10,000 Sheep All drawn by Turkers
   http://www.thesheepmarket.com/
   Dr. Suessify the News
   http://groups.csail.mit.edu/uid/deneme/?p=671
   Russia to launch 520-day mock mission to Mars ! Oh, The
   Places Russia Will Go!
   Fisherman’s wife breaks the silence ! One Fish Two Fish Sick
   Fish Who Fish
   BP tries again to cap well; protests set to start ! BP tries to
   cap with a hat while protesters start to raise hell about the well
   “Take a Bow” http://nowtakeabow.com/

This isn’t created with Mechanical Turk, but you can envision the
possibilities of utilizing Mechanical Turk workers for the creation of
a piecemeal video such as this.
Iterative Uses for Mechanical Turk
Exploring Iterative and Parallel Human Computation
Processes Greg Little1, Lydia B. Chilton2, Max Goldman1, Robert
C. Miller1

We are interested in human computation processes which
coordinate small contributions from many humans to achieve
larger goals. For example, an algorithm might coordinate many
workers to write a description for an image.
Ruby Integration
 RTurk gem http://github.com/mdp/rturk
 Creating HITs
 h = RTurk::Hit.create(:title => hit_title) do |hit|
   hit.assignments = num_assignments
   hit.description = hit_description
   hit.reward      = reward
   hit.lifetime    = duration.days.seconds.to_i
   hit.question(f_url, :frame_height => HIT_FRAMEHEIGHT)

 hit.qualifications.add :approval_rate, { :gt => 80 } end
 Reviewing HITs
 hits = RTurk::Hit.all_reviewable
 puts “#{hits.size} reviewable hits. n”
 unless hits.empty?
   puts “Reviewing all assignments”
   hits.each do |hit|
     hit.assignments.each do |assignment|
       puts assignment.answers[‘tags’]
       assignment.approve! if assignment.status == ‘Submitted’
     end
   end
 end
Rails Integration Turkee Gem
   Turkee gem http://github.com/aantix/turkee

The Turkee gem builds on top of RTurk providing a simple form
helper for posting/representing model data on Mechanical Turk’s
external servers.

The create_hit method takes a model,
creates a Mechanical Turk hit pointing it at
the new resource URL for the model
passed in. e.g. new_iteration_url
task = Turkee::TurkeeTask.create_hit(host, turkee_task[:hit_title], turkee_task[:
                                     turkee_task[:hit_num_assignments], turkee_ta


The form helper directs the form action
towards the external Turk url and expands
the fields in the given block in the same
manner that Rails’s form_for helper does.




<form accept-charset="UTF-8" action="https://workersandbox.mturk.com/mturk/extern
    <input type="hidden" id="assignmentId" name="assignmentId" value="2G2UC26DG67
    <input type="hidden" id="hitId" name="hitId" value="29E6NG1FS3NYNMBGNC6ZVZL21
    <input type="hidden" id="workerId" name="workerId" value="A3DWBLF5GT7ACT"/>
    <input type="hidden" id="turkSubmitTo" name="turkSubmitTo" value="https%3A%2F
    <input id="iteration_turkee_task_id" name="survey[turkee_task_id]" type="hidd
    <p><textarea cols="40" id="survey_value" name="survey[value]" rows="20"></tex
    <p><input name="commit" type="submit" value="Create" /></p>
</form>
Retrieving Data from Mechanical
Turk
The process_hits method retrieves the user posted data from
Mechanical Turk, reconstructs the data into a param hash (like a
Controller), determines the model for which the data represents,
and creates an entry for that model in the database.

Turkee::TurkeeTask.process_hits(@turkee_task)

rake turkee:get_all_results
Turkee Iterator (updated for Rails
3.1)
Turkee Iterator is a demo built on top of Turkee to sample some of
the capabilities of Mechanical Turk. Allows for flexible HIT requests
from Mechanical Turk.

https://github.com/aantix/turkee_iterator_rails31
Christmas Demos
Gift For Wife
Directions : Should be under $25 Should include a link to the
item on Amazon Should look like I really care.

Either vote for your favorite submission OR give me your own gift
idea (and Amazon link) for a chance to win a bonus 50 cents.

Thanks for the help!

http://turkee31.herokuapp.com/iterations?id=9
Difficult Dilemma
Directions: I’m having difficulty telling my kids that Santa doesn’t
really exist. On top of that, they’e been terrible this past year.

Please write a short letter to my kids from Santa explaining why he
can no longer give them gifts anymore(any reason will do). I will
transcribe the best one it and leave it in their stockings on
Christmas morning.

Merry Christmas!

http://turkee31.herokuapp.com/iterations?id=11
Let’s Do a Realtime Example, Say
Hello
http://turkee31.herokuapp.com/turkee/turkee_tasks/new

   Please say “Hello” to my colleagues and tell us where you’re
   from.
   I am demo’ing Mechanical Turk to the Ruby SF meetup group.
   Please tell send us a creative “hello” and where you’re from.
   A one or two line introduction would be nice. Tell us how the
   weather is in your city. Tell us about your family. Or just simply
   say hello.
   0.05
   100
   5
   It’s mostly sunny at 65 degrees. Hello from San Mateo, CA!
More Content Creation
Mechanical Turk is an untapped
market in terms of getting content
created for your application. Start
thinking outside the context of
scientific surveys and fact finding
and start thinking of this resource
as a pool of human workers with
emotions, opinions, and stories to
tell.
E.g.

   Iterative story telling where each Turker adds a piece to a
   never ending story.
   Have the Turker give their opinion on a topic or product and
   then have other Turkers respond to that opinion.
   Have the Turker describe some emotionally happy or painful
   point in their life. Ask emotionally charged question to have the
   Turker reveal the essence of who they are.
Example: Obesity Testimonials
(Design of Questions)
Upload a picture of yourself of when you
first considered yourself “fat”.




When the above picture was taken, were you aware that
you had been gaining weight? Describe the feeling you had
when you first glanced at the picture and noticed your
weight gain. Did anyone else notice or make snide
comments? How did you feel?

“As a child I was pretty average but I was lead to believe that I
was fat. I would request simple items, things that are status
symbols to the average kid, like jeans. These small requests would
be shot down, my mother telling me that I was too big to fit into a
pair of jeans. I remember her telling me that when I was eight. It
started a cycle in my life where I felt that I was too large, too fat,
for the things other children took for granted.”
Other Concepts
Micro Bonuses
The concept where the user is awarded a varying amount
depending upon how much time they take to answer a question or
how accurate it (relative to the other user answers)

E.g. Maybe there’s a trivia game where you record how long it
takes each user to answer a question and the fastest response for
the day gets 10 cents instead of 5.
Gambling
Allow users to gamble their potential pay.

E.g. Sentiment analysis.

Show the user all of the recent headlines for a given stock, stock
charts, etc.

You state that they can either take their payout now, OR, if they
think that the stock will close out higher than it is now at the end
of the day, they can “gamble”, and will be awarded the fee PLUS a
bonus at the end of the day. If it closes down from the point at
where it is, they receive nothing.

Does a future, larger payout increase the quality of data analysis?
Interesting Research Findings
Financial Incentives and the
Performance of Crowds
Winter Mason, Yahoo! Research

Figure 2 reveals two main findings: first, that across all difficulty
levels participants chose to complete more tasks on average when
the pay was higher (F(3,607) = 15.73, p < 0.001); and second,
that across all payment levels, the number of completed tasks
decreased with increasing difficulty.

As Figure 3 indicates, however, increasing compensation did
not improve accuracy, which we measured in two ways…
Toward Automatic Task Design: A
Progress Report
Eric Huang, School of Engineering and Applied Sciences
Harvard University
Task Search in a Human
Computation Market
   Lydia B. Chilton, University of Washington **

We found strong evidence that Turkers sort by the most number of
HITs available (so they can find one task, and then do 100
instances of them in a row) and the most recently posted HITS (so
they get the latest and greatest HITs).

More Related Content

Similar to Ruby, Turkee and Mechanical Turk

Cloudera Data Science Challenge 3 Solution by Doug Needham
Cloudera Data Science Challenge 3 Solution by Doug NeedhamCloudera Data Science Challenge 3 Solution by Doug Needham
Cloudera Data Science Challenge 3 Solution by Doug NeedhamDoug Needham
 
Idea2app
Idea2appIdea2app
Idea2appFlumes
 
What Are We Still Doing Wrong
What Are We Still Doing WrongWhat Are We Still Doing Wrong
What Are We Still Doing Wrongafa reg
 
Sports Day Essay - Jabarictx. Online assignment writing service.
Sports Day Essay - Jabarictx. Online assignment writing service.Sports Day Essay - Jabarictx. Online assignment writing service.
Sports Day Essay - Jabarictx. Online assignment writing service.Melody Rios
 
The Between Math And Sports
The Between Math And SportsThe Between Math And Sports
The Between Math And SportsBrandy Grant
 
How To Write A Compare And Contrast Essay Example
How To Write A Compare And Contrast Essay ExampleHow To Write A Compare And Contrast Essay Example
How To Write A Compare And Contrast Essay ExampleJennifer Hellmuth
 
Money Can Buy Happiness Essay Introduction
Money Can Buy Happiness Essay IntroductionMoney Can Buy Happiness Essay Introduction
Money Can Buy Happiness Essay IntroductionMimi Young
 
Prescriptive Motor Programs
Prescriptive Motor ProgramsPrescriptive Motor Programs
Prescriptive Motor ProgramsTania Knapp
 
Assignment 1 Studio Assignment
Assignment 1 Studio AssignmentAssignment 1 Studio Assignment
Assignment 1 Studio AssignmentVickie Western
 
Bridging Silos Between SEO, UX, and Content for Big Marketing Wins | #DSCHI
Bridging Silos Between SEO, UX, and Content for Big Marketing Wins | #DSCHIBridging Silos Between SEO, UX, and Content for Big Marketing Wins | #DSCHI
Bridging Silos Between SEO, UX, and Content for Big Marketing Wins | #DSCHIRebekah Baggs
 
Doing More with Less: Automated, High-Quality Content Generation
Doing More with Less: Automated, High-Quality Content GenerationDoing More with Less: Automated, High-Quality Content Generation
Doing More with Less: Automated, High-Quality Content GenerationHamlet Batista
 
Adding data sources to the reporter
Adding data sources to the reporterAdding data sources to the reporter
Adding data sources to the reporterRogan Hamby
 
With Great Nerdery Comes Great Responsibility
With Great Nerdery Comes Great Responsibility With Great Nerdery Comes Great Responsibility
With Great Nerdery Comes Great Responsibility John Anderson
 
Assignment 1.1 Personal Power Profile
Assignment 1.1 Personal Power ProfileAssignment 1.1 Personal Power Profile
Assignment 1.1 Personal Power ProfileMelissa Grant
 
Metadata in a Crowd: Shared Knowledge Production
Metadata in a Crowd: Shared Knowledge ProductionMetadata in a Crowd: Shared Knowledge Production
Metadata in a Crowd: Shared Knowledge ProductionKevin Rundblad
 
Operation Analytics and Investigating Metric Spike (1).pdf
Operation Analytics and Investigating Metric Spike (1).pdfOperation Analytics and Investigating Metric Spike (1).pdf
Operation Analytics and Investigating Metric Spike (1).pdfVaibhaviKhedekar1
 
Progress of JavaScript Architecture
Progress of JavaScript ArchitectureProgress of JavaScript Architecture
Progress of JavaScript ArchitectureTonya Mork
 

Similar to Ruby, Turkee and Mechanical Turk (20)

Cloudera Data Science Challenge 3 Solution by Doug Needham
Cloudera Data Science Challenge 3 Solution by Doug NeedhamCloudera Data Science Challenge 3 Solution by Doug Needham
Cloudera Data Science Challenge 3 Solution by Doug Needham
 
Idea2app
Idea2appIdea2app
Idea2app
 
What Are We Still Doing Wrong
What Are We Still Doing WrongWhat Are We Still Doing Wrong
What Are We Still Doing Wrong
 
Sports Day Essay - Jabarictx. Online assignment writing service.
Sports Day Essay - Jabarictx. Online assignment writing service.Sports Day Essay - Jabarictx. Online assignment writing service.
Sports Day Essay - Jabarictx. Online assignment writing service.
 
Gaps in the algorithm
Gaps in the algorithmGaps in the algorithm
Gaps in the algorithm
 
The Between Math And Sports
The Between Math And SportsThe Between Math And Sports
The Between Math And Sports
 
How To Write A Compare And Contrast Essay Example
How To Write A Compare And Contrast Essay ExampleHow To Write A Compare And Contrast Essay Example
How To Write A Compare And Contrast Essay Example
 
Money Can Buy Happiness Essay Introduction
Money Can Buy Happiness Essay IntroductionMoney Can Buy Happiness Essay Introduction
Money Can Buy Happiness Essay Introduction
 
Prescriptive Motor Programs
Prescriptive Motor ProgramsPrescriptive Motor Programs
Prescriptive Motor Programs
 
Assignment 1 Studio Assignment
Assignment 1 Studio AssignmentAssignment 1 Studio Assignment
Assignment 1 Studio Assignment
 
Bridging Silos Between SEO, UX, and Content for Big Marketing Wins | #DSCHI
Bridging Silos Between SEO, UX, and Content for Big Marketing Wins | #DSCHIBridging Silos Between SEO, UX, and Content for Big Marketing Wins | #DSCHI
Bridging Silos Between SEO, UX, and Content for Big Marketing Wins | #DSCHI
 
Doing More with Less: Automated, High-Quality Content Generation
Doing More with Less: Automated, High-Quality Content GenerationDoing More with Less: Automated, High-Quality Content Generation
Doing More with Less: Automated, High-Quality Content Generation
 
Adding data sources to the reporter
Adding data sources to the reporterAdding data sources to the reporter
Adding data sources to the reporter
 
The Phase Diagram
The Phase DiagramThe Phase Diagram
The Phase Diagram
 
With Great Nerdery Comes Great Responsibility
With Great Nerdery Comes Great Responsibility With Great Nerdery Comes Great Responsibility
With Great Nerdery Comes Great Responsibility
 
Data Science Machine
Data Science Machine Data Science Machine
Data Science Machine
 
Assignment 1.1 Personal Power Profile
Assignment 1.1 Personal Power ProfileAssignment 1.1 Personal Power Profile
Assignment 1.1 Personal Power Profile
 
Metadata in a Crowd: Shared Knowledge Production
Metadata in a Crowd: Shared Knowledge ProductionMetadata in a Crowd: Shared Knowledge Production
Metadata in a Crowd: Shared Knowledge Production
 
Operation Analytics and Investigating Metric Spike (1).pdf
Operation Analytics and Investigating Metric Spike (1).pdfOperation Analytics and Investigating Metric Spike (1).pdf
Operation Analytics and Investigating Metric Spike (1).pdf
 
Progress of JavaScript Architecture
Progress of JavaScript ArchitectureProgress of JavaScript Architecture
Progress of JavaScript Architecture
 

Recently uploaded

UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2DianaGray10
 
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - TechWebinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - TechProduct School
 
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENTSIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENTxtailishbaloch
 
IT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingIT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingMAGNIntelligence
 
Flow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameFlow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameKapil Thakar
 
Oracle Database 23c Security New Features.pptx
Oracle Database 23c Security New Features.pptxOracle Database 23c Security New Features.pptx
Oracle Database 23c Security New Features.pptxSatishbabu Gunukula
 
Stobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through TokenizationStobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through TokenizationStobox
 
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptxEmil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptxNeo4j
 
The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)codyslingerland1
 
Top 10 Squarespace Development Companies
Top 10 Squarespace Development CompaniesTop 10 Squarespace Development Companies
Top 10 Squarespace Development CompaniesTopCSSGallery
 
UiPath Studio Web workshop series - Day 1
UiPath Studio Web workshop series  - Day 1UiPath Studio Web workshop series  - Day 1
UiPath Studio Web workshop series - Day 1DianaGray10
 
Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.IPLOOK Networks
 
The Importance of Indoor Air Quality (English)
The Importance of Indoor Air Quality (English)The Importance of Indoor Air Quality (English)
The Importance of Indoor Air Quality (English)IES VE
 
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdfQ4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdfTejal81
 
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptxGraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptxNeo4j
 
3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud DataEric D. Schabell
 
How to release an Open Source Dataweave Library
How to release an Open Source Dataweave LibraryHow to release an Open Source Dataweave Library
How to release an Open Source Dataweave Libraryshyamraj55
 
My key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAIMy key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAIVijayananda Mohire
 
Novo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4jNovo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4jNeo4j
 
March Patch Tuesday
March Patch TuesdayMarch Patch Tuesday
March Patch TuesdayIvanti
 

Recently uploaded (20)

UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2
 
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - TechWebinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
 
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENTSIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
 
IT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingIT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced Computing
 
Flow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameFlow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First Frame
 
Oracle Database 23c Security New Features.pptx
Oracle Database 23c Security New Features.pptxOracle Database 23c Security New Features.pptx
Oracle Database 23c Security New Features.pptx
 
Stobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through TokenizationStobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
 
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptxEmil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
 
The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)
 
Top 10 Squarespace Development Companies
Top 10 Squarespace Development CompaniesTop 10 Squarespace Development Companies
Top 10 Squarespace Development Companies
 
UiPath Studio Web workshop series - Day 1
UiPath Studio Web workshop series  - Day 1UiPath Studio Web workshop series  - Day 1
UiPath Studio Web workshop series - Day 1
 
Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.
 
The Importance of Indoor Air Quality (English)
The Importance of Indoor Air Quality (English)The Importance of Indoor Air Quality (English)
The Importance of Indoor Air Quality (English)
 
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdfQ4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
 
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptxGraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
 
3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data
 
How to release an Open Source Dataweave Library
How to release an Open Source Dataweave LibraryHow to release an Open Source Dataweave Library
How to release an Open Source Dataweave Library
 
My key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAIMy key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAI
 
Novo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4jNovo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4j
 
March Patch Tuesday
March Patch TuesdayMarch Patch Tuesday
March Patch Tuesday
 

Ruby, Turkee and Mechanical Turk

  • 1. Who We Are Jim Jones ( jim.jones1@gmail.com ) Creator of the Turkee gem, a Rails form library that allows a developer to easily integrate with Mechanical Turk using the standard Rails form helpers. The gem also makes it easy to retrieve the data submitted to Mechanical Turk and map those values back to your existing models. Mark Percival ( m@mdp.im ) Creator of the RTurk gem, a simple wrapper and library for Amazon’s Mechanical Turk API.
  • 2. What is Mechanical Turk The Amazon Mechanical Turk (MTurk) is one of the suites of Amazon Web Services, a crowd-sourcing Internet marketplace that enables computer programmers (known as Requesters) to co- ordinate the use of human intelligence to perform tasks which computers are unable to do. The people that perform these tasks are informally referred to as “Turkers”.
  • 3. Mechanical Turk Demographics http://waxy.org/2008/11/the_faces_of_mechanical_turk/
  • 6. More Demographics http://behind-the-enemy-lines.blogspot.com/2010/03/new- demographics-of-mechanical-turk.html United States: 46.80% India: 34.00% Miscellaneous: 19.20%
  • 8. The Mechanical Turk Interface Worker Interface https://www.mturk.com/mturk/findhits? match=false Requester Interface https://requester.mturk.com/bulk/batches
  • 9. Common Uses Tagging of images Collection of survey data Sentiment analysis
  • 10. Creative Uses of Mechanical Turk 10,000 Sheep All drawn by Turkers http://www.thesheepmarket.com/ Dr. Suessify the News http://groups.csail.mit.edu/uid/deneme/?p=671 Russia to launch 520-day mock mission to Mars ! Oh, The Places Russia Will Go! Fisherman’s wife breaks the silence ! One Fish Two Fish Sick Fish Who Fish BP tries again to cap well; protests set to start ! BP tries to cap with a hat while protesters start to raise hell about the well “Take a Bow” http://nowtakeabow.com/ This isn’t created with Mechanical Turk, but you can envision the possibilities of utilizing Mechanical Turk workers for the creation of a piecemeal video such as this.
  • 11. Iterative Uses for Mechanical Turk Exploring Iterative and Parallel Human Computation Processes Greg Little1, Lydia B. Chilton2, Max Goldman1, Robert C. Miller1 We are interested in human computation processes which coordinate small contributions from many humans to achieve larger goals. For example, an algorithm might coordinate many workers to write a description for an image.
  • 12. Ruby Integration RTurk gem http://github.com/mdp/rturk Creating HITs h = RTurk::Hit.create(:title => hit_title) do |hit| hit.assignments = num_assignments hit.description = hit_description hit.reward = reward hit.lifetime = duration.days.seconds.to_i hit.question(f_url, :frame_height => HIT_FRAMEHEIGHT) hit.qualifications.add :approval_rate, { :gt => 80 } end Reviewing HITs hits = RTurk::Hit.all_reviewable puts “#{hits.size} reviewable hits. n” unless hits.empty? puts “Reviewing all assignments” hits.each do |hit| hit.assignments.each do |assignment| puts assignment.answers[‘tags’] assignment.approve! if assignment.status == ‘Submitted’ end end end
  • 13. Rails Integration Turkee Gem Turkee gem http://github.com/aantix/turkee The Turkee gem builds on top of RTurk providing a simple form helper for posting/representing model data on Mechanical Turk’s external servers. The create_hit method takes a model, creates a Mechanical Turk hit pointing it at the new resource URL for the model passed in. e.g. new_iteration_url task = Turkee::TurkeeTask.create_hit(host, turkee_task[:hit_title], turkee_task[: turkee_task[:hit_num_assignments], turkee_ta The form helper directs the form action towards the external Turk url and expands the fields in the given block in the same manner that Rails’s form_for helper does. <form accept-charset="UTF-8" action="https://workersandbox.mturk.com/mturk/extern <input type="hidden" id="assignmentId" name="assignmentId" value="2G2UC26DG67 <input type="hidden" id="hitId" name="hitId" value="29E6NG1FS3NYNMBGNC6ZVZL21 <input type="hidden" id="workerId" name="workerId" value="A3DWBLF5GT7ACT"/> <input type="hidden" id="turkSubmitTo" name="turkSubmitTo" value="https%3A%2F <input id="iteration_turkee_task_id" name="survey[turkee_task_id]" type="hidd <p><textarea cols="40" id="survey_value" name="survey[value]" rows="20"></tex <p><input name="commit" type="submit" value="Create" /></p> </form>
  • 14. Retrieving Data from Mechanical Turk The process_hits method retrieves the user posted data from Mechanical Turk, reconstructs the data into a param hash (like a Controller), determines the model for which the data represents, and creates an entry for that model in the database. Turkee::TurkeeTask.process_hits(@turkee_task) rake turkee:get_all_results
  • 15. Turkee Iterator (updated for Rails 3.1) Turkee Iterator is a demo built on top of Turkee to sample some of the capabilities of Mechanical Turk. Allows for flexible HIT requests from Mechanical Turk. https://github.com/aantix/turkee_iterator_rails31
  • 17. Gift For Wife Directions : Should be under $25 Should include a link to the item on Amazon Should look like I really care. Either vote for your favorite submission OR give me your own gift idea (and Amazon link) for a chance to win a bonus 50 cents. Thanks for the help! http://turkee31.herokuapp.com/iterations?id=9
  • 18. Difficult Dilemma Directions: I’m having difficulty telling my kids that Santa doesn’t really exist. On top of that, they’e been terrible this past year. Please write a short letter to my kids from Santa explaining why he can no longer give them gifts anymore(any reason will do). I will transcribe the best one it and leave it in their stockings on Christmas morning. Merry Christmas! http://turkee31.herokuapp.com/iterations?id=11
  • 19. Let’s Do a Realtime Example, Say Hello http://turkee31.herokuapp.com/turkee/turkee_tasks/new Please say “Hello” to my colleagues and tell us where you’re from. I am demo’ing Mechanical Turk to the Ruby SF meetup group. Please tell send us a creative “hello” and where you’re from. A one or two line introduction would be nice. Tell us how the weather is in your city. Tell us about your family. Or just simply say hello. 0.05 100 5 It’s mostly sunny at 65 degrees. Hello from San Mateo, CA!
  • 21. Mechanical Turk is an untapped market in terms of getting content created for your application. Start thinking outside the context of scientific surveys and fact finding and start thinking of this resource as a pool of human workers with emotions, opinions, and stories to tell. E.g. Iterative story telling where each Turker adds a piece to a never ending story. Have the Turker give their opinion on a topic or product and then have other Turkers respond to that opinion. Have the Turker describe some emotionally happy or painful point in their life. Ask emotionally charged question to have the Turker reveal the essence of who they are.
  • 22. Example: Obesity Testimonials (Design of Questions) Upload a picture of yourself of when you first considered yourself “fat”. When the above picture was taken, were you aware that you had been gaining weight? Describe the feeling you had when you first glanced at the picture and noticed your weight gain. Did anyone else notice or make snide comments? How did you feel? “As a child I was pretty average but I was lead to believe that I was fat. I would request simple items, things that are status symbols to the average kid, like jeans. These small requests would be shot down, my mother telling me that I was too big to fit into a pair of jeans. I remember her telling me that when I was eight. It started a cycle in my life where I felt that I was too large, too fat, for the things other children took for granted.”
  • 24. Micro Bonuses The concept where the user is awarded a varying amount depending upon how much time they take to answer a question or how accurate it (relative to the other user answers) E.g. Maybe there’s a trivia game where you record how long it takes each user to answer a question and the fastest response for the day gets 10 cents instead of 5.
  • 25. Gambling Allow users to gamble their potential pay. E.g. Sentiment analysis. Show the user all of the recent headlines for a given stock, stock charts, etc. You state that they can either take their payout now, OR, if they think that the stock will close out higher than it is now at the end of the day, they can “gamble”, and will be awarded the fee PLUS a bonus at the end of the day. If it closes down from the point at where it is, they receive nothing. Does a future, larger payout increase the quality of data analysis?
  • 27. Financial Incentives and the Performance of Crowds Winter Mason, Yahoo! Research Figure 2 reveals two main findings: first, that across all difficulty levels participants chose to complete more tasks on average when the pay was higher (F(3,607) = 15.73, p < 0.001); and second, that across all payment levels, the number of completed tasks decreased with increasing difficulty. As Figure 3 indicates, however, increasing compensation did not improve accuracy, which we measured in two ways…
  • 28. Toward Automatic Task Design: A Progress Report Eric Huang, School of Engineering and Applied Sciences Harvard University
  • 29. Task Search in a Human Computation Market Lydia B. Chilton, University of Washington ** We found strong evidence that Turkers sort by the most number of HITs available (so they can find one task, and then do 100 instances of them in a row) and the most recently posted HITS (so they get the latest and greatest HITs).