Talk at AAAI Human Computation 2013 Workshop on Scaling Speech, Language Understanding and Dialogue through Crowdsourcing (November 9, 2013): http://faculty.washington.edu/mtjalve/HCOMP2013.Workshop.html
Crowdsourcing Transcription Beyond Mechanical Turk
Beyond Mechanical Turk
Denys Baskov, Matthew Lease
School of Information
University of Texas at Austin
• Natural Speech: Opportunity & Challenge
• Strengths & Limitations of AMT research
– e.g. AMT-based Transcription
• Qualitative review of 8 transcription providers
• Quantitative evaluation of 4 providers
• Observations & Contributions
The Rise of Stored Natural Speech
• Conversational speech is the most ubiquitous
form of human communication on the planet
• We can now capture & store our
conversations in new ways & at massive scale
• But… need effective technology to search
massive conversational speech archives
• Oard: “Unlocking the Potential of the Spoken Word”
oh i'll you know are yeah yeah yeah yeah yeah yeah yeah
the very why don't we start with you saying anything in your
about grandparents great grandparents well as a small
child i remember only one of my grandfathers and his wife
his second wife he was selling flour and the type of
business it was he didn't even have a store he just a few
sacks of different flour and the entrance of an apartment
building and people would pass by everyday and buy a
chela but two killers of flour we have to remember related
times were there was no already baked bread so people
had to baked her own bread all the time for some strange
reason i do remember fresh rolls where everyone would
buy every day but not the bread so that was the business
that's how he made a living where was this was the name
of the town it wasn't shammay dish he ours is we be and
why i as i know in southern poland and alisa are close
Perfect ASR: Raw Transcription
I never left new York before I didn't know anything else
so some fellow I knew he said I have a friend that lives
in Tucson Arizona so I went to the map looked it up I
never heard of Tucson he says I'll write him a letter and
when you go there you could stay with him so he did he
wrote a letter and his friend he was a dentist he invited
me to come over there and spend a week with him
[so I didn't] * I never left New York before.
I didn't know anything else.
So some fellow I knew [mentioned that] <uh> * he said I have
a friend that lives [in Arizona] * in Tucson Arizona.
So I went to the map looked it up.
<um> I never heard of Tucson.
<uh and anyhow> He says <well> I'll write him a letter and
when you go there you could <uh> stay with him.
So he did.
He wrote a letter.
And his friend, he was a dentist.
He invited me to come over there and spend a week with him.
Transcription Research via AMT
Audhkhasi et al. (2011)
Evanini et al. (2010)
Gruenstein et al. (2009)
Lee et al. (2011)
Marge et al. (2010)
Novotney et al. (2010)
Parent et al. (2010)
Williams et al. (2011)
Why Eytan Adar hates MTurk
Research (CHI 2011 CHC Workshop)
• Overly-narrow focus on MTurk
– Identify general vs. platform-specific problems
– Academic vs. Industrial problems
• How much should we focus on “...writing
the user’s manual for MTurk ... struggl[ing]
against the limits of the platform...”?
HCOMP 2013 Panel
Anand Kulkarni: “How do we
dramatically reduce the complexity of
getting work done with the crowd?”
Greg Little: How can we post a task
and with 98% confidence know we’ll
get a quality result?
Beyond AMT: An Analysis of
Crowd Work Platforms
• Vakharia & Lease, arXiv online 2013
• Near-exclusive research focus on AMT
risks its particular vagaries and limitations
overly shaping our understanding of crowd
work and the research questions and
directions being pursued.
• We present a cross-platform content
analysis of seven crowd work platforms.
Errors Distribution in WER
• Management costs beyond Base Price
– Crowdsourcing studies rarely discuss other
costs (other costs dwarf crowd costs…)
• CW, TH and VF's price higher than oDesk
• But… oDesk: no management cost in the
price rate, but additional effort was needed
– communicate with workers to negotiate price
– clarify requirements, and monitor work
– take risk of low quality or late/no delivery
• Snapshot in time of current crowdsourcing
transcription providers & offerings beyond AMT
– Those looking for alternatives today
– Retrospective studies
• Quantitative WER vs. cost for spontaneous
speech transcription across providers
• Discussion of tradeoffs among quality, cost,
risk & effort in crowdsourcing transcription