Wrong confirmation ID

Email
Favorite
Favorited ×
Download
Embed
Private Content
Copy and paste this code into your blog or website Copy Customize… Close
We have emailed the verification/download link to "".
Login to your email and click the link to download the file directly.

To request the link at a different email address, update it here. Close
Validation messages. Success message. Fail message.

Check your bulk/spam folders if you can't find our mail.

Tweet

Loading…

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

Nishimoto Interspeech 2010 v3

by Takuya Nishimoto on Sep 23, 2010

503 views

The comparison between the Deletion-Based Method and the Mixing-Based Method for Audio CAPTCHAs

The comparison between the Deletion-Based Method and the Mixing-Based Method for Audio CAPTCHAs

Interspeech 2010, Mon-Ses2-P3, Makuhari, Tokyo

Accessibility

View text version

Categories

Technology

Upload Details

Uploaded via SlideShare as Adobe PDF

Usage Rights

© All Rights Reserved

File a copyright complaint

Statistics

Favorites: 0
Downloads: 2
Comments: 0
Embed Views: 0
Views on SlideShare: 503
Total Views: 503

Nishimoto Interspeech 2010 v3 — Presentation Transcript

The comparison between the Deletion-Based Method and the Mixing-Based Method for Audio CAPTCHAs Takuya NISHIMOTO (Univ. Tokyo, Japan) Takayuki WATANABE (TWCU, Japan) Interspeech 2010 Mon-Ses2-P3 1
CAPTCHA  Completely Automated Public Turing test to tell Computers and Humans Apart  popular security techniques on the Web  prevent automated programs from abusing  image-based CAPTCHAs  image containing distorted characters  preventing use of persons with visual disability  audio CAPTCHAs were created  create better audio CAPTCHA tasks  safeness: the difference of recognition performance  usability: mental workload of human in listening speech 2
Performance gap model  performance of machine should be lower  than the intelligibility of human  gap: safeness 100  should be large Human Intelligibility (%)  exposed ratio (ER)  0%: random answer ASR  chance-level; no gap  100%: best guess  easy for both; no gap  practical condition  0 < ER < 100 0 Exposed Ratio (%) 100 (Provided Information) 3
Safeness: ER control  machine is becoming strong  statistical ASR method is the mainstream  supervised machine learning (Hidden Markov Models)  techniques to cope with the noise  CAPTCHA tasks should be created systematically  it should not be created by trial and error  controllability of Exposed Ratio is essential  Mixing-based method: best way to control ER?  mixing noises / distorting signals  can hide portion of information, however...  difficult to measure the ER, performance is not easy to predict  alternatives must be investigated 4
Usability: Mental workload  CAPTCHAs should not increase mental workload  the workload may increase, if they are..  difficult to listen / memorize the task  long task (many characters)  difficult to remember  safer, but higher mental workload  requirements  information can be obtained in short time, easily  investigation required  human auditory sensation  language cognition 5
Top-down knowledge  incomplete stimulus  knowledge helps to guess the information  visual sensation  if part of image is missing, or part of the word is hidden  common knowledge can complement image  about the character and the vocabulary  speech perception  if "word familiarity" is high: easy to guess  phonemic restoration  may help the human listening 6
Deletion-based method  delete some parts on temporal axis little by little  if every 30 msec over a period of 100 msec is replaced with silence, the 30% of the information was deleted (D70)  if the ratio of remained sections go down, the degree of listening difficulty may increase.  Exposed Ratio can be controlled easily  however, not easy to understand.... deletion (original) Festival engine KAL (HMM-based) 7
Phonemic restoration  interrupted speech and noise maskers combined  the fence effect  continuity of speech signal perceived  may help human listening  does not affect machine performance  expected to enlarge the gap  performance difference of human and machine deletion + phonemic restoration 8
NASA-TLX evaluation  mental workload  rating 6 subscales  Mental, Physical, and Temporal Demands, Frustration, Effort, and Performance  range: 0-100  weights of subscales (6-1)  for each participant  placing an order how the 6 dimensions are related to personal definition of workload  weighted workload (WWL) 9
Deletion vs Mixing (Exp1)  objective: compare intelligibility and mental workload  Deletion-Based Method (DBM)  Mixing-Based Method (MBM)  effect of SNR (signal-to-noise ratio) in MBM  human intelligibility test  75 utterances: 3,4,5 digits numbers (3 x 25)  Japanese recorded speech  subjects: 15 (5 x 3) undergraduate students  mental workload (WWL) by NASA-TLX  normalized within every subject  their average and SD become 50 and 10 respectively  automatic speech recognition using HMM  task: numbers (1-7 digits) in Japanese  training: 8440 utterances, 18 states, 20 mixtures  evaluation: 1001 utterances, sentence recognition 10
Setup (Exp1)  compare DBM and MBM within a person  acoustic presentation: given by headphone  at the subject’s preferred reference loudness level  MBM disturbing signals  utterances of Japanese sentences fragmented as short periods, shuffled and combined MBM(Exp1): Sentence Group Trial 1: D30 Trial 2: M0, Mm10, Mm20 recognition using HTK (%) 80 G1 DBM 30% MBM SNR 0dB 60 G2 DBM 30% MBM SNR -10dB 40 G3 DBM 30% MBM SNR -20dB 20 0 M0 Mm10 Mm20 11
Performance (Exp1) DBM(T1)：marginally significant (p<0.1) (G1>G2) DBM 30% task is harder than MBM 0dB, -10dB, -20dB MBM(T2): effect of SNR conditions is significant, however, only between 0dB & -10dB (p<0.05) (G1>G2) DBM 30% vs DBM 30% vs DBM 30% vs 100 MBM 0dB MBM -10dB MBM -20dB 90 80 70 60 50 40 T1 T2 30 s101 s102 s103 s104 s105 s201 s202 s203 s204 s205 s301 s302 s303 s304 s305 12
Workload diffefence (Exp1)  WWL: individual difference cancelled  subtraction of DBM (D30) score from MBM (M0, Mm10 and Mm20) score was performed DBM 30% vs DBM 30% vs DBM 30% vs MBM 0dB MBM -10dB MBM -20dB 20 10 0 s101 s102 s103 s104 s105 s201 s202 s203 s204 s205 s301 s302 s303 s304 s305 -10 -20 average WWL difference -30 20 -40 0.7 1.0 0 -50 WWL: MBM 0db < DBM 30% ? -60 -20 no significance (ANOVA) (16.2) M0-D30 Mm10-D30 Mm20-D30 MBM: task difficulty is not easy to control 13
DBM exposed ratio (Exp2)  DBM: Exposed Ratio can control the gap size 100 70 90 Workload 60 80 70 50 60 50 Human Ave. (%) 40 40 Machine (%) 30 30 30% 50% 70% 30% 50% 70% DBM 30% gap is very large, however, Significant difference (p<0.05) workload is very high. 14
Discussion  D30 (DBM) & Mm10 (MBM) can be the benchmarks  for the purpose of comparison between MBM and DBM  performance difference are close (43.7pt & 44.8pt)  WWL are also very close (WMm10 - WD30 = 0.7) performance difference between human and machine (pt) 80 60 40 20 0 M0 Mm10 Mm20 D70 D50 D30 15
Conclusion  audio CAPTCHA task using phonemic restoration  deletion-based method (DBM)  evaluation of CAPTCHA task  performance + mental workload (NASA-TLX)  comparison between DBM and MBM  DBM: easier to control the task  future works  improve the noise  investigation of phonemic restoration  really improving performance? only decreasing workload?  word familiarity, speech rate, synthesized speech, ... 16