On the Viability of CAPTCHAs for Use
in Telephony Systems: A Usability
Field Study
Niharika	
  Sachdeva*,	
  Nitesh	
  Sax...
Overview
– MoDvaDon	
  	
  
– Research	
  quesDon	
  
– Study	
  Design	
  
– Experimental	
  setup	
  
– ParDcipants...
Some Attacks

3	
  
CAPTCHA
– Completely	
  Automated	
  Public	
  Turing	
  Test	
  to	
  
tell	
  Computers	
  and	
  Humans	
  Apart	
  
	...
Is it Really Useful ??
– FrustraDng	
  	
  	
  	
  	
  	
  	
  	
  	
  
– Lack	
  of	
  incenDve	
  	
  	
  	
  	
  
– ...
But CAPTCHA continues to Rule

CAPTCHA	
  for	
  RoboCalls	
  

6	
  
But CAPTCHA continues to Rule

CAPTCHA	
  for	
  RoboCalls	
  

7	
  
Audio CAPTCHA a solution?
– 	
  Yahoo	
  
	
  	
  	
  
– 	
  Google	
  
	
  
– Patent	
  CAPTCHA	
  

8	
  
Research Question
– QuanDfy	
  the	
  amount	
  of	
  inconvenience	
  
CAPTCHA	
  causes	
  to	
  users.	
  
– How	
  d...
Study Design

LaDn	
  	
  
Square	
  

Polakis,	
  G.	
  Kontaxis,	
  and	
  S.	
  Ioannidis.	
  CAPTCHuring	
  Automated	...
CAPTCHA Features
Category 	
  

Char.
Set 	
  

Word 	
   Repeat 	
   Duration 	
  

Nois
e 	
  

Voice 	
   Beep 	
  

Mi...
Deployment
Database

IVRS Playing CAPTCHA

VOIP
Java
Application
PSTN

Linksys Gateway
SPA 3102

Cellular
Network
Source (...
Participants
– 90	
  ParDcipants	
  
	
  
– Five	
  ciDes	
  
- Delhi	
  
- Mumbai	
  
- Chennai	
  
- Noida	
  
- Vello...
Results: Accuracy
CAPTCHA	
  

Category	
  

Accuracy	
  (%)	
  

Skip	
  (%)	
  

CD	
  

Telephony	
  

18.71	
  

35.67...
Results: Time taken
CAPTCHA	
  

Category	
  

Time	
  (s)	
  

CD	
  

Telephony	
  

96.11	
  

Math-­‐Func	
  

Telepho...
Results:H1
–  H1:	
  	
  Close	
  to	
  the	
  expected	
  /	
  correct	
  answers	
  even	
  
though	
  the	
  overall	
...
Results: H2

Math function, but we noticed a negative relationship with correlation coefficient r = 0.47 for
web-based capt...
Results: H3
–  H3:	
  	
  Users	
  will	
  take	
  more	
  Dme	
  responding	
  to	
  a	
  CAPTCHA	
  
that	
  requires	
...
User Experience
– User	
  friendliness	
  
Strongly#Agree#

Agree#

Nuetral#

Disagree#

#Strongly#disagree#
0%#

10%# 20...
100%#

User Experience
– User	
  preferred	
  scheme	
  
100%#
90%#
80%#

Par$cipants+(%)++

70%#
60%#
50%#
40%#
30%#
20%...
Guidelines
– One	
  Dme	
  instrucDon	
  
– Loss	
  /	
  Error	
  Tolerant	
  
– Feedback	
  
– Verbal	
  Responses	
 ...
Thank you!!
QuesDons	
  

22	
  
For any further information, please write to
pk@iiitd.ac.in	
  
precog.iiitd.edu.in	
  
	
  
Upcoming SlideShare
Loading in …5
×

On the Viability of CAPTCHAs for Use in Telephony Systems: A Usability Field Study

491 views

Published on

Telephony systems are imperative for information exchange offering low cost services and direct reach to million of customers. They have not only benefited users but have also provided a convenient medium for spammers. Voice spam is often encountered on telephony, such as in the form of an automated telemarketing call asking to call a number to win million of dollars. A large percentage of voice spam is generated through automated systems which introduces the classical challenge of distinguishing machines from humans on the telephony. CAPTCHA is a conventional solution used for distinguishing humans and machines, and audio-based CAPTCHAs have been proposed as a solution to curb voice spam. In this paper, we conduct a field study with 90 participants in order to answer two primary research questions: how much inconvenience does CAPTCHA cause to users, and how different features of the CAPTCHA, e.g., duration and size influence usability of CAPTCHA on telephony. Our results suggest that currently proposed CAPTCHAs are far from usable. We provide certain guidelines that may help improve existing CAPTCHAs for use in telephony systems.

Published in: Technology, Design
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
491
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

On the Viability of CAPTCHAs for Use in Telephony Systems: A Usability Field Study

  1. 1. On the Viability of CAPTCHAs for Use in Telephony Systems: A Usability Field Study Niharika  Sachdeva*,  Nitesh  Saxena,  Ponnurangam  Kumaraguru*   University  of  Alabama,  Birmingham*IIIT-­‐Delhi   InformaDon  Security  Conference,  2013  (Nov  13  –  15)  
  2. 2. Overview – MoDvaDon     – Research  quesDon   – Study  Design   – Experimental  setup   – ParDcipants   – Results   – Guidelines   2  
  3. 3. Some Attacks 3  
  4. 4. CAPTCHA – Completely  Automated  Public  Turing  Test  to   tell  Computers  and  Humans  Apart     ReCAPTCHA   Google         Math  FuncDon   Yahoo  
  5. 5. Is it Really Useful ?? – FrustraDng                   – Lack  of  incenDve           – Hard  to  recognize   – Difficult  to  solve   – NaDve  language     E.  Bursztein,  S.  Bethard,  C.  Fabry,  J.  Mitchell,  and  D.  Jurafsky.  How  Good  Are  Humans  at  Solving  CAPTCHAs?   A  Large  Scale  EvaluaDon.  SP  ’10,  pages  399–413.     J.  Yan,  A.  Ahmad.  Usability  of  CAPTCHAs  Or  usability  issues  in  CAPTCHA  design.  In  Symposium  On  Usable   Privacy  and  Security,  pages  44–52,  2008.       5  
  6. 6. But CAPTCHA continues to Rule CAPTCHA  for  RoboCalls   6  
  7. 7. But CAPTCHA continues to Rule CAPTCHA  for  RoboCalls   7  
  8. 8. Audio CAPTCHA a solution? –   Yahoo         –   Google     – Patent  CAPTCHA   8  
  9. 9. Research Question – QuanDfy  the  amount  of  inconvenience   CAPTCHA  causes  to  users.   – How  different  features  of  CAPTCHA,  e.g.   duraDon,  size  and  character  set  influence  the   users’  performance?     -  H1:  Close  to  the  expected  /  correct  answers  even  though  the  overall   CAPTCHA  solving  accuracy  is  low.       -  H2:  Accuracy  of  answering  the  CAPTCHA  correctly  on  telephony   decreases  as  the  number  of  key  presses  required  increases.       -  H3:  Users  will  take  more  Eme  responding  to  a  CAPTCHA  that  requires   more  key  presses  than  to  the  one  requiring  less  key  presses.   9  
  10. 10. Study Design LaDn     Square   Polakis,  G.  Kontaxis,  and  S.  Ioannidis.  CAPTCHuring  Automated  (Smart)  Phone  Aiacks.     In  SYSSEC,  2011.       10  
  11. 11. CAPTCHA Features Category   Char. Set   Word   Repeat   Duration   Nois e   Voice   Beep   Min Max length   length   Google   0-9   No   Yes   34.4   Yes   M   Yes   5   15   Ebay   0-9   No   No   3.7   Yes   V   No   6   6   Yahoo   0-9   No   No   18.0   Yes   Child   No   6   8   Recaptcha   a-z   Yes   No   10.6   Yes   F   No   6   6   Slashdot   a-z   Yes   No   2.9   No   M   No   1   1   CD   1-5   No   No   14   Yes   M   No   1   1   Mathfunction   0-9   No   No   6.0   No   M   No   4   3   RPC   0-9   No   No   20.0   No   M   No   3   2   C+CD   0-9   No   No   14.0   No   M   No   4   3   M  =  Male  ;  F  =  Female;  V=Various  Voices     11  
  12. 12. Deployment Database IVRS Playing CAPTCHA VOIP Java Application PSTN Linksys Gateway SPA 3102 Cellular Network Source (Legitimate or malicious) File System IP phone Linux Server acting as CAPTCHA Shield (With FreeSWITCH) Architecture  Diagram   12  
  13. 13. Participants – 90  ParDcipants     – Five  ciDes   - Delhi   - Mumbai   - Chennai   - Noida   - Vellore     – Real  world  deployment   13  
  14. 14. Results: Accuracy CAPTCHA   Category   Accuracy  (%)   Skip  (%)   CD   Telephony   18.71   35.67   Math-­‐FuncDon   Telephony   17.47   26.51   RPC   Telephony   15.47   40.33   C+CD   Telephony   4.57   40.10   Ebay   Web  (Number)   8.75   13.13   Google   Web  (Number)   0.00   43.83   Yahoo   Web  (Number)   7.74   20.24   ReCaptcha   Web  (Alphabet)   0.00   46.07   Slashdot   Web  (Alphabet)   13.73   30.06   14  
  15. 15. Results: Time taken CAPTCHA   Category   Time  (s)   CD   Telephony   96.11   Math-­‐Func   Telephony   90.23   RPC   Telephony   147.44   C+CD   Telephony   109.59   Ebay   Web  (Number)   80.25   Google   Web  (Number)   123.49   Yahoo   Web  (Number)   95.88   ReCaptcha   Web  (Alphabet)   120.64   Slashdot   Web  (Alphabet)   122.57   15  
  16. 16. Results:H1 –  H1:    Close  to  the  expected  /  correct  answers  even   though  the  overall  CAPTCHA  solving  accuracy  is   low.   50   Yahoo! eBay 45 Google Slashdot 40 RPC Number of Captcha 35 Math−Function CD 30 C+CD 25 20 15 10 5 0 1 2 3 4 Edit Distance 5 6 7 16  
  17. 17. Results: H2 Math function, but we noticed a negative relationship with correlation coefficient r = 0.47 for web-based captcha. Finally, we found significant difference (t-test, t-value = 5.30 p-value < 0.001) between Expected Key Press (Average DTMF) and accuracy in statistical results shows that these two were independent of each other. The results mentioned above do not approve our hypothesis H2. •  H2:  Accuracy  of  answering  the  CAPTCHA  correctly   on  telephony  decreases  as  the  number  of  key   presses  required  increases   Table 3: Presents the Average DTMF expected for captcha (Avg. DTMF), accuracy, time and Average DTMF input by users (Avg. User DTMF) of each captcha. Scheme Category Avg. Accuracy Time Avg. User DTMF DTMF CD Telephony 1.00 18.71 96.11 1.76 MathTelephony 2.05 17.47 90.23 2.71 function RPC Telephony 3.00 15.47 147.44 3.92 C + CD Telephony 2.06 4.57 109.59 2.65 Ebay Web 6.00 8.75 80.25 3.85 Google Web 6.36 0.00 123.49 4.68 Yahoo Web 7.09 7.74 95.88 4.99 Slashdot Web 15.34 13.73 120.64 6.02 ReCaptcha Web 64.93 0.00 122.57 10.97 H3 – Time vs. Number of key press: Table 3 shows that users spent varying amount of time 17  
  18. 18. Results: H3 –  H3:    Users  will  take  more  Dme  responding  to  a  CAPTCHA   that  requires  more  key  presses  than  to  the  one  requiring   less  key  presses.   80" 70" 60" 50" 40" 30" 20" 10" 0" Ebay"" Google"" Key"press"(#)" Yahoo"" Accuracy"(%)" Slashdot"" Recaptcha" Avg"play"Gme(sec)" 18  
  19. 19. User Experience – User  friendliness   Strongly#Agree# Agree# Nuetral# Disagree# #Strongly#disagree# 0%# 10%# 20%# 30%# 40%# 50%# 60%# 70%# 80%# 90%# 100%# Complexity# Frequently#use# Confidence# Technical#help# 19  
  20. 20. 100%# User Experience – User  preferred  scheme   100%# 90%# 80%# Par$cipants+(%)++ 70%# 60%# 50%# 40%# 30%# 20%# 10%# 0%# 18-24# 25-35# 36-50# 51-65# Age+ #Numeric# #Mathfunc;on# #Alphabets# 20  
  21. 21. Guidelines – One  Dme  instrucDon   – Loss  /  Error  Tolerant   – Feedback   – Verbal  Responses   21  
  22. 22. Thank you!! QuesDons   22  
  23. 23. For any further information, please write to pk@iiitd.ac.in   precog.iiitd.edu.in    

×