Federico	
  Maggi,	
  Alberto	
  Volpatto,	
  
Simone	
  Gasparini,	
  Giacomo	
  Boracchi,	
  Stefano	
  Zanero	
  
  Direct	
  attacks	
  
  Well-­‐known	
  in	
  both	
  literature	
  and	
  industry	
  
  Very	
  active	
  research	...
  Less	
  known	
  yet	
  very	
  effective	
  
  Digital	
  side-­‐channels	
  
  Example:	
  decrypting	
  SSL	
  thro...
  First	
  attempt	
  of	
  automatic	
  shoulder	
  surfing	
  
  Recovery	
  of	
  long	
  texts	
  
  2010	
  survey	
  on	
  2,252	
  US	
  citizens	
  
  72%	
  use	
  a	
  mobile	
  phone	
  for	
  texting	
  
  30%	...
  Non-­‐automated	
  
  not	
  interesting	
  
  time	
  consuming	
  
  Automated	
  
  Is	
  it	
  feasible?	
  
 ...
  Moving	
  target	
  
  Fixed	
  observation	
  point	
  not	
  always	
  feasible	
  
  Very	
  small	
  keyboards	
 ...
  Lack	
  of	
  tactile	
  feedback	
  
  Early	
  soft	
  keyboards	
  were	
  hard	
  to	
  use	
  
  UI	
  engineers...
  Old	
  dilemma	
  
  More	
  secure,	
  less	
  easy	
  to	
  use	
  
  Example:	
  Google's	
  2-­‐step	
  authentic...
Our	
  approach	
  
  Requirement	
  1	
  
  iPhone-­‐like	
  visual	
  feedback	
  mechanism	
  
  Requirement	
  2	
  
  Template	
  of	...
SCREEN	
  TEMPLATE	
   KEY	
  TEMPLATES	
  
Q
W
E
R
T
Y
(synthetic,	
  hi-­‐res)	
  
MAGNIFIED	
  LAYOUT	
  
(x,y-­‐coordi...
  Phase	
  1	
  
  Screen	
  detection	
  and	
  rectification	
  
  Phase	
  2	
  
  Magnified	
  key	
  detection	
  
...
  Input	
  
  Image	
  depicting	
  the	
  current	
  scene	
  (current	
  frame)	
  
  Output	
  
  Synthetic	
  imag...
  The	
  current	
  frame	
  is	
  searched	
  for	
  the	
  screen	
  
template	
  (Requirement	
  1)	
  
? +	
  
SCREEN...
  SURF	
  features	
  
  Edges	
  
  Corners	
  
  Invariant	
  to:	
  
  Rotation	
  
  Scale	
  
  Skew	
  
  Oc...
  Estimate	
  during	
  
screen	
  detection	
  
  Successfull	
  
matches	
  improve	
  
matches	
  in	
  
subsequent	
...
  Input	
  
  Image	
  of	
  the	
  rectified	
  screen	
  
  Output	
  
  Areas	
  where	
  magnified	
  keys	
  appear...
-	
   =	
  
CURRENT FRAME SCREEN TEMPLATE	
   FOREGROUND	
  
FOREGROUND	
  
HIGHLIGHTED KEY (MAGNIFIED-KEY CANDIDATE)	
  
OTHER FOREGROUND
ELEMENTS (NOISE)	
  
  Input	
  
  Magnified-­‐key	
  candidates	
  
  Output	
  
  Sequence	
  of	
  typed	
  symbols	
  
  Procedure	
  
...
  Known	
  keyboard	
  layout	
  (Requirement	
  2)	
  
  Centroid	
  identification	
  
  Match	
  centroids	
  with	
 ...
Q	
  W	
  E	
  R	
  T	
  Y	
  U	
  I	
  O	
  P	
  
A	
  S	
  D	
  F	
  G	
  H	
  J	
  K	
  L	
  
Z	
  X	
  C	
  V	
  B	
  ...
CENTROID	
  1	
  
CENTROID	
  2	
  
CENTROID	
  3	
  
E	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  R	
  	
  	
...
  Region	
  of	
  interest	
  
  Key	
  template	
  (Req.	
  2)	
  
E	
  	
  	
  R	
  	
  	
  T	
  	
  	
  G	
  	
  H	
 ...
  Computing	
  the	
  key	
  similarity	
  is	
  expensive	
  
  Black-­‐white	
  distribution	
  of	
  the	
  ROI	
  
...
  Find	
  maxima	
  of	
  the	
  key	
  similarity	
  function	
  
  Phase	
  1	
  
  C++	
  
  OpenCV	
  
  Phase	
  2-­‐3	
  
  Matlab	
  
  Compiled	
  into	
  C	
  
  Threshold	
...
DEMO	
  
http://www.youtube.com/watch?v=aPuS8kNI30U	
  
http://www.youtube.com/watch?v=t9BxB3dO0KQ	
  
  Types	
  of	
  text	
  
  Context-­‐free	
  
  Context-­‐sensitive	
  
  3	
  attackers,	
  3	
  victims	
  
  Goal...
  Typing	
  
  3	
  victims	
  are	
  given	
  the	
  input	
  text	
  
  Victims	
  type	
  text	
  on	
  their	
  iPh...
spent chapter foundation identified because
first which material notation summarized
time spent volume much technical little...
close your eyes and begin to relax take
a deep breath and let it out slowly
concentrate on your breathing with
each breath...
  Non-­‐magnifying	
  keys	
  
  Space	
  (on	
  iPhone	
  only)	
  
  Layout-­‐switching	
  keys	
  
  Mitigation	
  ...
  [Raguram,	
  CCS	
  2011]	
  
  Appeared	
  at	
  the	
  same	
  conference	
  
  Completely	
  different	
  approach	...
  Touchscreen	
  mobile	
  devices	
  are	
  widespread	
  
  Shoulder	
  surfing	
  is	
  automatable	
  
  Automatic	
...
  Challenge	
  
  How	
  to	
  detect	
  tapping?	
  
Federico	
  Maggi	
  
fmaggi@elet.polimi.it	
  
@vp_lab	
  	
  
Dipartimento	
  di	
  Elettronica	
  e	
  Informazione	
  ...
A Fast Eavesdropping Attack Against Touchscreens
A Fast Eavesdropping Attack Against Touchscreens
A Fast Eavesdropping Attack Against Touchscreens
A Fast Eavesdropping Attack Against Touchscreens
A Fast Eavesdropping Attack Against Touchscreens
A Fast Eavesdropping Attack Against Touchscreens
A Fast Eavesdropping Attack Against Touchscreens
A Fast Eavesdropping Attack Against Touchscreens
A Fast Eavesdropping Attack Against Touchscreens
A Fast Eavesdropping Attack Against Touchscreens
A Fast Eavesdropping Attack Against Touchscreens
A Fast Eavesdropping Attack Against Touchscreens
A Fast Eavesdropping Attack Against Touchscreens
Upcoming SlideShare
Loading in...5
×

A Fast Eavesdropping Attack Against Touchscreens

231

Published on

The pervasiveness of mobile devices increases the risk of exposing sensitive information on the go. In this paper, we arise this concern by presenting an automatic attack against modern touchscreen keyboards. We demonstrate the attack against the Apple iPhone—2010's most popular touchscreen device—although it can be adapted to other devices (e.g., Android) that employ similar key-magnifying keyboards. Our attack processes the stream of frames from a video camera (e.g., surveillance or portable camera) and recognizes keystrokes online, in a fraction of the time needed to perform the same task by direct observation or offline analysis of a recorded video, which can be unfeasible for large amount of data. Our attack detects, tracks, and rectifies the target touchscreen, thus following the device or camera's movements and eliminating possible perspective distortions and rotations In real-world settings, our attack can automatically recognize up to 97.07 percent of the keystrokes (91.03 on average), with 1.15 percent of errors (3.16 on average) at a speed ranging from 37 to 51 keystrokes per minute.

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
231
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
27
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "A Fast Eavesdropping Attack Against Touchscreens"

  1. 1. Federico  Maggi,  Alberto  Volpatto,   Simone  Gasparini,  Giacomo  Boracchi,  Stefano  Zanero  
  2. 2.   Direct  attacks     Well-­‐known  in  both  literature  and  industry     Very  active  research  community     Other  types  of  attacks     Social  engineering  attacks     Side-­‐channel  attacks     Difficult  to  mitigate  (if  not  through  awareness)  
  3. 3.   Less  known  yet  very  effective     Digital  side-­‐channels     Example:  decrypting  SSL  through  wifi  LAN  sniffing     Physical-­‐world  observation     Direct  observation   ▪  Shoulder  surfing     Indirect  observation   ▪  Sound  emanations   ▪  Reflections   ▪  Magnetic  radiations   ▪  Desk  surface  vibrations  
  4. 4.   First  attempt  of  automatic  shoulder  surfing     Recovery  of  long  texts  
  5. 5.   2010  survey  on  2,252  US  citizens     72%  use  a  mobile  phone  for  texting     30%  use  a  mobile  phone  for  instant  messaging     38%  use  a  mobile  phone  for  Web  browsing     (1970)  touchscreen  technology  was  invented     2010:  5  billion  US  dollars  market     159%  market  grow  rate     Q3  2010:  417  million  of  touchscreen  devices  sold  
  6. 6.   Non-­‐automated     not  interesting     time  consuming     Automated     Is  it  feasible?     Mobile  context  poses  several  constraints  
  7. 7.   Moving  target     Fixed  observation  point  not  always  feasible     Very  small  keyboards     No  visibility  of  pressed  keys     No  visible  key  occlusions  
  8. 8.   Lack  of  tactile  feedback     Early  soft  keyboards  were  hard  to  use     UI  engineers  came  up  with  usable  keyboards  
  9. 9.   Old  dilemma     More  secure,  less  easy  to  use     Example:  Google's  2-­‐step  authentication     Very  secure     Very  unusable   ▪  Wait  for  the  verification  code  every  time  you  do  email     Apply  also  in  this  context     Feedback-­‐less  touchscreen  keyboards   ▪  hard  to  type  on     Feedback-­‐rich  keyboard  keyboards   ▪  easy  to  type  on   ▪  eyes  follow  the  feedback  naturally  during  typing  
  10. 10. Our  approach  
  11. 11.   Requirement  1     iPhone-­‐like  visual  feedback  mechanism     Requirement  2     Template  of  the  target  screen  known  in  advance  
  12. 12. SCREEN  TEMPLATE   KEY  TEMPLATES   Q W E R T Y (synthetic,  hi-­‐res)   MAGNIFIED  LAYOUT   (x,y-­‐coordinates)  (screenshot)  
  13. 13.   Phase  1     Screen  detection  and  rectification     Phase  2     Magnified  key  detection     Phase  3     Keystroke  sequence  reconstruction  
  14. 14.   Input     Image  depicting  the  current  scene  (current  frame)     Output     Synthetic  image  of  the  rectified,  cropped  screen     Procedure     Screen  detection     Screen  rectification  
  15. 15.   The  current  frame  is  searched  for  the  screen   template  (Requirement  1)   ? +   SCREEN  TEMPLATE   CURRENT  FRAME   MATCHING  PATCH  
  16. 16.   SURF  features     Edges     Corners     Invariant  to:     Rotation     Scale     Skew     Occlusions     Homography  estimation   TEMPLATE CURRENT FRAME
  17. 17.   Estimate  during   screen  detection     Successfull   matches  improve   matches  in   subsequent   frames   CURRENT  FRAME   RECTIFIED  FRAME  
  18. 18.   Input     Image  of  the  rectified  screen     Output     Areas  where  magnified  keys  appeared     Procedure     Background  subtraction  
  19. 19. -   =   CURRENT FRAME SCREEN TEMPLATE   FOREGROUND  
  20. 20. FOREGROUND   HIGHLIGHTED KEY (MAGNIFIED-KEY CANDIDATE)   OTHER FOREGROUND ELEMENTS (NOISE)  
  21. 21.   Input     Magnified-­‐key  candidates     Output     Sequence  of  typed  symbols     Procedure     Approximate  neighbors  lookup     Best  matching  key  identification     Fast pruning     Key  sequence  analysis  
  22. 22.   Known  keyboard  layout  (Requirement  2)     Centroid  identification     Match  centroids  with  keyboard  layout  
  23. 23. Q  W  E  R  T  Y  U  I  O  P   A  S  D  F  G  H  J  K  L   Z  X  C  V  B  N  M  
  24. 24. CENTROID  1   CENTROID  2   CENTROID  3   E                            R                              T   N                          M   G                            H                            J  
  25. 25.   Region  of  interest     Key  template  (Req.  2)   E      R      T      G    H      J        N    M   LOW   HIGH   LOW   LOW   LOW   LOW  MED   MED  
  26. 26.   Computing  the  key  similarity  is  expensive     Black-­‐white  distribution  of  the  ROI     %B/W-­‐heuristic  is  way  faster   B            W   B            W   B            W   NOT  A  LETTER   NOT  A  LETTER   MAYBE  A  LETTER   B            W   (we  don’t  know  which  one,  yet)   B            W   B            W   ≠   ≠   =   CANDIDATE  FOUND   BASELINE  
  27. 27.   Find  maxima  of  the  key  similarity  function  
  28. 28.   Phase  1     C++     OpenCV     Phase  2-­‐3     Matlab     Compiled  into  C     Threshold  estimation     Confidence  interval  (mean,  variance)     Video  samples  collected  in  “no  typing”  conditions  
  29. 29. DEMO   http://www.youtube.com/watch?v=aPuS8kNI30U   http://www.youtube.com/watch?v=t9BxB3dO0KQ  
  30. 30.   Types  of  text     Context-­‐free     Context-­‐sensitive     3  attackers,  3  victims     Goals     Precision  and  speed     Resilience  to  disturbances  
  31. 31.   Typing     3  victims  are  given  the  input  text     Victims  type  text  on  their  iPhones     Recording     A  recording  camera  was  used  for  repeatability     Attack     3  attackers  are  provided  with  the  videos     Attackers  have  “infinite”  time  to  analyze  videos     Comparison     Automatic  attack  vs.  human  attackers  
  32. 32. spent chapter foundation identified because first which material notation summarized time spent volume much technical little system reference figured number measurement lorem referring abstract text introductory shown in the we observing request second objective books relationship astute formidable quantile convenient remainder between utilizable tool law resident minutes exemplified the product then temporarily number will per systematic average accumulated south specialty terminal numerous introduce
  33. 33. close your eyes and begin to relax take a deep breath and let it out slowly concentrate on your breathing with each breath you become more relaxed imagine a brilliant white light above you focusing on this light as it flows through your body allow yourself to drift off as you fall deeper and deeper into a more relaxed state of mind now as i
  34. 34.   Non-­‐magnifying  keys     Space  (on  iPhone  only)     Layout-­‐switching  keys     Mitigation   ▪ Device-­‐specific  heuristics   ▪ E.g.,  on  iPhone,  exploit  color-­‐changing  spacebar     Alternative  layouts  (minor  limitation)     Mitigation   ▪ Detect  switch   ▪ Loop  through  different  templates  during  detection  
  35. 35.   [Raguram,  CCS  2011]     Appeared  at  the  same  conference     Completely  different  approach     Classification-­‐based     They  require  training     Really,  the  very  same  accuracy  97~98%  
  36. 36.   Touchscreen  mobile  devices  are  widespread     Shoulder  surfing  is  automatable     Automatic  shoulder  surfing  is  precise  too     Counteract  these  attacks  with  privacy  screens     But…  
  37. 37.   Challenge     How  to  detect  tapping?  
  38. 38. Federico  Maggi   fmaggi@elet.polimi.it   @vp_lab     Dipartimento  di  Elettronica  e  Informazione   Politecnico  di  Milano  
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×