Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Improving Two-Thumb Text Entry on Touchscreen Devices


Published on

Presentation at ACM CHI'13 in Paris by Antti Oulasvirta (Max Planck Institute for Informatics). Work done in collaboration with Keith Vertanen (Montana Tech) and Per Ola Kristensson (University of St Andrews)

Published in: Technology, Education
  • Be the first to comment

Improving Two-Thumb Text Entry on Touchscreen Devices

  1. 1. Improving Two-Thumb Text Entryon Touchscreen DevicesA. Oulasvirta,A. Reichel,W. Li,Y. Zhang, M. BachynskyiK.VertanenP.O. KristenssonMAX PLANCK INSTITUTE FOR INFORMATICSMONTANA TECHOF THE UNIVERSITY OF MONTANAUNIVERSITY OF ST ANDREWS10. toukokuuta 13
  2. 2. 10. toukokuuta 13
  3. 3. Text input,more than anything else,is the problem Apple needsto solve toukokuuta 13
  4. 4. iPad Dial KeysiPad Dial Keys Android qwertyWindows 8 thumb kbd Windows 8 regularLarge variety of virtualkeyboards for tablets10. toukokuuta 13
  5. 5. Two-thumb typing10. toukokuuta 13
  6. 6. Survey of postures ontouchscreen devicesuser had no experience and 7 indicated the user was anexpert at entering text on soft smartphones keyboards.As shown in Figure 2, each of the three postures was usedat least “sometimes” by about 60% of the participants. Onthe other hand, no single method was used by allparticipants. Two thumbs, one thumb, and one finger were“almost never” used by 37%, 35% and 41% of theparticipants, respectively.Figure 2. Use of an index finger, two thumbs, and one thumbduring text entry on smartphones.The Kthe deto prevuser’swordsperforalmost always often sometimes almost never0102030#ofpeopleFrequency of use in text entry[Azenkot & Zhai MobileHCI 2012]N = 7510. toukokuuta 13
  7. 7. Natural in mobile use10. toukokuuta 13
  8. 8. 14 wpm[Oulasvirta et al. 2010][Goel et al. 2012]31 wpm21 wpmIn-house testing N=6...Text editingPresentationsSpreadsheetsData entryCoding(Long) EmailsShort messagesNote-takingSearch queriesURLsAddressesNames...Reported rates10. toukokuuta 13
  9. 9. 14 wpm[Oulasvirta et al. 2010][Goel et al. 2012]31 wpm21 wpmIn-house testing N=6...Text editingPresentationsSpreadsheetsData entryCoding(Long) EmailsShort messagesNote-takingSearch queriesURLsAddressesNames...Reported ratesFollowing best practices,how much can we boost typing performance?10. toukokuuta 13
  10. 10. Design spacex-positiony-positionbutton widthCerror correctionwidthheightA Borganization of lettersX10. toukokuuta 13
  11. 11. Layout?Letter-to-keyassignment?Error correction?Study of grips[Wagner et al. CHI’12]Model of two-thumbtapping [Clawson et al. CHI’07]U!G! T! O!K! A! L! Q!I! E! !J!D! N! F! V!R! Y! S! Z!P! X! C!M! B! W! H!Optimization[Zhai et al UIST’00]Language & movement model[Kristensson &Vertanen ISCA’11]Training?[Castellucci & MacKenzie CHI’11][Sears et al. B&IT ’01][Zhai et al. CHI’02]10. toukokuuta 13
  12. 12. How to hold the device?10. toukokuuta 13
  13. 13. Grip spaceα10. toukokuuta 13
  14. 14. Study 1: Grips6PROXIMAL DISTALRADIALULNAR543CORNEREDGE2110. toukokuuta 13
  15. 15. 10. toukokuuta 13
  16. 16. Study 1:Task++ appears randomlyon both sides onreachable areas6 participantsx 3,000 taps / gripx 6 grips= 108,000 taps10. toukokuuta 13
  17. 17. Study 1: Results123 456~10% faster than the worst grip~4% faster than a random grip6PROXIMAL DISTALRADIALULNAR543CORNEREDGE2110. toukokuuta 13
  18. 18. Winner grip57.6 mmSplit keyboard10. toukokuuta 13
  19. 19. Constrained movement10. toukokuuta 13
  20. 20. 58 mm5 mmGrid is incompatible with QWERTY10 mm99% CI for offset10. toukokuuta 13
  21. 21. Letter-to-key assignment?6 * 1033 alternatives!A B C D E F G H I J K L M N O PQ R S T U V X Y Z _10. toukokuuta 13
  22. 22. A previous model forphysical thumb kbds[Clarkson et al. CHI’07]!!"##$ !"#!!!, !"#! = ! + !!!" = ! + !!!"#! !!!+ 1 ,!!!(1)where D is the distance between keys, W is the width of keyn,ID is the index of difficulty derived from D and W, and a andb are empirical parameters.For alternate-side (switching) taps, the “idle” thumb is as-sumed to approach its next target aggressively. Its movementtime is affected by not only ID but also the time elapsed,telapsed, before its turn. After it presses keyn, the thumb imme-diately starts to approach keyn. If it has not yet reached itwhen its turn comes, the remaining movement is shorter thanif having to start from the beginning. If telapsed is long enoughfor the thumb to reach keyn, it can rest over or on it. Then,only a minimal time tmin is needed for pressing keyn. The totaltime Tn for the nth letter in a word is:!!!!!!!!!! =!!!! + !!"##$ !"#!!! − !"#! !!!!!!!!!!!!!!"#$!"# !!!!!!!"#!!!!!!!"##$(!"#!!!!!"#!)!!!!!!!""!#$%&!!!!!!!!!!!!!(2)In the case of touchscreens, resting on a key is impossiblebecause it would cause an erroneous tap. For one to benefittheir ktaps. Uusers tEach sTask asequenas posstargetsplanninbecausredoneProceder-overstructewhile won key(correcscreenModelilantic Drive, Atlanta GA 30332-0280ark, kent, jamer, thad}@cc.gatech.eduy introduced apert two–thumbIn this work wem a longitudinalpdate the modelawategiesFigure 1. The mini–QWERTY keyboardsprevious studies: Targus (left) and Dell (right).approachwaitpress10. toukokuuta 13
  23. 23. What abouttouchscreens?thumb cannotrest on a target10. toukokuuta 13
  24. 24. Model acquisitionthe N-return task2431520 participants 10 x 10 trials per sequence1 < N < 910. toukokuuta 13
  25. 25. Predictive modelLeft RightSame-side tapswith taps longer than 1,000 ms. Because in Step 3 we usepixel coordinates, we here report D in pixel units. In allmodels, we use eight ID conditions. For modeling of side-switch taps, we use six telapsed conditions.Same-Side TapsFor same-side taps, the subset of the fastest 7% of tap se-quences constitutes 25,296 data points, or 65% of all data.This indicates that performance in this task improved quick-ly, stabilizing near a user’s personal best. We model MTwith a polynomial:!"!"#$ = 319.5 − 89.0!!" + 36.7!!"!(3)!"!"#!! = 237.3 − 7.6!!" + 13.8!!"!(4)R2 = .94 R2 = .9510. toukokuuta 13
  26. 26. Hover-over behavior15hover and waitapproachpress10. toukokuuta 13
  27. 27. 2. The lowest-ID targets are slower than medium-ID tar-gets, in contrast to the standard Fitts’-law models.The need for a squared term can be explained by the obser-vation that a thumb at times occludes nearby targets (low-ID) and it needs to be moved away for seeing the target. Ifone limits to ID≥1.3, a first-order model suffices.Alternating TapsOut of 14,619 returning taps (the Nth tap) in data, filteringto the best 15% within a condition yielded 5,105 data points(35% of the total). The 5% threshold was chosen to addressthe fact that reaching the best performance in alternatingtaps requires quite a few repetitions, and we had fewer ob-servations of returning taps per sequence. Our model is abivariate quadratic function with telapsed (see Background)and ID as the predictive variables:!"!"#$ = 265.286 − 9.501!!" − 0.024!!!"#$%!& + 2.003!!"!!−0.007!!!"#$%!&!!" + 3.322 ∗ 10!!!!"#$%!&!(5)!"!"#!! = 142.601 + 86.564!!" + 0.062!!!"#$%!& − 17.949!!"!!!!!!!!!!!!!!!!−0.035!!!"#$%!&!!" + 1.930 ∗ 10!!!!"#$%!&!(6)fasterswitcher-ovetowardturn aprisingThe prdecreatelapsedsimiladue totentiontion grneedsDesigWe arr1. M2. In3. WclR2 = .79 R2 = .79Alternating taps10. toukokuuta 13
  28. 28. OptimizationGradient)Descent))500#itera)ons#Annealing))10#X#3,000#itera)ons#Gradient)Descent)10,000#itera)ons#5,000#random#layouts#100#best#10#best#1)winner)OptimizationMobile Enron datasetPredictive model[Vertanen & Kristensson MobileHCI’11]10. toukokuuta 13
  29. 29. Key row optimizationDextrMicrosoft’s new curved kbdA" A" A" A"A" A" A" A"A" A" A" A"A" A" A" A"A" A" A" A"A" A" A" A"A" A" A" A"A" A" A" A"All key row position combinations within thereachable area 4810. toukokuuta 13
  30. 30. U!G! T! O!K! A! L! Q!I! E! !J!D! N! F! V!R! Y! S! Z!P! X! C!M! B! W! H!Predicted rate of 49.0 wpm10. toukokuuta 13
  31. 31. U!G! T! O!K! A! L! Q!I! E! !J!D! N! F! V!R! Y! S! Z!P! X! C!M! B! W! H!Analysis4.1% better than qwerty6.1% better than alphabetical layout64% of long-gramsFewer lettersMost vowels (a,e,u,o,i)Spacebars centrallylocated54%46%62% switching10. toukokuuta 13
  32. 32. 10. toukokuuta 13
  33. 33. Error correctionTouch model[Kristensson &Vertanen ISCA’11]Language modelNormally distributed touch points Perplexity of 3.84!∗= arg!max! ! ! ! ! ! . (8)Movement ModelSince KALQ is a new keyboard layout there is no straight-forward method to collect representative touch point data.We could not train a likelihood model on the evaluationstudy’s touch point data as this would mean we would trainthe model on the same subjects. Therefore, we instead esti-mated the likelihood P(K|T) by using a prescriptive modelthat assumes normal distribution of touch points [13],which is justified by existing evidence [9]. The probabilityof a touch point belonging to a particular key is! ! ! = exp −!!!!!! , (9)where !! is the Euclidean distance between the touch pointand the center of the key and !! is an estimate of the vari-ance of the touch point distribution around that particularkey’s center. This parameter was estimated from trainingdata of Step 2 that is disjoint from the evaluation (Step 5).Language ModelThe prior probability P(K) was estimated using a statisticaland question7-gram langno count cumobile devisize. Our fiprobabilitiessize of 9 MBby using adevices [12]terms of avecates the avpossible nexthe MobileEits small sizpared to anters. This lato 3.44.STEP 5: TREmpirical etions in theturned out tclustered around the spacebar, whereas the left thumb hasonly a few fast-action keys while the rest are more diffuse.This exploits the unique switching characteristics observedin the N-return study. A typing example is given in Table 1.STEP 4: ERROR CORRECTIONPrevious work has shown improvements in text-entry accu-racy on mobile devices through error-correction techniquesthat consider linguistic context and movement characteris-tics [6,9,13]. Ideally, error correction should operate in realtime, correcting erroneous characters as they are typed.Building on previous work [13], we constructed an error-correction technique for KALQ* that utilizes both linguisticinformation and the movement model for two-thumb textentry. For each touch point T, the error-correction modelfinds the key !∗that maximizes the posterior probability:!∗= arg!max! ! ! ! ! ! . (8)Movement ModelSince KALQ is a new keyboard layout there is no straight-tatistical properties of typingargets (hue: slow fast)sparency 1–100).ovement times and the fre-demonstrates how the rightate, frequently pressed keyswhereas the left thumb hasle the rest are more characteristics observedexample is given in Table 1.NLetter% Hand% D%(px)% telapsed%(ms)% MT%S* L* ]* ]*O* R* **93* **266*U* R* **93* ]*N* L* **66* **485*D* L* **66* ]*S* L* **93* ]*_* R* **66* **782*G* R* 148* ]*O* R* 132* ]*O* R* ***0* ]*D* L* **93* 1015*Table 1. Predicted typing performance with K! ! = ! ! ! ≈ ! ! C!!!!!!),where C is all previously written text and !!!!!!!asix characters written.We trained our model on a sample of 778M mesvia Twitter (12/2010–6/2012). Duplicate tweetsand non-English-language tweets were eliminalanguage-identification module [18, 19] (with a CWe included only tweets written on mobile dU!G! T! O!K! A! L! Q!I! E! !J!778M tweetsInserts,  deletes,  subs-tutes  le.ers  for  the  last  6  characters.  10. toukokuuta 13
  34. 34. TrainingSaarland university (M 25 years, SD 3.52). They reportedhaving almost no experience with large touchscreen devicessuch as tablets, and only one was a touch-typist on physicalQWERTY keyboards. The participants were compensated at10€/hour, and the two best were given a bonus of €100.Session% Contents% Test% Goal%0% QWERTY*typing*test* I* Baseline**measurement*1% Grip,*idle*thumb*tech]nique,*spacebar*policy** Introduce*KALQ,*confirm*understanding*of*the*basics*1?3% The*alphabet*and*frequent*words** Type*the*alphabet*without*seeing*the*key*labels*3?8% Frequent*bigrams*and*words*II,*III* Learn*motor*techniques*for*frequent*text,*speed*up*9?13% Full*sentences,*frequent*bigrams*and*words*IV* Speed*up*gradually*13?19% As*above*but*extra*prac]tice*with*error*correction** Speed*up*while*keeping*error*rate*under*5%*Final% Final*evaluation* Va,*Vb* Personal*best*with*KALQ*6 students (non-natives in English!)[Castellucci & MacKenzie CHI’11][Sears et al. B&IT ’01][Zhai et al. CHI’02]10. toukokuuta 13
  35. 35. ResultsFigure 9. Development of text entry speed throug the trainingKALQQWERTY1 4 8 11 13-19Test daywpm37 wpm5% CER28 wpm9% CER10. toukokuuta 13
  36. 36. Conclusion10. toukokuuta 13
  37. 37. 10. toukokuuta 13
  38. 38. Row shifting: pred < 1%Grip: 4-10%Optimization: pred. 4%Error correction: 1.3 perc units in cerHover-over: 10-20%28 wpm (9% cer)37 wpm (5% cer)See the paper fordetails here10. toukokuuta 13
  39. 39. The challenge50 wpm10. toukokuuta 13
  40. 40. Thanks!oantti@mpi-inf.mpg.deWill have KALQ, code, and data toukokuuta 13