Getting in touch with text: Designing a mobile phone           application for illiterate users to harness SMS            ...
2. BACKGROUND                                                          Otherwise users might think the icons represent loc...
who had mobile phones restricted to making voice calls [11].             with illiterate people before and her involvement...
listen to the content on the phone through a human or machine            To compose a new message, the user could re-use w...
complicated, especially for illiterate users and gestures available     language schools. According to them none knew how ...
literate helper, as mentioned in study 1. Throughout the session        For the usability test, the video recordings were ...
the played sound. While the karaoke was playing, the woman             avoid misinterpretations especially when seeing the...
they usually try to hide their illiteracy. Our way to get in contact   approach for blind users. We used monochrome simple...
the main presentation of the inbox screen. No previous            
19. Smyth, T.N., Kumar, S., Medhi, I., and Toyama, K. Where       21. UNESCO. Gender and Education for All: The Leap to   ...
Upcoming SlideShare
Loading in …5

ACM DEV '12 Proceedings of the 2nd ACM Symposium on Computing for Development


Published on

Getting in touch with text: Designing a mobile phone application for illiterate users to harness SMS(Link)

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

ACM DEV '12 Proceedings of the 2nd ACM Symposium on Computing for Development

  1. 1. Getting in touch with text: Designing a mobile phone application for illiterate users to harness SMS Elsa Friscira Hendrik Knoche Jeffrey Huang EPFL EPFL EPFL Station 14, IC LDM Station 14, IC LDM Station 14, IC LDM 1015 Lausanne, Switzerland 1015 Lausanne, Switzerland 1015 Lausanne, Switzerland +33 63 112 3494 +41 21 693 1315 +41 21 693 1341 jeffrey.huang@epfl.chABSTRACT asynchronous communication channel of text messaging has beenA large number of illiterate people – 800 million worldwide – are inaccessible to illiterate or semi-literate people.currently excluded from the benefits of asynchronous and cheap Much of the previous work on illiterate user interfaces (UI) oncommunication through text messages also known as SMS. Smart mobile devices and computers has been pointing to shortcomingsphones with touch screen will soon be in financial reach of in terms of usability of UIs that were not adapted to this specialilliterate people in developing countries. Our application user group. However, the uptake of mobile phones in developingEasyTexting allows illiterate users to listen to received SMS and countries is staggering and research has shown that incentives incompose text messages by augmenting words with touch-initiated real world contexts were high enough [19] to overcome initialtext-to-speech support, icons for frequent phrases and by re-using usability hurdles in learning a UI or technology.words from previous messages. The application sends and While much of the ICT4D literature is at odds with the use of textreceives plain SMS and makes no assumption on second parties’ in user interfaces for illiterate users we see a great potential toSMS editors. We present the motivation for this application connect illiterate users through text messaging - a cheap,derived from interviews and the evolution of the design along asynchronous and convenient communication channel. Touchwith an exploratory evaluation of the interface both with illiterate screen smart phones offer a new opportunity for illiterate peopleimmigrants. to interact with textual content in connections with text-to-speechCategories and Subject Descriptors solutions. To that effect, we present the design evolution and evaluation of EasyTexting an application that enables illiterateH.5.1 [Information Systems]: Multimedia Information Systems – users to use this medium via text-to-speech for reading out wordsanimations, audio input/output. and by making composition possible through icons and reuse ofGeneral Terms previous words. The application sends and receives plain SMSDesign, Experimentation, Human Factors. and there is no requirement for a second party to utilize the application.Keywords The goal of this paper was to address the following questions:ICTD, mobile phones, touch screens, texting, SMS, illiterate How do illiterate people use mobile phones or other artifactspeople within their coping strategies in general and how SMS in particular? Which UI conventions from current SMS editors can1. INTRODUCTION be kept and how can illiterate users be enabled to use textAround 800 million people worldwide cannot read or write their messaging in conjunction with audio, text, and visuals through amother tongue. Most of them live in developing countries – touch screen interface? After the first prototypes were developed,mainly in rural areas. Mobile phones have been a phenomenal our goal was to understand how users would experience this novelsuccess in terms of sustainable development and its business interaction with text.model proved viable in many developing countries despite huge We addressed these questions by following a user-centered designinfrastructural shortcomings in terms of e.g. the availability of method with illiterate users in Switzerland consisting of initialelectricity. Coverage has seen huge improvements and most of the interviews, various prototyping with expert reviews and an initialgrowth for mobile provider now lies in attracting customers in exploration through task-based scenarios. Our study was focusedrural areas who mainly work in agriculture. Low-end smart on illiterate users in Western countries but we think that some ofphones with touch screens have already dropped below the $100 our findings will generalize to illiterate people in developingmark and will be soon within financial reach of less affluent rural regions.populations in developing countries. However, so far the cheap We summarize previous research in section 2 and our own results from interviews with illiterate immigrants in section 3. We detail Permission to make digital or hard copies of all or part of this work for our early design work, the design evolution of EasyTexting and an personal or classroom use is granted without fee provided that copies are exploratory evaluation with illiterate users in sections 3-5. We not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy discuss our findings in relation to previous research in section 6 otherwise, or republish, to post on servers or to redistribute to lists, and the final design in section 7. requires prior specific permission and/or a fee. DEV 12, March 11-12, Atlanta, GA Copyright © 2012 ACM 978-1-4503-1262-2/12/03... $10.00
  2. 2. 2. BACKGROUND Otherwise users might think the icons represent locations orLiteracy can be defined in many ways. The U.N. defines a literate objects, for instance a kitchen instead of cooking [13]. Prasad etperson as someone who can “…with understanding, both read al. found that the metaphor of a postcard symbol worked well in aand write a short simple statement in his or her everyday life” video mail application and helped users overcome their[21]. Semi-literates represent another large group struggling with difficulties in understanding the notion of asynchronicity in areading simple text passages [7]. Many illiterate people have basic video mail application [15]. Bhamidipaty & Deepak improvednumeracy skills, i.e., they can to some degree understand, read contact management for illiterates by adding symbols to theand write numbers. Huenerfauth distinguished between phone’s physical number keypad, which allowed users to filtertechnological illiteracy and written language illiteracy [4]. contacts through combinations of these symbols [1]. The meaning of an icon represents a learnt concept some of which can be moreA large number of development initiatives now evolve around the easily understood and recalled than others. However, if additionaluse of mobile phones but so far most of the commercial offerings modalities are available in the UI to explain their meaning(see [16] for an overview on mobile information services for designers often discount concerns about their lack ofagriculture) require their users to be literate. A number of ICTD appropriateness or intuitiveness.projects have aimed at improving rural communication andknowledge building for illiterate users, e.g. through audio wikis Audio feedback and voice annotation support represent two such[8], discussion forums that extend existing mass media coverage modalities, which are used in many designs for illiterate users e.g.such as community radios [14], and spoken web interfaces for [2], [11], [12], [18], [13], [14]. Audible instructions given touser generated content [9]. However, to the best of our knowledge illiterate users needed to be short and simple and instructionsno previous work has tried to empower illiterate users to use text- containing multiple steps have to be avoided [18], [10], [15].based communication through mobile phones. Possibly the closest When given audio instructions with multiple steps, illiterate usersto our idea is Shankar’s work on speech writing or spriting [17]. usually performed only the first or the last one. Findlater et al. reported that the combination of text and audio disturbed illiterateAs a hardware platform for ICT the mobile phone poses various users but that semi-literates - with rudimentary reading skills -challenges to illiterate users. According to Chipchase work benefitted from them in transitions from audio+text to text-onlyturning on the mobile phone and accepting incoming calls were interactions [3]. To illiterate users text in the UI simplythe most successfully completed tasks by illiterate users of low- represented visual noise. In the search for optimal audio-visualend or feature phones [2]. Dialing numbers for an outgoing call representations for illiterate users of health kiosks Medhi et al.already proved more difficult. More complicated features such as created visual representations of health symptoms. Voicecontact management or asynchronous text messaging were outside annotations helped the user in speed of comprehension andtheir current reach. To make mobile phones accessible for increased correct responses. If not told, however, someilliterate users he proposed: a design not recognizable as targeting participants did not understand that the voice annotations wereilliterate people due to the associated stigma and a minimal meant to explain the visuals [12].feature set by supporting only incoming and outgoing calls and asimplified way to store contacts through call logs [2]. The As an input method voice has yet to overcome some hurdles.Motorola Motofone F3 fit many checkboxes to be designed for During a longitudinal field trial of Avaaj Otalo - an interactivepoor, illiterate people. It was light, very rugged, and provided voice forum for small farmers accessed through voice calls – theaudio feedback for its functions from power on throughout its users could choose between voice commands and touchtone asmain (minimal) menu. Its e-ink screen could easily be read in input methods to navigate menus. Touchtone input was preferredbright sunlight, it had a phenomenal battery life (nominally 30 in the large majority of cases over voice and users unanimouslydays on standby) useful in rural areas with long power cuts and, at preferred touchtone navigation. Users found voice input morearound 20 USD it was affordable. However, it was not a success. error prone. However, this could have been due to the lowAccording to an unnamed Motorola source the company had accuracy of the speech recognizer, which was trained onunderestimated the aspirational aspects of the device. Given that American English and often faced inputs with noisy backgroundmany people see mobile phones as extensions of themselves they [14].did not want to be seen with a cheap phone. Common UI conventions and elements presented problems forMost studies that we surveyed employed illiterate users who were lingual and technologically illiterate people. Chipchase cautionednumerate but all agreed that they could not understand text-based against the use of soft-buttons and suggested that each hardwareUIs. Researchers advocated minimal use of text and some even button on mobile phones should map to one task only. Prasad ettext-free UIs [13]. In the greater socio-technical context, however, al. found that users were confused when faced with modes e.g.,many illiterate people rely on proxy-literacy and seek out literate when creating a mail required them to choose from: video, audio,helpers to mitigate encounters with text. These helpers can benefit drawn images and text as input methods [15]. Lalji & Good foundfrom the existence of text, making their involvement less onerous the use of lists far more effective than a hierarchical comparison to text-free UIs. Fore example, Chipchase argued According to this study, participants remembered that they couldfor the value of textual descriptions to accompany icons and use the ‘up’ and ‘down’ buttons but easily forgot how to accessdeemed icon-only interfaces inferior for use by illiterate users. features when presented with a menu-based interface accessible through soft keys [10]. They warned that color-coding wasThe value of icons in UIs for illiterate users has been insufficient if users’ instructions were based on identifyingacknowledged and demonstrated in many studies, e.g. [2], [12], different colored buttons. In their lab studies, users often pressed[10], [3] but they are not universally recognizable and need to be the green button when instructed to press the blue, and vice versaadapted culturally. For instance, participants understood the [10]. In a study by Prasad et al. participants were likely to click on“house” icon as a village hut and mistook musical notes for birds anything green when asked to click on a green arrow [15]. Medhi[10]. Hand-drawn icons were preferred to realistic photos in et al. mentioned that scrollbars were not initially understood in thestudies by Medhi et al. and she noted that icons, which indicate an sense that subjects did not realize that there were functionsaction may require visual cues for indicating motion [12]. displayed below the fold. These users coincided with the ones
  3. 3. who had mobile phones restricted to making voice calls [11]. with illiterate people before and her involvement eventuallyScreen navigation was an issue frequently quoted in previous proved helpful to establish a trusted connection with the [11], [10], [15]. To curb confusion from abrupt screenchanges, Prasad et al. proposed that navigation employed 3.1 Participantsanimation to transition from one screen to the next [15] - now We carried out semi-structured interviews (60-90 minutes insupported and common in e.g. iPhone and Android UIs. Katre duration) in cafés or if they felt comfortable with it in theargued focusing on thumb-based interaction in the design of participants’ home. All of the 9 participants (7f, 2m) living inapplications for semi- and illiterate users on smart phones with Switzerland had immigration backgrounds from Africa and Brasiltouch screen [5]. He claimed that this user group lacked fine and had only very recently started a course to learn how to readmotor skills due to non-practice in writing. This made stylus and and write. Most of them did not currently hold a job and wereindex finger based input slower compared with literates. supported by either partners or the state. Except one retired woman all participants had enrolled in the school to be able toMethodologically, involving illiterate users in HCI studies is more find jobs. They received 20 CHF/h as compensation for their time.challenging than with literate users in advanced economies.Participants in previous studies typically had no faith in The interview script included the description of a typical day intechnology [13], had difficulties understanding abstract questions their life, problems or inconveniences faced, technology used inand were not used to being tested [18], lacked self-confidence and the home, their use of means of communication, interacting withfelt they were not clever enough to use technology and wanted to necessary machinery e.g. automated teller machines (ATMs) andobserve and be taught [10]. Sherwani et al. proposed incremental with a special focus on the use of their mobile phones includingtutorials for participants before the study in order to better prepare receiving and placing calls, SMS, managing contacts and otherthem to use a UI [18]. Prasad et al. reported that congratulatory functionality messages after users performed a task seemed to produceencouragement and excitement in order for them to continue 3.2 Resultsnavigation the application with more self-confidence. For In this paper we focus on the use and coping mechanisms for textinstance, after successfully logging in to an application, an audio messaging. The broader results from the interviews are reported incongratulatory message informing the users that they had [1]. Living in a foreign country our Swiss participants needed tosuccessfully entered their inbox and that they could now retrieve stay in touch with family and friends in their home countries.their mails [15]. Calling abroad was expensive and they often used internet cafes’In summary previous research on designing for illiterate users has to make calls through special operators or VoIP which requiredproduced many recommendations, which are, however, often synchronizing with the called party to be at a place at a certainremind us of the problems faced by all novice users of computers time. Many regarded asynchronous communication such as SMSsuch as conventions of UIs and the affordances controls have. The as a convenient and cost-efficient alternative to stay in a touch. Asrecommendations were often derived from usability studies in one woman whose daughter was living in Morocco stated:which people encounter systems for the very first time and outside “I would love to send an SMS to my daughter such as ‘I’mthe context in which they typically discover and learn a new thinking of you’, but unfortunately, this is far too complicated fortechnology. Most of the previous work focused on mobile phones me.”with keypads that soon might become obsolete. In particular, Moreover, some had been asked by others to send texts rather thansome of the described hurdles in basic and feature phones e.g. the call. However, its reliance on literacy seemed an insurmountableproblems with soft keys could be overcome if illiterate users were barrier to using it to contact people. All of them had received textable to find out what effect a button press would have analogously messages often unsolicited. Dealing with received text messagesto a mouse-over help text, i.e. without the need for pressing the varied and depended to some degree on the content. Three of ourbutton and carrying out its action. In order to look at requirements interviewees had stored SMS that contained telephone numberswe wanted to find out more about illiterate people actual use, for months as another way of looking up contacts.specifically with respect to text messaging and possibly on moreadvanced mobile phones that were used six years ago when “I know X sent me this text message that has the telephoneChipchase conducted his seminal work on illiterate people’s number from a friend of mine in Togo. So I go back here [to themobile phone use. inbox of his messages] and need to find his message. Here this is it, he wrote this text in front of the number – my wife read it to me.3. STUDY 1 It’s the name of the friend.”We conducted interviews with illiterate immigrants in Switzerland Some had developed simple heuristics in detecting unsolicitedto study their use of mobile phones. We got access to them SMS through the length of the sending telephone numbers and thethrough schools in Switzerland that taught adults how to read and fact that it contained lots of text. Mostly interviewees respondedwrite French. We told the school directors that we were interested to an incoming SMS by calling the sender – either they hadin illiterate peoples’ coping strategies as well as their use of memorized how to do this through the context menu or they notedmobile phones and the tricks they employ to overcome their down the number and typed it into the phone again. Someinability to read and write. Coming from a scientifically reputable interviewees treated all text messages as spam and had learnedschool helped only to some extent as the teachers and directors of how to either exit the mode into which the phone switched onthe schools found it hard to understand what would come out of reception or how to quickly delete them without checking thethis study, whether their students would be treated with respect, content or their origin. Others asked for help with the content ofanonymity would be guaranteed and overall what benefit the the text messages. None of the interviewees felt bad about beingstudents and the school would enjoy in return. Some of the read to but one of them who was in a new relationship foundteachers were not even sure that many of their students were using asking close friends to read SMS with romantic content exciting atmobile phones. A retired researcher that had worked extensively first but recently increasingly annoying. One participant wondered whether it would be possible to forward the SMS to a service and
  4. 4. listen to the content on the phone through a human or machine To compose a new message, the user could re-use words fromvoice. previous messages or rely on icons. In the former case when theThe signing of receipts for the payment turned out to be a problem user tapped on the “Edit” button (see fig 2 left), the wholefor three of our participants. They never signed any documents message was added to the New Message editor area. By tappingunless a trusted person was present to make sure they are not on the pencil button the application switched into the edit modebeing taken advantage of. and the user could select only some parts of the previous messages. In the latter case by tapping on the smiley icon, the user4. EasyTexting could navigate to two selection screens:Inspired by the findings around illiterate people’s use and non-use 1. The Quick sender screen contained nine icons representing theof SMS and their interest in this form of communication we most frequent messages sent such as “ok”, “no”, “I miss you”.developed a prototype for a voice-assisted SMS application The icons did not have text labels, however, each of them haddubbed EasyTexting. This idea was born during a design course, a sound support. When the user selected one icon to be addedwhich evolved around four expert reviews. The first author of this to the message, the icon in itself was added to the messagepaper developed the conceptual idea iteratively during this course editor (see the question mark icon “Why” in Figure 1 left).and obtained feedback from interaction designers and researchers 2. The Customize screen contained multi-sentence iconswho had published in the area of ICT4D. arranged by topics such as “Places and activities, feelings”.First explorations of the concept were carried out with paper and Each icon had multiple meanings: For instance, the skyscraperpost-it notes to simulate screen navigation and later Powerpoint icon, had three associated sentences: “1_ I am at work. 2_ Islides to simulate interactivity with audio. These early evaluations cannot answer, I am busy. 3_ I am doing some shopping in thewere based on the idea of sending an SMS through icons only. city”. By long tapping on this icon the user could listen to all the sentences associated to it. If the user wanted to use the4.1 First prototype second sentence, he had to tap the icon twice. There was noIn order to be able to test the application with users we developed visual/audio feedback on how many times he had alreadyan interactive prototype (see Figure 1) with Microsoft Expression tapped. Voice prompts read out the content but did notBlend on an LG Optimus 7 - a WVGA (800x480 at 246ppi) multi- provide any action cues. There was the possibility to “add atouch screen phone running the Windows Phone 7 (WP7) new entry” to extend the repertoire and add sentences fromoperating system. It allowed for users to ‘read’ SMS through text- previous messages to some of the existing icons. For instance,to-speech audio rendition and compose SMS through a range of after reception of a message e.g. “I really miss you today”, theicons the user can drag into the message editor and by re-using user could add this sentence to his repertoire in the Feelingswords from previous messages. The icons represented common section.text messages such as ‘Yes’ and ‘No’. We used generic facesdrawing to represent contacts in the phone.This prototype was composed of five main screens: the threadoverview or Inbox screen (entry point), the individual thread orConversation screen (see Figure 1, left), the Quick Sender screen,the Multi-sentences icon or Customize icon screen and theMessage reviewing (see Figure 1, right).(1) The Inbox screen enabled users to see an overview of messagethreads by contact.(2) The Conversation screen allowed users to see all the messageswith a contact chronologically ordered. It also allowed users toread incoming messages and to create a new message by:selecting icons for most frequently used messages (in the Quicksender screen), selecting multi-sentence icons by category(Customize screen) and turning into an edit mode to reuse wordsfrom previous messages.(4) The Message reviewing screen helped users reviewing thecontent of messages during their creation. Figure 1: First prototype Conversation screen (left) and Message reviewing screen (right)(5) A part from selecting multi-sentences icons by category theCustomize screen enabled users to create new entries for multi- After composing a new message, the user could navigate to thesentence icons (to extend the repertoire of available sentences Message reviewing screen and listen to its audio rendition. Thethrough icons). icons were transformed into text and read out through text-to- speech. Each group of words played and its corresponding icon The Inbox screen contained all the inbox messages displayed as a was highlighted. For instance, in Figure 1 (right), the skyscrapervertical list. It contained an iconic picture of a person, their first icon and the sentence “I am at work” are highlighted while thename and envelopes that represented the new messages (closed) phone plays this group of words.and read messages (open). Tapping on a thread in the list broughtup the Conversation screen Figure 1 (left). We carried out walk-throughs with four experts, brainstorming sessions and corridor testing with students to improve the design.The Conversation screen represented the history of all the We did not test this prototype with any illiterate user. The biggestmessages the user exchanged with a particular person. Each concern was that there were too many screens to go through. Formessage could be listened to and visually contain combination of instance the Conversation screen could be combined with theicons and text (cf. Figure 1, left). Message reviewing screen. The edit mode was deemed too
  5. 5. complicated, especially for illiterate users and gestures available language schools. According to them none knew how to read oron touch screens such as scrolling and drag and drop could reduce write in their mother tongue either. A 40-year old woman fromthe amount of taps required. Angola was married and her husband was literate. She had a feature phone without a touch-screen. A 35-year old Moroccan4.2 Second prototype widow (her late husband was a researcher) and mother of a threeThe second prototype was simplified yet contained additional year old had had an iPhone, which broke after being thrown in theinformation. It included the same main screens as the first toilet by her toddler. She was planning on buying a newer iPhoneprototype: Inbox screen, Conversation screen, Quick sender, model with an Internet subscription in the near future but for theCustomize screen but we removed the Message reviewing screen. transition was using a Nokia feature phone. A 35-year oldThe Inbox screen included the date of the last message received Senegalese father of a six-month old, married to a literate nurseand the telephone numbers of the contact. By tapping on a thread, had an iPhone with an international subscription.the application brought up the Conversation screen that lists allSMS with a given contact. We removed the need for an editing 5.2 Methodmode by turning every word into a button. This allowed the users We invited the participants for lunch before their session to maketo single tap on them to listen to its spoken form and to reuse them them feel at ease. We started by introducing the motivation andby dragging them into the New message editor area that was fixed the purpose of the application. We guaranteed anonymity andon the bottom of the Conversation screen (see Figure 2). During explained that our goal was not to test them but to obtain theirthe audio playback of a read out word it was visually highlighted feedback as illiterate users. To boost their confidence, we stressedin synch. We will refer to this assistive function as karaoke from that they were ‘the experts’ who tested applications designed byhereon. This represented a new way of reviewing the message students. For data collection we used note taking by thecomposed and due to its fixed placement allowed for removing experimenter in-situ while the participants were performing thethe Message reviewing screen. tasks. With their permission we video recorded the interaction of their hands with the UI of the phone along with the soundtrack. Each session lasted about 40 minutes and consisted of four parts: 1. a socio-demographic questionnaire, 2. a semi-structured interview, 3. a usability test of the application including a participatory element around the design of the employed icons, and 4. a debrief interview. Before starting the questionnaire we introduced ourselves and tried to establish some common ground with the participants. The teachers assured us that the participants did not know how to read or write simple sentences in French. Hence we did not perform additional literacy tests. Since we tried to establish a setting in which the participants were encouraged to provide feedback in a Figure 2: Second prototype: Message details screen confident way we deemed literacy tests counterproductive to this end.To select icons, the user could horizontally scroll the top part ofthe Conversation screen to bring up the screens containing icons: The semi-structure interview focused on their use of mobileQuick sender, Feelings and Places and Activities. While the top phones and SMS in their every day lives. We asked them to showpart of the screen was horizontally scrollable, the New message us their mobile phones and the main functions they used.editor area remained fixed on the bottom of the screen. To select Specifically we probed how they checked call logs, if they storedan icon, the user simply had to drag it into the New message contacts on their phones and how they interacted with SMS. Weeditor and to listen to its meaning he had to single tap on it. As in used note taking by the experimenter in-situ and a video camerathe previous prototype, icons themselves were appended to the that recorded the participant’s actions on the mobile phone and themessage editor. discussions we had with them.We introduced a second approach for multi-sentences icons. It We started the usability test by demonstrating the application, therequired the user to tap and hold a multi-sentence icon to open a content of which was entirely in French. We demonstratedpop-up on the right hand side of the icon with all associated navigating through the different screens to check for newsentences. A small play button at the end of each sentence allowed messages, listen to a new message and reply to a message byfor playing it. To append the sentence the user had to tap on it in double tapping on icons and re-using existing words fromthe pop-up. We tried both this pop-up based version and the pre- previous messages. Before having them listen to the meaning oflisten version described in section 4.1 in study 2. selected icons, we asked them what their meaning might be. When they could not infer the meaning of an icon we used for a5. STUDY 2 particular phrase we asked them to sketch or to explain us how they would represent this idea visually. We then demonstrated5.1 Participants how the audio counterpart of an icon was invoked by tapping onWe conducted exploratory lab-based tests with three paid (20 it. After this demonstration we asked our participants to repeat theCHF/h) participants that had participated in study 1. All of them same actions and encouraged them to take the phone to scroll, tapspoke French as their second language and had recently started a and double tap to get familiar with the touch screen UI. Thiscourse to learn how to read and write in French. All three of them watch and repeat approach was supposed to emulate theirwere from a course for beginners from one of the aforementioned learning strategy when confronted with new technology with a
  6. 6. literate helper, as mentioned in study 1. Throughout the session For the usability test, the video recordings were our first source ofwe tried using simple, non-technical language for all explanations. data collection. We reviewed the recordings and for eachFor example, the participants were not familiar with terms like performed task reviewed the kind of errors they made and onapplication and icons. The pictograms and icons we referred to as which screen it occurred. Due to the low number of participants‘little pictures’, for example. We encouraged them to talk-out- we did not conduct any statistical tests, however. During theloud especially about any problems they encountered or parts they debrief interviews, we elicited if they found the application easyfound unclear. We stressed that if they did not understand the to use and whether it would be useful for them in their everydayapplication or parts of it was not their fault but the programmers. lives.Once we felt that they were confident and understood the mainfeatures of the application we started the usability test, which 5.3 Results All three participants were comfortable with their own phones.focused on multi-sentence icons, message composition and They navigated very quickly on it and used several functionalitiesreading, and specifically the karaoke feature. such as radio, photo camera apart from making and receivingWe started with the two different versions of the multi-sentence calls. The man from Senegal even used a football app on hisicons both of which were available from different icons on the iPhone to check the outcome of football matches and who scoredsame screen. We compared the pre-listen version we introduced in since he understood both the number format of scores and thethe first prototype with the pop-up based one introduced in the roster, which featured players’ head shots along with icons forsecond prototype. Recall that in the pre-listen version users had to goals scored. In terms of SMS all of them knew how to handletap and hold on the icon to listen to all the sentences associated to and open incoming SMS and used literate helpers for the in a row. Then, to select the sentence number i, users had to tap The three of them were numerate and knew how to read date andon the icon i times. In the pop-up version users were required to time but they found the latter easier from a digital than from antap and hold on the icon and a little pop-up appeared on the screen analog clock with handles. When asked if she knew how to searchwith all the associated sentences. To select a sentence, users for her messages one participant proudly showed how quick shesimply had to tap on it. For both versions we asked them to long- was at searching for new SMS. She knew how to create a newtap on the multi-sentence icon and queried whether they had an SMS but could not compose text in it. She used SMS very oftenidea on how to append one of the offered sentences to the message with the help of literate friends and had 256 SMS in her inbox. “Ieditor. know how to check the call logs, how to delete, how to do almostFor the composition, we situated them in the following scenario: everything on my cell phone, the only problem I have is reading“Let’s suppose you received an SMS from Amisha a friend of and writing SMS.” The Senegalese man never used SMS since ityours. You can see you have a new message from Amisha in your was too long and too complicated for him to try composing oneINBOX [participants are in the thread screen]. Now, you can tap but he had a number of SMS in his inbox, which mostly containedon this message to see why Amisha is sending you this SMS telephone numbers of people along with their names. His wife had[participants navigate to the message details screen (see Figure read the SMS to him and he consulted them when he needed the1)].” After they had navigated to the Conversation screen we phone number of that contact.made them listen to what Amisha had sent by tapping on the play The two iPhone owners succeeded in sending the SMS “Tonight,button next to the text message “Cinema tonight?” that was on top no”. The third participant seemed not as confident. She hardlyof the list. To double check that they had understood the audio touched the phone during the whole interview even if wemessage we asked them to explain why Amisha had sent them an encouraged her several times to do so. Worrying that this mightSMS. After their explanation, we asked them to reply that they embarrass or stress her too much we refrained from pushing herwere not free tonight with “Tonight, no.” To make it easier, we further through the scenario. This participant found it difficult tobroke this task into two subtasks through which we walked the come up with possible meanings of the icons and struggled withparticipants: the concept of text being associated with the icons. For her icons(1) We asked them to reuse the word “tonight” from the previous represented or were related to actions: “this [pointing to the message by first finding it in the previous message and to smiling emoticon] means I am talking with someone and this append it to the message editor. When necessary we reminded pointing to [sad emoticon] represents the person I am talking them to use a double tap on the word. with”. The two iPhone owners roughly understood the meaning of(2) We asked them to find the icon “No” from the list of icons in the icons but were not entirely sure. Asked about the meaning of the Quick sender screen and to append it to the editor. the call icon (depicting a receiver) one said: “This might meanBefore sending they had to review the composed message by ‘Call me’ or maybe ‘I will call you later’”. Hearing the audiotapping on the “play” button. counterpart removed any doubts for them.For reading we tested what happened if we removed the karaoke The idea of having multiple meaning for an icon and making themfunction (the words currently played were highlighted in red). We available (in both versions) through multiple taps was challenginghad two versions of the application: one with karaoke support and for all participants. None of them succeeded in appending aanother one in which the whole sentence was played out but with sentence to the editor and asked for help on what they had to visual feedback in the UI. We tested two sentences in French In the pre-listen version the length of the entire prompt “One:“When do you come back?“ (“Tu rentres quand?”) and “Cinema sentence 1, Two: sentence 2, Three: sentence 3” was too long andtonight?” (“Cine ce soir?”). First we asked the participants to play at the end the participants could not remember the first sentenceout the sentences and identify as many words as they could in the anymore. In the version with the pop-up, they were surprised by itkaraoke version. Then we asked them to repeat this with the same and did not know where to tap to listen to the several associatedsentences in the version without the karaoke. They could listen to sentences. The corresponding play buttons at the end of eachthe message as many times as they wanted. At the end of each sentence in the pop-up were relatively small but clearly visible.sub-task completed we provided congratulatory or encouragingfeedback. We tested playing back a message with and without the karaoke. With the karaoke, all of them succeeded matching some words to
  7. 7. the played sound. While the karaoke was playing, the woman avoid misinterpretations especially when seeing them for the firstfrom Morocco remarked: “Oh, yes, cinema, this word is cinema… time. Additionally, any mistakes could easily be corrected byCi ne ma” she pointed at the word and tapped on it to check she deleting erroneously added words from the editor. The corpus ofwas right. Without the karaoke, the participants did not even icons was limited but we hope that this will provide an initialrealize there was a link between what they were hearing and the entry point for illiterate users to create words through which theysentence played by the phone. can express themselves. Obviously a speech recognition facilityNone of our participants seemed uncomfortable with being tested could be more versatile and powerful.but their confidence varied. The woman from Angola often asked The participants struggled with the concept of multi-sentence“Am I right? Am I saying the right thing?” while the other two icons. For the pre-listen version none our participants understoodwere more self-confident. The man from Senegal immediately that the numbers “1, 2, 3” corresponded to the number of timeswanted to touch the phone, play the messages, drag some icons they had to tap on the icon to add the sentence to the messageinto the message editor and scroll to go through all the screens. editor. Thus, after long-tapping on the icons and listening to theWhen he and the other iPhone user succeeded in sending the three options, the users did not know what to do since the voiceSMS, they asked: “Thats it? Is my message really sent?” They prompts did not provide any action cues. Instead of having theseemed surprised by the simplicity. rather abstract guideline “One: I will be late. Two: …” we shouldFrom the beginning, the woman from Morocco was excited about have given an action cue such as “Tap this icon once for I will bethe application: “This could be wonderful for people like me, is it late, tap it twice for…”. The combined prompts were too long andpossible to get the application on my mobile phone today?” The in hindsight reminded us of Medhi et al.’s recommendations aboutother iPhone owner called us one hour after the interview to thank short and simple audio instructions [12]. Our participants did notus about dedicating our time to help “people like him” and succeed in memorizing the three different meanings for a singleexpressing his interest in obtaining the application. icon. Once reminded of the sentence and that the numbers corresponded on how many times they had to tap on the icon,At the end of the test, they seemed proud for helping us and for their main problem was that there was no feedback on how manybeing useful to help researchers from a respected university. The times they had already tapped on the icon. This behavior was alsofeedback we obtained from the teachers of the school was very inconsistent with how words and regular icons responded to taps.positive and conveyed that the man from Senegal was“transformed” after the session and for the first time he learned Compared to voice mail or voice-based SMS services (e.g. India’shis lesson for the next day. VoiceSMS) our application offers additional value. In voice mail, users need to have network access to compose a new SMS, with6. DISCUSSION our application however, users can review and compose theirAlthough our two studies were based on a small number of users SMS offline. Standard SMS are cheap or even free (e.g. a hundredwe found consistently how proficient illiterate users were in SMS per day) as part of certain prepaid contracts. Mostnavigating and using their mobile phones – be it low-end, feature importantly voice mails offer no potential learning whereas ouror smart phones. Since two out of three users in study 2 had application provides an audio-visual matching between text andiPhones already, our results were biased compared to users who audio, which can represent a source of learning for users.had never used smartphones before but it added to the existing According to Srivastava [20] an India NGO has startedevidence that using a smartphone proficiently is not a cognitive encouraging women to buy mobile phones English because of thematter but a matter of habits. Illiterate users who are used to potential to learn various alphabets through them. We do not wantsmartphones can be as proficient as literate users in using their to claim that illiterate users will learn how to read and write withmobile phones at least for the functions that are important to them. this application alone. But we see potential for it in providing additional encounters with text with concrete short-term goalsSimilar to Lalji & Good’s finding in which participants were providing reading practice and thereby incentivizing anduncomfortable with touch screens our feature phone owner from catalyzing literacy acquisition. Particularly the fact that ourAngola, despite encouragement was disinclined to touch the participants were not able to identify words after removing thephone. In the few cases when she did her styled, long and curved karaoke function convince us that illiterate, neo-literate and semi-fingernails made interactions with the touch screen seem a little literate users will find this application helpful. Every day exposureawkward because they would click on the screen first and the to text in conjunction with audio in same language subtitles ofangle for touching was quite low. But both iPhone users had no movie content was also shown to improve reading and writingproblem interacting with our touch screen based application skills in neo-literates [7]. Semi-literates in Findlater et al.’s studywhatsoever and with initial explanations managed to compose benefited from combination of text and audio and had superiormessages successfully. Like Medhi et al., we do not believe this to word recognition at the end of each session after the second day ofbe a cognitive issue since the other two participants were use [3].confident using the touch screen even with an application thatthey had no previous experience with. Before testing touch-screen Chipchase recommended that phones for illiterates should not beapplications users need to be taught the basics of touch screen recognizable as such because of the associated stigma [2]. Theinteraction. In contrast to Katre’s study, our participants had no only thing that might reveal a user’s illiteracy to by-standers whileproblems using their index fingers for interacting with the touch using our application is the sound played when tapping on wordsscreen although some of the icons were relatively small. We are and icons. This can be mitigated by headphone use. Moreover, allaware that these differences with Katre’s observations might be the SMS sent from our application are regular SMS. If andue to the difference between our users (Swiss immigrants from EasyTexting user sends an SMS there will be no way for thedeveloping countries) and the rural farmers he studied. recipient to know that it was written with an application for illiterates.As can be expected, the icons we used – although carefully chosen- were not self-explanatory. Each participant had his own Recruiting and running studies with illiterate users in Westernrepresentation of an idea. Audio support for icons was helpful to countries is a challenge since they are not numerous and since
  8. 8. they usually try to hide their illiteracy. Our way to get in contact approach for blind users. We used monochrome simple icons towith them was via schools. Establishing initial contacts with the mimic the WP7 metro design’s look and feel.schools and to gain the trust of the staff and teachers took time.Partly, they wanted to make sure we were going to treat theirstudents with respect and without a patronizing attitude. Despitethe testimonials of teachers only some of the students volunteeredto participate despite remuneration. Almost all of our participantswere financially relying on their partners.7. FINAL DESIGNThe entry screen of the application depicted in Figure 3 is theInbox screen, which contains all the threads of received messages.Each thread item contains the picture, phone number, name of thecontact and one line of the last exchanged message. The last threedigits are highlighted by putting them apart from the rest to aidrecognition of contacts by phone numbers as mentioned in [6].Tapping on a list item brings up all the messages exchanged withthis particular contact. Figure 4: Third prototype: Conversation screen (left) Quick sender sub-screen (right) The final design of EasyTexting application is composed of two main screens: the Inbox and the Conversation, the latter of which extends to the sub-screens providing access to icons (Quick sender, Feelings, Places and activites). The Inbox screen is similar to the existing SMS composition tool on WP7 except for the added contact picture and the last three digits of the phone numbers that are visually separated. The Conversation screen with the history of all the previous messages exchanged with someone differs in various points from the WP7 counterpart. The picture, phone number and name of the contact users are exchanging SMS with are displayed at the top of Figure 3: Third prototype: Inbox screen the page. While the middle of the screen is scrollable, this part atContrary to the other prototypes, we added a text label underneath the top is fixed. We followed the Windows convention andeach icon (cf. Figure 4, right). For the composition of a message displayed SMS in speech bubbles. However, each word is a buttonthe user can rely on icons, re-use of words or both. Double the user can reuse in a new message. This removes the need fortapping on a word in a previous message results in appending the copy and paste functionality of the regular SMS version. Butword to the message editor (the grey speech bubbles in Figure 4). words can only be added sequentially to the end of the messageIcons only have one meaning and the user has to scroll editor. The current application does not allow users to use thehorizontally to the Quick Sender screen as illustrated in Figure 4. keyboard, attach a picture (MMS) or to save it as a draft.Analogously to words, single taps on icons play the sound of the Comparing to the standard SMS application on WP7, ourwords or sentence associated with it. Contrary to the previous application includes sound support. Each word is a playableprototypes in which double tapping on an icon placed the icon button and each SMS can be played with karaoke support.itself on the message editor, double taps on icons place their Without this feature, users cannot “read” or understand thecorresponding words in the message editor. As with received content of an SMS by themselves and not make use of textmessages each word of the sentence under composition is a button messaging.and on tap delivers its audio. Double tapping on words in the From the Conversation screen, users can directly access the iconsmessage editor results in its deletion. We enlarged the word dictionaries screens by scrolling horizontally. Each icon isborders to improve tapping on single letter words and punctuation. playable and has a predefined sentence associated with it.Initially we had experimented with single taps on icons and wordsto add them to the editor and long taps for the audio. But after 8. CONCLUSIONsome corridor tests long-taps proved to be too time-consuming. Along this research we discovered that illiterate people did useSince the equivalent of a mouse over event does not exist on touch their mobile phones a lot but were unable to use text-basedscreens we needed to find a way to provide its audio rendition applications. Managing their contacts and dealing with SMS werewithout triggering another action. We settled with single taps to the two things they struggled with most or could not do at the sound and double tap to add the icon’s associated text to However when it came to SMS, they used some tricks such asthe message editor. Thus, a single tap was used to represent a asking their relatives to read SMS or calling back the senders. Onmouse over event. This is akin to the iPhones accessibility our prototype, we kept many UI conventions that we had found usable for illiterate users such as the threaded view of SMS and
  9. 9. the main presentation of the inbox screen. No previous on touch-screen phones for illiterate users were 2005/index.html.developed before. Our findings from two studies add to the 3. Findlater, L., Balakrishnan, R., and Toyama, K. Comparingevidence that using touch-screen phones does not represent a semiliterate and illiterate users’ ability to transition fromcognitive problem for illiterate users but only a problem in terms audio+text to text-only interaction. Proceedings of the 27thof lacking confidence or technological literacy. We found international conference on Human factors in computingpromising first evidence that illiterate users can use text systems, ACM (2009), 1751–1760.messaging in conjunction with audio, text and visuals when initial 4. Huenerfauth, M.P. Design approaches for developing user-training is provided. Overall, users we interviewed were interested interfaces accessible to illiterate users. University Collegein making use of text messaging and some of them wanted to take Dublin, Ireland, (2002).the application home with them. From our findings we argue that 5. Katre, D. One-handed thumb use on smart phones by semi-ICTD research should not reduce mobile phones to mere literate and illiterate users in India: A usability report withtelephones with simplified storage for contacts. This restrictive design improvements for precision and ease. Proceedings ofapproach would most likely fail in the market place because it Workshop on Cultural Usability and Human Work Interactiondenies illiterates to enjoy other functions such as entertainment Design, NordiCHI Conference, Lund, Sweden, (2008).through music, pictures and video. Touch screen phones with on- 6. Knoche, H., Huang, J. Text is not the enemy - How illiteratesdemand voice feedback can enable illiterate users to use use their mobile phones. NUIs for New Worlds: Newpotentially important information services by leveraging the Interaction Forms and Interfaces for Mobile Applications inaffordances of multimedia UIs on touch screen phones. Chipchase Developing Countries - CHI’2012 workshop, (2012).concluded that to improve literacy skills the best solution would 7. Kothari, B., Takeda, J., Joshi, A., and Pandey, A. Samebe a phone. We aimed at this by providing an application that language subtitling: a butterfly for literacy? Internationalallows illiterates to compose and listen to SMS. We combined Journal of Lifelong Education 21, 1 (2002), 55–66.icons, audio and text and in-synch highlighting of read out words 8. Kotkar, P., Thies, W., and Amarasinghe, S. An audio wiki forto aid recognition and possibly reading acquisition. In our publishing user-generated content in the developing world.application words are objects that react to taps and reveal their HCI for Community and International Developmentmeaning in audio form. Initial tests with touch-screen experienced (Workshop at CHI 2008), Florence, Italy, (2008).participants showed potential for this approach. 9. Kumar, A., Agarwal, S.K., and Manwani, P. The spoken web application framework: user generated content and service9. FUTURE WORK creation through low-end mobiles. Proceedings of the 2010We plan to further add to this application by improving the input International Cross Disciplinary Conference on Webof text a) through keyboard entries, e.g. for numbers b) through Accessibility (W4A), ACM (2010), 1–10.speech recognition c) by reusing words from previous SMS from 10. Lalji, Z. and Good, J. Designing new technologies for illiterateall threads d) providing tactile feedback when words are added to populations: A study in mobile phone interface design.the message composer e) by providing a movable insertion point. Interacting with Computers 20, 6 (2008), 574–586.We would like to improve the contact manager for illiterate users 11. Medhi, I., Gautama, S.N., and Toyama, K. A comparison ofboth for picking contacts and the management itself. Searching mobile money-transfer UIs for non-literate and semi-literatethrough a contact long list of contacts is time consuming for users. Proceedings of the 27th international conference onilliterate users since the search is based on alphabetic order. Human factors in computing systems, (2009), 1741–1750.Moreover, creating a new entry can be difficult when written 12. Medhi, I., Prasad, A., and Toyama, K. Optimal audio-visualnames are required for a contact – see [6] for more details. representations for illiterate users of computers. Proceedings of the 16th international conference on World Wide Web,We plan to port the application to the Android platform and (2007), 882.extend it with speech recognition for the composition of messages 13. Medhi, I., Sagar, A., and Toyama, K. Text-free user interfacesand carry out field studies. We would like to evaluate the for illiterate and semiliterate users. Information Technologiesapplication with illiterate and semi-literate users was well as and International Development 4, 1 (2007), 37–50.elderly. 14. Patel, N., Chittamuru, D., Jain, A., Dave, P., and Parikh, T.S. Avaaj Otalo—A Field Study of an Interactive Voice Forum10. ACKNOWLEDGMENTS for Small Farmers in Rural India. Proceedings of theWe would like to express our gratitude to Oscar Bolanos and Proceedings of the 28th international conference on HumanLukas Frelich for helping with the implementation; Anne factors in computing systems (Atlanta, GA, USA, 2010). ACM,Marquis, Catherine Wick, Annick Mello Spano and the teachers (2010).from Lire-et-écrire and Français-en-jeu and all interviewees for 15. Prasad, A., Medhi, I., Toyama, K., and Balakrishnan, R.their time; Jeffrey Huang, Jan Blom, Florian Egger, Mairi Willis, Exploring the feasibility of video mail for illiterate users.Daniel Keller, Gunnar Harboe and Saket Sathe for providing Proceedings of the working conference on Advanced visualvaluable feedback and guidance. This research has been funded by interfaces, (2008), 103–110.the Swiss Development Council in collaboration with 16. Rao, K.V. and Sonar, R.M. M4D Applications in Agriculture:cooperation@EPFL. Some Developments and Perspectives in India. Defining the “D”in ICT4D, (2009), 104–111.11. REFERENCES 17. Shankar, T.M.R. Speaking on the Record. 2004.1. Bhamidipaty, A. Symab: Symbol-based address book for the 18. Sherwani, J., Palijo, S., Mirza, S., Ahmed, T., Ali, N., and semi-literate mobile user. Human-Computer Interaction– Rosenfeld, R. Speech vs. touch-tone: Telephony interfaces for INTERACT 2007, (2007), 389–392. information access by low literate users. Proc. IEEE/ACM2. Chipchase, J. Understanding non-literacy as a barrier to Int’l Conference on Information and Communication mobile phone communication. Technologies and Development, (2009).
  10. 10. 19. Smyth, T.N., Kumar, S., Medhi, I., and Toyama, K. Where 21. UNESCO. Gender and Education for All: The Leap to there’s a will there’s a way: mobile media sharing in urban equality. 2003. india. Proceedings of the 28th international conference on Human factors in computing systems, (2010), 753–762. international-agenda/efareport/reports/20034-gender/.20. Srivastava, Kendra. Indian Women Learn Alphabets on Handsets. Mobiledia.