Recall - Evaluation of route learning software on Android for people with disabilities
RECALL Project 2009-504970-LLP-1-2009-1-UK-KA3-KA3MP Evaluation of route learning software on Android for people with disabilities – Jacqui Lewis, Camelia Popescu, Penny Standen and Maria SaridakiThe project is partially funded under Key Activity 3: Informational and Communication Technologies (ICT) Lifelong learning programme of the European Commission The content of this project does not necessarily reflect the position of the Commission, nor does it involve any responsibility on its part
Project Partners:Nottingham Trent University – UK Greenhat Interactive Ltd – UK BID Services - UKMarie Curie Association – Bulgaria University of Athens – Greece CPPC – Romania
RationaleEvidence presented by the Disability Rights Commission (DRC) indicatesthat people with disabilities experience considerable disadvantage in termsof transport and travel. For example:• People with disabilities are unable to use 70 per cent of buses and 40 per cent of the rail network.• 60 per cent of people with disabilities have no car in the household, compared with 27 per cent of the general population.• Almost half (49%) of people with disabilities are totally reliant on public transport.• Over half (56%) of people with disabilities said that they would like to go out more.• People with disabilities are twice as likely to turn down a job due to travel difficulties.• More than half (52%) of people with disabilities expressed some difficulties in getting to all essential services such as GPs and hospitals.• The research phase of our project showed that travel skills and confidence to learn new routes are core skills for leading independent lives; showing users their exact location relative to where they need to get to can help people who struggle with spatial skills.
Project Aims• To counteract exclusion that occurs when people with learning disabilities leave statutory services, exacerbated by the anxiety of carers about their vulnerability in the community.• To overcome physical and psychological barriers to independent travel and community inclusion.• To reengage people into active citizenship• To increase access to community activities by developing independent travel skills
Objectives• The project utilizes location based services in assistive technologies to allow people with learning disabilities and sensory impairments to develop lifelong learning skills of: route planning, memory, concentration, stress management, time-management & understanding targets and deadlines.• Three modes of RECALL allow our target audience to plan and use routes to reconnect with community opportunities; and to play to reinforce the connection between maps and the real world.
Description of Application• Provides a tool for developing and supporting routes that involves and enables the route learner• Allows carers and trainers to have additional confidence• Uses mainstream and contemporary location aware technology• Provides other travel training elements such as reminding the user when to leave• Provides geographically based prompts• Allows specific safety based prompt e.g. at road junctions• Provides a means to contact a carer in the event of an emergency• Not a replacement for travel training, but another tool trainers can use to engage and keep users on task
Route Mate vs Satnav approachRoute Mate attempts to Traditional routeuse technology to guidance (location based)enhance route learning turn by turn instruction – Route creation is part – Route generation is of the process automatic – User defines their own – Instructions presented waypoints to the user – Supports cognitive – Suppresses cognitive map formation map formation
Route Mate - ConsoleUsing a basic desktop/laptop and internetconnection, the user, parent or carer can:• Load the routes from a device• Preview and edit routes• Duplicate routes• Save/load routes to/from PC• Save new and modified routes to the phone• Delete routes from the phone.
Plan Mode•Plan mode gives the user the option to create a newroute, or load and modify an existing route with thehelp of a parent, carer or trainer. They can enter thestart point using selection on a map and can alsoenter the start time of their journey, set daily alarmsand end address of their new route.•An emergency contact can be specified and points ofinterest can also be set between the start and endpoints using the phone’s camera to break the journeyup into a number of smaller routes connected by keylandmarks and more effectively scaffold its learning.
Route Mate ‘Use’ Mode•The user can rehearse the route a number oftimes accompanied by a trainer or teacher beforeindependent travel. The Use Mode allows theuser to travel more independently and rely less onthe application and more on their own skills.•To facilitate independent movement, the screenturns off while travelling between points ofinterest in the Use Mode.•This is important because over reliance might bedangerous for example looking down at thescreen whilst crossing a road unaccompanied.
Challenge Mode - GamificationTo teach/reinforce the concept ofmaps and route learning, a gamesapproach has been adopted that:• Allows students to insert different icons to waypoints in planning mode (e.g., treasure)• The student is then challenged to find this treasure in the real world, using game play around treasure hunts, or pirates, for example.• Promotes the connection between the map and the real world
Project Evaluation Completed Ongoing/Future:• Issues Tracking for • Scientific Review functionality• Iterative Testing for • Internal and external product usability and impact measurement accessibility against key• Piloting performance indicators• Soft outcomes measurement
Scientific Review – RoleThe external evaluator will:1. ensure that we are able to demonstrate the scientific quality of our work so that others can trust our outputs by being able to provide enough evaluation detail to reassure them that our conclusions are sound and to enable them to judge whether our products would be suitable for their client group2. monitor our use of accepted inclusive design standards and also the use of the issues tracker for bug fixing3. produce a state of the art review of other developments to test for product innovation
Qualitative Evaluation –internal and externalFurther qualitative evaluation will take twoapproaches to outcome measurement, both ofwhich will measure the delivery of the projectagainst a number of key performance indicators formeasurement of sustainability and social impact ofthe project and its results.•the first of these will be conducted internally withproject partners•the second externally with project stakeholders.
Usability Testing methodologyBetween month 9 and month 30 of the project, iterative testing wasconducted by all partners using different tools for experts and final users.Tools:• Instructions for deploying testing• List of tasks that users should attempt with Route Mate• Observations sheet to record any usability issues• Template to collect dataThroughout this period of testing an issues tracker was used by all partnersand experts to record any problems with functionality of the device. Thisallowed the programmer to bug fix as we went along to keep the processflowing, so that the device would be ready for piloting
Testing Tasks• Testing with final users was conducted in teams including one assistant (to assist users and to ensure their safety) and one evaluator (to observe and record the data). Spontaneous feedback from the participants and their facilitators was recorded. The participants were given training in the use of the phone and were shown how to navigate the application.• Relatively quiet and short roads were chosen. A shorter route allowed more time to be spent gathering feedback. The length of routes chosen for testing varied between 200-1600 m and the time spent on the road varied between 10 – 30 minutes.• At least two testing sessions were carried out with every user and between 5 to 10 users tested the application in each country, following the same set routine of tasks.
Usability Testing ResultsAnalysis of TasksThe main characteristics of Route Mate tested were:• Open/ close the application• Use the virtual keyboard• Use the map (using arrows, using zoom buttons)• Set points• Take pictures• Use panic button• Take a routeA 4 level scale for registering answers was used:Level 1= could not complete even with helpLevel 2 = requiring considerable physical or verbal help to completeLevel 3 = requiring a little physical or verbal help (eg prompts) but could otherwise complete the task unaidedLevel 4 = completed with no helpThe amount of time spent looking at the device (percent of total time of walking) was:• 22 cases between 70% -90%• 6 cases between 20%-50%.
Testing Results• Analyzing the main characteristics of Route Mate we found out that for the users the application was in most of the cases easy and engaging to test and only in a few cases were no answers registered because of a functionality error.• Open/close the application was completed mostly with no help - 20 responses• Use the virtual keyboard was completed mostly requiring a little physical or verbal help -18 responses• Using the map (using arrows, using zoom buttons) was completed mostly requiring a little physical or verbal help - 17 responses• Set points was completed mostly requiring a little physical or verbal help - 20 responses• Take pictures was completed with requiring a little physical or verbal help - 16 responses• Use panic button was completed mostly requiring a little physical or verbal help - 21 responses. (4 non answers were recorded because of errors)• Take a route was completed requiring a little physical or verbal help - 13 responses. (8 non answers were recorded because of errors)
Main areas of physical difficultyTaking pictures was one of the characteristic where the answers variedthe most depending on which version of Route Mate was tested. Thisrequired considerable physical or verbal help to complete in somecases – 7 responses.Using the virtual keyboard and using the map (using arrows, usingzoom buttons) required sometimes considerable physical or verbalhelp to complete depending of the disability of the users:• Using the virtual keyboard –required considerable physical or verbal help to complete - 6 responses• Using the map (using arrows, using zoom buttons) required considerable physical or verbal help to complete - 8 responses
Users’ Typical ReactionsSome examples of reactions and behaviours of users:• Users weren’t always sure of menu layout and where to find the Route Mate icon.• Users were unsure how to locate the ‘close keyboard’ icon to be able to progress to the next screen.• When moving around the map screen in Plan mode, accidental clicking meant the option to add a point kept coming up, which caused confusion with users• Some users found it difficult to know where they were on the map while taking a route and, therefore, how to proceed.• Difficulties with using the keyboard due to fine motor limitations. This cannot be rectified within the project – it is a hardware issue.
Functionality• An online issues tracker (Red Mine) was used to allow the software engineer to be constantly updated on functionality problems encountered during iterative testing.• These issues were then prioritised and bugs fixed accordingly.• Testers were then informed as each issue was dealt with and signed off.
Functionality – Main Issues reportedMain issues have been: - Crashes / freezes - usually a coding mistake. Crash reporting tool (ACRA) integrated into the app to allow analysis of the events leading up to the crash - Photos not being recorded - only occurred on some devices, not others. Image capture code was rewritten. - Off route warning not consistently working - code corrected. Quite a few issues were determined to be device specific, so it was decided to pilot with a consistent configuration. Galaxy Nexus devices were used as they have an up to date version of android, and dont have any manufacturer (e.g. Sense UI (HTC) or TouchWiz (Samsung)) or carrier (e.g. hard coded home page, Telco apps) additions to the operating system.
Piloting TasksSCENARIOOutline Scenario• 3-5 Sessions• 5-10 end usersPiloting Scenario1st Session: Training and first use of Route Mate• Training by the researchers - max 15 mins• Soft Outcomes Star™ used by End Users (physical disabilities) or caretakers (learning disabilities) - max 15 mins• Piloting Process - Participant observation• Familiarize with the software• Create first route on console - by facilitator/end user• Use first route - by facilitator/end user/researcher• End of route, return to the premises - by facilitator/end user/researcher• Debriefing Process - max 15 mins
Piloting Tasks contd2nd Session: Semi-Autonomous use• Piloting Process - Participant observation• Use existing route - by facilitator/end user/researcher• End of route, return to the premises - by facilitator/end user/researcher• Debriefing Process - max 15 minsEnd Session - Semi-Autonomous (if possible)• Piloting Process - Participant observation• Use existing route - by end facilitator/end user/researcher• End of route, return to the premises - by facilitator/end user/researcher• Soft Outcomes Star™ used by End Users (physical disabilities) or carers (learning disabilities) - max 20 mins• Observational checklist - Likert Scale - max 20 mins
Observational checklist - Likert Scale• Documents feedback - open questions• Closed type questionnaire with interval scales• Invites open comments from both end users and facilitators
SOFT OUTCOMES STAR ™ tool The soft outcomes star™ is a tool to measure and record the difference a service/training has made to clients by showing progress made in qualitative areas of life across a period of time such as from the beginning to end of a course delivered. The tool is used with clients in a non-threatening way to help diagnose where they might need more support as well as what they have achieved. It can also be used as evidence of value of a project or course and to demonstrate a return on their investment to project funding agencies in areas that are hard to quantify.
Piloting Results – Likert Scale•Carers and users were very positive about Route Mate.•General attitude towards Route Mate was positive even incases of technical issues•Users are willing to use Route Mate when the app hasbeen finalised•Most of them credited Route Mate with an increase inconfidence and in helping relatives and families to bereassured that users are safe when travelling independentlyusing the app. The panic button was one of the mostimportant features for the family
Piloting Results – Likert Scale•There were insignificant differences in the way caretakers(carers, teachers, trainers, support workers) answered thequestionnaire.•The only demographic characteristic that was found to beimportant is experience in technology.•Experienced caretakers found Route Mate to be usable and easilyaccessible and had more positive attitude towards it.•Users that had previous experience with technology scored better•All categories of users showed significant improvement in thesecond measurement•Older users performed worse in the first measurement butimproved more than the other categories in the finalmeasurement•Impact was the same across disability categories
Piloting Results - Star Tool and Observation• Users with intellectual disabilities show no qualitative differentiation in contrast with individuals with other kinds of impairments/disabilities, providing us with an important indication that Route Mate can be an equally useful learning and assistive tool for users with different disabilities
Piloting Results – Star ToolStar Tool: All categories increase significantly in the second measurement
Piloting Results•During the piloting sessions and as the technical problemsstarted to slowly resolve: –Stress levels decreased –Users’ confidence increased –Users felt encouraged and motivated to use Route Mate.•As expected, previous experience with mobile technologyhad an effect on users’ self confidence and therefore theirease with Route Mate.•Games and game based learning elements reduce thesteepness of the learning curve and increase satisfaction andmotivation
Piloting ResultsReasons that increased the levels of stress:•Technological problems•Disbelief about technology’s reliability in general•Feelings of frustration about not being able toaccess the right technology (smartphone) becauseof prohibitive costs and accessibility•Risk (fear of being victimized when holding anexpensive mobile)According to Soft Outcomes Star™findings, these stress levels can be reducedthrough repeated use of Route Mate.
Piloting Methodology Results - ImpactEnd users could clearly see the immediate impact that Route Mate could have on their lives:• “Route Mate will be really helpful next year when I go to college”• “Will let my mother know that I can get there safely”, End users could see the potential that Route Mate could have on expanding and improving their daily lives:• “I can see my friends more often”• “I can learn new and unfamiliar routes”
Piloting Methodology Results - Motivation• Motivation to use Route Mate was also increased) by the “Where to go Next?” screen and the photo gallery, which according to most of the users made the route more interesting and enjoyable• Motivation from using Route Mate: two cases where the use of Route Mate prompted them to improve themselves or at least triggered the desire to improve themselves. One student with learning disabilities who was unable to read or write. After the first session with RM she expressed to her parents the desire to learn how to read. In a second case, after a RM session, the end user started to search on the internet for similar programs and assistive technologies.
Piloting Methodology ResultsBugs and optimization:•The intermediate stops have to be actual places, things that theend users can recall instantly and not commands or actions.•The pop up info about the next stop has to appear before theusers reach a street, otherwise they tend to freeze in the middle ofa street raising important safety issues.•For some students with sensory impairment or fine motor controlissues, the keys were too small•Individuals in wheelchairs could not use Route Mate while moving.They had to stop, use the application and then continue theirjourney
Piloting Methodology ResultsProposals for further improvement:• To add a voice command or to be able to activate audio commands• To add haptic feedback as an alternative for people with visual impairments• To enhance the more game-like aspects of the app.
Thank you for listening! RECALL teamhttp://recall-project.eu