The document analyzes research on the validity and reliability of apps for use in adult rehabilitation. It finds that while thousands of apps exist for potential use, there is limited research evaluating their efficacy. Only two studies were identified that examined app psychometrics, with one app showing good to moderate test-retest reliability but mixed convergent validity. Most virtual reality and computerized tools analyzed had more validity and reliability data reported. The document concludes that more research is needed on establishing assessment criteria and evaluating app psychometric properties to support their use as evidence-based tools in occupational therapy.

1. The Validity and Reliability of Apps in Adult Rehabilitation
Daniel M. Fichter, OTS and Dr. Rochelle Mendonca PhD, OTR/L
Introduction
There is a national push for the use of evidence-based practice
to quantify occupational therapy evaluation and intervention
processes. 7, 1
The use of technology is increasingly present in occupational
therapy rehabilitation settings, but with limited research on efficacy
for specific populations. 3
Thousands of apps exist that can potentially be utilized in
occupational therapy rehabilitation facilities. Virtual tools allow for
increased repetition, standardization, control and environmental
simulation. 7, 5, 2, 6
Studies report that there is a need for widespread evaluation and
intervention tools for cognitive skills such as neglect, executive
functioning and memory. 2. 1, 6, 8, 4
Methods
Research studies on the use of apps in rehabilitation settings were
gathered and analyzed using levels of evidence and statistical
reports to determine the clinical uses of apps and their validity and
reliability. All search terms were recorded by the researcher and
documented to reveal effectiveness. See the handout for a list of
search terms.
The articles chosen for this analysis mainly concern cognitive
performance skills within functional performance.
Results
Discussion
Research providing psychometric data on the use of apps in adult
rehabilitation is lacking in a time when it is necessary to utilize
evidence-based practice. Only two research articles studying the use
of apps were identified, one of which was created and tested by
researchers with good to moderate test-retest reliability (ICC=0.712)
and mixed convergent validity with the Life Satisfaction Index K
(r=-0.045 - 0.297) 7, 3. When a clinician chooses an existing app, he
or she must account for the lack of supporting psychometric data.
Many of the analysed tools run on other hand-held or virtual reality
platforms. Evidence on the use of virtual reality and reminder
systems on phones is more abundant and can be considered as a
model for practitioners using apps. Many of the virtual tools are
concerned with cognitive functioning and 8/9 present with detailed
descriptive sample data 1,2,3,4,5,6,7,8. Virtual assessments, including apps,
can be inherently higher in reliability and validity due to their abilities
to be systematically repeated and representative of more natural
environments than paper and pencil versions.
Conclusions
Research shows that the use of virtual assessment and intervention
tools can be perceieved as effective due to increased collaboration,
motivation, control, ecological validity and personalization 1,2,3,4,5,6,7,8.
Apps have inherent features that promote reliability and validity
Further research is necessary to progress the use of apps into an
evidence-based model.
Research on the psychometric properties of apps needs to begin
with the creation of assessment criteria, or an assessment tool to
rate their function with specific populations, settings and skill deficits.
Clinicians should be vigilant while using apps in rehabilitation
settings and, with strong clinical reasoning, utlize existing
population-specific research on virtual reality and hand-held systems
to generalize treatment outcomes to the use of apps.
Abstract
Research on the use of apps (iPad, iPod, iPhone) and virtual
systems was analyzed for their validity and reliability data to
provide further information on the evidence supporting their use in
outcome-based practice. The researcher searched multiple
scholarly databases and found that there is a lack of clinically valid
evidence pertaining to the use of apps in adult rehabilitation
settings. This analysis seeks to answer, “Is there reliable and valid
evidence to support the use of apps for occupational therapy
evaluation and intervention with adults in rehabilitation facilities?”
Platform Used Program Name Psychometric Data Reported? Detailed Description of Participants? Level of Design
iPad Aid for Decision-Making in Occupation
Choice (ADOC) (7)
Validity (V): Yes
Reliability (R): Yes
Yes Level III: Cohort
iPod Touch Clock App, Notes App, Video Camera App,
“Simply Being” App, VoCal Verbal
Reminder App, iCal App, Storykit App,
iReward App (3)
V: No
R: No
Yes Level IV: Multiple Case Study
Immersive camera and large screen using
Neuro VR software
Virtual Mltiple Errands Test (VMET) (5,
6)
V: Yes
R: No
Yes Level II: Cohort
Non-immersive laptop with keyboard
interface
Virtual Action Planning Supermarket
(VAP-S) (5, 4)
V: Yes
R: No
Yes Level II: Cohort
Not Reported Virtual Environment Technology (VET) (5) V: Yes
R: Yes
Not Reported (Literature Review) Not Reported (Literature Review)
Desktop computer with stereo
headphones, haptic feedback pen,
numeric keyboard, 19” monitor and
shutter glasses
VR-DiSTRO (2) V: Yes
R: No
Yes Level IV: Cohort
Palm PDA Auditory and Visual Reminders (1) V: No
R: No
Yes Level II: Repeated Measures Crossover
Microsoft Pocket PC Auditory and Visual Reminders (1) V: No
R: No
Yes Level II: Repeated Measures Crossover
Not Reported Computerized Cognitive Assessment
System (CCAS) (8)
V: Yes
R: No
Yes Level IV: Descriptive
Table: Two out of the nine virtual systems reported run on platforms that support apps. Two virtual tools report reliability and validity data, while four
report validity data only. Three virtual tools are reported with no psychometric data. All reported tools have detailed descriptions of participants.
(1) Dowds, M. M, Lee, P. H., Sheer, J. B., O’Neil-Pirozzi, T. M., Xenooulos-Oddsson, A., Goldstein, R., . . . & Glenn, M. B. (2011). Electronic reminding technology following traumatic brain injury: Effects on timely task
completion. Journal of Head Trauma Rehabilitation, 26, 339-347. doi: 10.1097/HTR.0b013e3181f2bf1d
(2) Fordell, H., Bodin, K., Bucht, G., & Malm, J. (2011). A virtual reality test battery for assessment and screening of spatial neglect. Acta Neurological Scandinavica, 123, 167-174. doi: 10.1111/j.1600-0404.2010.01390.x
(3) Gentry, T., Lau, S., Moinelli, A., Fallen, A., & Kriner, R. (2012). The Apple iPod Touch as a vocational support aid for adults with autism: Three case studies. Journal of Vocational Rehabilitation, 37, 75-85. doi:
10.3233/JVR-2012-0601
(4) Josman, N., Schenirderman, A. E., Klinger, E., & Shevil, E. (2009). Using virtual reality to evaluate executive functioning among persons with schizophrenia: A validity study. Schizophrenia Research, 115, 270-277. doi:
10.1016/j.schres.2009.09.015
(5) Poulin, V., Korner-Bitensky, N., Dawson, D. R. (2013). Stroke-specific executive function assessment: A literature review of performance-based tools. Australian Occupational Therapy Journal, 60, 3-19. doi:
10.1111/1440-1630.12024
(6) Raspelli, S., Pallavicini, F., Carelli, L., Morganti, F., Pedroli, E., Cipresso, P., . . . & Riva, G. (2012). Validating the neuro VR-based virtual verson of the Multiple Errands Test: Preliminary results. Presence, 21, 31-42.
Retrieved from: http://www.mitpressjournals.org/loi/pres
(7) Tomori, K., Saito, Y., Nagayama, H., Seshita, Y., Ogahara, K., Nagatani, R., & Higashi, T. (2013). Reliability and validity of individualized satisfaction score in aid for decision-making in occupation choice. Disability &
Rehabilitation, 35, 113-117. doi: 10.3109/09638288.2012.689919
(8) Yip, C. K., & Man, D. W. K. (2009). Validation of a computerized cognitive assessment system for persons with stroke: A pilot study. International Journal of Rehabilitation Research, 32, 270-278. doi:
10.1097/MRR.0b013e32832c0dbb
Presentation presented in partial fulfillment of the requirements for OT 668: Evidence-Based Practice.
Faculty: Colleen Maher, OTD, OTR/L, CHT