1. Steven R. Chan MD MBA, John Torous MD, Ladson Hinton MD, Peter Yellowlees MD MBBS
A Framework for Evaluating
Mobile phones are ubiquitous in society and amongst
psychiatric patients. Their versatility as a platform
can extend mental health services, but the efficacy
and reliability of publicly-available applications has
yet to be demonstrated. Numerous articles have noted
the need for rigorous evaluation of the efficacy and
clinical utility of smartphone applications, which are
largely unregulated.
Mental Health Smartphone Apps
Clinicians remain concerned about both efficacy and privacy.
While there is great potential for mobile applications for
psychiatric assessment and interventions, there is limited
data on efficacy), scarce research on application security, and
little empirical data on outcomes. In essence, little is known
about how these applications impact clinical care. However,
we also believe another critical factor limits the potential of
mobile applications: a lack of standardized evaluation and
rating tools that allow clinicians and patients to identify high
quality and safe mobile applications. Professional
organizations — including the American Psychiatric
Association, the American Psychology Association, National
Alliance on Mental Illness, and the Substance Abuse and
Mental Health Services Administration — do not provide
guidelines that patients or clinicians may use to evaluate
mobile applications.
Consequently, an unregulated free market exists in
which many applications are being developed that are of
uncertain quality and efficacy. Research on mobile
applications for mental health — for instance, smoking
cessation — has thus far demonstrated that smartphone
applications rarely conform to established guidelines or
integrate evidence-based practices. The most thorough meta-
analysis of mental health applications to date found only a
total of five applications meeting their inclusion criteria,
underscoring the lack of thorough evaluation and study of the
vast majority of mental health applications on the
marketplaces.
Criteria from engineering and informatics disciplines can assist in
creating guidelines. Heuristics governing software usability were
adapted for ambulance emergency medical services. The
Telehealth Usability Questionnaire adopted similar methods in an
assessment of a telepsychiatric system for adult autism
assessment. The Healthcare Information and Management
Systems Society (HIMSS) created checklists for mobile technologies
as part of their mHIMSS initiative. The American Telemedicine
Association also include similar criteria in the Lexicon of
Assessment and Outcome Measures for Telemental Health.
Patients and providers can use the following three dimensions of
evaluation criteria for mental health smartphone applications:
Usefulness dimension
➔ Validity and Accuracy — Does the application work as
advertised?
➔ Reliability — Will the application consistently function
from session to session?
➔ Effectiveness — Is the application clinically effective —
with demonstrated improved outcomes — for the target
population, disease, or disability?
➔ Time and number of sessions — What time is required
for the user to derive some benefit from the
application?
Usability dimension
➔ Satisfaction and Reward — Is the application
pleasurable and enjoyable to use, or does it discourage
repeat use?
➔ Usability — Can the user easily — or with minimal
training — use and understand the application?
OBJECTIVE / BACKGROUND
DEFICITS OF APPS
A POTENTIAL SOLUTION
PROPOSED FRAMEWORK
POSTER #
➔ Disability accessibility — Is the application usable by
those with disabilities (e.g. incorporates screen
readers for blind users, closed captions for the hard-
of-hearing and deaf communities)?
➔ Cultural accessibility — Does the application work
effectively with the user's culture (as defined by
factors such as ethnicity and language)?
➔ Socioeconomic and generational accessibility — Does
the application take into account socioeconomic
status? And, the user's age, with potential
implications for the user's digital health literacy?
Integration & Infrastructure dimension
● Security — Is the application's data encrypted on the
device and/or in transmission? Is it anonymized or
does it contain personal health information? If so,
what does it do?
● Workflow integration — Does the application work
within its user's workflow?
● Data integration — Does the application share data
with other applications, networks, and medical
record systems?
● Safety — Does the application take into account
patient safety, such as suicidality or homicidality?
Second, smartphone applications can target one or more of the
following stages in a provider’s workflow:
education & training
reference
history data input & output
physical data input & output
diagnosis
treatment and intervention
patient-provider communication
1
2
3