From 2014-2016, the National Networks of Libraries of Medicine, MidContinental Region ran the Mobile App Experience Project. This program provided funds for health science librarians to purchase apps via iTunes or Google Play. In exchange, participants were asked to thoroughly evaluate each app and share their experience in using them with the library community. Evaluators were equipped with an App Evaluation Report Form (AERF) to help develop a systematic and critical evaluation of mobile apps. The AERF guided the users to report on an app's: authority of information sources; accuracy and objectivity; currency of information; organization and usability; and purpose. After completing the project, participants were asked to respond to a brief questionnaire about their experience. This presentation will provide an overview of the project, its challenges and successes, and suggestions for how you can establish a similar program in your college or university.

1. REVIEWING APPS:
TAKING EVALUATION
SKILLS BEYOND THE
LIBRARY
Alicia Lillich
Kansas Outreach and Technology Coordinator
A.R. Dykes Library
University of Kansas Medical Center

2. BACKGROUND

3. NATIONAL NETWORK OF
LIBRARIES OF MEDICINE (NNLM)
NNLM, formerly Regional Medical Library Program
Established in the Medical Library Assistance Act (1965)
Mission
 Provide health professionals and the general public with equal access to
biomedical information
Coordinated by the National Library of Medicine
8 Regional Offices
 Support Network members to achieve NNLM mission
 5-year grants

4. NNLM MIDCONTINENTAL REGION
(MCR)
6 State Region
Based out of the University of Utah
Free Membership
 Mostly Academic Health Sciences and Hospital Libraries
Provide support in: education, community engagement,
advocacy, technology, and evaluation

5. METHODOLOGY

6. PURPOSE
Provide funds for librarians to purchase apps
Evaluate and share with the community
Librarians benefit from the experience
Improve Confidence

7. PARTICIPANT REQUIREMENTS
MCR Network member
Professional librarian*
Experiment and report on 4 appropriate apps
Submit reports by deadlines

8. RULES FOR APP SELECTION
Apps must be appropriate for their setting
 Purposely kept broad
Must cost money
 Or free apps with in-app purchases

9. REPORTING
Summaries of each review highlighted in the Plains to Peak Post (MCR
newsletter)

10. PARTICIPANTS
Year One 13
Library Type
Hospital 6
Academic 7
Card Type
iTunes 10
Google Play 3
State
Colorado 3
Kansas 3
Missouri 2
Nebraska 3
Utah 1
Wyoming 1
Year Two 19
Library Type
Hospital 5
Academic 14
Card Type
iTunes 15
Google Play 4
State
Colorado 3
Kansas 2
Missouri 7
Nebraska 4
Utah 2
Wyoming 1

11. COST AND CARDS DISTRIBUTED
Year 1
Distributed $650 = 13 x $50
 10 iTunes
 3 Google Play
Year 2
Year 2: $950 = 19 x $50
 15 iTunes
 4 Google Play

12. EVALUATION TOOL

18. RESULTS

19. APPS EVALUATED
Excellent
1Password
AnkiMobile
BioStat Decision Tool
CalenMob Pro for Google Calendar
Clear
Clinical Pattern Recognition: Knee Pain
Concussion Assessment & Response: Sport Version
Dexteria - Fine Motor Skills Development
Duet Display
Explain Everything
Family Medicine Study Guide
Family Practice Notebook
FRAX:WHO Fracture Risk Assessment Tool
GoodReader
IFTTT
iMoodJournal
Infectious Disease Compendium: A Persiflager's
Guide
iThoughts
Kidometer
Kidometer
KidsDoc
Medrills: Pediatric considerations
Micromedex Drug Reference Essentials
MindNode
Muscle Trigger Points by Real Bodywork
NCLEX-RN Alt-format
Patient Education Bundle
Patient Education Bundle-Spine
patient education bundle; Heart
Patient Education Bundle: Heart, Ortho and Spine
PDF Expert 5
Pocket Anatomy
Pocket Heart, version 2.3
Prognosis: Your Diagnosis
Research Methods for Library and Information
Science Professionals - Book by Martin Frické
Spine Pro III
SplashID
Stop Breathe & Think
The Mindfulness App
Transesophageal Echocardiography (TEE): Colour
Doppler
Visible Body Atlas
Very Good
ABCs of Medical Management of Stones
ArchieMD 3D Health Essentials
Buddhify
Byword
CBT Keeper
Diagnosaurus DDX
Differential diagnosis pocket. Also referred to as
DDX Pocket
Drawboard PDF
Encyclopedia of Alternative Medicine
English<>Spanish Medical
Evernote
Feed Baby
Flashcards Deluxe
GoScholar
ICU Rounds
iGeriatrics
igeriatrics
Infectious Disease Compendium: A Persiflager's
Guide
Interfaith Care for the Ill
JKZ Series2 and JKZ Series3 (Mindfulness Apps by Jon
Kabat-Zinn)
Journal Club for Android
Lab Values Pro
Lab Values Pro 3 in 1 Medical Reference
LastPass Password Mgr Premium
MD on Call
MedCalc - The Professional Medical Calculator
Medical Microbiology Pronunciations
Medical Spanish: Healthcare Phrasebook with Audio
Mediquations
Mediquations Medical Calculator
Memeo Connect Reader for ipad
MRSA eGuideline
My Wonderful Days: Daily Journal/Diary
NCLEX-RN Exam Prep 2015
Noteshelf
Nurse Tabs Complete
OnExam
Pediatric Quiz Game
Physical Exam Essentials
Pill Monitor – Medication Reminders and Logs
Prioritize Me
Prizmo Scanning & OCR app
Psych On Demand
PubMed on Tap
RH Medical Labs
SoundNote
The Mindfulness App
Todo
Fairly/Mildly Good
AHIMA Pocket Glossary
Animated Pocket Dictionary of Allergy
Dara Torres Health & Fitness
Fooducate
GoodReader
Instant Anatomy Flash Cards
Journal Club for Android
LabGear 3.0
Large Print Message Composer
Lippincott's NCLEX-RN
Marvin
Medical Abbreviations
Medical Doctor: Reference Tool
Medical Microbiology
Medical Term Pronunciation
Medical Terms Abbreviation
MobileDDx
mySymptoms
NCLEX-RN Practice Test & Questions
Osteopathic Medicine pocketcards By Börm
Bruckmeier Publishing LLC
Outliner
Pain Therapy
palmEM
Pediatric Blue Book
Polaris Office5
PSA (Psychological Symptoms Analyzer)
RN Cheat Sheet
Shelby's Quest
Shelved
SportsMed
TalkBoard
Ultimeyes
Not Good
Everyday First Aid
Instant Blood Pressure
Nursing History
T'ai Chi for Life

20. APP COSTS
Year 1
Total Spent: $305
Average Cost: $6.63
Highest App Cost: $24
Lowest App Cost: Free
Leftover Funds: $345
Year 2
Total Spent: $625
Average Cost: $8.22
Highest App Cost: $45
Lowest App Cost: Free
Leftover Funds: $325

21. App focus:
health/medicine
productivity
0
10
20
30
40
50
60
Health/Medicine Productivity
Year 1 Year 2
APP TYPES

22. APP RANKING
0
5
10
15
20
25
30
Excellent Very Good Fairly/Mildly Good Not Good
Year1 Year2

23. RATE OF PARTICIPATION
73% 74%
21%
26%
6%
Year 1 Year 2
No Report
Late
On Time

24. 0
5
10
15
20
25
30
35
A B C D E F G
Quality Grading of Evaluations Evaluations were down-
graded for:
 Blank Responses
 Using “Information Not Available”
response
Range A (0 deductions) to G
(6 deductions)
QUALITY OF EVALUATIONS

25. SELF-EVALUATION
11%
22%
31%
5%
5%58%
47%
54%
26%
26%
15%
Y2: My involvement in
this project benefited
or enhanced
my professional
development.
Y2: I now feel more
confident in
my ability to
evaluate mobile
apps.
Y1: Participating in this
project benefited my
program. Strongl
y
Agree
Agree Neither
Disagr
ee
No
Response
Positiv
ely
Very
Positiv
ely

26. LESSONS LEARNED

27. POSITIVES
Fun project to manage
Diverse apps were selected
Happy with the evaluations

28. PARTICIPANT COMMENTS
“You have made this a fun year!”
“This enabled me to critically evaluate apps that I otherwise
would not have downloaded.”

29. From the KUMC Mobile Apps LibGuide

30. CHALLENGES
Conservative on spending
Managing participants
 Meeting deadlines
 Quality of evaluations
Evaluation form problems
 Too comprehensive
 Confusion between fairly/mildly good rating

31. PARTICIPANT COMMENTS
“I found the evaluation form for this project hard to fill out”
“Sometimes it was difficult to decide which app to test drive.
Lots of information on free apps, but how do you find apps with
a fee that are appropriate?”

32. RECOMMENDATIONS
Limit funds
Revise evaluation form
 Shorten
 Require responses
More guidance in app selection

33. IDEAS FOR
COLLEGES/UNIVERSITIES
Faculty App Reviews
Student Group
Information Literacy / English Comp. Assignment
Partner with IT or other campus departments

34. ANY QUESTIONS?
This project has been funded in whole or in part with Federal funds from the Department of Health and Human Services, National
Institutes of Health, National Library of Medicine, under cooperative agreement number UG4LM012344 with the University of Utah