2. What is a good app ?
Effective (is it good for us )
Cost-effective
Feasible
Profitable
User-friendly
Safe
Relevant
Usable
Side effects shown
Dose response known
Mode of action known
4. What is a good app ?
Effective
Cost-effective
Feasible
Profitable
User-friendly
Safe
Relevant
Usable
Side effects shown
Dose response known
Mode of action known
How much
uncertainty / bias
do we accept ?
5. Should we delay access to massive
improvements over the status quo while we
wait for the perfect study to satisfy
academic criteria
Why all the fuss...
Isn’t innovation always good ?
Exposure to
innovation
6. Connecting the dots…
Aging
Chronic disease
Healthcare
utilization / costs
(super) Computers / networks
Robotica
Regenerative medicine
3d /4d Printing
Sensors / imaging
Omics
Artificial intelligence
+ Mobile technology / wearables
Prosperity
Autonomy claim /
paradigm shift
Smartphone use
Innovation
Challenges
Solutions
Changes
7. The Mhealth revolution.....
Over 97.000 medical applications available and growing big time….
…..and the sky is the limit regarding the possibilities
• Academic focused innovation Private sector focused innovation
‘Zolderkamer’ focused innovation
• High thresholds for implementation based on level of evicence Relatively
low thresholds for implementation based on:
• CE licence
• Expert ratings
• Subjective user ratings (i.e. 5 starr ratings)
9. Drawbacks / Risks
• Proliferation (overgrowth) of apps (+wearables) without adequate
regulation leaves both patients, healthcare providers and policy
makers behind in how to select apps
Lack of knowledge or transparency in:
o Level of evidence (minimal threshold = non-inferiority / cost-
neutrality
o Underlying principles and theoretical background
o Development / engineering processes
User centered principles / experts involved ?
Usability/acceptability/manageability ?
Technical features ? Both software and hardware
o Safety issues
o Ethical issues
o Medical costs issues
Introduces risk of non-use, low
effectiveness, mobile ‘kwakzalverij’
or even adverse / hazardous effects
10. What is a good app ?
Financial transactions and incentives
Conditional cash transfers
Insurance
Payment for services
Performance-based incentives
Savings accounts
Information systems
Data collection and reporting
Service delivery statistics
Household surveys
Surveillance (public health)
Electronic health records
Registries/vital events tracking
Service delivery and support
Electronic decision support
Disease diagnosis/Point-of-care diagnostics
Disease management
Disease prevention
Provider-to-provider communication
Referrals
Remote client-to-provider consultations (Telemedicine)
Social and behavior change communication
Appointment reminders
Health education or promotion
Hotlines and information services
Mass messaging campaigns
Treatment adherence
Supply management
Cold chain management
Commodity tracking/replenishment
Counterfeit prevention
Maintenance of equipment
◦tock out prevention
Workforce development and performance support
Constituent feedback on service quality
Human resource management
Provider training and education
Provider work planning and scheduling
Supportive supervision
11. Example (I)
• Opioid conversion apps
• N= 26 (Android, Appstore etc.)
• Findings:
• calculated dosages are highly variable, with statistically significant
differences in conversion outputs between apps with stated medical
involvement and those without in some cases
• few apps appear to identify the primary data source underlying their
calculation algorithm
• Unknown whether there has been involvement in app creation or
content of individuals who have practical experience in or insight into
the undertaking of these high-risk prescribing decisions
Haffey et al. Drug Saf (2013) 36:111–117
12. Example (II)
• Melanoma detection apps
• N= 4 scanning 60 melanoma cases and 128 benign lesion
controls
• Findings:
• The performance of smart phone applications in assessing melanoma
risk is highly variable, and 3 out of 4 smart phone applications
incorrectly classified 30% or more of melanomas as unconcerning.
• Reliance on these applications has the potential to delay the diagnosis
of melanoma and to harm users.
Wolf et al. JAMA Dermatol. 2013 April ; 149(4): 422–426
13. What is a good app ?
Financial transactions and incentives
Conditional cash transfers
Insurance
Payment for services
Performance-based incentives
Savings accounts
Information systems
Data collection and reporting
Service delivery statistics
Household surveys
Surveillance (public health)
Electronic health records
Registries/vital events tracking
Service delivery and support
Electronic decision support
Disease diagnosis/Point-of-care diagnostics
Disease management
Disease prevention
Provider-to-provider communication
Referrals
Remote client-to-provider consultations (Telemedicine)
Social and behavior change communication
Appointment reminders
Health education or promotion
Hotlines and information services
Mass messaging campaigns
Treatment adherence
Supply management
Cold chain management
Commodity tracking/replenishment
Counterfeit prevention
Maintenance of equipment
◦tock out prevention
Workforce development and performance support
Constituent feedback on service quality
Human resource management
Provider training and education
Provider work planning and scheduling
Supportive supervision
What/who is:
• the end-user
• the primary outcome,
• the sequence of
anticipated effects
• degree of acceptable
uncertainty
?
?
?
?
?
?
14. What is a good app ?
Regular care
mHealth
innovation
outcome = f (mHealth innovation) + (regular care)
What do we aim to evaluate ?
• superiority
• non-inferiority
• equivalence
Regular care
vs
mHealth
innovation
16. Example 1: Airstrip – obstetrical monitoring
Remote fetal monitoring
in at risk deliveries.
partly replacing regular
outpatient care
↓ delay in
diagnosis fetal
distress
↓ communication
breakdowns
among clinicians
↓ fetal deaths
↓ admission
days
Exposure Intermediates Outcomes
↓ delay in
treatment fetal
distress
17. Taxonomy
Proof of mechanism
(theory, requirement
analysis)
Proof of concept
(usability, safety,
dose-response )
Proof of principle
(effectiveness,
safety, process)
Mechanism of mHealth
application is reasonably:
relevant
safe
feasible
(cost-)effective
Concept of mHealth application is
provisionally:
safe & side effects known
feasible
usable
Mode of action as anticipated
+ dose-response known
Principle of mHealth application is
certainly:
safe
≥ non-inferior
feasible to implement
presumably/reasonably/potentially provisionally certainly
Pre-clinical Clinical
18. Engineering meets science
To develop and evaluate future according to pre-
defined quality standards and user-centered principles
(co-creation)
Development through science
(scientific engineering)
19. Advantages of scientific/academic engineering
State of the art:
• Access to theory (epidemiology, treatment and healthcare processes)
• Multidisciplinary expertise
• Research + (meta-)datamanagement facilities
• facilities + Platforms for knowledge dissemination
• ABOVE ALL : (early) access to large groups of
target users (patients + care providers)
20. No ivory towers…
Science
TechnologyCare
Collaborative research & development
Collaborative R&D
strategic alliances
reduces financial and
technical risk and encourages
knowledge exchange, supply
chain development and
parallel working on complex
challenges.
• Partnership fund
application
• Direct investments
cofinancing
• Crowd funding
21. Iterative and user-centered Mhealth
development
Background analysis &
design conceptualization
Alpha-usability
(paper prototype)
Field-usability
Iterative
software
development
Theory
(review)
Patients
Focus
group
+
interviews
Experts
Focus
group
Patients
Task analysis
+
interviews
Patients
Field study
(task analysis,
usability +
interviews
phase 2 phase 3 phase 4phase 1
22. Phase 1: design conceptualization
Background analysis &
design conceptualization
Theory
(review)
Patients
Focus
group
+
interviews
Experts
Focus
group
phase 1
Aim:
to conceptualize the preliminary design of the Mhealth intervention and to identify a set of
delivery components that are active and make a difference in the intervention outcome
23. Iterative and user-centered Mhealth
development
Background analysis &
design conceptualization
Alpha-usability
(paper prototype)
Theory
(review)
Patients
Focus
group
+
interviews
Experts
Focus
group
Patients
Task analysis
+
interviews
phase 2phase 1
Aim:
to observe the human interaction with user interfaces even before ‘real’
interfaces are designed and developed.
24. Iterative and user-centered Mhealth
development
Background analysis &
design conceptualization
Alpha-usability
(paper prototype) Iterative
software
development
Theory
(review)
Patients
Focus
group
+
interviews
Experts
Focus
group
Patients
Task analysis
+
interviews
phase 2 phase 3phase 1
Aim:
To carry out a standard iterative production process with information
derrived from phase 1 & 2
25. Phase 4: Field/lab Usability
Background analysis &
design conceptualization
Alpha-usability
(paper prototype)
Field/Lab-usability
Iterative
software
development
Theory
(review)
Patients
Focus
group
+
interviews
Experts
Focus
group
Patients
Task analysis
+
interviews
Patients
Field study
(task analysis,
usability +
interviews
phase 2 phase 3 phase 4phase 1
Aim:
to assess the degree to which a system is effective, efficient, accessible and favors positive
attitudes and responses from the intended users
1-1 meetings. Field or lab ‘think aloud’ task analysis with
n=5 patients
• Learnability
Assessment of how easy it is for users to accomplish basic tasks the first time through
and to be able to work quickly without spending much time searching for instructions and
screen interface commands.
• Efficiency
Refers to how quickly a user takes to perform each of sequential tasks.
• Memorability
Assessment of the ease at re-establishing each of the tasks after a period of nonuse.
• Errors
Assessment of user errors and ability to quickly recovery from errors.
• Satisfaction
Assessment of pleasantness for users.
26. Development of complex interventions
User-centered development
Testing / pilot
Evaluation
Implementation /
exploitation
Chief engineering = PhD student
Dissemination
of knowledge
27. Output
• Evidence-based innovative mHealth applications
• Far-reaching knowledge exchange
• Peer-reviewed / professional publications
• PhD thesis
• Etc.
• Valorisation and implementation
