Challenges in evaluating eHealth applications

Associate Professor Department of Health Policy, Management and Evaluation, University of Toronto; Canada Senior Scientist , Centre for Global eHealth Innovation, Division of Medical Decision Making and Health Care Research; Toronto General Research Institute of the UHN, Toronto General Hospital, Canada Challenges in evaluating eHealth applications Visiting Professor, Faculty of Behavioural Sciences University of Twente, The Netherlands Editor & Publisher, Journal of Medical Internet Research (www.JMIR.org) Keynote at “Supporting Health by Technology II”, Enschede, NL, May 28th, 2009 Gunther Eysenbach MD MPH Gunther Eysenbach MD MPH

The ethnic theory of plane crashes (Chapter in “Outliers” by Malcolm Gladwell)

“ The single most important variable in determining whether a plane crashes is not the plane, it's not the maintenance, it's not the weather, it's the culture the pilot comes from. Planes are flown safely when the pilot and co-pilot are in open and honest communication. And in cultures where it is difficult for a junior person to speak openly to a superior, you have lots of plane crashes.” (Source: Gladwell)

Power Distance Index (P.D.I.) [Geert Hofstede] = measurement of how much a particular culture values and respects authority. Countries with a high P.D.I. generally value being more deferential towards authority, and thus not contradicting a superior.

Analogies to Healthcare

Pilot = health care professional

Co-pilot = patient

Power Distance = Inverse Patient Empowerment

Eysenbach. Random Research Rants (Blog) http://www.webcitation.org/5h5jkooUX

Why patient empowerment / patient involvement?

Safety/Quality

Trust

Health Outcomes?

Community (listserv) Communication (email) Content (WWW) stress anxiety depression Public with peers (P2P) with health professionals (P2D) Private with family & friends Self- efficacy Doctor- patient- communication Virtual social support (weak ties) Realistic expectations satisfaction confusion Benefit pathway Harm pathway information Asking right questions Local social support (strong ties) Θ Increases Decreases Θ Θ Positive health outcomes knowledge Θ (?) Θ Shared decision making/ compliance Too much / low quality Tailored / high quality Θ Θ loneliness Θ Eysenbach G. The impact of the Internet on cancer outcomes. CA Cancer J Clin 2003; 53(6): 356-371 “ Internet” О + О + О + О + О + О + О + О + О + О + О + О + О + О + О + О + О + О + О + О + О + О + О + О +

“ It saved me spiritually and psychologically,” she says. “ I could compare notes with patients around the world, and I have made friends through [this mailing list] that rival lifelong relationships.” Anecdotes suggest that (p2p) online services have an impact on patients…

“ Until I discovered Internet support, my experience of parent support had been both positive and beneficial. The Internet changed all that. During a downturn in my daughter's health it was suggested to me to join the Internet community associated with her disorder and (…) I decided to give it a try. What followed has been an experience that added to my trauma beyond what words could express.” ..though not all of them are only positive…

Challenges in evaluating eHealth interventions

Meyer B, Berger T, Caspar F, Beevers CG, Andersson G, Weiss M Effectiveness of a Novel Integrative Online Treatment for Depression (Deprexis): Randomized Controlled Trial J Med Internet Res 2009;11(2):e15 URL: http://www.jmir.org/2009/2/e15/ The Gold Standard: Randomized Controlled Trial

Problem: Adherence => Attrition RCT open Eysenbach G The Law of Attrition J Med Internet Res 2005;7(1):e11 <URL: http://www.jmir.org/2005/1/e11/>

www.JMIR.org

“ Predictors of adherence included disease severity, treatment length, and chronicity. “ Christensen H, Griffiths KM, Farrer L Adherence in Internet Interventions for Anxiety and Depression J Med Internet Res 2009;11(2):e13 URL: http://www.jmir.org/2009/2/e13/

What drives / motivates consumers + patients? Healthy Acute Condition Chronic/Severe Condition Motivation

Ease of use Usability User-centered Design Other predictors for attrition / lack of adherence

Essential: Needs assessment, formative evaluation, usability testing (iterative & ongoing)

Focus Groups

Usability lab

In-depth interviews with stakeholders

Gaps between patient and provider needs / expectations

“ Patients are particularly likely to anticipate that shared records will be empowering (...).”

“ Physicians, by contrast, are especially likely to anticipate that laboratory results will confuse patients and that shared records will make patients worry more. “

Expectations of Patients and Physicians Regarding Patient-Accessible Medical Records Stephen E Ross, MD, Jamie Todd, MS-IV, Laurie A Moore, MPH, Brenda L Beaty, MSPH, Loretta Wittevrongel, Chen-Tan Lin, MD J Med Internet Res 2005 (May 24); 7(2):e13

Gaps between patient and provider needs / expectations Credits: Selina Brudnicki & Claudette DeLenardo

Ease of use People will not enter health information to a significant degree…

… (perhaps there are some exceptions)…

… rather, eHealth sites / PHR (or PHA platform) must be populated seamlessly and effortlessly… Web 2.0 (collaborative, data entered by others) Mobile technologies, SMS Domotics, Ambient, pervasive computing, Intelligent car Applications with geospatial awareness Electronic Medical Record (Provider) PHR / PHA Platform Natural speech interfaces Personal Monitoring Tools

Sorbi MJ, Mak SB, Houtveen JH, Kleiboer AM, van Doornen LJP Mobile Web-Based Monitoring and Coaching: Feasibility in Chronic Migraine J Med Internet Res 2007;9(5):e38 <URL: http://www.jmir.org/2007/5/e38/> Health issues do not only occur at home - mobile interfaces are essential

Intelligent spoon

But even ubiquitous computing / sensors is not always appropriate and accepted

“ I wouldn't want to track (a variable or in general) because tracking would… “

Not apply to me: (eg, smoking, alcohol drinking, pets)

Not provide new information: (ie, “I already know this”)

Not provide valuable information

Provide too much information (information overload)

Threaten self-image (“would feel criticized”)

Not provide actionable information

Lead to social conflict

Promote obsessive or unhealthy reactions: (“becoming obsessed”)

Force too much structure (“Approaching life too analytically”)

Not be suitable for particular activity or behavior

Be too complicated, error-prone, or disruptive

Beaudin JS, Intille SS, Morris ME To Track or Not to Track: User Reactions to Concepts in Longitudinal Health Monitoring J Med Internet Res 2006;8(4):e29 <URL: http://www.jmir.org/2006/4/e29/>

Communities / peer-pressure as a tool to enhance adherence

Social Uses of Personal Health Information Within PatientsLikeMe, an Online Patient Community: What Can Happen When Patients Have Access to One Another’s Data Jeana H Frost, Michael P. Massagli J Med Internet Res 2008 (May 27); 10(3):e15

http://en.wikipedia.org/wiki/Image:Web20_en.png

Medicine 2.0 (“next generation medicine”) From: Gunther Eysenbach. Medicine 2.0: Social Networking, Collaboration, Participation, Apomediation, and Openness J Med Internet Res 2008; 10(3):e22 http://dx.doi.org/ 10.2196/jmir.1030 DOI: 10.2196/jmir.1030 Gunther Eysenbach MD, MPH , www.medicine20congress.com Consumer / Patient Health Professionals Biomedical Researchers Science 2.0 Peer-review 2.0 Personal Health Record 2.0 Virtual Communities (peer-to-peer) Professional Communities (peer-to-peer) Health 2.0 HealthVault Google Health HealthBook Sermo WebCite CiteULike Medting WiserWiki eDoctr BioWizard Dissect Medicine E-learning PLoS One BMC JMIR Wikis Blogs RSS RDF, Semantic Web Virtual Worlds Web 2.0 Technologies & Approaches Apomediation Participation Social Networking Collaboration XML AJAX Openess Revolution Health PatientsLikeMe PeerClip Connotea ALIVE HealthMap caBIG Doctorshangout.com Asklepios

Gunther Eysenbach MD, MPH , www.medicine20congress.com September 17-18, 2009 

Venkatesh,Viswanath;Morris,Michael G.;Davis,Gordon B.;Davis,Fred D., “User acceptance of information technology: Toward a unified view”, MIS Quarterly, 2003, 27, 3, 425-478.

Teresa Chiu, PhD Thesis, University of Toronto

Reporting Attrition/Adherence

Attrition is the norm, not the exception - don’t try to cover it up

“ Attrition curves” now a standard for reporting website use over time (at least at the J Med Internet Res )

Image source: Meyer B, Berger T, Caspar F, Beevers CG, Andersson G, Weiss M Effectiveness of a Novel Integrative Online Treatment for Depression (Deprexis): Randomized Controlled Trial J Med Internet Res 2009;11(2):e15 URL: http://www.jmir.org/2009/2/e15/

Dealing with Attrition/Adherence

Use appropriate statistical methods for dealing with missing data

Refrain from LOCF (last observation carried forward)

Use models instead (ANOVA etc)

Challenge 2: The problem of “controlling” the control group

On the Internet, similar interventions may be accessible for the control group (e.g. smoking cessation)

difficult to “control” what the control group does

privacy / ethical concerns limit the amount of data which can be gathered (e.g. logging all accessed URLs)

Eysenbach G Issues in evaluating health websites in an Internet-based randomized controlled trial J Med Internet Res 2002;4(3):e17 <URL: http://www.jmir.org/2002/3/e17/>

Challenge 3: Re-registration, participant identity

re-registration of (anonymous) participants eager to get the intervention

“ neither requesting optional provision of offline contact details, nor monitoring IP addresses satisfactorily addressed the issue”

Murray E, Khadjesari Z, White IR, Kalaitzaki E, Godfrey C, McCambridge J, Thompson SG, Wallace P Methodological Challenges in Online Trials J Med Internet Res 2009;11(1):e9 URL: http://www.jmir.org/2009/2/e9/

Other challenges

Complexity of interventions

Data quality

Spam (reminder emails mistaken for spam)

Cybersquatting

Murray E, Khadjesari Z, White IR, Kalaitzaki E, Godfrey C, McCambridge J, Thompson SG, Wallace P Methodological Challenges in Online Trials J Med Internet Res 2009;11(1):e9 URL: http://www.jmir.org/2009/2/e9/

Thank you!