Associate Professor  Department of Health Policy, Management and Evaluation, University of Toronto; Canada Senior Scient...
 
 
The ethnic theory of plane crashes (Chapter in “Outliers” by Malcolm Gladwell)
“ The single most important variable in determining whether a plane crashes is not the plane, it's not the maintenance, it...
Power Distance Index (P.D.I.) [Geert Hofstede] = measurement of how much a particular culture values and respects authorit...
Analogies to Healthcare <ul><li>Pilot = health care professional </li></ul><ul><li>Co-pilot = patient </li></ul><ul><li>Po...
Eysenbach. Random Research Rants (Blog) http://www.webcitation.org/5h5jkooUX
Why patient empowerment / patient involvement? <ul><li>Safety/Quality </li></ul><ul><li>Trust </li></ul><ul><li>Health Out...
Community (listserv) Communication (email) Content (WWW) stress anxiety depression Public with peers  (P2P) with health pr...
“ It saved me spiritually and psychologically,” she says. “ I could compare notes with patients around the world, and I ha...
“ Until I discovered Internet support, my experience of parent support had been both positive and beneficial. The Internet...
Challenges in evaluating eHealth interventions
Meyer B, Berger T, Caspar F, Beevers CG, Andersson G, Weiss M Effectiveness of a Novel Integrative Online Treatment for De...
Problem: Adherence => Attrition RCT open Eysenbach G The Law of Attrition J Med Internet Res 2005;7(1):e11 <URL: http://ww...
www.JMIR.org
“ Predictors of adherence included disease severity, treatment length, and chronicity. “ Christensen H, Griffiths KM, Farr...
What drives / motivates consumers + patients? Healthy Acute Condition Chronic/Severe Condition Motivation
Ease of use Usability User-centered Design Other predictors for attrition / lack of adherence
Essential: Needs assessment, formative evaluation, usability testing (iterative & ongoing) <ul><li>Focus Groups </li></ul>...
 
Gaps between patient and provider needs / expectations <ul><li>“ Patients are particularly likely to anticipate that share...
Gaps between patient and provider needs / expectations Credits: Selina Brudnicki & Claudette DeLenardo
Gaps between patient and provider needs / expectations Credits: Selina Brudnicki & Claudette DeLenardo
Ease of use People will  not  enter health information to a significant degree…
… (perhaps there are some exceptions)…
… rather, eHealth sites / PHR (or PHA platform) must be populated seamlessly and effortlessly… Web 2.0 (collaborative, dat...
Sorbi MJ, Mak SB, Houtveen JH, Kleiboer AM, van Doornen LJP Mobile Web-Based Monitoring and Coaching: Feasibility in Chron...
Intelligent spoon
 
 
But even ubiquitous computing / sensors is not always appropriate and accepted
“ I wouldn't want to track (a variable or in general) because tracking would…  “ <ul><li>Not apply to me:  (eg, smoking, a...
Communities / peer-pressure as a tool to enhance adherence
Social Uses of Personal Health Information Within PatientsLikeMe, an Online Patient Community:  What Can Happen When Patie...
http://en.wikipedia.org/wiki/Image:Web20_en.png
Medicine 2.0  (“next generation medicine”) From: Gunther Eysenbach. Medicine 2.0: Social Networking, Collaboration, Partic...
Gunther Eysenbach MD, MPH ,  www.medicine20congress.com September 17-18, 2009 
Venkatesh,Viswanath;Morris,Michael G.;Davis,Gordon B.;Davis,Fred D., “User acceptance of information technology: Toward a ...
Teresa Chiu, PhD Thesis, University of Toronto
Reporting Attrition/Adherence <ul><li>Attrition is the norm, not the exception - don’t try to cover it up </li></ul><ul><l...
Dealing with Attrition/Adherence <ul><li>Use appropriate statistical methods for dealing with missing data </li></ul><ul><...
Challenge 2:  The problem of “controlling” the control group <ul><li>On the Internet, similar interventions may be accessi...
Challenge 3: Re-registration, participant identity <ul><li>re-registration of (anonymous) participants eager to get the in...
Other challenges <ul><li>Complexity of interventions </li></ul><ul><li>Data quality </li></ul><ul><li>Spam (reminder email...
Thank you!
Upcoming SlideShare
Loading in...5
×

Challenges in evaluating eHealth applications

3,372

Published on

Keynote at “Supporting Health by Technology II”, Enschede, NL, May 28th, 2009

2 Comments
8 Likes
Statistics
Notes
No Downloads
Views
Total Views
3,372
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
2
Likes
8
Embeds 0
No embeds

No notes for slide
  • Challenges in evaluating eHealth applications

    1. 1. Associate Professor  Department of Health Policy, Management and Evaluation, University of Toronto; Canada Senior Scientist ,  Centre for Global eHealth Innovation, Division of Medical Decision Making and Health Care Research;  Toronto General Research Institute of the UHN, Toronto General Hospital, Canada Challenges in evaluating eHealth applications Visiting Professor, Faculty of Behavioural Sciences University of Twente, The Netherlands Editor & Publisher, Journal of Medical Internet Research (www.JMIR.org) Keynote at “Supporting Health by Technology II”, Enschede, NL, May 28th, 2009 Gunther Eysenbach MD MPH Gunther Eysenbach MD MPH
    2. 4. The ethnic theory of plane crashes (Chapter in “Outliers” by Malcolm Gladwell)
    3. 5. “ The single most important variable in determining whether a plane crashes is not the plane, it's not the maintenance, it's not the weather, it's the culture the pilot comes from. Planes are flown safely when the pilot and co-pilot are in open and honest communication. And in cultures where it is difficult for a junior person to speak openly to a superior, you have lots of plane crashes.” (Source: Gladwell)
    4. 6. Power Distance Index (P.D.I.) [Geert Hofstede] = measurement of how much a particular culture values and respects authority. Countries with a high P.D.I. generally value being more deferential towards authority, and thus not contradicting a superior.
    5. 7. Analogies to Healthcare <ul><li>Pilot = health care professional </li></ul><ul><li>Co-pilot = patient </li></ul><ul><li>Power Distance = Inverse Patient Empowerment </li></ul>
    6. 8. Eysenbach. Random Research Rants (Blog) http://www.webcitation.org/5h5jkooUX
    7. 9. Why patient empowerment / patient involvement? <ul><li>Safety/Quality </li></ul><ul><li>Trust </li></ul><ul><li>Health Outcomes? </li></ul>
    8. 10. Community (listserv) Communication (email) Content (WWW) stress anxiety depression Public with peers (P2P) with health professionals (P2D) Private with family & friends Self- efficacy Doctor- patient- communication Virtual social support (weak ties) Realistic expectations satisfaction confusion Benefit pathway Harm pathway information Asking right questions Local social support (strong ties) Θ Increases Decreases Θ Θ Positive health outcomes knowledge Θ (?) Θ Shared decision making/ compliance Too much / low quality Tailored / high quality Θ Θ loneliness Θ Eysenbach G. The impact of the Internet on cancer outcomes. CA Cancer J Clin 2003; 53(6): 356-371 “ Internet” О + О + О + О + О + О + О + О + О + О + О + О + О + О + О + О + О + О + О + О + О + О + О + О +
    9. 11. “ It saved me spiritually and psychologically,” she says. “ I could compare notes with patients around the world, and I have made friends through [this mailing list] that rival lifelong relationships.” Anecdotes suggest that (p2p) online services have an impact on patients…
    10. 12. “ Until I discovered Internet support, my experience of parent support had been both positive and beneficial. The Internet changed all that. During a downturn in my daughter's health it was suggested to me to join the Internet community associated with her disorder and (…) I decided to give it a try. What followed has been an experience that added to my trauma beyond what words could express.” ..though not all of them are only positive…
    11. 13. Challenges in evaluating eHealth interventions
    12. 14. Meyer B, Berger T, Caspar F, Beevers CG, Andersson G, Weiss M Effectiveness of a Novel Integrative Online Treatment for Depression (Deprexis): Randomized Controlled Trial J Med Internet Res 2009;11(2):e15 URL: http://www.jmir.org/2009/2/e15/ The Gold Standard: Randomized Controlled Trial
    13. 15. Problem: Adherence => Attrition RCT open Eysenbach G The Law of Attrition J Med Internet Res 2005;7(1):e11 <URL: http://www.jmir.org/2005/1/e11/>
    14. 16. www.JMIR.org
    15. 17. “ Predictors of adherence included disease severity, treatment length, and chronicity. “ Christensen H, Griffiths KM, Farrer L Adherence in Internet Interventions for Anxiety and Depression J Med Internet Res 2009;11(2):e13 URL: http://www.jmir.org/2009/2/e13/
    16. 18. What drives / motivates consumers + patients? Healthy Acute Condition Chronic/Severe Condition Motivation
    17. 19. Ease of use Usability User-centered Design Other predictors for attrition / lack of adherence
    18. 20. Essential: Needs assessment, formative evaluation, usability testing (iterative & ongoing) <ul><li>Focus Groups </li></ul><ul><li>Usability lab </li></ul><ul><li>In-depth interviews with stakeholders </li></ul>
    19. 22. Gaps between patient and provider needs / expectations <ul><li>“ Patients are particularly likely to anticipate that shared records will be empowering (...).” </li></ul><ul><li>“ Physicians, by contrast, are especially likely to anticipate that laboratory results will confuse patients and that shared records will make patients worry more. “ </li></ul>Expectations of Patients and Physicians Regarding Patient-Accessible Medical Records Stephen E Ross, MD, Jamie Todd, MS-IV, Laurie A Moore, MPH, Brenda L Beaty, MSPH, Loretta Wittevrongel, Chen-Tan Lin, MD J Med Internet Res 2005 (May 24); 7(2):e13
    20. 23. Gaps between patient and provider needs / expectations Credits: Selina Brudnicki & Claudette DeLenardo
    21. 24. Gaps between patient and provider needs / expectations Credits: Selina Brudnicki & Claudette DeLenardo
    22. 25. Ease of use People will not enter health information to a significant degree…
    23. 26. … (perhaps there are some exceptions)…
    24. 27. … rather, eHealth sites / PHR (or PHA platform) must be populated seamlessly and effortlessly… Web 2.0 (collaborative, data entered by others) Mobile technologies, SMS Domotics, Ambient, pervasive computing, Intelligent car Applications with geospatial awareness Electronic Medical Record (Provider) PHR / PHA Platform Natural speech interfaces Personal Monitoring Tools
    25. 28. Sorbi MJ, Mak SB, Houtveen JH, Kleiboer AM, van Doornen LJP Mobile Web-Based Monitoring and Coaching: Feasibility in Chronic Migraine J Med Internet Res 2007;9(5):e38 <URL: http://www.jmir.org/2007/5/e38/> Health issues do not only occur at home - mobile interfaces are essential
    26. 29. Intelligent spoon
    27. 32. But even ubiquitous computing / sensors is not always appropriate and accepted
    28. 33. “ I wouldn't want to track (a variable or in general) because tracking would… “ <ul><li>Not apply to me: (eg, smoking, alcohol drinking, pets) </li></ul><ul><li>Not provide new information: (ie, “I already know this”) </li></ul><ul><li>Not provide valuable information </li></ul><ul><li>Provide too much information (information overload) </li></ul><ul><li>Threaten self-image (“would feel criticized”) </li></ul><ul><li>Not provide actionable information </li></ul><ul><li>Lead to social conflict </li></ul><ul><li>Promote obsessive or unhealthy reactions: (“becoming obsessed”) </li></ul><ul><li>Force too much structure (“Approaching life too analytically”) </li></ul><ul><li>Not be suitable for particular activity or behavior </li></ul><ul><li>Be too complicated, error-prone, or disruptive </li></ul>Beaudin JS, Intille SS, Morris ME To Track or Not to Track: User Reactions to Concepts in Longitudinal Health Monitoring J Med Internet Res 2006;8(4):e29 <URL: http://www.jmir.org/2006/4/e29/>
    29. 34. Communities / peer-pressure as a tool to enhance adherence
    30. 35. Social Uses of Personal Health Information Within PatientsLikeMe, an Online Patient Community: What Can Happen When Patients Have Access to One Another’s Data Jeana H Frost, Michael P. Massagli J Med Internet Res 2008 (May 27); 10(3):e15
    31. 36. http://en.wikipedia.org/wiki/Image:Web20_en.png
    32. 37. Medicine 2.0 (“next generation medicine”) From: Gunther Eysenbach. Medicine 2.0: Social Networking, Collaboration, Participation, Apomediation, and Openness J Med Internet Res 2008; 10(3):e22 http://dx.doi.org/ 10.2196/jmir.1030 DOI: 10.2196/jmir.1030 Gunther Eysenbach MD, MPH , www.medicine20congress.com Consumer / Patient Health Professionals Biomedical Researchers Science 2.0 Peer-review 2.0 Personal Health Record 2.0 Virtual Communities (peer-to-peer) Professional Communities (peer-to-peer) Health 2.0 HealthVault Google Health HealthBook Sermo WebCite CiteULike Medting WiserWiki eDoctr BioWizard Dissect Medicine E-learning PLoS One BMC JMIR Wikis Blogs RSS RDF, Semantic Web Virtual Worlds Web 2.0 Technologies & Approaches Apomediation Participation Social Networking Collaboration XML AJAX Openess Revolution Health PatientsLikeMe PeerClip Connotea ALIVE HealthMap caBIG Doctorshangout.com Asklepios
    33. 38. Gunther Eysenbach MD, MPH , www.medicine20congress.com September 17-18, 2009 
    34. 39. Venkatesh,Viswanath;Morris,Michael G.;Davis,Gordon B.;Davis,Fred D., “User acceptance of information technology: Toward a unified view”, MIS Quarterly, 2003, 27, 3, 425-478.
    35. 40. Teresa Chiu, PhD Thesis, University of Toronto
    36. 41. Reporting Attrition/Adherence <ul><li>Attrition is the norm, not the exception - don’t try to cover it up </li></ul><ul><li>“ Attrition curves” now a standard for reporting website use over time (at least at the J Med Internet Res ) </li></ul>Image source: Meyer B, Berger T, Caspar F, Beevers CG, Andersson G, Weiss M Effectiveness of a Novel Integrative Online Treatment for Depression (Deprexis): Randomized Controlled Trial J Med Internet Res 2009;11(2):e15 URL: http://www.jmir.org/2009/2/e15/
    37. 42. Dealing with Attrition/Adherence <ul><li>Use appropriate statistical methods for dealing with missing data </li></ul><ul><ul><li>Refrain from LOCF (last observation carried forward) </li></ul></ul><ul><ul><li>Use models instead (ANOVA etc) </li></ul></ul>
    38. 43. Challenge 2: The problem of “controlling” the control group <ul><li>On the Internet, similar interventions may be accessible for the control group (e.g. smoking cessation) </li></ul><ul><li>difficult to “control” what the control group does </li></ul><ul><li>privacy / ethical concerns limit the amount of data which can be gathered (e.g. logging all accessed URLs) </li></ul>Eysenbach G Issues in evaluating health websites in an Internet-based randomized controlled trial J Med Internet Res 2002;4(3):e17 <URL: http://www.jmir.org/2002/3/e17/>
    39. 44. Challenge 3: Re-registration, participant identity <ul><li>re-registration of (anonymous) participants eager to get the intervention </li></ul><ul><li>“ neither requesting optional provision of offline contact details, nor monitoring IP addresses satisfactorily addressed the issue” </li></ul>Murray E, Khadjesari Z, White IR, Kalaitzaki E, Godfrey C, McCambridge J, Thompson SG, Wallace P Methodological Challenges in Online Trials J Med Internet Res 2009;11(1):e9 URL: http://www.jmir.org/2009/2/e9/
    40. 45. Other challenges <ul><li>Complexity of interventions </li></ul><ul><li>Data quality </li></ul><ul><li>Spam (reminder emails mistaken for spam) </li></ul><ul><li>Cybersquatting </li></ul>Murray E, Khadjesari Z, White IR, Kalaitzaki E, Godfrey C, McCambridge J, Thompson SG, Wallace P Methodological Challenges in Online Trials J Med Internet Res 2009;11(1):e9 URL: http://www.jmir.org/2009/2/e9/
    41. 46. Thank you!

    ×