Proudfoot, J. et al.: Methodological Issues and Challenges in eHealth Research Panel This slideshow, presented at Medicine 2.0’08 , Sept 4/5 th , 2008, in Toronto, was uploaded on behalf of the presenter by the Medicine 2.0 team Do not miss the next Medicine 2.0 congress on 17/18th Sept 2009 ( www.medicine20congress.com ) Order Audio Recordings (mp3) of Medicine 2.0’08 presentations at http://www.medicine20congress.com/mp3.php

This slideshow, presented at Medicine 2.0’08 , Sept 4/5 th , 2008, in Toronto, was uploaded on behalf of the presenter by the Medicine 2.0 team

Do not miss the next Medicine 2.0 congress on 17/18th Sept 2009 ( www.medicine20congress.com )

Order Audio Recordings (mp3) of Medicine 2.0’08 presentations at http://www.medicine20congress.com/mp3.php

Methodological Issues and Challenges in eHealth Research Dr Judy Proudfoot , Black Dog Institute & School of Psychiatry, University of New South Wales, Sydney, Australia. Dr Lisa Whitehead , University of Otago, Christchurch, New Zealand. Dr Caryl Barnes , School of Psychiatry, University of New South Wales, Sydney, and Sentiens, Perth, Australia

Session Outline Introduction Systematic review of methodological and ethical issues in Internet research Sampling issues Ethical issues Measurement issues Discussion Conclusion

Introduction

Systematic review of methodological and ethical issues in Internet research

Sampling issues

Ethical issues

Measurement issues

Discussion

Conclusion

Systematic review of methodological and ethical issues in Internet research Dr Lisa Whitehead

Search strategy 8 databases (EBSCO,EMBASE, Medline, PsycInfo, CINAHL, PubMed, Cochrane and TRIP). Keywords: Internet, research, quality, credibility, reliability and validity. Quality appraisal (Dixon-Woods et al. 2005). Whitehead, L. (2007) Methodological and ethical issues in Internet-mediated research in the field of health: An integrated review of the literature. Social Science and Medicine 65: 782-791

8 databases (EBSCO,EMBASE, Medline, PsycInfo, CINAHL, PubMed, Cochrane and TRIP).

Keywords: Internet, research, quality, credibility, reliability and validity.

Quality appraisal (Dixon-Woods et al. 2005).

Whitehead, L. (2007) Methodological and ethical issues in Internet-mediated research in the field of health: An integrated review of the literature. Social Science and Medicine 65: 782-791

3 Key Issues Sampling issues Ethical issues Reliability and validity 3 key issues Potential for sampling bias Ethical issues in Internet research? Validity of the data- impact on the established psychometric properties of measures

3 key issues

Potential for sampling bias

Ethical issues in Internet research?

Validity of the data- impact on the established psychometric properties of measures

Sampling Issues Two approaches (open and selective) Open (convenience sampling) Enhance representation? Can large samples override potential bias? Presence and impact of self-selection can be traced to some degree

Two approaches (open and selective)

Open (convenience sampling)

Enhance representation?

Can large samples override potential bias?

Presence and impact of self-selection can be traced to some degree

Sampling issues Selective (if able to define population) Few studies based on a randomly selected sample Convenience sample random allocation Propensity scoring an option

Selective (if able to define population)

Few studies based on a randomly selected sample

Convenience sample random allocation

Propensity scoring an option

Response rates Unsolicited e-mail (with electronic return) generates lowest response rate, even when compared to unsolicited postal survey. Impact of presentation- browser type, download time, password, security issues, computer anxiety, computer self-efficacy

Unsolicited e-mail (with electronic return) generates lowest response rate, even when compared to unsolicited postal survey.

Impact of presentation- browser type, download time, password, security issues, computer anxiety, computer self-efficacy

Ethical Issues Seeking/obtaining consent Protecting participants’ anonymity Security of site Opportunistic research

Seeking/obtaining consent

Protecting participants’ anonymity

Security of site

Opportunistic research

Ethical issues in planned research Developing informed consent Ability to ensure a level of anonymity Security of site

Developing informed consent

Ability to ensure a level of anonymity

Security of site

Opportunistic research When and how is material posted on the web classed as public or private? Level of “disguise” in presenting results? Issue of perceived privacy Are researchers “lurkers”? Is the process of “non-alienation” acceptable?

When and how is material posted on the web classed as public or private?

Level of “disguise” in presenting results?

Issue of perceived privacy

Are researchers “lurkers”?

Is the process of “non-alienation” acceptable?

Reliability and validity Equivalence must be demonstrated not assumed Some evidence of differences in the distribution of scores and psychometric properties (e.g. factor loading)

Equivalence must be demonstrated not assumed

Some evidence of differences in the distribution of scores and psychometric properties (e.g. factor loading)

Reliability and credibility Layout of pages Linguistic competence Culturally driven interpretations Low attentiveness Repeat participation Incomplete submissions Misrepresentation of self

Layout of pages

Linguistic competence

Culturally driven interpretations

Low attentiveness

Repeat participation

Incomplete submissions

Misrepresentation of self

Construct validity Psychometric properties Mixed evidence How critical are the differences? Difficult to disentangle issues around sampling biases and possible effects of completing measures on-line.

Psychometric properties

Mixed evidence

How critical are the differences?

Difficult to disentangle issues around sampling biases and possible effects of completing measures on-line.

Sampling Issues in Internet-Mediated Research Dr Judy Proudfoot

Online Psychoeducation for Bipolar Disorder: Randomised Controlled Trial Dr Judy Proudfoot – Black Dog Institute & School of Psychiatry, University of New South Wales Amisha Jawawant - Black Dog Institute & School of Psychiatry, University of New South Wales Prof. Gordon Parker – Black Dog Institute & School of Psychiatry, University of New South Wales A/Prof Vijaya Manicavasagar – Black Dog Institute & School of Psychiatry, University of New South Wales A/Prof Meg Smith – University of Western Sydney

Dr Judy Proudfoot – Black Dog Institute & School of Psychiatry, University of New South Wales

Amisha Jawawant - Black Dog Institute & School of Psychiatry, University of New South Wales

Prof. Gordon Parker – Black Dog Institute & School of Psychiatry, University of New South Wales

A/Prof Vijaya Manicavasagar – Black Dog Institute & School of Psychiatry, University of New South Wales

A/Prof Meg Smith – University of Western Sydney

Background Bipolar Disorder is difficult to detect. Accurate diagnosis can take 10-20 years. Poor treatment adherence, inadequate self management, increased levels of disability are common. 40% of patients relapse in 1 year, 60% over 2 years and 73% over 5 years (Gitlin et al, Am J Psych, 1995) . Psycho-education reduces relapses and hospitalisations. Not previously delivered online.

Bipolar Disorder is difficult to detect. Accurate diagnosis can take 10-20 years.

Poor treatment adherence, inadequate self management, increased levels of disability are common.

40% of patients relapse in 1 year, 60% over 2 years and 73% over 5 years (Gitlin et al, Am J Psych, 1995) .

Psycho-education reduces relapses and hospitalisations. Not previously delivered online.

Research Study RANDOMISED CONTROLLED TRIAL Does an 8-module online education program + ‘treatment as usual’ for Bipolar Disorder improve psycho-social adjustment in adults with newly-diagnosed BD With and without online support from an apomediary (“informed supporter”). In comparison with online information about bipolar disorder + monitoring.

RANDOMISED CONTROLLED TRIAL

Does an 8-module online education program + ‘treatment as usual’ for Bipolar Disorder improve psycho-social adjustment in adults with newly-diagnosed BD

With and without online support from an apomediary (“informed supporter”).

In comparison with online information about bipolar disorder + monitoring.

Bipolar Disorder Education Program

Topics Diagnosing Bipolar Disorder Causes of Bipolar disorder Medications Psychological Treatments Stay-Well Plans (identifying triggers and early warning signs, and designing preemptive actions involving pt and family) Support Networks and Carers Omega-3 as a treatment for depression Person First Illness Last

Diagnosing Bipolar Disorder

Causes of Bipolar disorder

Medications

Psychological Treatments

Stay-Well Plans (identifying triggers and early warning signs, and designing preemptive actions involving pt and family)

Support Networks and Carers

Omega-3 as a treatment for depression

Person First Illness Last

Random Allocation Bipolar Disorder Education Program Email support from an apomediary + lnformation + Monitoring

Online interventions One online module per week over 8 weeks Workbook activities to assist participants to apply the information and strategies to their situation Self-monitoring between sessions

One online module per week over 8 weeks

Workbook activities to assist participants to apply the information and strategies to their situation

Self-monitoring between sessions

“Informed Supporters” People with bipolar disorder Effectively managing their condition for >2 years Trained to provide email support to participants under supervision of our clinical team.

People with bipolar disorder

Effectively managing their condition for >2 years

Trained to provide email support to participants under supervision of our clinical team.

Informed Support At least one and maximum two emails per week sent to each participant (max 300 words per email), with invitation to respond. Practical advice about how to set up a “well-being plan” (strategies to minimise risks and illness triggers and a contingency plan for times when unwell) as well as general issues about bipolar disorder. Posts from Informed Supporters are checked for adherence and safety. Posts from participants to Informed Supporters are monitored for adverse events.

At least one and maximum two emails per week sent to each participant (max 300 words per email), with invitation to respond.

Practical advice about how to set up a “well-being plan” (strategies to minimise risks and illness triggers and a contingency plan for times when unwell) as well as general issues about bipolar disorder.

Posts from Informed Supporters are checked for adherence and safety.

Posts from participants to Informed Supporters are monitored for adverse events.

Inclusion Criteria: 18+ years old Diagnosed with BD in previous 12 months Currently receiving regular care from psychiatrist or GP Has access to internet and printer, and is computer literate Able to read and write English Living in Australia

18+ years old

Diagnosed with BD in previous 12 months

Currently receiving regular care from psychiatrist or GP

Has access to internet and printer, and is computer literate

Able to read and write English

Living in Australia

Screening Bipolar Disorder Assessment Test (Parker et al., 2006) Sensitivity: Specificity: Potential for over-ride by Black Dog Institute Psychiatrists (who validated the BDAT)

Bipolar Disorder Assessment Test (Parker et al., 2006) Sensitivity: Specificity:

Potential for over-ride by Black Dog Institute Psychiatrists (who validated the BDAT)

Sampling Open (convenience) sampling: - Via Internet Black Dog Institute website, websites of professional organisations, websites of mental health organisations in Australia - Via print media Newspaper advertisements and articles

Open (convenience) sampling: - Via Internet Black Dog Institute website, websites of professional organisations, websites of mental health organisations in Australia - Via print media Newspaper advertisements and articles

Sampling (cont) Selective Sampling: - Via Black Dog Institute Mood Disorders Clinic – invited participation - Via leaflets at special events , eg Fun Runs, Flower Festivals, Training programs for GPs, School Counsellors

Selective Sampling:

- Via Black Dog Institute Mood Disorders Clinic – invited participation - Via leaflets at special events , eg Fun Runs, Flower Festivals, Training programs for GPs, School Counsellors

Sampling (cont) Variety of recruitment methods to minimise sampling bias and maximise response rates. But still not a random sample (ie each member of the population does not have equal probability of being selected). No recruitment by unsolicited email (lower response rate). But still low response rate (3-4 per week) - due to tight inclusion criteria? - due to phenomenology of BD diagnosis?

Variety of recruitment methods to minimise sampling bias and maximise response rates.

But still not a random sample (ie each member of the population does not have equal probability of being selected).

No recruitment by unsolicited email (lower response rate).

But still low response rate (3-4 per week) - due to tight inclusion criteria?

- due to phenomenology of BD diagnosis?

A Solution?

Methodological Issues and Ethical Challenges in e-Health Research: Panel Discussion Dr Caryl Barnes MBBS,FRANZCP Doctoral fellow University of New South Wales, Sydney, Australia ‘ HealthSteps (formally Recovery Road) for Bipolar Disorder’: an on-line relapse prevention program

Overview Bipolar disorder: role of internet in treatment Aims of RCT: ‘ Taking control of Bipolar Disorder: Staying well, Learning more’-12 month online relapse prevention program Ethical issues considered Privacy/Confidentiality/Anonymity Informed consent Analysis of potential risk to participants/ benefits Beneficence/ Duty-of-care Discussion

Bipolar disorder: role of internet in treatment

Aims of RCT: ‘ Taking control of Bipolar Disorder: Staying well, Learning more’-12 month online relapse prevention program

Ethical issues considered

Privacy/Confidentiality/Anonymity

Informed consent

Analysis of potential risk to participants/ benefits

Beneficence/ Duty-of-care

Discussion

Bipolar Disorder: role of internet in treatment Common 1-2% of the general population Chronic relapsing remitting illness 6th highest cause of disability world wide (WHO figures) Frequent relapses despite good adherence to medication Psychological interventions vital part of treatment e.g. psycho-education / CBT Disadvantages time consuming, expensive depending on area, difficult to access

Common 1-2% of the general population

Chronic relapsing remitting illness

6th highest cause of disability world wide (WHO figures)

Frequent relapses despite good adherence to medication

Psychological interventions vital part of treatment e.g. psycho-education / CBT

Disadvantages

time consuming,

expensive

depending on area, difficult to access

Research Team: collaboration between public/private University of New South Wales Dr Caryl Barnes-Chief Investigator, Consultant Psychiatrist/ Doctoral fellow Dusan Hadiz-Pavlovic, Senior Hospital Scientist Professor Philip Mitchell, Head of School of Psychiatry Assoc Professor Kay Wilhelm Sentiens Pty Ltd Dr Dennis Tannenbaum, CEO Dr Caroline Spiranovic, Research and Development Associate Gavin Pinto, IT Architect Michael Bosley-Smith, Research Officer Stacey Bosley-Smith, Research Officer Dr Robin Harvey, Research Manager Perth Western Australia Sydney, NSW

University of New South Wales

Dr Caryl Barnes-Chief Investigator, Consultant Psychiatrist/ Doctoral fellow

Dusan Hadiz-Pavlovic, Senior Hospital Scientist

Professor Philip Mitchell, Head of School of Psychiatry

Assoc Professor Kay Wilhelm

Sentiens Pty Ltd

Dr Dennis Tannenbaum, CEO

Dr Caroline Spiranovic, Research and Development Associate

Gavin Pinto, IT Architect

Michael Bosley-Smith, Research Officer

Stacey Bosley-Smith, Research Officer

Dr Robin Harvey, Research Manager

Aims of 12-month RCT To evaluate on-line adjunctive relapse prevention program ‘HealthSteps for Bipolar Disorder’ vs. placebo/control program ‘Virtual Highway for Bipolar Disorder’ Outcome measures: Time to relapse- Survival analysis Measures: BDI-II, ISS, Sheehan disability scale, Hospitalization,

To evaluate on-line adjunctive relapse prevention program ‘HealthSteps for Bipolar Disorder’ vs. placebo/control program ‘Virtual Highway for Bipolar Disorder’

Outcome measures:

Time to relapse- Survival analysis

Measures: BDI-II, ISS, Sheehan disability scale, Hospitalization,

Study Inclusion Criteria Over 18 years of age Meets criteria for bipolar disorder (BDSQ Confirmed) Diagnosis made by a mental health professional Currently under care of psychiatrist /GP Taking medication Access to internet and printer- at home/ work. Has an email account/ has email address. Able to understand written English Interested in using web based program

Over 18 years of age

Meets criteria for bipolar disorder (BDSQ Confirmed)

Diagnosis made by a mental health professional

Currently under care of psychiatrist /GP

Taking medication

Access to internet and printer- at home/ work.

Has an email account/ has email address.

Able to understand written English

Interested in using web based program

Study Inclusion Criteria Over 18 years of age Meets criteria for bipolar disorder (BDSQ Confirmed) Diagnosis made by a mental health professional Currently under care of psychiatrist /GP Taking medication Access to internet and printer- at home/ work. Has an email account/ has email address. Able to understand written English Interested in using web based program

Over 18 years of age

Meets criteria for bipolar disorder (BDSQ Confirmed)

Diagnosis made by a mental health professional

Currently under care of psychiatrist /GP

Taking medication

Access to internet and printer- at home/ work.

Has an email account/ has email address.

Able to understand written English

Interested in using web based program

Ethical Considerations: Getting started UNSW Human Research Ethics Committee (HREC) Formal application process Consist of a diverse panel includes ’lay’ members Issues Privacy/Confidentiality/Anonymity Informed consent Analysis of potential risk to participants/ benefits Beneficence/ Duty-of-care

UNSW Human Research Ethics Committee (HREC)

Formal application process

Consist of a diverse panel includes ’lay’ members

Issues

Privacy/Confidentiality/Anonymity

Informed consent

Analysis of potential risk to participants/ benefits

Beneficence/ Duty-of-care

Privacy/ Confidentiality Screening/Recruitment-BDSQ Only contact email required Minimal identifying details collected Contact participants via email only Secure database/server Professional standards for confidentiality Online Enrolment/Access to study Allocation 8 digit individual ID number/password Data entered during study De-identification Secure data storage -fire walled Limited access

Screening/Recruitment-BDSQ

Only contact email required

Minimal identifying details collected

Contact participants via email only

Secure database/server

Professional standards for confidentiality

Online Enrolment/Access to study

Allocation 8 digit individual ID number/password

Data entered during study

De-identification

Secure data storage -fire walled

Limited access

Privacy / Confidentiality

Informed Consent Consent for Study: Two-way process Provide information about requirements, risks and benefits of the study Ensure level of common understanding about study is reached Information via online ‘Participation Information statement’ start of BDSQ Used ‘click-button’ option. Consent to inform treating clinician optional Participant able to withdraw consent at any point Technical support given via s [email_address] .au

Consent for Study: Two-way process

Provide information about requirements, risks and benefits of the study

Ensure level of common understanding about study is reached

Information via online ‘Participation Information statement’ start of BDSQ

Used ‘click-button’ option.

Consent to inform treating clinician optional

Participant able to withdraw consent at any point

Technical support given via s [email_address] .au

Screen shots in here

Screen shots in here

Risks/Benefit Ratio Low risk of inducing psychological distress No risk of physical harm Vulnerable population recognized; inclusion criteria had to be under care treating mental health professional Program adjunctive only Belief participation would benefit participants

Low risk of inducing psychological distress

No risk of physical harm

Vulnerable population recognized; inclusion criteria had to be under care treating mental health professional

Program adjunctive only

Belief participation would benefit participants

Duty of Care Need to recognize worsening/ life threatening symptoms/ suicidal ideation Developed a RED FLAG system Outcome measures set to trigger flag within system Individual data entry reviewed, if required de-identification participant, review of consents Email to participant, if applicable to treating clinician

Need to recognize worsening/ life threatening symptoms/ suicidal ideation

Developed a RED FLAG system

Outcome measures set to trigger flag within system

Individual data entry reviewed, if required de-identification participant, review of consents

Email to participant, if applicable to treating clinician

Discussion Restraints caused by a research collaboration-private and public entities Privacy- not enough identifying details collected Informed consent process – was this sufficient? Risk/Benefit email Ongoing clinician contact?

Restraints caused by a research collaboration-private and public entities

Privacy- not enough identifying details collected

Informed consent process – was this sufficient?

Risk/Benefit

email

Ongoing clinician contact?

Discussion Duty of care Management of participants outside of Australia Red flag system appreciated-feedback received rated this most useful components of study Consent to contact clinician optional, is this sufficient? Control group going into HealthSteps lack of resources limited retention some features e.g. Red Flags

Duty of care

Management of participants outside of Australia

Red flag system appreciated-feedback received rated this most useful components of study

Consent to contact clinician optional, is this sufficient?

Control group going into HealthSteps lack of resources limited retention some features e.g. Red Flags

Acknowledgements Graduate Research School and Faculty of Medicine, University of New South Wales, Sydney, Australia Travel Scholarship and Doctoral Scholarship Dr Dennis Tannebaum, CEO and all Research Staff at Sentiens Pty Ltd Professor Philip Mitchell : Doctorate Supervisor For a sustainable future in research The 5.31 tonnes of Co2 produced by the author’s flights from Australia to Pittsburgh to present this poster for the 7 th ICBD have been offset with www.climatecare.org.

Graduate Research School and Faculty of Medicine, University of New South Wales, Sydney, Australia

Travel Scholarship and Doctoral Scholarship

Dr Dennis Tannebaum, CEO and all Research Staff at Sentiens Pty Ltd

Professor Philip Mitchell : Doctorate Supervisor

Carbon Neutral: for a sustainable future in research The 6.09 tonnes of Co2 produced by the presenter’s flights from Perth, Western Australia to Toronto for the Medicine 2.0 Conference has been offset with www.climatecare.org.

The health of tertiary students: A mixed method study Dr Lisa Whitehead, Dr Shelagh Dawson, Avin Panckhurst Centre for Postgraduate Nursing Studies, University of Otago, Christchurch, New Zealand

Methods Student population of 20,665 A random sample of 2000 students was taken and these were randomly assigned to either a postal (N=1000) or an on-line survey (N=1000). The survey was about physical and emotional health, anxiety, depression and fatigue

Student population of 20,665

A random sample of 2000 students was taken and these were randomly assigned to either a postal (N=1000) or an on-line survey (N=1000).

The survey was about physical and emotional health, anxiety, depression and fatigue

Sampling Issues Response rates

Response rates

Pattern of response

Sampling issues

Correlations between measures

Distribution of scores- Single fatigue item Table 4. Single item fatigue measure (I feel tired for no reason) X 2 =0.07 df=1 p=0.79 (Armitage-Cochran test for trend) 969 39 90 424 416 Total 497 19 (3.8%) 52 (10.4%) 212 (42.7%) 214 (43.1%) Postal 472 20 (4.2%) 38 (8.1%) 212 (44.9%) 202 (42.8%) On-line Total 3 (Most of the time) 2 (A good part of the time) 1 (Some of the time) 0 (none or a little of the time)

Mean differences

Construct validity FSI- Mean differences in fatigue between the on-line and postal group

Mean differences on FSI items on-line, postal and scale development group

Correlations among the fatigue intensity and duration ratings as an indication of the construct validity of the FSI

Reach Out Central Evaluation Swinburne eTherapy Unit www.swinburne.edu.au/lss/swinpsyche/etherapy

www.reachout.com.au

Online game-based youth mental health education program Target audience: young people aged 16-25, particularly young men. Objectives: Increase mental health literacy Increase skills to change thoughts and solve problems Application of learned skills to improve mental health ROC Overview

Online game-based youth mental health education program

Target audience:

young people aged 16-25, particularly young men.

Objectives:

Increase mental health literacy

Increase skills to change thoughts and solve problems

Application of learned skills to improve mental health

Methodology Single group, quasi-experimental repeated measures design (pre-, post-program and follow-up) N = 266 (176 females; 88 males) Questionnaires included: K10 (depression and anxiety) AUDIT (alcohol use) Coping Strategy Indicator – short form (subscales – problem solving , seeking support, avoidance) Resilience Scale – short form (RS – SF) Satisfaction with Life Scale (SWLS) Additional items examined drug use, mental health stigma, help seeking attitudes, program satisfaction and application of skills in day-to-day life.

Single group, quasi-experimental repeated measures design (pre-, post-program and follow-up)

N = 266 (176 females; 88 males)

Questionnaires included:

K10 (depression and anxiety)

AUDIT (alcohol use)

Coping Strategy Indicator – short form (subscales – problem solving , seeking support, avoidance)

Resilience Scale – short form (RS – SF)

Satisfaction with Life Scale (SWLS)

Additional items examined drug use, mental health stigma, help seeking attitudes, program satisfaction and application of skills in day-to-day life.

Issues for Discussion What is the minimal process considered sufficient for informed consent? Are we overconcerned with privacy? What are the minimal requirements for duty of care? What is your experience in the transfer of measures to the online setting? Should standard measures be robust enough to cope with minor variance? How generalisable are the results from Internet studies (convenience samples)? How can the generalisability be improved? Can inferences be drawn from randomised controlled trials within convenience samples?

What is the minimal process considered sufficient for informed consent?

Are we overconcerned with privacy?

What are the minimal requirements for duty of care?

What is your experience in the transfer of measures to the online setting?

Should standard measures be robust enough to cope with minor variance?

How generalisable are the results from Internet studies (convenience samples)?

How can the generalisability be improved?

Can inferences be drawn from randomised controlled trials within convenience samples?