Johns Hopkins University Department of Psychiatry
Research Conference
Baltimore, MD
Drawing upon successes in other fields to
inform opportunities for technology-based
mental health promotion
December 16, 2014
12:00 – 1:00 PM
Michele L. Ybarra MPH PhD
Center for Innovative Public Health Research
* Thank you for your interest in this presentation. Please note that analyses
included herein are preliminary. More recent, finalized analyses may be available
by contacting CiPHR for further information.
Opportunities for the mental health field
Medication reminders
https://itunes.apple.com/us/app/medcoach-medication-reminder/id443065594?mt=8
Opportunities for the mental health field
Symptom monitoring / facilitated communication
http://www.glasbergen.com/doctor-cartoons-cartoons-about-doctors/
Opportunities for the mental health field
Health literacy / disease management skills
http://blog.surroundhealth.net/2013/10/21/how-have-you-been-celebrating-health-literacy-month/
Benefits of using technology
 The wide adoption of technology
provides novel opportunities to go where
people are.
 Technology-based programs overcome
many structural issues of traditional
programs:
◦ Lack of services in the local area,
◦ Transportation,
◦ Competing life demands.
 They are cost-effective.
Text messaging versus Internet
 Messages are “pushed” to the user
rather than “pulled” as is the case for
online programs / “apps”
 Greater ubiquity because more people
have phones that can text than have
smart phones
 Text messaging-based delivery may be
superior because most you carry your
cell phone with you everywhere, making
intervention messages never far from
reach.
SMS Turkey / cebınız bırakın dıyor
SMS Turkey: Development steps
1. Survey of adult smokers to document
the demand for cessation programs
(n=148)
2. Content development
3. 1-arm trial for feasibility and
acceptability (n=75)
4. Pilot RCT to further test feasibility and
acceptability, preliminary efficacy
(n=151)
Despite high prevalence rates, smokers
want to quit
73%
47%
27%
53%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Seriously thinking
about quitting in the
next 30 days
1+ quit attempted in
the past year
Many want to quit smoking
No Yes
55%
57%
45%
43%
0% 20% 40% 60% 80% 100%
Text messaging
Online
Would try a smoking
cessation program if it
were available…
All other Somewhat/extremely likely to
SMS Turkey theoretical model -> content
map
Message type Description
Preparing to quit Describes steps to take in preparing to quit
smoking, including preparing oneself mentally and
physically, and preparing one’s own and others’
environment
Benifits of quitting Describes the health-related, social, and financial
benefits of becoming a nonsmoker
Coping and
coping strategies
Describes and encourages the effectiveness and
use of cognitive and behavioral strategies to avoid
smoking during a craving or impulse to smoke
Discomfort and
difficulties
Discusses discomfort associated with the quitting
process and how the participant may see his or
her discomfort as normal and how to cope with
such discomfort
Encouragement Offers motivation and support to the participant to
continue with quitting
.
.
.
Algorithm for cessation messages
Messag
e type
Pre-quit Quit day
and Day
2
Early
quit
Late
quit
Relapse Encoura
gement
Preparin
g to quit
17 0 0 0 0 0
Benefits
of
quitting
4 2 3 19 3 2
Coping
and
coping
strategie
s
14 4 11 6 7 1
Discomf
ort and
difficultie
s
1 6 3 1 2 0
Example SMS Turkey messages:
Encouragement arm
Timing Message text
Day 1a Most smokers try to quit 6–7 times before they quit for good.
Don’t quit quitting!
Day 1b It’s a great thing that you’ve tried to quit smoking. You
learned some things that you can apply to the next time you
try to quit. What worked? What didn’t?
Day 2a Quitting smoking is the single most important step you can
take to improve your health.
Day 2b Medicines that treat craving can double your success. Try
medicine next time you quit. If you used medicine, try a
different one next time. Ask your doctor.
Day 3a Smokers live an average of 7–12 years less than
nonsmokers. Consider quitting again!
Day 3b Whatever you decide about smoking, believe in yourself.
You CAN quit smoking if you put your mind to it and have a
plan for success.
Testing for feasibility and acceptability: 1-
arm trial
 No one requested that we stop sending text messages,
complained about the text messages, or requested to
be removed from the study. In contrast, when the
program had ended, 15 participants asked to receive
more text messages. Many participants told the
research staff that they kept the text messages on their
phones and referred back to them.
 Participants had favorable reactions to the text
messages.
◦ Most (89%) said the messages were easy to understand and
talked about what they were experiencing and feeling during the
quitting process (78%).
◦ Although 63% reported that there were too many text messages
received per day, respondents said that they read almost all of
their messages (an average of 9.7 on a scale of 1 being never,
and 10 being always.
Testing for feasibility and acceptability: 1-
arm trial
 Sixty-three participants (84%) provided
12-week post Quit day carbon monoxide
data at the study office.
 Based upon intention-to-treat, 13%
(n=10) of participants reported
continuous abstinence since their quit
date at 12-week follow up, confirmed by
carbon monoxide readings.
https://mfinocchiaro.files.wordpress.com/2012/06/cigarette-smash1.jpg
Preliminary efficacy: Pilot RCT
 Participants were randomly assigned to the
intervention (n=76) or control group (n=75)
 Intervention group:
◦ Received 6 weeks’ worth of text messages tailored to
where in the quitting process the person was
◦ On Day 2 and Day 7 post-quit, the RA called the
participant, asked if s/he had a cigarette and assigned
them the subsequent content “path” accordingly
 Control group participants received a brochure with
quitting tips
SMS Turkey pilot RCT cessation rates at 3-
months post quit-day
5%
0% 0%
11%
17%
14%
0%
2%
4%
6%
8%
10%
12%
14%
16%
18%
20%
Among all adults Among light smokers Among female
smokers
Control Intervention
Ybarra M1, Bağci Bosi AT, Korchmaros J, Emri S A text messaging-based smoking cessation program for adult
smokers: randomized controlled trial.. J Med Internet Res. 2012 Dec 27;14(6):e172.
Stop My Smoking (SMS) USA
SMS USA: Development steps
Translating the program for salience for young adult
smokers:
1. Focus groups to understand why young adults
smoke, what their common triggers are, what their
previous experiences are with quitting, and what
issues we need to be aware of when designing a
text messaging-based smoking program (n=35)
2. Young Adult Advisory Board to review and provide
feedback on the content messages (n=10)
3. Beta test the program and protocol (n=12, 28)
4. Pilot RCT for feasibility, acceptability, and
preliminary efficacy (n=164)
SMS USA: Lessons learned
 Young adults 18-24 years old are just as
addicted to cigarettes as older adults
 Young adults *want* to quit – for
themselves, for their families
 They have a lot going on: almost of our
participants had a job – even if they were
enrolled in tertiary education
 Many did not want to use NRT as it was
seen as “more nicotine”
 DO A COMPLETE BETA TEST!!
http://holykaw.alltop.com/smokers-cravings-get-triggered-by-seeing-e-cigarettes
SMS USA example program
messages
Day sent Text message
Day 1
(Pre-Quit)
Congratulations! The hardest part – deciding to quit – is already
behind you. Write down your quit date [insert day] and post it
where you can see it every day.
Day 2
(Pre-Quit)
Write down a list of reasons why you want to quit smoking. Put
the list in a place where you’ll see it every day.
Day 3
(Pre-Quit)
Got stress? Maybe it’s your cigarettes. In between cigarettes,
your body goes through nicotine withdrawal and makes you feel
stressed out and anxious.
Day 15 (Quit
Day)
I bet you’re feeling cranky and annoyed right now. This is all
normal. Just take a deep breath and get through the next 5
minutes.
Day 20 (Early
Quit)
Right now, you’re learning to quit. Just like learning to ride a
bike or drive a car, it takes time. Before you know it, you’ll learn
to be a non-smoker too.
Day 31
(Late Quit)
Encouragement from your friends and family might be starting
to wane about now, but remember that they are probably still
proud of you for quitting.
SMS USA cessation at 3-months post quit-
day
30%
26%
40%
45%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
50%
Among all youth Among youth who are not in a
tertiary setting
Control Intervention
Guy2Guy
Guy2Guy theoretical model -> message
development
Guy2Guy: Development steps
1. Focus groups to understand their
sexual decision making processes
(n=80)
2. Content Advisory team to test
messages and confirm tone,
understandability (n=56; twice)
3. Beta test to confirm protocol (n=18)
4. Pilot RCT to examine preliminary
efficacy (n=301)
Guy2Guy: Lessons Learned
 Facebook is a *great* place to recruit gay,
bisexual, and queer teen guys
 The “text buddy” is very popular. Safety is critical:
◦ Constant monitoring
◦ Geographical distance
 High interest and high engagement with the
program (e.g., texting back to the program with
“musings”)
BullyDown theoretical model -> message
development
Social emotional learning:
◦ Empathy and communication
◦ Attitudes toward bullying and attitudes
supportive of aggression
◦ Managing Anger, hostility, and impulsivity
◦ Problem solving and coping
◦ Perspective-taking, respect for diversity, and
intentions to intervene to help others
BullyDown: Development steps
1. Focus groups to understand youth’s current
exposure to bullying prevention programs in
their schools (n=40)
2. Content Advisory Team to review and
provide feedback on the text messages
(n=10)
3. Beta test to confirm protocol (n=21)
4. Pilot RCT to obtain preliminary estimates of
efficacy (n=?)
http://www.dreamstime.com/royalty-free-stock-photography-stepwise-progress-image13674027
BullyDown: Lessons learned
 Facebook is *not* a good place to recruit
youth for BullyDown, nor are youth likely
good recruitment targets
 Middle school students’ relationship with their
phone is different than older teens:
◦ They don’t have as much control over their
phone ‘ownership’
◦ They may not even know their own phone
number
◦ They don’t bring their phones to school
 Text buddy seems to be acceptable, but they
would prefer it to be with someone they don’t
know
Limitations to technology-based
interventions
 Use technology in a way that fits the
population. Be aware that technology
may not work for all populations.
 There’s sometimes an assumption that:
If you build it, will it come. Technology-
based programs require self-motivation
and interest.
 Can be costly to develop.
Conclusions
 There is support from other fields that
behavior change programs delivered
online and via text messaging can
change behavior
 Evidence-based programs for people
with serious mental illness that utilize
technology should be explored further
 Figuring out how to engage people
with SMI and keep them coming back,
may be key
http://eurout.org/2010/03/24/studying-lesbians-being-ignored-and-excluded-non-targeted-research
Acknowledgements
 SMS Turkey was funded by the Fogerty
International Center (R01 TW007918)
 SMS USA was funded by the National Institute of
Cancer (R21 CA135669)
 Guy2Guy is funded by the National Institute of
Mental Health (R01 MH096660)
 BullyDown was funded by HopeLab
In all cases, the content is solely the responsibility of
Dr. Ybarra and does not necessarily represent the
official views of the Funders.

Drawing upon successes in other fields to inform opportunities for technology-based mental health promotion

  • 1.
    Johns Hopkins UniversityDepartment of Psychiatry Research Conference Baltimore, MD Drawing upon successes in other fields to inform opportunities for technology-based mental health promotion December 16, 2014 12:00 – 1:00 PM Michele L. Ybarra MPH PhD Center for Innovative Public Health Research * Thank you for your interest in this presentation. Please note that analyses included herein are preliminary. More recent, finalized analyses may be available by contacting CiPHR for further information.
  • 2.
    Opportunities for themental health field Medication reminders https://itunes.apple.com/us/app/medcoach-medication-reminder/id443065594?mt=8
  • 3.
    Opportunities for themental health field Symptom monitoring / facilitated communication http://www.glasbergen.com/doctor-cartoons-cartoons-about-doctors/
  • 4.
    Opportunities for themental health field Health literacy / disease management skills http://blog.surroundhealth.net/2013/10/21/how-have-you-been-celebrating-health-literacy-month/
  • 5.
    Benefits of usingtechnology  The wide adoption of technology provides novel opportunities to go where people are.  Technology-based programs overcome many structural issues of traditional programs: ◦ Lack of services in the local area, ◦ Transportation, ◦ Competing life demands.  They are cost-effective.
  • 6.
    Text messaging versusInternet  Messages are “pushed” to the user rather than “pulled” as is the case for online programs / “apps”  Greater ubiquity because more people have phones that can text than have smart phones  Text messaging-based delivery may be superior because most you carry your cell phone with you everywhere, making intervention messages never far from reach.
  • 7.
    SMS Turkey /cebınız bırakın dıyor
  • 8.
    SMS Turkey: Developmentsteps 1. Survey of adult smokers to document the demand for cessation programs (n=148) 2. Content development 3. 1-arm trial for feasibility and acceptability (n=75) 4. Pilot RCT to further test feasibility and acceptability, preliminary efficacy (n=151)
  • 9.
    Despite high prevalencerates, smokers want to quit 73% 47% 27% 53% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Seriously thinking about quitting in the next 30 days 1+ quit attempted in the past year Many want to quit smoking No Yes 55% 57% 45% 43% 0% 20% 40% 60% 80% 100% Text messaging Online Would try a smoking cessation program if it were available… All other Somewhat/extremely likely to
  • 10.
    SMS Turkey theoreticalmodel -> content map Message type Description Preparing to quit Describes steps to take in preparing to quit smoking, including preparing oneself mentally and physically, and preparing one’s own and others’ environment Benifits of quitting Describes the health-related, social, and financial benefits of becoming a nonsmoker Coping and coping strategies Describes and encourages the effectiveness and use of cognitive and behavioral strategies to avoid smoking during a craving or impulse to smoke Discomfort and difficulties Discusses discomfort associated with the quitting process and how the participant may see his or her discomfort as normal and how to cope with such discomfort Encouragement Offers motivation and support to the participant to continue with quitting . . .
  • 11.
    Algorithm for cessationmessages Messag e type Pre-quit Quit day and Day 2 Early quit Late quit Relapse Encoura gement Preparin g to quit 17 0 0 0 0 0 Benefits of quitting 4 2 3 19 3 2 Coping and coping strategie s 14 4 11 6 7 1 Discomf ort and difficultie s 1 6 3 1 2 0
  • 12.
    Example SMS Turkeymessages: Encouragement arm Timing Message text Day 1a Most smokers try to quit 6–7 times before they quit for good. Don’t quit quitting! Day 1b It’s a great thing that you’ve tried to quit smoking. You learned some things that you can apply to the next time you try to quit. What worked? What didn’t? Day 2a Quitting smoking is the single most important step you can take to improve your health. Day 2b Medicines that treat craving can double your success. Try medicine next time you quit. If you used medicine, try a different one next time. Ask your doctor. Day 3a Smokers live an average of 7–12 years less than nonsmokers. Consider quitting again! Day 3b Whatever you decide about smoking, believe in yourself. You CAN quit smoking if you put your mind to it and have a plan for success.
  • 13.
    Testing for feasibilityand acceptability: 1- arm trial  No one requested that we stop sending text messages, complained about the text messages, or requested to be removed from the study. In contrast, when the program had ended, 15 participants asked to receive more text messages. Many participants told the research staff that they kept the text messages on their phones and referred back to them.  Participants had favorable reactions to the text messages. ◦ Most (89%) said the messages were easy to understand and talked about what they were experiencing and feeling during the quitting process (78%). ◦ Although 63% reported that there were too many text messages received per day, respondents said that they read almost all of their messages (an average of 9.7 on a scale of 1 being never, and 10 being always.
  • 14.
    Testing for feasibilityand acceptability: 1- arm trial  Sixty-three participants (84%) provided 12-week post Quit day carbon monoxide data at the study office.  Based upon intention-to-treat, 13% (n=10) of participants reported continuous abstinence since their quit date at 12-week follow up, confirmed by carbon monoxide readings. https://mfinocchiaro.files.wordpress.com/2012/06/cigarette-smash1.jpg
  • 15.
    Preliminary efficacy: PilotRCT  Participants were randomly assigned to the intervention (n=76) or control group (n=75)  Intervention group: ◦ Received 6 weeks’ worth of text messages tailored to where in the quitting process the person was ◦ On Day 2 and Day 7 post-quit, the RA called the participant, asked if s/he had a cigarette and assigned them the subsequent content “path” accordingly  Control group participants received a brochure with quitting tips
  • 16.
    SMS Turkey pilotRCT cessation rates at 3- months post quit-day 5% 0% 0% 11% 17% 14% 0% 2% 4% 6% 8% 10% 12% 14% 16% 18% 20% Among all adults Among light smokers Among female smokers Control Intervention Ybarra M1, Bağci Bosi AT, Korchmaros J, Emri S A text messaging-based smoking cessation program for adult smokers: randomized controlled trial.. J Med Internet Res. 2012 Dec 27;14(6):e172.
  • 17.
    Stop My Smoking(SMS) USA
  • 18.
    SMS USA: Developmentsteps Translating the program for salience for young adult smokers: 1. Focus groups to understand why young adults smoke, what their common triggers are, what their previous experiences are with quitting, and what issues we need to be aware of when designing a text messaging-based smoking program (n=35) 2. Young Adult Advisory Board to review and provide feedback on the content messages (n=10) 3. Beta test the program and protocol (n=12, 28) 4. Pilot RCT for feasibility, acceptability, and preliminary efficacy (n=164)
  • 19.
    SMS USA: Lessonslearned  Young adults 18-24 years old are just as addicted to cigarettes as older adults  Young adults *want* to quit – for themselves, for their families  They have a lot going on: almost of our participants had a job – even if they were enrolled in tertiary education  Many did not want to use NRT as it was seen as “more nicotine”  DO A COMPLETE BETA TEST!! http://holykaw.alltop.com/smokers-cravings-get-triggered-by-seeing-e-cigarettes
  • 20.
    SMS USA exampleprogram messages Day sent Text message Day 1 (Pre-Quit) Congratulations! The hardest part – deciding to quit – is already behind you. Write down your quit date [insert day] and post it where you can see it every day. Day 2 (Pre-Quit) Write down a list of reasons why you want to quit smoking. Put the list in a place where you’ll see it every day. Day 3 (Pre-Quit) Got stress? Maybe it’s your cigarettes. In between cigarettes, your body goes through nicotine withdrawal and makes you feel stressed out and anxious. Day 15 (Quit Day) I bet you’re feeling cranky and annoyed right now. This is all normal. Just take a deep breath and get through the next 5 minutes. Day 20 (Early Quit) Right now, you’re learning to quit. Just like learning to ride a bike or drive a car, it takes time. Before you know it, you’ll learn to be a non-smoker too. Day 31 (Late Quit) Encouragement from your friends and family might be starting to wane about now, but remember that they are probably still proud of you for quitting.
  • 21.
    SMS USA cessationat 3-months post quit- day 30% 26% 40% 45% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% Among all youth Among youth who are not in a tertiary setting Control Intervention
  • 22.
  • 23.
    Guy2Guy theoretical model-> message development
  • 24.
    Guy2Guy: Development steps 1.Focus groups to understand their sexual decision making processes (n=80) 2. Content Advisory team to test messages and confirm tone, understandability (n=56; twice) 3. Beta test to confirm protocol (n=18) 4. Pilot RCT to examine preliminary efficacy (n=301)
  • 25.
    Guy2Guy: Lessons Learned Facebook is a *great* place to recruit gay, bisexual, and queer teen guys  The “text buddy” is very popular. Safety is critical: ◦ Constant monitoring ◦ Geographical distance  High interest and high engagement with the program (e.g., texting back to the program with “musings”)
  • 26.
    BullyDown theoretical model-> message development Social emotional learning: ◦ Empathy and communication ◦ Attitudes toward bullying and attitudes supportive of aggression ◦ Managing Anger, hostility, and impulsivity ◦ Problem solving and coping ◦ Perspective-taking, respect for diversity, and intentions to intervene to help others
  • 27.
    BullyDown: Development steps 1.Focus groups to understand youth’s current exposure to bullying prevention programs in their schools (n=40) 2. Content Advisory Team to review and provide feedback on the text messages (n=10) 3. Beta test to confirm protocol (n=21) 4. Pilot RCT to obtain preliminary estimates of efficacy (n=?) http://www.dreamstime.com/royalty-free-stock-photography-stepwise-progress-image13674027
  • 28.
    BullyDown: Lessons learned Facebook is *not* a good place to recruit youth for BullyDown, nor are youth likely good recruitment targets  Middle school students’ relationship with their phone is different than older teens: ◦ They don’t have as much control over their phone ‘ownership’ ◦ They may not even know their own phone number ◦ They don’t bring their phones to school  Text buddy seems to be acceptable, but they would prefer it to be with someone they don’t know
  • 29.
    Limitations to technology-based interventions Use technology in a way that fits the population. Be aware that technology may not work for all populations.  There’s sometimes an assumption that: If you build it, will it come. Technology- based programs require self-motivation and interest.  Can be costly to develop.
  • 30.
    Conclusions  There issupport from other fields that behavior change programs delivered online and via text messaging can change behavior  Evidence-based programs for people with serious mental illness that utilize technology should be explored further  Figuring out how to engage people with SMI and keep them coming back, may be key http://eurout.org/2010/03/24/studying-lesbians-being-ignored-and-excluded-non-targeted-research
  • 31.
    Acknowledgements  SMS Turkeywas funded by the Fogerty International Center (R01 TW007918)  SMS USA was funded by the National Institute of Cancer (R21 CA135669)  Guy2Guy is funded by the National Institute of Mental Health (R01 MH096660)  BullyDown was funded by HopeLab In all cases, the content is solely the responsibility of Dr. Ybarra and does not necessarily represent the official views of the Funders.

Editor's Notes

  • #6 Indeed, over half (63%) of 12- to 17-year-olds text every day, with adolescent women the most avid texters: They send and receive a median of 100 text messages per day.68 Cell phone ownership is high across racial and ethnic groups (78% White, 75% Black, 68% Hispanic), and income levels (62% of youth from households with incomes of <=$30,000; 76-81% among youth from higher income households).69 This is the primary mode of communication between peers and is preferred over other communication modes.69 : Compared to the high personnel and infrastructure costs of in-person interventions, Girl2Girl is scalable and costs <5 cents per message
  • #7 Indeed, over half (63%) of 12- to 17-year-olds text every day, with adolescent women the most avid texters: They send and receive a median of 100 text messages per day.68 Cell phone ownership is high across racial and ethnic groups (78% White, 75% Black, 68% Hispanic), and income levels (62% of youth from households with incomes of <=$30,000; 76-81% among youth from higher income households).69 This is the primary mode of communication between peers and is preferred over other communication modes.69 : Compared to the high personnel and infrastructure costs of in-person interventions, Girl2Girl is scalable and costs <5 cents per message
  • #8 SMS Turkey was supported by Award Number R01TW007918 from the Fogarty International Center.. SMS USA was supported by Award Number R21CA135669 from the National Cancer Institute at the National Institutes of Health. In both cases, the content is solely the responsibility of the author and does not reflect those of the funder. Stop My Smoking was piloted in both Ankara, Turkey among adult smokers and in the United States among young adult smokers. Completion rates were high, and text messaging appeared to be a useful method of data collection for young adults as well – it resulted in an 89% response rate at 3-months.
  • #10 148 adult smokers took part in a community-based survey in Ankara, Turkey. Two in five (43%) respondents reported typically smoking their first cigarette within 30 minutes of waking. Many participants expressed a desire to quit smoking: 27% reported seriously thinking about quitting in the next 30 days 53% reported at least one quit attempt in the past year. Two in five smokers wanting to quit reported they were somewhat or extremely like to try a smoking cessation program if it were accessible via text messaging (45%) or online (43%).
  • #14 Participants were recruited through mall intercepts, word-of-mouth, and posters at the Hacettepe University Medical School campus. Half (49%) were female by design, and 48% were married. Average of 37.6 years of age (Range: 19 - 62 years; SD: 10.8 years). Median monthly household income was 2000-4000 YTL (approximately US$1250 - $2500). All participants said that quitting was very important to them (Range: 1-10; M: 9.3, SD: 1.3) although their confidence in being able to quit was generally low (Range: 1-10; M: 5.9, SD: 2.6) at baseline. The average participant was addicted to cigarettes based upon the Fagerström score (Range: 0-8; M: 4.3, SD: 2.4).
  • #17 Remember that in the 1 arm trial, 13% were quit; so this seems to be about the % that we can hope to affect
  • #23 Guy2Guy is supported by Award Number R01 MH096660 from the National Institute of Mental Health. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institute of Mental Health. Development steps: Focus groups to understand their sexual decision making processes Content Advisory team to test messages and confirm tone, understandability Beta test to confirm protocol Pilot RCT to examine preliminary efficacy
  • #25 Focus groups to understand youth’s current exposure to bullying prevention programs in their schools
  • #27 Focus groups to understand youth’s current exposure to bullying prevention programs in their schools
  • #31 http://eurout.org/2010/03/24/studying-lesbians-being-ignored-and-excluded-non-targeted-research