In 2013, the New York City Department of Health and Mental Hygiene launched a mobile app to help NYC youth identify and access sexual and reproductive health information and services such as contraception, pregnancy testing, and STI screening. Two years later, we have learned significant lessons about the strengths and limitations of mobile app monitoring and evaluation activities, as well as applications of mHealth user data for health programs. This presentation pays special attention to describing the information programs can gather via an mHealth tool, the strengths and limitations of this data, and how to leverage findings into actionable changes to mHealth tools and health programming.
Call Girl Coimbatore Prisha☎️ 8250192130 Independent Escort Service Coimbatore
Process evaluation of a youth-focused mHealth tool
1. Process Evaluation of a
Youth-Focused mHealth Tool
NYC Department of Health and Mental Hygiene
Bureau of Maternal, Infant and Reproductive Health
Sexual & Reproductive Health Unit
Sarah P. Roberts, MPH (Presenter); Allyna Steinberg, MPH;
Marybec Griffin-Tomas, MA, MPH; Alzen Whitten, MPA
2. SRH: Sexual and Reproductive Health
DOHMH: NYC Department of Health and Mental Hygiene
BMIRH: Bureau of Maternal, Infant and Reproductive Health
SRHU: Sexual & Reproductive Health Unit
Alphabet Soup
2
3. Brief overview of the Teens in NYC mobile app
Evaluation goals
Strategies and results
Evaluation successes
Challenges faced
Conclusions + future directions
Agenda
3
5. Launched in 2013
Free on Android and Apple devices
Users can locate teen-friendly SRH
services, find information on birth
control, and learn about what to
expect at a clinic
Part of larger teen-focused SRH
program that includes print resource
guides and referrals via call-center
Teens in NYC Mobile App
5
8. App Process Evaluation
What is the reach of the mobile app?
What is the uptake of the mobile app and its
components (e.g. searches for services)?
To what extent does the program influence
teens age 13-19 to access sexual and
reproductive health services?
8
9. Subway campaign survey
App user survey
Downloads and installs
In-app searches
Evaluation Strategies and Results
9
11. Subway Campaign Online Survey
Conducted following two-month NYC online & subway
ad campaign that ran April 28 –June 30, 2014
~35 questions with a representative sample of NYC
adolescents (age 15-24; N=406)
Questions included:
Have you seen the ad campaign?
What do you think about the ads?
As a result of the ads, did you…download the app? Visit
a clinic? Use dual protection?
How likely are you to do these things in the future?
11
12. Subway Campaign Survey Results
Reach of ads is greatest among:
Black/African-American and Hispanic/Latino respondents
High school and college students
Frequent users of subway and bus
Sexually active respondents
Respondents who are in high school or college (vs. those not in
school) are more likely to say they “learned something new” and
“intend to download app”
Although print ads have greater reach, online ads seem to prompt
greater behavioral response, e.g. downloading app
12
14. User Survey Goals
Inspired by The Hookup SMS outreach service evaluation
(detailed by Rebecca Braun, Jenna Gaarde, and Bhupendra
Sheoran in their 2012 Sex::Tech presentation)
Gain understanding of…
Who is using the app
Where users heard about the app
What features users find helpful
If users have gone or intend to go to a clinic
14
15. User Survey Questions
1. First, a little about you. (Age, Gender, Borough)
2. Did you go to a clinic after finding it in this app?
3a. What information in the Teens in NYC app was helpful to you?
3b. Tell us in your own words: How has the Teens in NYC app been
helpful to you?
4. How did you find out about the Teens in NYC app?
5. [If Age=20 and over] How have you used the Teens in NYC app?
[Otherwise] Help us know if Teens in NYC is useful for teens, whether
they’ve had sex or not. Choose the statement that applies to you…
6. One more thing! How would you make the Teens in NYC app better?
15
16. User Survey Results: Quantitative
Response Target:
500 users who have ever installed or upgraded to Version 2.0
Total Surveys Completed (i.e. all items answered): 289
As of April 20, 2015
58% of our target number of surveys
4% of total Android installs, since launch
1% of total estimated installs (iOS+Android users)
*As of 3/21/2015; install proportions are estimated based on download proportions by device type
16
17. User Survey Results: Health Behavior
17
17%
47%
20%
16%
Clinic Visit Behavior and Intentions
(N = 349*)
Already visited a clinic from the app
Plans to visit a clinic from the app in
the next year
Does not plan to visit a clinic for sexual
health services in the next year
Already has a clinic that they go to for
sexual health servicesData collected
between 6/9/2014
and 4/14/2015 *N = no. respondents that completed survey item #2
18. User Survey Results: Qualitative Responses
How the app has been helpful:
“Teens in NYC app helped me by letting me know that I
don't have to be afraid or embarrassed about getting
tested.” (F, 15-17, Bronx)
“It helped answer some of the questions i [wouldn't] feel
comfortable asking my family.” (F, 15-17, Manhattan)
“It helped me find where to go to get emergency
contraception” (F, 15-17, Queens)
Suggestions for improvement:
“Have a survey that people could take and at the end it
tells you the best method of birth control for you” (F, 15-
17, Queens)
“Make it more interactive.” (F, 20+, Brooklyn) 18
20. Annual Downloads & Total Installs
Total Annual Downloads, U.S.
Current installs, Android: 926
Est. current installs, iOS: 1,509
Total Android installs, since
launch: 8,236
Installs
2013: 10,029
2014: 5,322
2015: 4,011*
2016, as of March 20: 100*
Total: 19,462
*iOS download data not available after Sep 27, 2015
20
22. Total Annual Searches
2013: 11,164
2014: 8,817
2015: 7,573
2016, as of March 20: 949
22
23. Teens in NYC Mobile App Users Search For…
2/25/13 to 6/29/15
Birth control or
Gyn 29%
STD testing and
treatment
21%
Pregnancy
testing
18%
Abortion
17%
HIV testing
9%
Counseling
4%
LGBTQ-specific
services
2%
23Data collected
between 2/27/2013
and 3/20/2016
24. We Are Reaching Teens Proportionately
*2010 US Census Data
Brooklyn
33%
Queens
26%
Bronx
21%
Manhattan
14%
Staten Island
6%
Total Adolescents 12-19*
(n = 818,671)
2/25/13 to 6/29/15
Brooklyn
27%
Queens
29%
Bronx
20%
Manhattan
21%
Staten Island
3%
Current Location Searches by Borough
24Data collected
between 2/27/2013
and 3/20/2016
26. Process Evaluation, Positive Results:
Qualitative user survey feedback indicate the app is helpful
Data indicate 20,000+ downloads and searches from 2013-2016
We are reaching teens where they’re at: users’ current location data searches
mirrors the distribution of adolescents across boroughs
Teens are using search function as we had hoped, i.e. to identify both services
(e.g. ‘birth control’) and methods (e.g. ‘the pill’)
The app may help teens discover less popular methods of birth control:
While only 17% of NYC adolescents report using methods besides condoms or the pill, 52% of
single-method searches in the app are for other forms of contraception (data not shown)
26
28. Evaluation Pain Points: Big Picture
Population-specific concerns (e.g. privacy)
Outcome-specific concerns (e.g. measuring behavior change)
Resources (i.e. time, money, technical support)
No analytics platform integrated into app
28
29. Evaluation Data Sources
Domain Type of Data Source
Reach
Downloads App Annie
Current installs (Android) Google Console
User characteristics User survey
Marketing
Referral source User survey
Subway campaign recall Subway campaign online survey
Social media impressions and clicks HN/OPAD Media reports
User Interactions
Clinic searches DIIT Analytics
Video shares Facebook Insights
Acceptability
Usability Usability testing sessions
User preferences User survey
Ratings & reviews Apple Store, Google Play Store
Behavior Change Clinic visits & visit intentions User survey, subway campaign survey
29
30. Evaluation Pain Points: On the Ground
Diversity of strategies—potential strength became a liability
Fragmentation of data sources (e.g. iOS vs. Android metrics)
Glitches, bugs, unreliable data, and ‘workarounds’
Over-reliance on download and search data
How do we implement an analytics solution while also protecting
users’ privacy?
30
31. Delayed timeline and
limited IT resources
Mobile app bugs
Survey button and
notifications
Survey crashes
Other app issues
Setting and achieving
response target
User Survey Issues
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
% Downloads % Survey Completes
Percentage of App Downloads and
User Survey Completes, by Platform
iOS
Android
31
33. Lessons Learned
Future campaigns should employ both social science theory and
principles of user experience design
A sustainable evaluation plan must be developed well before the
launch of promotional activities
Adequate and sustainable technology resources must be in place
during development, launch and maintenance of app
Analytics, analytics, analytics!
33
34. Applications for Future Work
Formative research and/or literature reviews needed on user
engagement and retention strategies for youth-focused mHealth tools
Important to consider data privacy and security and address concerns
related to target audience e.g. minors, sensitive content
This experience has influenced the development of an adult-focused
SRH app, launching soon, which has built analytics into the product
Most of all, this evaluation process has taught us to….
34