In 2013, the New York City Department of Health and Mental Hygiene launched a mobile app to help NYC youth identify and access sexual and reproductive health information and services such as contraception, pregnancy testing, and STI screening. Two years later, we have learned significant lessons about the strengths and limitations of mobile app monitoring and evaluation activities, as well as applications of mHealth user data for health programs. This presentation pays special attention to describing the information programs can gather via an mHealth tool, the strengths and limitations of this data, and how to leverage findings into actionable changes to mHealth tools and health programming.

1. Process Evaluation of a
Youth-Focused mHealth Tool
NYC Department of Health and Mental Hygiene
Bureau of Maternal, Infant and Reproductive Health
Sexual & Reproductive Health Unit
Sarah P. Roberts, MPH (Presenter); Allyna Steinberg, MPH;
Marybec Griffin-Tomas, MA, MPH; Alzen Whitten, MPA

2.  SRH: Sexual and Reproductive Health
 DOHMH: NYC Department of Health and Mental Hygiene
 BMIRH: Bureau of Maternal, Infant and Reproductive Health
 SRHU: Sexual & Reproductive Health Unit
Alphabet Soup
2

3.  Brief overview of the Teens in NYC mobile app
 Evaluation goals
 Strategies and results
 Evaluation successes
 Challenges faced
 Conclusions + future directions
Agenda
3

4. TEENS IN NYC APP
4

5.  Launched in 2013
 Free on Android and Apple devices
 Users can locate teen-friendly SRH
services, find information on birth
control, and learn about what to
expect at a clinic
 Part of larger teen-focused SRH
program that includes print resource
guides and referrals via call-center
Teens in NYC Mobile App
5

6. Screenshots of App
6

7. EVALUATION GOALS
7

8. App Process Evaluation
 What is the reach of the mobile app?
 What is the uptake of the mobile app and its
components (e.g. searches for services)?
 To what extent does the program influence
teens age 13-19 to access sexual and
reproductive health services?
8

9.  Subway campaign survey
 App user survey
 Downloads and installs
 In-app searches
Evaluation Strategies and Results
9

10. SUBWAY CAMPAIGN
10

11. Subway Campaign Online Survey
 Conducted following two-month NYC online & subway
ad campaign that ran April 28 –June 30, 2014
 ~35 questions with a representative sample of NYC
adolescents (age 15-24; N=406)
 Questions included:
 Have you seen the ad campaign?
 What do you think about the ads?
 As a result of the ads, did you…download the app? Visit
a clinic? Use dual protection?
 How likely are you to do these things in the future?
11

12. Subway Campaign Survey Results
 Reach of ads is greatest among:
 Black/African-American and Hispanic/Latino respondents
 High school and college students
 Frequent users of subway and bus
 Sexually active respondents
 Respondents who are in high school or college (vs. those not in
school) are more likely to say they “learned something new” and
“intend to download app”
 Although print ads have greater reach, online ads seem to prompt
greater behavioral response, e.g. downloading app
12

13. APP USER SURVEY
13

14. User Survey Goals
 Inspired by The Hookup SMS outreach service evaluation
(detailed by Rebecca Braun, Jenna Gaarde, and Bhupendra
Sheoran in their 2012 Sex::Tech presentation)
 Gain understanding of…
 Who is using the app
 Where users heard about the app
 What features users find helpful
 If users have gone or intend to go to a clinic
14

15. User Survey Questions
1. First, a little about you. (Age, Gender, Borough)
2. Did you go to a clinic after finding it in this app?
3a. What information in the Teens in NYC app was helpful to you?
3b. Tell us in your own words: How has the Teens in NYC app been
helpful to you?
4. How did you find out about the Teens in NYC app?
5. [If Age=20 and over] How have you used the Teens in NYC app?
[Otherwise] Help us know if Teens in NYC is useful for teens, whether
they’ve had sex or not. Choose the statement that applies to you…
6. One more thing! How would you make the Teens in NYC app better?
15

16. User Survey Results: Quantitative
 Response Target:
 500 users who have ever installed or upgraded to Version 2.0
 Total Surveys Completed (i.e. all items answered): 289
 As of April 20, 2015
 58% of our target number of surveys
 4% of total Android installs, since launch
 1% of total estimated installs (iOS+Android users)
*As of 3/21/2015; install proportions are estimated based on download proportions by device type
16

17. User Survey Results: Health Behavior
17
17%
47%
20%
16%
Clinic Visit Behavior and Intentions
(N = 349*)
Already visited a clinic from the app
Plans to visit a clinic from the app in
the next year
Does not plan to visit a clinic for sexual
health services in the next year
Already has a clinic that they go to for
sexual health servicesData collected
between 6/9/2014
and 4/14/2015 *N = no. respondents that completed survey item #2

18. User Survey Results: Qualitative Responses
 How the app has been helpful:
 “Teens in NYC app helped me by letting me know that I
don't have to be afraid or embarrassed about getting
tested.” (F, 15-17, Bronx)
 “It helped answer some of the questions i [wouldn't] feel
comfortable asking my family.” (F, 15-17, Manhattan)
 “It helped me find where to go to get emergency
contraception” (F, 15-17, Queens)
 Suggestions for improvement:
 “Have a survey that people could take and at the end it
tells you the best method of birth control for you” (F, 15-
17, Queens)
 “Make it more interactive.” (F, 20+, Brooklyn) 18

19. DOWNLOADS & INSTALLS
19

20. Annual Downloads & Total Installs
Total Annual Downloads, U.S.
 Current installs, Android: 926
 Est. current installs, iOS: 1,509
 Total Android installs, since
launch: 8,236
Installs
 2013: 10,029
 2014: 5,322
 2015: 4,011*
 2016, as of March 20: 100*
 Total: 19,462
*iOS download data not available after Sep 27, 2015
20

21. IN-APP SEARCHES
21

22. Total Annual Searches
 2013: 11,164
 2014: 8,817
 2015: 7,573
 2016, as of March 20: 949
22

23. Teens in NYC Mobile App Users Search For…
2/25/13 to 6/29/15
Birth control or
Gyn 29%
STD testing and
treatment
21%
Pregnancy
testing
18%
Abortion
17%
HIV testing
9%
Counseling
4%
LGBTQ-specific
services
2%
23Data collected
between 2/27/2013
and 3/20/2016

24. We Are Reaching Teens Proportionately
*2010 US Census Data
Brooklyn
33%
Queens
26%
Bronx
21%
Manhattan
14%
Staten Island
6%
Total Adolescents 12-19*
(n = 818,671)
2/25/13 to 6/29/15
Brooklyn
27%
Queens
29%
Bronx
20%
Manhattan
21%
Staten Island
3%
Current Location Searches by Borough
24Data collected
between 2/27/2013
and 3/20/2016

25. APP EVALUATION
SUCCESSES
25

26. Process Evaluation, Positive Results:
 Qualitative user survey feedback indicate the app is helpful
 Data indicate 20,000+ downloads and searches from 2013-2016
 We are reaching teens where they’re at: users’ current location data searches
mirrors the distribution of adolescents across boroughs
 Teens are using search function as we had hoped, i.e. to identify both services
(e.g. ‘birth control’) and methods (e.g. ‘the pill’)
 The app may help teens discover less popular methods of birth control:
 While only 17% of NYC adolescents report using methods besides condoms or the pill, 52% of
single-method searches in the app are for other forms of contraception (data not shown)
26

27. CHALLENGES FACED
27

28. Evaluation Pain Points: Big Picture
 Population-specific concerns (e.g. privacy)
 Outcome-specific concerns (e.g. measuring behavior change)
 Resources (i.e. time, money, technical support)
 No analytics platform integrated into app
28

29. Evaluation Data Sources
Domain Type of Data Source
Reach
Downloads App Annie
Current installs (Android) Google Console
User characteristics User survey
Marketing
Referral source User survey
Subway campaign recall Subway campaign online survey
Social media impressions and clicks HN/OPAD Media reports
User Interactions
Clinic searches DIIT Analytics
Video shares Facebook Insights
Acceptability
Usability Usability testing sessions
User preferences User survey
Ratings & reviews Apple Store, Google Play Store
Behavior Change Clinic visits & visit intentions User survey, subway campaign survey
29

30. Evaluation Pain Points: On the Ground
 Diversity of strategies—potential strength became a liability
 Fragmentation of data sources (e.g. iOS vs. Android metrics)
 Glitches, bugs, unreliable data, and ‘workarounds’
 Over-reliance on download and search data
 How do we implement an analytics solution while also protecting
users’ privacy?
30

31.  Delayed timeline and
limited IT resources
 Mobile app bugs
 Survey button and
notifications
 Survey crashes
 Other app issues
 Setting and achieving
response target
User Survey Issues
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
% Downloads % Survey Completes
Percentage of App Downloads and
User Survey Completes, by Platform
iOS
Android
31

32. CONCLUSIONS
+ FUTURE DIRECTIONS
32

33. Lessons Learned
 Future campaigns should employ both social science theory and
principles of user experience design
 A sustainable evaluation plan must be developed well before the
launch of promotional activities
 Adequate and sustainable technology resources must be in place
during development, launch and maintenance of app
 Analytics, analytics, analytics!
33

34. Applications for Future Work
 Formative research and/or literature reviews needed on user
engagement and retention strategies for youth-focused mHealth tools
 Important to consider data privacy and security and address concerns
related to target audience e.g. minors, sensitive content
 This experience has influenced the development of an adult-focused
SRH app, launching soon, which has built analytics into the product
 Most of all, this evaluation process has taught us to….
34

35. Be Flexible!
35

36. Acknowledgments
 Radhika Gore, Consultant, Winter/Spring 2015
 Silvia Beltran, Intern, Winter/Spring 2015
 Whitney Wilson, Intern, 2014-2015
 Lauren Birnie, Intern/Consultant, Fall 2013-Summer 2014
 Desiree Abu-Odeh, Intern, Summer 2013
36

37. THANK YOU!
QUESTIONS?
37

38.  Sarah P. Roberts, MPH
 NYC Health Department
 (347) 396-4437
 sroberts2@health.nyc.gov
 https://www.linkedin.com/in/sarahphoeberoberts
38