Paradata (i.e. intervention usage metrics or web/app analytics) have been associated with differential treatment outcomes in online health behavior interventions, yet they remain underutilized and underreported in technology-based HIV prevention and care studies. Paradata collection and analyses can help researchers identify how components in technology-based health interventions lead to behavior change, offer insights into the frequency of usage across intervention components, and provide opportunities to compare the efficacy of specific components across trials. As an emergent area within technology-based health interventions, paradata analyses will require multidisciplinary collaborations between experts in intervention development, bioinformatics, and biostatistics. Building on lessons learned from big data analytics in ecommerce and social media, these partnerships will be critical to determining what and how paradata will be collected and analyzed.

1. Measuring Engagement
in eHealth Interventions
Jesse Golinkoff, MPH
University of Pennsylvania
agolinko@upenn.edu

2. What We Know
● Collect website/app analytics
○ Improve design, UX, site performance
○ Attract and retain users
○ Assess performance of marketing efforts
○ Measure profit
● eHealth interventions à behavior change
○ Behavior change occurs from exposure to and
engagement with intervention

3. The Problem
● Efficacy of interventions is typically assessed by
looking at specific behavioral measures before
and after exposure (i.e. HIV/STI testing behaviors)
● Common practice to report participant dropout
rates or 1 measure of use (i.e. total time spent)
● Limitations to this approach:
○ Assumes similar exposure
○ Assumes similar order of exposure
○ Does not explain which intervention components might be
associated with any behavior change
Harris, S. (1977). What’s So Funny About Science?: Cartoons. Los Altos, CA:
William Kaufmann.

4. What’s Needed
● Intervention analytics: paradata
○ Examine engagement (time spent) and use (features explored) within
an intervention
○ Understand dose-response (i.e. level of exposure at which behavior
change occurs)
○ Assess impact of intervention components (i.e. fact-based content vs.
interactive forums)
○ Discover differences in engagement and use by participant
characteristics (i.e. age, race, education, relationship status, etc)
● Traditional intervention analysis + paradata analysis
for a more complete picture

5. Capturing Paradata
● Establish baseline paradata metrics and
measurement tools
○ Specific paradata needs dictated by intervention design
and goals
● Research teams and developers work
together to ensure necessary user actions
will be recorded
● Work through how to measure engagement
in non-directive eHealth interventions (e.g.
free-form navigation vs. linear progression)

6. Putting Paradata to Use
● Encourage research community to include
paradata collection and reporting as an
element of study design and study results
● In addition to using paradata to complement
traditional analyses, paradata can enable
data-driven decision-making:
○ Intervention iteration
○ Intervention scale-up
○ Platform change/update
○ Intervention adaptation (i.e. targeting different population)

7. Moving Forward
● Collaborations between disciplines to optimize paradata metrics, analyses, and
dissemination (i.e. expand the research team to include experts in computer
science, biostatistics, or learning technologies)
● Expansion into user interface and information architecture-specific research inquiry
(i.e. linear vs. free-form intervention design, otherwise known as A/B testing)
● Nimble/dynamic study design – feed paradata back into an intervention during a
study to deliver tailored content or new/updated components that may better meet
participants’ needs
● Conversations at like these to move the field forward – please share your
experiences and ideas!

8. Jesse Golinkoff, MPH
University of Pennsylvania
agolinko@upenn.edu
Kathryn E. Muessig, PhD
Lisa B. Hightow-Weidman, MD, MPH
Keith J. Horvath, PhD
José A. Bauermeister, MPH, PhD

9. Resources
● Allison, S., Bauermeister, J. A., Bull, S., Lightfoot, M., Mustanski, B., Shegog, R., & Levine, D. (2012). The intersection of
youth, technology, and new media with sexual health: moving the research agenda forward. The Journal of Adolescent
Health: Official Publication of the Society for Adolescent Medicine, 51(3), 207–212.
https://doi.org/10.1016/j.jadohealth.2012.06.012
● Baltierra, N. B., Muessig, K. E., Pike, E. C., LeGrand, S., Bull, S. S., & Hightow-Weidman, L. B. (2016). More than just
tracking time: Complex measures of user engagement with an internet-based health promotion intervention. Journal of
Biomedical Informatics, 59, 299–307. https://doi.org/10.1016/j.jbi.2015.12.015
● Crutzen, R., Roosjen, J. L., & Poelman, J. (2012). Using Google Analytics as a process evaluation method for Internet-
delivered interventions: an example on sexual health. Health Promotion International.
https://doi.org/10.1093/heapro/das008
● Horvath, K. J., Ecklund, A. M., Hunt, S. L., Nelson, T. F., & Toomey, T. L. (2015). Developing Internet-based health
interventions: a guide for public health researchers and practitioners. Journal of Medical Internet Research, 17(1), e28.
https://doi.org/10.2196/jmir.3770
● Pugatch, J., Grenen, E., Surla, S., Schwarz, M., & Cole-Lewis, H. (2018). Information Architecture of Web-Based
Interventions to Improve Health Outcomes: Systematic Review. Journal of Medical Internet Research, 20(3), e97.
https://doi.org/10.2196/jmir.7867
● Zheng, G., & Peltsverger, S. (2015). Web Analytics Overview. In Encyclopedia of Information Science and Technology (3rd
ed., pp. 7674–7683). Hershey, PA: IGI Global. https://doi.org/10.4018/978-1-4666-5888-2.ch756