The document describes the development of an online HIV prevention program evaluation training for grantees. It details the history of moving from in-person trainings to developing online training modules and webinars. Usage data is provided showing over 1,700 training lessons completed and high satisfaction rates among participants. Lessons learned include webinars promoting online training usage and the need for the training to evolve with changes in HIV prevention.

1. Developing National On-Line HIV Prevention
Program Evaluation Training
David Davis, Ph.D. ,and the NHM&E Training Team
Program Evaluation Branch, Division of HIV/AIDS Prevention
National HIV Prevention Conference
August 15, 2011
National Center for HIV/AIDS, Viral Hepatitis, STD, and TB Prevention
Division of HIV/AIDS Prevention

2. Outline
 NHM&E Training Project
 History
 Contents
 Data
 Lessons Learned

3. National HIV Prevention Program Monitoring and
Evaluation (NHM&E) Training Project
 Consists of
 On-line, on-demand training modules
 Webinars (live and recorded)
 Covers
 Basics of monitoring and evaluation
 NHM&E standardized variables
 Data reporting system (PEMS)
 Local use of data

4. The Old Way
 Two people per grantee agency (59 HDs, 130 CBOs)
 In person, in Atlanta
 Week long training sessions
 About 30 people at a time
 Done after major changes to data reporting system

5. The New Way
 On-line, web-based, multi-media, interactive training
modules
 Available on demand to all grantee and subcontractor
staff
 Webinars for most training modules as they are
released
 Webinars recorded for playback on demand
 Staff can take only those modules they need

6. Contents
 Sixteen “courses” or “ modules” with one or more
lessons
 Fift y-nine planned lessons, 43 currently available
 Sixteen current lessons specific to PEMS software

7. Topics
 M&E Guidance and Tools
 Data Qualit y Assurance
 HIV Prevention Program Indicators
 Communit y Planning
 Partner Services
 Referrals

8. More Topics
 Client-Level Services
 Aggregate-Level Services
 HIV Testing
 Agency Information
 Program Planning
 Data Securit y

9. Even More Topics
 Introduction to NHM&E Training
 Data Submission
 Data Transfer

10. Usage (5/2010-5/2011)
 1,794 lessons completed
 6,162 visits by 899 visitors
 Average of 120 visits per week
 Average of 17 first-time users per week
 Average of about 15 minutes per visit
 About 1.7 hours per visitor average total time on site

11. Most Completed Lessons
 Training Navigation Tutorial
 Monitoring and Evaluation Overview
 Data Submission
 PEMS Admin Module
 Data Qualit y Assurance

12. On-Demand Training Feedback
Participants’ Satisfaction With On-Demand Training Delivery
Statement n Not at All Somewhat Neutral For the Most Very Much N/A
Part So
The assistance I have received related to 42 0 (0%) 7 (17%) 4 (10%) 16 (38%) 10 (24%) 5 (12%)
the on-demand training met my needs.
Do the elements (e.g., the navigation 42 0 (0%) 6 (14%) 4 (10%) 19 (45%) 13 (31%) N/A
text, buttons on the play-bar) on the
screen enable you to move through the
on-demand training with ease?
The visual aids (e.g., graphics and 42 2 (5%) 6 (14%) 6 (14%) 15 (36%) 13 (31%) N/A
animation) have kept me engaged in the
on-demand training.
The Quick Checks have contributed to 42 3 (7%) 5 (12%) 6 (14%) 18 (43%) 10 (24%) N/A
my knowledge of the subject matter.
The on-demand training was well 42 0 (0%) 5 (12%) 7 (17%) 18 (43%) 12 (29%) N/A
organized.

13. More On-Demand Feedback
Participants’ Satisfaction With Relevance to Job
Statement n Not at All Somewhat Neutral For the Most Part Very Much So
The training has provided me 41 1 (2%) 10 (24%) 3 (7%) 23 (56%) 4 (10%)
with practical insights into the
content area (i.e., best practices
and/or lessons learned).
The on-demand training has 41 3 (7%) 6 (15%) 13 (32%) 9 (22%) 10 (24%)
provided useful tools (e.g.,
worksheets).
I would recommend this training 41 4 (10%) 6 (15%) 4 (10%) 12 (29%) 15 (37%)
to my colleagues.

14. Comments on On-Demand Training
 2% said training was too basic and 8% said too
advanced.
 Lack of time was most common barrier to using the
training.
 Site navigation was a problem for a few people.

15. Webinars
 Program Planning – June 24, 2010
 Agency Information – July 28, 2010
 M&E Guidance and Tools – October 14, 2010
 Data Qualit y Assurance and Securit y – November 18,
2010
 Client-Level Services Data – January 20, 2011
 Aggregate-Level Services Data – March 31, 2011
 Average 65 website logins per webinar (multiple
participants on some logins)

16. Webinar Delivery Dates and Web Traffic
300 Program Planning
Webinar Held
Security Measures and Data
M&E Guidance and
250 Quality Assurance Webinar Held
Tools Webinar Held
Aggregate-Level
200 Services Course
Client-Level Services
Webinar Held
Agency Information
Number of Visits
Webinar Held
150 Visits
100
50
0
*Number of individual sessions initiated by all visitors to the site.

17. Post Webinar Feedback
Participants’ Perceptions of Overall Webinar Quality
Program Agency M&E Guidance Data Quality Client-Level Aggregate-
Planning Information and Tools Assurance and Services Level Services
Security Measures
(n = 49) (n = 25) (n = 21] (n = 50] (n = 24) (n = 12)
The Webinar was well mean = 4.31 mean = mean = 4.38 mean = mean = 4.04 mean = 4.83
organized. 4.48 4.18
The Webinar met my learning mean = 4.78 mean = mean = 3.81 mean = mean = 3.83 mean = 4.58
needs. 3.72 3.71

18. More Post Webinar Feedback
Participants’ Perceptions of How Well the Webinar Has Prepared Them to Better Perform Their Jobs
Program Agency M&E Guidance Data Quality Client-Level Aggregate-
Planning Information and Tools Assurance and Services Level Services
Security Measures
(n = 49) (n = 25) (n = 21] (n = 51] (n = 24) (n = 12)
Poor 4 (8%) 1 (4%) 1 (5%) 1 (2%) 0 (0%) 0 (0%)
Marginal 2 (4%) 2 (8%) 0 (0%) 5 (10%) 1 (4%) 0 (0%)
Fair 7 (14%) 4 (16%) 4 (19%) 8 (16%) 5 (21%) 2 (17%)
Good 22 (45%) 10 (40%) 7 (33%) 20 (39%) 8 (33%) 2 (17%)
Excellent 12 (24%) 7 (28%) 9 (43%) 14 (27%) 9 (38%) 7 (58%)
N/A 2 (4%) 1 (4%) 0 (0%) 3 (6%) 1 (4%) 1 (8%)

19. Grantee suggestions for webinars
 Have grantees present in webinars on best practices
from the field.
 Don’t repeat on-demand training, supplement it.
 Separate webinars for different skill and experience
levels.

20. Lessons Learned
 Webinars prompt use of the on-demand training.
 Grantees like to hear from other grantees.
 Not being away from the office creates time conflicts to
taking on-line training.
 On-line training appears to deal with staff and grantee
turnover.
 As HIV prevention changes, the training will need to
change as well.

21. Acknowledgements
 CDC Staff
 Antonya Rakestraw
 ICF Macro Staff
 Emily Baughman
 Sonya Shropshire
 Micheal Raines
 Jessica Wals
 Altus Waters
 Kira Sloop
 Brieon Arthur
 Betty Treschitta
 Katherine Akopov
 Elizabeth Jacobs
 John Afon
 Nkateko Okwera

22. Thank you
For more information please contact David Davis at the Centers for Disease Control and
Prevention
Mail Stop E-59
1600 Clifton Road NE, Atlanta, GA 30333
Telephone: 1-404-639-0938
E-mail: dad5@ cdc.gov
The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the
Centers for Disease Control and Prevention.
National Center for HIV/AIDS, Viral Hepatitis, STD , and TB Prevention
Division of HIV/AIDS Prevention