• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Preparation For Outcome Monitoring 2010
 

Preparation For Outcome Monitoring 2010

on

  • 1,824 views

 

Statistics

Views

Total Views
1,824
Views on SlideShare
1,817
Embed Views
7

Actions

Likes
1
Downloads
26
Comments
0

1 Embed 7

http://www.slideshare.net 7

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Evaluation is: A systematic collection of data and information Why Evaluate? Assists in determining program effectiveness Must be done consistently Informs you for future program improvement 4. Helps make you accountable 5. Social justice
  • Participants will be completing a logic model for their intervention. They have already begun this process throughout the last two days. Refer participants to the newsprint and activity “Linking FIBs to Interventions” that they completed the previous day. Participants will pick one risk behavior to address in their expanded logic model. Participants are encouraged to complete the six-column logic model sheet before transposing it to newsprint. Instructions for doing this activity are on the following two pages. Participants get back to the groups they began the training with. Refer to the participant handout #9 “Logic Model” (with six columns) and pass out newsprint. 45 minutes-1 hour to complete the logic model.
  • Answer any final questions! Address final Parking Lot issues if not yet addressed. Mention that any tools we have used in this training can be located on the HIV/STD program website. Let them know that you will be available for technical assistance if they have any further questions about outcome monitoring. Jeff Wagers is also available for technical assistance (his name and phone number are listed in the manual). DSHS field operations and regional staff are also available for TA. Remind participants to complete the evaluation forms and leave them on the table before they leave. Thank participants for their attention and participation! Manual pg. 60 Trainers write their names and phone numbers/ email addresses on newsprint for Evaluations and TA, or can hand out business cards.

Preparation For Outcome Monitoring 2010 Preparation For Outcome Monitoring 2010 Presentation Transcript

  • Preparation for Outcome Monitoring
  • Presenters: 1. Jeff Wagers: Tx DSHS Program/ Training Specialist 2. Ken Ripperger-Suhler, Ph D: University of Texas at Austin, Researcher
  • Introductions
    • Presenters
    • Audience
    • Agenda
    • Objectives
  • What is an Outcome?
    • The resulting changes in the client during or after a client attends an intervention.
  • Brainstorm-Outcomes!
    • What outcomes would you like to see in your intervention?
    • Each intervention will report
  • Agenda
    • Review Basics of Interventions and OM
    • Walk through a sample intervention from logic model to analysis
    • Readiness to do Outcome Monitoring
    • Small Group Breakout by Intervention
    • Administering surveys in a trusting atmosphere
    • Practice inputting survey data
    • Data analysis and Implications
    • Final questions and wrap-up
  • Objectives
    • Review basics of interventions and OM
    • Discuss parameters and preparation for OM in Summer 2010
    • Examine and assess OM pre/post surveys by intervention
    • Learn how to input and analyze data
    • Brainstorm challenges and solutions to OM
    • Reassess readiness to do OM
  • Evidence-Based Interventions (EBI)
    • Clearly defined audience, goals & objectives
    • Based in behavioral theory
    • Focused on specific risk behavior
    • Could be an individual, group, or community level intervention
  • Evidence-Based Interventions (EBI)
    • Individual-Level Interventions: CRCS
    • Group Level Interventions
    • Community Level Interventions
  • Key Terms
    • Internal Logic
    • Core elements
    • Fidelity
    • Behavioral Determinants/ Factors that Influence Behavior/ Risk Factors
    • Tailoring
    • Adapting
    • Reinventing
  • Behavioral Determinants; Factors Influencing Behavior (FIBs)
    • The “why”
    • Reasons a client or population is engaged in the risk behavior
    • Come from the client through:
      • CRCS: asking clients questions
      • GLI, CLI: community assessment
    • Interventions focus on the BD/FIB as well as the behavior
  • Program Evaluation
    • What is evaluation?
    • Why evaluate?
  • Program Evaluation
    • Systematic, consistent collection of data
    • Shows program effectiveness
    • Accountability
    • Program Improvement
    • Increases Capacity
    • Defines parameters
    • Identifies unmet needs
    • Process vs. outcomes
  • What is Process?
    • “ How” you provide your intervention
    • Fidelity to the intervention
    • Quality Assurance issues
    • Skill level of agency staff
    • Numbers (who attended, how many sessions, # of target population, etc)
  • Process Questions
    • Is the location accessible?
    • Is the program implemented as designed?
    • Was the curriculum followed?
    • How many of the target population attended the sessions?
    • How well is the RRS able to keep the session focused on risk behavior?
    • Was staff culturally competent?
    • Others?
  • What is an Outcome?
    • The resulting changes in the client during or after a client attends an intervention.
  • Outcome Questions
    • Did the client
      • Learn anything?
      • Improve their skills?
      • Change their behavior?
      • Realize they are at risk of getting HIV?
  • Outcomes measure:
    • S - skills
    • K - knowledge
    • A - attitude
    • B - belief
    • B - behavior
  • Logic Models
  • Problem Statement Risk Behaviors Influencing Factors Intervention/ Activities Outcomes Immediate Intermediate
    • Outcomes
    • Immediate Outcomes: Immediate
    • results of the intervention, such as
    • changes in knowledge, attitudes,
    • beliefs, and skills.
    • Increased perception of HIV risk
    • - Increased condom use skills
    • - Increased self efficacy to use condoms
    • Intermediate Outcomes: Intervention
    • Results that occur some time after the
    • intervention is completed, such as
    • changes in behaviors, policies, and
    • environmental barriers.
    • Increased condom use
    • Decreased number of partners
  • Factors Influencing Behavior STD rates are high among young Hispanic MSM Problem Statement Young Hispanic MSM don’t use condoms Young Hispanic MSM have multiple partners Risk Behaviors Young Hispanic MSM lack skills to properly use condoms Young Hispanic MSM don’t know how to negotiate condom use with their partners Influencing Factors It’s the norm for young Hispanic MSM to have a lot of sex partners
  • Intervention Activities STD rates are high among young Hispanic MSM Problem Statement Young Hispanic MSM don’t use condoms Risk Behaviors Young Hispanic MSM lack skills to properly use condoms Young Hispanic MSM don’t know how to negotiate condom use with their partners Safe-sex negotiation scenarios & role plays Self-assertion skills training Intervention Activities Influencing Factors Young Hispanic MSM have multiple partners Condom skills demonstration and practice
  • Immediate and Intermediate Outcomes Young Hispanic MSM lack skills to properly use condoms Young Hispanic MSM don’t know how to negotiate condom use with their partners Influencing Factors Safe-sex negotiation scenarios & role plays Self-assertion skills training Intervention Activities Immediate Outcomes Young Hispanic MSM know how to negotiate condom use with their partners Young Hispanic MSM use condoms Intermediate Outcomes Condom demo and practice Young Hispanic MSM know how to use condoms
  • Problem Statement Risk Behaviors Influencing Factors Intervention/ Activities Outcomes Immediate Intermediate Impacts Impacts Impacts: Long-term results of one or more interventions over time, such as changes in HIV infection, morbidity, and mortality - Decreased HIV rates
  • Outcome Monitoring Data Collection And Analysis
  • Steps to Outcome Monitoring
    • Develop *SMART outcome objectives
      • Specific, Measurable, Appropriate, Realistic, Time-Based
    • Develop/ review/ modify surveys or other methods that collect outcomes
    • Gathering survey/ method data
    • Input data into spreadsheets
    • Analyze and report data (Outcome Monitoring)
    • Make program adjustments if needed
  • Example of Process Objective
    • Process objective : “Clients will complete SISTA”
    • Revised : “By December 31, 2010, 50 African-American women will complete SISTA as documented by client data system.”
  • Example of Outcome Objective
    • Outcome objective : “Clients who complete SISTA will show an increase self efficacy”
    • Revised : “75% of participants who complete SISTA will increase self-efficacy to correctly put on a condom as measured by pre/post survey.”
  • Ways to Collect Outcomes
  • Collect Outcomes
    • Skills
      • Correct condom use, needle-bleaching
      • Negotiation, Communication, Assertiveness
    • Questions
      • Knowledge
      • Attitudes
      • Beliefs
      • Behaviors
  • Measuring Skills
    • Demonstrations and Practice
    • Role Plays
  • Demonstrations
    • Examples:
      • Condom Use Skills
      • Needle Bleaching Skills
    • Checklists
      • Consistent
      • Provide “clean” pre-test
      • Demonstrate correct condom use
      • Practice
      • Next session, do a post-test and compare results
  • Role Plays
    • Examples:
      • Condom or bleaching negotiation skills
      • Self Assertiveness skills
    • Observations:
      • Consistent
      • Provide “clean” pre-role play (or reverse role play)
      • Demonstrate correct method
      • Practice
      • Next session, do a post-role play and compare results
  • Types of Questions
    • Open Ended
    • Closed Ended
      • True/False or Yes/No
      • Fill in the Blank
      • Likert Scale
      • Frequency Indicator
      • Multiple Choice
  • Question Answer Options MC Multiple Choice OE Open Ended FI Frequency Indicators S Scale FB Fill in the Blank Y/N Yes/No T/F True/False Answer Code Answer Option
  • Sample Questions
    • Perceived susceptibility
      • “Given my current behavior, I am worried that I might get HIV” (T/F) (Y/N)
      • “Given my current behavior, how worried am I that I might get HIV? (Scale)
      • 1 2 3 4 5
      • Not worried Very worried
  • Sample Questions
    • Expected outcome related to condom use
      • “I will get HIV from my main partner without using a condom” (T/F) (Y/N)
      • “How likely is it that I will get HIV from my main partner without using a condom?”
      • 1 2 3 4 5
      • Not likely at all Very likely
  • Steps to Outcome Monitoring
    • Gathering survey data
      • Individual interviews
      • Written questionnaires
      • Focus groups
    • Input data into spreadsheets
    • Analyze and report data (Outcome Monitoring)
    • Make program adjustments
  • Putting It All Together
  • Example-VOICES/VOCES
    • VOICES/ VOCES is a one-session group level intervention for HRH men and women in which groups watch culturally appropriate videos of condom negotiation, condom use and assertive communication, and practice.
    • Participants role-play condom use scenarios and practice putting on condoms
  • Logic Model Increased Condom Use Increased Condom Use Knowledge Increased Condom Use Self-Efficacy Increased Condom Use Intention INTERMEDIATE OUTCOMES (BEHAVIOR) IMMEDIATE OUTCOMES ACTIVITIES INFLUENCING FACTORS Practice Condom Use; Distribute condoms View Videos; Condom Negotiation Role Plays RISK BEHAVIORS Unprotected Sex Condom Use Knowledge Condom Use Self-Efficacy Condom Use Intention
    • At least 40% of clients will demonstrate greater knowledge of condom usage by identifying on a written test at the end of the session the correct answer (yes or no) with regard to the following knowledge questions :
    • a. Are specific lubricants appropriate for use with condoms;
    • b. Whether or not to unroll a condom before putting it on a man’s penis.
    Objective (Knowledge of Condom Use)
  • Survey Items (Knowledge of Condom Use) 3 2 1 2. You should unroll a condom before putting it on a man’s penis. 3 2 1 1. Vaseline and lotion are good lubricants for condoms/rubbers. Don’t Know No Yes
  • Spreadsheet for storing data yes  yes  -  GAY  M  27  H  A03  yes  yes  -  BI  M  25  AA  A02    yes yes -  STR  M  30    AA A01 Pre ConKnow 2 Pre ConKnow 1 HIV Status Sexual Orient Gender Age at intake Race/ Ethnic Client ID
  • Frequencies P r e & Post 0 (0%) 20 (100%) 0 (0%) (POST) 0 (0%) 0 (0%) 20 (100%) (PRE) Don’t Know (%) No (%) Yes (%) Item Vaseline and lotion are good lubricants for condoms/rubbers.
  • Vaseline and lotion are good lubricants for condoms/rubbers. 20 (0%) 0 20 0 Total 0 (0%) 0 0 0 Don’t Know 0 (0%) 0 0 0 No Responses 20 (100%) 0 20 0 Yes Pre Total Don’t Know No Yes Post Responses
  • Frequencies Pre & Post 0 (0%) 0 (0%) 20 (100%) (POST) 0 (0%) 0 (0%) 20 (100%) (PRE) Don’t Know (%) No (%) Yes (%) Item You should unroll a condom before putting it on a man’s penis.
  • You should unroll a condom before putting it on a man’s penis. 20 (100%) 0 (0%) 0 (0%) 20 (100%) Total 0 (0%) 0 (0%) 0 (0%) 0 (0%) Don’t Know 0 (0%) 0 (0%) 0 (0%) 0 (0%) No Responses 20 (100%) 0 (0%) 0 (0%) 20 (100%) Yes Pre Total Don’t Know No Yes Post Responses
  • Table 1. Change in Knowledge Scores from Pre to Post 0 0 0 0 2. You should unroll a condom before putting it on a man’s penis. 100 0 100 100 1. Vaseline and lotion are good lubricants for condoms/rubbers. % change to correct answers % changed to incorrect % changed to correct % correct post Item
  • Outcome Monitoring: VOICES/VOCES Report
    • Condom Use Knowledge
    • The results of cross tab analysis find that
    • 100% of clients showed increased knowledge of appropriate lubricants for condom use.
    • 0% of clients showed increased knowledge regarding when to unroll a condom to put on a man’s penis.
  • Planning for Outcome Monitoring
  • Planning for Outcome Monitoring
      • Readiness for Outcome Monitoring
    • Done on an individual agency basis
    • Stable implementation
    • Data collection instruments developed by Ken with agency input
    • Outcomes may change over time
  • DSHS Parameters for Outcome Monitoring
    • Skills-building, knowledge, behavioral determinants ( AKA FIBs)
    • BD/ FIBs identified in your intervention based on research
    • Pre/post survey (unless an agency wants to do observations)
    • Based on logic model of intervention
    • 3-4 outcomes for GLI
  • Some Differences in Data Collection….
    • CRCS
      • Intermediate and Immediate Outcomes
      • Use questions from behavioral assessment
      • No standardized survey
      • One initial assessment, up to two reassessments
      • Driven by prevention plan and goals
      • Designated outcomes
      • Pre/post done on same individual
      • Percentages for outcomes 15-50%
      • Not all outcomes will apply to all individuals
  • Some Differences in Data Collection….
    • Group Level
      • Immediate Outcomes
      • Pre/post tests
      • Standardized survey
      • 3-4 outcomes individualized per intervention
      • Pre/post done on same individual
      • Percentages for outcomes 40-75%
      • Can measure any SKABB
  • Some Differences in Data Collection….
    • Community Level: 2 components
      • Group Component- measured first
        • 2-4 outcomes
        • Will measure M-Groups or training of peer volunteers
        • Standardized survey
        • Pre/post; percentages 50-80%
        • Knowledge, skill-level, able to communicate very important
        • Will be collected first stage
  • Some Differences in Data Collection….
    • Community Level:
      • Community Component-measured second
        • 2-4 Outcomes
        • Done during community assessment
        • Pre/Post tests; Venue-based
        • Mainly measures knowledge, attitudes, beliefs, community norms
        • Percentages up to 15%
        • Gathered yearly
        • Will be collected in second stage
  • Outcome Monitoring Surveys
    • Ask 3-4 questions per outcome
    • 10-20 questions per survey
    • Can measure skills through survey
    • Knowledge can be a component
    • Include key outcomes of intervention, in line with behavioral determinants
    • Consistent data collection
  • Facilitated Group Activity 1
    • CLI: Discuss group and community level components of intervention
    • GLI: Discuss and review OM surveys by intervention
    • CRCS: Brainstorm problems and possible solutions to gathering OM data
  • Guided Group Discussion
    • Ways to Collect Outcomes
    • Giving Instructions
    • Oral Interviews
    • Written Questions
    • Focus Groups
  • Activity 2: Data Input and Analysis
    • Input sample data into a spreadsheet
      • Sample data is located in packet
      • Input into your laptop using the sample spreadsheet (partner with someone if needed)
    • Analyze the data
      • Spreadsheet calculates the data
      • What can you determine from analysis?
  • Next Steps Per intervention, brainstorm challenges and solutions to Outcome Monitoring
    • Final Questions?
    • Please complete your course evaluations
    • Thank you for your participation!
    Thank You!
  • Contact information
    • Jeff Wagers, TxDSHS
      • [email_address]
      • 512-533-3022
    • Ken Ripperger-Suhler, University of Texas
      • [email_address]
      • 254-624-1028