Evaluating Social Marketing Campaigns

Evaluating Social Marketing Campaigns Jorge A. Montoya, Ph.D. Los Angeles County Department of Public Health

Overview
Formative vs. Evaluative Research
Causality
Common Non-Experimental Research Designs
A Common Quasi-Experimental Research Design
Survey Instrument Development
Key Measures
Some Evaluation Results

We Are typically Concerned with Two Questions
How do we do this (in the best way possible)?
Formative research
Descriptive methods
Exploratory methods
Is what we are doing (or did) working?
Evaluation Research
Strive for cause-effect
But settle for strong associations (relationships)

Common Non-Experimental Research Designs
Cross-Sectional
Data are collected at one point in time
Longitudinal
Data are collected at two or more points in time
Repeated Cross-sectional design (Different groups)
Fixed-sample panel design (Same group)
X-------O A O A ------X-------O B ------O C O A ------X-------O A ------O A

Common Quasi-Experimental Research Design
Non-Equivalent Control group design
Experimental and control groups are designated before treatment occurs but not created by random assignment
Experimental: O A ------X-------O A Control: O B ---------------O B

Causal Validity: Research Design
Make Non-Experimental Designs rigorous by using…
Open-ended questions (when appropriate)
Statistical control (Multivariate Statistics)
Repeated measures
Causality Association
Experimental
Random assignment
At least 2 groups
Control of variation in IV before measure of DV
Non-Experimental
Non-random assignment
1 group only
Natural variation in IV before measure of DV
Quasi-Experimental
Non-random assignment
At least 2 groups
Control variation in IV before measure of DV

Basic Considerations
Define research objectives
Define population of interest
Determine data collection method(s)
Self-administered Survey (e.g., online, paper & pencil, CASI)
Interview Survey (e.g., face-to-face, phone)
Determine sample size needed & sampling method
Keep data collection costs down
Have the statistical power to reject the null when it is indeed false
Determine data collection period

Basic Survey Structure
Not absolute, just a general template
Screener
Survey Questions
Funneled
Avoid contamination
Demographics
Brief explanation and consent
Make sure you’re talking to member from population of interest
Collect information you need to address your research objectives
Provides data to answer research questions
Not answers to research questions
Collect data to describe sample and control for possible confounders
Introduction Awareness

Some Survey Question Tips
Use appropriate lead-ins for scale changes or new sections
Avoid loaded or leading questions
Were you more likely to call 911 for emergencies only as a result of seeing this campaign?
Response choices should be mutually exclusive and exhaustive (i.e., “Other” option)
How often do you go hiking on weekends?
Always
Frequently
Sometimes
Occasionally
Avoid open-ended questions – unless that’s what you’re looking for

Some Survey Question Tips (Cont.)
Keep response options balanced (Interval measures)
Poor
Fair
Good
Very Good
Excellent
Avoid double/triple barreled questions
Are automobile consumers looking for high fuel economy and extended warranties?
Avoid unfamiliar and confusing words or jargon in questions
Are you in a monogamous relationship?
How concerned are you with infertility?
Avoid long questions
Don’t force answers

Key Campaign Awareness Measures
Unaided Awareness
Name product category and ask for brand recall
First mentions (top of mind)
Aided Awareness
Show campaign elements for recognition
Modes of Exposure
Where did you see it? What mediums?
Exposure Frequency
How many times did you see it?
Key Message
What was the campaign about? What message?

Some Key Potential Co-Factor Measures
General Demographics
Including children, marital status/relationship status, SES questions, etc.
Risk Factors
Drug use, previous STD diagnosis, sexual orientation, sex venues
Sexual Practices
Sexually active, number of partners, condom use, type of sex, gender of sex partners, etc.
Medical Access
Health insurance, regular doctor, etc.
Living Situation
Who do you live with?

Some Key Outcome Measures
Primary Behavioral Outcome Measure
Based on the primary objective of your campaign…
Condom Usage, Safer-sex, Needle Sharing, Drug Use, STD Testing, etc.
Can opt to measure intention as a fall back
Knowledge
About STD/HIV transmission, detection, treatment, most impacted groups, curability, awareness of, etc.
Perceived Vulnerability
Measures likelihood perception of respondent being impacted by disease/infection
Self – Efficacy
Typically a scale to measure the perceived capability of respondent that he/she can engage in desired behavior

Beyond the Survey: Process Measures
Web Site Visits
Google Analytics, individual page views for key pages with timeline
Hotline Calls
Number of calls regarding campaign topic/message with timeline
Clinic Visits & STD/HIV Testing Volume
Select clinics with a steady high volume of testing/visits
Number of tests/visits with timeline

Evaluation of Check Yourself: A Social Marketing Campaign for MSM

Campaign Awareness (Endline=306) General Syphilis Reference *May not equal to 100%, multiple mentions allowed. General HIV/AIDS Reference Blair Underwood – Man Up Safer Sex/Protection/Use Condom General Testing Reference General Crystal Meth Reference 88 Check Yourself/Ad mention General STD Reference Other (Less than 2% mention) Stop the Sores
Only 12% of sample was unaware of the campaign

Appeal, Motivation, and Relevance of Campaign Ads

Average Number of Syphilis Tests Performed Per Month Among MSM at LAGLC Clinics Jan 06 Jul 06 Jan 07 Jul 07 Jan 08 Jul 08 Jan 09 Jul 09 Check Yourself Campaign

Evaluation of I Know: A Social Marketing Campaign for Young Women of Color

Campaign Awareness (n=599) General HIV/AIDS Reference BET – Rap It Up HPV/Cervical Cancer/Gardasil Ref. Safer Sex/Protection/Use Condom General Advertising Reference Blair Underwood – Man Up I Know/Don’t Think Know Get Tested/Get Checked Other (Less than 2% mention) General STD Reference 63

Mode of Exposure to Campaign Among Those Aware (n=599) Billboards Television Posters Bus Signage Radio Other (Less than 2% mentions) Magazine *Multiple mentions allowed – May not add to 100%.

Results - STS Campaign Exposure and Coverage Billboard in bus Bathhouses Magazine Poster Outdoor Billboard Outreach Event Phil the Sore Other Sources of campaign exposure: %
Among those aware of campaign:
13% had seen cable TV commercial
Average exposure to campaign 8 times; (median = 2 times)
Television Website / internet * Not equal to 100%, multiple mentions allowed.

Knowledge – Heard of Chlamydia? By Baseline & Endline (Unaware vs. Aware of Campaign) Baseline (n=301) Endline Aware (n=375) Endline Unaware (n=224) Endline Total (n=599) Desired Answer = Yes p<.001 p<.001

CT & GC Testing In Last 6 Months By Campaign Awareness (Endline = 599) Aware (n=375) Not Aware (n=224) * Chi-Square test significant at p=0.011

Predictors of CT & GC Testing In Last 6 Months (n=591) 95% CI Variable Sig. OR Lower Upper Campaign Awareness (Not aware vs. Aware) .041 1.56 1.02 2.38 Language (Spanish vs. English) .001 3.00 1.56 5.75 Age ns 1.05 0.96 1.16 Relationship (No vs. Yes) .015 1.65 1.10 2.48 Children (No vs. Yes) .010 2.04 1.18 3.51 Ethnicity (Black vs. Latino) ns 0.90 0.77 1.05 Medical Insurance (No vs. Yes) ns 0.89 0.57 1.38 Previous STD Diagnosis (No vs. Yes) .000 3.83 1.85 7.94 None REF ---- ---- ---- One .000 12.87 4.45 37.18 Multiple .000 21.54 7.47 62.09 Some High School or Less REF ---- ---- ---- Completed High School ns 1.33 0.70 2.52 Some College/Trade .042 2.00 1.02 3.91 College Graduate + ns 1.95 0.7 4.95 Number of Partners Level of Education

Home Test Kit Orders by Week from June 8, 2009 through February 9, 2010 Week Number of Orders Ad Campaign 71% of total orders Average Weekly = 153 Average Weekly = 30 PR

donthinkknow.org Visits by Week from June 8, 2009 through February 9, 2010 Week Number of Orders Ad Campaign 58% of total visits Average Weekly = 1,126 Average Weekly = 398 PR awareness measure

*How did you hear about DTK? (n=576) Family or friends Online Ad TV Commercial Poster Magazine Article/ Marie Claire No Response Bus Ad – Inside Bus Browsing online/ Search/ Google Other (Less than 2% mentions) Info. Card Bus Ad – Outside Bus *Multiple mentions allowed – May not add to 100%, from dontthinkknow.org web site users. School/ Teacher/ Class Presentation

Contact Info: Jorge A. Montoya, Ph.D. (p)213.864.8959 (f) 866.614.6075 [email_address] Sponsor: www.sentientresearch.net

We Are Concerned with Three Types of Validity
Measurement Validity
Exists when a measure measures what we intend it to measure
Generalizability (External Validity)
Exists when the results and conclusions from a sample can be generalized to the population it was drawn from
Causal Validity (Internal Validity)
Exists when the conclusion that A leads to or results in B is correct A  B

Measurement Validity: (Operationalization Process)
Construct (Latent variable)
HIV Risk Taking Behavior
Variables (Manifest Variables)
Drug Use
Sexual Behavior
Questions (Indicators)
How many different people have used a needle before you in the last month?
How often, in the last month, have you cleaned needles before re-using them?
How many people, including clients, have you had sex with in the last month?
How often have you used condoms when you had sex with casual partners?

4 Levels of Measurement
Nominal – Categorical (represent a quality), values have no mathematical interpretation, just codes
Ordinal – For greater than and less than distinctions, values represent a ranking by distance between them is not equal
Interval – Fixed values with relatively equal distance between them (e.g., Likert scale), but no absolute zero point (-, +)
Ratio – Fixed values with equal distance and an absolute zero value so that ratios can be calculated (-,+, x, ÷)
Levels are progressive so that they are mathematically more complex than the previous
Each level has all the benefits of the former plus some additional ones

Properties of Measurement Levels: Permissible Statistics Increase With Each Level A is equal to (not equal to) B = (≠) √ √ √ √ A is greater than (less than) B > (<) √ √ √ A is three more than (less than) B + (-) √ √ A is twice (half) as large as B x ( ÷) √ Appropriate math operations Examples of comparison statements Relevant Level of Measurement Nominal Ordinal Interval Ratio

Evaluation of Stop the Sores : A Social Marketing Campaign for MSM

Results – Campaign Awareness (1) Safer Sex Reference * Not equal to 100%, multiple mentions allowed. Syphilis Reference Get Tested HIV Stops with Me HIV/AIDS Reference Know Your Partner Healthy Penis 71% Stop the Sore/Phil the Sore AIDS Walk Smoking Crystal Meth Other STD/General reference HIV Testing 3

Syphilis Testing In Last 6 Months & Campaign Awareness
Results – Syphilis Testing (1) Aware (n=211) Not Aware (n=86) * Chi-Square test significant at p=0.011

Multivariate analysis (n=277)
Predictors of syphilis test in the last 6 months
Results – Syphilis Testing (2) * (includes both aided and unaided awareness) p=0.033 ** CSV visits included due to testing availability at these venues. Ethnicity/ Race HIV Status 0.91 1.43 1.14 Internet Partners 0.89 1.17 1.02 ** CSV Visits 0.97 1.03 1.00 Age 0.14 3.20 .661 Refused 0.01 0.61 .073 Unsure 0.83 3.19 1.63 Positive --- --- REF Negative 0.51 1.71 0.93 Latino/Hispanic --- --- REF White 0.23 2.14 0.70 Asian/Pacific Islander 0.51 2.07 1.03 African-American 1.05 3.20 1.83 * Campaign Awareness 95% CI Lower Upper OR Variable

Evaluating Social Marketing Campaigns

by CDC NPIN

Accessibility

Categories

Upload Details

Usage Rights

2 Embeds 49

Statistics