Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Evaluating Social Marketing Campaigns

7,043 views

Published on

Evaluating Social Marketing Campaigns

Published in: Health & Medicine
  • Be the first to comment

Evaluating Social Marketing Campaigns

  1. 1. Evaluating Social Marketing Campaigns Jorge A. Montoya, Ph.D. Los Angeles County Department of Public Health
  2. 2. Overview <ul><li>Formative vs. Evaluative Research </li></ul><ul><li>Causality </li></ul><ul><li>Common Non-Experimental Research Designs </li></ul><ul><li>A Common Quasi-Experimental Research Design </li></ul><ul><li>Survey Instrument Development </li></ul><ul><li>Key Measures </li></ul><ul><li>Some Evaluation Results </li></ul>
  3. 3. We Are typically Concerned with Two Questions <ul><li>How do we do this (in the best way possible)? </li></ul><ul><ul><li>Formative research </li></ul></ul><ul><ul><ul><li>Descriptive methods </li></ul></ul></ul><ul><ul><ul><li>Exploratory methods </li></ul></ul></ul><ul><li>Is what we are doing (or did) working? </li></ul><ul><ul><li>Evaluation Research </li></ul></ul><ul><ul><ul><li>Strive for cause-effect </li></ul></ul></ul><ul><ul><ul><li>But settle for strong associations (relationships) </li></ul></ul></ul>
  4. 4. Common Non-Experimental Research Designs <ul><li>Cross-Sectional </li></ul><ul><ul><li>Data are collected at one point in time </li></ul></ul><ul><li>Longitudinal </li></ul><ul><ul><li>Data are collected at two or more points in time </li></ul></ul><ul><ul><ul><li>Repeated Cross-sectional design (Different groups) </li></ul></ul></ul><ul><ul><ul><li>Fixed-sample panel design (Same group) </li></ul></ul></ul>X-------O A O A ------X-------O B ------O C O A ------X-------O A ------O A
  5. 5. Common Quasi-Experimental Research Design <ul><li>Non-Equivalent Control group design </li></ul><ul><ul><li>Experimental and control groups are designated before treatment occurs but not created by random assignment </li></ul></ul>Experimental: O A ------X-------O A Control: O B ---------------O B
  6. 6. Causal Validity: Research Design <ul><li>Make Non-Experimental Designs rigorous by using… </li></ul><ul><ul><li>Open-ended questions (when appropriate) </li></ul></ul><ul><ul><li>Statistical control (Multivariate Statistics) </li></ul></ul><ul><ul><li>Repeated measures </li></ul></ul>Causality Association <ul><li>Experimental </li></ul><ul><li>Random assignment </li></ul><ul><li>At least 2 groups </li></ul><ul><li>Control of variation in IV before measure of DV </li></ul><ul><li>Non-Experimental </li></ul><ul><li>Non-random assignment </li></ul><ul><li>1 group only </li></ul><ul><li>Natural variation in IV before measure of DV </li></ul><ul><li>Quasi-Experimental </li></ul><ul><li>Non-random assignment </li></ul><ul><li>At least 2 groups </li></ul><ul><li>Control variation in IV before measure of DV </li></ul>
  7. 7. Survey Instrument Development <ul><li>Basic Considerations </li></ul><ul><li>Define research objectives </li></ul><ul><li>Define population of interest </li></ul><ul><li>Determine data collection method(s) </li></ul><ul><ul><li>Self-administered Survey (e.g., online, paper & pencil, CASI) </li></ul></ul><ul><ul><li>Interview Survey (e.g., face-to-face, phone) </li></ul></ul><ul><li>Determine sample size needed & sampling method </li></ul><ul><ul><li>Keep data collection costs down </li></ul></ul><ul><ul><li>Have the statistical power to reject the null when it is indeed false </li></ul></ul><ul><li>Determine data collection period </li></ul>
  8. 8. Survey Instrument Development <ul><li>Basic Survey Structure </li></ul><ul><li>Not absolute, just a general template </li></ul>Screener <ul><li>Survey Questions </li></ul><ul><li>Funneled </li></ul><ul><li>Avoid contamination </li></ul>Demographics <ul><li>Brief explanation and consent </li></ul><ul><li>Make sure you’re talking to member from population of interest </li></ul><ul><li>Collect information you need to address your research objectives </li></ul><ul><ul><li>Provides data to answer research questions </li></ul></ul><ul><ul><li>Not answers to research questions </li></ul></ul><ul><li>Collect data to describe sample and control for possible confounders </li></ul>Introduction Awareness
  9. 9. Survey Instrument Development <ul><li>Some Survey Question Tips </li></ul><ul><li>Use appropriate lead-ins for scale changes or new sections </li></ul><ul><li>Avoid loaded or leading questions </li></ul><ul><ul><li>Were you more likely to call 911 for emergencies only as a result of seeing this campaign? </li></ul></ul><ul><li>Response choices should be mutually exclusive and exhaustive (i.e., “Other” option) </li></ul><ul><ul><li>How often do you go hiking on weekends? </li></ul></ul><ul><ul><ul><li>Always </li></ul></ul></ul><ul><ul><ul><li>Frequently </li></ul></ul></ul><ul><ul><ul><li>Sometimes </li></ul></ul></ul><ul><ul><ul><li>Occasionally </li></ul></ul></ul><ul><li>Avoid open-ended questions – unless that’s what you’re looking for </li></ul>
  10. 10. Survey Instrument Development <ul><li>Some Survey Question Tips (Cont.) </li></ul><ul><li>Keep response options balanced (Interval measures) </li></ul><ul><ul><ul><li>Poor </li></ul></ul></ul><ul><ul><ul><li>Fair </li></ul></ul></ul><ul><ul><ul><li>Good </li></ul></ul></ul><ul><ul><ul><li>Very Good </li></ul></ul></ul><ul><ul><ul><li>Excellent </li></ul></ul></ul><ul><li>Avoid double/triple barreled questions </li></ul><ul><ul><li>Are automobile consumers looking for high fuel economy and extended warranties? </li></ul></ul><ul><li>Avoid unfamiliar and confusing words or jargon in questions </li></ul><ul><ul><li>Are you in a monogamous relationship? </li></ul></ul><ul><ul><li>How concerned are you with infertility? </li></ul></ul><ul><li>Avoid long questions </li></ul><ul><li>Don’t force answers </li></ul>
  11. 11. Key Campaign Awareness Measures <ul><li>Unaided Awareness </li></ul><ul><ul><li>Name product category and ask for brand recall </li></ul></ul><ul><ul><ul><li>First mentions (top of mind) </li></ul></ul></ul><ul><li>Aided Awareness </li></ul><ul><ul><li>Show campaign elements for recognition </li></ul></ul><ul><li>Modes of Exposure </li></ul><ul><ul><li>Where did you see it? What mediums? </li></ul></ul><ul><li>Exposure Frequency </li></ul><ul><ul><li>How many times did you see it? </li></ul></ul><ul><li>Key Message </li></ul><ul><ul><li>What was the campaign about? What message? </li></ul></ul>
  12. 12. Some Key Potential Co-Factor Measures <ul><li>General Demographics </li></ul><ul><ul><li>Including children, marital status/relationship status, SES questions, etc. </li></ul></ul><ul><li>Risk Factors </li></ul><ul><ul><li>Drug use, previous STD diagnosis, sexual orientation, sex venues </li></ul></ul><ul><li>Sexual Practices </li></ul><ul><ul><li>Sexually active, number of partners, condom use, type of sex, gender of sex partners, etc. </li></ul></ul><ul><li>Medical Access </li></ul><ul><ul><li>Health insurance, regular doctor, etc. </li></ul></ul><ul><li>Living Situation </li></ul><ul><ul><li>Who do you live with? </li></ul></ul>
  13. 13. Some Key Outcome Measures <ul><li>Primary Behavioral Outcome Measure </li></ul><ul><ul><li>Based on the primary objective of your campaign… </li></ul></ul><ul><ul><ul><li>Condom Usage, Safer-sex, Needle Sharing, Drug Use, STD Testing, etc. </li></ul></ul></ul><ul><ul><li>Can opt to measure intention as a fall back </li></ul></ul><ul><li>Knowledge </li></ul><ul><ul><li>About STD/HIV transmission, detection, treatment, most impacted groups, curability, awareness of, etc. </li></ul></ul><ul><li>Perceived Vulnerability </li></ul><ul><ul><li>Measures likelihood perception of respondent being impacted by disease/infection </li></ul></ul><ul><li>Self – Efficacy </li></ul><ul><ul><li>Typically a scale to measure the perceived capability of respondent that he/she can engage in desired behavior </li></ul></ul>
  14. 14. Beyond the Survey: Process Measures <ul><li>Web Site Visits </li></ul><ul><ul><li>Google Analytics, individual page views for key pages with timeline </li></ul></ul><ul><li>Hotline Calls </li></ul><ul><ul><li>Number of calls regarding campaign topic/message with timeline </li></ul></ul><ul><li>Clinic Visits & STD/HIV Testing Volume </li></ul><ul><ul><li>Select clinics with a steady high volume of testing/visits </li></ul></ul><ul><ul><li>Number of tests/visits with timeline </li></ul></ul>
  15. 15. Evaluation of Check Yourself: A Social Marketing Campaign for MSM
  16. 16. Campaign Awareness (Endline=306) General Syphilis Reference *May not equal to 100%, multiple mentions allowed. General HIV/AIDS Reference Blair Underwood – Man Up Safer Sex/Protection/Use Condom General Testing Reference General Crystal Meth Reference 88 Check Yourself/Ad mention General STD Reference Other (Less than 2% mention) Stop the Sores <ul><li>Only 12% of sample was unaware of the campaign </li></ul>
  17. 17. Appeal, Motivation, and Relevance of Campaign Ads
  18. 18. Average Number of Syphilis Tests Performed Per Month Among MSM at LAGLC Clinics Jan 06 Jul 06 Jan 07 Jul 07 Jan 08 Jul 08 Jan 09 Jul 09 Check Yourself Campaign
  19. 19. Evaluation of I Know: A Social Marketing Campaign for Young Women of Color
  20. 20. Campaign Awareness (n=599) General HIV/AIDS Reference BET – Rap It Up HPV/Cervical Cancer/Gardasil Ref. Safer Sex/Protection/Use Condom General Advertising Reference Blair Underwood – Man Up I Know/Don’t Think Know Get Tested/Get Checked Other (Less than 2% mention) General STD Reference 63
  21. 21. Mode of Exposure to Campaign Among Those Aware (n=599) Billboards Television Posters Bus Signage Radio Other (Less than 2% mentions) Magazine *Multiple mentions allowed – May not add to 100%.
  22. 22. Results - STS Campaign Exposure and Coverage Billboard in bus Bathhouses Magazine Poster Outdoor Billboard Outreach Event Phil the Sore Other Sources of campaign exposure: % <ul><li>Among those aware of campaign: </li></ul><ul><ul><li>13% had seen cable TV commercial </li></ul></ul><ul><ul><li>Average exposure to campaign 8 times; (median = 2 times) </li></ul></ul>Television Website / internet * Not equal to 100%, multiple mentions allowed.
  23. 23. Knowledge – Heard of Chlamydia? By Baseline & Endline (Unaware vs. Aware of Campaign) Baseline (n=301) Endline Aware (n=375) Endline Unaware (n=224) Endline Total (n=599) Desired Answer = Yes p<.001 p<.001
  24. 24. CT & GC Testing In Last 6 Months By Campaign Awareness (Endline = 599) Aware (n=375) Not Aware (n=224) * Chi-Square test significant at p=0.011
  25. 25. Predictors of CT & GC Testing In Last 6 Months (n=591) 95% CI Variable Sig. OR Lower Upper Campaign Awareness (Not aware vs. Aware) .041 1.56 1.02 2.38 Language (Spanish vs. English) .001 3.00 1.56 5.75 Age ns 1.05 0.96 1.16 Relationship (No vs. Yes) .015 1.65 1.10 2.48 Children (No vs. Yes) .010 2.04 1.18 3.51 Ethnicity (Black vs. Latino) ns 0.90 0.77 1.05 Medical Insurance (No vs. Yes) ns 0.89 0.57 1.38 Previous STD Diagnosis (No vs. Yes) .000 3.83 1.85 7.94 None REF ---- ---- ---- One .000 12.87 4.45 37.18 Multiple .000 21.54 7.47 62.09 Some High School or Less REF ---- ---- ---- Completed High School ns 1.33 0.70 2.52 Some College/Trade .042 2.00 1.02 3.91 College Graduate + ns 1.95 0.7 4.95 Number of Partners Level of Education
  26. 26. Home Test Kit Orders by Week from June 8, 2009 through February 9, 2010 Week Number of Orders Ad Campaign 71% of total orders Average Weekly = 153 Average Weekly = 30 PR
  27. 27. donthinkknow.org Visits by Week from June 8, 2009 through February 9, 2010 Week Number of Orders Ad Campaign 58% of total visits Average Weekly = 1,126 Average Weekly = 398 PR awareness measure
  28. 28. *How did you hear about DTK? (n=576) Family or friends Online Ad TV Commercial Poster Magazine Article/ Marie Claire No Response Bus Ad – Inside Bus Browsing online/ Search/ Google Other (Less than 2% mentions) Info. Card Bus Ad – Outside Bus *Multiple mentions allowed – May not add to 100%, from dontthinkknow.org web site users. School/ Teacher/ Class Presentation
  29. 29. Contact Info: Jorge A. Montoya, Ph.D. (p)213.864.8959 (f) 866.614.6075 [email_address] Sponsor: www.sentientresearch.net
  30. 30. We Are Concerned with Three Types of Validity <ul><li>Measurement Validity </li></ul><ul><ul><li>Exists when a measure measures what we intend it to measure </li></ul></ul><ul><li>Generalizability (External Validity) </li></ul><ul><ul><li>Exists when the results and conclusions from a sample can be generalized to the population it was drawn from </li></ul></ul><ul><li>Causal Validity (Internal Validity) </li></ul><ul><ul><li>Exists when the conclusion that A leads to or results in B is correct A  B </li></ul></ul>
  31. 31. Measurement Validity: (Operationalization Process) <ul><li>Construct (Latent variable) </li></ul><ul><li>HIV Risk Taking Behavior </li></ul><ul><li>Variables (Manifest Variables) </li></ul><ul><li>Drug Use </li></ul><ul><li>Sexual Behavior </li></ul><ul><li>Questions (Indicators) </li></ul><ul><li>How many different people have used a needle before you in the last month? </li></ul><ul><li>How often, in the last month, have you cleaned needles before re-using them? </li></ul><ul><li>How many people, including clients, have you had sex with in the last month? </li></ul><ul><li>How often have you used condoms when you had sex with casual partners? </li></ul>
  32. 32. 4 Levels of Measurement <ul><li>Nominal – Categorical (represent a quality), values have no mathematical interpretation, just codes </li></ul><ul><li>Ordinal – For greater than and less than distinctions, values represent a ranking by distance between them is not equal </li></ul><ul><li>Interval – Fixed values with relatively equal distance between them (e.g., Likert scale), but no absolute zero point (-, +) </li></ul><ul><li>Ratio – Fixed values with equal distance and an absolute zero value so that ratios can be calculated (-,+, x, ÷) </li></ul><ul><ul><li>Levels are progressive so that they are mathematically more complex than the previous </li></ul></ul><ul><ul><ul><li>Each level has all the benefits of the former plus some additional ones </li></ul></ul></ul>
  33. 33. Properties of Measurement Levels: Permissible Statistics Increase With Each Level A is equal to (not equal to) B = (≠) √ √ √ √ A is greater than (less than) B > (<) √ √ √ A is three more than (less than) B + (-) √ √ A is twice (half) as large as B x ( ÷) √ Appropriate math operations Examples of comparison statements Relevant Level of Measurement Nominal Ordinal Interval Ratio
  34. 34. Evaluation of Stop the Sores : A Social Marketing Campaign for MSM
  35. 35. Results – Campaign Awareness (1) Safer Sex Reference * Not equal to 100%, multiple mentions allowed. Syphilis Reference Get Tested HIV Stops with Me HIV/AIDS Reference Know Your Partner Healthy Penis 71% Stop the Sore/Phil the Sore AIDS Walk Smoking Crystal Meth Other STD/General reference HIV Testing 3
  36. 36. <ul><li>Syphilis Testing In Last 6 Months & Campaign Awareness </li></ul>Results – Syphilis Testing (1) Aware (n=211) Not Aware (n=86) * Chi-Square test significant at p=0.011
  37. 37. <ul><li>Multivariate analysis (n=277) </li></ul><ul><li>Predictors of syphilis test in the last 6 months </li></ul>Results – Syphilis Testing (2) * (includes both aided and unaided awareness) p=0.033 ** CSV visits included due to testing availability at these venues. Ethnicity/ Race HIV Status 0.91 1.43 1.14 Internet Partners 0.89 1.17 1.02 ** CSV Visits 0.97 1.03 1.00 Age 0.14 3.20 .661 Refused 0.01 0.61 .073 Unsure 0.83 3.19 1.63 Positive --- --- REF Negative 0.51 1.71 0.93 Latino/Hispanic --- --- REF White 0.23 2.14 0.70 Asian/Pacific Islander 0.51 2.07 1.03 African-American 1.05 3.20 1.83 * Campaign Awareness 95% CI Lower Upper OR Variable

×