Evaluating Social Marketing Campaigns
Upcoming SlideShare
Loading in...5
×
 

Evaluating Social Marketing Campaigns

on

  • 7,148 views

Evaluating Social Marketing Campaigns

Evaluating Social Marketing Campaigns

Statistics

Views

Total Views
7,148
Views on SlideShare
7,099
Embed Views
49

Actions

Likes
0
Downloads
142
Comments
0

2 Embeds 49

http://www.slideshare.net 47
http://cc.bingj.com 2

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Formative research typically involves Qualitative methods : focus groups, one-on-one interviews, participant observation, etc. Qualitative research in terms of focus groups will be covered in more detail by VIVIAN MARLES. Formative research can also include quantitative methods `Market Segmentation (i.e., Hierarchical Cluster Analysis) `Trade-off Analysis (i.e., Choice-Based Conjoint Analysis) I will focus on Quantitative Evaluation Research .
  • Cross-Sectional One Shot Case Study Longitudinal Designs… Repeated Cross-Sectional – Usually manifested as trend studies (e.g., global changes in public attitudes such as racial tolerance, abortion, gun control; political polling such as favorability ratings of the president). Fixed-Sample Panel – This design follows the same individuals over time so that it allows us to observe changes in the DV in each individual who acts is his/her own control. Considered better than Repeated Cross-Sectional for testing causal hypotheses. Like Before-and-After or One-Group Pretest Posttest Design. Cohort study : Where you select those with exposure and those without exposure and follow forward (or backwards in time) for the development/occurrence of disease/outcome. A cohort is a group of people who share a common characteristic or experience within a defined period (e.g., are born, leave school, lose their job, are exposed to a drug or a vaccine, etc.). Thus a group of people who were born on a day or in a particular period, say 1948, form a birth cohort. The comparison group may be the general population from which the cohort is drawn, or it may be another cohort of persons thought to have had little or no exposure to the substance under investigation, but otherwise similar. Alternatively, subgroups within the cohort may be compared with each other. A research study that compares a particular outcome (such as lung cancer) in groups of individuals who are alike in many ways but differ by a certain characteristic (for example, female nurses who smoke compared with those who do not smoke). The other problem with cohort studies is that they can end up taking a very long time, since the researchers have to wait for the conditions of interest to develop. Researchers are, of course, anxious to have meaningful results as soon as possible, but another disadvantage with long studies is that things tend to change over the course of the study. So research turn to the case-control study. Case-Control Study: case-control, you identify your cases (have outcome) and controls (don't have outcome) and then you can look look back in time or at current exposure, so can have longitudinal component that is retrospective. could also have prospective, eg., measures of biomarkers forward in time. A study that compares two groups of people: those with the disease or condition under study (cases) and a very similar group of people who do not have the disease or condition (controls). Researchers study the medical and lifestyle histories of the people in each group to learn what factors may be associated with the disease or condition. For example, one group may have been exposed to a particular substance that the other was not. Also called retrospective study
  • -Usually the best alternative to an experimental design in maximizing internal validity is quasi-experimental design. - No consistent definition of quasi-experimental design in the literature, but generally it is one that is like an experimental design with the exception that groups are not randomly created. Statistical Control: Conduct multivariate statistics and control for -Demographics -Other relevant variables to the DV Use of pre-test and post-test so that subjects serve as their own control.
  • Refusal rates Screen outs Funnelled…Broad general questions in the beginning, warm-ups…more specific questions later..sensitive questions much later too. Example with does syphilis have any effect on HIV? What kind of effect?
  • Examples of bad questions: Double barreled: How relevant and motivational do you find the advertisements?
  • WOC Unaided: There are many announcements and ads to keep people informed about issues. Can you recall any ads or public events during the last year that provided information about sexually transmitted diseases? Unaided awareness question from STS: Can you recall any campaigns or advertisements in Los Angeles in the past year that were about sexual health in your community? Aided: Do you recall hearing or seeing one of these ads (SHOW ADS)? Modes of Exposure: Where do you recall seeing or hearing this advertising? Exposure frequency: About how many times did you see/hears this/these ads? _____ # (best guess) Key Message(s): What is/are the main message(s) in this advertising? (DO NOT READ OPTIONS)
  • A Co-Factor (or a potential confounder) is anything that can have an influence in your outcome besides you campaign. Now this can be a lot of things, so we have to be practical and use a lot of common sense to keep the list of questions at a reasonable length. FOR example, for WOC potential co-factors was covered with about 15 questions.
  • Process measures that measure implementation vs. process measures that help assess outcome
  • Aware=269, 30=Unaware There are many announcements and publicity in the media that are directed at the public to keep them informed about issues in their community. Can you recall any ads or public events that provided information about sexual health during the last year here in Los Angeles? If yes, please describe the campaign or key messages as best as you can. (WRITE IN NAME OR DESCRIPTION VERBATIM GIVEN BY RESPONDENT IN ORDER ANSWERED. DO NOT PROMPT OR PROBE
  • 10% is a watermark – there were no billboards in bus, so this is a false listing. We estimate that anything above 10% should be a valid percentage. Multiple answers were allowed for last question. Magazines was most commonly given answer. (people listed frontiers, adelante, and odyssey.) Television + TV commercial were combined for this graph to equal 12 %. Only 7% mentioned the STS commercial unaided. Internet + website were also combined to equal 7%. Only 4% mentioned the STS website unaided. Other: consisted of mostly places, where people would have seen various outreach materials. These places could have included places where the mobile testing van was located. In a few cases, they actually mentioned outreach materials (posters, stress grip, palm cards, comic book.) We can’t say specifically what these outreach materials are, however. Other – unique mentions of less than 3%. Median is only 2 because there were a lot of 0’s for seeing campaign in last 3 months.
  • Baseline vs. Endline Total significant at p=.005 Baseline vs. Endline Aware significant at p<.001
  • 80% drop in average weekly orders after the campaign ended. 71% of orders occurred during the campaign or 1,885/2,572
  • 65% drop in average weekly visits after marketing campaign Visits during campaign = 13,508 / After = 9,939
  • This is how you avoid GIGO (Garbage in, Garbage out). This is also where sampling, instrument development, and design come into play. Measurement Validity is the first concern in establishing the validity of research results. Usually remedied by using proven scales (i.e., self-efficacy, etc.) Quality of instrument design will determine measurement validity. Causal Validity considerably much more important in evaluative research. Remedied with a sound research design and clearly defined dependent and independent variables. Generalizability Sampling strategies can remedy generalizability. Two aspects of Generalizability: - Sample Generalizability : Exists when a conclusion based on a sample , or subset, of a larger population holds to for that population - Cross-population generalizability : Exists when findings about one group or population or setting hold true for other groups or populations or settings (Also called External Validity ) External validity is absolutely meaningless if you do not have internal validity.
  • Additional manifest variables with subsequent questions may include Sexual Addiction, Self-Esteem, Self-Efficacy with regards to Safer Sex Practices, etc. Self-Efficacy: Person’s belief about his/her ability or capacity to succeed in a particular situation (accomplish a task or to deal with the challenges of life). Most often you can use available data, theories or scales that have already defined constructs and have a specific set of questions associated with them, or scales. The use of scales in research will be covered in greater detail by LAUREN FRANK How are manifest variables identified?
  • You can elaborate on how there is a loss of information as you move from ratio to nominal, so whenever possible you try to collect ratio info, which you can recategorize to become nominal, ordinal, etc. My favorite example of this is when someone gave me age, which was collected in categories, and asked me to compute the mean age!! perfect example of loss of info.
  • Respondents were asked if they could recall any ads or public events that provided information about sexual health issues since June of last year in Los Angeles. All of these responses are unaided awareness, except that aided questions were asked regarding STS and Healthy Penis. Top bars represent mentions of specific health campaigns. Healthy Penis awareness also went up from the 2002-2003 evaluation from 5% awareness to 21%. 27% unaided awareness is larger even than general HIV/AIDS reference.
  • Here you can see that syphilis testing was significantly associated with campaign awareness. We are comparing those who were exposed to the campaign versus those who weren’t. You can see that >>>>>>>> And this difference is stat. sig.
  • Here you can see that campaign awareness is the only factor that is significantly related to syphilis testing, with an odds ratio of 1.83. We did a multivariate analysis. This allows us to control for other variables which might be assoc. with syphilis testing behavior. After controlling for XXXX variables, we found that those exposed to our campaign were nearly twice as likely to be tested for syphilis. Campaign awareness is the only sig. variable.

Evaluating Social Marketing Campaigns Evaluating Social Marketing Campaigns Presentation Transcript

  • Evaluating Social Marketing Campaigns Jorge A. Montoya, Ph.D. Los Angeles County Department of Public Health
  • Overview
    • Formative vs. Evaluative Research
    • Causality
    • Common Non-Experimental Research Designs
    • A Common Quasi-Experimental Research Design
    • Survey Instrument Development
    • Key Measures
    • Some Evaluation Results
  • We Are typically Concerned with Two Questions
    • How do we do this (in the best way possible)?
      • Formative research
        • Descriptive methods
        • Exploratory methods
    • Is what we are doing (or did) working?
      • Evaluation Research
        • Strive for cause-effect
        • But settle for strong associations (relationships)
  • Common Non-Experimental Research Designs
    • Cross-Sectional
      • Data are collected at one point in time
    • Longitudinal
      • Data are collected at two or more points in time
        • Repeated Cross-sectional design (Different groups)
        • Fixed-sample panel design (Same group)
    X-------O A O A ------X-------O B ------O C O A ------X-------O A ------O A
  • Common Quasi-Experimental Research Design
    • Non-Equivalent Control group design
      • Experimental and control groups are designated before treatment occurs but not created by random assignment
    Experimental: O A ------X-------O A Control: O B ---------------O B
  • Causal Validity: Research Design
    • Make Non-Experimental Designs rigorous by using…
      • Open-ended questions (when appropriate)
      • Statistical control (Multivariate Statistics)
      • Repeated measures
    Causality Association
    • Experimental
    • Random assignment
    • At least 2 groups
    • Control of variation in IV before measure of DV
    • Non-Experimental
    • Non-random assignment
    • 1 group only
    • Natural variation in IV before measure of DV
    • Quasi-Experimental
    • Non-random assignment
    • At least 2 groups
    • Control variation in IV before measure of DV
  • Survey Instrument Development
    • Basic Considerations
    • Define research objectives
    • Define population of interest
    • Determine data collection method(s)
      • Self-administered Survey (e.g., online, paper & pencil, CASI)
      • Interview Survey (e.g., face-to-face, phone)
    • Determine sample size needed & sampling method
      • Keep data collection costs down
      • Have the statistical power to reject the null when it is indeed false
    • Determine data collection period
  • Survey Instrument Development
    • Basic Survey Structure
    • Not absolute, just a general template
    Screener
    • Survey Questions
    • Funneled
    • Avoid contamination
    Demographics
    • Brief explanation and consent
    • Make sure you’re talking to member from population of interest
    • Collect information you need to address your research objectives
      • Provides data to answer research questions
      • Not answers to research questions
    • Collect data to describe sample and control for possible confounders
    Introduction Awareness
  • Survey Instrument Development
    • Some Survey Question Tips
    • Use appropriate lead-ins for scale changes or new sections
    • Avoid loaded or leading questions
      • Were you more likely to call 911 for emergencies only as a result of seeing this campaign?
    • Response choices should be mutually exclusive and exhaustive (i.e., “Other” option)
      • How often do you go hiking on weekends?
        • Always
        • Frequently
        • Sometimes
        • Occasionally
    • Avoid open-ended questions – unless that’s what you’re looking for
  • Survey Instrument Development
    • Some Survey Question Tips (Cont.)
    • Keep response options balanced (Interval measures)
        • Poor
        • Fair
        • Good
        • Very Good
        • Excellent
    • Avoid double/triple barreled questions
      • Are automobile consumers looking for high fuel economy and extended warranties?
    • Avoid unfamiliar and confusing words or jargon in questions
      • Are you in a monogamous relationship?
      • How concerned are you with infertility?
    • Avoid long questions
    • Don’t force answers
  • Key Campaign Awareness Measures
    • Unaided Awareness
      • Name product category and ask for brand recall
        • First mentions (top of mind)
    • Aided Awareness
      • Show campaign elements for recognition
    • Modes of Exposure
      • Where did you see it? What mediums?
    • Exposure Frequency
      • How many times did you see it?
    • Key Message
      • What was the campaign about? What message?
  • Some Key Potential Co-Factor Measures
    • General Demographics
      • Including children, marital status/relationship status, SES questions, etc.
    • Risk Factors
      • Drug use, previous STD diagnosis, sexual orientation, sex venues
    • Sexual Practices
      • Sexually active, number of partners, condom use, type of sex, gender of sex partners, etc.
    • Medical Access
      • Health insurance, regular doctor, etc.
    • Living Situation
      • Who do you live with?
  • Some Key Outcome Measures
    • Primary Behavioral Outcome Measure
      • Based on the primary objective of your campaign…
        • Condom Usage, Safer-sex, Needle Sharing, Drug Use, STD Testing, etc.
      • Can opt to measure intention as a fall back
    • Knowledge
      • About STD/HIV transmission, detection, treatment, most impacted groups, curability, awareness of, etc.
    • Perceived Vulnerability
      • Measures likelihood perception of respondent being impacted by disease/infection
    • Self – Efficacy
      • Typically a scale to measure the perceived capability of respondent that he/she can engage in desired behavior
  • Beyond the Survey: Process Measures
    • Web Site Visits
      • Google Analytics, individual page views for key pages with timeline
    • Hotline Calls
      • Number of calls regarding campaign topic/message with timeline
    • Clinic Visits & STD/HIV Testing Volume
      • Select clinics with a steady high volume of testing/visits
      • Number of tests/visits with timeline
  • Evaluation of Check Yourself: A Social Marketing Campaign for MSM
  • Campaign Awareness (Endline=306) General Syphilis Reference *May not equal to 100%, multiple mentions allowed. General HIV/AIDS Reference Blair Underwood – Man Up Safer Sex/Protection/Use Condom General Testing Reference General Crystal Meth Reference 88 Check Yourself/Ad mention General STD Reference Other (Less than 2% mention) Stop the Sores
    • Only 12% of sample was unaware of the campaign
  • Appeal, Motivation, and Relevance of Campaign Ads
  • Average Number of Syphilis Tests Performed Per Month Among MSM at LAGLC Clinics Jan 06 Jul 06 Jan 07 Jul 07 Jan 08 Jul 08 Jan 09 Jul 09 Check Yourself Campaign
  • Evaluation of I Know: A Social Marketing Campaign for Young Women of Color
  • Campaign Awareness (n=599) General HIV/AIDS Reference BET – Rap It Up HPV/Cervical Cancer/Gardasil Ref. Safer Sex/Protection/Use Condom General Advertising Reference Blair Underwood – Man Up I Know/Don’t Think Know Get Tested/Get Checked Other (Less than 2% mention) General STD Reference 63
  • Mode of Exposure to Campaign Among Those Aware (n=599) Billboards Television Posters Bus Signage Radio Other (Less than 2% mentions) Magazine *Multiple mentions allowed – May not add to 100%.
  • Results - STS Campaign Exposure and Coverage Billboard in bus Bathhouses Magazine Poster Outdoor Billboard Outreach Event Phil the Sore Other Sources of campaign exposure: %
    • Among those aware of campaign:
      • 13% had seen cable TV commercial
      • Average exposure to campaign 8 times; (median = 2 times)
    Television Website / internet * Not equal to 100%, multiple mentions allowed.
  • Knowledge – Heard of Chlamydia? By Baseline & Endline (Unaware vs. Aware of Campaign) Baseline (n=301) Endline Aware (n=375) Endline Unaware (n=224) Endline Total (n=599) Desired Answer = Yes p<.001 p<.001
  • CT & GC Testing In Last 6 Months By Campaign Awareness (Endline = 599) Aware (n=375) Not Aware (n=224) * Chi-Square test significant at p=0.011
  • Predictors of CT & GC Testing In Last 6 Months (n=591) 95% CI Variable Sig. OR Lower Upper Campaign Awareness (Not aware vs. Aware) .041 1.56 1.02 2.38 Language (Spanish vs. English) .001 3.00 1.56 5.75 Age ns 1.05 0.96 1.16 Relationship (No vs. Yes) .015 1.65 1.10 2.48 Children (No vs. Yes) .010 2.04 1.18 3.51 Ethnicity (Black vs. Latino) ns 0.90 0.77 1.05 Medical Insurance (No vs. Yes) ns 0.89 0.57 1.38 Previous STD Diagnosis (No vs. Yes) .000 3.83 1.85 7.94 None REF ---- ---- ---- One .000 12.87 4.45 37.18 Multiple .000 21.54 7.47 62.09 Some High School or Less REF ---- ---- ---- Completed High School ns 1.33 0.70 2.52 Some College/Trade .042 2.00 1.02 3.91 College Graduate + ns 1.95 0.7 4.95 Number of Partners Level of Education
  • Home Test Kit Orders by Week from June 8, 2009 through February 9, 2010 Week Number of Orders Ad Campaign 71% of total orders Average Weekly = 153 Average Weekly = 30 PR
  • donthinkknow.org Visits by Week from June 8, 2009 through February 9, 2010 Week Number of Orders Ad Campaign 58% of total visits Average Weekly = 1,126 Average Weekly = 398 PR awareness measure
  • *How did you hear about DTK? (n=576) Family or friends Online Ad TV Commercial Poster Magazine Article/ Marie Claire No Response Bus Ad – Inside Bus Browsing online/ Search/ Google Other (Less than 2% mentions) Info. Card Bus Ad – Outside Bus *Multiple mentions allowed – May not add to 100%, from dontthinkknow.org web site users. School/ Teacher/ Class Presentation
  • Contact Info: Jorge A. Montoya, Ph.D. (p)213.864.8959 (f) 866.614.6075 [email_address] Sponsor: www.sentientresearch.net
  • We Are Concerned with Three Types of Validity
    • Measurement Validity
      • Exists when a measure measures what we intend it to measure
    • Generalizability (External Validity)
      • Exists when the results and conclusions from a sample can be generalized to the population it was drawn from
    • Causal Validity (Internal Validity)
      • Exists when the conclusion that A leads to or results in B is correct A  B
  • Measurement Validity: (Operationalization Process)
    • Construct (Latent variable)
    • HIV Risk Taking Behavior
    • Variables (Manifest Variables)
    • Drug Use
    • Sexual Behavior
    • Questions (Indicators)
    • How many different people have used a needle before you in the last month?
    • How often, in the last month, have you cleaned needles before re-using them?
    • How many people, including clients, have you had sex with in the last month?
    • How often have you used condoms when you had sex with casual partners?
  • 4 Levels of Measurement
    • Nominal – Categorical (represent a quality), values have no mathematical interpretation, just codes
    • Ordinal – For greater than and less than distinctions, values represent a ranking by distance between them is not equal
    • Interval – Fixed values with relatively equal distance between them (e.g., Likert scale), but no absolute zero point (-, +)
    • Ratio – Fixed values with equal distance and an absolute zero value so that ratios can be calculated (-,+, x, ÷)
      • Levels are progressive so that they are mathematically more complex than the previous
        • Each level has all the benefits of the former plus some additional ones
  • Properties of Measurement Levels: Permissible Statistics Increase With Each Level A is equal to (not equal to) B = (≠) √ √ √ √ A is greater than (less than) B > (<) √ √ √ A is three more than (less than) B + (-) √ √ A is twice (half) as large as B x ( ÷) √ Appropriate math operations Examples of comparison statements Relevant Level of Measurement Nominal Ordinal Interval Ratio
  • Evaluation of Stop the Sores : A Social Marketing Campaign for MSM
  • Results – Campaign Awareness (1) Safer Sex Reference * Not equal to 100%, multiple mentions allowed. Syphilis Reference Get Tested HIV Stops with Me HIV/AIDS Reference Know Your Partner Healthy Penis 71% Stop the Sore/Phil the Sore AIDS Walk Smoking Crystal Meth Other STD/General reference HIV Testing 3
    • Syphilis Testing In Last 6 Months & Campaign Awareness
    Results – Syphilis Testing (1) Aware (n=211) Not Aware (n=86) * Chi-Square test significant at p=0.011
    • Multivariate analysis (n=277)
    • Predictors of syphilis test in the last 6 months
    Results – Syphilis Testing (2) * (includes both aided and unaided awareness) p=0.033 ** CSV visits included due to testing availability at these venues. Ethnicity/ Race HIV Status 0.91 1.43 1.14 Internet Partners 0.89 1.17 1.02 ** CSV Visits 0.97 1.03 1.00 Age 0.14 3.20 .661 Refused 0.01 0.61 .073 Unsure 0.83 3.19 1.63 Positive --- --- REF Negative 0.51 1.71 0.93 Latino/Hispanic --- --- REF White 0.23 2.14 0.70 Asian/Pacific Islander 0.51 2.07 1.03 African-American 1.05 3.20 1.83 * Campaign Awareness 95% CI Lower Upper OR Variable