Formative research typically involves Qualitative methods : focus groups, one-on-one interviews, participant observation, etc. Qualitative research in terms of focus groups will be covered in more detail by VIVIAN MARLES. Formative research can also include quantitative methods `Market Segmentation (i.e., Hierarchical Cluster Analysis) `Trade-off Analysis (i.e., Choice-Based Conjoint Analysis) I will focus on Quantitative Evaluation Research .
Cross-Sectional One Shot Case Study Longitudinal Designs… Repeated Cross-Sectional – Usually manifested as trend studies (e.g., global changes in public attitudes such as racial tolerance, abortion, gun control; political polling such as favorability ratings of the president). Fixed-Sample Panel – This design follows the same individuals over time so that it allows us to observe changes in the DV in each individual who acts is his/her own control. Considered better than Repeated Cross-Sectional for testing causal hypotheses. Like Before-and-After or One-Group Pretest Posttest Design. Cohort study : Where you select those with exposure and those without exposure and follow forward (or backwards in time) for the development/occurrence of disease/outcome. A cohort is a group of people who share a common characteristic or experience within a defined period (e.g., are born, leave school, lose their job, are exposed to a drug or a vaccine, etc.). Thus a group of people who were born on a day or in a particular period, say 1948, form a birth cohort. The comparison group may be the general population from which the cohort is drawn, or it may be another cohort of persons thought to have had little or no exposure to the substance under investigation, but otherwise similar. Alternatively, subgroups within the cohort may be compared with each other. A research study that compares a particular outcome (such as lung cancer) in groups of individuals who are alike in many ways but differ by a certain characteristic (for example, female nurses who smoke compared with those who do not smoke). The other problem with cohort studies is that they can end up taking a very long time, since the researchers have to wait for the conditions of interest to develop. Researchers are, of course, anxious to have meaningful results as soon as possible, but another disadvantage with long studies is that things tend to change over the course of the study. So research turn to the case-control study. Case-Control Study: case-control, you identify your cases (have outcome) and controls (don't have outcome) and then you can look look back in time or at current exposure, so can have longitudinal component that is retrospective. could also have prospective, eg., measures of biomarkers forward in time. A study that compares two groups of people: those with the disease or condition under study (cases) and a very similar group of people who do not have the disease or condition (controls). Researchers study the medical and lifestyle histories of the people in each group to learn what factors may be associated with the disease or condition. For example, one group may have been exposed to a particular substance that the other was not. Also called retrospective study
-Usually the best alternative to an experimental design in maximizing internal validity is quasi-experimental design. - No consistent definition of quasi-experimental design in the literature, but generally it is one that is like an experimental design with the exception that groups are not randomly created. Statistical Control: Conduct multivariate statistics and control for -Demographics -Other relevant variables to the DV Use of pre-test and post-test so that subjects serve as their own control.
Refusal rates Screen outs Funnelled…Broad general questions in the beginning, warm-ups…more specific questions later..sensitive questions much later too. Example with does syphilis have any effect on HIV? What kind of effect?
Examples of bad questions: Double barreled: How relevant and motivational do you find the advertisements?
WOC Unaided: There are many announcements and ads to keep people informed about issues. Can you recall any ads or public events during the last year that provided information about sexually transmitted diseases? Unaided awareness question from STS: Can you recall any campaigns or advertisements in Los Angeles in the past year that were about sexual health in your community? Aided: Do you recall hearing or seeing one of these ads (SHOW ADS)? Modes of Exposure: Where do you recall seeing or hearing this advertising? Exposure frequency: About how many times did you see/hears this/these ads? _____ # (best guess) Key Message(s): What is/are the main message(s) in this advertising? (DO NOT READ OPTIONS)
A Co-Factor (or a potential confounder) is anything that can have an influence in your outcome besides you campaign. Now this can be a lot of things, so we have to be practical and use a lot of common sense to keep the list of questions at a reasonable length. FOR example, for WOC potential co-factors was covered with about 15 questions.
Process measures that measure implementation vs. process measures that help assess outcome
Aware=269, 30=Unaware There are many announcements and publicity in the media that are directed at the public to keep them informed about issues in their community. Can you recall any ads or public events that provided information about sexual health during the last year here in Los Angeles? If yes, please describe the campaign or key messages as best as you can. (WRITE IN NAME OR DESCRIPTION VERBATIM GIVEN BY RESPONDENT IN ORDER ANSWERED. DO NOT PROMPT OR PROBE
10% is a watermark – there were no billboards in bus, so this is a false listing. We estimate that anything above 10% should be a valid percentage. Multiple answers were allowed for last question. Magazines was most commonly given answer. (people listed frontiers, adelante, and odyssey.) Television + TV commercial were combined for this graph to equal 12 %. Only 7% mentioned the STS commercial unaided. Internet + website were also combined to equal 7%. Only 4% mentioned the STS website unaided. Other: consisted of mostly places, where people would have seen various outreach materials. These places could have included places where the mobile testing van was located. In a few cases, they actually mentioned outreach materials (posters, stress grip, palm cards, comic book.) We can’t say specifically what these outreach materials are, however. Other – unique mentions of less than 3%. Median is only 2 because there were a lot of 0’s for seeing campaign in last 3 months.
Baseline vs. Endline Total significant at p=.005 Baseline vs. Endline Aware significant at p<.001 Baseline vs. Endline Unaware NS Endline Unaware vs. Endline Aware significant at <.001
80% drop in average weekly orders after the campaign ended. 71% of orders occurred during the campaign or 1,885/2,572
65% drop in average weekly visits after marketing campaign Visits during campaign = 13,508 / After = 9,939
This is how you avoid GIGO (Garbage in, Garbage out). This is also where sampling, instrument development, and design come into play. Measurement Validity is the first concern in establishing the validity of research results. Usually remedied by using proven scales (i.e., self-efficacy, etc.) Quality of instrument design will determine measurement validity. Causal Validity considerably much more important in evaluative research. Remedied with a sound research design and clearly defined dependent and independent variables. Generalizability Sampling strategies can remedy generalizability. Two aspects of Generalizability: - Sample Generalizability : Exists when a conclusion based on a sample , or subset, of a larger population holds to for that population - Cross-population generalizability : Exists when findings about one group or population or setting hold true for other groups or populations or settings (Also called External Validity ) External validity is absolutely meaningless if you do not have internal validity.
Additional manifest variables with subsequent questions may include Sexual Addiction, Self-Esteem, Self-Efficacy with regards to Safer Sex Practices, etc. Self-Efficacy: Person’s belief about his/her ability or capacity to succeed in a particular situation (accomplish a task or to deal with the challenges of life). Most often you can use available data, theories or scales that have already defined constructs and have a specific set of questions associated with them, or scales. The use of scales in research will be covered in greater detail by LAUREN FRANK How are manifest variables identified?
You can elaborate on how there is a loss of information as you move from ratio to nominal, so whenever possible you try to collect ratio info, which you can recategorize to become nominal, ordinal, etc. My favorite example of this is when someone gave me age, which was collected in categories, and asked me to compute the mean age!! perfect example of loss of info.
Respondents were asked if they could recall any ads or public events that provided information about sexual health issues since June of last year in Los Angeles. All of these responses are unaided awareness, except that aided questions were asked regarding STS and Healthy Penis. Top bars represent mentions of specific health campaigns. Healthy Penis awareness also went up from the 2002-2003 evaluation from 5% awareness to 21%. 27% unaided awareness is larger even than general HIV/AIDS reference.
Here you can see that syphilis testing was significantly associated with campaign awareness. We are comparing those who were exposed to the campaign versus those who weren’t. You can see that >>>>>>>> And this difference is stat. sig.
Here you can see that campaign awareness is the only factor that is significantly related to syphilis testing, with an odds ratio of 1.83. We did a multivariate analysis. This allows us to control for other variables which might be assoc. with syphilis testing behavior. After controlling for XXXX variables, we found that those exposed to our campaign were nearly twice as likely to be tested for syphilis. Campaign awareness is the only sig. variable.
Evaluating Social Marketing Campaigns Jorge A. Montoya, Ph.D. Los Angeles County Department of Public Health
Google Analytics, individual page views for key pages with timeline
Number of calls regarding campaign topic/message with timeline
Clinic Visits & STD/HIV Testing Volume
Select clinics with a steady high volume of testing/visits
Number of tests/visits with timeline
Evaluation of Check Yourself: A Social Marketing Campaign for MSM
Campaign Awareness (Endline=306) General Syphilis Reference *May not equal to 100%, multiple mentions allowed. General HIV/AIDS Reference Blair Underwood – Man Up Safer Sex/Protection/Use Condom General Testing Reference General Crystal Meth Reference 88 Check Yourself/Ad mention General STD Reference Other (Less than 2% mention) Stop the Sores
Only 12% of sample was unaware of the campaign
Appeal, Motivation, and Relevance of Campaign Ads
Average Number of Syphilis Tests Performed Per Month Among MSM at LAGLC Clinics Jan 06 Jul 06 Jan 07 Jul 07 Jan 08 Jul 08 Jan 09 Jul 09 Check Yourself Campaign
Evaluation of I Know: A Social Marketing Campaign for Young Women of Color
Campaign Awareness (n=599) General HIV/AIDS Reference BET – Rap It Up HPV/Cervical Cancer/Gardasil Ref. Safer Sex/Protection/Use Condom General Advertising Reference Blair Underwood – Man Up I Know/Don’t Think Know Get Tested/Get Checked Other (Less than 2% mention) General STD Reference 63
Mode of Exposure to Campaign Among Those Aware (n=599) Billboards Television Posters Bus Signage Radio Other (Less than 2% mentions) Magazine *Multiple mentions allowed – May not add to 100%.
Results - STS Campaign Exposure and Coverage Billboard in bus Bathhouses Magazine Poster Outdoor Billboard Outreach Event Phil the Sore Other Sources of campaign exposure: %
Among those aware of campaign:
13% had seen cable TV commercial
Average exposure to campaign 8 times; (median = 2 times)
Television Website / internet * Not equal to 100%, multiple mentions allowed.
Knowledge – Heard of Chlamydia? By Baseline & Endline (Unaware vs. Aware of Campaign) Baseline (n=301) Endline Aware (n=375) Endline Unaware (n=224) Endline Total (n=599) Desired Answer = Yes p<.001 p<.001
CT & GC Testing In Last 6 Months By Campaign Awareness (Endline = 599) Aware (n=375) Not Aware (n=224) * Chi-Square test significant at p=0.011
Predictors of CT & GC Testing In Last 6 Months (n=591) 95% CI Variable Sig. OR Lower Upper Campaign Awareness (Not aware vs. Aware) .041 1.56 1.02 2.38 Language (Spanish vs. English) .001 3.00 1.56 5.75 Age ns 1.05 0.96 1.16 Relationship (No vs. Yes) .015 1.65 1.10 2.48 Children (No vs. Yes) .010 2.04 1.18 3.51 Ethnicity (Black vs. Latino) ns 0.90 0.77 1.05 Medical Insurance (No vs. Yes) ns 0.89 0.57 1.38 Previous STD Diagnosis (No vs. Yes) .000 3.83 1.85 7.94 None REF ---- ---- ---- One .000 12.87 4.45 37.18 Multiple .000 21.54 7.47 62.09 Some High School or Less REF ---- ---- ---- Completed High School ns 1.33 0.70 2.52 Some College/Trade .042 2.00 1.02 3.91 College Graduate + ns 1.95 0.7 4.95 Number of Partners Level of Education
Home Test Kit Orders by Week from June 8, 2009 through February 9, 2010 Week Number of Orders Ad Campaign 71% of total orders Average Weekly = 153 Average Weekly = 30 PR
donthinkknow.org Visits by Week from June 8, 2009 through February 9, 2010 Week Number of Orders Ad Campaign 58% of total visits Average Weekly = 1,126 Average Weekly = 398 PR awareness measure
*How did you hear about DTK? (n=576) Family or friends Online Ad TV Commercial Poster Magazine Article/ Marie Claire No Response Bus Ad – Inside Bus Browsing online/ Search/ Google Other (Less than 2% mentions) Info. Card Bus Ad – Outside Bus *Multiple mentions allowed – May not add to 100%, from dontthinkknow.org web site users. School/ Teacher/ Class Presentation
Nominal – Categorical (represent a quality), values have no mathematical interpretation, just codes
Ordinal – For greater than and less than distinctions, values represent a ranking by distance between them is not equal
Interval – Fixed values with relatively equal distance between them (e.g., Likert scale), but no absolute zero point (-, +)
Ratio – Fixed values with equal distance and an absolute zero value so that ratios can be calculated (-,+, x, ÷)
Levels are progressive so that they are mathematically more complex than the previous
Each level has all the benefits of the former plus some additional ones
Properties of Measurement Levels: Permissible Statistics Increase With Each Level A is equal to (not equal to) B = (≠) √ √ √ √ A is greater than (less than) B > (<) √ √ √ A is three more than (less than) B + (-) √ √ A is twice (half) as large as B x ( ÷) √ Appropriate math operations Examples of comparison statements Relevant Level of Measurement Nominal Ordinal Interval Ratio
Evaluation of Stop the Sores : A Social Marketing Campaign for MSM
Results – Campaign Awareness (1) Safer Sex Reference * Not equal to 100%, multiple mentions allowed. Syphilis Reference Get Tested HIV Stops with Me HIV/AIDS Reference Know Your Partner Healthy Penis 71% Stop the Sore/Phil the Sore AIDS Walk Smoking Crystal Meth Other STD/General reference HIV Testing 3