Planning Campaigns with Evaluation in mind By Jorge Restrepo M.G.A Eureka Facts LLC

Design Clearly define the intended outcomes of a program and ensure that clear measures can be obtained. The key challenges in program evaluation stem from unclear goals or promised outcomes that go beyond the scope of a grant proposal. Educational, language and cultural barriers are necessary considerations in good evaluation design. Know what data you need that the respondent will not answer, and develop alternative methods of collection or inference.

Clearly define the intended outcomes of a program and ensure that clear measures can be obtained. The key challenges in program evaluation stem from unclear goals or promised outcomes that go beyond the scope of a grant proposal.

Educational, language and cultural barriers are necessary considerations in good evaluation design.

Know what data you need that the respondent will not answer, and develop alternative methods of collection or inference.

Design The time crunch – this most often impacts design and instrument testing. Carefully consider timelines for data collection- especially when programs require pre-post measures. Consider other methods to make maximum use of the time required for program deployment. Pre-post evaluation are complex so consider their use before promising a specific method in a campaign proposal. Evaluate outcome not just activity– changes in awareness, knowledge, behavior, are also good indicators of the extent of a campaign’s outcome.

The time crunch – this most often impacts design and instrument testing. Carefully consider timelines for data collection- especially when programs require pre-post measures. Consider other methods to make maximum use of the time required for program deployment.

Pre-post evaluation are complex so consider their use before promising a specific method in a campaign proposal.

Evaluate outcome not just activity– changes in awareness, knowledge, behavior, are also good indicators of the extent of a campaign’s outcome.

Methods - data collection Tips for in-person data collection Protocols and interviewer training are key Supervision and redundant verifications are important Dress-codes and interviewer profile Compensation/ Investment Controls Timing Response rate

Tips for in-person data collection

Protocols and interviewer training are key

Supervision and redundant verifications are important

Dress-codes and interviewer profile

Compensation/ Investment

Controls

Timing

Response rate

Methods - data collection Mail surveys Lists – sample frame Who is your respondent? Sponsor, package, timing, incentive Respondent burden Education/language Coding visibly, invisible coding Flexibility Timing Response rate - examine non-response closely

Mail surveys

Lists – sample frame

Who is your respondent?

Sponsor, package, timing, incentive

Respondent burden

Education/language

Coding visibly, invisible coding

Flexibility

Timing

Response rate - examine non-response closely

data collection Online Who is your respondent? - Your sample frame is everything Spam filters are a bigger concern than non-response Web links create disproportionate sampling Make web links and pop-ups as unobtrusive as possible Respondent fatigue CANSPAM COPA

Online

Who is your respondent? - Your sample frame is everything

Spam filters are a bigger concern than non-response

Web links create disproportionate sampling

Make web links and pop-ups as unobtrusive as possible

Respondent fatigue

CANSPAM

COPA

data collection Telephone Who is your respondent? - Your sample frame is everything RDD sampling – often no longer representative Combine methods No land line households Reach Incentives Cooperation Do not call registry

Telephone

Who is your respondent? - Your sample frame is everything

RDD sampling – often no longer representative

Combine methods

No land line households

Reach

Incentives

Cooperation

Do not call registry

Challenges in Analysis Most distributions – especially in public education efforts are not normal so common centrality statistics (mean, median) are not very meaningful. People often articulate categories (i.e. well, not well) better than continuum scales.

Most distributions – especially in public education efforts are not normal so common centrality statistics (mean, median) are not very meaningful.

People often articulate categories (i.e. well, not well) better than continuum scales.

Analysis Think outside the box – examine findings spatially Stars represent respondents to a campaign

Think outside the box – examine findings spatially

Analysis Think outside the box – examine findings spatially Flat findings examined by other means revealed strong differences by county

Think outside the box – examine findings spatially

Simplify constructs Build your assessment on existing frameworks and models, but create high level aggregates as well This evaluation measures knowledge and action

Build your assessment on existing frameworks and models, but create high level aggregates as well

High and Low Impact Groups A powerful tool is the use of classification trees to identify pockets of a population with high or low impact.

A powerful tool is the use of classification trees to identify pockets of a population with high or low impact.

Compare - Always Mirrored instruments are especially valuable in examining outcomes

Mirrored instruments are especially valuable in examining outcomes

Segment for Success Outcomes are rarely evenly spread out throughout a population

Outcomes are rarely evenly spread out throughout a population

Common cross-tabbing analysis Ethnicity Hispanic Origin Gender Age Education Urban locale Generational cohort Socioeconomic status Life stage, career stage Primary language Geography

Ethnicity

Hispanic Origin

Gender

Age

Education

Urban locale

Generational cohort

Socioeconomic status

Life stage, career stage

Primary language

Geography

Who we are Eureka Facts brings you the smart marketing Information you need to discover opportunities maximize results identify issues make decisions reduce market risks

Eureka Facts brings you the smart marketing Information you need to

discover opportunities

maximize results

identify issues

make decisions

reduce market risks

Smart Evaluation www.EurekaFacts.com EurekaFacts LLC 451 Hungerford Drive Suite 515 Rockville MD 20850 (301) 610-0590