This document discusses monitoring and evaluation of social marketing programs. It begins by defining the key differences between monitoring, which is conducted during program implementation, and evaluation, which is conducted after completion. The document emphasizes that developing a monitoring and evaluation plan is important to answer questions about why the program is being done, what will be measured, how and when measurements will be taken, and how much it will cost. It also stresses the importance of developing a model or logic model of the expected effects to ensure the right outcomes are measured and how change is expected to occur through specific interventions. The document outlines the types of measures as outputs, outcomes, and impacts and provides examples of each. It also discusses methods for measurement and considerations around budget and using

1. Monitoring &
Evaluation
SOCIAL MARKETING
OCTOBER 2010

2. October 2010 2
Not everything that
can be counted
counts,
and not everything
that counts can be
counted.
ALBERT EINSTEIN
If you can not
measure it, you can
not improve it.
LORD KELVIN

3. Monitoring+
Evaluation=k
WHAT ARE WE
TALKING ABOUT?
3

4. Monitoring & evaluation: differences
Monitoring: conducted
as the initiative is implemented.
Evaluation: conducted
at the end of the initiative.
4October 2010

5. Monitoring & evaluation answer Qs
Monitoring & Evaluation Plans
address these questions:
 Why are we doing this?
 What will we measure?
 How will we measure it?
 When will we measure it?
 How much will it cost?
5October 2010

6. Model of Effect
WHY DO WE USE MONIT+EVAL &
FOR WHAT ?
6

7. Model of Effect
Social marketing programs
are rarely as simple as:
‘We’ll tell them, they’ll learn
it, and they’ll change.’
7October 2010

8. So the question is:
Is the program expected to
result in changes in…
1. individual behaviour?
2. social networks/norms?
3. institutions?
8October 2010

9. Model of Effect
Mapping out how the program is
expected to work is critical, because
it ensures that the right outcomes are
measured.
DONER, 2003
9October 2010

10. Model of Effect
Define how we expect change to occur.
Describe how specific product, price,
place and promotional interventions with
priority audience segments will have an
impact on given factors.
10October 2010

11. What shall we
measure?
MEASUREMENT IN A NUTSHELL
11

12. What shall we measure?
There are 3 types of measures:
1. Outputs
2. Outcomes
3. Impacts
12October 2010

13. OUTPUTS

14. Outputs = quantity
Output/Process Measures:
 These are measures that quantify what we’ve done.
 Typically done by counting or recording our activities.
14October 2010

15. Outputs = frequencies/descriptives
Typical types:
 Numbers of materials distributed (how many)
 Reach and frequency (how many were
exposed and how often)
 Media coverage (did they pick up the story?
How much space/time was allocated)
 Impressions (how much would that have cost
per person?)
 Implementation (did we do everything? On
time? On budget? )
 Others?
15October 2010

16. OUTCOMES

17. Outcomes what measure
Relate directly to your goals and
objectives & measure things such as:
 Changes in Behavior
 Changes in Knowledge
 Changes in Belief
 Responses to Campaign Elements
 Campaign Awareness
 Customer Satisfaction Levels
 Partnerships and Contributions Created
 Policy Changes
 Others?
17October 2010

18. IMPACT

19. Impact measures
Impact measures the influence of the initiative on
health outcomes. Typically focus on outcomes like:
 Lives saved
 Diseases prevented
 Injuries avoided
 Water quality improved
 Air quality improved
 Landfill reduced
 Wildlife and habitats protected
 Crimes prevented
 Etc.
19October 2010

20. How &
when
shall we
measure?

21. How & when will we measure?
 Quantitative:
 Surveys through telephone, mail and online, Intercept
interviews, Controlled experiments, Content Analysis,
Records, Databases, National Registries, Survey
Centres
 Qualitative:
 Interviews, Focus Groups, Observational Research,
Content Analysis
21October 2010

22. Questions for you!
Which methods do you use for
Outputs?
Outcomes?
Impact?
WHEN will you measure WHAT using
WHICH METHOD?
22October 2010

23. BUDGET
MONITORING & EVALUATION COSTS
23

24. Monitoring & evaluation Cost
Deciding what we need to do and
what we can do depends on budgets.
So, what can we do?
1st decide what we need to do
2nd determine if we can afford to do that
 Does the benefit justify the cost?
 If so, find the money
 If not, revise your plan
24October 2010

25. LOGIC
EVALUATION MODELLED
25

26. Logic Model - 1
Displays the sequence of actions
that describe what the program is
and will do – how investments link
to results.
26October 2010

27. Logic model core components
1. INPUTS: resources, contributions, investments that go into
the program
2. OUTPUTS: activities, services, events and products that
reach people who participate or who are targeted
3. OUTCOMES: results or changes for individuals, groups,
communities, organizations, communities, or systems
4. ASSUMPTIONS: the beliefs we have about the program, the
people involved, and the context and the way we think
the program will work
5. EXTERNAL FACTORS: the environment in which the program
exists includes a variety of external factors that interact
with and influence the program action.
27October 2010

28. From the highly recommended site:
www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html#more
28October 2010

29. VERB It’s what you do
CASE STUDY
29
http://www.cdc.gov/youthcampaign/

30. Goals
 Increase knowledge and improve attitudes and beliefs
about tweens’ regular participation in physical activity.
 Increase parental and influencer support and
encouragement of tweens’ participation in physical
activity.
 Heighten awareness of options and opportunities for
tween participation in physical activity.
 Facilitate opportunities for tweens to participate in
regular physical activity.
 Increase and maintain the number of tweens who
regularly participate in physical activity.
30October 2010

31. VERB Logic Model
http://www.cdc.gov/youthcampaign/
31October 2010

32. VERB Channels – integrated
communication mix
Mass Media Adv Campaign
 Television commercials (Viacom, Disney, etc)
 Magazines, Newspapers
 Large format print ads
 Radio
 Flyers, Handouts
Internet Promotions
32October 2010

33. VERB Channels - cont’d
Public Relations
 Media relations
 Community involvement
 Policy (lobbying)
 Partnerships
National and Community Outreach
 Schools
 Student Groups
 Recreation Centers
33October 2010

34. Results 2002-2003
 40% awareness goal for first year nearly doubled
 Approx. 75% of tweens surveyed aware
 Tweens understand the message
 Approx. 33% increase in tween activity
http://pediatrics.aappublications.org/cgi/content/full/116/2/e277
34October 2010

35. WEB 2.0 METRICS
FOCUS ON
35

36. Why is it appropriate?
Social media is about people and relationships,
something that is challenging to “measure”.
[…]
With social media, qualitative and quantitative
must be used hand in hand to tell the story. [...]
The reality is that quantitative measures can
only tell a small part of the social media story
and qualitative measures are very important.
FICHTER & WISNIEWSKI, 2009:56
36October 2010

37. Measuring social media success
The Trinity Approach (Kaushik, 2006),
as seen on Fichter (2008):
 Behavioral Data (how much?) > Output
 Outcome (so what?) > Outcome + Impact
 Experience (why?) > Impact
37October 2010

38. Social Media metrics
(cfr. Fichter & Wisniewski, 2009)
 Search Engines
 Focus on specific results
 Technorati/Google Blogsearch/Liquida/OkNo/Wikio
 Delicious
 Twitter/Twitter Search
 Google Trends/Twist
 Social Mention
 Web analytics
 Google Analytics
 Webalizer, Awstats, WP Stats, etc.
38October 2010

39. LET’S GET
DOWN TO IT
HANDS ON SOCIAL MEDIA METRICS
39

40. October 2010 40

41. References
 CDC, Centers for Disease Control and Prevention. Social media tools for consumers and
partners - tools. Retrieved 10/5/2009, 2009, from
http://www.cdc.gov/SocialMedia/Tools/index.html
 Doner, L. (2003). Approaches to evaluating social marketing programs. Social Marketing
Quarterly, 9(3), 18‐26.
 Fichter, D. (2008). Social media metrics: Making the case for making the effort. Online,
32(6), 54-57.
 Fichter, D., & Wisniewski, J. (2009). Social media metrics: Tracking your impact. Online,
33(1), 54-57.
 Kaushik, A. (2006). Trinity: A mindset & strategic approach. Retrieved 10/20/2009, 2009, from
http://www.kaushik.net/avinash/2006/08/trinity-a-mindset-strategic-approach.html
 Program Development and Evaluation Unit,University of Winsconsin. Evaluation logic model. Retrieved
10/20/2009, 2009, from http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html#more
October 2010 41

42. Useful Web 2.0 resources & tools
 Getclicky. Real Time web analytics: http://getclicky.com/
 Google Analytics: http://www.google.com/analytics
 Google Blogsearch: http://blogsearch.google.com/
 Google Trends: http://www.google.com/trends
 Google Documents (Forms): http://docs.google.com
 Social Mention. Real-time social media search and analysis: http://www.socialmention.com/
 Technorati blogsearch: http://technorati.com/search
 Trendistic. See trends in Twitter: http://trendistic.com/
 Twitter Search. See what's happening: http://search.twitter.com/
 Survs.com: http://www.survs.com
 Surveymonkey: http://www.surveymonkey.com
October 2010 42