The document discusses evaluation in public relations practice. It defines evaluation and outlines some key principles of evaluation according to Patton, including that evaluation is research-based, looks at both inputs and outcomes, is dependent on user needs and context, can examine both short and long-term impacts, allows for comparisons, and takes a multifaceted approach. Several case studies are presented, including one on a campaign to increase dog adoptions by teaching shelter dogs to drive miniature cars. Evaluation of the campaign found it increased awareness of the organization's partnership and positively changed perceptions of shelter dog behavior.

1. PUBLIC RELATIONS PRACTICE
2014
Week 6
!
DR KANE HOPKINS

3. Evaluation defined
The practice of evaluation involves the systematic
collection of information about the activities,
characteristics, and outcomes of programs, personnel,
and products for use by specific people to reduce
uncertainties, improve effectiveness, and make
decisions with regard to what those programs,
personnel, or products are doing and affecting
!
Patton (1982, p. 15)

4. 1. Evaluation is research
2. Evaluation looks both ways
3. Evaluation is user- and situation-dependent
4. Evaluation is short term
5. Evaluation is long term
6. Evaluation is comparative
7. Evaluation is multi-faceted
Principles of evaluation

5. 1. Evaluation is research
2. Evaluation looks both ways
3. Evaluation is user- and situation-dependent
4. Evaluation is short term
5. Evaluation is long term
6. Evaluation is comparative
7. Evaluation is multi-faceted
Poindexter says:!
if you want more
info the title is a
link to the article
Principles of evaluation

6. Evaluation: the reality
• Practitioners viewed evaluation very narrowly and
lack confidence in promoting evaluation methods to
employers and clients
• Most relied on output measurement of media
coverage. Few undertook research or pre-testing
when preparing campaigns
• Evaluation was not undertaken because of a lack of
time, budget and knowledge of methods
• Evaluation was feared because it could challenge
the logic of practitioners’ advice and activities

7. Evaluation: the reality
• 75% claimed to undertake some form of evaluation,
with 62% of programmes being evaluated, mainly by
the project manager
• Little was spent on evaluation, with 74% indicating
that the total proportion of budget was zero to 5%
• The picture that emerged was of the practitioner as a
‘doer’ rather than an advisor

8. The Barcelona Declaration of
Research Principles
• A declaration of standards and practices to guide
measurement and evaluation of PR was first adopted at
the 2nd European Summit on Measurement in
Barcelona, Spain, in June 2010
• Organised by the International Association for
Measurement and Evaluation of Communication and
the Institute for Public Relations
• The Barcelona Principles will be the subject of ongoing
dialogue as industry professionals seek to apply the
standards to the public relations practice

9. Seven measurement principles
1. Goal setting and measurement are fundamental aspects of any
public relations programme
2. Measuring the effect on outcomes is preferred to measuring outputs
3. The effect of public relations programs on business results can and
should be measured where possible
4. Media measurement requires quantity and quality. Overall clip
counts and general impressions are meaningless
5. Advertising Value Equivalents (AVEs) do not portray the value of
public relations and do not inform future activity
6. Social media came in should be measured, although there is no
single ‘metric’
7. Transparency and replicability are paramount to sound
measurement

10. Goal setting & measurement
• Goal-setting and measurement are fundamental
aspects of any public relations program
• Goals should be as quantitative as possible and
address who, what, when and how much the PR
program is intended to affect
• Measurement should take a holistic approach,
including representative traditional and social media;
changes in awareness among key stakeholders,
comprehension, attitude, and behaviour as
applicable; and effect on business results

11. Measuring the effect on outcomes
• Outcomes include:
– shifts in awareness
– comprehension
– attitude and behaviour related to purchase
– donations
– brand equity
– corporate reputation
– employee engagement
– investment decisions,
– other shifts in stakeholders regarding a company, NGO,
government or entity, as well as the stakeholder’s own
beliefs and behaviours

12. Measuring the effect on outcomes
• Practices for measuring the effect on outcomes should
be tailored to the business objectives of the PR activities.
Quantitative measures such as benchmark and tracking
surveys, are often preferable. However, qualitative
methods can be well suited or used to supplement
quantitative measures
• Standard best practices in survey research including
sample design, question wording and order, and
statistical analysis should be applied in total
transparency

13. Measuring business results
• To measure business results from consumer or
brand marketing, models that determine the effects
of the quantity and quality of PR outputs on sales or
other business metrics, while accounting for other
variables, are a preferred choice

14. Measuring quantity & quality
• Overall clip counts and general impressions are usually
meaningless. Instead, media measurement, whether in
traditional or online channels, should account for:
– Impressions among the stakeholder or audience
– Quality of the media coverage including:
• Tone
• Credibility and Relevance of the Medium to the Stakeholder or
Audience
• Message Delivery
• Inclusion of a 3rd party or company spokesperson
• Prominence as Relevant to the Medium
– Quality can be negative, positive, or neutral

15. AVEs are not the value of PR
• AVEs measure the cost of media space and are rejected as a
concept to value public relations
• Where a comparison has to be made between the cost of
space from earned versus paid media, validated metrics
should be used, stated for what they are, and reflect:
– negotiated advertising rates relevant to the client, as available
– quality of the coverage, including negative results
– physical space of the coverage, and the portion of the
coverage that is relevant
• Multipliers intended to reflect a greater media cost for earned
versus paid media should never be applied unless proven to
exist in the specific case

16. Measuring social media
• Social media measurement is a discipline, not a tool; but there
is no “single metric”
• Organisations need clearly defined goals and outcomes for
social media
• Media content analysis should be supplemented by web and
search analytics, sales and CRM data, survey data and other
methods
• Evaluating quality and quantity is critical, just as it is with
conventional media
• Measurement must focus on “conversation” and
“communities” not just “coverage”
• Understanding reach and influence is important, but existing
sources are not accessible, transparent or consistent enough
to be reliable; experimentation and testing are key to success

17. Transparency & replicability
• PR measurement should be done in a manner that is
transparent and replicable for all steps in the process,
including specifying:
– Media Measurement:
• Source of the content (print, broadcast, internet, consumer
generated media) along with criteria used for collection
• Analysis methodology – for example, whether human or automated,
tone scale, reach to target, content analysis parameters
– Surveys:
• Methodology – sampling frame and size, margin of error, probability
or non-probability
• Questions – all should be released as asked (wording and order)
• Statistical methodology – how specific metrics are calculated

18. Use of output evaluation methods
in campaigns
Evalua'on
category
Evalua'on
methods %
use
Output Distribu'on
sta's'cs 28.81
Media
monitoring 64.41
Media
content
analysis 45.53
Audience
analysis 4.24
Sta's'cal
analysis 6.78
Response
rates 66.10
Coding
material 8.47
AItude
and
image
studies 4.24
Communica'on
audit 1.69
Organisa'onal
culture
study 0.85
Analysis
of
complaints 2.54

19. Use of output evaluation methods
in campaigns
Evalua'on
category
Evalua'on
methods %
use
Output Distribu'on
sta's'cs 28.81
Media
monitoring 64.41
Media
content
analysis 45.53
Audience
analysis 4.24
Sta's'cal
analysis 6.78
Response
rates 66.10
Coding
material 8.47
AItude
and
image
studies 4.24
Communica'on
audit 1.69
Organisa'onal
culture
study 0.85
Analysis
of
complaints 2.54

20. Use of output evaluation methods
in campaigns
Evalua'on
category
Evalua'on
methods %
use
Output Distribu'on
sta's'cs 28.81
Media
monitoring 64.41
Media
content
analysis 45.53
Audience
analysis 4.24
Sta's'cal
analysis 6.78
Response
rates 66.10
Coding
material 8.47
AItude
and
image
studies 4.24
Communica'on
audit 1.69
Organisa'onal
culture
study 0.85
Analysis
of
complaints 2.54
Poindexter says:!
That’s a link to
more info on the
methods

21. Use of output & outcome
evaluation methods in campaigns
Evalua'on
category
Evalua'on
methods %
use
Outcome Focus
group
discussions 4.24
In-­‐depth
interviews
surveys 4.24
Pre
and
post
tests 3.39
Unobtrusive
data
collec'on 17.80
Quasi-­‐experimental
study 0.00
Ac'vity
outcome 29.66

22. Use of output & outcome
evaluation methods in campaignsEvalua'on
category
Evalua'on
methods %
use
Outcome Focus
group
discussions 4.24
In-­‐depth
interviews
surveys 4.24
Pre
and
post
tests 3.39
Unobtrusive
data
collec'on 17.80
Quasi-­‐experimental
study 0.00
Ac'vity
outcome 29.66

23. Evaluation as a new beginning
• Measuring the effectiveness of a public relations effort
frequently provides new direction and emphasis for an
on-going programme.
• Even if a campaign is cancelled, the lessons learned
concerning its effectiveness will be useful in future
activities.
• Knowledge gained through careful evaluation is an
important payoff to prevent future mistakes.
• Careful measurement of successful efforts will help
reproduce positive elements in future programmes.

24. Case study: Driving Dogs

25. Case study: Driving Dogs
• Issue
– In New Zealand, the number of dogs in SPCA
shelters far outweighs public interest in their
adoption
• Research
– People think because a dog is in a shelter, it must be
damaged goods
– A media coverage audit revealed that exposure of
bad dog behaviour in the news was up 44%
– It also suggested that news media would be key in
changing perceptions

26. Case study: Driving Dogs
• Objectives
1. Change people’s negative perceptions of SPCA
shelter dog behaviour, in order to drive interest in
SPCA adoptions
2. Increase awareness of MINI’s association with
SPCA by 10%
3. Double engagement with the MINI brand on
Facebook: from 5% to 10%

27. Case study: Driving Dogs
• Audience: Potential dog adoptees nationwide
– People who had ever considered, or could consider
adopting a dog or encouraging someone they know to
do so
• Messages
– “Dogs this smart deserve a home”
• The SPCA in association with MINI is proving that you
can teach a rescue dog new tricks
• SPCA wants to showcase just how gifted, intelligent and
talented SPCA animals really are
• MINI is a long term supporter of the SPCA and wanted
to do something significant to drive adoptions

28. Case study: Driving Dogs
• Strategy
– To deliver an extreme display of SPCA’s dogs’ intelligence and
trainability to PROVE they didn’t have behavioural issues
– Prove that SPCA dogs are smart by teaching them to drive a mini
– They used news and social media influencers to take the nation
on a journey from disbelief to belief in three steps
1. Create disbelief – use a credible platform to engage NZ with the idea
that dogs could drive MINIs
2. Engage the nation – drive engagement with this idea and encourage
people to get behind the dogs through news, social and paid media
3. Prove it to the nation – create one mass, undeniable, media moment to
prove that the idea is actually possible – Dogs this smart deserve a
home

29. Case study: Driving Dogs
• Implementation
– New Zealand’s #1 animal trainer and selected three
SPCA shelter dogs to undergo a world first eight week
“doggy driving” training course in a modified MINI
• Create disbelief: They convinced Campbell Live to be our
editorial partner on the project. They would be given the
“reveal” exclusive and therefore be given access to the
dogs during the training process to capture the back
story
• Engage the nation: Facebook, YouTube channel, A MINI
USB stick on a SPCA branded dog collar was sent to15
New Zealand media “super influencers”

30. • Prove it to the nation: staged a demonstration of
SPCA dogs driving a MINI live on Campbell Live
– ensured those who had already engaged with us via
social media or had run the teaser story received a
copy of the video so as many people as possible
could see it with their own eyes and further share the
content
– This included online news sites, journalists, Twitter
followers and Facebook fans.
– Also shared it through our YouTube platform
Case study: Driving Dogs

31. Results
• Objective: Change people’s negative perceptions of
shelter dog behaviour, in order to drive interest in
SPCA adoptions.
– Perceptions were changed
• People were twice as likely to overcome perception
barriers once they’d seen the dogs driving

32. Evaluation: Survey
• To test the efficacy of the key messages and to
document whether or not they had achieved our
objectives:
– Driving Dogs drove a 40% increase in awareness of
MINI’s association with SPCA
– There was a 113% increase in the number of people
with a positive perception of shelter dog behaviour
– Once people had seen the campaign, they were twice
as likely to have overcome their perception barrier

33. Evaluation: Media Analysis
• Campbell Live ran the story not twice as originally
planned but four times to their audience of 304,000
people nationwide
• A total of 483 media articles
• 213 minutes of broadcast coverage ran altogether on the
story
• Broadcast coverage was syndicated to 1008 media
outlets globally
• Reach via media articles to an estimated audience of
230,540,469 worldwide

34. CONTROLLED
COMMUNICATION

35. What is that exactly?
Controlled communication is any form of
media that allows the communicator to
control the content, timing, frequency, format
and placement of the communication

36. Controlled media can be...
• Videos
• Posters
• Websites
• Personally addressed letters
• Brochures + booklets
• Advertising
• Fact sheets
• Exhibitions and displays
• Newsletters – external + internal audiences

39. Posters
• Fast and effective communication
• The difference is size and cost
• Advances in computers make them easy and cheap
to produce

46. Brochures and booklets
Purpose:
• Arouse interest
• Answer questions
• Direct to further information
• Sit under Public Information or Publicity models of PR
• Usually function as support pieces in a larger campaign
• Display rack
• Press kit
• Direct mail

51. Video
• A good way to repurpose content, such as media
releases, events etc
• Creating a video news release (VNR)
• Political campaigns
• Product launches
• Building trust and credibility

52. Video
• A good way to repurpose content, such as media
releases, events etc
• Creating a video news release (VNR)
• Political campaigns
• Product launches
• Building trust and credibility

53. The Galifianakis bump

55. Fact Sheet
– A quick reference tool for reporters
– Summarises the key points about and event,
product, issue or organisation
– Don’t include too much information
– Supports other material

58. Email
• Low cost
• Immediacy
• Ease of distribution
• Target messages
• Easily allow for feedback

63. Exhibitions and Displays
• Rely primarily on visual messages.
• Use many of the tactics talked about here
• Valuable for many types of industries
• Tourism
• Electronic
• Motor vehicles

67. TO READ THE READINGS
THIS WEEK. I MEAN,
WOULD IT KILL YOU?