1. PA 685: Strategic Management of
Public Communications
Module 7:
Evaluating Communications Efforts
2. The Purpose of Evaluation
The fourth step of the public relations process is
evaluation—the measurement of
established objectives.
results against
Evaluation is the measurement of results against
objectives; this can enhance future performance and
also establish whether the goals of management by
objective have been met.
Evaluation is well described as the systematic
assessment of a program and its result; it is a means
for practitioners to offer accountability to clients and
to themselves.
3. Objectives: A Prerequisite for
Evaluation
Before any public communication program can be
properly evaluated, it’s important to have a clearly
established set of measurable objectives; these should be
part of the program plan but first some points need to be
reviewed.
Public communications personnel and management should
agree on the criteria that will be used to evaluate success in
attaining objectives.
Don’t wait until the end of a campaign to determine how it will be
evaluated. Evaluating impact starts in the planning stage as you
break down the problem into measurable goals and objectives,
and then again after implementing the program when you
measure the results against the objectives.
4. Objectives: A Prerequisite for
Evaluation
If the objective focuses on knowledge-building
(awareness), measurement techniques must show how
successfully information was communicated to target
audiences.
Attitude-changing or motivational objectives are more
difficult to accomplish. Research should be done before
and after the public communication activity to measure
the percentage of change.
Behavior-related objectives can be measured using
appropriate measures of the desired behavior (e.g.,
increased recycling, reduction in the incidents of texting
while driving).
5. Objectives: A Prerequisite for
Evaluation
Evaluation Questions to Consider
Adequately planned?
Message understood?
How more effective?
All primary and secondary audiences reached?
Organizational objective achieved?
Unforeseen circumstances?
Within budget?
Steps to improve?
6. Current Status of Measurement
and Evaluation
The proportion of public communication budgets
devoted to measurement and evaluation grew over the
past decade to about 5 percent.
On the most basic level, practitioners can measure
message distribution and media placements.
The second level would be measurement of audience
awareness, message comprehension, and retention.
The most advanced level is the measurement of
change in attitudes, opinions, and behaviors.
7. Measurement: Production
One elementary form of evaluation is simply to count
how many news releases, feature stories, photos,
letters-to-the-editor and the like are produced in
given period of time.
This kind of evaluation is supposed to give
management and idea of staff’s productivity and
output.
Some criticize that this evaluation isn’t very
a
meaningful, because it emphasizes quantity instead of
quality. It may be more cost-effective to write fewer
news releases and spend more time on the few that are
newsworthy.
8. Measurement: Production
Another side of production measurement approach is
to specify what the communication staff should
accomplish in obtaining media coverage.
Such evaluation criteria, however, are difficult to use
as measures of success since media gatekeepers make
the decision as to what is published—not the public
communicators.
Management argues that such placement goals
provide incentives to staff and are tangible criteria in
employee performance evaluation.
9. Measurement: Exposure
Why measure media efforts?
Quantify public communication efforts in
meaningful ways
Gauge message frequency; location
Use as benchmark
10. The most widely practiced form of evaluating public
communication efforts is the compilation of press
clippings from print publications and online news
outlets as well as a summary of mentions in broadcast
outlets (radio, television, and online streaming media)
and in social media conversations.
Local agencies may focus on smaller scale exposure
measures by using staff to scan and record local
media.
However, larger state and national agencies will often
hire clipping services to scan large numbers of media
outlets—online and offline.
Measurement: Exposure
11. Measurement: Exposure
Measurement of exposure relies on:
Media impressions.
Hits on the internet
Advertising value equivalency.
Content analysis of news coverage.
Systematic tracking.
Requests for 800 numbers.
Cost per person.
Audience attendance.
12. Media Impressions
In addition to the number of media placements, public
communication staff are interested in knowing how many
people have been exposed to the message.
These numbers are described as media impressions, the
potential audience reached by a periodical or a broadcast
program.
National news can generate millions of impressions by
simple multiplication of each placement in specific media.
Media impressions are commonly used to give a rough
estimate of how many people are exposed to a message.
13. Media Impressions
Impressions
Opportunities to See (OTS)
Pass-along rates—Media are often left for others to read instead of being
recycled or thrown out.
The “Opportunity to See” is an estimate of how large the audience might
actually be rather than just the subscriber/impression number.
Newspapers: Industry average is 2.1
Magazines: Vary considerably
Only six out of top 250 magazines in 2012 average less than 2.
Top two: Bride Guide (25.71) and Truckin’ (25.22).
14. Hits on the Internet
The cyberspace version of the media impression is the
number of people reached via an organization’s website or
home page.
Each instance of a person accessing a particular site is
called a hit or visit.
Social media also provide different internet metrics that
can be used to evaluate your efforts.
Facebook and Twitter allow people to like/share and
favorite/retweet messages, respectively.
YouTube also allows people to give a “thumbs up” to a
video.
Comments are another measure of internet success.
15. Advertising Value Equivalency
Another approach is to calculate the value of message
exposure.
This is done by converting stories in the regular news
columns or on the air into equivalent advertising costs.
Some recommend calculating the cost of advertising for the
same amount of space and then multiplying that total three
to six times to reflect a number of research studies that
show that a new story has greater credibility than an
advertisement.
16. AVE is a flawed approach.
No consideration for message quality.
Forces equivalency with advertising.
Multipliers representing media endorsement.
Does not take social media influence into
consideration.
Advertising Value Equivalency
17. Content Analysis of News Coverage
One way to avoid criticism of relying on advertising
value equivalency is to analyze the coverage received
from media outlets.
You can analyze content for a variety of different
characteristics, including but not limited
Type of media outlet.
Tone of news story.
Conveyed company message points.
Feature quotations from company.
Company name in headline.
Visual media featured.
to:
18. Content Analysis of News Coverage
By deciding what news characteristics are
most important for your agency, you can
create a formula that helps determine the
overall impact of each media placement.
This can help steer future decisions on what
media outlets to approach for future news.
19. Requests and 800 numbers
Another measure of media exposure is to compile the
number of requests for more information.
A story in a newspaper or an appearance of a company
spokesperson on a broadcast often provides information
as to where people can get more information about a
subject.
Creating a microsite (a standalone website separate from
your organization’s website) can be used to direct people
for specific information about the effort and has easily
trackable information about web hits and requests.
Requests for materials can also come from phone
requests, such as with 1-800 numbers, and they all show
the effectiveness of a public communication program.
20. Cost Per Person
Another way to evaluate to the message is to determine
the cost of reaching each the audience members. The
technique is commonly used in advertising to place costs
in perspective.
Cost effectiveness, as this technique is known, also is
used in public communications when advertising space is
bought—whether that is a commercial, print
advertisement, radio spot, billboards, or online ads.
Cost-per-thousand (CPM) is calculated by taking the cost
of the purchased and dividing it by the total media
impressions (discussed earlier).
21. Audience Attendance
Counting attendance at events is a relatively simple way
of evaluating the effectiveness of pre-event publicity.
Poor attendance at a meeting or event can indicate
inadequate publicity and promotion.
Another major cause is lack of public interest, even when
people are aware that a meeting or event is taking place.
Low attendance usually results in considerable finger-
pointing; thus an objective evaluation of exactly what
happened—or didn’t happen—is a good policy.
22. Measurement of Audience Awareness
Techniques of measuring audience exposure and accurate
dissemination have been discussed.
A higher level of evaluation is to determine whether the audience
actually became aware of the message and understood it.
The tools of survey research are needed to answer such questions.
Members of the target audience must be asked about the message
and what they remember about it.
Public awareness of what agency sponsors events also is important.
Another way of measuring audience awareness and comprehension
is day-after recall.
Under this method, participants are asked to view a specific TV
program; the next day they are interviewed to learn which message
they remembered .
23. Measurement: Attitudes
Closely related to audience awareness and
understanding of a message are changes in
audiences perception and attitudes.
A major technique to determine such changes is the
baseline study, a measurement of attitudes before,
after, and during a campaign.
It’s also called benchmark study.
26. Communication Audits
Communication activity of an organization should be
evaluated at least once a year to make sure that every
primary and secondary public is receiving appropriate
messages.
A communication audit could include the following:
Determine the key communication areas to be examined.
Collect and evaluate past communications.
Look outward: Ask your stakeholders what worked, what
didn’t, and what media channels they use.
Review what worked for you and what didn’t; see what
new channels exist that you haven’t used before.
Take all of the research in steps 2-4 and put together a
communication plan for the future.
1.
2.
3.
4.
5.
27. Pilot and Split Message Testing
A variation of pre-testing is the pilot test, before going
national with public communication messages and
message points.
Agencies will often test the message and copy points in
selected cities to learn how the media accept the
message and how the public reacts.
The split message approach is common in direct mail
campaigns. This approach has one message design sent
to one group of stakeholders and another design sent to
the remaining stakeholders.
The idea is to split the message and see which is more
effective.
28. Marketing Collateral Readership
Writers of marketing collateral (newsletters, brochures,
Facebook updates, blog posts, flyers, etc.) should
evaluate readership annually.
Such an evaluation can help ascertain:
Reader perception of the material.
The degree to which information is accurate,
truthful.
Whether information shared is helpful; determine
what else is needed.
The credibility of the publication and agency.
The extent to which the materials are meeting
organizational objectives.
29. Analysis of Marketing Collateral
Your communication materials
in a number of ways:
can be evaluated
1.
2.
3.
Content analysis
Readership interest surveys
Article recall
30. Content Analysis
As described on an earlier slide, a content analysis helps you
decide which characteristics of the communication piece are
most important and measure how prevalent they are.
In this situation, you would want to assess various pieces of
marketing collateral by taking representative samples of past
pieces (e.g., issues of a newsletter, tweets made on Twitter,
pictures shared on your website) and determine what
characteristics of each communication piece you wish to
measure.
Systematic analysis will show what percentage of the
publication is devoted to each category.
Combining your content analysis with information from your
stakeholders can show you areas where you are succeeding
with your efforts and identify areas for improvement.
31. Readership Interest Surveys
The purpose of these surveys is to get feedback about the
types of material stakeholders are most interested in
reading.
The reading can occur in newsletters, your website, your
social media updates.
You are simply wanting to identify what types of messages
and topics your stakeholders want to read.
The most common survey method is simply to provide a long
list of topics and have people rate their level of interest along
a scale, where 1 is “Not at all Interested” and 5 is “Very
Interested.”
This way, you can compare how your stakeholders view
various topics so that you can better appeal to their
interests.
32. Article Recall
When you use newsletters, magazines, annual
reports, or other lengthy types of messages, you may
decide to conduct an “Article Recall” test.
This measurement occurs when communication staff
ask a sampling of stakeholders what they have read
and remember from the latest issue of the
publication, latest blog update, or latest YouTube
channel.
What people actually remember may be very
different from what they say is important to them.
Both pieces of information should be used to help
decide what future message strategies should be.
