Jim Macnamara presents latest international developments in standards and models for evaluation as part of the PR Measurement Summit which was held last October 12th-13th in the Address Dubai Marina, Dubai, UAE.

1. Latest International
Developments In Standards And
Models For Evaluation
Prof.Jim Macnamara
PhD, FAMI, CPM, FAMEC, FPRIA
Associate Dean, University of Technology,
Sydney - Link

2. Latest International Developments
in Standards and Models for Evaluation
Breakthroughs at Last!
Professor Jim Macnamara PhD, FAMEC, FAMI, CPM, FPRIA
Professor of Public Communication, University of Technology Sydney
Visiting Professor, London School of Economics and Political Science

3. Measurement and Evaluation – The Answer!

4. This Presentation
■ Review some fundamentals
―Key principles
―The basis of evaluation frameworks and models – where do they come from?
■ New frameworks for evaluation
―EU evaluation framework
―US Task Force on Standards
―UK Cabinet Office evaluation framework
―AMEC Integrated Evaluation Framework
■ A taxonomy of evaluation – mapping the stages, steps, metrics, and methods
―A tool to apply evaluation
This presentation

5. • Goal setting and measurement are fundamental for
communication and PR
• Measuring communication outcomes is recommended
versus only measuring outputs
• The effect on organizational performance can and should be
measured where possible
• Measurement and evaluation require both qualitative and
quantitative methods
• AVEs are not the value of communications
• Social media can and should be measured
• Measurement and evaluation should be transparent,
consistent and valid

6. MEASUREMENT
• The taking of measures
• The collection and analysis of data in relation to
a particular object, process, or condition
• SO WHAT?
EVALUATION
• “Making a judgement” about the value or
significance of something (Oxford and Merriam-
Webster dictionaries)
• Evaluation is the systematic acquisition and
assessment of information to provide useful
feedback about some object” (Trochim, 2006,
para, 3)
• “The systematic application of research
procedures to understand the conceptualization,
design, implementation, and utility of
interventions (Valente, 2001, p. 106)
• Assessing results against objectives

7. SMART objectives
■ Contain numbers, percentages, dates
■ Design measurement in the planning stage
■ Be realistic
■ Linked to organisational objectives and goals
■ Achieved by a specific date

8. Three Types of Evaluation
Formative
Process
Summative
• Baseline (e.g., what is the
current level of awareness)
• Precedents (have other similar
programs worked in the past?)
• Pre-testing (will what we
propose work?)
• CEA (is it the most cost-
effective approach?)
• Insights / learning (what has
been learned that can
inform future strategy,
policy, etc?)
• Impact (what has happened
as a result of the
communication?)
• Outcomes – intermediate
and long-term (what
attitude or behaviour change
has occurred?)
• Monitoring and tracking
outputs (e.g., are we reaching
the audience?)
• Monitoring and tracking
immediate outcomes (e.g., what
response are we getting?

9. Why Stages?
Six stages of communication (W. J. McGuire, 1968, 1969)
ComprehensionPresentation Change/actionRetentionAttention Acceptance

10. Stages of communication
The AIDA model of advertising (Strong, 1925)
AWARENESS
INTEREST
DESIRE
ACTION

11. Stages of communication
■ Exposure
■ Attention
■ Interest and/or liking
■ Comprehension
■ Cognition, particularly cognitive elaboration (thinking about the message)
■ Acquiring skills or knowledge required to deal with the issue (if necessary)
■ Attitude change, particularly to agreement (what McGuire calls “yielding”)
■ Storing information in memory (retention)
■ Retrieving information (i.e., recall)
■ Deciding to act in accordance with information (intention)
■ Action/behaviour
■ Cognitive integration of behaviour (e.g., reinforcement)
■ Encouraging others to behave similarly – what McGuire called “proselytizing” and what is commonly regarded in modern marketing as
‘advocacy’ (McGuire, 1999, 2001)

12. Frameworks/models of evaluation
The PII model of evaluation (Cutlip, Center, & Broom, 1985)

13. Frameworks/models of evaluation
Macro model of evaluation (Macnamara, 1992)

14. Frameworks/models of evaluation
Pyramid model of PR research (Macnamara, 2002, 2005)
OUTCOMES
(Functional &
organisational evaluation)
OUTPUTS
(Process & program
evaluation)
INPUTS
(Formative
research)
Number who . change ..…..……………………………………………….. Quantitative surveys (large scale structured)
behaviour …...…………………………………………….. Sales; Voting results; Adoption rates; Observation
Number who ….. change
attitudes ……….…………………………………………… Focus groups; Surveys (targeted) (eg Customer, Employee
Number who understand messages..……………………………… Focus groups; Interviews; Complaint decline; Experiments
Number who retain messages …….………………………………… Interviews; Focus groups; Mini-surveys; Experiments
Number who consider messages ……....….. ………………………… Response mechanisms (1800, coupons); Inquiries
Number & type of messages reaching target audience ……………………… Media Content Analysis; Communication Audits
Number of messages in the media ……………..….`…………………… Media Monitoring (clippings, tapes, transcripts)
Number who received messages ………………...……………………….. Circulations; Event attendances; Web visits & downloads
Number of messages sent …………………………………..……………….. Distribution statistics; Web pages posted
Quality of message presentation …………………………………………………. Expert analysis; Peer review; Feedback; Awards
Appropriateness of message content …………………………………….………….. Feedback; Readability tests (eg. Fog, Flesch); Pre-testing
Appropriateness of the medium selected ………………………………………….…….. Case studies; Feedback; Interviews; Pre-testing (eg. PDFs)
How does target audience prefer to receive information? …………….…………..……… Academic papers; Feedback; Interviews; Focus groups
What does target audience know, think, feel? What do they need/want? ………………… Observations; Secondary data; Advisory groups; Chat rooms
& online forums; Databases (eg. Customer complaints)
or Shareholder Satisfaction); Reputation studies
Measurement Methodologies:
(formal & informal)
Key Steps/Stages in Communication:
OUT-TAKES
(Proposed by some as a 4th stage)

15. Frameworks/models of evaluation
■ PR effectiveness yardstick (Lindenmann, 1993)
Measuring:
Behaviour change
Attitude change
Opinion change
ADVANCED
Measuring:
Retention
Comprehension
Awareness
Reception
INTERMEDIATE
Measuring:
Target audiences
Impressions
Media placements
OUTPUT
LEVEL #3
LEVEL #2
LEVEL #1
PUBLICRLEATIONSEFFECTIVENESSYARDSTICK
OUTCOMES
OUTGROWTHS
OUTPUTS

16. Frameworks/models of evaluation
Noble & Watson’s united model (Noble & Watson, 1999)
INPUT STAGE
Planning & Preparation
OUTPUT STAGE
Messages & Targets
IMPACT STAGE
Awareness & Information
EFFECT STAGE
Motivation & Behaviour
Tactical feedback
Management feedback

17. Frameworks/models of evaluation
Communication controlling (DPRG/GPRA, 2000; DPRG/IPV, 2009)

18. Frameworks/models of evaluation
■ The European Commission’s (EC) Better Regulation Guidelines uses the terms:
―Inputs
―Outputs
―Results
―Impact
■ European Commission model (EC, 2015)

19. 10 political priorities
Communication
objectives
Activities Relevance Output Outtake Outcome
Recall indicators
measuring the extent
to which the
communication
activity is likely to
have satisfied,
captured the attention
of audiences, or raised
knowledge and
awareness focussing
on the audience
directly reached
Political match
Of the communication
activities undertaken
with the 10 political
priorities
• Organising events
• Providing
information
• Working with
information
networks and desks
Reach indicators
measuring the extent
to which extent the
communication
activity is likely to
have reached the right
target audience
quantitatively and
qualitatively
Engagement Trust
indicators measuring
the extent to which
the communication
activity led to either a
discernible action
being taken or the
desired change in the
target audience’s
perception
Inputs Results
Did the activity match the
10 priorities?
Did the activity reach the
target?
Was the message received?
Did the activity meet the
expectations of the target
audience?
Did the action lead to the
expected result?
Did the action change the
perception of the EU?
Q.

20. Terms
• Inputs / preparation
• Activities / implementation
• Outputs
• Outtakes
• Outcomes / outflows / outgrowths
• Results / effects / impact

21. Program theory and program logic models
Basic program model evolved since the 1970s (Kellogg Foundation, 2004, p. 1)

22. Program theory and program logic models
University of Wisconsin University Cooperative Extension Program (UWEX)
(Taylor-Power & Henert, 2008, p. 5)

23. What are the standards for stages/terms?
Program and PR evaluation models (1985–early 2000s)
Basicprogramlogicmodel–
UWEX(Taylor-Power&Henert,
2008)
Expandedprogramlogicmodel
–UWEX(Taylor-Power&
Henert,2008)
ClassicProgramLogicModel
(e.g.,KelloggFoundation
(1998/2004)
Cutlip,Center&Broom(1985)
‘PIImodel’
Macnamara’sMacromodel’
(1992)
Lindenmann’sEffectiveness
Yardstick(1993,1997a)
IPRAGoldPaperonEvaluation
(1994,pp.10,18–19)
Fairchild(1997,2001);Fairchild
&O’Connor,IPRToolkit(1999)
LindenmannStandardsand
Guidelines,1997b,1997c)
Noble&Watson’s‘Unified
Model’(1999)
Grunig&Hon‘relatinships’
model
DPRG/GPRA(2000),DPRG/ICV
(2009)models
LikelyPerformance
MeasurementFramework
(2000)
Macnamara’s‘Pryamid’model
(2000,2002a,2002b)
Lindenmann’s‘guidelines'
(2002/2003)
Input Inputs Inputs Inputs Inputs Inputs
(hinted)
Input Input Inputs
Activities Planning
Output Outputs
 Activities
 Participation
Outputs Implement-
ation
Outputs Outputs
(basic)
Outputs Output Outputs Output Outputs Output Outputs Outputs PR Outputs
Outtake Outtakes Outtakes PR Outtakes
Outgrowths
(inter-
mediate)
Outcomes Outcomes
 Short-term
 Intermediate
 Long-term
Outcomes Outcomes
(advanced)
Outcomes Outcome Outcomes Outcomes Outcome Outcomes Outcomes PR Outcomes
Impact Impact Results Business /
organization
outcomes
Impact Relationships Outflow Outgrowths Business /
organization
outcomes
Effect
9
14
4
1
12
5
1
(Macnamara, 2016)

24. Theory of Change
INPUTS ACTIVITIES OUTPUTS OUTCOMES IMPACT
Communication
• Objectives
• Target audiences
• Strategy
Actions
Short | Medium | Long term
The Organisation
Organisation goals and objectives
Feedback loops to monitor and adjust strategy and tactics if required
Program Theory

25. INPUTS ACTIVITIES OUTPUTS OUTCOMES OUTCOMES OUTCOMES
Short term Intermediate Long term
OUTTAKES OUTCOMES IMPACT
Communication
• Objectives
• Target audiences
• Strategy
Feedback loops to planning
Preparation Production Distribution Exposure/Reception Response Effects Results Organisation/Stakeholder Impact
ORGANISATION
Organisation goals and objectives
Theory of Change
Program Theory

27. AMEC Integrated Evaluation Framework (AMEC, 2016)

32. INPUTS ACTIVITIES OUTPUTS OUTCOMES OUTCOMES OUTCOMES
Short term Intermediate Long term
OUTTAKES OUTCOMES IMPACT
Communication
• Objectives
• Target audiences
• Strategy
Feedback loops to planning
Preparation Production Distribution Exposure/Reception Response Effects Results Organisation/Stakeholder Impact
ORGANISATION
Organisation goals and objectives
Theory of Change
Program Theory
? ? ? ? ? ?

34. Inputs Activities Outputs Outtakes Outcomes Impact

35. INPUTS ACTIVITIES OUTPUTS OUTCOMES IMPACT
The Organisation
Organisation goals and objectives
Feedback loops to monitor and adjust strategy and tactics if required
Short | Medium | Long term
Communication
• Objectives
• Target audiences
• Strategy Theory of Change
Program Theory

36. INPUTS ACTIVITIES OUTPUTS OUTCOMES IMPACT
The Organisation
Organisation goals and objectives
• Budget
• Personnel
• Expertise
• Other resources
• Formative research
• Creative design
• Pre-testing
• Planning
• Production
• Media relations
• Paid advertising
• Media publicity
• Publications (e.g.,
newsletters, reports)
• Web sites
• Social media posts
• Events
• Sponsorships
• Community projects
Feedback loops to monitor and adjust strategy and tactics if required
Preparation Production Distribution Exposure/Reception Response Results
Short | Medium | Long term
• Recall
• Awareness
• Interest (e.g., follows,
likes, retweets)
• Engagement
• Satisfaction
• Attitude change
• Trust
• Preference
• Intentions
• Inquiries / leads
• Advocacy
• Reputation
• Relationships
• Public support
• Sales / revenue
• Donations
• Cost savings
• Staff retention
• Customer retention
• Complying behaviour
(e.g., give blood; drive
safely, get fit)
• Quality of life / well-
being improvements
Communication
• Objectives
• Target audiences
• Strategy Theory of Change
Program Theory

37. INPUTS ACTIVITIES OUTPUTS OUTCOMES IMPACT
The Organisation
Organisation goals and objectives
• Budget
• Personnel
• Expertise
• Other resources
• Formative research
• Creative design
• Pre-testing
• Planning
• Production
• Media relations
• Paid advertising
• Media publicity
• Publications (e.g.,
newsletters, reports)
• Web sites
• Social media posts
• Events
• Sponsorships
• Community projects
Stakeholders, Publics, Society
Feedback loops to monitor and adjust strategy and tactics if required
• Sales/revenue data
• Behaviour tracking
• CRM data
• Surveys
• Cost benefit / ROI
• Well-being metrics
• Surveys (awareness,
reputation, trust)
• Interviews
• Social media qual
analysis (likes, shares,
comments)
• Net Promoter Score
• Media metrics (reach,
impressions, OTS, TARPs,
CPM)
• Content analysis
(e.g., messages,
tone/sentiment)
• Web site data
• Social media stats
• Attendance figures
• Reader surveys
• Pre- surveys
• Focus groups
• Interviews
• Baseline data
collection (e.g.,
databases)
• Pre-testing
• Peer review / expert
review
EVALUATION
METHODS
Preparation Production Distribution Exposure/Reception Response Results
Short | Medium | Long term
• Recall
• Awareness
• Interest (e.g., follows,
likes, retweets)
• Engagement
• Satisfaction
• Attitude change
• Trust
• Preference
• Intentions
• Inquiries / leads
• Advocacy
• Reputation
• Relationships
• Public support
• Sales / revenue
• Donations
• Cost savings
• Staff retention
• Customer retention
• Complying behaviour
(e.g., give blood; drive
safely, get fit)
• Quality of life / well-
being improvements
Communication
• Objectives
• Target audiences
• Strategy Theory of Change
Program Theory

38. Questions