Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Measure What Matters: Monitoring and Measurement
in the New Media World
Katie Delahaye Paine, CEO
Paine Publishing
Jan. 22...
Paine Publishing:
Providing communications’
professionals the knowledge and
information they need to navigate
their path t...
Don’t Measure What’s Easy
3
Cartoon by Rob Cottingham
Measure What Matters
25/02/2015News Group International 4
Cartoon by Rob Cottingham
Welcome to the
“I want what I want, not what
you send me” era
aka: “The Relationship Era” ®
Measuring Attribution in an era of all media everywhere
Web Analytics
eCommerce
iCloud &
Google+
Map My Run/Social
Sharing...
Conversations
$$$$$$$$$$$$$$$$$$$$$$$$$$
Savings, shorter cycles, more
renewals, better ideas,
Product
Support HRIR SalesC...
We need a new Attribution Model
ROI
Other
Paid
Marketing
Digital/
Online
Media
Buy
Print/TV
Media
Buy
ROI
Media
Relations
...
What has changed?
 Collapse of mass media
 Growth of media everywhere
 Intolerance for messaging
 The Barcelona Princi...
The Barcelona Principles, The Conclave, and Industry Standards
1. Importance of goal setting & measurement
2. Measuring th...
Social Media Measurement Standards have been published for :
Content Sourcing & Methods1
Reach & Impressions2
Engagement3
...
Eyeball
counting
HITS Outcomes
MSM Online
Social
Media
Impressions are not awareness…
Where’s the “So what?”
Likes are not engagement
AdvocacyCommitmentTrial/Consideration
FollowersLikesImpressions
• Influence happens when you are persuaded to change behavior or opinion that
would otherwise not have changed.
• Influenc...
#5 Standards for Measuring Tone/Sentiment
 Whatever process is defined and applied, it must be used
consistently througho...
• Impact and value will always be dependent on client objectives
• Need to define outcomes in advance – will likely span m...
Good Relationships Are More Cost Effective
 Type I love Zappos into Google, and you find
1.19 million references
 Type C...
ROI
 ROI is:
 Cost savings
 Greater efficiency
 Lower legal costs
 Faster time to implementation
 Lower cost per imp...
ACA Measurement Program Parameters
 Goal: Move to Industry Standards Metrics and away
from media value and ad value equiv...
Definitions of “Success”
 Workshop defined the criteria
 All criteria linked back to the goals:
High-quality media cover...
Reality: Negative coverage of AC is constant & growing
0
100
200
300
400
500
600
700
Jan Feb Mar Apr May Jun Jul Aug Sep O...
Solution: ACA Programs generate predominantly positive
coverage
0
20
40
60
80
100
120
140
160
Jan Feb Mar Apr May Jun Jul ...
Without ACA events, OCS scores for Atlantic City would
have been significantly lower
4
3.25
2.75
2.99
3.65
2.96
3.36 3.24
...
ACA has made a demonstrable difference in the quality of
media coverage
 If an ACA or one of its
programs was mentioned
i...
OCS scores for Atlantic City significantly improved thanks
to ACA events
36
278
458
334
537
432
342
225
995
329 324
247
13...
ACA maintains high OCS score with
proactive program coverage Lady Antebellum,
Non-Gambling Fun,
Hello Summer,
GMA segment
...
14
39
23
36
133
235
8 7 4 2 10 158 11 1 2 9 2
24
70
21
73
88
110
13
104
74
138
87
48
Jan Feb Mar Apr May Jun
Qtr1 Qtr2
201...
Quality of coverage has improved in all markets
4.00
4.25
8.00
3.62
-1.00
4.00
6.50
4.65
5.07
5.88
6.50
5.70
7.17
5.13 5.1...
Free entertainment generated the highest OCS scores in Q2
5.26
5.73
6.36
6.54
6.71
7.15
7.31
7.36
7.50
7.67
8.00
8.10
8.68...
When ACA programs received media coverage, traffic followed
0
50
100
150
200
250
300
0
20000
40000
60000
80000
100000
1200...
ACA programs drive higher OCS scores, which correlate
highly with web visits
0.41
0.44
0.47
AC items
ACA Items
ACA OCS Sco...
PR is having a positive impact on preference & perceptions
 People who say they have recently heard news about Atlantic
C...
6 steps to standards-compliant
measurement
Step 1: Define your goal(s)
What outcomes is this strategy or tactic going to a...
Definitions
 Monitoring – process by which data are systematically and regularly
collected about a program over time.
 M...
Why do we communicate?
Outtakes
(Intermediary Effects)
• Awareness
• Knowledge/Education
• Understanding
Outcomes
(Target ...
Step 1: Define the goals
 What return is expected? – Define in terms of the
mission.
 Define your champagne moment. If y...
Goals & Suggested Metrics
Goals
Goal 1: Meet
sales targets
Goal 2: Reduce
risk/threats
Goal 3: Increase
market share in
ne...
Step 2: Understand the parameters
What are management’s priorities?
Who are you are trying to reach?
How do your effort...
Step 3: Establish benchmarks
 Past Performance Over Time
 Think 3
 Whatever keeps your
C-suite up at night
Step 4: Pick your Kick-Butt Index
The Perfect KBI
 Is actionable
 Is there when you need it
 Continuously improves your...
Defining High Quality Coverage
The OCS Score
Desirable Criteria Score Undesirable Score Score
Positive sentiment 1 Negativ...
Step 5: Pick the right measurement tools
 If you want to measure messaging, positioning,
themes, sentiment:
Content analy...
Step 5 continued: Selecting a measurement tool
43
Objective KBI Tool
Increase message
communications
Increase percent of i...
Testing the accuracy of coding
48.94%
68.31%
88.64%
0.00% 20.00% 40.00% 60.00% 80.00% 100.00%
SDL
UberVu
NetBase
% Agreeme...
Everyone agrees on “Neutral”…
But, not the case with “Negative” & “Positive”
100.00%
62.32%
14.82%
72.36%
0.00%
37.68%
85....
Step 6: Be data informed, not data driven
 Rank order results from worst to best
 Ask “So What?” at least 3 times
 Put ...
Photo Event
High Message Content
Resource Use
Low
High
Medium
Veryhigh
Level of Engagement
VeryHigh
Medium
High
Low
High R...
Real-time metrics focus on outcomes
10 Ways to find the money for measurement
1. Don’t call it measurement – It’s Research
2. Consolidate – Social? IR? Agenci...
Remember these points:
It’s not about the media, it’s about the business and the customer.1
It’s not about Big Data, it’s ...
Thank You!
 For more information on measurement, read my blog:
http://www.painepublishing.com/blog
 For a copy of this p...
Upcoming SlideShare
Loading in …5
×

PRSA Connecticut, January 2015

1,088 views

Published on

Slides from "Measure What Matters: Monitoring & Measurement in the New Media World" as presented to the Southern Connecticut Chapter of PRSA in January 2015.

Published in: Business
  • Login to see the comments

PRSA Connecticut, January 2015

  1. 1. Measure What Matters: Monitoring and Measurement in the New Media World Katie Delahaye Paine, CEO Paine Publishing Jan. 22, 2015 www.painepublishing.com measurementqueen@gmail.com | @queenofmetrics
  2. 2. Paine Publishing: Providing communications’ professionals the knowledge and information they need to navigate their path to perfect measurement  Newsletters  Training Courses  Consulting  painepublishing.com Katie Delahaye Paine: Helping communications professionals define and measure success for 25 years.  Founder of:  The Delahaye Group  KDPaine & Partners  Paine Publishing  Author of:  Measuring the Networked Non-Profit  Measure What Matters  Measuring Public Relationships
  3. 3. Don’t Measure What’s Easy 3 Cartoon by Rob Cottingham
  4. 4. Measure What Matters 25/02/2015News Group International 4 Cartoon by Rob Cottingham
  5. 5. Welcome to the “I want what I want, not what you send me” era aka: “The Relationship Era” ®
  6. 6. Measuring Attribution in an era of all media everywhere Web Analytics eCommerce iCloud & Google+ Map My Run/Social Sharing CRM/Convio
  7. 7. Conversations $$$$$$$$$$$$$$$$$$$$$$$$$$ Savings, shorter cycles, more renewals, better ideas, Product Support HRIR SalesCIR&D Mktg Research Prod. MktgPR//Mkt 7 Social media isn’t media, it is your business
  8. 8. We need a new Attribution Model ROI Other Paid Marketing Digital/ Online Media Buy Print/TV Media Buy ROI Media Relations /Social Media/PR Digital/ Online Media Buy Print/TV Media Buy Media Relations /Social Media/PR
  9. 9. What has changed?  Collapse of mass media  Growth of media everywhere  Intolerance for messaging  The Barcelona Principles
  10. 10. The Barcelona Principles, The Conclave, and Industry Standards 1. Importance of goal setting & measurement 2. Measuring the effect on outcomes is preferred to measuring outputs 3. The effect on business results can & should be measured where possible 4. Media measurement requires quantity & quality 5. Earned Media Value/AVEs are not the value of public relations 6. Social media can & should be measured 7. Transparency & replicability are paramount to sound measurement All standards are available on http://www.painepublishing.com/ standards-central Conclave Members
  11. 11. Social Media Measurement Standards have been published for : Content Sourcing & Methods1 Reach & Impressions2 Engagement3 Influence & Relevance4 Opinion & Advocacy5 Impact & Value6
  12. 12. Eyeball counting HITS Outcomes MSM Online Social Media Impressions are not awareness… Where’s the “So what?”
  13. 13. Likes are not engagement AdvocacyCommitmentTrial/Consideration FollowersLikesImpressions
  14. 14. • Influence happens when you are persuaded to change behavior or opinion that would otherwise not have changed. • Influence can be attributed to either an outlet or an individual. • Influence happens online and off line and both should be considered • Should include some combination of the following five elements: • Reach • Engagement around individual • Relevance to topic • Frequency of posts around the topic • Audience impact as measured by the ability to get the target audience to change behavior or opinion. • If an individual scores a 0 on one element, they have no influence • Influence cannot be expressed in a single score or algorithm #4: Influence & Relevance
  15. 15. #5 Standards for Measuring Tone/Sentiment  Whatever process is defined and applied, it must be used consistently throughout any analysis.  Sentiment reliability varies by vendor and approach – therefore coding definitions, consistency and transparency are critical.  Opinions, recommendations and other qualitative measures are typically more valuable than raw sentiment and increasingly measurable:  Opinions (“It’s a good product”)  Recommendations (“try it” or “avoid it”)  Feeling/Emotions (“That product makes me feel happy”)  Intended action (“I’m going to buy that product tomorrow”)
  16. 16. • Impact and value will always be dependent on client objectives • Need to define outcomes in advance – will likely span multiple business goals, especially for social (crosses disciplines) • “ROI” should be strictly limited to measurable financial impact; “total value” can be used for financial and non-financial impact combination • Value can be calculated in positive returns (sales, reputation, etc.) or avoided negative returns (risk mitigated, costs avoided) • Key performance indicators and balanced scorecards are helpful to connect social media impact to business results/language #6: Impact & Value
  17. 17. Good Relationships Are More Cost Effective  Type I love Zappos into Google, and you find 1.19 million references  Type Citibank and you get 21,000 references. Citibank spends 100 times more a year on advertising than Zappos.  Cost per delegate acquired:  Obama: $6,024  Clinton: $147,058  Romney: $2,389,464
  18. 18. ROI  ROI is:  Cost savings  Greater efficiency  Lower legal costs  Faster time to implementation  Lower cost per impact vs. other marketing channels  Lower cost per message exposure vs. other channels  ROI is not:  Impressions  Earned media value or AVE  Number of shares, views or likes
  19. 19. ACA Measurement Program Parameters  Goal: Move to Industry Standards Metrics and away from media value and ad value equivalency  Measure quality not just quantity  Began in January 2013  Established a benchmark: January-March 2013  Customized “Quality Score” that indexes message content, tone, and media importance on a scale of +10 to -10  Correlated with web traffic to measure outcomes  Correlated with longitudinal research to determine impact on preference and intent to visit  Data used continuously in strategic decision making
  20. 20. Definitions of “Success”  Workshop defined the criteria  All criteria linked back to the goals: High-quality media coverage Intent to visit Improved reputation Visits/non-gaming revenue
  21. 21. Reality: Negative coverage of AC is constant & growing 0 100 200 300 400 500 600 700 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Tone of Coverage about Atlantic City Over Time Negative Neutral Positive Miss America
  22. 22. Solution: ACA Programs generate predominantly positive coverage 0 20 40 60 80 100 120 140 160 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Sentiment of ACA Coverage Over Time Negative Neutral Positive
  23. 23. Without ACA events, OCS scores for Atlantic City would have been significantly lower 4 3.25 2.75 2.99 3.65 2.96 3.36 3.24 2.34 2.37 2.43 1.30 -1.24 0.37 -0.05 0.28 0.28 -1.56 4.91 3.92 2.99 3.58 4.14 4.1 4.27 4.12 4.29 2.78 2.56 1.53 -0.29 0.61 0.20 1.44 1.77 0.63 -2 -1 0 1 2 3 4 5 6 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May June AverageOCSScore Average OCS Score Over Time Atlantic City OCS without ACA Atlantic City OCS with ACA The red line represents coverage of Atlantic City minus all mentions of ACA and its programs
  24. 24. ACA has made a demonstrable difference in the quality of media coverage  If an ACA or one of its programs was mentioned in a media story, it was:  More likely to contain an endorsement  More likely to be positive  More likely to contain key messages  Less likely to be negative 0.22 0.00 0.62 1.42 4.61 0.08 0.08 0.01 -0.61 -0.66 Endorsements Positive Visual Negative Visual Tone Message Communication Average of OCS Differences in Quality of Coverage with and without ACA Atlantic City Atlantic City Alliance
  25. 25. OCS scores for Atlantic City significantly improved thanks to ACA events 36 278 458 334 537 432 342 225 995 329 324 247 139 419 260 476 538 925 4.89 3.92 2.99 3.58 4.14 4.10 4.27 4.12 4.29 2.78 2.56 1.55 -0.29 0.61 0.20 1.44 1.77 0.63 -1.00 0.00 1.00 2.00 3.00 4.00 5.00 6.00 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May June 0 200 400 600 800 1000 1200 OCSScore Numberofitems AC Mentions vs. OCS Over Time AC Mentions AC OCS Dead whale, non- gaming promotions, Caesar’s CEO quote, Lady Antibellum Revel Casino value, Electric Jitney test, Fake Chips, Phil Ivey suit, Blake Shelton Showboat closing, Revel facing 2nd bankruptcy
  26. 26. ACA maintains high OCS score with proactive program coverage Lady Antebellum, Non-Gambling Fun, Hello Summer, GMA segment 14 44 31 93 87 75 75 40 95 28 7 27 17 18 12 107 122 254 5.5 7.18 6.3 5.02 6.24 6.42 7.42 7.21 6.54 5.27 4.25 5.08 6.47 5.94 5.33 4.75 6.87 6.41 0 1 2 3 4 5 6 7 8 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May June 0 50 100 150 200 250 300 OCSScore Numberofmentions ACA Mentions vs. OCS Over Time ACA Mentions ACA OCS Miss America Statue, ACA new campaign, Blake Shelton, World Poker Championship
  27. 27. 14 39 23 36 133 235 8 7 4 2 10 158 11 1 2 9 2 24 70 21 73 88 110 13 104 74 138 87 48 Jan Feb Mar Apr May Jun Qtr1 Qtr2 2014 Atlantic City has something for everyone Atlantic City is a clean and safe place to visit Atlantic City is a year-round destination spot Atlantic City is making a comeback Atlantic City offers more than just gaming Communication of the “Making a Comeback” & “Something for Everyone” messages increased as ACA announced new programs
  28. 28. Quality of coverage has improved in all markets 4.00 4.25 8.00 3.62 -1.00 4.00 6.50 4.65 5.07 5.88 6.50 5.70 7.17 5.13 5.13 4.67 7.31 7.02 6.25 6.89 5.00 5.29 6.40 7.55 9.00 Jan Feb Mar Apr May Jun New Markets Yield High Quality Results Houston Chicago National Baltimore Philadelphia New York DC
  29. 29. Free entertainment generated the highest OCS scores in Q2 5.26 5.73 6.36 6.54 6.71 7.15 7.31 7.36 7.50 7.67 8.00 8.10 8.68 9.30 Miss America Meet AC DO AC July 4th Fireworks Air Show Blake Shelton Miss'd America Pageant Sand Blast Hello Summer Boardwalk Hall Light Show Challenge Triathlon Sand Sculpting World Cup Lady Antebellum Concert Free Entertainment Top Programs By OCS
  30. 30. When ACA programs received media coverage, traffic followed 0 50 100 150 200 250 300 0 20000 40000 60000 80000 100000 120000 140000 Relationship between ACA Program Mentions and Site Visits Site Visits Program Mentions
  31. 31. ACA programs drive higher OCS scores, which correlate highly with web visits 0.41 0.44 0.47 AC items ACA Items ACA OCS Scores Correlations between Web Visits and PR Metrics Pearson r. value
  32. 32. PR is having a positive impact on preference & perceptions  People who say they have recently heard news about Atlantic City are very likely to perceive AC as fun – a key driver of preference  Respondents who remember seeing news reports about Atlantic City are also very likely to associate key messages and positioning statements such as “place I am excited to go to” and “fun place to hang out with friends.”  The same respondents are also more likely to recommend Atlantic City to friends.
  33. 33. 6 steps to standards-compliant measurement Step 1: Define your goal(s) What outcomes is this strategy or tactic going to achieve? What are your measurable objectives? Step 2: Define the parameters Who are you are trying to reach? How do your efforts connect with those audiences to achieve the goal. Step 3: Define your benchmarks Who or what are you going to compare your results to? Step 4: Define your metrics What are the indicators to judge your progress? Step 5: Select your data collection tool(s) Step 6: Analyze your data. Turn it into action, measure again Six Steps to Success 1 2 3 4 5 6
  34. 34. Definitions  Monitoring – process by which data are systematically and regularly collected about a program over time.  Measurement – a way of giving an activity a precise dimension, generally by comparison to some standard; usually done in a quantifiable or numerical manner; see also: data, scale  Metrics – an agreed upon number or percentage that indicates progress towards a goal  Evaluation – a form of research that determines the relative effectiveness of a public relations campaign or program by measuring program* *IPR Dictionary of Measurement Terms, Dr. Donald Stacks www.instituteforpr.org
  35. 35. Why do we communicate? Outtakes (Intermediary Effects) • Awareness • Knowledge/Education • Understanding Outcomes (Target Audience Action) • Revenue • Leads • Engagement • Advocacy ActivitiesHow does what you do contribute to the bottom line?
  36. 36. Step 1: Define the goals  What return is expected? – Define in terms of the mission.  Define your champagne moment. If you are celebrating complete 100% success a year from now, what is different about the organization? 36
  37. 37. Goals & Suggested Metrics Goals Goal 1: Meet sales targets Goal 2: Reduce risk/threats Goal 3: Increase market share in new market Communications Contribution • Expand the marketable universe • Reduce sales cycle • Increase trust • Increase advocacy • Expand the marketable universe Metrics • % increase in desirable share of voice • % increase in awareness • % decrease in undesirable voice • % increase in trust scores • % increase in desirable voice in new market • % increase in awareness of brand in new market
  38. 38. Step 2: Understand the parameters What are management’s priorities? Who are you are trying to reach? How do your efforts connect with those audiences to achieve the goal? What influences their decisions? What’s important to them? What makes them act? 38 Goal: Get the cat to stop howling Options: Local? Cheap? Convenient? Strategy: Buy cat food
  39. 39. Step 3: Establish benchmarks  Past Performance Over Time  Think 3  Whatever keeps your C-suite up at night
  40. 40. Step 4: Pick your Kick-Butt Index The Perfect KBI  Is actionable  Is there when you need it  Continuously improves your processes & gets you where you want to go  You become what you measure, so pick your KBI carefully 40
  41. 41. Defining High Quality Coverage The OCS Score Desirable Criteria Score Undesirable Score Score Positive sentiment 1 Negative sentiment -2 Contains one or more positive message(s) 3 Contains one or more negative messages(s) -3 Event/program is mentioned 2 No event/program is mentioned 0 Appears in Tier 1 Media 2 Negative mention in Tier 1 -1 Third party endorsement 1 Recommends competition -2 Contains a desirable visual 1 Contains undesirable visual -2 Total Score 10 Total -10
  42. 42. Step 5: Pick the right measurement tools  If you want to measure messaging, positioning, themes, sentiment: Content analysis  If you want to measure awareness, perception, relationships, preference: Survey research  If you want to measure engagement, action, purchase: Web analytics  If you want predictions and correlations you need two out of three
  43. 43. Step 5 continued: Selecting a measurement tool 43 Objective KBI Tool Increase message communications Increase percent of items containing one or more message(s) CyberAlert, Prime Research Increase awareness/preference % of audience preferring your brand to the competition Survey Monkey, Survata Engage marketplace % increase in engagement on website and/or social sites Simply Measured, Unmetric, Google Analytics, Site Catalyst, Network Analysis Become a thought leader on a specific issue/market/topic % increase in desirable share of voice on issue/market/top Little Bird, Trackr, CyberAlert
  44. 44. Testing the accuracy of coding 48.94% 68.31% 88.64% 0.00% 20.00% 40.00% 60.00% 80.00% 100.00% SDL UberVu NetBase % Agreement with human coding SDL UberVu NetBase 33.12% 58.00% 7.84% 11.95% 26.53% 13.00% 0.32% 9.46% 40.35% 30.00% 92.11% 78.00% SDL Beyond NetBase UberVu % positive %negative %neutral
  45. 45. Everyone agrees on “Neutral”… But, not the case with “Negative” & “Positive” 100.00% 62.32% 14.82% 72.36% 0.00% 37.68% 85.18% 27.64% Mixed Negative Neutral Positive Degree to which agreement occurred No Yes
  46. 46. Step 6: Be data informed, not data driven  Rank order results from worst to best  Ask “So What?” at least 3 times  Put your data into an overall framework consistent with C-Suite expectations  Find your “Data Geek”  Compare to last month, last quarter, 13-month average
  47. 47. Photo Event High Message Content Resource Use Low High Medium Veryhigh Level of Engagement VeryHigh Medium High Low High Resources No Message Low Resources Webinar Status update Link Ultimate Road Trip Google + Chat Media Day Corporate Video Advocacy vs. resource use Resource Use Low High Medium Veryhigh VeryHigh Medium High Low High ResourcesLow Resources Webinar Status update Link Ultimate Road Trip Google + Chat Media Day Corporate Video
  48. 48. Real-time metrics focus on outcomes
  49. 49. 10 Ways to find the money for measurement 1. Don’t call it measurement – It’s Research 2. Consolidate – Social? IR? Agencies? Research? 3. Crowdsource – Who else has tools or surveys? 4. Spread the cost over 2 fiscal years 5. Reach out to universities 6. Take advantage of free stuff 7. Monitor only what matters 8. Narrow your Top Tier media list to what you CAN afford 9. Google Analytics 10. Facebook Insights 11. Twitter Analytics
  50. 50. Remember these points: It’s not about the media, it’s about the business and the customer.1 It’s not about Big Data, it’s about how you use it.2 You need to be data informed, not data-driven.3 It’s not how loud you’re shouting, it’s about relationships.4 Standards are a reality, not an excuse to hide behind.5
  51. 51. Thank You!  For more information on measurement, read my blog: http://www.painepublishing.com/blog  For a copy of this presentation or to subscribe to our newsletter, give me your card or email me at measurementqueen@gmail.com  Follow me on Twitter: @queenofmetrics  “Like” Paine Publishing’s Facebook account  Call me at 1-603-682-0735

×