Successfully reported this slideshow.

2009 Measurement 101


Published on

This is my Measurement 101 presentation from The IPR Measurement Summit 09 in Portsmouth, NH

Published in: Business, Technology
  • Be the first to comment

2009 Measurement 101

  1. 1. Measurement 101 A Presentation to the IPR Measurement Summit October 2009 Katie Delahaye Paine Member, IPR Measurement Commission CEO [email_address]
  2. 2. Why Measure? <ul><li>“ The main reason to measure objectives is not so much to reward or punish </li></ul><ul><li>individual communications manager for success or failure as it is to learn from the </li></ul><ul><li>research whether a program should be continued as is, revised, or dropped in favor of another approach ” </li></ul><ul><li>James E. Grunig, Professor Emeritus, University of Maryland </li></ul>“ If we can put a man in orbit, why can’t we determine the effectiveness of our communications? The reason is simple and perhaps, therefore, a little old-fashioned: people, human beings with a wide range of choice. Unpredictable, cantankerous, capricious, motivated by innumerable conflicting interests, and conflicting desires.” Ralph Delahaye Paine, Publisher, Fortune Magazine , 1960 speech to the Ad Club of St. Louis
  3. 3. The Ah-Ha moment, the Lotus PR Report Percent of impressions containing messages by product
  4. 4. The Ah-Ha moment, the Lotus PR Report Cost per message communicated
  5. 5. Victory Over Martians Confirmed! Share of exposure over time 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Jan Mar May Jul Sep Nov Jan Mar May Jul Sep Nov Jan IN TI MO NS
  6. 6. Interviews and media advisories generated best coverage 0% 20% 40% 60% 80% 100% Application articles Contract wins Exec Interview Media advisory Release + conference Press release plus VNR Product review Industry issue Trade show/event No Message Negative Message Positive Message
  7. 7. Comparing the effectiveness of different tactics
  8. 8. Correlation exists between traffic to the ASPCA web site and the organization’s overall media exposure
  9. 9. Correlations also exist between online donations to the ASPCA and the organization’s overall media exposure
  10. 10. Facebook: Correlating MSM, CGM and signups <ul><li>Strong correlation </li></ul><ul><li>Non-negative discussion only </li></ul>
  11. 11. A measurement timeline
  12. 12. 10 signs that it’s the end of measurement as we know it <ul><li>BMC Software measures communications effectiveness based on contribution to EPS </li></ul><ul><li>Procter & Gamble is now paying for engagement, not eyeballs </li></ul><ul><li>Sodexo measures success of its Twitter program by showing $300K reduction in recruitment costs </li></ul><ul><li>USO compares PR and PSAs to determine effectiveness of fund raising </li></ul><ul><li>HSUS measures Social media success by calculating $650,000 in new donations </li></ul><ul><li>Immunize BC counts calculates awareness + shots delivered </li></ul><ul><li>The Red Cross measures the effectiveness of Twitter via lives saved and harm avoided </li></ul><ul><li>IBM can calculate more sales from a $500 podcast than a $40K ad program </li></ul><ul><li>BestBuy measures 85% lower turnover as a result of its Blue Shirt community </li></ul><ul><li>Advertisers are starting to admit that all their measures are flawed </li></ul>
  13. 13. What do you need to measure to get to the Holy Grail of ROI measurement? Data. <ul><li>Outputs? </li></ul><ul><ul><li>Did you get the volume of coverage & visibility you wanted? </li></ul></ul><ul><ul><li>Did you produce the promised materials on time and on budget? </li></ul></ul><ul><li>Outtakes? </li></ul><ul><ul><li>Did your target audience see the messages? </li></ul></ul><ul><ul><li>Did they believe the messages? </li></ul></ul><ul><li>Outcomes? </li></ul><ul><ul><li>Did audience behavior change? </li></ul></ul><ul><ul><li>Did the right people show up? </li></ul></ul><ul><ul><li>Did your relationship change? </li></ul></ul><ul><ul><li>Did sales increase? </li></ul></ul>
  14. 14. Goals, Actions and Metrics Goal Action Output Metric Outtake Metric Outcome Metric Increased on-line donations Revamp website Amount of content on web site % perceiving ASPCA as a reliable source % increase in web traffi and donations #1 most trusted source for information on companion animals Increase staffing and resources for communications Increased exposure of “trusted source” message Increased perception of ASPCA as trustworthy and comprehensive source % increase in agreement with the statement Website is preferred site for information Add content, features to web site, keep up to date % increase in traffic % agreeing with the statement # 1 rankings, and time spent on site Integration of department Reorganize department More integrated materials produced on time and on budget Internal perception is “one department” Consistent messaging throughout
  15. 15. The 7 steps to PR ROI <ul><li>Define the “R” – Define the expected results? </li></ul><ul><li>Define the “I” -- What’s the investment? </li></ul><ul><li>Understand your audiences and what motivates them </li></ul><ul><li>Define the metrics (what you want to become) </li></ul><ul><li>Determine what you are benchmarking against </li></ul><ul><li>Pick a tool and undertake research </li></ul><ul><li>Analyze results and glean insight, take action, measure again </li></ul>
  16. 16. Step 1: Define the “R” what return do you expect? s <ul><li>What were you hired to change? </li></ul><ul><li>If you are celebrating complete 100% success a year from now, what is different about the organization? </li></ul><ul><li>If you eliminated your department or failed utterly, what would be different? </li></ul>
  17. 17. <ul><li>Generates Revenue, Sales, Profit </li></ul><ul><ul><ul><li>Marketing Public Relations drives sales </li></ul></ul></ul><ul><ul><ul><li>Investor Public Relations drives investment </li></ul></ul></ul><ul><ul><ul><li>Public Relations drives donations & membership for relevant organizations </li></ul></ul></ul><ul><li>Drives Efficiency </li></ul><ul><ul><ul><li>Better audience targeting </li></ul></ul></ul><ul><ul><ul><li>Reaching more people with a credible message for less money </li></ul></ul></ul><ul><li>Avoids Catastrophic Cost </li></ul><ul><ul><ul><li>Quality counsel helps to mitigate impacts of crises </li></ul></ul></ul>How PR Impacts Financial Performance
  18. 18. <ul><li>Increases likelihood to purchase / consider your brand(s) </li></ul><ul><li>Minimizes the effects of a crisis </li></ul><ul><li>Reinforces communication of organizational values </li></ul><ul><li>Rebuilds trust after a crisis </li></ul><ul><li>Establishes credibility of new products / companies; ease of market entry </li></ul><ul><li>Commanding higher prices, lower costs, premium on stock price </li></ul><ul><li>Enhances recommendations / word of mouth leading to faster adoption </li></ul><ul><li>Increases customer loyalty / renewals / satisfaction </li></ul><ul><li>Improves the attracting / retaining of talent </li></ul><ul><li>Lowers legal costs </li></ul>How PR Positively Affects Reputation / Brand Equity
  19. 19. <ul><li>Increases employee satisfaction and engagement, leading to greater efficiency, increased retention, lower turnover rates, lower recruitment costs, and higher productivity </li></ul><ul><li>Lowers legal costs </li></ul><ul><li>Change employee behaviors such as greater levels of focus on key areas such as safety, quality, call response times </li></ul><ul><li>Provides greater transparency and commitment to and from employees </li></ul><ul><li>Creates a platform should it be necessary to communicate bad news at some stage in the future </li></ul><ul><li>* Note items here can also refer to other internal publics such as trade association members . </li></ul>How PR Impacts Employees
  20. 20. <ul><li>Creates public awareness, understanding, and support for legislation, regulation, and political candidates </li></ul><ul><li>Affects voter behavior </li></ul><ul><li>Helps pass legislation, regulation, and initiatives </li></ul><ul><li>Affects specific companies and industries through appropriations, tax impacts, and regulatory changes that can affect any and all aspects of a business </li></ul><ul><li>Instigates and perpetuates grassroots or grasstops campaigns </li></ul>How PR Affects Public Policy
  21. 21. Step 2: Define your investment <ul><li>You can’t divide by $0 </li></ul><ul><li>People time </li></ul><ul><li>Opportunity Cost </li></ul><ul><li>Executive time/goodwill </li></ul>
  22. 22. Step 3: Understand what motivates your stakeholders and priorities the stakeholders <ul><li>What motivates customers to purchase, members to join, students to apply, etc. </li></ul><ul><li>What is your role in the outcome? </li></ul><ul><li>Prioritize based on benefits </li></ul>
  23. 23. Step 4: Define your Key Performance Indicators (KPIs) <ul><li>The Perfect KPI </li></ul><ul><ul><ul><li>Gets you where you want to go (achieves corporate goals) </li></ul></ul></ul><ul><ul><ul><li>Is actionable </li></ul></ul></ul><ul><ul><ul><li>Continuously improves your processes </li></ul></ul></ul><ul><ul><ul><li>Is there when you need it </li></ul></ul></ul>
  24. 24. Step 4: Define your Key Performance Indicators (KPIs) carefully because you become what you measure <ul><li>Cost savings </li></ul><ul><li>Efficiency </li></ul><ul><ul><li>Cost per message communicated </li></ul></ul><ul><ul><li>Cost per new lead/customer acquired </li></ul></ul><ul><li>Productivity: </li></ul><ul><ul><li>Increase in employee engagement/morale </li></ul></ul><ul><ul><li>Lower turnover/recruitment costs </li></ul></ul><ul><li>Engagement: </li></ul><ul><ul><li>Ratio of posts to comments </li></ul></ul><ul><ul><li>% of repeat visitors </li></ul></ul><ul><ul><li>% of 5+min visitors </li></ul></ul><ul><ul><li>% of registrations </li></ul></ul><ul><li>Trust: </li></ul><ul><ul><li>Improvement in relationship /reputation scores with customers and communities (Loyalty/Retention) </li></ul></ul><ul><li>Thought leadership: </li></ul><ul><ul><li>Share of quotes </li></ul></ul><ul><ul><li>Share of opportunities </li></ul></ul><ul><li>Message penetration </li></ul><ul><ul><li>Positioning on key issues </li></ul></ul><ul><ul><li>Improvement in favorable/unfavorable ratio </li></ul></ul><ul><ul><li>Improvement in Optimal Content Score (OCS) </li></ul></ul>
  25. 25. Step 4: Potential KPIs for Media Relations <ul><ul><li>Cost Per Message Communicated </li></ul></ul><ul><ul><li>Increase in % of discussion containing key messages </li></ul></ul><ul><ul><li>% increase in Optimal Content Score </li></ul></ul><ul><ul><li>Share of discussion vs. the competition </li></ul></ul><ul><ul><li>Share of brand visibility vs. the competition </li></ul></ul><ul><ul><li>Share of spokesperson visibility vs. the competition </li></ul></ul><ul><ul><li>Share of desirable coverage (positive+neutral coverage) vs. the competition </li></ul></ul><ul><ul><li>Share of negatives </li></ul></ul><ul><ul><li>Share of favorable positioning on key topics/battles </li></ul></ul><ul><ul><li>Share of mentions by key media (including A-list bloggers) </li></ul></ul>
  26. 26. Potential KPIs for thought leader visibility <ul><ul><li>Our share of quotes in mass media </li></ul></ul><ul><ul><li>Our share of papers in scientific/academic/trade journals </li></ul></ul><ul><ul><li>Our leaders appointed to key professional boards over time </li></ul></ul><ul><ul><li>Our share of papers at key conferences </li></ul></ul><ul><ul><li>% awareness among “influentials” </li></ul></ul><ul><ul><li>Total opportunities to see our message via speaking engagements </li></ul></ul>
  27. 27. Potential KPIs for Community Relations <ul><ul><li>Ratio between positive and negative press in local media </li></ul></ul><ul><ul><li>% of articles in local media that contain our key messages </li></ul></ul><ul><ul><li>% improvement in relationships scores between us and local community and those that influence the local community. </li></ul></ul>
  28. 28. Step 5: Define your benchmarks <ul><li>Existing benchmarks </li></ul><ul><li>Past Performance </li></ul><ul><li>Peer companies </li></ul><ul><li>Whatever keeps the C-suite up at night </li></ul><ul><li>Think 3-5: </li></ul><ul><ul><li>A stretch goal </li></ul></ul><ul><ul><li>The underdog who’s nipping at your heels </li></ul></ul><ul><ul><li>Peer organizations </li></ul></ul><ul><ul><li>Anyone that you compete with for share of mind or share of wallet </li></ul></ul>
  29. 29. Past performance: tonality of blog content
  30. 30. The competitive landscape
  31. 31. Non-Profit industry benchmarks in social media % of Desirable Coverage 96% % of Key Message Communication 37% Number of Messages tracked 11.5 Words per Key Message 14 Number of Key Messages tracked 11.5 Most frequent conversation types Express support (69%) Making an observation (28%) % rallying support 3% % asking a question 4% % of exclusive mentions 17% % mentioning brand in title 8% % of discussions in Blogs 51% % of discussion on Twitter 30% % of visibility from YouTube & Flickr 10%
  32. 32. Overview of Key Metrics for Higher Ed Bookmark. Facebook Ext. Blogs Inst. Blogs YouTube MSM SOV 2% — 8% 9% 11% 7% Popularity 230 bkmks 500/mo. — 20 links 150k views — Engagement 59 cmts 1 day 13 cmts 2-12 cmts 2 cmts — % Positive 20% 32% 54% 50% 15% 15% % Negative 0% 0% 4% 0% 1% 2% Strat. Mess. 40% † 18% † 42% 42% † 18% 38%
  33. 33. Step 6: Conduct research (if necessary) <ul><li>First: find out what already exists </li></ul><ul><ul><li>Web traffic </li></ul></ul><ul><ul><li>Customer Satisfaction data </li></ul></ul><ul><ul><li>Customer Loyalty data </li></ul></ul><ul><li>Second: Decide what research is needed to give you the information you need: </li></ul>
  34. 34. Step 6: Selecting a measurement tool Objective KPI Tool Increase inquiries, web traffic, recruitment % increase in traffic #s of clickthrus or downloads Omniture, Google Analytics, Web Trends Increase awareness/preference % of audience preferring your brand to the competition SurveyMonkey, Zoomerang Engage marketplace Conversation index greater than .8 Rankings TypePad, Technorati , Radian6, Communicate messages % of articles containing key messages Total opportunities to see key messages Cost per opportunity to see key messages Media content analysis – Dashboards % aware of or believing in key message Survey
  35. 35. Your tool box needs: <ul><li>A content source: </li></ul><ul><ul><li>Google News/Google Blogs, RSS feeds </li></ul></ul><ul><ul><li>Technorati, Social Mention, Twazzup, </li></ul></ul><ul><ul><li>Cyberalert, CustomScoop, e-Watch </li></ul></ul><ul><ul><li>Radian 6, Techrigy, Sysymos, Visible Technologies, Scout Labs </li></ul></ul><ul><ul><li>Survey Monkey/Zoomerang </li></ul></ul>
  36. 36. Your tool box also needs to include: <ul><li>2. A way to analyze that content </li></ul><ul><ul><li>Automated vs. Manual </li></ul></ul><ul><ul><li>Census vs random sample </li></ul></ul><ul><ul><li>The 80/20 rule – Measure what matters because 20% of the content influences 80% of the decisions </li></ul></ul><ul><ul><li>Dashboards to aggregate data </li></ul></ul>
  37. 37. The Content Analysis Process Training Are we all on the same page? Content Collection Did we get everything? Coding Start analyzing (computers/humans) Final Reliability (Resource Permitting) Reporting What does it all mean? Adapted with edits from The Content Analysis Guidebook (Neuendorf , 2002) .
  38. 38. Why an Optimal Content Score? <ul><li>You decide what’s important: </li></ul><ul><li>Benchmark against peers and/or competitors </li></ul><ul><li>Track activities against OCS over time </li></ul>
  39. 39. Optimal content score for media coverage <ul><li>Positive: </li></ul><ul><ul><li>Mentions of the brand </li></ul></ul><ul><ul><li>Positive brand mentions </li></ul></ul><ul><ul><li>Key messages </li></ul></ul><ul><ul><li>Customer quoted positive </li></ul></ul><ul><ul><li>Analyst quoted </li></ul></ul><ul><ul><li>Positioned as trusted partner </li></ul></ul><ul><ul><li>Positioned as trustworthy, ethical </li></ul></ul><ul><ul><li>Positioned as vendor of choice </li></ul></ul><ul><ul><li>Positioned as global </li></ul></ul><ul><ul><li>Positioned as one company </li></ul></ul><ul><li>Negative </li></ul><ul><ul><li>Omitted </li></ul></ul><ul><ul><li>Negative tone </li></ul></ul><ul><ul><li>No key message </li></ul></ul><ul><ul><li>No quote </li></ul></ul><ul><ul><li>No analyst quote </li></ul></ul>
  40. 40. How to calculate Optimal Content
  41. 41. Charting OCS over time between divisions
  42. 42. Trend against competition with OCS
  43. 43. Standard classifications of discussion <ul><li>Acknowledging receipt of information </li></ul><ul><li>Advertising something </li></ul><ul><li>Answering a question </li></ul><ul><li>Asking a question </li></ul><ul><li>Augmenting a previous post </li></ul><ul><li>Calling for action </li></ul><ul><li>Disclosing personal information </li></ul><ul><li>Distributing media </li></ul><ul><li>Expressing agreement </li></ul><ul><li>Expressing criticism </li></ul><ul><li>Expressing support </li></ul><ul><li>Expressing surprise </li></ul><ul><li>Giving a heads up </li></ul><ul><li>Responding to criticism </li></ul><ul><li>Giving a shout-out </li></ul><ul><li>Making a joke </li></ul><ul><li>Making a suggestion </li></ul><ul><li>Making an observation </li></ul><ul><li>Offering a greeting </li></ul><ul><li>Offering an opinion </li></ul><ul><li>Putting out a wanted ad </li></ul><ul><li>Rallying support </li></ul><ul><li>Recruiting people </li></ul><ul><li>Showing dismay </li></ul><ul><li>Soliciting comments </li></ul><ul><li>Soliciting help </li></ul><ul><li>Starting a poll </li></ul><ul><li>Validating a position </li></ul>
  44. 44. Standard classifications of videos <ul><li>Advertisement </li></ul><ul><li>Animation </li></ul><ul><li>Demonstration </li></ul><ul><li>Event/Performance </li></ul><ul><li>Fiction </li></ul><ul><li>Film </li></ul><ul><li>Home Video </li></ul><ul><li>Instructional Video </li></ul><ul><li>Interview </li></ul><ul><li>Lecture </li></ul><ul><li>Montage </li></ul><ul><li>Music Video </li></ul><ul><li>News Broadcast </li></ul><ul><li>Promotional Video </li></ul><ul><li>Sightseeing/Tour </li></ul><ul><li>Slideshow </li></ul><ul><li>Speech </li></ul><ul><li>Television Show </li></ul><ul><li>Video Log </li></ul>
  45. 45. Your tool box also needs to include: <ul><ul><li>3. A way to measure engagement </li></ul></ul><ul><ul><li>The conversation index </li></ul></ul><ul><ul><li>Links/trackbacks </li></ul></ul><ul><ul><li>Time spent </li></ul></ul><ul><ul><li>Relationship studies </li></ul></ul><ul><ul><li>Questionnaires </li></ul></ul>
  46. 46. Share of conversation vs share of engagement Page 2 2 1 2 1 6 5 3 1 1 1 1 1 1 1 1 1 1 2 2 2 1 2 1 2 1 4 2 1 4 2 1 1 4 1 6 7 6 2 2 2 2 1 3 2 3 1 0 2 4 6 8 10 12 14 16 18 20 Faculty Students Research, Physical Sciences Courses Research, Earth Sciences Projects, Non - Research Financials Alumni Topics Research, Life Sciences Staff Admissions Legal News Other Research, Agriculture Policies Institution, Overall Campus Life Research, Social Sciences Share of Subject Peer 1 Michigan State Peer 2 Peer 3 Peer 4 15.3% 68.7% 100.0% 4.4% 33.3% 96.8% 28.6% 34.9% 12.5% 43.3% 28.6% 13.0% 38.3% 100.0% 23.6% 66.7% 6.3% 28.6% 20.8% 2.3% 95.6% 33.2% 5.8% 28.6% 100.0% 86.8% 13.0% 31.0% 22.1% 3.2% 71.4% 43.5% 18.8% 94.2% 56.7% 14.2% 13.2% 53.2% 28.4% 21.1% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Admissions Alumni Topics Campus Life Community Relations Courses Events Faculty Financials Institution, Overall Inventions Legal News Other Partnerships Policies Projects, Non - Research Research, Agriculture Research, Earth Sciences Research, Life Sciences Research, Other Research, Physical Sciences Research, Social Sciences Staff Students Share of Engagement by Subject - ,External Blogs Peer 1 Michigan State Peer 2 Peer 3 Peer 4
  47. 47. The vast majority of discussion in external blogs is neutral. Page
  48. 48. One definition of engagement <ul><li>% increase or decrease in unique visits </li></ul><ul><li>In the past  month,  what % of all sessions represent more than 5 page views </li></ul><ul><li>% of sessions that are greater than 5 minutes in duration </li></ul><ul><li>% of visitors that come back for more than 5 sessions </li></ul><ul><li>% of sessions that arrive at your site from a Google search, or a direct link from your web site or other site that is related to your brand </li></ul><ul><li>% of visitors that become a subscriber </li></ul><ul><li>% of visitors that download something from the site </li></ul><ul><li>% of visitors that provide an email address </li></ul>Courtesy of Eric Peterson
  49. 49. Your tool box also needs to include: <ul><li>5. Measuring ROI </li></ul><ul><ul><li>Membership </li></ul></ul><ul><ul><li>Registrations </li></ul></ul><ul><ul><li>WebSide story/Web trends/Clicktraks </li></ul></ul><ul><ul><li>Sales tracking </li></ul></ul><ul><ul><li>Marketing Mix Modeling </li></ul></ul>
  50. 50. <ul><li>Research without insight is just trivia </li></ul><ul><li>What works, what doesn’t </li></ul><ul><li>What needs to be done? </li></ul><ul><li>What are you communicating? </li></ul><ul><li>What tools work best? </li></ul>Step 7: Analysis
  51. 51. Data mining the numbers you have <ul><li>Look for failures first </li></ul><ul><li>Then look for exceptional success </li></ul><ul><li>Compare to last month, last quarter, last year </li></ul><ul><li>Figure out what worked and what didn’t work </li></ul>
  52. 52. Actionable Conclusions A sk for money Get C ommitment Manage T iming I nfluence decisions Get O utside help Just Say N o
  53. 53. Measuring doesn’t have to be complex High Impact Low Cost High Costs Low Impact Blogging Intentional Leaks SEO-optimized press releases e-newsletter Advertising Press conference Party
  54. 54. Take action and measure again <ul><li>Make sure data is ready when you need it </li></ul><ul><li>Work around regular reporting schedules </li></ul><ul><li>Keep questions and criteria consistent </li></ul>Benchmark Conclusions Report
  55. 55. Mistakes made most often in the measurement process <ul><li>Lack of agreed upon goals </li></ul><ul><li>Questionable methodology </li></ul><ul><li>Unclear definition of tone/sentiment </li></ul><ul><li>Incorrect search strings </li></ul><ul><li>Incomplete or out-of-date media list (no CGM) </li></ul><ul><li>Incorrect circulation figures </li></ul><ul><li>New products/spokespeople not added to analysis parameters </li></ul><ul><li>Promising a Jaguar on a bicycle budget </li></ul><ul><li>Not allowing enough time to do measurement right </li></ul><ul><li>Insufficient so what – looking at the trees not the forest </li></ul>
  56. 56. Thank You! <ul><li>For more information on measurement, read my blog: or give me your card and we’ll send you The Measurement Standard, </li></ul><ul><li>To start developing your own dashboard or for a copy of this presentation go to: http:// </li></ul><ul><li>Or call me at 1-603-868-1550 </li></ul><ul><li>Follow me on Twitter: @kdpaine </li></ul><ul><li>Friend me on Facebook: Katie Paine </li></ul><ul><li>Skype: KDPaine </li></ul>