Measurement mini workshop


Published on

Published in: Technology, Business
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Welcome – count down
  • In KD Paine’s decades of experience, the measurement process looks like this – no matter what you’re measuring to get at measuring what matters … The strategy piece includes creating a plan, it might include research (audience research or benchmarking w/peers). For some of you, your pilot might be “prospective” – as part of planning a future campaign or program. Or it might be analyzing the past – something you’ve already implemented but want to go back and measure results/value for planning next reiteration or to do present to senior management to pave the way for more incorporating social media into your program delivery or communications channel. The do part – your tactics, tools – The measure is your data collection – as part of your strategy you define this – it includes analysis – you don’t just collect data, you figure out what it means in terms of improving your results – and then you adapt your plan/strategy and the cycle continues.Make sense? Does this sound familiar? Is this way you measure other programs or communications efforts that do not include social media?
  • These are the basic steps that you will follow for your measurement pilot …. Or any project that involves social media ..The challenge here is that you are not just measuring, but measuring around your practice or strategy.The pilot is making it very simple.
  • Start with your results and incorporate social media into it .. Keep asking to what end? Why is this important? AwarenessReputationRelationshipsActionsDonationsVolunteersBehavior ChangePartner Engagement
  • Theory of Change series of if, then statements that are laid out as a path …This might be a useful approach for those of you working more on policy or program outcomes – and how to integrate social media For example:Kansas Action for Children has helped shape policy that puts children first.
  • What are you going to do to get those results with an integrated social media plan?SMART ObjectivesMeasurement PlanAudience Definition: Target Audience InfluencersEnvironmentMessaging/ContentChannelsBudgetTimeline
  • A strategy is a plan of action designed to achieve a particular objective. Tactics and tools are the means by which a strategy is carried out – these are on the ground methods to executive strategy. Ways to be strategic …Smart Objectives many by when?Results, Capacity, or TacticsGuides, not report cards
  • Key performance indicators are 3-5 metrics that are critical to knowing you have been reach your result or not ….
  • Key Performance IndicatorsMetrics
  • Key Performance Indicators: Key performance indicators are the most important metrics to understand how successful your are ..This is the conversation to have with your team …..Metrics
  • USA for UNHCR (my client) is a 501(c)(3) headquartered in Washington, D.C. They raise funds and awareness in the United States for the lifesaving work that more than 6,000 staffers of Geneva-based UNHCR undertake for refugees around the world, 24/7.   Given that there are more than 43.3 million refugees worldwide, this is pretty important work… and did I mention lifesaving? (Just look at what’s happening in Somalia right now.) The challenge is that typically refugees are so far removed from our “normal” lives that we don’t automatically think of giving to organizations that work to support them.USA for UNHCR created the Blue Key campaign as a way to drive awareness of this global issue in the US. The $5 blue key pin or pendant symbolizes the one thing most of us have and that refugees don’t: a key to their own home.My role is to work with USA for UNHCR on building out the digital and social media aspects of the campaign, particularly from an online community-building point of view. That is where the Blue Key Champion strategy came in; to bring on board US-based bloggers who are active online, and cause-passionate, to help spread the word and, hopefully, get more keys ordered.
  • The Blue Key site was only launched in December 2010, and its social/digital aspects were relatively new, so there was not a lot of data to base KPIs on. Overall, when we went into the first phase of the campaign, we had two goals: to secure at least three Blue Key Champions, and  to get 6,000 keys ordered between May 9 and June 20 (World Refugee Day).
  • to implement your social media pilot – including measurement time! We know that most of the software/platforms are free, so the cost is going to be an investment in staff time. What is the level of time?What is the opportunity cost?Are you adding resources or shifting them?What are the consequences?How can you be as efficient as possible in your execution time? Data collection time?You’ll need this for reporting … and also helps you get better and more efficient
  • measurement tools are the techniques you will use to collect data – the key performance indicators – These are covered in a later session. Remember – any tool is useless unless it covers the sort of data that help you evaluate progress towards to your goalsContent analysis of social or traditional media- Primary audience via online, mail, or phone surveys- AnalyticsAffordable ….
  • is a continuous improvement process – feedback loopsAny measurement to work, you need to assess results, make changes, and see if those changes had impact, make more changes, and so on. Valid, actionable conclusionsAvoid focusing on only the most exciting, look for failures …You need a regular reporting schedule – not just do a data dump at the end – important for buy-in
  • These are the basic steps that you will follow for your measurement pilot ….
  • These are the basic steps that you will follow for your measurement pilot ….
  • that it is “not fun” Creating is more appealing then analysis to most people who work in nonprofits. Sadly, many people are more likely to build instead of measure even if they are building on a fault line.  That comes as no surprise because most people don’t decide to go into non-profit work because they want to sit and look at spreadsheets all day. Even if there is a staff position for analytics, it tends to be in finance not communications. And not all communications departments invest in professional development in measurement for staff members responsible.  1/in/photostream/
  • of consequences The nonprofit world can be a scary place in a fragile economy.No one wants to take risks if the results could lose you a promotion or worse, lose your job. So inherently measuring results is a scary prospect.  What if we discover that our campaign didn’t get the results we thought it would or even worse, that our precious time was wasted.   
  • often, there is a fear that by setting objectives, you will be held accountable for some else’s unreasonable expectations. This is the result of “SMART” objectives – which we will discuss in depth in the next chapter. Not setting measurable objectives is akin to running a marathon in an earthquake blindfolded.
  • is a common misperception that valid measurement requires highly trained specialists. Or if it is done in house, it is considered time-consuming and just one more chore on a seemingly endless list. However, if nonprofits match their data collection approaches to their capacity, chances are measurement will save time and resources. It is better to start small with a few data points and strategically add metrics to the spreadsheet dashboard. Nonprofits need to think of the measurement task as training for marathon.   If you are just starting to get into shape, you need to build up your strength before you run a 35-mile marathon.
  • nonprofits default to the “measure everything” approach on the mistaken belief that more data (or even bogus data (is better than nothing. Worse they collect data “just in case” Either way, these results in bloated spreadsheets to which no one pays attention.  so a data-driven organization needs to be comfortable in just saying “No!” to too much data. In a recent interview, Jodi Nelson, who heads up the Bill & Melinda Gates Foundation’s Impact Planning and Improvement Unit, said, “Even when you’ve figured out the metrics, it doesn’t necessarily mean that the data you collect end up getting used. It may sound intuitive, but the key is figuring out how to guarantee the data are both relevant and used. You’d be amazed at how much data collection goes on in the world that produces reports that are never read or used to do anything differently. “
  • 1. Visualize success and failure2. Spend more time identifying what you want to measure, not how to measure it 3. Measure in context – don’t ever collect data unless you have SMART objectives and a benchmark to measure it against.4. Don’t wait until the end of the pilot to gather and analyze your data. Build measurement in from the start and then collect in regular weekly or monthly increments.5. Don’t ever just shovel data over the fence and onto your executive director’s desk. Share high level insights, make recommendations that can spark ideas on improvement6. Less is more. Discipline yourself to measure only one objective or one channel and don’t collect more than data points.7 . Start with a small measurement pilot that helps your organization create a habit of collecting, analyzing, and applying data. We describe in this great detail in the next sections.
  • Measurement mini workshop

    1. Measurement Trick or Treat<br />Beth Kanter, Beth’s Blog -- October, 19, 2011<br />
    2. KD Paine<br />Measurement<br />Basics<br />
    3. KD Paine’s Basic Steps<br />Define ResultsStrategy BenchmarksMetrics – KPICosts<br />Select Right Tool To Collect DataTurn Data in Action <br />
    4. Define Results<br />Keep asking to what end?<br />AwarenessReputationRelationshipsActionDonationsVolunteersBehavior Change<br />
    5. Theory of Change Lite: To What End? <br />Kidds<br />
    6. Key Result: Kids get health care<br />Gets better at communication, proactive planning, partner engagement, message discipline and social media integration<br />More and better partners, more and better relationships with reporters and more or better policy maker relationships<br />Communications strategy has more impact<br />More attention from policymakers<br />Policies are adopted<br />Kids get health care coverage<br />
    7. StrategySMART ObjectivesMeasurement PlanAudience Definition:<br /> Target Audience InfluencersEnvironmentMessaging/ContentChannelsBudgetTimeline<br />
    8. SMART objectives<br />1.How many?<br />2. By when?<br />
    9. Integrated Social Media StrategyFacebook Measurement Pilot<br />
    10. Target Audience: AdvocatesIntermediary Audience: Policy Makers<br />Advocates send 200 emails, make 300 phone calls, and post on FB page to endorse an issue related to ACAIdentify and establish relationships with “influencers” (lawmakers, parents, etc) through Facebook so they spread stories of Children’s Health Care by 1/2012<br />Increase number of fans for Facebook page by 10% by 2012<br />Increase engagement with Facebook fans to 3 comments per post by 2012<br />Repurpose policy paper content into series of regular Facebook posts by 2012 <br />Two aligned partners collaborate on an online event on Facebook to draw attention to issue by 2012<br />
    11. Benchmarks<br />"The greatest danger for most of us is not that our aim is too high and we miss it but that it is too low and we reach it." Michelangelo<br />
    12. DYI Benchmark Study of Peers<br />
    13. Use industry studies<br />
    14. Define Your Key Performance Indicators <br />What are the metrics you will report with? <br />
    15. There is no shortage of metrics you could use …….<br />27.  Social bookmarks<br />28.  Subscriptions (RSS, podcasts, video series)<br />29.  Pageviews (for blogs, microsites, etc)<br />30.  Effective CPM based on spend per impressions received<br />31.  Change in search engine rankings for the site linked to through social media<br />32.  Change in search engine share of voice for all social sites promoting the brand<br />33.  Increase in searches due to social activity<br />34.  Percentage of buzz containing links<br />35.  Links ranked by influence of publishers<br />36.  Percentage of buzz containing multimedia (images, video, audio)<br />37.  Share of voice on social sites when running earned and paid media in same environment<br />38.  Influence of consumers reached<br />39.  Influence of publishers reached (e.g., blogs)<br />40.  Influence of brands participating in social channels<br />41.  Demographics of target audience engaged with social channels<br />42.  Demographics of audience reached through social media<br />43.  Social media habits/interests of target audience<br />44.  Geography of participating consumers<br />45.  Sentiment by volume of posts<br />46.  Sentiment by volume of impressions<br />47.  Shift in sentiment before, during, and after social marketing programs<br />48.  Languages spoken by participating consumers<br />49.  Time spent with distributed content<br />50.  Time spent on site through social media referrals<br />1.     Volume of consumer-created buzz<br />2.     Amount of buzz 3.     Shift in buzz over time<br />4.     Buzz by time of day <br />5.     Seasonality of buzz<br />6.     Competitive buzz<br />7.     Buzz by category <br />8.     Buzz by social channel 9.     Buzz by stage in purchase funnel <br />10.  Asset popularity <br />11.  Mainstream media mentions<br />12.  Fans<br />13.  Followers<br />14.  Friends<br />15.  Growth rate of followers friends<br />16.  Rate of pass-along<br />17.  Change in rates <br />18.  Second-degree reach <br />19.  Embeds / Installs<br />20.  Downloads<br />21.  Uploads<br />22.  User-initiated views <br />23.  Ratio of embeds to views<br />24.  Likes / favorites<br />25.  Comments<br />26.  Ratings<br />Source: 100 Ways to Measure Social Media by David Berkowitz <br />
    16. There is no shortest of metrics you could use …….<br />75.  Customers assisted<br />76.  Savings per customer assisted through direct social media interactions compared to other channels (e.g., call centers, in-store)<br />77.  Savings generated by enabling customers to connect with each other<br />78.  Impact on first contact resolution (FCR) (hat tip to Forrester Research for that one)<br />79.  Customer satisfaction<br />80.  Volume of customer feedback generated<br />81.  Research & development time saved based on feedback from social media<br />82.  Suggestions implemented from social feedback<br />83.  Costs saved from not spending on traditional research<br />84.  Impact on online sales<br />85.  Impact on offline sales<br />86.  Discount redemption rate<br />87.  Impact on other offline behavior (e.g., TV tune-in)<br />88.  Leads generated<br />89.  Products sampled<br />90.  Visits to store locator pages<br />91.  Conversion change due to user ratings, reviews<br />92.  Rate of customer/visitor retention<br />93.  Impact on customer lifetime value<br />94.  Customer acquisition / retention costs through social media<br />95.  Change in market share<br />96.  Earned media's impact on results from paid media<br />97.  Responses to socially posted events<br />98.  Attendance generated at in-person events<br />99.  Employees reached (for internal programs)<br />100.  Job applications received<br />51.  Method of content discovery<br />52.  Clicks<br />53.  Percentage of traffic generated from earned media<br />54.  View-throughs<br />55.  Number of interactions<br />56.  Interaction/engagement rate<br />57.  Frequency of social interactions per consumer<br />58.  Percentage of videos viewed<br />59.  Polls taken / votes received<br />60.  Brand association<br />61.  Purchase consideration<br />62.  Number of user-generated submissions received<br />63.  Exposures of virtual gifts<br />64.  Number of virtual gifts given<br />65.  Relative popularity of content<br />66.  Tags added<br />67.  Attributes of tags 68.  Registrations from third-party social logins <br />69.  Registrations by channel (<br />70.  Contest entries<br />71.  Number of chat room participants<br />72.  Wiki contributors<br />73.  Impact of offline marketing/events<br />74.  User-generated content created that <br />75.  Customers assisted<br />What matters most?<br />Source: 100 Ways to Measure Social Media by David Berkowitz <br />
    17. Cost<br />
    18. The right tool for the job<br />Content analysis of social or traditional media- Primary audience via online, mail, or phone surveys- Analytics<br />
    19. GRIST.ORG<br />KPI: Footprint: The reach of their activities, both online and offline<br />ViewsGoogle Analytics<br />Engagement: Readers engage with their content<br />Comments, Virility, RetweetsChart BeatFacebook InsightsTwitter Crowd<br />Individual Behavior Change: Impact on users behaviors, purchase decisions, and daily lives that are in line with sustainability<br />Questions about habits<br />Survey Monkey<br />Societal Change: Impact on society, policy discussions, and conversations that advance sustainable practices.<br />Anecdotal stories<br />
    20. Turn Data in Action<br />
    21. KD Paine’s Basic Steps<br />Define ResultsStrategy BenchmarksMetrics – KPICosts<br />Select Right Tool To Collect DataTurn Data in Action <br />
    22. Measure Value<br />Direct RevenueCost Savings<br />Traffic/Search Engine Rankings<br />Social Capital<br />
    23. Why does calculating give us the collywobbles?<br />The 5 Deadly Fears That Get in the Way of Good Measurement<br />
    24. Fear that isn’t fun<br />
    25. Fear of Consequences<br />
    26. Fear of objectives<br />
    27. Fear of being overwhelmed<br />
    28. Fear of the data swamp<br />
    29. The 7 Habits of Highly Effective Nonprofits<br />1. Visualize success and failure<br />2. Spend more time identifying what you want to measure, not how to measure it <br />3. Measure in context – don’t ever collect data unless you have SMART objectives<br />4. Don’t wait until the end of project to collect data<br />5. Don’t ever just shovel data over the fence and onto your executive director’s desk <br />6. Less is more<br />
    30. Thank you<br /><br /><br /><br />