FCN Presentation - Measuring Communication - Nov 2012

1,135 views

Published on

MITRE Principal Consultant Jeff Brooke shared a presentation on how to think about measurement in everyday communication planning.

Published in: Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,135
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
20
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

FCN Presentation - Measuring Communication - Nov 2012

  1. 1. Measuring Communication -Simple Approaches That CanChange Your ImagePresentation to theFederal Communicators NetworkJeffrey Brooke, ABCPrincipal ConsultantOrganizational Communication & Change ManagementJbrooke@MITRE.orgP r e s en tati on a p p r ove d f o r P u b l i c Re l e a se :12 -46 11 . Di s t r i bu tio n Un l i mi t ed*ABC – Accredited in Business Communication by the International Association of Business Communicators © 2012 The MITRE Corporation. All rights reserved.
  2. 2. |2|Agenda Logistics – 30 minute presentation – Q&A – via text box – Presentation will be forwarded Illustration - a typical communicator dilemma What to measure – Conceptual framework for measurement How to measure – Other examples and tools
  3. 3. |3|Communication measurementis a way of thinking Measurement is about: – Gathering data – To tell a story – That allows people to make decisions Not about: – Complicated math
  4. 4. |4|Start with internal communication Focusing on internal communication today An organization’s brand/reputation, comes from the inside out. Employees treat customers as management treats employees. – Effective communication improves employee engagement – Engaged employees deliver better for the customer Here’s an example.
  5. 5. |5|Scenario - Call center at “the Bureau”
  6. 6. |6|Call center agent, Jill, answers question andcompletes service applications for citizens
  7. 7. |7|Goal: communication with Jill will align herperformance with the Bureau’s goals DesiredCommunication interaction with citizens
  8. 8. |8|Head of call centers needs quality improved(response accuracy)
  9. 9. |9|His e-mails, speeches, newsletters have stressedquality – but haven’t produced change
  10. 10. | 10 |He wants your helpgetting his quality message through “What other communication should I do? I thought maybe a special newsletter each month focused just on quality, like industry best practices. Then maybe a contest between units and then spotlight the quality superstars in the newsletter each month. What do you think?”
  11. 11. | 11 |Stepping back for a moment…The communication flow – classic model Communication FlowSender Message Channel Receiver Results Noise Feedback
  12. 12. | 12 |How does the basic model work in ourlarge and complex organizations?
  13. 13. | 13 |Organizational roles that tieto steps in the model Communication FlowSender Message Channel Receiver Results Noise Feedback Communicator Stakeholder Perspective Client Program Owner Perspective (e.g. employees) Perspective
  14. 14. | 14 |Planned communication in an organizationSteps that operationalize the model Communicator Stakeholder Perspective Client Program Owner Perspective (e.g. employees) Perspective Plan Deploy Frame of Communication Stakeholder Results Communication Reference Behaviors Activities Outcomes Communication Flow
  15. 15. | 15 |Planning & measures are intertwinedPlanning starts at the end, works backward Communicator Stakeholder Perspective Client Program Owner Perspective (e.g. employees) Perspective Plan Deploy Stakeholder Communication Stakeholder Results Communication Assessment Behaviors Activities Outcomes (Frame of (Knowledge & Reference) Attitude) Planning Flow
  16. 16. | 16 |Communication planningExplore measures for all steps, not just the end Communicator Stakeholder Perspective Client Program Owner Perspective (e.g. employees) Perspective Plan Deploy Stakeholder Communication Stakeholder Results Assessment Outcomes Behaviors (Frame of (Knowledge & Reference) Attitude) Planning Flow Measures Measures Measures Measures Measures Measures
  17. 17. | 17 |How to Measure CommunicationOverview of Key Tools Qualitative – Interviews – Focus Groups (and Structured Group Interviews) – Polls Quantitative – Surveys – Counting
  18. 18. | 18 |Qualitative Data Gathering Top tools Best for Examples understanding• Focus groups • Depth—by probing • What people know• Interviews• Polls • To understand the • How people feel range of issues on a about a topic topic • Frame of reference • Story-telling nature of on a topic dialogue can uncover forces around change—enablers and barriers to change
  19. 19. | 19 |Quantitative Data Gathering Top tools Best for ExamplesSurveys Prediction • Comparison over • Breadth—how widely time, and between shared is an opinion organizations • Precise, for tracking • Correlation analysis changes over time. for predictionCounting Outputs • Number of articles • Event attendance • Percentage of articles dedicated to each strategy
  20. 20. | 20 |Communication planning1. Results Stakeholder Perspective Client Program OwnerCommunicator Perspective (e.g. employees) Perspective Results Plan What to measure How to measuresQuestions to ask: Examples: Examples:Outcomes: • Customer satisfaction • Survey• What needs to change and why? • Budget requirements • Financial statements• How will you know if you’ve • Congressional • GAO/committee accomplished your goal? requirement feedbackOutputs: • Number • Counting• What changes in outputs are needed • Quality • Sampling to achieve the goals? • Cost • Cost accounting• How will you measure progress?
  21. 21. | 21 |ResultsOutcomes and OutputsWill be measured by the program owner Outcome: Higher satisfaction from – GAO/Congress – Citizens Output: Higher quality calls
  22. 22. | 22 |Communication planning2. Targeted stakeholder behavior Stakeholder Perspective Client Program OwnerCommunicator Perspective (e.g. employees) Perspective Stakeholder Behaviors Plan What to measure How to measureQuestions to ask Examples Examples• Who needs to do what differently to Actual behaviors Business process produce the output changes? • Applying new training measures• How will you measure the behaviors? • Using new procedures• What will change to enable the new behaviors? Behavioral indicators Survey questions
  23. 23. | 23 |BehaviorsNot clearRon assumes the agents have leeway.Sees his communication as motivation.
  24. 24. | 24 |What to tell Ron?One approach “Yes! I can help you with a quality-focused newsletter and other ways to get out your message.”“AND— – how do you feel about first letting me do some feedback sessions to get a sense of what will and won’t work. I might find better options for you to consider.”
  25. 25. | 25 |Communication planning3. Stakeholder assessment – change readiness Stakeholder Perspective Client Program OwnerCommunicator Perspective (e.g. employees) Perspective Stakeholder Assessment 1. Frame of Reference for the change Plan What to measure How to measureQuestions to ask the program owner Examples ExamplesClient hypothesis about reaction to the Validate with the Focus groupchange: stakeholder: Interview• What changes will enable the new • Will you be able to do behaviors or remove barriers? this, in light of the• What is the stakeholder’s frame of planned support? reference on what you are asking them to do differently?• What might they see as enabling or blocking them?
  26. 26. | 26 |Conduct stakeholder assessmentFocus groups at call centers
  27. 27. | 27 |Finding:Quality measured quarterly, at the office level
  28. 28. | 28 |Finding: Quantity Metric–Quantity measured every 30 minutes, at the group level Every 30 minutes, Group Level
  29. 29. | 29 | Frame of reference determines what Jill will notice, how she will interpret itCommunication Frame of Reference (Enablers & Barriers) Resulting interaction with citizens
  30. 30. | 30 |Stakeholder assessment:Focus groups determine Jill’s frame of referenceElements of frame ofreference• Leadership Frame of• Mission & Vision Reference• Culture• Mgt. practices (Enablers & Resulting Barriers) interaction• Policies & procedures with citizens• Work unit climate• Individual needs• Task & individual skills (See Burke-Litwin model)
  31. 31. | 31 |What to tell Ron?  What would you tell Ron?  What would you recommend to him as a next step?  Might this change how Ron sees you?
  32. 32. | 32 |Communication planning3. Stakeholder assessment – communication fitness Stakeholder Perspective Client Program OwnerCommunicator Perspective (e.g. employees) Perspective Stakeholder “Fitness” measures. Assessment What do they think of 2. Frame of Reference on our stuff? Does it “fit” communication their needs?” Plan What to measure How to measureDuring deployment: “Fitness” of communication During deploymentHow to assess the effectiveness of eachcommunication activity during Examples: • Surveydeployment. - Clarity • Poll - Right channels • Focus groupPast experience: - AmountUsefulness of communication channels - Timing Past experienceand approaches with this stakeholder. - Exposure • Compilation of previous data • Communication audit
  33. 33. | 33 |Example:Ongoing assessment of "fitness" of messages Weekly survey of employees and managers Hottest topic of the week Statistically reliable sample
  34. 34. | 34 |Example "Fitness" measuresMessage penetration and usefulness
  35. 35. | 35 |Example "Fitness" measuresTimeliness, satisfaction, importance, amount Q4. To what extent do you feel this communication was timely? Timely Not timely (Running average for "Timely" is 82%) Q5. Overall, how satisfied were you with this communication? Satisfied Dissatisfied(Running average for "Satisfied" is 86%) Q6. How important is the topic of"new system procedures" to you? Important Unimportant(Running average for "Important" is 65%) Q7. How would you rate the amount of communication youve About right Too much Too little received on this topic? (Running average for "About right" is 68%) 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
  36. 36. | 36 |Communication planning4. Communication outcomes Stakeholder Perspective Client Program OwnerCommunicator Perspective (e.g. employees) Perspective Communication Outcomes Knowledge & Attitude Plan What to measure How to measureWhat knowledge and attitude will Examples Examplesproduce the targeted behaviors, as seenthrough the stakeholder’s frame of • Knowledge and attitudes • Surveyreference? (perceived) • Poll • Actual knowledge • Focus group
  37. 37. | 37 |Example knowledge measure
  38. 38. | 38 |Example measure of feelings To what extent do you feel the new Performance Plan system will help the agency reach its goals? 40% 38% 35% 31% 30% 25% 25% 25% 23% 20% 20% Employees 17% Sups 15% 13% 10% 9% 5% 0% 0% A great deal Somewhat A little None I’m not sure yet.
  39. 39. | 39 |Example: Measuring knowledge and feelingsfor course correction Situation: – Executive Committee interactive video town hall Topic: – Transformation update – two years into effort  Update on each of five change strategies  Scripts weighted heavily on “why this strategy is important” Measurement Plan – Pre-event random survey – two questions for each strategy  Know: “How important do you feel this is to the agency transformation?”  Feel: “How effectively do you think this strategy is being implemented?” – Recommend possible script changes – Post-event repeat survey
  40. 40. | 40 |Measuring knowledge and feelingsPre-event survey (two days prior)Several strategies showed:• good scores for importance (feeling)• lower scores for implementation (knowledge)Comments indicated lack ofknowledge about earlyimplementation efforts andresults.Updated script for thesestrategies to weight more heavilyon pilots and earlyimplementation efforts.
  41. 41. | 41 |Measuring knowledge and feelingsPost-event survey (one hour after event)Scripts adjustments weresuccessful.
  42. 42. | 42 |Communication planning5. Deploy communication activities Stakeholder Perspective Client Program Owner Communicator Perspective (e.g. employees) Perspective Deploy communication activities Plan What to measure How to measureWhat types of communication will best Communication Outputs Examplesconvey the targeted knowledge andattitudes? Examples: • Counting • # of articles and events • % of content supporting each business strategy • % of stakeholders who provide input on a change • # of awards for a desired behavior • # informal leader conversations
  43. 43. | 43 | Targeting value change withPrimary communication activities messages Content, such as intranet pages, publications and formal presentations. Interaction, such as input to decisions, dialogue with managers, and collaboration with colleagues. Leadership example, such as how leaders signal what they value through promotions, recognition, budget, their time and informal comments.
  44. 44. | 44 |Output measuresContent analysis – weekly data collection % of total editorialThemes Totals space E-Bulletin Managers Bulletin Magazine WebbieTotal Length 66586 37702 15582 7289 6013Vision 222 0% 0 0% 222 1% 0 0% 0Mission 239 0% 239 1% 0 0% 0 0% 0Values 12026 18% 2249 6% 498 3% 6165 85% 336Value: Integrity 208 0% 0 0% 0 0% 0 0% 0Value: Teamwork 10712 16% 2053 5% 498 3% 6165 85% 336Value: Commitment 1423 2% 196 1% 0 0% 0 0% 0Value: Dependability 389 1% 0 0% 0 0% 0 0% 0Goals 39169 59% 16872 45% 11046 71% 2330 32% 1911Goal 1: Develop a flexibledigital… 3421 5% 1092 3% 0 0% 1284 18% 0Goal 2: Prepare andequip… 6601 10% 3694 10% 1076 7% 1046 14% 0Goal 3: Continue totransition the… 784 1% 567 2% 0 0% 0 0% 0Goal 4: To serve thepublics needs by… 892 1% 342 1% 352 2% 0 0% 23Goal 5: Develop anintegrated enterpriseapproach to… 32496 49% 14603 39% 10694 69% 0 0% 1888Quotes 16629 25% 5112 14% 3973 25% 7090 97% 315Photos/Images 497 55 9 24 25 188 63 10 60Total articles or services 786
  45. 45. Example output measure: Percentage of total editorial space,by category, for all six channels50%45%40%35%30% 28% 27% 25%25%20% 17% 14%15%10% 6%5% 1% 0%0% Vision Mission Values Goals Strategic Administrative Recognition Quotes Priorities
  46. 46. Example output measure: Percentage of total editorial space,by category, for E-Bulletin50%45%40%35%30% 25%25% 23% 19%20%15% 13%10% 8%5% 3% 1% 1%0% Vision Mission Values Goals Strategic Administrative Recognition Quotes Priorities
  47. 47. Example output measure: Percentage of total editorialspace, deep dive in “goals” coverage50%45%40%35%30% 28% 25%25%20%15%10% 5% 5% 3% 2% 2% 0% Goals Total Goal 1: Develop Goal 2: Prepare Goal 3: Continue Goal 4: To serve Goal 5: Develop a… and equip… to transition the the publics an enterprise … needs by… approach to…
  48. 48. | 48 |Communication MeasuresIntegrated with Communication Planning Communicator Stakeholder Client Program Owner Perspective Perspective Perspective 5. 3. 4. 2. 1.Communi-cation Tactics to Stakeholder Communication Stakeholder Results Deploy Assessment Outcomes Behaviorsplanelements (Frame of Reference)Measures Communication Communication Knowledge Specific Outcomes Outputs Fitness and behaviors and outputsfor eachelement Feelings needed Changes that will enable the behaviors Change Readiness
  49. 49. | 49 |Does anyone really care?Measures can move your “brand” from overhead, toward operations.Communications Executive Council survey: 2012 vs. 2007 data Responsibilities of communication function – #1 increase: Social Media 80% (from zero) – #2 increase: Measurement 82% (from 64%) Top communication strategic priorities for 2013 – #1: Employee engagement – #2: Improving leadership communication – #3: Building company presence in social media Strategies to improve communication function effectiveness – #1 Improved consultative partnership with business leaders – #2 Deeper business acumen on the team CEC 2012 report: Taking Your Communications Budgeting and Planning to the Next Level
  50. 50. | 50 |What next? Deeper topics – Design and use tools for valid and reliable data – Communication audits Broader topics – Looking at measures across organizations and time – What communication function can do to add value Feedback on today’s presentation Reach out to me and FCN community Q&A

×