|2|Agenda Logistics – 30 minute presentation – Q&A – via text box – Presentation will be forwarded Illustration - a typical communicator dilemma What to measure – Conceptual framework for measurement How to measure – Other examples and tools
|3|Communication measurementis a way of thinking Measurement is about: – Gathering data – To tell a story – That allows people to make decisions Not about: – Complicated math
|4|Start with internal communication Focusing on internal communication today An organization’s brand/reputation, comes from the inside out. Employees treat customers as management treats employees. – Effective communication improves employee engagement – Engaged employees deliver better for the customer Here’s an example.
|6|Call center agent, Jill, answers question andcompletes service applications for citizens
|7|Goal: communication with Jill will align herperformance with the Bureau’s goals DesiredCommunication interaction with citizens
|8|Head of call centers needs quality improved(response accuracy)
|9|His e-mails, speeches, newsletters have stressedquality – but haven’t produced change
| 10 |He wants your helpgetting his quality message through “What other communication should I do? I thought maybe a special newsletter each month focused just on quality, like industry best practices. Then maybe a contest between units and then spotlight the quality superstars in the newsletter each month. What do you think?”
| 11 |Stepping back for a moment…The communication flow – classic model Communication FlowSender Message Channel Receiver Results Noise Feedback
| 12 |How does the basic model work in ourlarge and complex organizations?
| 13 |Organizational roles that tieto steps in the model Communication FlowSender Message Channel Receiver Results Noise Feedback Communicator Stakeholder Perspective Client Program Owner Perspective (e.g. employees) Perspective
| 14 |Planned communication in an organizationSteps that operationalize the model Communicator Stakeholder Perspective Client Program Owner Perspective (e.g. employees) Perspective Plan Deploy Frame of Communication Stakeholder Results Communication Reference Behaviors Activities Outcomes Communication Flow
| 15 |Planning & measures are intertwinedPlanning starts at the end, works backward Communicator Stakeholder Perspective Client Program Owner Perspective (e.g. employees) Perspective Plan Deploy Stakeholder Communication Stakeholder Results Communication Assessment Behaviors Activities Outcomes (Frame of (Knowledge & Reference) Attitude) Planning Flow
| 16 |Communication planningExplore measures for all steps, not just the end Communicator Stakeholder Perspective Client Program Owner Perspective (e.g. employees) Perspective Plan Deploy Stakeholder Communication Stakeholder Results Assessment Outcomes Behaviors (Frame of (Knowledge & Reference) Attitude) Planning Flow Measures Measures Measures Measures Measures Measures
| 17 |How to Measure CommunicationOverview of Key Tools Qualitative – Interviews – Focus Groups (and Structured Group Interviews) – Polls Quantitative – Surveys – Counting
| 18 |Qualitative Data Gathering Top tools Best for Examples understanding• Focus groups • Depth—by probing • What people know• Interviews• Polls • To understand the • How people feel range of issues on a about a topic topic • Frame of reference • Story-telling nature of on a topic dialogue can uncover forces around change—enablers and barriers to change
| 19 |Quantitative Data Gathering Top tools Best for ExamplesSurveys Prediction • Comparison over • Breadth—how widely time, and between shared is an opinion organizations • Precise, for tracking • Correlation analysis changes over time. for predictionCounting Outputs • Number of articles • Event attendance • Percentage of articles dedicated to each strategy
| 20 |Communication planning1. Results Stakeholder Perspective Client Program OwnerCommunicator Perspective (e.g. employees) Perspective Results Plan What to measure How to measuresQuestions to ask: Examples: Examples:Outcomes: • Customer satisfaction • Survey• What needs to change and why? • Budget requirements • Financial statements• How will you know if you’ve • Congressional • GAO/committee accomplished your goal? requirement feedbackOutputs: • Number • Counting• What changes in outputs are needed • Quality • Sampling to achieve the goals? • Cost • Cost accounting• How will you measure progress?
| 21 |ResultsOutcomes and OutputsWill be measured by the program owner Outcome: Higher satisfaction from – GAO/Congress – Citizens Output: Higher quality calls
| 22 |Communication planning2. Targeted stakeholder behavior Stakeholder Perspective Client Program OwnerCommunicator Perspective (e.g. employees) Perspective Stakeholder Behaviors Plan What to measure How to measureQuestions to ask Examples Examples• Who needs to do what differently to Actual behaviors Business process produce the output changes? • Applying new training measures• How will you measure the behaviors? • Using new procedures• What will change to enable the new behaviors? Behavioral indicators Survey questions
| 23 |BehaviorsNot clearRon assumes the agents have leeway.Sees his communication as motivation.
| 24 |What to tell Ron?One approach “Yes! I can help you with a quality-focused newsletter and other ways to get out your message.”“AND— – how do you feel about first letting me do some feedback sessions to get a sense of what will and won’t work. I might find better options for you to consider.”
| 25 |Communication planning3. Stakeholder assessment – change readiness Stakeholder Perspective Client Program OwnerCommunicator Perspective (e.g. employees) Perspective Stakeholder Assessment 1. Frame of Reference for the change Plan What to measure How to measureQuestions to ask the program owner Examples ExamplesClient hypothesis about reaction to the Validate with the Focus groupchange: stakeholder: Interview• What changes will enable the new • Will you be able to do behaviors or remove barriers? this, in light of the• What is the stakeholder’s frame of planned support? reference on what you are asking them to do differently?• What might they see as enabling or blocking them?
| 26 |Conduct stakeholder assessmentFocus groups at call centers
| 27 |Finding:Quality measured quarterly, at the office level
| 28 |Finding: Quantity Metric–Quantity measured every 30 minutes, at the group level Every 30 minutes, Group Level
| 29 | Frame of reference determines what Jill will notice, how she will interpret itCommunication Frame of Reference (Enablers & Barriers) Resulting interaction with citizens
| 30 |Stakeholder assessment:Focus groups determine Jill’s frame of referenceElements of frame ofreference• Leadership Frame of• Mission & Vision Reference• Culture• Mgt. practices (Enablers & Resulting Barriers) interaction• Policies & procedures with citizens• Work unit climate• Individual needs• Task & individual skills (See Burke-Litwin model)
| 31 |What to tell Ron? What would you tell Ron? What would you recommend to him as a next step? Might this change how Ron sees you?
| 32 |Communication planning3. Stakeholder assessment – communication fitness Stakeholder Perspective Client Program OwnerCommunicator Perspective (e.g. employees) Perspective Stakeholder “Fitness” measures. Assessment What do they think of 2. Frame of Reference on our stuff? Does it “fit” communication their needs?” Plan What to measure How to measureDuring deployment: “Fitness” of communication During deploymentHow to assess the effectiveness of eachcommunication activity during Examples: • Surveydeployment. - Clarity • Poll - Right channels • Focus groupPast experience: - AmountUsefulness of communication channels - Timing Past experienceand approaches with this stakeholder. - Exposure • Compilation of previous data • Communication audit
| 33 |Example:Ongoing assessment of "fitness" of messages Weekly survey of employees and managers Hottest topic of the week Statistically reliable sample
| 34 |Example "Fitness" measuresMessage penetration and usefulness
| 35 |Example "Fitness" measuresTimeliness, satisfaction, importance, amount Q4. To what extent do you feel this communication was timely? Timely Not timely (Running average for "Timely" is 82%) Q5. Overall, how satisfied were you with this communication? Satisfied Dissatisfied(Running average for "Satisfied" is 86%) Q6. How important is the topic of"new system procedures" to you? Important Unimportant(Running average for "Important" is 65%) Q7. How would you rate the amount of communication youve About right Too much Too little received on this topic? (Running average for "About right" is 68%) 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
| 36 |Communication planning4. Communication outcomes Stakeholder Perspective Client Program OwnerCommunicator Perspective (e.g. employees) Perspective Communication Outcomes Knowledge & Attitude Plan What to measure How to measureWhat knowledge and attitude will Examples Examplesproduce the targeted behaviors, as seenthrough the stakeholder’s frame of • Knowledge and attitudes • Surveyreference? (perceived) • Poll • Actual knowledge • Focus group
| 38 |Example measure of feelings To what extent do you feel the new Performance Plan system will help the agency reach its goals? 40% 38% 35% 31% 30% 25% 25% 25% 23% 20% 20% Employees 17% Sups 15% 13% 10% 9% 5% 0% 0% A great deal Somewhat A little None I’m not sure yet.
| 39 |Example: Measuring knowledge and feelingsfor course correction Situation: – Executive Committee interactive video town hall Topic: – Transformation update – two years into effort Update on each of five change strategies Scripts weighted heavily on “why this strategy is important” Measurement Plan – Pre-event random survey – two questions for each strategy Know: “How important do you feel this is to the agency transformation?” Feel: “How effectively do you think this strategy is being implemented?” – Recommend possible script changes – Post-event repeat survey
| 40 |Measuring knowledge and feelingsPre-event survey (two days prior)Several strategies showed:• good scores for importance (feeling)• lower scores for implementation (knowledge)Comments indicated lack ofknowledge about earlyimplementation efforts andresults.Updated script for thesestrategies to weight more heavilyon pilots and earlyimplementation efforts.
| 41 |Measuring knowledge and feelingsPost-event survey (one hour after event)Scripts adjustments weresuccessful.
| 42 |Communication planning5. Deploy communication activities Stakeholder Perspective Client Program Owner Communicator Perspective (e.g. employees) Perspective Deploy communication activities Plan What to measure How to measureWhat types of communication will best Communication Outputs Examplesconvey the targeted knowledge andattitudes? Examples: • Counting • # of articles and events • % of content supporting each business strategy • % of stakeholders who provide input on a change • # of awards for a desired behavior • # informal leader conversations
| 43 | Targeting value change withPrimary communication activities messages Content, such as intranet pages, publications and formal presentations. Interaction, such as input to decisions, dialogue with managers, and collaboration with colleagues. Leadership example, such as how leaders signal what they value through promotions, recognition, budget, their time and informal comments.
Example output measure: Percentage of total editorial space,by category, for all six channels50%45%40%35%30% 28% 27% 25%25%20% 17% 14%15%10% 6%5% 1% 0%0% Vision Mission Values Goals Strategic Administrative Recognition Quotes Priorities
Example output measure: Percentage of total editorial space,by category, for E-Bulletin50%45%40%35%30% 25%25% 23% 19%20%15% 13%10% 8%5% 3% 1% 1%0% Vision Mission Values Goals Strategic Administrative Recognition Quotes Priorities
Example output measure: Percentage of total editorialspace, deep dive in “goals” coverage50%45%40%35%30% 28% 25%25%20%15%10% 5% 5% 3% 2% 2% 0% Goals Total Goal 1: Develop Goal 2: Prepare Goal 3: Continue Goal 4: To serve Goal 5: Develop a… and equip… to transition the the publics an enterprise … needs by… approach to…
| 48 |Communication MeasuresIntegrated with Communication Planning Communicator Stakeholder Client Program Owner Perspective Perspective Perspective 5. 3. 4. 2. 1.Communi-cation Tactics to Stakeholder Communication Stakeholder Results Deploy Assessment Outcomes Behaviorsplanelements (Frame of Reference)Measures Communication Communication Knowledge Specific Outcomes Outputs Fitness and behaviors and outputsfor eachelement Feelings needed Changes that will enable the behaviors Change Readiness
| 49 |Does anyone really care?Measures can move your “brand” from overhead, toward operations.Communications Executive Council survey: 2012 vs. 2007 data Responsibilities of communication function – #1 increase: Social Media 80% (from zero) – #2 increase: Measurement 82% (from 64%) Top communication strategic priorities for 2013 – #1: Employee engagement – #2: Improving leadership communication – #3: Building company presence in social media Strategies to improve communication function effectiveness – #1 Improved consultative partnership with business leaders – #2 Deeper business acumen on the team CEC 2012 report: Taking Your Communications Budgeting and Planning to the Next Level
| 50 |What next? Deeper topics – Design and use tools for valid and reliable data – Communication audits Broader topics – Looking at measures across organizations and time – What communication function can do to add value Feedback on today’s presentation Reach out to me and FCN community Q&A