Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Communications Audit


Published on

Presentation to library schools in Tallinn and Stuttgart

Published in: Education, Technology, Business
  • Be the first to comment

Communications Audit

  1. 1. Demystifying the communications audit Barbie E. Keiser US Consulate – Stuttgart November 17, 2008
  2. 2. Because the world changes so quickly, sooner or later we all need to rebuild or refine … <ul><li>An intranet, extranet/public website, or e-newsletter </li></ul><ul><li>The Library-related portions of an organization’s intranet or public website </li></ul><ul><li>Intranet or extranet-delivered services and content </li></ul><ul><li>Communications programs (internal or external) </li></ul><ul><li>Web based client relationship management </li></ul><ul><li>AKA CLIENT-BASED CONTINUOUS IMPROVEMENT </li></ul>
  3. 3. What worked well last year may no longer be ideal <ul><li>Input may &quot;happen our way,&quot; indicating that improvements are needed </li></ul><ul><li>But it would be risky to wait for – or rely solely on such input! </li></ul><ul><li>We need to engage in planned and well-executed audits </li></ul>
  4. 4. &quot;Audit&quot; is not a scary word <ul><li>But if it seems to be, just choose another name! </li></ul><ul><ul><li>Strategic Planning Review </li></ul></ul><ul><ul><li>Resource Assessment </li></ul></ul><ul><ul><li>Business Process Planning </li></ul></ul><ul><ul><li>Communications Checkup </li></ul></ul><ul><li>Translation: Regularly repeated, s ystematic examination of current practices and future needs with the focus on the client </li></ul>
  5. 5. The Audit is supplemented by ongoing monitoring: <ul><li>Mechanisms to catch evidence of: </li></ul><ul><ul><li>Shifts in client needs due to changes in internal operations, changes in their target markets, and developments in industries in which they operate </li></ul></ul><ul><ul><li>Opportunities created through the application of advanced technology </li></ul></ul><ul><li>The objective is to put in place a &quot;360 °&quot; system so as never to be caught off guard </li></ul>
  6. 6. The methods we choose to employ in conducting the Audit depend on: <ul><li>Existing client relationships </li></ul><ul><li>Corporate culture </li></ul><ul><li>Time and money tradeoffs </li></ul><ul><ul><li>Benefits of a comprehensive effort </li></ul></ul><ul><ul><li>Options for a phased-approach </li></ul></ul><ul><li>Skill sets of staff involved </li></ul><ul><li>Available time frame </li></ul><ul><li>Complexity of what is being examined </li></ul>
  7. 7. Key consideration <ul><li>How can we engage informants in a non-burdensome way? </li></ul><ul><li>How do we choose the right method for each target user group? </li></ul>
  8. 8. Tell-it-to-me … or Easter Eggs? <ul><li>Common techniques involve a mix: </li></ul><ul><ul><li>Interviews </li></ul></ul><ul><ul><li>Discussion (focus) groups </li></ul></ul><ul><ul><li>Surveys </li></ul></ul><ul><ul><li>Spot checks </li></ul></ul><ul><li>But what people say isn’t always “the goods” </li></ul><ul><li>We may need more “evidentiary” methods </li></ul><ul><ul><li>Logs (behavior trails) </li></ul></ul><ul><ul><li>Stats (behavior trends) </li></ul></ul><ul><ul><li>Observed behavior as-it-happens </li></ul></ul>
  9. 9. All efforts are designed to help us: <ul><li>Understand where we need / want / are able to go in order to meet evolving client needs </li></ul><ul><li>Build a business case for new or redirected efforts/investments </li></ul><ul><li>Create a plan for how we get there from here </li></ul>
  10. 10. Focus for today <ul><li>Designing a process that will yield maximum insight into client perceptions and priorities </li></ul><ul><ul><li>Involving the least effort - for us and for our clients </li></ul></ul><ul><ul><li>Resulting in the greatest degree of confidence in the findings </li></ul></ul>
  11. 11. Case example 1 <ul><li>The Corporate Library’s Intranet presence is not cutting it! </li></ul><ul><li>It may once have been the cat’s meow, but time has gone by and “grafting” has resulted in a difficult-to-use site </li></ul>
  12. 12. We know some challenges, based on symptoms <ul><li>Usage has dropped off significantly </li></ul><ul><li>Users are confused </li></ul><ul><li>Navigation is not intuitive </li></ul><ul><li>Users &quot;miss&quot; important announcements and new sources </li></ul><ul><li>Users go elsewhere for information </li></ul>
  13. 13. But we need to dig deeper <ul><li>Our audit must not only point toward solutions for known concerns (for staff and users), but be designed to uncover details that may be &quot;hiding&quot; from us </li></ul>
  14. 14. What are the elements to be examined? <ul><li>What specific &quot;defects&quot; must we address? </li></ul><ul><li>What content and services would be priority offerings in the eyes of our clientele? </li></ul><ul><li>What functionality do clients consider essential vs. &quot;nice to have&quot;? </li></ul><ul><li>Any &quot;sacred cows&quot; we can drop? </li></ul><ul><li>What are &quot;good&quot; library intranets doing that we aren’t? </li></ul><ul><ul><li>How are those intranets maintained and by whom? </li></ul></ul><ul><li>Are there good models we could use for inspiration? </li></ul>
  15. 15. Audit design preamble <ul><li>Balance what we already know and what we don't yet know - avoid skewing results </li></ul><ul><li>Who can tell us? Who should find out? </li></ul><ul><ul><li>Who on our staff should be involved in the process? </li></ul></ul><ul><ul><li>What outside assistance will we need? </li></ul></ul>
  16. 16. Typical process <ul><li>Review documented (web logs) & anecdotal evidence, and any marketing collateral distributed </li></ul><ul><li>Identify &quot;regular user&quot; / fan base informants </li></ul><ul><li>Identify &quot;non-user who should be user&quot; informants </li></ul><ul><li>Identify individuals others view as role models </li></ul><ul><li>Devise interaction structures </li></ul><ul><ul><li>Two-on-one interviews </li></ul></ul><ul><ul><li>Discussion/focus groups </li></ul></ul><ul><li>Devise direct observation events </li></ul><ul><li>Test interview/survey instruments </li></ul><ul><li>Set up logistics </li></ul>
  17. 17. A note on communication <ul><li>Must &quot;ace&quot; the invitation: WIIFM </li></ul><ul><li>Explain the why </li></ul><ul><li>If possible, make the effort fun, not arduous </li></ul><ul><li>Accommodate participants' schedules </li></ul><ul><li>Gently remind </li></ul><ul><li>Demonstrate later that input got results </li></ul>
  18. 18. Tell me … or may I watch? <ul><li>Common techniques involve questions </li></ul><ul><li>But what people say isn’t always &quot;the goods&quot; </li></ul><ul><li>We may need other methods - such as observation of what people actually do in a specific situation </li></ul>
  19. 19. &quot;Easter Eggs&quot; <ul><li>A specific task answerable through the intranet site or other vehicle being examined </li></ul><ul><li>Forcing respondents to not just say “looks nice” but to dive in and answer a specific question </li></ul><ul><li>What path used to solve question? </li></ul><ul><li>Where found answer - or abandoned? </li></ul>
  20. 20. At-the-elbow &quot;Process Deconstruction&quot; <ul><li>An attempt to trace a logical decision path </li></ul><ul><li>&quot;Show me a typical task you need to accomplish&quot; </li></ul><ul><li>What do you do first? Why? </li></ul><ul><li>Stop – why did you look there ? </li></ul><ul><li>Stop – then what made you decide to look here ? </li></ul><ul><li>Why did you not check here first? </li></ul>
  21. 21. Benefits of observation <ul><li>Discovery of &quot;hidden knowledge&quot; not documented and thus unavailable to new users </li></ul><ul><li>Reality check to guard against owners' sense that &quot;it's obvious&quot; </li></ul><ul><li>Insights useful in further communication and design </li></ul>
  22. 22. Case example 2 <ul><li>A Library Consortium’s Communications Program is not as effective as it could be </li></ul><ul><li>Too many vehicles are employed </li></ul><ul><li>Many services are under-utilized </li></ul>
  23. 23. Project design <ul><li>Initial orientation meeting </li></ul><ul><li>Survey of members </li></ul><ul><li>Focus groups </li></ul><ul><li>Needs assessment and usability-lite test for the Consortium’s website </li></ul><ul><li>Audit and benchmarking </li></ul><ul><li>Synthesis of data </li></ul>
  24. 24. Survey <ul><li>Identified survey content, designed survey, and coordinated Consortium’s review of draft survey </li></ul><ul><li>Identified survey pretest participants, completed survey pretest, and revised questionnaire </li></ul><ul><li>Developed and implemented Web-based survey </li></ul><ul><ul><li>Addressed security & privacy issues by observing professional protocols for information collection </li></ul></ul><ul><ul><li>Attained buy-in and announced survey </li></ul></ul><ul><li>Hosted and monitored Web-based collection tools and systems </li></ul><ul><li>Monitored survey completion and followed-up </li></ul><ul><li>Analyzed data </li></ul><ul><ul><li>Overall (Example) </li></ul></ul><ul><ul><li>Type of library (Example) </li></ul></ul><ul><ul><li>Portraits of _____ Library (Example) </li></ul></ul>
  25. 25. Focus groups <ul><li>Obtain information and clarification on: </li></ul><ul><ul><li>Which issues are a priority </li></ul></ul><ul><ul><li>What features are important </li></ul></ul><ul><ul><li>Extent to which members will have influence </li></ul></ul><ul><ul><li>Ways to measure success </li></ul></ul><ul><ul><li>Role of interactivity </li></ul></ul><ul><li>Conduct exercises to discern: </li></ul><ul><ul><li>Awareness of the services offered to and valued by the participants </li></ul></ul><ul><ul><li>Adequacy of communication about those services </li></ul></ul><ul><ul><li>Availability and accessibility of services </li></ul></ul><ul><ul><li>Opportunities for improvement </li></ul></ul><ul><li>Present results (to organization, participants, and members) </li></ul>
  26. 26. &quot;Usability-lite&quot; test of the Website <ul><li>Does our website succeed in communicating clearly? Are users having trouble? </li></ul><ul><li>Usability-lite tests will help you determine: </li></ul><ul><ul><li>Actuals / Optimals </li></ul></ul><ul><ul><li>Drivers / Incentives </li></ul></ul><ul><ul><li>Barriers / Potential solutions </li></ul></ul><ul><li>A combination of telephone interviews (for pre-screening candidates) and in-person interviews in the participants’ normal work environment </li></ul><ul><ul><li>Ask participants to “think aloud” as they explore the website </li></ul></ul><ul><ul><li>Personae and specific situations </li></ul></ul><ul><ul><li>Ask some follow-up questions </li></ul></ul>
  27. 27. Audit and benchmarking: Objective website review <ul><li>Reviewed the site from a member’s perspective vis-à-vis stated goals </li></ul><ul><li>Analyzed extant data (e.g. logfile data) </li></ul><ul><li>Evaluated typical navigation </li></ul><ul><li>Examined usability/human factors </li></ul><ul><li>Assessed the calls to action and flow of copy </li></ul><ul><li>Identified interactive techniques </li></ul><ul><li>Provided recommendations for: </li></ul><ul><ul><li>Navigation, technical, and usability functions </li></ul></ul><ul><ul><li>Marketing copy </li></ul></ul><ul><ul><li>Interactive techniques </li></ul></ul><ul><ul><li>Access to other information systems and services </li></ul></ul>
  28. 28. The result: User- and usage-centered design
  29. 29. Before…
  30. 30. After…
  31. 31. Tips for interviews and focus groups <ul><li>Explain if/how an audio/video recording of the discussion will be utilized </li></ul><ul><li>Have a set of questions but be flexible </li></ul><ul><li>Assure complete confidentiality </li></ul><ul><li>Let them talk </li></ul><ul><li>Be comfortable with silence, but ready to &quot;prime the pump“ </li></ul><ul><li>Elicit elaboration by validating: Interesting, you are not the first to say so </li></ul><ul><li>Use &quot;others-find&quot; technique (you too?) </li></ul><ul><li>Be aware of interpersonal dynamics and politics </li></ul><ul><li>Recognize that some may not want to &quot;look bad&quot; & may tailor comments to what is thought &quot;correct&quot; </li></ul>
  32. 32. Tips for surveys <ul><li>Short - Fast – Easy – did we mention short ! </li></ul><ul><li>Clear, unambiguous questions in sections that flow </li></ul><ul><li>Ranking of personal priorities (What means more to you?) vs. &quot;easy middle choice&quot; </li></ul><ul><li>&quot;How much do you love us on a 1-10&quot; yields less valuable insight (no one wants to offend) </li></ul><ul><li>Minimize the number of open-ended questions </li></ul><ul><li>Do you agree with these statements made by your peers? </li></ul><ul><li>People shouldn’t agonize over responses, so provide an &quot;out&quot; using Don’t know/Not sure options </li></ul>
  33. 33. Tips for the direct-observation portion of a web usability test <ul><li>Explain that the findings from the evaluation will be used &quot;for good purpose&quot;- no need to be polite </li></ul><ul><li>Explain that you will be collecting data by taking written notes </li></ul><ul><li>Stress that the website is being tested, they aren't </li></ul><ul><li>Remind interviewees to articulate their thoughts </li></ul><ul><li>Stay neutral </li></ul><ul><li>Help users in distress </li></ul><ul><li>Ask if they have any questions before the interview begins </li></ul>
  34. 34. What do we have in hand when it's all done? <ul><li>Ideally, the elements for our business case </li></ul><ul><li>The justification for action and investment </li></ul><ul><li>The credibility that &quot;the users spoke&quot; </li></ul>
  35. 35. Audit Report sets out ( in one place )… <ul><li>Drivers for the Audit (why did we) </li></ul><ul><li>Goals (what looked for) </li></ul><ul><li>Methodology / Informants (how did we) </li></ul><ul><li>Findings (broken down by major topic area) – factual, dispassionate </li></ul><ul><li>Conclusions that take the findings and group them into themes </li></ul><ul><li>Recommendations flowing from the findings (not just a crazy idea) </li></ul><ul><li>Business Case that includes a “how we will” implement change & risks of not implementing </li></ul>
  36. 36. Some of what we uncovered may be similar to your own experiences <ul><li>Overall communications strategy: NSLS is responsive to member requests, but with some interesting effects </li></ul><ul><li>Target market: Probability of messages being opened, read, and acted upon increases dramatically as the targeting becomes more precise </li></ul><ul><li>Design/layout: Members normally find what they need on the NSLS website, but not easily, and they often forget where pages used in the past are located </li></ul><ul><li>Orientation/navigation: Consistency and clarity </li></ul>
  37. 37. Some of what we uncovered may be similar to your own experiences (con’t) <ul><li>Search: Search functionality provided by local Google and sitemap </li></ul><ul><li>Content: Targeted content; relationships established; sourcing of content </li></ul><ul><li>Networking: Member &quot;control&quot; over physical and virtual (listservs vs. CoPs) </li></ul><ul><li>Continuing education: Professional development opportunities with series of courses for career paths </li></ul>
  38. 38. Access to technology <ul><li>Issues surrounding technology focused on convenience in accessing NSLS services rather than not having access to required technology to make full use of them </li></ul><ul><li>Self-described as not being “explorers,” merely accepting what is on the screen in the way in which it appears </li></ul><ul><li>Preference for accessing NSLS while working, and not at home (nights or weekends) </li></ul>
  39. 39. Moving from findings and conclusions to recommended action <ul><li>Document findings </li></ul><ul><ul><li>Known challenges and potential solutions </li></ul></ul><ul><ul><li>Existing technical barriers and other constraints </li></ul></ul><ul><li>Synthesize/interpret findings to draw conclusions </li></ul><ul><ul><li>What works </li></ul></ul><ul><ul><li>Areas for improvement </li></ul></ul><ul><ul><li>Opportunities for growth </li></ul></ul><ul><li>Make recommendations </li></ul><ul><ul><li>Implement immediately </li></ul></ul><ul><ul><li>Implement in the near-term </li></ul></ul><ul><ul><li>Longer-term initiatives </li></ul></ul>
  40. 40. What have we learned today? <ul><li>Why organizations should conduct audits </li></ul><ul><li>Options for soliciting client input </li></ul><ul><li>The importance of communicating with our clients (and especially those who give time to the audit process) </li></ul><ul><li>Alternative ways to say “no” </li></ul><ul><li>Evolution, not revolution </li></ul><ul><ul><li>No surprises </li></ul></ul>
  41. 41. Questions?
  42. 42. Thank You … feel free to be in touch! <ul><li>Barbie Keiser </li></ul><ul><li>[email_address] </li></ul>