Communications Audit


Published on

Presentation to library schools in Tallinn and Stuttgart

Published in: Education, Technology, Business
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Are we hearing them? Responding to their concerns in a timely fashion? First presented at Internet Librarian International in 2006
  • Communications Audit

    1. 1. Demystifying the communications audit Barbie E. Keiser US Consulate – Stuttgart November 17, 2008
    2. 2. Because the world changes so quickly, sooner or later we all need to rebuild or refine … <ul><li>An intranet, extranet/public website, or e-newsletter </li></ul><ul><li>The Library-related portions of an organization’s intranet or public website </li></ul><ul><li>Intranet or extranet-delivered services and content </li></ul><ul><li>Communications programs (internal or external) </li></ul><ul><li>Web based client relationship management </li></ul><ul><li>AKA CLIENT-BASED CONTINUOUS IMPROVEMENT </li></ul>
    3. 3. What worked well last year may no longer be ideal <ul><li>Input may &quot;happen our way,&quot; indicating that improvements are needed </li></ul><ul><li>But it would be risky to wait for – or rely solely on such input! </li></ul><ul><li>We need to engage in planned and well-executed audits </li></ul>
    4. 4. &quot;Audit&quot; is not a scary word <ul><li>But if it seems to be, just choose another name! </li></ul><ul><ul><li>Strategic Planning Review </li></ul></ul><ul><ul><li>Resource Assessment </li></ul></ul><ul><ul><li>Business Process Planning </li></ul></ul><ul><ul><li>Communications Checkup </li></ul></ul><ul><li>Translation: Regularly repeated, s ystematic examination of current practices and future needs with the focus on the client </li></ul>
    5. 5. The Audit is supplemented by ongoing monitoring: <ul><li>Mechanisms to catch evidence of: </li></ul><ul><ul><li>Shifts in client needs due to changes in internal operations, changes in their target markets, and developments in industries in which they operate </li></ul></ul><ul><ul><li>Opportunities created through the application of advanced technology </li></ul></ul><ul><li>The objective is to put in place a &quot;360 °&quot; system so as never to be caught off guard </li></ul>
    6. 6. The methods we choose to employ in conducting the Audit depend on: <ul><li>Existing client relationships </li></ul><ul><li>Corporate culture </li></ul><ul><li>Time and money tradeoffs </li></ul><ul><ul><li>Benefits of a comprehensive effort </li></ul></ul><ul><ul><li>Options for a phased-approach </li></ul></ul><ul><li>Skill sets of staff involved </li></ul><ul><li>Available time frame </li></ul><ul><li>Complexity of what is being examined </li></ul>
    7. 7. Key consideration <ul><li>How can we engage informants in a non-burdensome way? </li></ul><ul><li>How do we choose the right method for each target user group? </li></ul>
    8. 8. Tell-it-to-me … or Easter Eggs? <ul><li>Common techniques involve a mix: </li></ul><ul><ul><li>Interviews </li></ul></ul><ul><ul><li>Discussion (focus) groups </li></ul></ul><ul><ul><li>Surveys </li></ul></ul><ul><ul><li>Spot checks </li></ul></ul><ul><li>But what people say isn’t always “the goods” </li></ul><ul><li>We may need more “evidentiary” methods </li></ul><ul><ul><li>Logs (behavior trails) </li></ul></ul><ul><ul><li>Stats (behavior trends) </li></ul></ul><ul><ul><li>Observed behavior as-it-happens </li></ul></ul>
    9. 9. All efforts are designed to help us: <ul><li>Understand where we need / want / are able to go in order to meet evolving client needs </li></ul><ul><li>Build a business case for new or redirected efforts/investments </li></ul><ul><li>Create a plan for how we get there from here </li></ul>
    10. 10. Focus for today <ul><li>Designing a process that will yield maximum insight into client perceptions and priorities </li></ul><ul><ul><li>Involving the least effort - for us and for our clients </li></ul></ul><ul><ul><li>Resulting in the greatest degree of confidence in the findings </li></ul></ul>
    11. 11. Case example 1 <ul><li>The Corporate Library’s Intranet presence is not cutting it! </li></ul><ul><li>It may once have been the cat’s meow, but time has gone by and “grafting” has resulted in a difficult-to-use site </li></ul>
    12. 12. We know some challenges, based on symptoms <ul><li>Usage has dropped off significantly </li></ul><ul><li>Users are confused </li></ul><ul><li>Navigation is not intuitive </li></ul><ul><li>Users &quot;miss&quot; important announcements and new sources </li></ul><ul><li>Users go elsewhere for information </li></ul>
    13. 13. But we need to dig deeper <ul><li>Our audit must not only point toward solutions for known concerns (for staff and users), but be designed to uncover details that may be &quot;hiding&quot; from us </li></ul>
    14. 14. What are the elements to be examined? <ul><li>What specific &quot;defects&quot; must we address? </li></ul><ul><li>What content and services would be priority offerings in the eyes of our clientele? </li></ul><ul><li>What functionality do clients consider essential vs. &quot;nice to have&quot;? </li></ul><ul><li>Any &quot;sacred cows&quot; we can drop? </li></ul><ul><li>What are &quot;good&quot; library intranets doing that we aren’t? </li></ul><ul><ul><li>How are those intranets maintained and by whom? </li></ul></ul><ul><li>Are there good models we could use for inspiration? </li></ul>
    15. 15. Audit design preamble <ul><li>Balance what we already know and what we don't yet know - avoid skewing results </li></ul><ul><li>Who can tell us? Who should find out? </li></ul><ul><ul><li>Who on our staff should be involved in the process? </li></ul></ul><ul><ul><li>What outside assistance will we need? </li></ul></ul>
    16. 16. Typical process <ul><li>Review documented (web logs) & anecdotal evidence, and any marketing collateral distributed </li></ul><ul><li>Identify &quot;regular user&quot; / fan base informants </li></ul><ul><li>Identify &quot;non-user who should be user&quot; informants </li></ul><ul><li>Identify individuals others view as role models </li></ul><ul><li>Devise interaction structures </li></ul><ul><ul><li>Two-on-one interviews </li></ul></ul><ul><ul><li>Discussion/focus groups </li></ul></ul><ul><li>Devise direct observation events </li></ul><ul><li>Test interview/survey instruments </li></ul><ul><li>Set up logistics </li></ul>
    17. 17. A note on communication <ul><li>Must &quot;ace&quot; the invitation: WIIFM </li></ul><ul><li>Explain the why </li></ul><ul><li>If possible, make the effort fun, not arduous </li></ul><ul><li>Accommodate participants' schedules </li></ul><ul><li>Gently remind </li></ul><ul><li>Demonstrate later that input got results </li></ul>
    18. 18. Tell me … or may I watch? <ul><li>Common techniques involve questions </li></ul><ul><li>But what people say isn’t always &quot;the goods&quot; </li></ul><ul><li>We may need other methods - such as observation of what people actually do in a specific situation </li></ul>
    19. 19. &quot;Easter Eggs&quot; <ul><li>A specific task answerable through the intranet site or other vehicle being examined </li></ul><ul><li>Forcing respondents to not just say “looks nice” but to dive in and answer a specific question </li></ul><ul><li>What path used to solve question? </li></ul><ul><li>Where found answer - or abandoned? </li></ul>
    20. 20. At-the-elbow &quot;Process Deconstruction&quot; <ul><li>An attempt to trace a logical decision path </li></ul><ul><li>&quot;Show me a typical task you need to accomplish&quot; </li></ul><ul><li>What do you do first? Why? </li></ul><ul><li>Stop – why did you look there ? </li></ul><ul><li>Stop – then what made you decide to look here ? </li></ul><ul><li>Why did you not check here first? </li></ul>
    21. 21. Benefits of observation <ul><li>Discovery of &quot;hidden knowledge&quot; not documented and thus unavailable to new users </li></ul><ul><li>Reality check to guard against owners' sense that &quot;it's obvious&quot; </li></ul><ul><li>Insights useful in further communication and design </li></ul>
    22. 22. Case example 2 <ul><li>A Library Consortium’s Communications Program is not as effective as it could be </li></ul><ul><li>Too many vehicles are employed </li></ul><ul><li>Many services are under-utilized </li></ul>
    23. 23. Project design <ul><li>Initial orientation meeting </li></ul><ul><li>Survey of members </li></ul><ul><li>Focus groups </li></ul><ul><li>Needs assessment and usability-lite test for the Consortium’s website </li></ul><ul><li>Audit and benchmarking </li></ul><ul><li>Synthesis of data </li></ul>
    24. 24. Survey <ul><li>Identified survey content, designed survey, and coordinated Consortium’s review of draft survey </li></ul><ul><li>Identified survey pretest participants, completed survey pretest, and revised questionnaire </li></ul><ul><li>Developed and implemented Web-based survey </li></ul><ul><ul><li>Addressed security & privacy issues by observing professional protocols for information collection </li></ul></ul><ul><ul><li>Attained buy-in and announced survey </li></ul></ul><ul><li>Hosted and monitored Web-based collection tools and systems </li></ul><ul><li>Monitored survey completion and followed-up </li></ul><ul><li>Analyzed data </li></ul><ul><ul><li>Overall (Example) </li></ul></ul><ul><ul><li>Type of library (Example) </li></ul></ul><ul><ul><li>Portraits of _____ Library (Example) </li></ul></ul>
    25. 25. Focus groups <ul><li>Obtain information and clarification on: </li></ul><ul><ul><li>Which issues are a priority </li></ul></ul><ul><ul><li>What features are important </li></ul></ul><ul><ul><li>Extent to which members will have influence </li></ul></ul><ul><ul><li>Ways to measure success </li></ul></ul><ul><ul><li>Role of interactivity </li></ul></ul><ul><li>Conduct exercises to discern: </li></ul><ul><ul><li>Awareness of the services offered to and valued by the participants </li></ul></ul><ul><ul><li>Adequacy of communication about those services </li></ul></ul><ul><ul><li>Availability and accessibility of services </li></ul></ul><ul><ul><li>Opportunities for improvement </li></ul></ul><ul><li>Present results (to organization, participants, and members) </li></ul>
    26. 26. &quot;Usability-lite&quot; test of the Website <ul><li>Does our website succeed in communicating clearly? Are users having trouble? </li></ul><ul><li>Usability-lite tests will help you determine: </li></ul><ul><ul><li>Actuals / Optimals </li></ul></ul><ul><ul><li>Drivers / Incentives </li></ul></ul><ul><ul><li>Barriers / Potential solutions </li></ul></ul><ul><li>A combination of telephone interviews (for pre-screening candidates) and in-person interviews in the participants’ normal work environment </li></ul><ul><ul><li>Ask participants to “think aloud” as they explore the website </li></ul></ul><ul><ul><li>Personae and specific situations </li></ul></ul><ul><ul><li>Ask some follow-up questions </li></ul></ul>
    27. 27. Audit and benchmarking: Objective website review <ul><li>Reviewed the site from a member’s perspective vis-à-vis stated goals </li></ul><ul><li>Analyzed extant data (e.g. logfile data) </li></ul><ul><li>Evaluated typical navigation </li></ul><ul><li>Examined usability/human factors </li></ul><ul><li>Assessed the calls to action and flow of copy </li></ul><ul><li>Identified interactive techniques </li></ul><ul><li>Provided recommendations for: </li></ul><ul><ul><li>Navigation, technical, and usability functions </li></ul></ul><ul><ul><li>Marketing copy </li></ul></ul><ul><ul><li>Interactive techniques </li></ul></ul><ul><ul><li>Access to other information systems and services </li></ul></ul>
    28. 28. The result: User- and usage-centered design
    29. 29. Before…
    30. 30. After…
    31. 31. Tips for interviews and focus groups <ul><li>Explain if/how an audio/video recording of the discussion will be utilized </li></ul><ul><li>Have a set of questions but be flexible </li></ul><ul><li>Assure complete confidentiality </li></ul><ul><li>Let them talk </li></ul><ul><li>Be comfortable with silence, but ready to &quot;prime the pump“ </li></ul><ul><li>Elicit elaboration by validating: Interesting, you are not the first to say so </li></ul><ul><li>Use &quot;others-find&quot; technique (you too?) </li></ul><ul><li>Be aware of interpersonal dynamics and politics </li></ul><ul><li>Recognize that some may not want to &quot;look bad&quot; & may tailor comments to what is thought &quot;correct&quot; </li></ul>
    32. 32. Tips for surveys <ul><li>Short - Fast – Easy – did we mention short ! </li></ul><ul><li>Clear, unambiguous questions in sections that flow </li></ul><ul><li>Ranking of personal priorities (What means more to you?) vs. &quot;easy middle choice&quot; </li></ul><ul><li>&quot;How much do you love us on a 1-10&quot; yields less valuable insight (no one wants to offend) </li></ul><ul><li>Minimize the number of open-ended questions </li></ul><ul><li>Do you agree with these statements made by your peers? </li></ul><ul><li>People shouldn’t agonize over responses, so provide an &quot;out&quot; using Don’t know/Not sure options </li></ul>
    33. 33. Tips for the direct-observation portion of a web usability test <ul><li>Explain that the findings from the evaluation will be used &quot;for good purpose&quot;- no need to be polite </li></ul><ul><li>Explain that you will be collecting data by taking written notes </li></ul><ul><li>Stress that the website is being tested, they aren't </li></ul><ul><li>Remind interviewees to articulate their thoughts </li></ul><ul><li>Stay neutral </li></ul><ul><li>Help users in distress </li></ul><ul><li>Ask if they have any questions before the interview begins </li></ul>
    34. 34. What do we have in hand when it's all done? <ul><li>Ideally, the elements for our business case </li></ul><ul><li>The justification for action and investment </li></ul><ul><li>The credibility that &quot;the users spoke&quot; </li></ul>
    35. 35. Audit Report sets out ( in one place )… <ul><li>Drivers for the Audit (why did we) </li></ul><ul><li>Goals (what looked for) </li></ul><ul><li>Methodology / Informants (how did we) </li></ul><ul><li>Findings (broken down by major topic area) – factual, dispassionate </li></ul><ul><li>Conclusions that take the findings and group them into themes </li></ul><ul><li>Recommendations flowing from the findings (not just a crazy idea) </li></ul><ul><li>Business Case that includes a “how we will” implement change & risks of not implementing </li></ul>
    36. 36. Some of what we uncovered may be similar to your own experiences <ul><li>Overall communications strategy: NSLS is responsive to member requests, but with some interesting effects </li></ul><ul><li>Target market: Probability of messages being opened, read, and acted upon increases dramatically as the targeting becomes more precise </li></ul><ul><li>Design/layout: Members normally find what they need on the NSLS website, but not easily, and they often forget where pages used in the past are located </li></ul><ul><li>Orientation/navigation: Consistency and clarity </li></ul>
    37. 37. Some of what we uncovered may be similar to your own experiences (con’t) <ul><li>Search: Search functionality provided by local Google and sitemap </li></ul><ul><li>Content: Targeted content; relationships established; sourcing of content </li></ul><ul><li>Networking: Member &quot;control&quot; over physical and virtual (listservs vs. CoPs) </li></ul><ul><li>Continuing education: Professional development opportunities with series of courses for career paths </li></ul>
    38. 38. Access to technology <ul><li>Issues surrounding technology focused on convenience in accessing NSLS services rather than not having access to required technology to make full use of them </li></ul><ul><li>Self-described as not being “explorers,” merely accepting what is on the screen in the way in which it appears </li></ul><ul><li>Preference for accessing NSLS while working, and not at home (nights or weekends) </li></ul>
    39. 39. Moving from findings and conclusions to recommended action <ul><li>Document findings </li></ul><ul><ul><li>Known challenges and potential solutions </li></ul></ul><ul><ul><li>Existing technical barriers and other constraints </li></ul></ul><ul><li>Synthesize/interpret findings to draw conclusions </li></ul><ul><ul><li>What works </li></ul></ul><ul><ul><li>Areas for improvement </li></ul></ul><ul><ul><li>Opportunities for growth </li></ul></ul><ul><li>Make recommendations </li></ul><ul><ul><li>Implement immediately </li></ul></ul><ul><ul><li>Implement in the near-term </li></ul></ul><ul><ul><li>Longer-term initiatives </li></ul></ul>
    40. 40. What have we learned today? <ul><li>Why organizations should conduct audits </li></ul><ul><li>Options for soliciting client input </li></ul><ul><li>The importance of communicating with our clients (and especially those who give time to the audit process) </li></ul><ul><li>Alternative ways to say “no” </li></ul><ul><li>Evolution, not revolution </li></ul><ul><ul><li>No surprises </li></ul></ul>
    41. 41. Questions?
    42. 42. Thank You … feel free to be in touch! <ul><li>Barbie Keiser </li></ul><ul><li>[email_address] </li></ul>