Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Measuring government websites


Published on

Measuring online performance can be tricky, and there's not a lot of guidance out there for non-commerce sites. This framework has been developed specifically for government organizations to help them measure, track and improve their online services.

Published in: Business, Technology, Design
  • Be the first to comment

Measuring government websites

  1. 1. Creating a Web Performance Measurement Framework for Government sites Providing evidence for making decisions to ensure citizen-focused websites Developed by Laura Wesley with help and research by tons of others both known and unknown to me. Find me online at or on Twitter @resultsjunkie
  2. 2. Table of Contents <ul><ul><li>Objectives of providing this framework </li></ul></ul><ul><ul><ul><li>Goals of the framework </li></ul></ul></ul><ul><ul><ul><li>Common challenges of measuring online performance </li></ul></ul></ul><ul><ul><li>Implementing the framework </li></ul></ul><ul><ul><ul><li>Measuring efficiency </li></ul></ul></ul><ul><ul><ul><li>Measuring effectiveness </li></ul></ul></ul><ul><ul><ul><li>Measuring satisfaction </li></ul></ul></ul><ul><ul><li>Web Performance Measurement Framework </li></ul></ul><ul><ul><ul><li>Sample Web Performance Measurement Strategy </li></ul></ul></ul><ul><ul><ul><li>Evaluation criteria & rationale for each indicator </li></ul></ul></ul><ul><ul><ul><li>Reporting for decision-making </li></ul></ul></ul>
  3. 3. Objectives & common challenges of measuring online performance
  4. 4. Goal of providing this framework <ul><li>Reduce duplication, error rates & cost of gathering and using client feedback to improve online communication and service to the public by: </li></ul><ul><ul><li>Providing a Web Performance Measurement Framework that can be customized and re-used by any department to increase ability to make decisions using facts and figures. </li></ul></ul><ul><ul><li>Proposing a set of performance indicators that can be used consistently across the Government of Canada to compare progress and success. </li></ul></ul>
  5. 5. Goals of the framework… <ul><li>Develop an approach that is consistent with: </li></ul><ul><ul><ul><li>Government of Canada’s Management Accountability Framework, specifically Citizen-focused Service element. </li></ul></ul></ul><ul><ul><ul><li>Government Service Reference Models (GSRM) </li></ul></ul></ul><ul><ul><ul><li>Private sector approach to client-centred design and testing </li></ul></ul></ul><ul><ul><ul><li>Results-based Management (RBM) concept & tools </li></ul></ul></ul><ul><ul><ul><li>Research from the Institute on Citizen-Centric Service (ICCS) and other academic forums. </li></ul></ul></ul><ul><li>Provide a framework to assess web channel performance </li></ul><ul><ul><ul><li>Effectiveness of website </li></ul></ul></ul><ul><ul><ul><li>Efficiency of web tools & processes that support this channel </li></ul></ul></ul><ul><ul><ul><li>Satisfaction of website visitors </li></ul></ul></ul><ul><li>Develop an evidenced-based approach to guide long-term site improvements. </li></ul>
  6. 6. Common challenges <ul><li>This framework is meant to assist with these common challenges to collecting and using data to inform decisions: </li></ul><ul><ul><li>Too much/not enough data. </li></ul></ul><ul><ul><li>Not sure what information to gather or analyze. </li></ul></ul><ul><ul><li>Focus on data instead of trends that lead to insights. </li></ul></ul><ul><ul><li>Difficulty turning data into insights and then into concrete actions. </li></ul></ul><ul><ul><li>Mistake website as a goal in itself rather than a means by which a goal is achieved or communicated. </li></ul></ul>
  7. 7. Limitations of this framework <ul><li>This framework only covers measuring performance of your own site to inform ongoing improvements. </li></ul><ul><li>Doesn't provide additional steps required to: </li></ul><ul><ul><li>Evaluate impact of offline change in the world (outcomes) as a result of providing services or information online. </li></ul></ul><ul><ul><li>Measure performance of off-site content, that is, social media or other forms of online content that are not contained on your website. </li></ul></ul><ul><ul><li>Diagnose specific site ailments. </li></ul></ul>
  8. 8. Develop and implement your web performance measurement strategy
  9. 9. <ul><li>Efficiency </li></ul><ul><ul><li>Demonstrate that the resources required to create, maintain & continuously improve the website are worth associated costs. </li></ul></ul><ul><li>Effectiveness </li></ul><ul><ul><li>Demonstrate that visitors are able to complete the necessary and desired tasks that are available on the website. </li></ul></ul><ul><li>Satisfaction </li></ul><ul><ul><li>Demonstrate that site visitors are happy with the online experience. </li></ul></ul>What do we need to measure?
  10. 10. Measure the efficiency of your online communication and services
  11. 11. How do we measure e fficiency ? <ul><li>Suggested indicator = Cost per visit/interaction. Compare resources invested to qualified visits on the site. </li></ul><ul><li>How?* </li></ul><ul><ul><li>Track all costs associated with developing the site, plus the number of visits to the site. </li></ul></ul><ul><ul><li>Take it one step further by breaking down cost by: </li></ul></ul><ul><ul><ul><li>types of content – eg. video, podcasts or newsletters </li></ul></ul></ul><ul><ul><ul><li>transactional vs. informational services </li></ul></ul></ul><ul><ul><ul><li>service channels (phone, web, in-person) </li></ul></ul></ul>*It should be noted that I have not yet seen this done well. If you have, please call me!
  12. 12. How do we use this info? <ul><li>As government organizations, we have an obligation to serve citizens in the way they want to be served, but we also have to provide services as efficiently as possible. Chances are, you can't eliminate the highest cost channel or activities but you can: </li></ul><ul><ul><li>Reduce time spent on inefficient activities </li></ul></ul><ul><ul><li>Promote use of most efficient channel </li></ul></ul><ul><ul><li>Promote services where feedback is positive, but costs still outweigh the number of people using it. </li></ul></ul>
  13. 13. Measure the effectiveness of your online communication and services
  14. 14. How do we measure effectiveness? <ul><li>Suggested indicators: accessible, credible, valuable, desirable, useful, usable, findable* </li></ul><ul><li>How? </li></ul><ul><ul><li>Collect data to support user expectations mentioned above. Taken together, these expectations should help deconstruct elements of end-user satisfaction. </li></ul></ul><ul><ul><li>Develop a research plan to collect all the data required to validate user expectations. This becomes your web performance measurement strategy. </li></ul></ul>*From Peter Morville’s Semantic Studios :
  15. 15. How do we use this info? <ul><li>Actual numbers are less important than trends, but both are useful. Additional research may be required to further investigate actual improvements but by collecting data to measure user expectations, you can: </li></ul><ul><ul><li>Create a starting point to report on level of improvements over time (baseline) or compare to similar organizations (benchmark). </li></ul></ul><ul><ul><li>Look for consistent trends across feedback mechanisms (qualitative/quantitative) to prioritize necessary changes. </li></ul></ul><ul><ul><li>Test assumptions and theories about what is working. </li></ul></ul><ul><ul><li>Understand clients' needs and expectations better in order to develop new products and services. </li></ul></ul>
  16. 16. Sample web performance measurement strategy
  17. 17. Components of User Experience From Peter Morville’s Semantic Studios : Context Clients Content Findable Accessible Usable Useful Desirable Credible Valuable MAF: citizen-focused service
  18. 18. Goal is to be… Key Performance Indicator (KPI) Suggested This year's target Data Source Who will collect when? Share info with… Accessible Comply with W3C accessibility guidelines as measured by the percentage of CLF-compliant pages 95% CLF compliance CLF validation reports Web services (Annually) Treasury Board (MAF13) Valuable Site advances our mandate as measured by conversion rates of on-site tasks. 20% of visitors take desired action (on-site) User survey /Various third party Web services (Post-release) Senior Managers Desirable Improved perception of quality of service as measured by percentage of visitors who agree that their visit was positive. 85% satisfaction User survey Analyst (Annually) Site owner /Content owner Usable More easily complete tasks as measured by ease, time and error rate of task completion. 80% composite score of ease, time, error rate Usability testing Web Comms (Pre-release) Site owner /Content owner Useful Increased satisfaction as measured by the percentage of respondents who agree with the statement “This page was useful”. 80% say YES On-page feedback form Analyst (Monthly) Content owner Credible Increased trust in the reliability of the content as measured by the percentage of pages reviewed within past 6 months. 95% of pages reviewed within past 6 months Content Mgt System Web services (Bi-annually) Content owner Findable More easily locate information through external search by measuring percentage of traffic referred to site from search engines. 60% visits referred by search engines Web Analytics software Analyst (Monthly) Content owner
  19. 19. Accessibility Evaluation Criteria Comply with Common Look and Feel (CLF) Policy and W3C accessibility guidelines. Percentage of pages that are compliant to CLF priority checkpoints. 95% of pages are 95% compliant. CLF validation reports. This information will be presented to channel managers and senior management. It will also be reported to Treasury Board annually. Report Determine performance indicators Collect data to measure against target # Set target Identify goal
  20. 20. Valuable Evaluation Criteria Strategic objectives are met or enhanced through online service delivery and information sharing. Conversion rate on pre-determined tasks or workflows. Increase conversion rates by 2% for at least 3 tasks or workflows. Define important tasks and determine desired paths or workflows through the site then map & track through web analytics. Baseline and final reports will be provided to channel managers with each release. Final report to senior management. Report Determine performance indicators Collect data to measure against target # Set target Identify goal
  21. 21. Desirable Evaluation Criteria End-users are satisfied with the website. Percentage of users who are satisfied with the Web site. 80% of respondents are satisfied. Annual online survey. This information will be reported to programs and presented to channel managers and senior managers annually. Report Determine performance indicators Collect data to measure against target # Set target Identify goal
  22. 22. Usability Evaluation Criteria Users can complete desired tasks as measured by efficiency, effectiveness and satisfaction of task completion. A minimum of 80% should be reached in 4 out of 5 test cases. Task-based usability testing should be performed with the proposed site mock-ups before every major release. This information will be presented to content managers and channel managers. The indicator will be expressed as a composite of the speed, error rate and user satisfaction of users regarding each task. Report Determine performance indicators Collect data to measure against target # Set target Identify goal
  23. 23. Usefulness Evaluation Criteria Content provided is useful to site visitors. Percentage of respondents who agree with the statement “This page was useful”. 80% of respondents agree that “This page was useful”. “ Was this page useful?” on-page feedback form. This information will be reported to content managers and presented to channel managers and senior managers quarterly. Report Determine performance indicators Collect data to measure against target # Set target Identify goal
  24. 24. Credibility Evaluation Criteria Visitors trust the reliability of the content. Percentage of pages that have been reviewed in the previous six months. 95% of pages have been reviewed and updated (if necessary) within previous 6 months. Reports run from the Content Management System on the ‘last reviewed’ information for each page. This information will be reported to content managers, channel managers and senior managers twice a year. Report Determine performance indicators Collect data to measure against target # Set target Identify goal
  25. 25. Findability Evaluation Criteria Easily locate information on the site. Percentage of traffic from search engines is maintained throughout project, increased after project. At least 60% of site traffic referred from search engines. Web Analytics Information will be presented to content managers quarterly. Report Determine performance indicators Collect data to measure against target # Set target Identify goal
  26. 26. Reporting for decision making
  27. 27. Reporting to Content Owners <ul><li>To empower content owners to use data to inform decisions and priorities they should be provided with: </li></ul><ul><ul><li>Tombstone data about visitors (total visits, user demographics) </li></ul></ul><ul><ul><li>User experience indicators – updated as and when available </li></ul></ul><ul><ul><ul><li>Monthly: Findability, Credibility, Usefulness. </li></ul></ul></ul><ul><ul><ul><li>Pre/post tests or surveys: Usability, Value. </li></ul></ul></ul><ul><ul><ul><li>Annually: Accessibility, Desirability. </li></ul></ul></ul><ul><li>Reports should be specific to only the content for which each content owner is responsible. </li></ul><ul><li>Presentations to governance committees should also highlight trends across data collected and suggested improvements for their specific area of the site. </li></ul>
  28. 28. Reporting to Channel Owners <ul><li>To empower channel owners to use data to inform decisions and priorities reports should include: </li></ul><ul><ul><li>Same info as provided to content owners, but rolled up for the entire website. </li></ul></ul><ul><ul><li>Costs – ideally in the form of efficiency ratios described on slide 14 </li></ul></ul><ul><li>Presentations to governance committees should highlight trends across data collected. Web analysts should suggest improvements. This information should be used to inform prioritization of functionality and content needs for each release cycle. </li></ul>
  29. 29. Reporting to Senior Management <ul><li>To empower senior managers (of content or channels) to use data to inform decisions and priorities reports should include </li></ul><ul><ul><li>report summaries mentioned on previous 2 slides depending on their role in organization. </li></ul></ul><ul><li>Presentations to governance committees should demonstrate how data is being used to inform decision-making at other levels. </li></ul><ul><li>Senior managers should be asked to help set priorities based on data provided and other externalities (e.g. resource availability, other unrelated activities that also demand resources). </li></ul>
  30. 30. Reporting to Treasury Board Secretariat* <ul><li>Report to Treasury Board Secretariat through your Departmental Performance Report (DPR) to demonstrate good management practices, results and improvements based on performance data. </li></ul><ul><li>Relevant areas of the Management Accountability Framework (MAF) include: </li></ul><ul><ul><li>AoM 12: Effectiveness of Information Management </li></ul></ul><ul><ul><ul><li>Public can find info/services to complete tasks that meet their needs </li></ul></ul></ul><ul><ul><li>AoM 13: Effectiveness of Information Technology </li></ul></ul><ul><ul><ul><li>Common Look & Feel Standards on Accessibility, Usability </li></ul></ul></ul><ul><ul><li>AoM 20: Citizen-focused Service </li></ul></ul><ul><ul><ul><li>Using client feedback to continuously improve services </li></ul></ul></ul>*This step only relevant for Canadian federal departments who report through this balanced-scorecard approach. See details:
  31. 31. Implementation <ul><li>This framework was developed for implementation as part of a website redesign project for a federal government department if you'd like to use the following steps are suggested: </li></ul><ul><ul><li>Develop an implementation plan for the data that needs to be collected, including when and how it will be used to improve the site. </li></ul></ul><ul><ul><li>Submit data collection needs to technical lead as part of requirements gathering phase to ensure ability to collect data is added over time. </li></ul></ul><ul><ul><li>If starting from scratch, add an indicator or two to reporting cycle as capacity builds, for example, with each release or annual reporting cycle include a new measure. It may not be practical or feasible to implement and collect data all at once. </li></ul></ul><ul><ul><li>Customize the approach for your own implementation. </li></ul></ul>
  32. 32. Customization <ul><li>Specific indicators, targets, data sources may vary from the sample performance measurement framework provided but the goals described herein are probably the same. </li></ul><ul><li>Why? The goals used here describe the expectations most users have of how the web works. </li></ul><ul><li>Be practical. Don't collect data you can't turn into an improvement. </li></ul><ul><li>Add one indicator at a time and build the ability to collect data into each technical release. </li></ul>
  33. 33. Use this framework if… <ul><li>You're not yet collecting any or all data required. </li></ul><ul><li>Your governance or business processes do not enable continuous improvements. </li></ul><ul><li>You're spending money maintaining or updating your site but not sure of the value. </li></ul><ul><li>You don’t know how to define success or use the data you do have to inform decisions. </li></ul><ul><li>You're making decisions based on what the boss wants rather than what site visitors and clients want. </li></ul>
  34. 34. Questions & comments can be forwarded to Laura Wesley via Twitter: @resultsjunkie or online via