Assessing UserSatisfaction for theDigimap Service:Surveys, spread sheets and a differentkind of chart.
The 2011 Winter Survey • Required as part of the JISC services portfolio review • Assess user satisfaction as part of the review of all their services • Common questions across all JISC services • In house design for interface and database • Bristol Online Surveys used in the past • Detailed review for Geo-Services in 2011-12 • Survey made more prominent in a pop-up • Users made to wait 10 seconds for the pop-up to disappear • Results were made publicly available online: http://edina.ac.uk/impact/
Considerations before starting a survey Planning Surveys are exercises in Public Engagement, be prepared! • Don’t carry them out lightly, have a plan • Survey are an inconvenience, weigh the costs with the benefits • Plan carefully how you are going to carry out the survey • Plan how you are going to give feedback • Plan how you are going to handle complaints Good and swift feedback can turn someone with a negative view of your service into your best customer; if they have been part of an improvement they will often help promote your services. “Tell me and Ill forget; show me and I may remember; involve me and Ill understand.” Chinese Proverb
Considerations before starting a survey (2) Timing Survey fatigue can dent your response rate and increase the amount of overly negative responses you get through misrepresentation: • When possible, avoid times that you know your customers/users are having to fill in lots of other surveys. • Don’t survey too often / balance with length. • Find out the best time of year to survey your users. Length Keep it as short as possible, and use yes/no answers and options as much as you can to minimise the time users need to fill it in.
Considerations before starting a survey (3) Misrepresentation This can be further mitigated by the amount of information you give respondents: • Tell them why they are being surveyed • Tell them how there responses will be used • If possible, show how previous results have affected the service • Feedback the results of the survey • Feedback how the results have changed the service The more the respondents understand why they are doing the survey and what impact it is going to have, the more useful the responses they are likely to give.
The Questions – Who are the users? 1. Where do you study/work? 2. In which role are you using Digimap? Student, Postgraduate, Lecturer, Researcher, Information professional 3. Please indicate your subject area: Agriculture, food, and forestry Law Architecture and planning Mathematics and computer science Biological sciences Medicine including dentistry Business and management studies Modern languages and area studies Communication and media studies Nursing, midwifery and allied health Creative and performing arts Physical sciences Education and research methods Psychology Engineering Social sciences Geography and environment Veterinary medicine Humanities Automatically answered for registered users
The Questions – How do you find the service? Q1. On average, how often do you use this service? – Several times a week, Once a week, Once or twice a month, Less often, This is my first use Q2. How do you find using Digimap? – Very easy, Easy, Neither easy nor difficult, Difficult, Very difficult Q3. My work would take longer if Digimap were not available. – Strongly agree, Agree, Neither agree nor disagree, Disagree, Strongly disagree Q3b. [optional] Why do you say that?
The Questions – What else is there? Q4. Would you use an alternative if Digimap was not available? – Yes, No, Don’t Know Q4b. If yes, what would you use? Q5. I would recommend Digimap to others. – Strongly agree, Agree, Neither agree nor disagree, Disagree, Strongly disagree Q5b. [optional] Please tell us why you would, or would not, recommend this service:
The Questions – What do you use Digimap for? Q6a. What do you use Digimap for? Tick all that apply to you: – For Research – For teaching the use of maps, spatial data or GIS – For Post Graduate Thesis – For teaching materials in a general subject – For Undergraduate Thesis – For supporting students use of maps, spatial data or GIS – For Coursework – For personal interest Q6b. [optional] We’d really like to know more about what you are doing with Digimap, please give as much detail about what you use the service for below: Can we contact you about using your example as a case study?
The Questions – What do you use the most? Q7a. Rate the following elements of the service in terms of how important they are to your studies, research or work: Essential, Important, Useful, Little value, I don’t use it Viewing Online Maps Downloading Data Accessing Streamed Maps Gazetteer and Postcode Query tools Support Materials Q7b. [optional] We’d really like to know how important the service is to you; please share any stories you may have about how Digimap has helped you:
The Questions – What do you want in the future? Q8a. What would you like Digimap to provide next? Rate the following options in terms of how useful they would be to your studies, research or work: Essential, Important, Useful, Little value, I wouldn’t use it Mobile Maps More Data Products More Data Formats Advanced Online GIS Tools More printing options More mapping support More GIS support Q8b. [optional] Please give us any details about the choices you would use from the table above such as which products, formats, options or support you would like to see:
The Results • 8537 responses received • The published results are based on the 7000+ responses from returning users (omitting those who indicated this was their first time): • OS = 4969 • Historic = 1462 • Geology = 776 • Marine = 117 • Results in full: http://edina.ac.uk/impact/ • Thank you to anyone who filled one in!
Doughnut Charts Digimap User Disciplines 2010-11 & 2011-12 Agriculture, food, and forestry Architecture and planning Biological sciences Business and management studies 2011 - 12 Communication and media studies Creative and performing arts Education and research methods Engineering Geography and environment 2010 - 11 Humanities Law Mathematics and computer science Medicine including dentistry Modern languages and area studies Nursing, midwifery and allied health Physical sciences Psychology Social sciences Veterinary medicine Academic Support / Other Mmm... Donuts!
Diverse Users Agriculture, food, and forestry OS Architecture and planning Historic Biological sciences Business and management studies Architecture and planning 8% Communication and media studies 12% Creative and performing arts Geography and environment Education and research methods Engineering Humanities Geography and environment Humanities 19% Law 22% Mathematics and computer science 53% Medicine including dentistry 47% Modern languages and area studies Nursing, midwifery and allied health Physical sciences Psychology Social sciences Veterinary medicine Academic Support English Lit, History, Fine Art, Equine Studies, Medicine Geology Architecture and planning Marine Architecture and planning Engineering 14% Biological sciences 20% Geography and environment 30% Engineering Physical sciences 41% Geography and environment 19% Physical sciences 15% 17% 19% 13%
Easy To Use? OS Historic Very Easy 1% Very Easy 8% 1% 16% 5% 16% Easy Easy Neither Neither 31% 30% Difficult Difficult Very Difficult Very Difficult 46% 47% Geology YES Marine 1% 4% Very Easy 1% 9% Very Easy 19% 14% Easy Easy 26% Neither Neither Difficult Difficult Very Difficult 32% 44% Very Difficult 50%
Difficulties Users Experienced • About 62% of respondents found the service easy or very easy to use… but what about the others? • These were the main areas of concern: • Waiting for maps • Printing • Complex interfaces • Registration
Priorities for the Future OS Historic Geology Marine 1. Printing 1. Printing 1. Printing 1. GIS Tools 2. Mapping Support 2. Mapping Support 2. Mapping Support 2. GIS Support 3. Data Formats 3. Data Products 3. GIS Support 3. Data Formats Over 300 people commented in this section
Priorities for the Future (2) • The Print interface needs to be enhanced. • New functionality: Legends, 2up maps & more formats • We need to offer downloads in many more formats. • Pre-generate data & on the fly transformations • Mapping and GIS Support • The new home page resource centre • New Datasets • We are continually looking to add to Digimap • Environment Digimap
Survey Results Summary • The service used by an incredibly broad range of people from across academia. • The service is well liked and very well used: ...but... • There is more that can be done to make the service: • Faster • Easier to use • Meet all it’s users requirements • We continually need feedback to make sure Digimap continues to meet its users needs.
Lessons Learned from the survey JISC want us to carry out the survey every year. Changes we need to make: • No survey at all for first time users • Easier to ignore • Tighten up the wording of certain questions • Timing of the appearance of the pop-up • More thorough testing Important considerations before making changes: • Surveys need to be comparable to previous years (especially questions 1-6) • We need a good amount of responses
Beyond the Annual Survey Continual Feedback Via: • Helpdesk • No news is good news! • “Comment on this service” links • A mechanism for people to let us know about something while they are doing it • Surprising number of glowing reviews from this • Shorter, one question pop-ups to gauge opinion on certain features etc. are also being considered. • This has proved successful in the past for users waiting for downloads • Social media • Popular with certain groups of users • Needs careful monitoring as it is all public
Beyond the Annual Survey (2) More in-depth user engagement during service development • Usage statistics are being used to create personae • Users who fit into the categories are asked to test services in development • Iterative testing and design phases are used to hone the user experience • USeD Project: http://used.blogs.edina.ac.uk/