Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How was it for you? Evaluating the iSkillZone. Bisset & Brain


Published on

Presented at LILAC 2010

Published in: Education
  • Be the first to comment

  • Be the first to like this

How was it for you? Evaluating the iSkillZone. Bisset & Brain

  1. 1. LILAC 2010 How was it for you? Evaluating the iSkillZone Presentation by James Bisset & Sarah Brain Library Services, University of the West of England
  2. 2. University of the West of England • Post-1992 university • Five campuses in and around Bristol • 29,300 students • Over 600 courses offered • Partnerships
  3. 3. Graduate Development Programme “To make academic and personal development, the learning of skills and employability explicit in order to improve the quality of the student experience through retention, progression and achievement.” University of the West of England (2008). Staff Guide to the Graduate Development Programme, p8.
  4. 4. Graduate Development Programme • Level 1: Sense of belonging and the development of learning skills • Level 2: Employability and personal development planning • Level 3: Academic achievement and progression to employment or further study
  5. 5. What is the iSkillZone? • Launched by UWE Library Services Sept 2008 • Ten Information Literacy units
  6. 6. Why evaluate? • Impact on Graduate Development Programme • Impact on students • Ease of use and navigation • Inform future developments and direction
  7. 7. Statistics Visits during 2009
  8. 8. Overall usage Total page views Jan-Dec 2009: 481,408
  9. 9. Statistics
  10. 10. Evaluation Methodology
  11. 11. How and why? • Usability testing of students - Is the source easy to navigate? - Do users find what they want, where they expect to find it? • Academic staff interviews - How is it being used in teaching? - What is working, what could be improved? • Questionnaires for staff and students - How is it being used and how is it being received by all users?
  12. 12. Usability testing
  13. 13. • 5 students participated • Set 7 tasks to find information held on site. • Morae software records entire session. • Pre- & post questionnaires to gauge experience of student and impressions afterwards. Usability testing
  14. 14. Usability testing 0 5 10 15 20 25 Task 1 Task 2 Task 3 Task 4 Task 5 Task 6 Task 7 NumberofMouseclicks iSkillZone Evaluation - Student Usability Testing Mouse clicks for each task Minimum clicks S1 S2 S3 S4 S5
  15. 15. Usability testing– preliminary findings • Liked the colours, lay-out and interactivity • Less text • More images, bullet points and interactive quizzes • Less jargon & more use of ‘student language’ • Need better search facility within the resource • ‘Basic’ & ‘Advanced’ versions • Would use in future – especially for ‘Referencing’, ‘Plagiarism’ and ‘Finding journal articles’
  16. 16. Questionnaires
  17. 17. Questionnaires Staff Questionnaire (June 2009) – 161 respondents Student Questionnaire (April 2009) – 493 respondents Library Staff Questionnaire (June 2009) – 39 respondents
  18. 18. Questionnaires Staff Questionnaire 59 out of 161 had used in teaching sessions (37%) Student Questionnaire: 193 out of 493 had used in teaching sessions (39%) Library Staff Questionnaire: 29 out of 39 had used in teaching / service points (74%)
  19. 19. Usage by section: Academic Staff 0 10 20 30 40 50 60 70 80 90 100 Percentage of staff responses
  20. 20. Usage by section: Staff & Students 0 10 20 30 40 50 60 70 80 90 100 Percentage of staff responses Percentage of student responses
  21. 21. Staff responses How has the iSkillZone been received?:- • Q14 asked respondents if they planned to use in the future? • 100% of respondents who had used the iSkillZone planned to use it again. • 60% of respondents who had not previously used it in teaching indicated that they planned to in the future.
  22. 22. 0 10 20 30 40 50 60 70 80 Recommend to students Link to from Blackboard Embed Quizzes in teaching materials GDP Level 1 GDP Level 2 GDP Level 3 No of responses Intended future useage
  23. 23. Questionnaires Conclusions from staff questionnaire Conclusions from student questionnaire Improve publicity and expand introduction to the iSkillZone’s purpose. Need better guidance on what it is and why/if it is important. Staff who use the iSkillZone generally think it is a good resource and plan to continue using it in the future. Majority of students who use it like it. Possibly offer alternative ‘context’ with no tutorials and just quick links for use to support teaching. Students introduced via structured sessions rate it more highly, get more out of it & more likely to use in future. Navigation needs to be improved. Navigation on referencing section needs to be improved.
  24. 24. Questionnaires Conclusions from library staff questionnaire Journal articles section possibly needs a redesign or different context options. Reduce ‘wordiness’ of sections. Consult and provide training on sections appropriate for service desk usage. Navigation needs to be improved plus the ability to search across the site.
  25. 25. Questionnaires “The online quizzes are very useful in GDP sessions – the Harvard referencing system one is particularly good.” “...great resource and mostly very intuitive” “I will definitely use it in next year’s teaching” “Too wordy” “I have found it so helpful – thank you!” “Its fun, user- friendly and informative!” “We need a better understanding of what it is and what it offers” “The backwards navigation needs to be sorted out”“Needs clearer and quicker links”
  26. 26. Questionnaires “What do students want?” “The assumption of this questionnaire is that skills are taught – this is not always the case. I would encourage students to use the iSkillZone to facilitate their own learning.” “We would like to work more with you with regards an academic skills module” “...[use a similar format] to support other generic skills development, e.g. writing skills, presentation skills, career development etc.” “Continue to respond to student needs when requested.”
  27. 27. Academic Staff Interviews
  28. 28. • Six members of staff interviewed in June 2009 • Business, Land and property management, Film studies, Cultural studies, Humanities, Equine science •Purpose: identify impact of iSkillZone in GDP and other teaching Academic Staff Interviews
  29. 29. • Impact on GDP • Impact on other teaching • What works well • What doesn’t work well • Improvements Questions
  30. 30. Impact on GDP and other teaching • Four respondents used the iSkillZone • Key sections used in GDP: ‘Referencing’, ‘Journals’, ‘Plagiarism’ • Key sections used in other teaching: ‘Literature searching’, ‘Evaluating resources’ • Varied level of use • Meets the required needs
  31. 31. What works well • Content • Navigation and usability • Appearance
  32. 32. What doesn’t work well • Navigation • Terminology • Level
  33. 33. Improvements • Introduce a range of levels from basic to advanced •Incorporation of study skills •Increase interactivity
  34. 34. Conclusions
  35. 35. Some key findings •Staff who use the iSkillZone generally think it is a good resource, find it suits most of their needs for GDP and plan to continue using it in the future. • Still room for improvement and a need for promotion and publicity. • Introduction to students in structured teaching sessions seems to improve students’ usage and response to the iSkillZone. • Need to meet varying needs, ranging from quick links to more interactive quizzes and tutorials.
  36. 36. Some key findings •Significant demand for study skills resources to be incorporated into the iSkillZone. • Need to address appropriateness, given the current focus of the iSkillZone and study skills materials already available. • Various navigation issues need to be addressed (e.g. search function, lay-out, wordiness). • Reassess and clarify guidance and terminology used throughout the iSkillZone.
  37. 37. How has evaluation helped? • Identified some key navigation issues to be addressed. • Helped prioritise which areas we need to develop / redesign (what is used most, what people like the least, what people want to see added). • Additional publicity and promotion.
  38. 38. Changes already implemented… • Navigation on Referencing section redesigned. • ‘Evaluate’ and ‘Information on the Net’ sections redesigned as the new ‘Evaluating Information’ section.
  39. 39. The Future… • Plans in development to engage with ‘Study Skills’ provision at UWE and look at embedding resources already developed, in collaboration with academic staff. • Identifying and prioritising future developments from the findings of the evaluation process.