1.
LILAC 2010
How was it for you? Evaluating the
iSkillZone
Presentation by
James Bisset & Sarah Brain
Library Services, University of the West of England

2.
University of the West of
England
• Post-1992 university
• Five campuses in and around Bristol
• 29,300 students
• Over 600 courses offered
• Partnerships

3.
Graduate Development
Programme
“To make academic and personal development, the
learning of skills and employability explicit in order to
improve the quality of the student experience
through retention, progression and achievement.”
University of the West of England (2008). Staff Guide to the Graduate Development
Programme, p8.

4.
Graduate Development
Programme
• Level 1: Sense of belonging and the development
of learning skills
• Level 2: Employability and personal development
planning
• Level 3: Academic achievement and progression
to employment or further study

5.
What is the iSkillZone?
• Launched by UWE Library Services Sept 2008
• Ten Information Literacy units
http://www.uwe.ac.uk/library/

6.
Why evaluate?
• Impact on Graduate Development Programme
• Impact on students
• Ease of use and navigation
• Inform future developments and direction

7.
Statistics
Visits during 2009

8.
Overall usage
Total page views Jan-Dec 2009:
481,408

9.
Statistics

10.
Evaluation
Methodology

11.
How and why?
• Usability testing of students
- Is the source easy to navigate?
- Do users find what they want, where they expect to find it?
• Academic staff interviews
- How is it being used in teaching?
- What is working, what could be improved?
• Questionnaires for staff and students
- How is it being used and how is it being received by all users?

12.
Usability testing

13.
• 5 students participated
• Set 7 tasks to find information held on site.
• Morae software records entire session.
• Pre- & post questionnaires to gauge experience of
student and impressions afterwards.
Usability testing

14.
Usability testing
0
5
10
15
20
25
Task 1 Task 2 Task 3 Task 4 Task 5 Task 6 Task 7
NumberofMouseclicks
iSkillZone Evaluation - Student Usability Testing
Mouse clicks for each task
Minimum clicks
S1
S2
S3
S4
S5

15.
Usability testing–
preliminary findings
• Liked the colours, lay-out and interactivity
• Less text
• More images, bullet points and interactive quizzes
• Less jargon & more use of ‘student language’
• Need better search facility within the resource
• ‘Basic’ & ‘Advanced’ versions
• Would use in future – especially for ‘Referencing’,
‘Plagiarism’ and ‘Finding journal articles’

16.
Questionnaires

17.
Questionnaires
Staff Questionnaire (June 2009) – 161
respondents
Student Questionnaire (April 2009) – 493
respondents
Library Staff Questionnaire (June 2009) – 39
respondents

18.
Questionnaires
Staff Questionnaire
59 out of 161 had used in teaching sessions (37%)
Student Questionnaire:
193 out of 493 had used in teaching sessions (39%)
Library Staff Questionnaire:
29 out of 39 had used in teaching / service points (74%)

19.
Usage by section: Academic Staff
0
10
20
30
40
50
60
70
80
90
100
Percentage
of staff
responses

20.
Usage by section: Staff & Students
0
10
20
30
40
50
60
70
80
90
100
Percentage of
staff responses
Percentage of
student
responses

21.
Staff responses
How has the iSkillZone been received?:-
• Q14 asked respondents if they planned to use in the
future?
• 100% of respondents who had used the iSkillZone planned to use it
again.
• 60% of respondents who had not previously used it in teaching
indicated that they planned to in the future.

22.
0
10
20
30
40
50
60
70
80
Recommend to
students
Link to from
Blackboard
Embed Quizzes
in teaching
materials
GDP Level 1 GDP Level 2 GDP Level 3
No of
responses
Intended future useage

23.
Questionnaires
Conclusions from staff
questionnaire
Conclusions from student
questionnaire
Improve publicity and expand
introduction to the iSkillZone’s purpose.
Need better guidance on what it is and
why/if it is important.
Staff who use the iSkillZone generally
think it is a good resource and plan to
continue using it in the future.
Majority of students who use it like it.
Possibly offer alternative ‘context’ with no
tutorials and just quick links for use to
support teaching.
Students introduced via structured
sessions rate it more highly, get more out
of it & more likely to use in future.
Navigation needs to be improved. Navigation on referencing section needs
to be improved.

24.
Questionnaires
Conclusions from library staff questionnaire
Journal articles section possibly needs a redesign or different
context options.
Reduce ‘wordiness’ of sections.
Consult and provide training on sections appropriate for service
desk usage.
Navigation needs to be improved plus the ability to search
across the site.

25.
Questionnaires
“The online quizzes are very
useful in GDP sessions –
the Harvard referencing
system one is particularly
good.”
“...great resource and
mostly very intuitive”
“I will definitely use
it in next year’s
teaching”
“Too wordy”
“I have found it
so helpful –
thank you!”
“Its fun, user-
friendly and
informative!”
“We need a better
understanding of
what it is and what it
offers”
“The backwards
navigation needs to
be sorted out”“Needs clearer and
quicker links”

26.
Questionnaires
“What do students
want?”
“The assumption of this questionnaire is that
skills are taught – this is not always the
case. I would encourage students to use the
iSkillZone to facilitate their own learning.”
“We would like to work
more with you with
regards an academic
skills module”
“...[use a similar format] to support other
generic skills development, e.g. writing
skills, presentation skills, career
development etc.”
“Continue to respond
to student needs
when requested.”

27.
Academic Staff
Interviews

28.
• Six members of staff interviewed in June 2009
• Business, Land and property management, Film
studies, Cultural studies, Humanities, Equine
science
•Purpose: identify impact of iSkillZone in GDP and
other teaching
Academic Staff Interviews

29.
• Impact on GDP
• Impact on other teaching
• What works well
• What doesn’t work well
• Improvements
Questions

30.
Impact on GDP and other
teaching
• Four respondents used the iSkillZone
• Key sections used in GDP: ‘Referencing’,
‘Journals’, ‘Plagiarism’
• Key sections used in other teaching: ‘Literature
searching’, ‘Evaluating resources’
• Varied level of use
• Meets the required needs

31.
What works well
• Content
• Navigation and usability
• Appearance

32.
What doesn’t work well
• Navigation
• Terminology
• Level

33.
Improvements
• Introduce a range of levels from basic to
advanced
•Incorporation of study skills
•Increase interactivity

34.
Conclusions

35.
Some key findings
•Staff who use the iSkillZone generally think it is a good resource, find
it suits most of their needs for GDP and plan to continue using it in the
future.
• Still room for improvement and a need for promotion and publicity.
• Introduction to students in structured teaching sessions seems to
improve students’ usage and response to the iSkillZone.
• Need to meet varying needs, ranging from quick links to more
interactive quizzes and tutorials.

36.
Some key findings
•Significant demand for study skills resources to be incorporated into
the iSkillZone.
• Need to address appropriateness, given the current focus of the
iSkillZone and study skills materials already available.
• Various navigation issues need to be addressed (e.g. search
function, lay-out, wordiness).
• Reassess and clarify guidance and terminology used throughout the
iSkillZone.

37.
How has evaluation
helped?
• Identified some key navigation issues to be addressed.
• Helped prioritise which areas we need to develop / redesign (what is
used most, what people like the least, what people want to see
added).
• Additional publicity and promotion.

38.
Changes already
implemented…
• Navigation on Referencing section redesigned.
• ‘Evaluate’ and ‘Information on the Net’ sections
redesigned as the new ‘Evaluating Information’
section.

39.
The Future…
• Plans in development to engage with ‘Study
Skills’ provision at UWE and look at embedding
resources already developed, in collaboration
with academic staff.
• Identifying and prioritising future developments
from the findings of the evaluation process.