evaluating courses with our elearning framework

1. Do you REALLY know how good your modules are?
Kate Boardman

3. Contents
• Me and My Blackboard 
• Minimum Standards
• “Evaluating E-learning: does it work?”
• E-learning Framework
• Scary results
• Over to you

4. Let me tell you a story…
• If you happened to be stopped by one of your
Deputy Vice Chancellors and asked about the state of
e-learning across campus, what would you say?
• If he felt you were providing a diplomatic answer and
asked you to be candid, what would you say?
• In the office, within your team, what would you say?

6. Are you...
• Feeling smug sitting here?
• Bit nervous in case there are home truths?
• Here to be reassured it’s not just you  ?

7. Teesside University
• Regional • 25000 students
• Demography • 60% part time
• FT/PT balance • 60% mature
• HEBP – FE network • 60% Health
• NHS contracts • Increasing WBL
• Police training provision • 6 autonomous Schools

8. Teesside University
• 2005 – 80% modules had a Blackboard site
…using it?
…using it well?
“98% of students said the most useful thing would be
if their other lecturers used it”
• Does it make a difference to learning?
How can you tell?

9. LMS expansion
• Durham model
• York model
• Tees model
“step-change”

10. LMS sites must have:
• Menu areas (appropriately labelled, not visible if empty)
• Staff information and current contact details for all staff involved in the
delivery of the module
• A module guide (including learning outcomes, assessment details and
assessment criteria)
• An overview of how the module will be delivered (structure, scheduling,
balance of face-to-face and online)
• Assessment details for the module, including the process for submission
• A link to DISSC Live
• A link to an up-to-date reading list normally via the L&IS reading list
software
• Copies of relevant presentations, documents or media used to facilitate
learning and teaching (appropriately structured into folders/learning units
eg. by topic or time)
• Communication tools used to provide notices/announcements and
commentary on the progression of the module

11. …and:
• Module information should be checked regularly and
kept up to date e.g. announcements/notices should be
regularly reviewed.
• Links should be appropriately labelled and regularly
checked to ensure currency/no dead links
• Where discussion boards / blogs / wikis are made
available, module leaders should agree protocols for
use with students.
• A link to any commonly used School or discipline
support site should be included as appropriate
• Ensure the content does not infringe copyright
legislation. University advice is available…

12. Minimum Standards
• Clear navigation menu
• Organise content in folders
• Avoid empty areas
• Module Guide or Handbook
• Module delivery schedule
• Assessment information
• Submission instructions
• Assessment feedback
• Staff contact details
• Copy of teaching materials
• Regular Announcements
• Link to DISSC Live
• Link to current reading list

13. Theory
Minimum Standards / Threshold Quality
Standards exist
therefore
modules meet them…
Right?

14. LMS evaluation
• Courseware development evaluation
• Evaluation Cookbook
• Bb Exemplary Course Rubric
• Extensive checklists
• Pathfinder E-learning Framework
• Peer Review/Observation
• Student e-assistants

15. Pathfinder Project
• Looking at “e-quality” – provision and QA of e-
learning…
…across Schools
…across Levels
…across Subject Groups
…across Staff…

16. Framework
Level Presentation Organisation Communication Construction
1 Minimum VLE template – no Documents placed online with no No or little online communication. No or little opportunity for
customisation. or little organisation. Announcements/ notices only. students to construct their
Online materials are own resources. Material is
standard documents written by teaching staff.
rather than created
specifically for the module.
2 Customised items Some use of folders to organise Regular appropriate announcements Learning materials are not
documents. Descriptions with module links. constructed or added to by
added to file attachments. Opportunities for basic peer- students. Construction
Some materials are to-peer online does occur through online
created specifically for the communication. student discussion of
module. subject.
3 Banner image, some images in module. Consistent thematic folder use, Increased use of online More construction of ideas by
appropriate range of communication. Use of blogs students through activities
materials, some interactive for regular notices, peer-to- online such as those
activities, potential for peer online communication. related to group work or
electronic submission. Opportunity to offer online any online activity that
Most materials are created feedback. enables students to test or
specifically for the module. apply the subject.
4 Thematic graphics in module, edited Alternative formats or styles with Online communication is likely to It is an integral part of this level
menu, student-focused design. a broad range of digital play an important role in the that students use online
interactive media. Specially module. Use of a ‘what’s new’ features to create and
designed and integrated strategy. Integrated peer-to- contribute to items as they
learning activities. peer online communication, are then using their
integrated online feedback learning experience to
throughout the module. construct new ideas.

17. Presentation
• Presentation is important in creating an
interesting online module presence. A higher
quality of online module site would be
expected to have a clear and attractive layout
which has been designed specifically with
both the module and the respective students
in mind.

18. Organisation
• One of the aspects of online modules which
students comment on most as a problem is
sites which are badly organised. A well
constructed site should be very easy for a
student to navigate and should have clearly
labelled files and folders so that a student can
find what they are looking for as quickly as
possible.

19. Communication
• In planning how one will use communication in your
module, it is important to consider both
communication in terms of how to communicate
with the students, and how it might be helpful to
enable students to communicate with each other.
Communication about the module that comes from
staff to the students will obviously be important to
the students and will therefore have to be easy to
find for the student. There should be clear rules of
conduct provided for any online student discussions
and these should be moderated by staff.

20. Construction
• Construction is one element of this model in
which it may be appropriate to consider the level
of module that is being taught as it refers to
learning activities where the students construct
new materials or ideas online. In a higher level of
module students would obviously be expected to
demonstrate a higher level of knowledge and
understanding of the subject matter meaning
that such activities may well be most appropriate
for higher level modules.

21. Level 1
Level Presentation Organisation Communication Construction
1 Minimum VLE Documents placed No /little online No or little
template – no online with no communication. opportunity
customisation. or little Announcements/ for students
organisation. notices only. to construct
Online their own
materials are resources.
standard Material is
documents written by
rather than teaching staff.
created
specifically for
the module.

22. Presentation
• Query: how many modules have changed
their menu style from the provided default?
Module design changed from default
changed,
914, 26%
unchanged,
2579, 74%

23. Organisation
• Query: how many empty folders or menu
areas are there in the module?
Num ber of em pty folders/m enu areas
2%
31% 0
37% 1
2
3
4
5+
14%
6%
10%

24. Communication
• Query: how many Announcements (available)
are there in the module?
Num ber of visible Announcem ents
6% 3%
0
24% 1
2-10
60% 11-20
20+
7%

25. Construction
• Query: How many opportunities are there for
students to construct content/knowledge in
the module? Opportunities for Student Construction
0%
1%
5%
5%
0
1
2-5
6-10
11+
89%

26. Are you...
• Feeling smug sitting here?
• Bit nervous in case there are home truths?
• Here to be reassured it’s not just you  ?

28. Smug people...
• Still smug – you’re sure you’re far ahead?
• Still smug, but maybe not by much?
• Gritting your teeth, smiling and keeping up
smug anyway?

29. My friends...
• Relieved, you definitely aren’t this bad?
• Yes, you’re pretty much feeling our pain?
• What, truthfully, honestly – maybe you’re
worse?

31. Where next?
• What do you do with those results?
– More staff
– New resources
– More ‘desk sessions’
• Nothing?
– nothing different
– ‘any intervention makes people think about their
teaching’

32. Return to sound principles
• Encourages contacts between students and faculty
• Develops reciprocity and cooperation among
students
• Uses active learning techniques
• Gives prompt feedback
• Emphasizes time on task
• Communicates high expectations
• Respects diverse talents and ways of learning
Chickering A W & Gamson Z F (1987), "Seven Principles for Good Practice in Undergraduate Education" AAHE Bulletin

33. Keep on chipping away…
• Promotion of learning through discussion
groups
• Self-test assessment with feedback
• Media-rich content
• Development of ‘e-tivities’
• Back to basics
• Thinking out of the box (hard)
– Andy Ramsden’s “up with…”, “small things”, “ditch workshops”, write
up case studies”, “just-in-time work with individuals”

34. Level 2?
Level Presentation Organisation Communication Construction
2 Customised Some use of folders Regular appropriate Learning materials
items to organise announcements are not
documents. with module constructed or
Descriptions links. added to by
added to file Opportunities students.
attachments. for basic peer-to- Construction
Some materials peer online does occur
are created communication. through online
specifically for the student
module. discussion of
subject.

35. Discuss…
• Are we alone?
• Is our framework any good?
• Where does it go from here?
• How do you quantify the quality?
• Can we usefully collaborate as a community?

36. Presentation
• Query: how many modules have changed
their menu style from the provided default?
Module design changed from default
changed,
914, 26%
unchanged,
2579, 74%

37. Organisation
• Query: how many empty folders or menu
areas are there in the module?
Num ber of em pty folders/m enu areas
2%
31% 0
37% 1
2
3
4
5+
14%
6%
10%

38. Communication
• Query: how many Announcements (available)
are there in the module?
Num ber of visible Announcem ents
6% 3%
0
24% 1
2-10
60% 11-20
20+
7%

39. Construction
• Query: How many opportunities are there for
students to construct content/knowledge in
the module? Opportunities for Student Construction
0%
1%
5%
5%
0
1
2-5
6-10
11+
89%

40. Discuss…
• Are we alone?
• Is our framework any good?
• Where does it go from here?
• How do you quantify the quality?
• Can we usefully collaborate as a community?

42. Resources
• E-learning Framework
• Framework Report
• Evaluating E-learning (Interact)
• Slides
• Web access to vote in the polls!
• Posting for comments on Connections
• Thanks! Kate x

43. Resources
• E-learning Framework
• Framework Report
• Evaluating E-learning (Interact)
• Slides
• Bb9 eat.tees.ac.uk
• Twitter elearningatees