2.
There is a plethora of learning materials of all
sorts now available on-line.
The range of ways in with these materials are
produced and consumed means that there
cannot be one single way to evaluate these
materials.
However there is a major issue with quality
control with regard to this material.
Continued:
3.
A great deal of material is developed without
any evidence of it ever having being tested on
learners.
A logical way to evaluate the quality of
learning materials is to try it out on learners
and get their feedback (both formative and
summative).
It perhaps make most sense to submit this
learner feedback in conjunction with the
learner materials to verify that the materials
have been tested by learners.
4.
There are two main evaluation tools for
learning materials
◦ MERLOT
◦ LORI
Two slightly less used tools are
◦ MicroSoft Evaluators Guide
◦ National Council for Educational Technology for
evaluating CD-DROM
5. MERLOT
(www.merlot.org) is a
repository containing educational
resources classified into seven
broad subject categories: Arts;
Business; Education; Humanities;
Mathematics and Statistics;
Science and Technology; Social
Sciences.
6. There
are three general
categories of evaluation
standards used within
MERLOT:
1. Quality of Content
2. Potential Effectiveness as a
Teaching-Learning Tool
3. Ease of Use
7.
Quality of Content
There are two general elements to quality of
content:
1. Does the software present valid (correct)
concepts, models, and skills?
2. Does the software present educationally
significant concepts, models, and skills for the
discipline?
Continued:
8.
Quality of Content
To evaluate the educational significance of
the content, reviews use the following
guidelines:
Continued:
9.
Quality of Content
◦ Content is core curriculum within the discipline.
◦ Core curriculum topics are typically covered to
some degree in the introductory classes within the
discipline and/or "Everyone teaches it" and/or it is
identified as a core area by the discipline's
professional organizations
◦ Content is difficult to teach and learn.
◦ Content is a pre-requisite for understanding more
advanced material in the discipline
10.
Potential Effectiveness as a
Teaching-Learning Tool
Evaluating POTENTIAL effectiveness is asking
the Peer Reviewer to judge, based on his
expertise as a teacher, whether the learning
material is likely to improve teaching and
learning given the ways the faculty and
students could use the tool.
Continued:
11.
Potential Effectiveness as a TeachingLearning Tool
What stage(s) in the learning process/cycle could
the materials be used?
◦ Explanation or description of the topic/stating the
problem
◦ Demonstration of the curriculum/exploration of the
problem
◦ Practice using the curriculum/analysis of the outcomes
from solving the problem
◦ Applying the curriculum to "new" problems/application
of the outcomes to other problems
Continued:
12.
Potential Effectiveness as a
Teaching-Learning Tool
What is(are) the learning objective(s)? What should
students be able to do after successfully learning
with the materials?
What are the characteristics of the target learner(s)
Continued:
13.
Potential Effectiveness as a TeachingLearning Tool
There are other general elements to effectiveness as
a teaching-learning tool that MERLOT asks reviewers
to consider:
◦ Does the interactive/media-rich presentation of material
improve faculty and students' abilities to teach and learn
the materials?
◦ Can the use of the software be readily integrated into
current curriculum and pedagogy within the discipline?
◦ Can the software be used in a variety of ways to achieve
teaching and learning goals?
◦ Are the teaching-learning goals easy to identify?
◦ Can good learning assignments for using the software
application be written easily?
14. Ease of Use
The basic question underlying the ease of use
standard is: how easy it is for teachers and
students to use the software for the first time?
Elements that affect ease of use include:
Continued:
15. Ease
of Use
◦ Are the labels, buttons, menus, text, and general layout of the
computer interface consistent and visually distinct?
◦ Does the user get trapped in the material?
◦ Can the user get lost easily in the material?
◦ Does the module provide feedback about the system status
and the user's responses?
◦ Does the module provide appropriate flexibility in its use?
◦ Does the learning material require a lot of
documentation, technical support, and/or instruction for most
students to successfully use the software?
◦ Does the material present information in ways that are familiar
for students?
◦ Does the material present information in ways that would be
attractive to students?
16.
eLera is a distributed group that researches
and evaluates e-learning. Their specific
interests include learning objects, eportfolios, and learning design
specifications, and related topics.
eLera provides tools and information for
learning object evaluation and
research, maintains a database of learning
object reviews, and supports communication
and collaboration among
researchers, evaluators and users of online
learning resources.
17.
The goals of eLera are to:
◦ improve the quality of online learning resources
through better design and evaluation
◦ develop effective pedagogical models that
incorporate learning objects
◦ help students, teachers, professors, instructional
designers and others to select pedagogical models
and digital resources that meet their requirements
18.
The Learning Object Review Instrument (LORI)
is used to evaluate the quality of e-learning
resources. LORI is an online form consisting
of rubrics, rating scales and comment fields.
The current version of LORI available from
eLera is version 1.5.
http://www.elera.net/eLera/Home/Articles/L
ORI%201.5.pdf
19. Nine Dimensions of LORI
1.
2.
3.
4.
5.
6.
7.
8.
9.
Content Quality
Learning Goal Alignment
Feedback and Adaptation
Motivation
Presentation Design
Interaction Usability
Accessibility
Reusability
Standards Compliance
20. Nine Dimensions of LORI
1. Content Quality:
Veracity, accuracy, balanced presentation
of ideas, and appropriate level of detail
21. Nine Dimensions of LORI
2. Learning Goal Alignment: Alignment among
learning
goals, activities, assessments, and learner
characteristics
22. Nine Dimensions of LORI
3. Feedback and Adaptation: Adaptive content
or feedback driven by differential learner
input or learner modelling
23. Nine Dimensions of LORI
4. Motivation: Ability to motivate and interest
an identified population of learners
24. Nine Dimensions of LORI
5. Presentation Design: Design of visual and
auditory information for enhanced
learning and efficient mental processing
25. Nine Dimensions of LORI
6. Interaction Usability: Ease of
navigation, predictability of the user
interface, and quality of the interface help
features
26. Nine Dimensions of LORI
7. Accessibility: Design of controls and
presentation formats to accommodate
disabled and mobile learners
27. Nine Dimensions of LORI
8. Reusability: Ability to use in varying
learning contexts and with learners from
differing backgrounds
28. Nine Dimensions of LORI
9. Standards Compliance: Adherence to
international standards and specifications
29.
30.
diagram here is showing individual
reviews merging to a panel review
and then published on web.
31.
INSTRUCTIONAL QUALITY
The purpose of the package is well
defined
The package achieves its defined
purpose
Presentation of content is clear and
logical
The level of difficulty is appropriate to
the target audience
Graphics/colour/sound are used for
appropriate instructional reasons
Use of package is motivational
The package effectively stimulates
student creativity
Feedback on student responses is
effectively employed
The learner controls the rate and
sequence of presentation and review
Instruction is integrated with previous
student experience
Learning is generalisable to an
appropriate range of situations
CONTENT
The content is accurate.
The content has educational
value
The content is free of
race, ethnic, sex, and other
stereotypes
TECHNICAL QUALITY
The user support materials are
comprehensive
The user support materials are
effective
Informative displays are effective
Intended users can easily and
independently operate the
program
Teachers can easily employ the
package
The program appropriately uses
relevant computer capabilities
The program is reliable in
normal use
32.
1. Which computer system will the disc run on?
2. Will your computer system do justice to the illustrations?
3. Is the operation by keyboard or mouse, or both?
4. Can we have the disc for a trial period?
5. Is the language and spelling on the disc Queen’s English or American
English?
6. How much bias is there in the content of the disc?
7. Is printing out easy and intuitive?
8. Can the selected material readily be down-loaded to disc?
9. Can subsections of the disc be searched?
10. Is the software to control the CD-ROM on the disc itself or is it supplied on
a separate floppy disc?
11. Does the software manage memory resources well?
12. What search procedures are available?
13. What is the language level on the disc?
14. Is the user interface tolerant of typing and spelling errors?
15. Can you select exactly what you want to print out or save to disc?
16. Are there any supporting features?
17. Can the illustrations be printed out?
18. Can images be readily transferred?
19. Is there a sound capability to accompany the pictures?
