This document discusses key aspects of quality as they relate to learning resources and learning objects. It begins with definitions of quality from different perspectives. It then examines various prerequisites of quality learning resources, including content quality, learning goal alignment, feedback and adaptation, motivation, presentation design, interaction usability, accessibility, reusability, and standards compliance. Each of these aspects is discussed in one or more paragraphs. The document also introduces the Learning Object Review Instrument (LORI) as a tool for evaluating learning objects. Finally, it addresses the importance of metadata quality for discoverability and reuse of learning objects.

1. Eleventh Floor
Introduction to Quality11

2. Structure
• Introduction to Quality
• Quality for Learning Resources
• Quality Dimensions
• Evaluating Quality of Learning Resources
• The European view on e-learning
• Quality in Metadata

3. Notion of Quality
• “character with respect to fineness, or grade
of excellence”
• “quality is the customers' perception of the
value of the suppliers' work output”
• “when something is what you expect it to be
then it is perceived as quality. Quality is a
fulfillment of expectation”

4. Some more…
• Quality is fitness for use (Juran, 1974)
• Quality means conformance to
requirements (Crosby, 1979)
• Quality, an inherent or distinguishing
characteristic, a degree or grade of excellence
(American Heritage Dictionary, 1996)

5. Quality is a state of mind?
• “quality is a momentary perception that
occurs when something in our environment
interacts with us, in the pre-intellectual
awareness that comes before rational
thought takes over and begins establishing
order. Judgment of the resulting order is then
reported as good or bad quality value”

6. Working Definition
• Quality is the ongoing process of building and
sustaining relationships by assessing,
anticipating, and fulfilling stated and implied
needs
http://www.qualitydigest.com/html/qualitydef.html

7. Interesting Approach
• There are two forms of quality, and therefore two
definitions and two forms of measurement.
– OBJECTIVE quality is the degree of compliance of a
process or its outcome with a predetermined set of
criteria, which are presumed essential to the ultimate
value it provides
– SUBJECTIVE quality is the level of perceived value
reported by the person who benefits from a process
or its outcome. It may subsume various intermediate
quality measures, both objective and subjective

8. Quality Aspects of a
Learning Resource
!MUST HAVE!

9. Prerequisites (1/3)
• Content quality: Veracity, accuracy, balanced
presentation of ideas, and appropriate level of
detail
• Learning goal alignment: Alignment among
learning goals, activities, assessments, and
learner characteristics
• Feedback and adaptation: Adaptive content
or feedback driven by differential learner
input or learner modeling

10. Prerequisites (2/3)
• Motivation: Ability to motivate and interest
an identified population of learners
• Presentation Design: Design of visual and
auditory information for enhanced learning
and efficient mental processing
• Interaction Usability: Ease of navigation,
predictability of the user interface, and the
quality of the interface help features

11. Prerequisites (3/3)
• Accessibility: Design of controls &
presentation formats to accommodate
disabled and mobile learners
• Reusability: Ability to use in varying learning
contexts and with learners from different
backgrounds
• Standards compliance: Adherence to
international standards and specifications

12. Content Quality (1/2)
• A learning resource is of little or no use if it is
well designed in all other respects but its
content is inaccurate or misleading
• MERLOT’s general evaluation standards divide
quality into three parts:
– content validity,
– potential effectiveness as a teaching-learning tool,
– ease of use

13. Content Quality (2/2)
• Gollnick and Chinn (1991), identified six forms
of bias that are often present in learning
materials:
– invisibility,
– stereotyping,
– selectivity and imbalance,
– unreality,
– fragmentation and isolation,
– language bias

14. Learning goal alignment
• Improving instructional alignment between
teaching and testing in teacher-designed
materials can boost student achievement
• Approaches used to assess alignment at the
broad curricular level require a significant
investment of time,
– Not that appropriate for the smaller chunk sizes of
most learning objects

15. Alignment Triangle
• The learning activities should be aligned with
the stated goals
• The activities should be sufficient to provide
learners with the knowledge and skills to be
successful in the assessments
• The assessments should measure student
achievement of the learning goals

16. Feedback & Adaptation
• Generating effective feedback and adapting to
learner characteristics have been understood
as important goals for educational technology
since at least the early 1960s
• Feedback is a limited form of adaptation in
which the object presents information in
relation to a localized action of the learner

17. Adaptation
• More powerful forms of adaptation use
comprehensive information about the learner
such as:
– Performance history,
– Measures of aptitude,
– Self-reports of preference, aptitude, or mental
state
to individualize the learning environment

18. Examples
• The number of examples presented in interrogatory or
expository format during concept learning (Tennyson &
Christensen, 1988),
• The number of problems to be solved in learning LISP
programming (Anderson, Corbett, Koedinger, & Pelletier, 1995),
• The number and difficulty of test items (Wainer, 2000)
Very few adaptive resources have been made available
to teachers and students outside the research projects
for which they were constructed

19. Motivation
• The motivational quality of a learning object
affects the amount of effort a learner will be
willing to invest in working with and learning
from the object
• Expectancy-value theory
– motivation is a function of the value one places on
a task, one’s expectations about the task, and the
perceived cost of the task

20. How to motivate?
• A learning object that is perceived to be too difficult or too
easy may result in low motivation because learners expect
it to be boring, not possible to complete, or not worth
completing
• Multimedia-heavy interfaces are introduced in an attempt
to bolster learner motivation, but may squander and
misdirect cognitive resources if their use is not integral to
the learning goals of the resource
• Gaining and retaining attention by presenting highly
relevant material and authentic activities that are
meaningful to learners are two approaches that can
increase intrinsic motivation

21. “Motivating” Objects
• Objects that allow learners some control over their own
activities and learning that provide opportunities for high
levels of interactivity and encourage learner participation
and that present game-like challenges will have high
motivational value
• Objects that partition content into discrete components or
levels matched to the ability of the learner lead to
increased self-efficacy and motivation
• Multimedia possibilities afforded by learning objects, which
help learners to visualize complex information and
processes, have also been shown to increase student
motivation

22. Presentation Design
• High-quality presentations incorporate
aesthetics, production values, and design of
instructional messages in ways that are
consistent with principles from research and
theory in cognitive psychology and multimedia
learning and with established conventions for
multimedia design

23. Cognitive Load Theory
• In CLT, intrinsic cognitive load is described as being an
inherent part of the learning task that results from the
interactivity among the elements of to-be-learned
material; this component of cognitive load cannot be
reduced without impacting the learning objectives
• Effective presentation design can increase germane
cognitive load, which can contribute to learning and
schema development. Poor presentation design can
lead to increased extraneous cognitive load, which will
reduce the capacity available for other cognitive
processing

24. Principles for lessening cognitive load
• Coherence principles: exclude unneeded or
irrelevant materials,
• Contiguity principles: Present elements that
the learner must mentally integrate close
together in space and time,
• Modality principle: Explain animations with
audio narration rather than text

25. Interaction Usability (1/2)
• Usability has long been recognized as a critical
issue in software quality and in educational
software in particular
• Usability efforts focus on error prevention, yet
instructional activities are often designed to
encourage students to make and learn from
mistakes (?)

26. Interaction Usability (2/2)
• There are two types of interactions that occur
when a student uses a learning object:
– Interaction with the interface
– Interaction with the content
• Usable designs build on learners’ prior
knowledge of common interface patterns and
symbols and require recognition, rather than
recall, in navigational tasks

27. Accessibility
• People with disabilities may be inadvertently
excluded from the potential benefits of online
learning
– Learning object developers must consider and
accommodate issues of accessibility in the design of
learning objects
• Survey of major providers of instructional
software found that none of the 19 companies
responding to the survey provided accessible
products
– Only 2 companies were enacting plans to address
accessibility issues

28. Accessibility Policies
• In the US, Section 508 of the Rehabilitation Act requires
that federal agencies make their information technologies
accessible to people with disabilities
• At the international level, accessibility is being advanced by
the 14 Web Content Accessibility Guidelines established by
the W3C (World Wide Web Consortium, 1999)
• The IMS Global Learning Consortium has provided
Guidelines for Developing Accessible Learning Applications
(IMS, 2002). They recommend the use of standard technical
formats, especially W3C’s Extensible Hypertext Markup
Language (XHTML)

29. Reusability
• The possibility of creating digital learning
objects that are more granular and more
adaptive to different contexts and learner
needs than conventional materials increases
the opportunities for reuse

30. Reusability in a nutshell
• Learning objects should not contain reference to
specific contextual information such as instructor
names, class locations, or course dates;
• Designing reusable learning objects demands a
thoughtful balance between reusability and fit to
context
– “learning objects need to be produced in such a way that
they are large enough to make sense educationally, but
small enough to be flexibly reused.” Campbell (2003)
• Consider the needs of diverse learners in that context
— those with different backgrounds, abilities, and
disabilities

31. Standards Compliance (1/2)
• The requirement for standards is
incontrovertible. From baseballs to railroad
tracks, standard dimensions and approaches
to design are essential if the cogs of today’s
technological world are to intermesh

32. Standards Compliance (2/2)
• In the context of learning objects, the primary thrust of
standardization efforts has been in the area of
metadata
• It is this metadata that potential users search when
looking for learning objects, yet there are often
discrepancies in approaches to metadata across objects
and repositories
• Even when a consistent metadata template is used,
there can be variation in the quality of the information
entered into each field
• All of these issues directly impact the searchability and
reusability of learning objects

33. Learning Object Review
Instrument (LORI)

35. Evaluating Learning Objects
Questions to ask yourself

36. Ask yourself (1/3)
• Is the learning object appealing overall?
– Aesthetically pleasing, interesting, challenging?
• Is the experience of using the learning object
a pleasant one?
– Post-experience
• Are the technical requirements easily
understood and easily met?
– Is it easy to use, or does it require an intricate
process?

37. Ask yourself (2/3)
• Is it easy to find your way around the learning
object?
– Is the process intrinsic or difficult and tedious?
• Is the content complete and correct?
– Theoretically sound
• Are the activities appropriate to the content?
– Do the activities make sense when compared to
what the content presents?

38. Ask yourself (3/3)
• Is the scope of the learning object suitable:
neither too limited, nor too general for your
purposes?
– Fitness for use
• Does it meet the educational goal you
decided upon?
– Can you use it for a specific purpose you have in
mind?

39. Quality in Metadata

40. Metadata VS Learning Objects
• Metadata affect heavily the adoption of
Learning Objects
– Use & Reuse
– Discovery, etc.
• High-quality metadata for a learning object,
favor its usage
– An intricate part of the learning object economy

41. Metadata Quality Aspects
• See:
– http://www.slideshare.net/nikospala/metadata-
quality-issues-in-learning-repositories
– http://www.slideshare.net/nikospala/metadata-
quality-in-learning-repositories-issues-
considerations
– http://www.slideshare.net/nikospala/metadata-
quality-in-digital-repositories

42. Eleventh Floor
Learning Repository
Business Models
11