A Practical Method for
Courseware Evaluation
Aldo de Moor
CommunitySense
PragWeb 2007

The Pragmatic Web of what?
Communal effective use of
Information
Functionalities

Courseware and the Pragmatic Web
Pragmatic Web = context
How to evaluate web functionalities in their context
of use?
Courseware provides an interesting domain
technological environments consisting of multiple
functionality components, together offering a
complete system of info/comm services required for
supporting course needs
Functionality evaluation needed
Too much (costly) functionality
Gaps between required and available functionality
Conflicting functionality requirements

Functionality
Functionality
A set of functions and their specified properties that satisfy
stated or implied needs
Levels of granularity
Systems
Courseware environments
Tools
Blackboard
Modules
Announcements
Functions
Post announcement

A context model of courseware
evaluation

A practical courseware evaluation
method
Portfolio methods
Scores weighed by infrastructure/architecture of organization.
Bedell’s method for ICT investment selection
Functionalities scored on both effectiveness and importance for the
activities to be supported.
Practical method
Simplification of Bedell
No higher-order analyses
Actors (users in their roles) provide, interpret and decide
upon scores
Context: courseware evaluation:
Actors: students, software manager
Tool system level: module
Two questions
How well are course activities supported by various functionality
components?
To what extent are the functionality modules used?

Scores
Elements
I(a) = importance of activity
E.g. I(Information Collection) = 9
I(f,a) = importance of functionality f in supporting
activity a
E.g. I(Virtual Chat, Information Collection) = 4
Q(f,a) = quality of functionality in supporting an
activity
E.g. Q(File Transfer, Submission of Results) = 8

Activity and functionality scores
Activity scores
Σ I(fi,a) * Q(fi,a), for all functionalities 1..i.
Usefulness of the combined technologies for a
particular activity
Relevant for technology users (lectures, students)
Functionality scores
Σ I(aj) * I(f,aj) * Q(f,aj), for all activities 1..j
Usefulness of a particular functionality component
for the combined activities
Relevant for technology developers/maintainers

Experiment: evaluating group
assignment functionality
Two courseware tools: Blackboard, CourseFlow
Goal: making group assignments
Four activities, 11 functionality modules
Actors: 2nd year Information Management students,
software manager
2002: 62 students, 16 groups
2003: 46 students, 12 groups
Questions
Quality of tools for various group assignment activities?
Usefulness of various functionality modules?

Activity scores 2002/2003
Avg. Activity Scores in 2002 and 2003
450
400
350
300
Avg. score
2002
250
200 2003
150
100
50
0
Info coll. Discuss. Subm. Feedback
Activity

Avg. score
se
nd
di
0
200
400
600
800
1000
1200
1400
sc e-
m
us
ai
si
l
on
bo
ar
d
vi
rtu
di al
sc st ch
ud
us at
en
si
on tr
os
bo
te
r
ar
d
vi
rtu (g
ro
al
up
ch
)
fi l
e at
(g
tra
ro
ns
an up
fe )
no r(
un gr
co ce ou
m
ur p)
se
Functionality
en
in ts
fo
co (i n
rm
ur fo
se )
at
io
n
do
cu (i n
m fo
)
en
as ts
si
Avg. Functionality Scores in 2002 and 2003
(in
gn
m fo
)
en
ts
(in
Functionality scores 2002/2003
fo
)
2003
2002

Activity Scores BB/CF (2003)
Avg. Activity Scores Blackboard and CourseFlow (2003)
450
400
350
300
Avg. score
Blackboard
250
200 CourseFlow
150
100
50
0
Info coll. Discuss. Subm. Feedback
Activity

Avg. score
di s en
sc
0
200
400
600
800
1000
1200
1400
1600
us d e
s i -m
on a
b il
di v i oa
r
sc
us s t tua rd
l
u
si
on de c h
at
b nt
v i oa ros
rtu rd te
a r
fi l l c (g ro
e h u
an tra at ( p )
g
nn
c o o u s fe rou
ur nc r (g p)
e
s r
(2003)
c o e i n m e ou
p
n
f
ur
s e orm ts ( )
Functionality
do ati i nfo
)
cu o n
as me (i n
f
si
gn nt s o)
m (in
en fo
)
ts
(in
fo
)
Avg. Functionality Scores Blackboard and CourseFlow
Blackboard
CourseFlow
Functionality Scores BB/CF (2003)

Conclusion
Practical method, shown to be useful for initial
courseware functionality selection
Context-bound, in toto, ex-post evaluation
Simple measures useful for quickscan and
discussion purposes
Open source can perform just as well
“E-learning organization”: continuous evolution /
evaluation needed