Published on

Published in: Economy & Finance, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide


  1. 1. A Practical Method for Courseware Evaluation Aldo de Moor CommunitySense PragWeb 2007
  2. 2. The Pragmatic Web of what? Communal effective use of Information Functionalities
  3. 3. Courseware and the Pragmatic Web Pragmatic Web = context How to evaluate web functionalities in their context of use? Courseware provides an interesting domain technological environments consisting of multiple functionality components, together offering a complete system of info/comm services required for supporting course needs Functionality evaluation needed Too much (costly) functionality Gaps between required and available functionality Conflicting functionality requirements
  4. 4. Functionality Functionality A set of functions and their specified properties that satisfy stated or implied needs Levels of granularity Systems Courseware environments Tools Blackboard Modules Announcements Functions Post announcement
  5. 5. A context model of courseware evaluation
  6. 6. A practical courseware evaluation method Portfolio methods Scores weighed by infrastructure/architecture of organization. Bedell’s method for ICT investment selection Functionalities scored on both effectiveness and importance for the activities to be supported. Practical method Simplification of Bedell No higher-order analyses Actors (users in their roles) provide, interpret and decide upon scores Context: courseware evaluation: Actors: students, software manager Tool system level: module Two questions How well are course activities supported by various functionality components? To what extent are the functionality modules used?
  7. 7. Scores Elements I(a) = importance of activity E.g. I(Information Collection) = 9 I(f,a) = importance of functionality f in supporting activity a E.g. I(Virtual Chat, Information Collection) = 4 Q(f,a) = quality of functionality in supporting an activity E.g. Q(File Transfer, Submission of Results) = 8
  8. 8. Activity and functionality scores Activity scores Σ I(fi,a) * Q(fi,a), for all functionalities 1..i. Usefulness of the combined technologies for a particular activity Relevant for technology users (lectures, students) Functionality scores Σ I(aj) * I(f,aj) * Q(f,aj), for all activities 1..j Usefulness of a particular functionality component for the combined activities Relevant for technology developers/maintainers
  9. 9. Experiment: evaluating group assignment functionality Two courseware tools: Blackboard, CourseFlow Goal: making group assignments Four activities, 11 functionality modules Actors: 2nd year Information Management students, software manager 2002: 62 students, 16 groups 2003: 46 students, 12 groups Questions Quality of tools for various group assignment activities? Usefulness of various functionality modules?
  10. 10. Activity scores 2002/2003 Avg. Activity Scores in 2002 and 2003 450 400 350 300 Avg. score 2002 250 200 2003 150 100 50 0 Info coll. Discuss. Subm. Feedback Activity
  11. 11. Avg. score se nd di 0 200 400 600 800 1000 1200 1400 sc e- m us ai si l on bo ar d vi rtu di al sc st ch ud us at en si on tr os bo te r ar d vi rtu (g ro al up ch ) fi l e at (g tra ro ns an up fe ) no r( un gr co ce ou m ur p) se Functionality en in ts fo co (i n rm ur fo se ) at io n do cu (i n m fo ) en as ts si Avg. Functionality Scores in 2002 and 2003 (in gn m fo ) en ts (in Functionality scores 2002/2003 fo ) 2003 2002
  12. 12. Activity Scores BB/CF (2003) Avg. Activity Scores Blackboard and CourseFlow (2003) 450 400 350 300 Avg. score Blackboard 250 200 CourseFlow 150 100 50 0 Info coll. Discuss. Subm. Feedback Activity
  13. 13. Avg. score di s en sc 0 200 400 600 800 1000 1200 1400 1600 us d e s i -m on a b il di v i oa r sc us s t tua rd l u si on de c h at b nt v i oa ros rtu rd te a r fi l l c (g ro e h u an tra at ( p ) g nn c o o u s fe rou ur nc r (g p) e s r (2003) c o e i n m e ou p n f ur s e orm ts ( ) Functionality do ati i nfo ) cu o n as me (i n f si gn nt s o) m (in en fo ) ts (in fo ) Avg. Functionality Scores Blackboard and CourseFlow Blackboard CourseFlow Functionality Scores BB/CF (2003)
  14. 14. Conclusion Practical method, shown to be useful for initial courseware functionality selection Context-bound, in toto, ex-post evaluation Simple measures useful for quickscan and discussion purposes Open source can perform just as well “E-learning organization”: continuous evolution / evaluation needed