A Practical Method for Courseware Evaluation Aldo de Moor   CommunitySense PragWeb 2007
The Pragmatic Web of what? <ul><li>Communal effective use of </li></ul><ul><li>Information  </li></ul><ul><li>Functionalit...
Courseware and the Pragmatic Web <ul><li>Pragmatic Web = context </li></ul><ul><ul><li>How to evaluate web functionalities...
Functionality <ul><li>Functionality </li></ul><ul><ul><li>A set of functions and their specified properties that satisfy s...
A context model of courseware evaluation
A practical courseware evaluation method  <ul><li>Portfolio methods </li></ul><ul><ul><li>Scores weighed by infrastructure...
Scores <ul><li>Elements </li></ul><ul><ul><li>I(a)  = importance of activity </li></ul></ul><ul><ul><ul><li>E.g. I(Informa...
Activity and functionality scores <ul><li>Activity scores </li></ul><ul><ul><li>   I(f i ,a) * Q(f i ,a), for all functio...
Experiment: evaluating group assignment functionality <ul><li>Two courseware tools: Blackboard, CourseFlow </li></ul><ul><...
Activity scores 2002/2003
Functionality scores 2002/2003
Activity Scores BB/CF (2003)
Functionality Scores BB/CF (2003)
Conclusion <ul><li>Practical method, shown to be useful for initial courseware functionality selection </li></ul><ul><li>C...
Upcoming SlideShare
Loading in …5
×

A Practical Method for Courseware Evaluation

145 views

Published on

A. de Moor (2007). A Practical Method for Courseware Evaluation. In Proc. of the 2nd International Conference on the Pragmatic Web (PragWeb 2007), Tilburg, the Netherlands, October 22-23, 2007. ACM International Conference Proceedings Series, Vol. 280, pp.57-63.

Published in: Technology, Education
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
145
On SlideShare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
3
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • A Practical Method for Courseware Evaluation

    1. 1. A Practical Method for Courseware Evaluation Aldo de Moor CommunitySense PragWeb 2007
    2. 2. The Pragmatic Web of what? <ul><li>Communal effective use of </li></ul><ul><li>Information </li></ul><ul><li>Functionalities </li></ul>
    3. 3. Courseware and the Pragmatic Web <ul><li>Pragmatic Web = context </li></ul><ul><ul><li>How to evaluate web functionalities in their context of use? </li></ul></ul><ul><li>Courseware provides an interesting domain </li></ul><ul><ul><ul><li>technological environments consisting of multiple functionality components, together offering a complete system of info/comm services required for supporting course needs </li></ul></ul></ul><ul><li>Functionality evaluation needed </li></ul><ul><ul><ul><li>Too much (costly) functionality </li></ul></ul></ul><ul><ul><ul><li>Gaps between required and available functionality </li></ul></ul></ul><ul><ul><ul><li>Conflicting functionality requirements </li></ul></ul></ul>
    4. 4. Functionality <ul><li>Functionality </li></ul><ul><ul><li>A set of functions and their specified properties that satisfy stated or implied needs </li></ul></ul><ul><ul><li>Levels of granularity </li></ul></ul><ul><ul><ul><li>Systems </li></ul></ul></ul><ul><ul><ul><ul><li>Courseware environments </li></ul></ul></ul></ul><ul><ul><ul><li>Tools </li></ul></ul></ul><ul><ul><ul><ul><li>Blackboard </li></ul></ul></ul></ul><ul><ul><ul><li>Modules </li></ul></ul></ul><ul><ul><ul><ul><li>Announcements </li></ul></ul></ul></ul><ul><ul><ul><li>Functions </li></ul></ul></ul><ul><ul><ul><ul><li>Post announcement </li></ul></ul></ul></ul>
    5. 5. A context model of courseware evaluation
    6. 6. A practical courseware evaluation method <ul><li>Portfolio methods </li></ul><ul><ul><li>Scores weighed by infrastructure/architecture of organization. </li></ul></ul><ul><li>Bedell’s method for ICT investment selection </li></ul><ul><ul><li>Functionalities scored on both effectiveness and importance for the activities to be supported. </li></ul></ul><ul><li>Practical method </li></ul><ul><ul><li>Simplification of Bedell </li></ul></ul><ul><ul><ul><li>No higher-order analyses </li></ul></ul></ul><ul><li>Actors (users in their roles) provide , interpret and decide upon scores </li></ul><ul><ul><li>Context: courseware evaluation: </li></ul></ul><ul><ul><li>Actors: students, software manager </li></ul></ul><ul><ul><li>Tool system level: module </li></ul></ul><ul><li>Two questions </li></ul><ul><ul><li>How well are course activities supported by various functionality components? </li></ul></ul><ul><ul><li>To what extent are the functionality modules used? </li></ul></ul>
    7. 7. Scores <ul><li>Elements </li></ul><ul><ul><li>I(a) = importance of activity </li></ul></ul><ul><ul><ul><li>E.g. I(Information Collection) = 9 </li></ul></ul></ul><ul><ul><li>I(f,a) = importance of functionality f in supporting activity a </li></ul></ul><ul><ul><ul><li>E.g. I(Virtual Chat, Information Collection) = 4 </li></ul></ul></ul><ul><ul><li>Q(f,a) = quality of functionality in supporting an activity </li></ul></ul><ul><ul><ul><li>E.g. Q(File Transfer, Submission of Results) = 8 </li></ul></ul></ul>
    8. 8. Activity and functionality scores <ul><li>Activity scores </li></ul><ul><ul><li> I(f i ,a) * Q(f i ,a), for all functionalities 1..i. </li></ul></ul><ul><ul><li>Usefulness of the combined technologies for a particular activity </li></ul></ul><ul><ul><li>Relevant for technology users (lectures, students) </li></ul></ul><ul><li>Functionality scores </li></ul><ul><ul><li> I(a j ) * I(f,a j ) * Q(f,a j ), for all activities 1..j </li></ul></ul><ul><ul><li>Usefulness of a particular functionality component for the combined activities </li></ul></ul><ul><ul><li>Relevant for technology developers/maintainers </li></ul></ul>
    9. 9. Experiment: evaluating group assignment functionality <ul><li>Two courseware tools: Blackboard, CourseFlow </li></ul><ul><li>Goal: making group assignments </li></ul><ul><li>Four activities, 11 functionality modules </li></ul><ul><li>Actors: 2 nd year Information Management students, software manager </li></ul><ul><ul><li>2002: 62 students, 16 groups </li></ul></ul><ul><ul><li>2003: 46 students, 12 groups </li></ul></ul><ul><li>Questions </li></ul><ul><ul><li>Quality of tools for various group assignment activities? </li></ul></ul><ul><ul><li>Usefulness of various functionality modules? </li></ul></ul>
    10. 10. Activity scores 2002/2003
    11. 11. Functionality scores 2002/2003
    12. 12. Activity Scores BB/CF (2003)
    13. 13. Functionality Scores BB/CF (2003)
    14. 14. Conclusion <ul><li>Practical method, shown to be useful for initial courseware functionality selection </li></ul><ul><li>Context-bound, in toto, ex-post evaluation </li></ul><ul><li>Simple measures useful for quickscan and discussion purposes </li></ul><ul><li>Open source can perform just as well </li></ul><ul><li>“ E-learning organization”: continuous evolution / evaluation needed </li></ul>

    ×