2009 IEEE/WIC/ACM International Joint Conferences on Web Intelligence and Intelligent Agent Technologies
         2009 IEE...
•  support the teacher, which is a very valuable yet limited           tokens that can exchange with good answers. Wheneve...
IV. S OCIAL C OLLABORATIVE P ROJECTS                             A WF is a set of projects {Pj }j∈(1,...,nWF )
project Pj ; each grade, as well as the principal one given             3) Self evaluation through open-answers quizzes: W...
Upcoming SlideShare
Loading in …5

Collaborative Projects And Self Evaluation Within A Social Reputation Based Exercise Sharing System


Published on

Collaborative Projects And Self Evaluation Within A Social Reputation Based Exercise Sharing System

Published in: Education
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Collaborative Projects And Self Evaluation Within A Social Reputation Based Exercise Sharing System

  1. 1. 2009 IEEE/WIC/ACM International Joint Conferences on Web Intelligence and Intelligent Agent Technologies 2009 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology Collaborative projects and self evaluation within a social reputation-based exercise-sharing system Andrea Sterbini Marco Temperini Dept. of Computer Science Dept. of Computer and Systems Science ”La Sapienza” University, Rome, Italy ”La Sapienza” University, Rome, Italy sterbini@di.uniroma1.it marte@dis.uniroma1.it Abstract—We present the design issues and motivations of an where they can interact, exchange information and collaborate enhanced version of the web-based system S OCIAL X, supporting over common problems (e.g. mandatory exercises in a subject collaborative and social aspects of learning. This web application matter). allows to share solutions to exercises and development of project- (possibly group-) work, through the management of a reputation In this paper we deal mainly with the designing aspects of a system. With the aim of enhancing collaboration and to help web application, called S OCIAL X, taking care of collaborative students working on exercises, we introduce contextual FAQs and social aspects of learning in the aforementioned system and micro-forums and a currency-based concretization of the and allowing the management of an augmented learning perceived usefulness of other’s answers. The tokens exchanged model, based on the analysis of learner behavior in a specifi- are used also to help the teacher/tutor in choosing the best question/answer pairs to be promoted to the FAQ. To introduce cally devised reputation system environment. group responsibilities, peer-pressure and self-evaluation we de- II. T HE OLD S OCIAL X fine group-based projects with self/peer-evaluated phases. The different phases of a project are given to different groups, so Our previous system [5], paired a reputation system together that the produced deliverables are both self-evaluated when they with an exercise-sharing web tool. It aimed at: are submitted and peer-evaluated by the group working on the • increasing the motivation of students in doing home- next phase. The system is its last stages of development and will be tested with real students in the next academic year. works, • increasing/encouraging higher cognitive learning activi- ties (as in the Bloom cognitive taxonomy [1]), both by I. I NTRODUCTION rewarding the student grading other’s solutions, and by Cooperative learning is an indispensable element (in the rewarding the reuse and correction of other’s solutions e-learning field) to help learners sharing and combining ex- To obtain this we built a reputation-based system, within pertise, with the goal to prepare to join team-based working which a student was able to work on homeworks, share environments [2]. Whereas cooperative learning is usually his/hers solutions and judge and reuse other’s solutions. A rep- discussed and applied on small groups, a further aspect of utation system is normally used to motivate interaction and to interest is then in the vision of e-learning as a community elicit good behaviors by awarding points to the user’s actions and social activity [7]. Cooperative learning can improve that are deemed more useful to the community. In our case the teaching and learning considerably; moreover its implications student’s reputation is a blend of five facets that describe how and effects in the extension of present e-learning standard well s/he is working within the class: involvement, usefulness, (namely the IMS Learning Design [4]) have just started to competence, judgment, and critical thinking (see later for be considered [8]. Fundamental didactic tools in collaborative details). Reports of the student’s reputations can be shown learning are reputation systems, that capture (and make evident both at the course and topic level, with details displaying all to the learner) the contributions s/he is giving to the group, to the facets to allow the student to improve his/her reputation the class and to the course. A reputation system is both a mo- by focusing on the type of social activity s/he likes more. tivational tool and a way to evaluate and understand learner’s psychological preferences, relations with others, ability to III. T HE NEW S OCIAL X analyze/judge others’ work, and thus conceptual competences. SocialX is being extended with the added goals of: We are also working on a comprehensive approach to the • increasing collaboration and peer-based help by intro- management of personalized courses [3], [6], exploiting social ducing contextual micro-forums within which ”direct aspects of learning to enrich the definition of learner’s model. rewards” (tokens) are used to explicitly capture the per- In particular, opposite to the usual approaches to collaboration ceived usefulness of other’s help, based on small groups, we are defining a model including • introducing peer-pressure and responsibilities towards the the idea that learners are participating to a social network, group by managing group-based projects. 978-0-7695-3801-3/09 $26.00 © 2009 IEEE 243 239 DOI 10.1109/WI-IAT.2009.273
  2. 2. • support the teacher, which is a very valuable yet limited tokens that can exchange with good answers. Whenever s/he resource, by exploiting to our best all the recorded social needs an information s/he can pose a question (consuming interactions among the intervening students. one token) and reward the best answer received. Tokens are a Moreover, we introduce a new facet in the reputation system limited resource, and thus a student needing answers should to explicitly capture the self judgment ability of the student. first ”work” for the community to collect the tokens needed Definition 1: The reputation of a learner is an overall to ask more questions. The total number of tokens received representation of certain learner’s qualities as they come out is a direct indication of the usefulness of the student in the from his/her interaction with the S OCIAL X system. It can be community, and thus it contributes to the usefulness and to calculated at different levels of detail in the system: course the competence factors of his/her reputation. The number of topic, whole course, and whole system (encompassing several tokens spent, instead, counts how many times the student has courses). There are six basic aspects that are taken care of in asked questions to the community, and thus it contributes to the system: the involvement factor of his/her reputation. • involvement: the degree of active participation in the To avoid students cheating the system (e.g. by exchanging system, measurable by the amount of work that the useless questions/answers) we mildly discourage ”off-topic” learner has been available to submit, also in terms of and ”dummy” discussions. The teacher/tutor flags this kind of participation (such ad the number of solutions submitted, useless exchanges so that they contribute zero to the reputation questions proposed and grades given, as well as the and the token spent to create the question is lost (for both propriety and extension of judgments); the students involved). Discouraged exchanges may affect the • usefulness: how the learner’s work is beneficial to others reputation of both parties involved. in the system (such as the reuse of learner’s solutions, Therefore the participation of the students at the contextual and the appreciation of her/his questions); micro-forums produces reputation through the rules: • competence: an appraisal of the skills shown by the 1) the answers given to others (even if not awarded with the learner (deriving from the grades and judgments coming token) contribute to the student’s involvement factor, from peer students and from the teacher; 2) the tokens received show how much a student has been • judgment: how well the student has evaluated other’s useful to the others, increasing her/his usefulness factor, solutions, questions, answers and products (with respect 3) the tokens spent to propose questions show how much to the teacher’s grades and evaluations) the student has participated, and contribute to his/her • self-judgment: how well the student has evaluated her/his involvement factor, own answers and products (with respect to the teacher’s 4) a Q/A promoted to FAQ shows that that contribution is grades and evaluations) important, thus contributing to the answering student’s • active critical thinking: a measure of the conceptual competence factor, work issued to understand and critically appraise others’ 5) ”dummy” and ”off-topic” discussions are completely work, in order to modify, reuse, and start from such ignored and loose the corresponding token work (such as when a solution is the first produced for a To make the best use of the teacher’s time we highlight the problem, or is the correction of another) token exchanges to help him/her to evaluate faster the dummy A. Increasing collaboration and FAQ candidates. We introduce both contextual micro-forums attached to each C. The teacher is the bottleneck exercise, so that students help each other by asking/answering questions, and FAQs to collect the most interesting discus- The teacher’s work in the system is a crucial, limited sions. The students exchanges are moderated by the teacher, resource. S/he should correct solutions, moderate answers, that can ”promote” the discussion threads by refactoring the promote good Q/As to the FAQs, manage the group projects. most interesting pairs of questions/answers to the exercise’s We must make the best use of the teacher’s expertise, even if FAQ. When a discussion is refactored as a new FAQ entry, the s/he would be able to check/test/correct just a small part of students involved in the originating discussion are rewarded by the solutions submitted. To this aim we exploit the network of increasing their usefulness and competence reputation levels. social exchanges between the students to guide the teacher by To make the best use of the teacher’s time, we highlight the selecting the most interesting items to be evaluated. Then, the exchanges that the students have already selected as the most social network is used to propagate the evaluation results to the interesting/appropriate. neighbor items to adjust the authors’ reputations accordingly. E.g., the tokens exchanged in the contextual micro-forums B. Perceived usefulness are used (also) to highlight the most useful answers, thus To enhance the motivation of the students in helping each- reducing the number of Q/As to peruse while looking for other, and to make explicit the perceived usefulness of others good candidates for FAQ promotion. In the exercise evalu- we apply the classical currency-based approach (tokens) that ation the teacher is guided by the judgments expressed, their students can use to acquire services by other students or by agreement/disagreement, and the reputation of the intervening the teacher. Each student is awarded an initial number of students. 244 240
  3. 3. IV. S OCIAL C OLLABORATIVE P ROJECTS A WF is a set of projects {Pj }j∈(1,...,nWF ) A SCP-path in a WF is a set of tasks As seen, S OCIAL X allows the use of a reputation system in an e-learning environment, supporting the development of {ti,j }i∈(1,...,nT ),j∈(1,...,nWF ) collaborative-social exercising activities within a potentially where ti,j is the i-th task in the j-th project of the work-field. large group of students. Such “exercising activities” so have been made of single exercises, freely reusable by each learner. In a work-field, the projects are supposed to share a common So, in the context of S OCIAL X the learning activity is a trade- structure, meaning that the number of steps and their logical off between individual work (selection and comprehension sequence are homogeneous, so that it is acceptable that a SCP- of others’ work, reuse and adaptation, development of new path provides group learners with a reasonably standard and solutions) and social exchange. complete project activity in the course topic. Once a suitable However in certain courses the development of projects is a work-field is defined, SCP-paths can be assigned to groups. relevant part of learning, in both cases of an activity performed The following is an example of path assigned to a group g: by the individual learner or a collaborative work carried on by a small group of learners. So we extended S OCIAL X to {t1,k1 t2,k2 · · · tnT ,kn } g g g where T g embed also the support to a partially social approach to the ∀h ∈ (1 . . . nT ) kh ∈ (1, . . . , nWF ) development of projects. The approach is called “partially” (the path is made by nT tasks (to make the overall activity social collaborative because, while a small group (possibly complete according to the course topic definition); each i-th singleton) of learners is still the basic operating unit, the task is the i-th task in one of the projects of the work-field). products of such units are submitted to social exchange with If we can assume that each task in a project depends on the the other units (to be reused and assessed). A project is usually previous and is depended upon by the following, we can also a prolonged and organized activity, made of a sequence of assume that for almost each task undertaken by a group in tasks, each one depending on previous one and depended its SCP-path, the group is going to depend on the work done upon by the following ones. Usually a project is entrusted by other groups and will produce material for other groups to to a small group of learners, and collaborative work among use. This gives the social dimension to the activities in a SCP- them is instructed and supported, to produce the deliverable work-field, and gives also the opportunity to add feedbacks for the whole project. We add to S OCIAL X the support to over the reputation of learners, beyond the evaluation of their a partially social collaborative approach to the development technical skills related to project deliverables. (In considering of projects. Instead of having a small group working on the the dependences of a tasks from others, we limit the scope various steps of a single project, the idea is to have the to those immediately preceding and succeeding, in order to group working on different steps of different projects: all simplify a bit the notation, with no prejudice for the general the projects share a similar structure, made of a sequence of discussion). tasks (the steps); the n-th task of a project is expected to be “similar” to the n-th task of another (wrt the general learning Definition 3: (fulfillment of a task by a group) goals related to the project development methodology); so the Given a task ti,j assigned to group g = {lp }p∈Ig , and group would be assigned a path of tasks, each one possibly assuming that the previous and successive tasks in the same involving a step in a different project; at each step the group project Pj , ti−1,j and ti+1,j , are resp. assigned to groups g should deliver a product; moreover, the learners in the group and g, g fulfills ti,j when it provides the system with provide evaluations of the product(s) received from earlier step(s) in the same project (from which the group should • A product p(g, ti,j ), start to work on its task) and of the deliverable released by • A set of evaluations { VAL (lp , g, ti−1,j )}p∈Ig over the the group (to show self-evaluation skills). We define a social product received from the previous task in the project collaborative project (SCP), in a given course topic T , as a (one explicit evaluation for each member of the group), set of tasks P T = {ti }i∈(1,...,nT ) . Each task is assigned to • A set of self-evaluations { AVAL (lp , g, ti,j )}p∈Ig over a group of learners (g T = {li }i∈(1,...,ngT ) ), that will do the the product released by the group itself (one explicit corresponding learning activity (such as the construction of evaluation for each member of the group), a deliverable product). Moreover, the sequence of tasks in When the group g = {lq }q∈Ig has fulfilled its task a SCP provides a complete span of learning activities about ti+1,j another set of evaluations {VAL(lq , g, ti,j )}q∈Ig ) will the related project methodology. (Henceforth, where possible be available over the work of group g. we’ll assume that projects are all on the same T and avoid the related indexes.) So, from the work of a group of learners g in a SCP-work- In the following definition, an SCP-path is a sequence of field, and from Def.3, many items may produce a feedback tasks, selected from different projects in such a way to provide over the reputation of the group members: the aforementioned complete span of learning activities. 1) for each task ti,j of the SCP-path assigned to g we have a set of evaluations of the product p(g, ti,j ), issued by Definition 2: (work-field - WF - and SCP -path) the members of group g that followed g in the same 245 241
  4. 4. project Pj ; each grade, as well as the principal one given 3) Self evaluation through open-answers quizzes: We want by teachers, is imparted to the whole group and can to introduce open-form quizzes with a very simple mechanism easily be spread, mediated by the teachers’ judgment, to that allow the student both to engage in self-evaluation and feedback over usefulness and competence of each lp ∈ g. to do high-level cognitive work (respect to the Bloom’s 2) for each task, ti,j , we also have the evaluations issued hierarchy). The student is proposed a question, which s/he by group members about the product p(g, ti−1,j ) inher- answers. Then s/he is proposed a selection of peer’s answers ited from the previous task in the same project: those to the same question (including his own) from which s/he are single learner’s evaluations, that can be compared could choose the best answer. In doing this, the student is with teachers’, affecting both learner’s competence and analyzing his and the other’s answers, comparing them to judgment. each other. The above evaluations could be wrong, because 3) the various evaluations mentioned at point 1) are also to it’s affected by the student competence on the topic. The be taken into account to measure the ability of group preferences expressed in the system are then used to analyze g to build a good product, basing on the one they such level of competence. Answers that collect high number received from previous task: the relationship between the of choices are probably more correct, and contribute to the grades of the former, p(g, ti,j ), and those of the latter, competence part of the author’s reputation. As the preference p(g, ti−1,j ) can provide a feedback over the active criti- relation expressed should be transitive, if all preferences are cal thinking component in the reputation of the members correct they should build a poset or a total order. If a student of group g. Of course, as it is apparent that the evalua- has a limited knowledge of a topic and makes a wrong choice tions over the previous product, {VAL(lp , g, ti−1,j )}p∈Ig then s/he could introduce a preference going in reverse order are coming from g’s members, for it only the teachers’ respect to the ”correct” order, which could introduce cycles. grades will be taken into account. Thus, cycles in the preference graph highlight the presence of 4) finally, for each ti,j , task assigned to g, we also have a misunderstanding and could be used to select which answers the evaluations issued by group members about their should be examined first to find the mistake (and to correct own product p(g, ti,j ): those are single learner’s self- the corresponding student’s competency level on that topic). evaluations, that can be compared with teachers’ evalu- Moreover, while the teacher evaluates part of the answers, ations, affecting learner’s self-judgment. the graph could be used to propagate the marks given to student to other (yet not examined) answers and to assess the V. F UTURE W ORK competency levels of others. At a given moment, depending on the preferences expressed so far on the presented answers, We have presented the new SocialX system, which in- the answers can be ranked as: 1) best answers, which have troduces collaborative group projects, and contextual micro- been chosen by many; 2) worst answers, which have been forums with rewards for best answers within its reputation proposed but never chosen; 3) unseen answers, which haven’t system. The system is its last stages of development and will been proposed yet. Therefore, the choice of answers to propose be tested with real students during the next academic year. In to the student is better done by choosing an appropriate mix a near future, we intend to continue the SocialX expansion in of the above three types: 1) some best answers: to allow the several directions: switch to a better answer; 2) some worst answers: to show 1) The teacher as a quality rater: Our initial approach good distractors; 3) some unseen answers: to evaluate all the uses the token exchanges as a simple indicator of hot topics, answers. while the student’s judgments are used to pinpoint the most important solutions to mark. R EFERENCES Our final aim is to transform the teacher into the ”quality [1] B.S. Bloom (Ed). Taxonomy of Educational Objectives. David McKay Company Inc, New York (1964). assessor” of the system, by efficiently highlighting the most [2] Y. Cheng, H. Ku. An investigation of the effects of reciprocal peer important didactic decisions and by leveraging the student’s tutoring. Comp.in H.Behav. 25 (2009). social network structure with its reputation levels, instead than [3] G. Fernandez, A. Sterbini, M. Temperini. On the Specification of Learning Objectives for Course Configuration. Proc. Int. Conf. on keeping him engaged in tedious repetitive tasks. Web-Based Education (WBE), (2007) 2) Student’s Fairness: As we have seen with the discour- [4] IMS Learning Design Best Practice and Implementation Guide; aged exchanges, to keep a high level of quality we discourage IMS Learning Design Information Model; IMS Learning Design XML Binding. http://www.imsglobal.org/learningdesign/index.cfm. misbehaviors. The penalization should be done very mildly to [5] A. Sterbini, M. Temperini. Learning from peers: motivating stu- avoid discouraging also normal participation, thus we currently dents through reputation systems. Int. Symp. on Applications and just make all misbehaviors void. We would like to introduce the Internet, Social and Personal Computing for Web-Supported Learning Communities (SPeL). Turku, Finland, (2008). a fairness factor to capture how much the student agrees [6] A. Sterbini, M. Temperini. Adaptive Construction and Delivery with the ”didactic pact”, i.e. with the proper behavior rules of Web-Based Learning Paths. accepted for publication in Proc. in the course. This factor is probably meaningful only for the Frontiers in Education (FIE). San Antonio, Texas, (2009). [7] E.Wenger. Communities of practice: Learning, meaning, and iden- teacher/tutor, and is updated whenever the student misbehaves tity. Cambridge Un. Press (1998). within the system, either by annoying others or by trying to [8] Yu D., Chen X. Supporting Collaborative Learning Activities with cheat the system. IMS LD. Proc. ICACT2007 (2007). 246 242