Hello and good afternoon. My name is Nikos Michailidis, I’m a PhD candidate in the Informatics department of Aristotle University of Thessaloniki in Greece and today I’m going to present you an evaluation study of a CSCL script authoring tool, which is called Collage editor. Co-authors of this work are assistant professor Stavros Demetriadis from AUTH and Professor Yannis Dimitriadis from University of Valladolid.
More specifically, the contents of today’s presentation, are as follows: First of all I’m going to present you the main goals & objectives of the case study together with a short introduction of scripting in the field of CSCL. Then, Collage editor, the script authoring tool under investigation will be presented together with its previous evaluation studies. Following that, the context and the added value of this particular study will be described, and the most important evaluation results will be analyzed and discussed. Finally the main conclusions of the study will be summarized and the future research efforts will be presented.
As I mentioned already, this talk will present an evaluation study of Collage editor. The main objective of the study was to provide additional evaluation data regarding the usability & efficiency of Collage editor as a tool for supporting mainly beginner CSCL script designers (for example teachers). Moreover, the case study attempts to draw conclusions related on the possible ways to enhance and improve Collage, both in terms of technical and pedagogical features.
Ok. So let us first provide a short introduction and recap some basic research conclusions so far regarding the CSCL field. First of all, social interactions are considered as a key element for learning. Moreover, it has been concluded that free collaboration does not necessarily produce fruitful interactions and thus effective CL. For that reason, scripting or learning design has been researched as an effective mechanism, that aims to guide & structure students’ collaborative activity so that learning interactions occur . Having all these in mind, an important question appeared for the CSCL researchers. “ How can teachers create “potentially effective” “computer-interpretable” collaboration scripts according to the necessities of their situation?” The answer to the above question led to the development of several script authoring tools in order to facilitate the design and creation of pedagogically effective CSCL scripts.
A characteristic example of such a script authoring tool is Collage editor, which was developed & proposed by the GSIC research group in the University of Valladolid. Collage is a Graphic-based high-level specialized authoring tool for CL, that help users in the process of creating their own CSCL scripts by starting from existing well-known collaborative patterns, employing representations and abstractions that are easy to understand and use. For further information you can visit the official site of the tool at the GSIC group.
Collaborative Learning Flow Patterns can be understood as a collection of broadly accepted techniques repetitively used by CL practitioners (best practices) when structuring the flow of activities involved in CL. In other words they are a way of communicating CL expertise. Moreover, they can be applied to a wide range of scenarios, since they are neutral to specific domain content. An example is the Jigsaw CLFP, which it structures collaboration for a context in which several small groups are facing the study of a lot of information for the resolution of the same problem
Collage supports designers through a pattern-based script design process of reusing and customizing CLFPs. This process involves the selection of one or more CLFPs, according to certain learning objectives, types of problems or complexity, as well as the configuration of the embedded activities and resources to the characteristics of the scenario. The tool creates a computer-interpretable representation of the script, using the IMS-LD specification, thanks to which scripts generated with Collage can be interpreted with any IMS-LD compliant system. However, the educator performs all those steps by means of a GUI that eliminates the complexity of the XML-based formalism of IMS-LD.
This table illustrates the systematic multicase studies performed so far in order to assess the impact of Colalge editor on the CSCL authoring process. 3 main studies with different objectives revealed important and interesting evaluation results.
The present case study was organized in the context of a postgraduate course in the Informatics Department at Aristotle University of Thessaloniki, Greece. The activity was completed during the spring semester 2009 with 21 students participating, during the course “Technology-enhanced constructivist Learning Environments”. The case study implemented a 2 weeks collaborative learning activity with a synchronous laboratory session in between.
As mentioned already, the main goal of this study is to further explore the efficiency and usability of Collage editor. The study contributes towards a more consistent and thorough evaluation, since firstly it employs non-expert users that are not biased by any previous knowledge neither on the CLFP-based approach nor on the Collage editor itself and secondly most of these users are teachers in “real” life. Furthermore, the study focuses on the CLFP-based design process implied in Collage and particularly on whether the main aspects of students’ collaborative scenarios can be modeled with the editor. The ultimate objective is to identify possible design constraints that limit flexibility and adaptability and thus the case study attempts to draw conclusions related on the possible ways to enhance and improve Collage editor, both in terms of technical and pedagogical features.
The activity pursued the following objectives: Firstly, it was expected that students understand the concept of collaborative scripts through the design of a real collaborative scenario (i.e. Collage users employ the tool as a means for learning about scripts). Secondly, students were offered the opportunity to familiarize themselves with a script authoring tool such as Collage editor, involving them in its use as much as possible. A third objective was to provide further evaluation evidence regarding the strong and weak points of Collage, exploring how it can facilitate beginner practitioners in the implementation of effective CSCL scripts.
Students were invited to form pairs and work collaboratively by outlining and suggesting ideas for the plan and design of their CSCL scripts, first on paper and then authoring them using Collage. As a basic requirement the generated CSCL scripts should be designed in such a way in order to incorporate at least two CLFPs from those available in Collage, and more specifically it was decided to use Jigsaw CLFP in combination with at least 1 more CLFP. Moreover, for the fulfilment of the case study’s objectives, it was decided that the topic for the design of the script should be open and thus, students could freely choose from any subject domain, but they had to have elements of originality & interdisciplinary. Originality was evaluated based on the combination of the different CLFPs used, whereas interdisciplinarity was evaluated through the different subject domains involved in the final scenario.
Special care was given to students’ support and guidance throughout the implementation of the study. Thus, prior to the beginning of the activities, a meeting was held with the participating students, where the activity and its objectives were introduced. Moreover, students were given detailed tutorials and user guides of the tool, which was also presented to them during a live demonstration & training. Finally, students were asked to validate the produced Collage UoLs of their CSCL scripts, by running them on an IMS-LD compatible enactment tool, such as the provided Reload LD Player.
Towards the fulfillment of case study’s main objective, an evaluation process including three data categories was implemented. The first category is related to the evaluation of the case study’s activity itself and on the educational motivation and innovation, if any, that Collage represents with respect to students’ previous experience. The second category focuses on the general use of Collage, as well as on the technical characteristics of the tool (i.e. information on usability issues). This category may show whether Collage is suitable and useful to non-technical practitioners for use in educational settings. The third data category is related to the pedagogical evaluation of Collage and particularly to the implied CLFP-based design process. This category is intended to acquire evidence on whether the script generated with Collage, actually reflects the designer’s intention. Table shows the different data sources used for the evaluation.
The next slides will summarize the main findings, their partial results and supported data, extracted from the case study. First of all, students perceived the experience of the scenario as a positive improvement over their previous courses. More specifically, …… for example…. Quantitative data ……..qualitative comments…..
Another evaluation results was that Collage & the overall design process are useful & suitable for CSCL designers and especially non-experienced users. More specifically, …… for example…. Quantitative data ……..qualitative comments…..
Moreover, CLFPs in Collage facilitate & guide users through the CSCL script design process . More specifically, …… for example…. Quantitative data ……..qualitative comments…..
Finally, it was found that Collage editor could be further improved in terms of technical usability, pedagogical usefulness and flexibility. More specifically, …… for example…. Quantitative data ……..qualitative comments…..
This paper has presented a case study in which Collage editor has been used to produce collaboration scripts following a pattern-based design approach. It was concluded that participating students highly appreciated the collaborative learning activity of the case study and perceived the overall experience as a positive improvement over their previous non-collaborative courses. Moreover, the evaluation so far has provided important evidence that Collage is a useful tool for facilitating and guiding non-experienced designers in the creation of their own CSCL scripts, according to the requirements of particular learning situations while reducing the complexity of manipulating the IMS-LD specification by means of a user-friendly GUI.
Summarizing Collage editor has no major usability or technical drawbacks, although certain improvements could further enhance its potential. Moreover, these findings could be utilized for the future design of other CSCL script authoring tools. Specifically, students suggested several further improvements for Collage editor both in terms of pedagogical usefulness and flexibility : First of all, in order to further enrich and make the CLFPs structure more flexible, it is necessary to implement support for the IMS-LD levels B & C in Collage. That would enable the use of properties and global elements in order to define some flexible elements to be determined at run-time. Moreover, the integration of Collage with enactment & delivery systems or “mini-players” should also provide extra flexibility and minimize implementation errors. An additional possible solution to the script flexibility issue could be the integration of adaptive characteristics to tools for scripted collaboration. The need for including assessment methods in the pattern-based design process of Collage also emerged. Finally, upgrading Collage with new CLFPs and new ways of combining them should further increase its overall functionality.
With the aim of further evaluating and validating the above findings, new case studies should be carried out in different educational and technological contexts. Moreover, new case studies are needed to further evaluate the applicability and suitability of assessment and adaptation patterns in systems for scripted CSCL. Furthermore, it is important to investigate into the possibility of implementing additional and different types of learning design patterns in Collage (apart from the flow-level CLFPs) such as: activity-level, resource-level. Finally, future research may explore the usability of complementary tools like the InstanceCollage authoring tool, for assisting educators in the task of assigning learners to groups at instantiation time.
Nikos Michailidis 1 , Stavros Demetriadis 1 , Yannis Dimitriadis 2 1 Multimedia Lab, Aristotle University of Thessaloniki 2 GSIC-EMIC group, University of Valladolid ICALT 2010 5 - 7 July 2010 Sousse, Tunisia Scripted Collaborative Learning Based on Collaborative Learning Flow Patterns: A Case Study Using Collage Editor 5 – 7 July, Sousse, Tunisia ICALT 2010 / 24
ICALT 2010 (5/7/2010) Michailidis, Demetriadis & Dimitriadis / 24 Learning objectives Type of problems / tasks Complexity (CL experience)
Previous Collage Evaluations ICALT 2010 (5/7/2010) Michailidis, Demetriadis & Dimitriadis / 24 “ Collage Workshops” “ Network Management” “ Planet Game” Description 2 experiences with target audience and 2 mini-cases, using a mixed evaluation method. Authentic collaborative learning scenario at the laboratory of University of Valladolid. A CLFP-based approach to the “Planet Game” scenario, during the ICALT 2006 workshop “Comparing educational modeling language on a case study”. Objectives Creating CSCL scripts based on CLFPs using Collage, focusing on topics such as the pattern-based design process, the use of Collage and the characteristics of the potential audience. Get evidence on whether the CSCL script created with Collage can be successfully used in real situations. Solving a third-party scenario using Collage and identifying the added value and/or weaknesses of this approach. Results The use of CLFPs facilitates the design task and the editor itself provides an intuitive and easy way for non-expert users to design complex collaborative learning scenarios. Need for making the description of CLFPs more flexible. Significant evidence on the suitability of Collage editor for facilitating the introduction of collaborative learning, since the tool can support the creation of meaningful and effective CSCL scripts. Collage succeeded in implementing the scenario and also pointed out some significant advantages in terms of design reusability, generality and user-friendliness.
General use & technical characteristics of Collage
Pedagogical evaluation of Collage & particularly the CLFP-based design process
ICALT 2010 (5/7/2010) Michailidis, Demetriadis & Dimitriadis / 24 Data source Type of data Hard-copy questionnaire Quantitative ratings & qualitative explanations of the students Focus group interviews Qualitative students’ opinions Student deliverables IMS-LD based CSCL produced as results of the activities
76% of the participants stated that had very little experience with CL activities as educators & 85% stated having never used another CSCL script authoring tool [quest].
90.5% of students rated Collage as an easy-to-learn & u ser-friendly tool [quest].
“ Collage is a great tool for facilitating the design of CSCL scripts especially for users without great experience […] the usage of buttons & graphics is quite obvious [focus]”.
Overall evaluation of Collage: 5.24, deviation: 1.73 (max.: 7) [quest].
“ The interface of Collage editor Window was functional, user-friendly and helped me in the development of my CSCL script design by simply editing and customizing all the provided fields in a straightforward manner […] [focus]”.
“ The CLFP Selector helped me to choose the CLFP best suited for the desired educational and collaborative objectives of my scenario […] [focus]”.
Satisfaction with CLFP representation and advice: 5.38, deviation: 1.46 (max.: 7) [quest].
“ Collage helped me to structure the activities flow for my scenario […] the right-click feature for replacing a CLFP phase with another CLFP is simple & functional [focus]”.
8 out of 10 scripts generated were successfully validated with Reload LD Player [script].
Satisfaction with produced scripts: 5.05, deviation: 1.73 (max.: 7) [quest].
45% of students found the structure of available CLFPs too coercive, [quest].
Design process and CLFP structure flexibility: 4.4, deviation: 2.21 (max.:7) [quest].
“ […], although my final Collage UoL reflects my original design intentions, I had to adjust my CL scenario because I couldn’t modify the number of Problem Solvers & Listeners on each phase of the TAPPS CLFP […] similar for the TPS [focus]”.
P. Dillenbourg (2002). “Over-scripting CSCL: The risks of blending collaborative learning with instructional design,”. In Three Worlds of CSCL: Can We Support CSCL?, P. A. K. Heerlen, Ed. Heerlen, The Netherlands: Open University Nederland, 2002, pp. 61–91.
D. Hernández-Leo, E. D. Villasclaras-Fernández, I. M. Jorrín-Abellán, J. I. Asensio-Pérez, Y. Dimitriadis, I. Ruiz-Requies, & B. Rubia-Avi. (2006). “COLLAGE, a collaborative learning design editor based on patterns,” Educ. Technol. Soc., vol. 9, no. 1, pp. 58–71, 2006.
D. Hernández-Leo, D. Burgos, C. Tattersall & R. Koper. (2007a). “Representing Computer-Supported Collaborative Learning macro-scripts using IMS Learning Design”, Proc. of the 2nd European Conference on Technology Enhanced Learning, CEUR Workshop Proceedings, EC-TEL'07, Crete, Greece, September 2007.
D. Hernández-Leo, E.D. Villasclaras-Fernández, J.I. Asensio-Pérez & Y.A. Dimitriadis. (2007b). “Some findings from the evaluation of the Collage authoring tool”, TENCompetence Open Workshop on current research on IMS-LD and Lifelong Competence Development Infrastructures, pp. 27-32, Barcelona, Spain.
N. P. Michailidis & S. Demetriadis. (2009). “Technology tools for scripted collaborative learning: The case of the Reload LD Player”, Proc. 4th Balkan Conference on Informatics (BCI 09), Thessaloniki, Greece, September 17-19 2009, pp 163-168
A. Karakostas & S. Demetriadis. (2009). “Adaptation Patterns in Systems for Scripted Collaborative Learning”, in Proc. of the International Conference on Computer Supported Collaborative Learning 2009, (CSCL 2009), pp. 477-481.
E. D. Villasclaras-Fernández, D. Hernández-Leo, J.I. Asensio-Pérez & Y. Dimitriadis. (2009a). “Incorporating assessment in a pattern-based design process for CSCL scripts”, Computers in Human Behavior, 2009, 25(5), pp. 1028-1039
E.D. Villasclaras-Fernández, J.A. Hernández-Gonzalo, D. Hernández-Leo, J.I. Asensio-Pérez, Y.A. Dimitriadis & A. Martinez-Mones. (2009b) “InstanceCollage: A Tool for the Particularization of Collaborative IMS-LD Scripts”, Educational Technology & Society, 2009, vol.12, no. 3, pp. 56-70.