It's all about time: towards the real time evaluation of collaborative activities

513 views

Published on

In this paper, we propose a method for the real time evaluation of collaborative activities. The method aims to classify collaborative activities using a memory-based learning model and to evaluate them using a reference set of pre-evaluated activities. The proposed approach uses time series to represent activities and classifies them regarding their similarity. In order to evaluate the results, the method is used as an automatic rater of collaboration quality and the ratings are compared to ratings of human experts.

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
513
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

It's all about time: towards the real time evaluation of collaborative activities

  1. 1. It's all about time: towards the real time evaluation of collaborative activities Irene-Angelica Chounta, Nikolaos Avouris {houren, avouris}@upatras.gr HCI Group, University of Patras
  2. 2. Real time evaluation Why... ● To support teachers orchestrate classrooms (monitoring) ● To support awareness and provide feedback to teams (mirroring) How... ● Multiple views of common workspaces/dialogues ● Metrics of interaction, statistics of participation & graphs ● Insight of work progress via comparison
  3. 3. Objective ● To evaluate the quality of collaboration using time series classification – User interaction represented as time series – Evaluation through classification – Activities are classified with respect to time series similarity ● To study the effectiveness of the method with respect to the duration of the activity
  4. 4. Related work ● Time series classification was used for the post-evaluation of collaboration quality (Chounta, Avouris 2012) ● Activity metrics that correlate with collaboration quality used for time series construction of activity ● No need for hand-coding or specialized data manipulation
  5. 5. Collaborative setting ● Dyads of students – 210 sessions ● Duration of the activity: 90 minutes ● Task: joint construction of algorithmic flowcharts over a common workspace ● Synchronous communication over a chat tool
  6. 6. Real-time evaluation method ● A memory based learning model (TSCMoCA) was used as an automatic rater of collaboration quality:
  7. 7. Real-time evaluation method ● The model TSCMoCA is called on demand by the teacher ● An evaluative value for the quality of collaboration CQA is returned
  8. 8. Quality of collaboration ● The quality of collaboration is defined on six dimensions: ● Collaboration flow ● Sustaining mutual understanding ● Knowledge exchange ● Argumentation ● Structuring the problem solving process ● Cooperative orientation ● The collaborative activities were evaluated by human experts (Kahrimanis, Chounta, et.al 2009) ● The evaluative value of collaboration quality ranged between [-2, +2] Communication Joint information processing Coordination Interpersonal Relationship
  9. 9. Method of the study ● Real time evaluation of collaboration quality on 3 time instances: – 25 minutes ( ≈1/4 of total duration) – 40 minutes ( ≈1/2 of total duration) – Total Duration (≈90 minutes) ● Real time evaluation on six collaborative dimensions, after the first half of the activity ● The results of the automatic rater are compared with the ratings of human experts
  10. 10. Results (1) 25 minutes 40 minutes Total duration Correlation (p<0.05) 0.27 0.51 0.59 Mean Absolute Error 0.99 0.94 0.91 ● In all cases, the ratings of the model correlate significantly with the ratings of human experts ● Weak correlations for the case of 25 minutes ● Big improvement for the case of 40 minutes ● Similar results for the evaluation on total duration
  11. 11. Results (2) Collaborative Dimensions Correlation MAE Collaboration Flow 0.53 0.92 Sustaining mutual understanding 0.42 1.02 Knowledge exchange 0.5 1.17 Argumentation 0.44 1.25 Structuring the Problem Solving Process 0.46 1.17 Cooperative orientation 0.47 1.07 ● In all cases, the ratings of the model correlate significantly with the ratings of human experts ● The method performs effectively for the dimensions that reflect communication on a low level ● Low correlations/high error for complicated constructs or dimensions that reflect collaboration on the long run
  12. 12. Results (2) Collaborative Dimensions Correlation MAE Collaboration Flow 0.53 0.92 Sustaining mutual understanding 0.42 1.02 Knowledge exchange 0.5 1.17 Argumentation 0.44 1.25 Structuring the Problem Solving Process 0.46 1.17 Cooperative orientation 0.47 1.07 ● In all cases, the ratings of the model correlate significantly with the ratings of human experts ● The method performs effectively for the dimensions that reflect communication on a low level ● Low correlations/high error for complicated constructs or dimensions that reflect collaboration on the long run
  13. 13. Results (2) Collaborative Dimensions Correlation MAE Collaboration Flow 0.53 0.92 Sustaining mutual understanding 0.42 1.02 Knowledge exchange 0.5 1.17 Argumentation 0.44 1.25 Structuring the Problem Solving Process 0.46 1.17 Cooperative orientation 0.47 1.07 ● In all cases, the ratings of the model correlate significantly with the ratings of human experts ● The method performs effectively for the dimensions that reflect communication on a low level ● Low correlations/high error for complicated constructs or dimensions that reflect collaboration on the long run
  14. 14. Conclusions ● Real-time evaluation of collaboration by classifying activities with respect to their time-series similarity ● Adequate performance when the evaluation is carried out on about half of the duration ● The method can be used to evaluate low-level collaborative aspects unlike more advanced constructs
  15. 15. Discussion and future work ● The proposed method was studied for a specific collaborative setting (dyads/synchronous communication/collaborative diagram building) ● Generalization of the findings for a wider range of collaborative activities ● Application of the method in a real-classroom for teacher support
  16. 16. Thank you
  17. 17. Time Series Construction Metric of Activity #messages or #workspace actions rate of #messages or #workspace actions #role alternation Rate of #role alternation 17
  18. 18. Memory based learning model TSCMoCA Time series of Collaborative Activity x Training set DTW similarity comparison of time series Similarity Matrix Classification of Collaborative Activity x with respect to KNN algorithm
  19. 19. TSCMoCA – Teacher Interface

×