In this paper, we propose a method for the real time evaluation of collaborative activities. The method aims to classify collaborative activities using a memory-based learning model and to evaluate them using a reference set of pre-evaluated activities. The proposed approach uses time series to represent activities and classifies them regarding their similarity. In order to evaluate the results, the method is used as an automatic rater of collaboration quality and the ratings are compared to ratings of human experts.

1. It's all about time:
towards the real time evaluation of
collaborative activities
Irene-Angelica Chounta, Nikolaos Avouris
{houren, avouris}@upatras.gr
HCI Group, University of Patras

2. Real time evaluation
Why...
●
To support teachers
orchestrate classrooms
(monitoring)
●
To support awareness
and provide feedback
to teams (mirroring)
How...
●
Multiple views of
common
workspaces/dialogues
●
Metrics of interaction,
statistics of participation
& graphs
●
Insight of work progress
via comparison

3. Objective
●
To evaluate the quality of collaboration using
time series classification
– User interaction represented as time series
– Evaluation through classification
– Activities are classified with respect to time series
similarity
●
To study the effectiveness of the method with
respect to the duration of the activity

4. Related work
●
Time series classification was used for the
post-evaluation of collaboration quality
(Chounta, Avouris 2012)
●
Activity metrics that correlate with
collaboration quality used for time series
construction of activity
●
No need for hand-coding or specialized data
manipulation

5. Collaborative setting
●
Dyads of students – 210
sessions
●
Duration of the activity:
90 minutes
●
Task: joint construction
of algorithmic
flowcharts over a
common workspace
●
Synchronous
communication over a
chat tool

6. Real-time evaluation method
●
A memory based learning model (TSCMoCA)
was used as an automatic rater of
collaboration quality:

7. Real-time evaluation method
●
The model TSCMoCA is called on demand by
the teacher
●
An evaluative value for the quality of
collaboration CQA is returned

8. Quality of collaboration
●
The quality of collaboration is defined on six
dimensions:
●
Collaboration flow
●
Sustaining mutual understanding
●
Knowledge exchange
●
Argumentation
●
Structuring the problem solving process
●
Cooperative orientation
●
The collaborative activities were evaluated by
human experts (Kahrimanis, Chounta, et.al
2009)
●
The evaluative value of collaboration quality
ranged between [-2, +2]
Communication
Joint information processing
Coordination
Interpersonal Relationship

9. Method of the study
●
Real time evaluation of collaboration quality
on 3 time instances:
– 25 minutes ( ≈1/4 of total duration)
– 40 minutes ( ≈1/2 of total duration)
– Total Duration (≈90 minutes)
●
Real time evaluation on six collaborative
dimensions, after the first half of the activity
●
The results of the automatic rater are
compared with the ratings of human experts

10. Results (1)
25 minutes 40 minutes Total duration
Correlation
(p<0.05)
0.27 0.51 0.59
Mean
Absolute Error
0.99 0.94 0.91
●
In all cases, the ratings of the model correlate
significantly with the ratings of human
experts
●
Weak correlations for the case of 25 minutes
●
Big improvement for the case of 40 minutes
●
Similar results for the evaluation on total
duration

11. Results (2)
Collaborative Dimensions Correlation MAE
Collaboration Flow 0.53 0.92
Sustaining mutual understanding 0.42 1.02
Knowledge exchange 0.5 1.17
Argumentation 0.44 1.25
Structuring the Problem Solving Process 0.46 1.17
Cooperative orientation 0.47 1.07
●
In all cases, the ratings of the model correlate
significantly with the ratings of human experts
●
The method performs effectively for the dimensions that
reflect communication on a low level
●
Low correlations/high error for complicated constructs
or dimensions that reflect collaboration on the long run

12. Results (2)
Collaborative Dimensions Correlation MAE
Collaboration Flow 0.53 0.92
Sustaining mutual understanding 0.42 1.02
Knowledge exchange 0.5 1.17
Argumentation 0.44 1.25
Structuring the Problem Solving Process 0.46 1.17
Cooperative orientation 0.47 1.07
●
In all cases, the ratings of the model correlate
significantly with the ratings of human experts
●
The method performs effectively for the dimensions that
reflect communication on a low level
●
Low correlations/high error for complicated constructs
or dimensions that reflect collaboration on the long run

13. Results (2)
Collaborative Dimensions Correlation MAE
Collaboration Flow 0.53 0.92
Sustaining mutual understanding 0.42 1.02
Knowledge exchange 0.5 1.17
Argumentation 0.44 1.25
Structuring the Problem Solving Process 0.46 1.17
Cooperative orientation 0.47 1.07
●
In all cases, the ratings of the model correlate
significantly with the ratings of human experts
●
The method performs effectively for the dimensions that
reflect communication on a low level
●
Low correlations/high error for complicated constructs
or dimensions that reflect collaboration on the long run

14. Conclusions
●
Real-time evaluation of collaboration by
classifying activities with respect to their
time-series similarity
●
Adequate performance when the evaluation is
carried out on about half of the duration
●
The method can be used to evaluate low-level
collaborative aspects unlike more advanced
constructs

15. Discussion and future work
●
The proposed method was studied for a
specific collaborative setting
(dyads/synchronous
communication/collaborative diagram
building)
●
Generalization of the findings for a wider
range of collaborative activities
●
Application of the method in a real-classroom
for teacher support

16. Thank you

17. Time Series Construction
Metric of Activity
#messages or #workspace actions
rate of #messages or #workspace actions
#role alternation
Rate of #role alternation
17

18. Memory based learning model TSCMoCA
Time series of Collaborative Activity x
Training set
DTW similarity comparison of time series
Similarity Matrix
Classification of Collaborative Activity x with
respect to KNN algorithm

19. TSCMoCA – Teacher Interface