CRIWG2014 - In this paper we explore the application of network analysis techniques in order to analyze synchronous collaborative activities of dyads. The collaborative activi-ties are represented and visualized as networks. We argue that the characteristics and properties of the networks reflect the quality of collaboration and therefore can support the analysis of collaborative activities in an automated way. To sup-port this argument we studied the collaborative practice of 228 dyads based on network graphs. The properties of each graph were evaluated in comparison to ratings of collaboration quality as assessed by human experts. The activities were also examined with respect to the solution quality. The paper presents the method and the findings of the study.

1. Two make a network:
using graphs to assess the quality of
collaboration of dyads
Irene-Angelica Chounta1, Tobias Hecking1, H. Ulrich Hoppe1,
Nikolaos Avouris2
1Collide, University of Duisburg-Essen, Germany
2HCI Group, University of Patras, Greece
{chounta, giemza, hoppe}@collide.info
avouris@upatras.gr

2. Collaborative Activities: Analysis and Evaluation
Analysis & Evaluation:
To map collaborative mechanisms
To improve the learning outcome
To support teachers in class orchestration
Learning Analytics: „the measurement, collection, analysis
and reporting of data about learners and their contexts”
(LAK 2011)

3. Collaborative Activities: Analysis and Evaluation
Methods Tools:
Logfile Analysis
activity metrics retrieved from logfiles
Interaction Analysis
metrics of interaction among users while working
together
Social Network Analysis and Graph Theory
user activity and interaction is represented through
graph representations

4. Objectives of the study
To represent the collaborative activities of
dyads using networks
To use network metrics in order to assess
qualitative aspects of collaboration
Research hypothesis:
The properties and characteristics of
networks that represent collaborative
activities, reflect the quality of collaboration

5. Method of the study
Network generation from logfiles of
previously evaluated collaborative activities
Visual inspection of networks representing
good vs. bad collaboration quality
Study of the relation of human ratings and
network properties in a systematic way
(correlation analysis)

6. Collaborative Activities of Dyads
Team: two students working over a shared-spaces
(Synergo)
Task: the construction of an algorithmic flowchart
Time: synchronous communication for 90 minutes
• Dataset of 228 Collaborative
Sessions
• Pre-evaluated with
qualitative and quantitative
methods with respect to
collaboration quality

7. Related Work
(a) Two human experts evaluated the dataset [1]
Quality of Collaboration (CQA) was evaluated on a
5-point Likert scale [-2, +2]
Well-established results for inter-rater reliability and
consistency
(b) The dataset was used to validate an automatic rater
(time-series classification) [2]
Meaningful Activity takes place in time frames of
15 – 30 seconds
Time series depict the quality of collaboration
[1] Kahrimanis, G., Meier, A., Chounta, I.-A., Voyiatzaki, E., Spada, H., Rummel, N. et al.: Assessing collaboration quality in synchronous CSCL
problem-solving activities: Adaptation and empirical evaluation of a rating scheme (2009)
[2] Chounta, I.-A.,Avouris, N.: Time Series Analysis of Collaborative Activities. CRIWG2012 (2012)

8. Quality of Collaboration
Rating scheme[3] for the assessment of the quality
of collaboration (CQA):
General aspects of
collaboration
Collaborative Dimensions
Communication
Collaboration flow (CF)
Sustaining mutual understanding (SMU)
Joint information processing
Knowledge exchange (KE)
Argumentation (Ar)
Coordination
Structuring the problem solving process
(SPSP)
Interpersonal relationship Cooperative orientation (CO)
CQA = average(CF, SMU, KE, Ar, SPSP, CO)
[3] Kahrimanis, G., Meier, A., Chounta, I.-A., Voyiatzaki, E., Spada, H., Rummel, N. et al.: Assessing collaboration quality in synchronous CSCL
problem-solving activities: Adaptation and empirical evaluation of a rating scheme (2009)

9. Network generation from logfiles
Network maps generated from logfiles of
collaborative activities:
- Nodes represent user actions
- Edges represent dependencies among actions
• To track dependencies:
(a) The time distance between actions ranges from
10 to 30 seconds
(b) Relevance on temporal and spatial proximity
(c) The identity of the actor should differ

10. Network generation from logfiles
Logfile example:
Resulting Network:

11. Visual inspection of networks
The SiSOB Workbench[4] was used for the
visualization and analysis of the network graphs
Good Collaboration Quality Bad Collaboration Quality
[4] Göhnert, T., Harrer, A., Hecking, T., Hoppe, H. U.: A workbench to construct and re-use network analysis workflows: concept, implementation,
and example case. (2013)

12. Network metrics
• The number of nodes (N) and the number of edges (E)
• The diameter of the network (d)
• The average path length (APL)
• The density of the network (D) = (
• The Power Law Fit (PLF)
k
P(k)
Hubs

13. Results
Comparison of Network metrics and Ratings of
human experts for Quality of Collaboration (CQA)
Most metrics correlate (p.05) to CQA and to
individual collaborative dimensions
#Nodes #Edges Density Diameter (PLF) (APL)
Quality of
Collaboration
(CQA)
0.446 0.18 -0.394 0.294 -0.233 0.243

14. Results
#Nodes correlates highly with collaboration quality
Sessions with intense activity point to better
collaboration
...In particular to successful argumentation and
knowledge exchange efficient communication
Good collaboration produces to larger event
networks
CF SMU KE Ar SPSP CO CQA
#Nodes (N) 0.358 0.351 0.4 0.416 0.339 0.41 0.446
#Edges (E) 0.179 0.19 0.169 0.156 0.195 0.18

15. Results
Diameter and Average path length correlate positively
with collaboration quality
Good quality leads to :
- Bigger network maps
- Longer paths, thus long uptake chains
- Faster unfolding networks
CF SMU KE Ar SPSP CO CQA
Diameter (d) 0.267 0.246 0.228 0.273 0.185 0.295 0.294
(APL) 0.226 0.205 0.191 0.223 0.154 0.257 0.243

16. Results
PLF (Power Law Degree Distribution): a smaller
value indicates a better fit
Correlates negatively to the Quality of Collaboration
Activities of good quality appear to lead to Scale-free
networks
Good collaborations contain key events that cause
activity bursts
CF SMU KE Ar SPSP CO CQA
(PLF) -0.229 -0.192 -0.225 -0.208 -0.196 -0.233

17. Results
Density is a negatively correlated to collaboration
quality
Dense Networks point to poor collaboration quality
But why?
Density is anti-proportional to number of nodes in scale-free
networks (
_

18. [5]
Thus, a low density indicates a bigger network ☺
CF SMU KE Ar SPSP CO CQA
Density (D) -0.405 -0.275 -0.34 -0.36 -0.268 -0.323 -0.394
[5] Hoppe, H. U., Engler, J., Weinbrenner, S.: The Impact of Structural Characteristics of Concept Maps on Automatic Quality Measurement.
(2012)

19. Results – Solution Quality
The diagram was graded for the correctness of
the solution from 0 to 10
Detailed solutions (big networks) are evaluated
as good
no correlation with power law fit or network
density… But a correct solution does not
presuppose good collaboration and vice versa
#Nodes
(N)
#Edges
(E)
Diameter
(d)
(APL)
solution
grade 0.319 0.305 0.202 0.189

20. Discussion
Efficient collaboration results in bigger
networks.
Good practices unfold faster in time and form
longer chains of actions.
Dense networks do not portray good
collaboration
Good collaboration results in scale-free
networks: a key action leads to reciprocal
interplay
The size of the network is a good indicator for
the solution

21. Conclusion
Dyads interaction as represented by networks
can indicate the meaningful interplay and
successful collaborative practice
Certain properties of networks reflect the
quality of collaboration
Future work
Content analysis to refine the relations and
connections between user actions
Network metrics as a tool for automatic
assessment of collaborative activities

22. Questions.
Thank you ☺
chounta@collide.info