Analysis of interaction in collaborative activities; the Synergo approach


Published on

Keynote talk at INCOS 2010
Analysis of interaction in collaborative activities: the Synergo trail
It provides background information on Synergo a collaborative learning environment more at

Published in: Technology, Education
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Analysis of interaction in collaborative activities; the Synergo approach

  1. 1. INCoS 2010 – Thessaloniki November 24thAnalysis of interaction in collaborative activities: the Synergo trail Nikolaos Avouris University of Patras, GR Keynote Talk 1/60
  2. 2. outline- on analysis of collaboration- the synergo testbed- synergo studies- models from synergo data 2/60
  3. 3. On analysis ofcollaborative activities 3/60
  4. 4. Typical analysis objectivesmethod focusUsability evaluation Collaborative technologyPre-post testing Learning outcomesQuantitative, qualitative,sequential methods Interaction processInquiry methods Participant’s perceptions 4/60
  5. 5. Focus on the interaction process– Dillenbourg: “the basic instrument for understanding collaborative learning is understanding the interaction that takes place during a learning process”– Koschmann: “CSCL research is not focused on instructional efficacy, but it is studying instruction as enacted practice” 5/60
  6. 6. Quantitative analysis• Frequency counts of events such as: - messages posted per student per period of time - hits on particular discussion forum pages - actions taken on objects of a shared workspace - number of files read in a shared file system etc.• Defining metrics (indicators) that combine different kinds of frequency counts• Suitable for all kinds of collaborative learning• They can lead to models of interaction (e.g. Social Networks etc.) 6/60
  7. 7. Qualitative content analysis• “Content analysis refers to any process that is a systematic replicable technique for compressing many words of text into fewer content categories based on explicit rules of coding” (Kripendorf, 1980)• Suitable for every means of dialogue oriented collaborative learning (synchronous & asynchronous, collocated & distant) 7/60
  8. 8. Content analysis models• Henri’s scheme• Garrison’s model• Gunawardena’s Interaction Analysis Model• Language/action OCAF 8/60
  9. 9. Content analysis resources • The content analysis guidebook http://academic.csuohio.e du/kneuendorf/content/ 9/60
  10. 10. Small group synchronousinteraction: Integration ofdialogue and action• Treats language acts and actions taken to objects in an integrated way• Uniform annotation (eg. the OCAF framework)• Shifts the focus to the objects of a shared workspace• Objects have an ‘owner’ just like language acts• Can visualize uptaking actions (Suthers 05) 10/60
  11. 11. Dialogue: Chat tool affordances• Visual and/or auditory cues are not available• No production blocking->overlapping exchanges• Persistence of messages – substantiation of conversation• Loose inter-turn connectedness - but possibility of simultaneous engagement in multiple threads• Verbal deixis spans throughout the whole history of dialogue (no restricted time window is adequate for analysis)• Posters may reply rapidly, using short messages and split long messages to increase referent/message coherency (Garcia and Jacobs 1999)• Participants begin new topics fairly much at will in a manner that would not happen in a formal face-to-face group discussion (O’Neil & Martin, 2003) 11/60
  12. 12. Action: Shared Activity spaceaffordances• Feedthrough (Dix et. al., 1993)• Various degrees of coupling (Salvador et. al., 1996)• Workspace can be used as an external representation of the task that allows efficient nonverbal communication• Workspace artefacts act as conversational props (Hutchins, 1990) 12/60
  13. 13. Types of communication acts /gestures in shared workspace • Deictic references • Demonstrations • Manifesting actions • Visual evidence (Gutwin, Greenberg, 2002) 13/60
  14. 14. Grounding through actions on aworkspace representation(Suthers, 2006)Sequences of actions :(1) one participant’s action in a medium…(2) is taken up by another participant in a manner that indicates understanding of its meaning, and(3) the first participant signals acceptance 14/60
  15. 15. Merging Action and dialogueAnnotated model=collection ofobjects (OCAF Avouris et al. 2003)MEF = {Entities= E (ABC) = 1/EP, FA , EI E (VELO) = 2/ EP, FA , EI E (TRUCK) = 3/FP, FI E (STOREHOUSE) = 4/FP EC, FA, FI E (STORE) = 5/FP EC, FA, FI Ε(DELIVERY)= 11/ FP, EX, FIRelations= R (VELO-owns-SH) = 9/FPI R (VELO-owns-ST) = 10/FPI R(TRUCK-transports- DELIVERY)=17/ EP, FI, EC A (storehouse) A (store) R(SH-are-suppplied-by-TR) = 18/ FIM A(id) 7/EP, FC R (ABC-owns-TR) = 25/ FPI 6/EP, FC 24/ FI A (truck) R(ST-owns-SH) = 24/ EP FP FI EC, EM 8/FP, EX R (ABC-owns-TR) = 25/ FPI R E(STORE- E(VELO) HOUSE)Attributes= A ( = 13/FIM 2/EP, FA , EI 9/FPI 4/FP EC, FI A (DEL.volume) = 14/FIM A (DEL.Weight) = 15/FI E(ABC) A (DEL.Destination) = 16/FI R R R 1/EP, FA , EI A (TR.Max_Weight ) = 19/FI 18/FIM R 10/FPI 24/EP FPI, EM R A ( ) = 21/EP , FI 25/FPI 12/EP, FR A (TR.Journey_id ) = 23/FI A (TR.volume ) = 20FIM E(DE- E(STORE) E(TRUCK) R LIVERY) A ( ) = 24/FI 5/FP , EC, FAI 17/EP,F I,EC EP, FI 11/FP, EX, FIItems not in the final solution -R (SH-DEL) = 12/EP , FR , A(Max_ A(Journey A(Id) A(volume) -A(VELO.Storehouse)=6/ EP , FC weight) _id) 19/FI 23/FI 13/FIM 14/FIM -A(VELO.Store)= 7/ EP , FC -A(ABC.Truck)= 8/ FP , EX A (max- A(volu journeys/week A(id) A(Weight A(destina -A (TR.max_journeys_per_week) = 22/EP , FR } me) tion) 22/EP, FR 20/FI,M 21/EP, FI 15/FI 16/FI 15/60
  16. 16. Synergo ChatAvouris et al. 2004 16/60
  17. 17. Synergo Partner selection toolDrawingobjectslibraries Chat toolSharedActivitySpace 17/60
  18. 18. Synergo Drawing libraries Concept maps Entity-Relationship Diagrams Flow charts Free Drawing 18/60
  19. 19. Activity logging used for :• Build a history of interaction at server• support latecomers during synchronouscollaboration• analysis and playback of the activity•Support replication/ reduce bandwidthrequirements 19/60
  20. 20. Analysis tools 20 20/60
  21. 21. Log Data PreprocessorAnalysistools 21 21/60
  22. 22. Typed events automaticallyannotate the diagram E i = (t i , Aa , [O o ], [Tt ])i Object A I C M R Actor A Actor B Actor C Types of events I (Insert), M (Modify), D (Delete) C (Contest) 22/60
  23. 23. Playback of annotated view 23/60
  24. 24. What about the chat? Can weannotate chat automatically?One approach is to ask the user to do it - open sentences(e.g. Epsilon (Soller et al. 97) 24/60
  25. 25. Annotation of chat events Deleted objects (b) Model objects Dialogue messages Abstract objects 25/60
  26. 26. Define types of actions (annotation scheme) 26/60
  27. 27. Overview: Visualization oflogged actions 27/60
  28. 28. Teachers view and tool support• E. Voyiatzaki, M. Margaritis, N. Avouris, Collaborative Interaction Analysis: The teachers perspective, Proc.ICALT 2006 - The 6th IEEE International Conference on Advanced Learning Technologies. July 5-7, 2006 – Kerkrade , Netherlands, pp. 345-349. 28/60
  29. 29. Teacher support (supervisor tools) 29/60
  30. 30. Study of the use of tools by teachers Level of Education Computer Engineering University degree program (A’ Semester) Teachers involved 1 Teacher + 5 Teaching Assistants Learners involved 80 students (46 students 2004-2005, 34 students 2005-2006) Collaborative Activity Problem solving activity: Development and Exploration of an Algorithm Students in Dyads , no roles assigned Typical Laboratory lesson (2 didactic hours) Collaborative Tools SYNERGO Collaborative Environment SYNERGO Analysis Tools 30/60
  31. 31. The Teachers Used the proposed views and gave feedback… Quantified Overview: Class and Groupteacher The Process Teachers: “The View (Playback process view is of the the most activity) important tool for in depth insight .” Qualitative viewresearcher Row data 31/60
  32. 32. studiesVrachneika Gymnasio-3rd year UnivPatras Algorithms 32/60
  33. 33. Typical tasks- Collaborative Cognitive Walkthroughof an interactive system- Designing Data bases (ER-D)- Building and exploring Flow Charts 33/60
  34. 34. Joint Univ Patras -UnivDuisburg croos-nationalcollaborative activities(2004-2005)• A. Harrer, G. Kahrimanis, S. Zeini, L. Bollen, N. Avouris, Is there a way to e-Bologna? Cross-National Collaborative Activities in University Courses, Proceedings EC-TEL, Crete, October 1-4, 2006, LNCS vol. 4227/2006, pp. 140-154, Springer Berlin 34/60
  35. 35. Similar models with differenttools (Synergo, Freestyler) 35/60
  36. 36. Findings of the Patras-Duisburg study• Mixture of synchronous and asynchronous approaches.• Only partly use of the provided tools• Engaging activities - examples of sessions of many hours (4-5 h) in joint activity and discussion• Innovative use of media and coordination mechanisms• Good strategies for division of labor• Excellent social dynamics and group spirit. 36/60
  37. 37. A distance learning courseof Hellenic Open University(HOU) (2003-2004)M. Xenos, N. Avouris, D. Stavrinoudis, and M. Margaritis,Introduction of synchronous peer collaboration activities ina distance learning course, IEEE Transactions inEducation, vol. 52 ( 3), Aug. 2009, pp. 305 - 311, 37/60
  38. 38. Mixed media and collaboration approaches Asynchronous group activity Post assignments, Respond to technical and form groups organizational problems – follow activity Tutor ODL Server (forum, exchange of material, ODL repository help desk) Asynchronous interaction Submit final Facilitator Activity Record solution Synergo activity server logging Synchronous Synchronous activity interaction (share Synergo drawing / chat Synergo client communication) client Student #1 Student #2 Arrangements on sessions plan- direct contact Group 38/60
  39. 39. Synergo- Discussion forum 39/60
  40. 40. Findings of the HOU study• Infrastructure overhead higher than expected (unforeseen technical problems)• Peer tutoring patterns emerged in higher degree than younger students• Multiple media engaged• Strong social aspects of community building 40/60
  41. 41. Study on Mecahnics ofCollaboration:Coordination protocol 200 180 160 GROUP A (with key) GROUP Β (without key) Number of events 140 120 100 Group A Explicit floor Group B No floor control: 80 control: Only the key all partners can act in the 60 40 owner can act in the shared shared work space 20 work space 0 Critical Insert Delete Move Chats T+ T- Type of events 41/60
  42. 42. Findings of the study§ Explicit floor control of the shared activity areadid not inhibit problem solving§ Similar patterns of activity in both groups.§ group T- was more active than T+§ T+ students have been obliged to negotiate onpossession of the activity enabling key and thusargue at the meta-cognitive level of the activityand externalise their strategies, a fact that helpedthem deepen their collaboration 42/60
  43. 43. models 43/60
  44. 44. #1 Support for GroupAwareness through a MachineLearning ApproachTrain a classifier to be used for estimation ofthe quality of collaboration using historicaldata of problem solving activities ofstudents engaged in building concept mapsand flow-chart diagrams in Hellenic OpenUniversity and University of Patras M. Margaritis, N. Avouris, G. Kahrimanis, On Supporting Users’ Reflection during Small Groups Synchronous Collaboration, 12th International Workshop on Groupware, CRIWG 2006 Valladolid, Spain, September 17-21, 2006, LNCS 4154 44/60
  45. 45. Logfile segmentationL={S1, S2, … Sk} NE quality of collaboration per segment (bad, average, good) 45/60
  46. 46. Correlation based feature selection(CFS) for different segment sizes NE=60 NE=80 NE=100 NE=200 (2) num_chat (2) num_chat (2) num_chat (2) num_chat (3)symmetry_chat (3)symmetry_chat (4) altern_chat (4) altern_chat (4) altern_chat (4) altern_chat (5) avg_words (5) avg_words (5) avg_words (5) avg_words (6) num_quest (6) num_quest (6) num_quest (7) num_draw (7) num_draw (7) num_draw (7) num_draw Correlation based Feature Selection (CFS)NE= number of technique:events per segment makes use of a heuristic algorithm along with a gain function to validate the effectiveness of feature subsets. 46/60
  47. 47. Performance of classification algorithms• Naïve Bayesian Network 90• Logistic Regression Success rate (%) 85• Bagging NaiveBayes Logistic 80 Bagging• Decision Trees SimpleLogistic RandomForest NNge• Nearest Neighbor 75 60 80 100 Fragmentation factor NE 200 47/60
  48. 48. Visualization of group awareness indicator State ofCollaboration 48/60
  49. 49. Evaluation study• 11 groups of 3 students each were given a collaborative task.• 6 of these groups were provided with the group awareness mechanibsm.• 5 groups did not have that facility• The mean values of collaboration symmetry were significanlty different between the two sets (p=0,0423). 49/60
  50. 50. Side-effect• in four (4) out of the six (6) groups there was an explicit discussion about the group awareness mechanism.• A side-effect: 50/60
  51. 51. #2 Measuring quality ofcollaboration in Synergoactivities using a rating schemeand an automatic rating modelBased on:Meier, A., Spada, H., & Rummel, N. (2007). A ratingscheme for assessing the quality of computer-supportedcollaboration processes. International Journal ofComputer-Supported Collaborative Learning, 2, 63–86. 51/60
  52. 52. Meier et al. (2007) rating scheme Original setting New setting Desktop-videoconferencing Synergo: shared whiteboardCSCL tool system with shared text and chat editorDomain Medical decision making Computer programming (algorithm building)Collaborators Intermediates; Beginners; complementary prior similar prior knowledge knowledge (psychology and medicine) 52/60
  53. 53. Meier et al (2007) rating schemedimensions 53/60
  54. 54. Kahrimanis et al. (2009) adaptedcollaboration rating schemeAspect of DimensionscollaborationCommunication 1. Collaboration Flow 2. Sustaining Mutual UnderstandingJoint information 3. Knowledge Exchangeprocessing 4. ArgumentationCoordination 5 .Structuring the Problem Solving ProcessInterpersonal 6 .Cooperative OrientationRelationshipMotivation 7. Individual Task Orientation (for dyad mean or absolute difference) 54/60
  55. 55. Development of a CollaborationQuality Estimation ModelData set used• 350 students of 1st year working in dyads to solve an algorithmic problem using Synergo (academic year 2007- 2008) duration of activity 45’ to 75’• 260 collaborative sessions• Grading according to the quality of solution and quality of collaboration 55/60
  56. 56. 36 derived metrics used(Kahrimanis et al. 2010) 56/60
  57. 57. Quality of Collaboration Estimator(Kahrimanis et al. 2010)Partial Least Squares Regression Modelcollaboration_quality_avg =0.460 + 0.004*C4 - 0.005*C6 + 0.011*C8_17.5 - 0.012*C7 + 0.602*EV3 + 0.447*STC - 0.001*MW1 + 0.008*MW6 Observed vs. Estimated CQ average VIPs (1 Comp / 95% conf. interval) 3 1.8 Observed (collaboration quality avg) 1.6 1.4 2 1.2 1 1 VIP 0.8 0.6 0 0.4 -2 -1 0 1 2 3 0.2 -1 Stone & Geisser 0 Coefficient -2 Variable (cross validation) Estim ated(collaboration quality avg) 57/60
  58. 58. Use of Quality of Collaboration Estimator as discriminator between cases of good and bad collaboration• The model scored between 76.6% to 79.2%, with the exception of one dimension of lower quality. 58/60
  59. 59. Use of Quality of Collaboration Estimator as automatic rater• The model had acceptable performance as rater as the inter-rater reliability with human raters had the following values: ICC=.54, Cronbach’s α=.70, Spearman’s ρ=.62 acceptable for α και ρ (George, & Mallery, 2003; Garson, 2009), not for ICC (.7) (Wirtz & Caspar, 2002) . This applies both for the average collaboration quality value and the individual dimensions. 59/60
  60. 60. Current developments• Study of tablet-based collaboration patterns (synergo v. 5)• Study of Attention mechanisms (Chounta et al. 2010) 60/60
  61. 61. More on 61/60
  62. 62. Some more key references• Avouris N., Margaritis M., & Komis V. (2004). Modelling interaction during small-group synchronous problem-solving activities: The Synergo approach, 2nd Int. Workshop on Designing Computational Models of Collaborative Learning Interaction, ITS2004, Maceio, Brasil, September 2004.• Κahrimanis, G., Meier, A., Chounta, I.A., Voyiatzaki, E., Spada, H., Rummel, N., & Avouris, N. (2009). Assessing collaboration quality in synchronous CSCL problem-solving activities: Adaptation and empirical evaluation of a rating scheme. Lecture Notes in Computer Science, 5794/2009, 267-272, Berlin: Springer-Verlag.• Kahrimanis G., Chounta I.A., Avouris N., (2010) Determining relations between core dimensions of collaboration quality - A multidimensional scaling approach, In the 2nd International Conference on Intelligent Networking and Collaborative Systems (INCoS 2010) 62/60