Automatic Assessment of Collaborative Chat Conversations with PolyCAFe  Traian Rebedea 1 , Mihai Dascalu 1 , Stefan Trausa...
Overview <ul><li>Context </li></ul><ul><li>Polyphonic Framework </li></ul><ul><li>System Architecture </li></ul><ul><li>Ut...
Context <ul><li>CSCL </li></ul><ul><li>Online textual interactions using web communication technologies </li></ul><ul><li>...
Context – Analysis Tools <ul><li>The need for automatic   Feedback </li></ul><ul><li>  Assessment </li></ul><ul><li>There ...
Problems <ul><li>P1.  Need to  integrate feedback   on different levels: conversation, utterance, participant </li></ul><u...
Polyphonic Framework <ul><li>Classical NLP discourse Monologs </li></ul><ul><li>Dialogues </li></ul><ul><li>Coherence is e...
Polyphonic Principles <ul><li>Floor may be shared   by different participants at the same time </li></ul><ul><li>Allows th...
Words, Voices and Discussion Threads <ul><li>The topic of the discussion defines a list of important concepts (words/stems...
Implicit Links  <ul><li>In order for the analysis to be effective, it is important to discover implicit links between utte...
Conversation Graph <ul><li>The conversation can be modeled as a graph: </li></ul><ul><ul><li>Utterances – nodes </li></ul>...
Proposed Solution - PolyCAFe <ul><li>Chat and Forum Analysis and Feedback System </li></ul><ul><li>PolyCAFe =  Poly phony-...
Technical architecture
Utterance evaluation
Utterance Evaluation & Threads Utterance Past Future Thread Coherence Future Impact Content In-Degree Out-Degree Social Cu...
Conversation Visualization
Utterance Feedback
Validation Experiment <ul><li>35 students part of the HCI course </li></ul><ul><ul><li>Experimental group: 25 </li></ul></...
Assignment Example <ul><li>A debate about the best collaboration tool for the web: chat, blog, wiki, forums and Google Wav...
Tutor Efficiency <ul><li>VT1:  Tutors/facilitators  spend less time preparing feedback for learners compared with traditio...
Quality and Consistency of Feedback <ul><li>VT2:  Learners  perceive that the feedback received from the system contribute...
Quality and Consistency of Feedback Validation statement Mean Standard deviation % Agree / Strongly agree The information ...
Quality of Educational Output <ul><li>VT3:  Learner  performance in online discussions is improved in the areas of content...
Quality of Educational Output Experimental group Control group Improvement over control group Average score for a chat con...
Verification Experiments <ul><li>Utterance scoring </li></ul><ul><li>Participant ranking </li></ul><ul><li>Speech acts cla...
Utterance Scoring <ul><li>Chat 1 (331 utterances): </li></ul><ul><li>Scores: 1 (not important) – 4  (very important) </li>...
Participant Ranking Rank  Student 1 Student 2 Student 3 Student 4 Student 5 Student 1 - 2 3 1 4 Student 2 2 - 3 1 4 Studen...
Participant Ranking Rankings compared  Correlation Precision Average distance Tutors – System  94% 77% 0.23 Students – Sys...
Transferability Issues <ul><li>Domain </li></ul><ul><ul><li>The topic of the conversation should be easily solved using di...
Conclusions <ul><li>Learners use web communication technologies </li></ul><ul><li>Need tools to harvest this data </li></u...
Test it! <ul><li>http://ltfll-lin.code.ro/ltfll/wp5/ </li></ul><ul><li>Follow link to test PolyCAFe </li></ul><ul><li>Scre...
THANK YOU! <ul><li>Questions  </li></ul><ul><li>  & </li></ul><ul><li>Feedback </li></ul><ul><li>Special thanks to FP7 REG...
Upcoming SlideShare
Loading in …5
×

Automatic assessment of collaborative chat conversations with PolyCAFe - EC-TEL2011

1,059 views

Published on

Published in: Education, Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,059
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Automatic assessment of collaborative chat conversations with PolyCAFe - EC-TEL2011

  1. 1. Automatic Assessment of Collaborative Chat Conversations with PolyCAFe Traian Rebedea 1 , Mihai Dascalu 1 , Stefan Trausan-Matu 1 , Gillian Armitt 2 & Costin Chiru 1 1 - Politehnica University of Bucharest 2 - University of Manchester
  2. 2. Overview <ul><li>Context </li></ul><ul><li>Polyphonic Framework </li></ul><ul><li>System Architecture </li></ul><ul><li>Utterance Assessment </li></ul><ul><li>Validation Experiment </li></ul><ul><li>Verification Results </li></ul><ul><li>Transferability & Conclusions </li></ul>
  3. 3. Context <ul><li>CSCL </li></ul><ul><li>Online textual interactions using web communication technologies </li></ul><ul><li>Education through dialogue </li></ul><ul><li>Focus on chat conversations in small groups </li></ul><ul><li>There is usually no feedback from tutors (difficult, time-consuming, etc.) </li></ul>
  4. 4. Context – Analysis Tools <ul><li>The need for automatic Feedback </li></ul><ul><li> Assessment </li></ul><ul><li>There is a wide variety of research for analyzing chat conversations: </li></ul><ul><ul><li>topic detection and extraction </li></ul></ul><ul><ul><li>concept formation in a group discussion </li></ul></ul><ul><ul><li>summarization </li></ul></ul><ul><ul><li>argumentation and transactivity acts in each posting </li></ul></ul><ul><ul><li>utterance classification based on concept coverage </li></ul></ul><ul><li>However, there is no complete analysis tool for online interactions (like chat, forums) of students </li></ul>
  5. 5. Problems <ul><li>P1. Need to integrate feedback on different levels: conversation, utterance, participant </li></ul><ul><li>S1. Mixture of techniques: </li></ul><ul><ul><li>Natural Language Processing </li></ul></ul><ul><ul><li>Social Network Analysis </li></ul></ul><ul><ul><li>Information Retrieval </li></ul></ul><ul><li>P2. Supplementary, focus on measuring collaborative discourse as a key characteristic of successful online textual interaction </li></ul><ul><li>S2. New theory of discourse for chat conversations with multiple participants </li></ul>
  6. 6. Polyphonic Framework <ul><li>Classical NLP discourse Monologs </li></ul><ul><li>Dialogues </li></ul><ul><li>Coherence is essential in the evolution of discourse (segmentation, topic change, etc.) </li></ul><ul><li>Dialogues are also based on a two interlocutors model </li></ul><ul><ul><li>Speech acts or dialogue acts </li></ul></ul><ul><ul><li>Adjacency pairs </li></ul></ul><ul><ul><li>Transacts </li></ul></ul><ul><li>However, chat conversations with multiple participants are different ! </li></ul>
  7. 7. Polyphonic Principles <ul><li>Floor may be shared by different participants at the same time </li></ul><ul><li>Allows the evolution of parallel “discussions” </li></ul><ul><li>Looking at the “serial” resulting text by concatenating the utterances by post time => the discourse may look incoherent </li></ul><ul><li>Start from Bakhtin’s dialogic theory for discourse analysis </li></ul><ul><ul><li>Discussion threads <=> voices that are more or less powerful than others throughout the conversation </li></ul></ul><ul><ul><li>Influence between different voices </li></ul></ul>
  8. 8. Words, Voices and Discussion Threads <ul><li>The topic of the discussion defines a list of important concepts (words/stems/…) => initial voices </li></ul><ul><li>Other voices appear naturally during the conversation </li></ul><ul><ul><li>Some may be related to the initial ones </li></ul></ul><ul><li>Voices can influence each other through explicit or implicit links between utterances </li></ul><ul><li>One or more similar voices can define a discussion thread </li></ul><ul><li>Discussion threads = a set of utterances that are linked implicitly or explicitly and correspond to a specific voice or more similar voices </li></ul>
  9. 9. Implicit Links <ul><li>In order for the analysis to be effective, it is important to discover implicit links between utterances </li></ul><ul><li>Very difficult task: trade-off precision-recall </li></ul><ul><li>Repetitions (expanded using ontologies) </li></ul><ul><li>Semantic similarity (including lexical chains) </li></ul><ul><li>Adjacency pairs (using speech acts) </li></ul><ul><li>Cue phrases ( “that’s a good idea”, “as <x> said”, …) </li></ul>
  10. 10. Conversation Graph <ul><li>The conversation can be modeled as a graph: </li></ul><ul><ul><li>Utterances – nodes </li></ul></ul><ul><ul><li>Links – edges </li></ul></ul><ul><li>Each edge has a trust factor </li></ul><ul><li>The conversation graph is fundamental to the analysis: </li></ul><ul><ul><li>Can be used as a social network </li></ul></ul><ul><ul><li>Can be used to compute conversation threads, coherence, collaborative discourse, etc. </li></ul></ul>
  11. 11. Proposed Solution - PolyCAFe <ul><li>Chat and Forum Analysis and Feedback System </li></ul><ul><li>PolyCAFe = Poly phony-based C ollaboration A nalysis and Fe edback generation </li></ul><ul><li>Provides feedback to learners, tutors and teachers related to the interaction of students in online discussions </li></ul><ul><li>Takes into account both: </li></ul><ul><ul><li>Content of the conversation (related to a domain or topic) </li></ul></ul><ul><ul><li>Collaboration (related to the conversation, participation, etc.) </li></ul></ul>
  12. 12. Technical architecture
  13. 13. Utterance evaluation
  14. 14. Utterance Evaluation & Threads Utterance Past Future Thread Coherence Future Impact Content In-Degree Out-Degree Social Current Relevance Completeness Centrality
  15. 15. Conversation Visualization
  16. 16. Utterance Feedback
  17. 17. Validation Experiment <ul><li>35 students part of the HCI course </li></ul><ul><ul><li>Experimental group: 25 </li></ul></ul><ul><ul><li>Control group: 10 </li></ul></ul><ul><li>Divided in teams of 5 </li></ul><ul><li>Had two distinct assignments that were correlated </li></ul><ul><li>6 tutors had to provide manual feedback for the students: using and not using PolyCAFe </li></ul>
  18. 18. Assignment Example <ul><li>A debate about the best collaboration tool for the web: chat, blog, wiki, forums and Google Wave. Each student shall choose one of the 5 tools and shall present its advantages and the disadvantages of the other tools. Thus, you will act as a &quot; sales person &quot; for your tool and try to convince the others that you have the best offer. You must also defend your product whenever possible and criticize the other products if needed. </li></ul>
  19. 19. Tutor Efficiency <ul><li>VT1: Tutors/facilitators spend less time preparing feedback for learners compared with traditional means </li></ul><ul><li>Likert questionnaire: everyone agreed that “they find the information needed to write the feedback for the learners more quickly using PolyCAFe than without it” (m=4.7, sd=0.52, agree=100%) </li></ul><ul><li>Comparison between average time needed to prepare feedback for a conversation with and without the system: </li></ul><ul><ul><li>without PolyCAFe: 84 minutes </li></ul></ul><ul><ul><li>with PolyCAFe: 55 minutes </li></ul></ul><ul><ul><li>Improvement: 35% </li></ul></ul>
  20. 20. Quality and Consistency of Feedback <ul><li>VT2: Learners perceive that the feedback received from the system contributes to informing their study activities </li></ul><ul><li>Logging: 285 visits to PolyCAFe and 1447 page-views, that results in more than 40 page-views in average per student. </li></ul>
  21. 21. Quality and Consistency of Feedback Validation statement Mean Standard deviation % Agree / Strongly agree The information the system provides me is accurate enough for helping me perform my learning tasks. 3.7 0.52 60% P olyCAFe's feedback is sufficiently accurate to inform my study activities. 3.8 0.88 64% PolyCAFe provides feedback that is useful to my study activities. 3.8 0.85 72% P olyCAFe provides feedback that is relevant to my study activities. 3.9 0.91 72% I trust PolyCAFe to provide helpful feedback. 4.0 0.87 80%
  22. 22. Quality of Educational Output <ul><li>VT3: Learner performance in online discussions is improved in the areas of content coverage and collaboration when using PolyCAFe </li></ul><ul><li>Measurements computed for the second chat assignment, by comparing experimental with control groups </li></ul>
  23. 23. Quality of Educational Output Experimental group Control group Improvement over control group Average score for a chat conversation (collaboration + content) 6.80 6.37 6.8% Average importance of the most important 20 concepts 0.194 0.192 1.2% Average number of utterances 351 338 3.8% Average distribution of (implicit and explicit) links between utterances 1.12 0.87 29%
  24. 24. Verification Experiments <ul><li>Utterance scoring </li></ul><ul><li>Participant ranking </li></ul><ul><li>Speech acts classification </li></ul><ul><li>Evaluation of collaboration score </li></ul><ul><li>& more </li></ul>
  25. 25. Utterance Scoring <ul><li>Chat 1 (331 utterances): </li></ul><ul><li>Scores: 1 (not important) – 4 (very important) </li></ul><ul><li>Tutor 1–Tutor 2 (inter-rater) correlation: 61% </li></ul><ul><li>Tutor 1 – PolyCAFe correlation: 60% </li></ul><ul><li>Tutor 2 – PolyCAFe correlation: 51% </li></ul><ul><li>Tutor average – PolyCAFe correlation: 57% </li></ul>
  26. 26. Participant Ranking Rank Student 1 Student 2 Student 3 Student 4 Student 5 Student 1 - 2 3 1 4 Student 2 2 - 3 1 4 Student 3 2 3 - 1 4 Student 4 1 2 3 - 4 Student 5 1 2 4 3 - Student average 2 3 4 1 5 Tutor 1 4 1 5 2 3 Tutor 2 4 2 5 1 3 Tutor average 4 1-2 5 1-2 3 PolyCAFe 4 2 5 1 3
  27. 27. Participant Ranking Rankings compared Correlation Precision Average distance Tutors – System 94% 77% 0.23 Students – System 84% 66% 0.43 Tutors – Students 84% 71% 0.40
  28. 28. Transferability Issues <ul><li>Domain </li></ul><ul><ul><li>The topic of the conversation should be easily solved using discussions, no graphics or formulas </li></ul></ul><ul><li>Language </li></ul><ul><ul><li>Need for the components of the NLP pipe </li></ul></ul><ul><ul><li>Corpus for training the LSA </li></ul></ul><ul><ul><li>Maybe, a domain ontology </li></ul></ul><ul><li>Activity </li></ul><ul><ul><li>Collaborative activity </li></ul></ul><ul><ul><li>Teams of 4-15 students (in the current design) </li></ul></ul>
  29. 29. Conclusions <ul><li>Learners use web communication technologies </li></ul><ul><li>Need tools to harvest this data </li></ul><ul><li>Want to replace tutors? </li></ul><ul><ul><li>No! Just to provide provisory feedback to learners </li></ul></ul><ul><ul><li>Support tutors to provide final feedback </li></ul></ul><ul><ul><li>Enhance the usage of web conversations by the participants </li></ul></ul><ul><li>The feedback that is provided still needs to be improved </li></ul><ul><ul><li>Corpora with good and bad conversations manually annotated by tutors </li></ul></ul>
  30. 30. Test it! <ul><li>http://ltfll-lin.code.ro/ltfll/wp5/ </li></ul><ul><li>Follow link to test PolyCAFe </li></ul><ul><li>Screencasts for all LTfLL services: </li></ul><ul><ul><li>http://augur.wu.ac.at/screencasts/v1/ </li></ul></ul>
  31. 31. THANK YOU! <ul><li>Questions </li></ul><ul><li> & </li></ul><ul><li>Feedback </li></ul><ul><li>Special thanks to FP7 REGPOT ERRIC </li></ul>

×