Your SlideShare is downloading. ×
Automatic assessment of collaborative chat conversations with PolyCAFe - EC-TEL2011
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Automatic assessment of collaborative chat conversations with PolyCAFe - EC-TEL2011

706
views

Published on

Published in: Education, Technology

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
706
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Automatic Assessment of Collaborative Chat Conversations with PolyCAFe Traian Rebedea 1 , Mihai Dascalu 1 , Stefan Trausan-Matu 1 , Gillian Armitt 2 & Costin Chiru 1 1 - Politehnica University of Bucharest 2 - University of Manchester
  • 2. Overview
    • Context
    • Polyphonic Framework
    • System Architecture
    • Utterance Assessment
    • Validation Experiment
    • Verification Results
    • Transferability & Conclusions
  • 3. Context
    • CSCL
    • Online textual interactions using web communication technologies
    • Education through dialogue
    • Focus on chat conversations in small groups
    • There is usually no feedback from tutors (difficult, time-consuming, etc.)
  • 4. Context – Analysis Tools
    • The need for automatic Feedback
    • Assessment
    • There is a wide variety of research for analyzing chat conversations:
      • topic detection and extraction
      • concept formation in a group discussion
      • summarization
      • argumentation and transactivity acts in each posting
      • utterance classification based on concept coverage
    • However, there is no complete analysis tool for online interactions (like chat, forums) of students
  • 5. Problems
    • P1. Need to integrate feedback on different levels: conversation, utterance, participant
    • S1. Mixture of techniques:
      • Natural Language Processing
      • Social Network Analysis
      • Information Retrieval
    • P2. Supplementary, focus on measuring collaborative discourse as a key characteristic of successful online textual interaction
    • S2. New theory of discourse for chat conversations with multiple participants
  • 6. Polyphonic Framework
    • Classical NLP discourse Monologs
    • Dialogues
    • Coherence is essential in the evolution of discourse (segmentation, topic change, etc.)
    • Dialogues are also based on a two interlocutors model
      • Speech acts or dialogue acts
      • Adjacency pairs
      • Transacts
    • However, chat conversations with multiple participants are different !
  • 7. Polyphonic Principles
    • Floor may be shared by different participants at the same time
    • Allows the evolution of parallel “discussions”
    • Looking at the “serial” resulting text by concatenating the utterances by post time => the discourse may look incoherent
    • Start from Bakhtin’s dialogic theory for discourse analysis
      • Discussion threads <=> voices that are more or less powerful than others throughout the conversation
      • Influence between different voices
  • 8. Words, Voices and Discussion Threads
    • The topic of the discussion defines a list of important concepts (words/stems/…) => initial voices
    • Other voices appear naturally during the conversation
      • Some may be related to the initial ones
    • Voices can influence each other through explicit or implicit links between utterances
    • One or more similar voices can define a discussion thread
    • Discussion threads = a set of utterances that are linked implicitly or explicitly and correspond to a specific voice or more similar voices
  • 9. Implicit Links
    • In order for the analysis to be effective, it is important to discover implicit links between utterances
    • Very difficult task: trade-off precision-recall
    • Repetitions (expanded using ontologies)
    • Semantic similarity (including lexical chains)
    • Adjacency pairs (using speech acts)
    • Cue phrases ( “that’s a good idea”, “as <x> said”, …)
  • 10. Conversation Graph
    • The conversation can be modeled as a graph:
      • Utterances – nodes
      • Links – edges
    • Each edge has a trust factor
    • The conversation graph is fundamental to the analysis:
      • Can be used as a social network
      • Can be used to compute conversation threads, coherence, collaborative discourse, etc.
  • 11. Proposed Solution - PolyCAFe
    • Chat and Forum Analysis and Feedback System
    • PolyCAFe = Poly phony-based C ollaboration A nalysis and Fe edback generation
    • Provides feedback to learners, tutors and teachers related to the interaction of students in online discussions
    • Takes into account both:
      • Content of the conversation (related to a domain or topic)
      • Collaboration (related to the conversation, participation, etc.)
  • 12. Technical architecture
  • 13. Utterance evaluation
  • 14. Utterance Evaluation & Threads Utterance Past Future Thread Coherence Future Impact Content In-Degree Out-Degree Social Current Relevance Completeness Centrality
  • 15. Conversation Visualization
  • 16. Utterance Feedback
  • 17. Validation Experiment
    • 35 students part of the HCI course
      • Experimental group: 25
      • Control group: 10
    • Divided in teams of 5
    • Had two distinct assignments that were correlated
    • 6 tutors had to provide manual feedback for the students: using and not using PolyCAFe
  • 18. Assignment Example
    • A debate about the best collaboration tool for the web: chat, blog, wiki, forums and Google Wave. Each student shall choose one of the 5 tools and shall present its advantages and the disadvantages of the other tools. Thus, you will act as a &quot; sales person &quot; for your tool and try to convince the others that you have the best offer. You must also defend your product whenever possible and criticize the other products if needed.
  • 19. Tutor Efficiency
    • VT1: Tutors/facilitators spend less time preparing feedback for learners compared with traditional means
    • Likert questionnaire: everyone agreed that “they find the information needed to write the feedback for the learners more quickly using PolyCAFe than without it” (m=4.7, sd=0.52, agree=100%)
    • Comparison between average time needed to prepare feedback for a conversation with and without the system:
      • without PolyCAFe: 84 minutes
      • with PolyCAFe: 55 minutes
      • Improvement: 35%
  • 20. Quality and Consistency of Feedback
    • VT2: Learners perceive that the feedback received from the system contributes to informing their study activities
    • Logging: 285 visits to PolyCAFe and 1447 page-views, that results in more than 40 page-views in average per student.
  • 21. Quality and Consistency of Feedback Validation statement Mean Standard deviation % Agree / Strongly agree The information the system provides me is accurate enough for helping me perform my learning tasks. 3.7 0.52 60% P olyCAFe's feedback is sufficiently accurate to inform my study activities. 3.8 0.88 64% PolyCAFe provides feedback that is useful to my study activities. 3.8 0.85 72% P olyCAFe provides feedback that is relevant to my study activities. 3.9 0.91 72% I trust PolyCAFe to provide helpful feedback. 4.0 0.87 80%
  • 22. Quality of Educational Output
    • VT3: Learner performance in online discussions is improved in the areas of content coverage and collaboration when using PolyCAFe
    • Measurements computed for the second chat assignment, by comparing experimental with control groups
  • 23. Quality of Educational Output Experimental group Control group Improvement over control group Average score for a chat conversation (collaboration + content) 6.80 6.37 6.8% Average importance of the most important 20 concepts 0.194 0.192 1.2% Average number of utterances 351 338 3.8% Average distribution of (implicit and explicit) links between utterances 1.12 0.87 29%
  • 24. Verification Experiments
    • Utterance scoring
    • Participant ranking
    • Speech acts classification
    • Evaluation of collaboration score
    • & more
  • 25. Utterance Scoring
    • Chat 1 (331 utterances):
    • Scores: 1 (not important) – 4 (very important)
    • Tutor 1–Tutor 2 (inter-rater) correlation: 61%
    • Tutor 1 – PolyCAFe correlation: 60%
    • Tutor 2 – PolyCAFe correlation: 51%
    • Tutor average – PolyCAFe correlation: 57%
  • 26. Participant Ranking Rank Student 1 Student 2 Student 3 Student 4 Student 5 Student 1 - 2 3 1 4 Student 2 2 - 3 1 4 Student 3 2 3 - 1 4 Student 4 1 2 3 - 4 Student 5 1 2 4 3 - Student average 2 3 4 1 5 Tutor 1 4 1 5 2 3 Tutor 2 4 2 5 1 3 Tutor average 4 1-2 5 1-2 3 PolyCAFe 4 2 5 1 3
  • 27. Participant Ranking Rankings compared Correlation Precision Average distance Tutors – System 94% 77% 0.23 Students – System 84% 66% 0.43 Tutors – Students 84% 71% 0.40
  • 28. Transferability Issues
    • Domain
      • The topic of the conversation should be easily solved using discussions, no graphics or formulas
    • Language
      • Need for the components of the NLP pipe
      • Corpus for training the LSA
      • Maybe, a domain ontology
    • Activity
      • Collaborative activity
      • Teams of 4-15 students (in the current design)
  • 29. Conclusions
    • Learners use web communication technologies
    • Need tools to harvest this data
    • Want to replace tutors?
      • No! Just to provide provisory feedback to learners
      • Support tutors to provide final feedback
      • Enhance the usage of web conversations by the participants
    • The feedback that is provided still needs to be improved
      • Corpora with good and bad conversations manually annotated by tutors
  • 30. Test it!
    • http://ltfll-lin.code.ro/ltfll/wp5/
    • Follow link to test PolyCAFe
    • Screencasts for all LTfLL services:
      • http://augur.wu.ac.at/screencasts/v1/
  • 31. THANK YOU!
    • Questions
    • &
    • Feedback
    • Special thanks to FP7 REGPOT ERRIC