Computer-Based Support and Feedback for  Collaborative Chat Conversations and           Discussion Forums        PhD Candi...
Contents• Introduction• Inter-animation in Online Conversations      – Types of Interactions between Utterances• Discourse...
Introduction “Time changes all things; there is no reason why     language should escape this universal law” (            ...
Introduction• Evolution of online communication (and the  linguistic artifacts): chat conversation, online  forums, blogs,...
Introduction• Inter-disciplinary research topic      – Computer Supported Collaborative Learning        (CSCL)      – Natu...
Research Question• How to design and implement a system to provide  automatic feedback and support for learners and  tutor...
Why?• Which of the previous two messages does  reply 68 continue?      – Does Loredana continue her own message?      – Do...
Why?14 December 2012,      PhD Defense - Computer-Based Support and Feedback forBucharest Romania       Collaborative Chat...
Why?• Collaboration & parallel discussion threads14 December 2012,          PhD Defense - Computer-Based Support and Feedb...
Novelty• Inter-animation framework for online conversation  analysis• Study for detecting implicit links in discussions• B...
Inter-animation in Collaborative            Chat Conversations   “The text lives only by coming into contact with     anot...
Scope• Inter-animation framework of analysis      – Developed for multi-party online conversations        (especially chat...
The Utterance• “unit of speech communication […] determined by a  change of speaking subjects, that is, a change of  speak...
Inter-animation• Inter-animation: “the meaning of an utterance is  not reducible to the intentions of the speaker or  to t...
In Online Conversations• Interactions or links among utterances and  among different discussion topics• Explicit links    ...
Problems in Online Conversations• Multiple parallel discussion floors• Different discussion threads, running in parallel, ...
Similar Problems• Identifying implicit links in      • Newsgroups      • Blog posts      • Chats• Thread detection in chat...
Classification of Utterances•   Speech acts•   Dialog acts•   Argumentation acts•   Rhetoric structures•   Transacts14 Dec...
Example: Argumentation in Chat34 Mona              Conversation              Ok! So... main the reason I sustain this tech...
Classification of Interactions between                  Utterances• Interactions based on Lexical Cohesion      – Reiterat...
Lexical Chains in Chats64       and if you do not have a good internet connection, clasic chat chould also be a solution.....
Semantic Similarity• Knowledge-based methods      – Dictionaries, thesauri, ontologies, other semantic        networks (DO...
Semantic Similarity• Corpus-based methods      – Statistical analysis of occurrences and co-occurrences of        words in...
Discourse Models for WebConversations like Chats & Forums                    “Practically everything is a graph”          ...
Types of Discourses•   OLAC (Open Language Archives Community)•   Dialogue: “conversations, interviews, correspondence, co...
Interactions/Links among Utterances• “The base condition governing the operation  of meaning in any utterance. It is that ...
Modelling Conversations as a                    Graph• Many discourse theories define links between  discourse units      ...
Conversation Graphs14 December 2012,              PhD Defense - Computer-Based Support and Feedback forBucharest Romania  ...
Conversation Graphs14 December 2012,              PhD Defense - Computer-Based Support and Feedback forBucharest Romania  ...
Explicit Links• Corpus with 8 chats, 2800+ utterances,  1900+ explicit links              0.25               0.2          ...
Shallow Links• Broken utterances / split turns• Reasons:      – Limitations in the length of the utterance caused by the  ...
29      how could we forget the                                                Michael      Example 1     30      blog    ...
Lexical Links14 December 2012,            PhD Defense - Computer-Based Support and Feedback forBucharest Romania          ...
Semantic Links         0.12          0.1         0.08         0.06                                                        ...
WordNet – Resnik   0.25    0.2   0.15                                                                                     ...
148     ok.... so what we were talking about before the             Mona           connection issue?   149   150     meeti...
How useful are semantic links for       implicit links’ detection?                              131   132   133   134     ...
Open Issues in Building the               Conversation Graph• Automatic Identification of Utterance Types      – Not just ...
Developing PolyCAFe   “In theory, theory and practice are the same. In  practice, they are not.” (attributed to various, i...
Problems• P1. Need to integrate feedback on different  levels: conversation, utterance, participant• S1. Mixture of techni...
Proposed Solution - PolyCAFe• Chat and Forum Analysis and Feedback System• PolyCAFe = Polyphony-based Collaboration Analys...
Implicit Links in PolyCAFe•   Repetitions (expanded using ontologies)•   Lexical chains (using WordNet, LSA and ESA)•   Se...
Technical architecture14 December 2012,                PhD Defense - Computer-Based Support and Feedback forBucharest Roma...
Utterance Evaluation & Threads                    Thread Coherence    Content              Future Impact                  ...
Utterance evaluation14 December 2012,               PhD Defense - Computer-Based Support and Feedback forBucharest Romania...
Lexical Chains Using ESA   employees employe employee employer employers   audit auditing statistics audits business   pro...
Conversation Visualization14 December 2012,            PhD Defense - Computer-Based Support and Feedback forBucharest Roma...
Utterance Feedback14 December 2012,              PhD Defense - Computer-Based Support and Feedback forBucharest Romania   ...
PolyCAFe Validation Experiments          «Validation ensures that “you built the right          thing”. Verification ensur...
Validation Experiment• 35 students part of the HCI course      – Experimental group: 25      – Control group: 10• Divided ...
Tutor Efficiency• VT1: Tutors/facilitators spend less time preparing  feedback for learners compared with traditional mean...
Quality and Consistency of Feedback• VT2: Learners perceive that the feedback  received from the system contributes to  in...
Quality and Consistency of Feedback                                                                      % Agree /        ...
Quality of Educational Output• VT3: Learner performance in online  discussions is improved in the areas of content  covera...
Quality of Educational Output                            Experimental      Control          Improvement over              ...
Verification Experiments•   Utterance scoring•   Participant ranking•   Speech acts classification•   Evaluation of collab...
Utterance Scoring• Chat 1 (331 utterances):• Scores: 1 (not important) – 4 (very important)• Tutor 1–Tutor 2 (inter-rater)...
Participant Ranking  Rank                Student 1 Student 2 Student 3 Student 4 Student 5  Student 1                   - ...
Participant Ranking          Rankings                                                    Average                        Co...
Using Semantics for Improving       Relevance of Text Documents         “All our work, our whole life is a matter of      ...
Wikipedia-Based Semantics for        Improving Search Relevance                Refined Rank = Rank(Google) + p * S        ...
Improved Semantics    Source Concept     Destination Concept                          Score    avl tree           self bal...
Relevance for Ranking Video            Comments on YouTube• The Beatles- Here Comes The Sun• On YouTube we have two option...
Solution14 December 2012,         PhD Defense - Computer-Based Support and Feedback forBucharest Romania          Collabor...
Our results Comment                                                                                      Relevance maybe y...
Overall Conclusions   “Im sorry I wrote you such a long letter; I didnt     have time to write a short one.” (attributed t...
••     Conference proceedings and journal articles                                                             Publication...
Publications (2)•   Book Chapters•   Rebedea T., Trausan-Matu S., Chiru C. - Inter-animation between Utterances in Collabo...
THANK YOU!                    Questions                           &                           Feedback14 December 2012,   ...
Upcoming SlideShare
Loading in...5
×

PhD Defense: Computer-Based Support and Feedback for Collaborative Chat Conversations and Discussion Forums

1,849
-1

Published on

Published in: Education
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,849
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide
  • Utterance = interventia
  • “ Within the arena of almost every utterance an intense interaction and struggle between one's own and another's word is being waged, a process in which they oppose or dialogically interanimate each other.” (Bakhtin, 1981)
  • A fost un drum lung, interesant si cateodata anevoios, Insa am cunoscut multi oameni extraordinari, cu care am mers mai departe – colegi din facultate, din proiecte, studenti Lor trebuie sa le multumesc, dar si familiei si prietenilor, dl.prof. Trausan pentru sfaturile primite in ultimii multi ani, precum si comisiei de doctorat pentru efortul intreprins in a intelege si imbunatati anumite aspecte ale tezei.
  • PhD Defense: Computer-Based Support and Feedback for Collaborative Chat Conversations and Discussion Forums

    1. 1. Computer-Based Support and Feedback for Collaborative Chat Conversations and Discussion Forums PhD Candidate: Traian Rebedea Scientific Advisor: Ștefan Trăușan-Matu
    2. 2. Contents• Introduction• Inter-animation in Online Conversations – Types of Interactions between Utterances• Discourse Models for Online Conversations – Conversation Graph• PolyCAFe – A Chat & Forum Analysis and Feedback System• PolyCAFe Validation Experiments• Using Semantics for Improving Relevance of Text Documents• Conclusions14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    3. 3. Introduction “Time changes all things; there is no reason why language should escape this universal law” ( de Saussure, 1915, p. 77)14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    4. 4. Introduction• Evolution of online communication (and the linguistic artifacts): chat conversation, online forums, blogs, social networks• New type of discourse – Highly collaborative – Many participants – Different from narratives, but also from dialogues• Parallel discussion threads, topic shifts, new language, etc.• Not investigated enough, there is no linguistic theory built for this type of discourse14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    5. 5. Introduction• Inter-disciplinary research topic – Computer Supported Collaborative Learning (CSCL) – Natural Language Processing – Collaborative Knowledge Building• CSCL – Uses lots of online communication – Massive Open Online Courses (MOOCs)14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    6. 6. Research Question• How to design and implement a system to provide automatic feedback and support for learners and tutors involved in online discussions? – How to (best) analyze collaborative online discussions using chat and forums? – What elements of the proposed theoretical framework can be used to design a system for automatic analysis of chats and forums in a CSCL situation? – How accurate is the feedback offered by such a system? – Is the system efficient for the learners and tutors that are using it? – Can semantics be used in other domains as well?14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    7. 7. Why?• Which of the previous two messages does reply 68 continue? – Does Loredana continue her own message? – Does she continue the previous message (67)? 66 yes, i think chat is more for brainstorming Loredana 67 chats greatest advantage is the simplicity it proposes Alex 68 or maybe finding out faster and direct ways of dealing Loredana with a certain domain14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    8. 8. Why?14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    9. 9. Why?• Collaboration & parallel discussion threads14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    10. 10. Novelty• Inter-animation framework for online conversation analysis• Study for detecting implicit links in discussions• Building a conversation graph using natural language processing (NLP) techniques• Development of PolyCAFe – POLYphony-based system for Collaboration Analysis and Feedback generation• Validation experiments and results• Using Wikipedia to extract semantics that may be used to improve the relevance of online text documents14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    11. 11. Inter-animation in Collaborative Chat Conversations “The text lives only by coming into contact with another text. Only at the point of this contact between texts does a light flash…” (Bakhtin, 1986)14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    12. 12. Scope• Inter-animation framework of analysis – Developed for multi-party online conversations (especially chat and other short text streams) – Focus on discovering interactions/links/references among utterances – Determining discussion threads – Compute interactions between threads and the degree of inter-animation or collaborative discourse14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    13. 13. The Utterance• “unit of speech communication […] determined by a change of speaking subjects, that is, a change of speakers.” (Bakhtin)• A unit of speech communication – not be related to existing boundaries in written text, such as phrases, sentences, paragraphs, etc.• An utterance does not have a specific length (may vary from a word to a novel or a treaty).• Any utterance should be meaningful on its own.• “Responsive understanding” – other utterances are linked to previous ones in the same discourse.14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    14. 14. Inter-animation• Inter-animation: “the meaning of an utterance is not reducible to the intentions of the speaker or to the response of the addressee but emerges between these two.” (Wegerif, 2008)• Polyphony, heteroglossia, inter-animation• Intertextuality• Interdiscourse• Ideal text, memex, hypertext, WWW14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    15. 15. In Online Conversations• Interactions or links among utterances and among different discussion topics• Explicit links – #tag – @username – Other means for linking utterances• Implicit links – Must be determined automatically14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    16. 16. Problems in Online Conversations• Multiple parallel discussion floors• Different discussion threads, running in parallel, inter-animating or competing• “competition for attention or control of the discourse” (Cherny, 1999)• Patterns that have been identified in specific conversations: speaker-and-supporter, collaborative and multiple conversational floors14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    17. 17. Similar Problems• Identifying implicit links in • Newsgroups • Blog posts • Chats• Thread detection in chat and chat-like conversations• Thread reconstruction in discussion forums• Uptake analysis framework14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    18. 18. Classification of Utterances• Speech acts• Dialog acts• Argumentation acts• Rhetoric structures• Transacts14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    19. 19. Example: Argumentation in Chat34 Mona Conversation Ok! So... main the reason I sustain this technology is its accesibility CLAIM35 Mona Every person that knows how to type can succesfully be part of a chat DATA (LINK TO 34)36 Cristi indeed .... chat is everywhere DATA (LINK TO 34)37 Cristi nowadays almost everyone uses one chat client DATA QUALIFIER (LINK TO 34)38 Stefan Why do you say that? Every other techology is everywhere. CLAIM2 REBUTTAL (LINK TO 34, 36)39 Corina true.... (LINK TO 39)40 Mona By accesibility I was also referring to the way you use it... chat is one of the WARRANT most simple means of communication nowadays (LINK TO 34)41 Stefan thats true. (LINK TO 40)42 Cristi is not very complex ... mainly is just typing BACKING (LINK TO 40)43 Mona Even a 7 year old kid can use messenger for example BACKING (LINK TO 40)14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    20. 20. Classification of Interactions between Utterances• Interactions based on Lexical Cohesion – Reiterations – Collocations – Lexical chains• Adjacency pairs• Coherence relations• Interactions based on Semantic Similarity• Anaphora and Coreference Resolution14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    21. 21. Lexical Chains in Chats64 and if you do not have a good internet connection, clasic chat chould also be a solution... Mona65 agree...videochat requires some bandwidth Cristi66 ok...but what about information, what do you think it is the most reliable source of Diana information?67 chat is fast, simple to use and it does not require such a good internet connection Mona68 if it is an inside network i dont think this should be a problem Stefan69 well depends on the type of the meeting... Corina70 what do you mean by inside network? Mona71 in the same building? Mona72 LAN Stefan73 nowadays a fast internet connection is everywhere :) Cristi74 or VPN Stefan75 true.... internet connection is not really a problem nowadays... Mona76 so we all agree that videochat is the best solution for a "virtual" meeting Mona77 undoubtedly Cristi78 yes! Corina79 so.... one activity solved.... Mona80 December 2012, meetings we have video chat14 ok so for PhD Defense - Computer-Based Support and Feedback for StefanBucharest Romania Collaborative Chat Conversations and Discussion Forums
    22. 22. Semantic Similarity• Knowledge-based methods – Dictionaries, thesauri, ontologies, other semantic networks (DOLCE, SUMO, Cyc)• The length of the path between the two concepts• The depth of the node in the IS-A taxonomy• The lowest common ancestor (LCA or lowest super- ordinate – LSO)• The information content of a concept• The commonality and difference between the two concepts in the IS-A taxonomy14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    23. 23. Semantic Similarity• Corpus-based methods – Statistical analysis of occurrences and co-occurrences of words in a large collections of documents• PMI (Point-wise mutual information)• GND (Google Normalized Distance)  WWW-based• LSA (Latent Semantic Analysis)• LDA (Latent Dirichlet Allocation)• ESA (Explicit Semantic Analysis)  Wikipedia-based14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    24. 24. Discourse Models for WebConversations like Chats & Forums “Practically everything is a graph” XKCD comics:14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    25. 25. Types of Discourses• OLAC (Open Language Archives Community)• Dialogue: “conversations, interviews, correspondence, consultations, greetings and leave-takings”• Drama: “plays, skits, and enacted scenes”• Formulaic: “prayers, curses, blessings, charms, curing rituals, marriage vows, and oaths”• Ludic: “play languages, jokes, secret languages, and speech disguises”• Oratory: “Public speaking, or of speaking eloquently according to rules or conventions”• Narrative: “historical, traditional, and personal narratives, myths, folktales, fables, and humorous stories”• Procedural: “recipes, instructions, and plans”• Report: “news reports, essays, and commentaries”• Singing• Unintelligible speech14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    26. 26. Interactions/Links among Utterances• “The base condition governing the operation of meaning in any utterance. It is that which insures the primacy of context over text. At any given time, in any given place, there will be a set of conditions – social, historical, meteorological, physiological – that will insure that a word uttered in that place and at that time will have a meaning different than it would have under any other conditions” (Bakhtin, 1981, p. 428)14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    27. 27. Modelling Conversations as a Graph• Many discourse theories define links between discourse units – Rhetorical structure – Coherence relations – Adjacency pairs• Stavrianou: post-opinion graph for online forums• Suthers: contingency graphs, uptake graph14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    28. 28. Conversation Graphs14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    29. 29. Conversation Graphs14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    30. 30. Explicit Links• Corpus with 8 chats, 2800+ utterances, 1900+ explicit links 0.25 0.2 0.15 Distribution of explicit links Poly. (Distribution of explicit 0.1 links) 0.05 2 R = 0.9927 0 0 5 10 15 2014 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    31. 31. Shallow Links• Broken utterances / split turns• Reasons: – Limitations in the length of the utterance caused by the environment – Incomplete turns due to haste – Trying to grab the discussion floor and the attention of the other participants although an utterance is still not finalized – Lengthy turns are seldom broken into multiple shorter ones – Allowing the other participants to continue the utterance that is not finalized or even incomplete ideas14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    32. 32. 29 how could we forget the Michael Example 1 30 blog Michael LINK TO 29 … 36 lets try to focus on the subject matter Michael Example 2 37 if you had to choose ONE reason Michael 38 for choosing your technology Michael LINK TO 37 39 what would that one reason be ? Michael LINK TO 37 … 66 yes, i think chat is more for brainstorming Loredana Example 3 67 chats greatest advantage is the simplicity it proposes Alex 68 or maybe finding out faster and direct ways of dealing with a Loredana LINK TO 67 certain domain … 70 true, however, in most projects you need a data format with Michael Example 4 more structure 71 and that is why i think Wave offers a good trade-off between Michael LINK TO 70 structure and interactivity … 75 A chat or a chat room has a more private nature than blogs or Alex Example 5 waves or wikis … 78 but privacy isnt really the best choice if you have an urgent Cristi LINK TO 75 matter and youre looking for answers … 123 of course Michael Example 6 124 but to make things interesting Michael LINK TO 123 125 were in different cities/countries/timezones Michael LINK TO 124 126 i.e. we cant be in the same room Michael LINK TO 12514 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    33. 33. Lexical Links14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    34. 34. Semantic Links 0.12 0.1 0.08 0.06 LSA similarity score between utterances 0.04 0.02 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 2014 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    35. 35. WordNet – Resnik 0.25 0.2 0.15 WordNet similarity score between utterances 0.1 0.05 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 2014 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    36. 36. 148 ok.... so what we were talking about before the Mona connection issue? 149 150 meeting board LSA for implicit links as long as it depends on an internet connection... Mona Cristi LINK TO 147 LINK TO score(150, 148) = 148 0.18 = max 151 about different stages of a project... a client must Corina LINK TO know about them..so wiki is a good solution 148 152 meeting? Corina LINK TO score(152, 150) = 150 1.00 = max 153 ok Stefan LINK TO 151 154 so we agree that wiki is a good solution when we want Corina score(154, 151) = to present a product/the evolution of a project to a 0.5 = max client ? 155 what about selling our products..what technologies we Diana should use for this? … 166 to sell our products blog is the best solution Corina LINK TO score(166, 155) = 155 0.38 = max … 169 A blog would be a good way to advertise our products Mona LINK TO score(169, 166) = 166 0.48 = max … 180 A forum would be useful for offering solutions to some Mona problems that our customers have 181 I agree...also other people can offer solutions, not only Corina LINK TO score(181, 180) = us 181 0.27 = max14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    37. 37. How useful are semantic links for implicit links’ detection?  131 132 133 134 135 136 143 An5 TotReferences  greater  than  0.44 0.47 0.41 0.45 0.42 0.43 0.45 0.43 0.4410-turn medianReferences  greater  than  0.40 0.47 0.41 0.47 0.42 0.45 0.46 0.45 0.4420-turn medianPearson  correlation between  number  of  0.83 0.90 0.81 0.88 0.76 0.92 0.78 0.83 0.84references  and  average similarity score14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    38. 38. Open Issues in Building the Conversation Graph• Automatic Identification of Utterance Types – Not just speech acts, but also argumentation – Machine learning vs rule-based• Questions and Answers in Online Conversations – Questions are easier to identify – Pairing answers to questions is much more difficult• Conversation-Focused Coreference Resolution14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    39. 39. Developing PolyCAFe “In theory, theory and practice are the same. In practice, they are not.” (attributed to various, in press)14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    40. 40. Problems• P1. Need to integrate feedback on different levels: conversation, utterance, participant• S1. Mixture of techniques: – Natural Language Processing – Social Network Analysis – Information Retrieval• P2. Supplementary, focus on measuring collaborative discourse as a key characteristic of successful online textual interaction• S2. New theory of discourse for chat conversations with multiple participants14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    41. 41. Proposed Solution - PolyCAFe• Chat and Forum Analysis and Feedback System• PolyCAFe = Polyphony-based Collaboration Analysis and Feedback generation• Provides feedback to learners, tutors and teachers related to the interaction of students in online discussions• Takes into account both: – Content of the conversation (related to a domain or topic) – Collaboration (related to the conversation, participation, etc.)14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    42. 42. Implicit Links in PolyCAFe• Repetitions (expanded using ontologies)• Lexical chains (using WordNet, LSA and ESA)• Semantic similarity (including lexical chains)• Adjacency pairs (using speech acts)• Cue phrases (“that’s a good idea”, “as <x> said”, …)14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    43. 43. Technical architecture14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    44. 44. Utterance Evaluation & Threads Thread Coherence Content Future Impact Relevance Completeness Utterance Centrality In-Degree Social Out-Degree Past Current Future14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    45. 45. Utterance evaluation14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    46. 46. Lexical Chains Using ESA employees employe employee employer employers audit auditing statistics audits business provides provide service communication executives state world constitution companies company corporate market redundant information technology chats clients irc chat client comes google come news refers friends live life status work policies prfit public policy real complex analysis number programs computer software marketing profitability discussing forums forum personal inter-personal users interface window question ask questions room rooms chat-rooms research university brainstorming mind music song artist14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    47. 47. Conversation Visualization14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    48. 48. Utterance Feedback14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    49. 49. PolyCAFe Validation Experiments «Validation ensures that “you built the right thing”. Verification ensures that “you built it right”.» (in press and in project meetings)14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    50. 50. Validation Experiment• 35 students part of the HCI course – Experimental group: 25 – Control group: 10• Divided in teams of 5• Had two distinct assignments that were correlated• 6 tutors had to provide manual feedback for the students: using and not using PolyCAFe14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    51. 51. Tutor Efficiency• VT1: Tutors/facilitators spend less time preparing feedback for learners compared with traditional means• Likert questionnaire: everyone agreed that “they find the information needed to write the feedback for the learners more quickly using PolyCAFe than without it” (average=4.7, agree=100%)• Comparison between average time needed to prepare feedback for a conversation with and without the system: – without PolyCAFe: 84 minutes – with PolyCAFe: 55 minutes – Improvement: 35%14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    52. 52. Quality and Consistency of Feedback• VT2: Learners perceive that the feedback received from the system contributes to informing their study activities• Logging: 285 visits to PolyCAFe and 1447 page-views, that results in more than 40 page- views in average per student.14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    53. 53. Quality and Consistency of Feedback % Agree / Validation statement Average Strongly agree The information the system provides me is accurate enough for helping me perform 3.7 60% my learning tasks. PolyCAFes feedback is sufficiently accurate to inform my study activities. 3.8 64% PolyCAFe provides feedback that is useful to my study activities. 3.8 72% PolyCAFe provides feedback that is relevant to my study activities. 3.9 72% I trust PolyCAFe to provide helpful feedback. 4.0 80%14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    54. 54. Quality of Educational Output• VT3: Learner performance in online discussions is improved in the areas of content coverage and collaboration when using PolyCAFe• Measurements computed for the second chat assignment, by comparing experimental with control groups14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    55. 55. Quality of Educational Output Experimental Control Improvement over group group control groupAverage score for a chatconversation(collaboration + content) 6.80 6.37 6.8%Average importance ofthe most important 20 0.194 0.192 1.2%conceptsAverage number ofutterances 351 338 3.8%Average distribution of(implicit and explicit) 1.12 0.87 29%links between utterances14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    56. 56. Verification Experiments• Utterance scoring• Participant ranking• Speech acts classification• Evaluation of collaboration score• & more14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    57. 57. Utterance Scoring• Chat 1 (331 utterances):• Scores: 1 (not important) – 4 (very important)• Tutor 1–Tutor 2 (inter-rater) correlation: 61%• Tutor 1 – PolyCAFe correlation: 60%• Tutor 2 – PolyCAFe correlation: 51%• Tutor average – PolyCAFe correlation: 57%14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    58. 58. Participant Ranking Rank Student 1 Student 2 Student 3 Student 4 Student 5 Student 1 - 2 3 1 4 Student 2 2 - 3 1 4 Student 3 2 3 - 1 4 Student 4 1 2 3 - 4 Student 5 1 2 4 3 - Student average 2 3 4 1 5 Tutor 1 4 1 5 2 3 Tutor 2 4 2 5 1 3 Tutor average 4 1-2 5 1-2 3 PolyCAFe 4 2 5 1 314 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    59. 59. Participant Ranking Rankings Average Correlation Precision compared distance Tutors – System 94% 77% 0.23 Students – System 84% 66% 0.43 Tutors – Students 84% 71% 0.4014 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    60. 60. Using Semantics for Improving Relevance of Text Documents “All our work, our whole life is a matter of semantics, because words are the tools with which we work, the material out of which laws are made […]. Everything depends on our understanding of them.” (Felix Frankfurter)14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    61. 61. Wikipedia-Based Semantics for Improving Search Relevance Refined Rank = Rank(Google) + p * S score14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    62. 62. Improved Semantics Source Concept Destination Concept Score avl tree self balanced binary tree 0.703385 avl tree amortization analysis 0.605471 avl tree donald knuth 0.557814 avl tree b tree 0.546726 avl tree binary tree 0.527878 avl tree computer science 0.527694 avl tree persistent data structure 0.500780 avl tree red black tree 0.441281 avl tree associative array 0.43768314 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    63. 63. Relevance for Ranking Video Comments on YouTube• The Beatles- Here Comes The Sun• On YouTube we have two options1. Top comments – Really? Who wouldve thought? – i think the beatles are better then one derection1. Chronologic – Pathetic attempt to get thumbs up – Why are people comparing The Beatles to all these other musicians? one direction and Justin Bieber are musicians too, but we prefer The Beatles. – yes of course they are?! –…14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    64. 64. Solution14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    65. 65. Our results Comment Relevance maybe your friend should know that being english, have a picture in abbey road 662 and "sing" all you need is love" wont make one direction? a group like the beatles... my mom said she doesnt like the beatles and she said that john was only good 968 to look at? not to hear. my dad said, " haha so true!." im an orphan now. you shouldnt be listening to the beatles since these seem to turn your friends 983 into enemies! beatles are all about peace!? you are not getting their message! please read this ! hey i know u just wanna listen to the song but i still have to 1309 write this hoping someone will see it and that someone will care .im a? young musician from croatia so this spam is my only chance to get noticed.please check out my channel and i promise u wont be sorry.i appreciate your time because music means everything to me, thank you! ? i didnt mean fight other places. i meant focus on the hurt people in your own 1639 country first, then expand to the others. if people dont agree with peace thats an opinion. not a fact, and people often take offense to opinions. there isnt? anything to take offense to, they say something thats all it is. they said it, dont put meaning to it. world peace - i meant the whole world having peace there14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    66. 66. Overall Conclusions “Im sorry I wrote you such a long letter; I didnt have time to write a short one.” (attributed to Blaise Pascal, in press) Inter-animation, implicit links, theoretical framework, conversation graph, PolyCAFe system, validation experiments, measurements, semantics for improved relevance14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums
    67. 67. •• Conference proceedings and journal articles Publications Trausan-Matu S., Dascalu M., Rebedea T. - A System for the Automatic Analysis of Computer-Supported Collaborative Learning Chats, accepted at ICALT 2012 (ISI Web of Knowledge, Category B in ARC Conference Ranking)• Rebedea T., Dascalu M., Trausan-Matu S., Armitt G., Chiru C.G. - Automatic Assessment of Collaborative Chat Conversations with PolyCAFe. EC-TEL 2011, LNCS 6964, Springer, 2011, pp. 299-312 (ISI Web of Knowledge)• Dascalu M., Rebedea T., Trausan-Matu S., Armitt G. - PolyCAFe: Collaboration and Utterance Assessment for Online CSCL Conversations, in CSCL 2011 Conference Proceedings, Hong Kong, vol 2, pp. 781-785 (ISI Web of Knowledge, Category A in ARC Conference Ranking)• Banica D., Trausan-Matu S., Rebedea T. - Detecting Collaboration Regions in a Chat Session, in CSCL 2011 Conference Proceedings, vol 2, Hong Kong, pp. 771-775 (ISI Web of Knowledge, Category A in ARC Conference Ranking)• Dessus P., Trausan-Matu S., Wild F., Dupré D., Loiseau M., Rebedea T., Zampa V. - Un environnement personnel dapprentissage évaluant des distances épistémiques et dialogiques. Distances & Savoirs, 9(4), 473–492• Chiru C., Cojocaru V., Trausan-Matu S, Rebedea T., Mihaila D. - Repetition and Rhythmicity Based Assessment for Chat Conversation. ISMIS 2011, LNAI 6804, Springer, 2011, pp 513-523.• (ISI Web of Knowledge, Category C in ARC Conference Ranking)• Wild F., Ullmann T., Scott P., Rebedea T., Hoisl, B. (2011) - Applicability of the technology acceptance model for widget-based personal learning environments. In: 1st Workshop on Exploring Fitness and Evolvability of Personal Learning Environments (EFEPLE11), 2011, France, http://www.ceur-ws.org/Vol-773/EFEPLE-10-Wild.pdf• Trausan-Matu S., Rebedea T. - A Polyphonic Model and System for Inter-Animation Analysis in Chat Conversations with Multiple Participants, CICLING 2010, LNCS 6008, Springer, 2010, pp.354-363 (ISI Web of Knowledge, Category B in ARC Conference Ranking)• Rebedea T. et al. - Overview and Preliminary Results of Using PolyCAFe for Collaboration Analysis and Feedback Generation, EC-TEL 2010, LNCS 6383, Springer, 2010, pp. 420-425 (ISI Web of Knowledge)• Rebedea T., Posea V., Dascalu M., Trausan-Matu S. - Web Services for Supporting the Interactions of Learners in the Social Web, accepted at 9th International Conference Roedunet 2010 (ISI Web of Knowledge)• Scheau C., Rebedea T., Costin C., Trausan-Matu S. - Improving the Relevance of Search Engine Results by Using Semantic Information from Wikipedia, 9th International Conference Roedunet 2010, pp. 151-156 (ISI Web of Knowledge)• Rebedea T., Trausan-Matu S. - Automatic Feedback System for Collaborative Learning Using Chats and Forums, CSEDU 2010 (ISI Web of Knowledge)• Chiru C.G., Cojocaru V., Rebedea T., Trausan-Matu S. - Malapropisms Detection and Correction Using a Paronyms Dictionary, a Search Engine and WordNet, ICSOFT 2010 (ISI Web of Knowledge, Category B in ARC Conference Ranking)• Chiru C.G., Hanganu A., Rebedea T., Trausan-Matu S. - Filling the Gaps Using Google 5-Grams Corpus, ICSOFT 2010 (ISI Web of Knowledge, Category B in ARC Conference Ranking)• Trausan-Matu S., Posea V., Rebedea T., Chiru C. - Using the Social Web to Supplement Classical Learning. In Advances in Web Based Learning – ICWL 2009, LNCS 5686, Springer, 2009, pp. 386-389 (ISI Web of Knowledge)• Rebedea T., Trausan-Matu S. - Computer-Assisted Evaluation of CSCL Chat Conversations, CSCL 2009 (poster) (ISI Web of Knowledge, Category A in ARC Conference Ranking)• Dessus P., Trausan-Matu S., Zampa V., Rebedea T., Dascalu M. - Vers un environnement-tuteur d’apprentissage dialogique, 2e Colloque Echanger pour Apprendre en Ligne (EPAL’09), Grenoble : France, http://hal.archives-ouvertes.fr/hal-00404842/fr/• Trausan-Matu S., Rebedea T. - Ontology-Based Analyze of Chat Conversations. An Urban Development Case, Towntology Conference 2009. In Teller (ed.) Proceedings of Towntology Conference, Liege, 2009• Rebedea T., Trausan-Matu S., Chiru C.G. - Extraction of Socio-semantic Data from Chat Conversations in Collaborative Learning Communities, EC-TEL 2008. In P. Dillenbourg and M. Specht (Eds.): EC-TEL 2008, LNCS 5192, Springer, 2008, pp. 366–377 (ISI Web of Knowledge)• Rebedea T., Trausan-Matu S. - Autonomous News Clustering and Classification for an Intelligent Web Portal, ISMIS 2008. In A. An et al. (Eds.): ISMIS 2008, LNAI 4994, Springer, 2008, pp. 477–486 (ISI Web of Knowledge, Category C in ARC Conference Ranking)• Trausan–Matu S., Rebedea T., Stahl G. - Polyphonic Analysis of CSCL Chats, ICLS Workshop: A Common Framework for CSCL Interaction Analysis, Utrecht, The Netherlands, 200814 December 2012,• Trausan-Matu S., Rebedea T., Dragan A., Alexandru C. - Visualisation of Learners’ Contributions in Chat Conversations, Workshop on Blended Learning at ICWL 2007. and Feedback for PhD Defense - Computer-Based Support In Fong., J., Wang, P.BucharestBlended Learning, Pearson Prentice Hall, 2007, pp. 215-224 (Eds.), Romania Collaborative Chat Conversations and Discussion Forums
    68. 68. Publications (2)• Book Chapters• Rebedea T., Trausan-Matu S., Chiru C. - Inter-animation between Utterances in Collaborative Chat Conversations, submitted for Innovative Methods for Electronic Discourse Analysis, IGI Global, 2013, to appear• Trausan-Matu S., Rebedea T., Dascalu M. - Analysis of discourse in collaborative Learning Chat Conversations with Multiple Participants, ỉn Tufis D., Forascu C. (eds.), Multilinguality and Interoperability in Language Processing with Emphasis on Romanian, Editura Academiei, 2010• Trausan-Matu S., Rebedea T. - Polyphonic Inter-Animation of Voices in VMT, in Stahl G. (ed.), Studying Virtual Math Teams, Springer, 2009• Rebedea T., Trausan-Matu S. - Identification of Discussion Threads in Chat Conversations, in Trausan-Matu S. (ed.), Conversational Analysis in Online Collaborative Systems, Matrix-Rom, 2008 (in Romanian)• More than 20 papers co-authored in international workshops, Romanian conference proceedings and journals• 11 deliverables co-authored within the FP7 ICT Language Technologies for Lifelong Learning14 December 2012, PhD Defense - Computer-Based Support and Feedback for (LTfLL) projectBucharest Romania Collaborative Chat Conversations and Discussion Forums
    69. 69. THANK YOU! Questions & Feedback14 December 2012, PhD Defense - Computer-Based Support and Feedback forBucharest Romania Collaborative Chat Conversations and Discussion Forums

    ×