The slides present a model and an application that can be used to assess chat conversations according to their content, which is related to a number of imposed topics, and to the personal involvement of the participants. The main theoretical ideas that stand behind this application are Bakhtin’s polyphony theory and Tannen’s ideas related to the use of repetitions. The results of the application are validated against the gold standard provided by two teachers from the Human-Computer Interaction evaluating the same chats and after that the verification is done using another teacher from the same domain. During the verification we also show that the model used for chat evaluation is dependent on the number of participants to that chat

1. Autor Conducător științific
Universitatea
Politehnica
București
Facultatea de
Automatică și
Calculatoare
Catedra de
Calculatoare
Repetition and Rhythmicity Based
Assessment Model for Chat
Conversations
Costin-Gabriel Chiru - costin.chiru@cs.pub.ro
Valentin Cojocaru
Stefan Trausan-Matu
Traian Rebedea
Dan Mihaila

2. Contents
• The Problem and Our Solution
• Insights about Repetition
• The Application
• Extracted Information
• Training Corpus
• Validation and Verification
• Conclusions
30.06.2011 ISMIS 2011 2

3. The Problem
• Instant messaging systems (chats), blogs, forums,
etc. are intensely used for informal talks in our
spare time.
• They are suitable for collaborative knowledge
building: “many people prefer to view learning as
becoming a participant in a certain discourse”.
• These tools do not provide analysis facilities  a
lot of collaborative conversations that cannot be
assessed.
30.06.2011 ISMIS 2011 3

4. The System
• Purpose: Evaluate the quality of the whole
conversation considering participants’
involvement in the conversation and the
effectiveness of the conversation regarding some
given key-concepts.
• The system starts from Bakhtin’s polyphony
theory. (Bakhtin, 1984)
• By voice we understand either a participant to
the chat, or an idea (inspired by Tannen’s ideas
(1989): repetitions as a measure of involvement).
30.06.2011 ISMIS 2011 4

5. More about Repetition
• “Dialogue combines with repetition to create rhythm.” (Tannen,
1989)
• Four major functions of repetitions in conversations (Tannen,
1989):
– Production: repetition allows for better prepared discourse.
– Comprehension: discourse not so dense; only important elements are
repeated.
– Connection: linking mechanism for connecting the phrases from the
text.
– Interaction: connects the participants; the author is trying to induce
the same feelings in the audience.
• The establishment of interpersonal involvement: participating in a
conversation, one shows his/her response along with his/her
attitude to what others have said.
30.06.2011 ISMIS 2011 5

6. The Application
30.06.2011 ISMIS 2011 6

7. Extracted Information (1)
• How interesting is the conversation for the users - the
number of a user’s replies (more interesting ↔ more
replies).
• Persistence of the users - the total number of the user’s
consecutive replies (penalize monologue).
• Explicit connections between the users’ words – explicit
references made by the participants (ConcertChat).
• Activity of a user - the average number of uttered
characters per reply for that user (favor elaborated
answers to maximize the chance of encountering important
concepts).
30.06.2011 ISMIS 2011 7

8. Extracted Information (2)
• Absence of a user from the conversation - the average time
between a user’s consecutive replies (penalize absence from the
discussion).
• On topic - to what degree the participants debated the imposed
topics (penalize off-topic).
• Repetition - how often a participant repeats the concepts
introduced by others (salience of not-preferred concepts).
• Usefulness of a user - how often the concepts launched by a user
have been used by a different participant (rank participants).
• Topic rhythmicity - the number of replies between two consecutive
occurrences of the same topic (penalize off-topic).
30.06.2011 ISMIS 2011 8

9. Extracted Information (3)
Graphical representation of topic’s rhythmicity
30.06.2011 ISMIS 2011 9
High rhythmicity for
all topics –
these were debated
in parallel as it can
be seen by the lack
of flat lines near the
left side of the
representation.
Low rhythmicity – flat
lines on the left side
of the graphic
showing that the
topic that they
represent has not
been debated in
those parts of the
chat.

10. Training Corpus
• 6 chats consisting of 1886 replies.
• Undergraduate students in the senior year involved in the
HCI course divided in small groups of 4-5 students.
• Every participant had to present a web-collaboration
platform (chat, forum, blog, wiki) and prove its advantages
over the ones chosen by the other participants.
• Chats purpose: to facilitate the understanding of the pros
and cons of all the given platforms and to find the best
communication and collaboration technologies to be used
by a company to support team work.
30.06.2011 ISMIS 2011 10

11. Validation (1)
• Made by 2 HCI teachers, considering 2 criteria: the
quality of the content related to the given concepts
and the participants’ involvement.
• The application’s grades are much smaller than the
ones provided by the reviewers and therefore we
increased these grades by the average of the
difference between the reviewers’ grades and the
scores provided by the application.
30.06.2011 ISMIS 2011 11
Chat Reviewer 1 Reviewer 2 Average Application score Modified app.
Score
Chat 1 7.8 7.74 7.77 2.72 7.08
Chat 2 10 9.3 9.65 7.25 10
Chat 3 9 8.9 8.95 3.44 7.8
Chat 4 8.4 8.6 8.5 4.1 8.46
Chat 5 10 9 9.5 5.08 9.44
Chat 6 9 9 9 4.62 8.98

12. Validation (2)
• Correlation:
– between reviewers: 0.8829;
– between the reviewers’ average grades and the
scores provided by the application: 0.8389.
30.06.2011 ISMIS 2011 12

13. Verification (1)
Two different verification methods for our application:
1) Model verification - meant to demonstrate that the model
depends very much on the number of participants - 4 chats
consisting of 1250 replies that had between 6 and 8
participants similar to the previous ones.
30.06.2011 ISMIS 2011 13
Name of test Chat 7 Chat 8 Chat 9 Chat 10 Min 6-8 Max 6-8 Min 4-5 Max 4-5
0. Utterance number 138 380 473 259 138 473 176 559
1. Most interested 0.31 0.27 0.21 0.19 0.19 0.31 0.3 0.4
2. Least interested 0.06 0.04 0.07 0.02 0.02 0.07 0.08 0.18
3. Users persistence 0.3 0.02 0.004 0.003 0.003 0.3 0.07 0.31
4. Explicit connections 1.04 1.01 1.01 1.03 1.01 1.04 0.19 0.79
5. Absence 0.12 0.058 0.028 0.134 0.134 0.028 0.011 0.047
6. Minimum activity 1 2 16 1 1 16 6 52
7. Maximum activity 21 137 90 46 21 137 48 345
8. On topic - lex. chain 0.2 0.19 0.29 0.34 0.19 0.34 0.19 0.28
9. Repetitions (min) 0.07 0.03 0.06 0.06 0.03 0.07 0.08 0.15
10. Repetitions (max) 0.13 0.09 0.09 0.13 0.09 0.13 0.12 0.18
11. Least useful user 2.77 3.53 4.24 4.09 2.77 4.09 1.73 2.69
12. Most useful user 3.39 4.96 5.1 4.81 3.39 5.1 2.12 2.95
13. Topic rhythmicity 1.54 1.46 0.51 0.69 0.51 1.46 0.76 1.69

14. Verification (2)
In order to prove that the chats were similar to the previously
seen ones, we modified them by considering the persons
debating the same topics as being a single participant and
thus we ended up having again chats with 4 participants.
They were evaluated and the results showed that they fit
well enough in the model with only 4-5 participants.
30.06.2011 ISMIS 2011 14
Test No. Mod 7 Mod 8 Mod 9 Mod 10 Min mod Max
mod
Min 6-8 Max 6-8 Min 4-5 Max 4-5
Test 1 0.42 0.35 0.28 0.3 0.28 0.42 0.19 0.31 0.3 0.4
Test 2 0.13 0.13 0.22 0.19 0.13 0.22 0.02 0.07 0.08 0.18
Test 3 0.17 0.1 0.05 0.02 0.02 0.17 0.003 0.3 0.07 0.31
Test 4 1.02 1.01 1 1.01 1 1.02 1.01 1.04 0.19 0.79
Test 5 0.055 0.018 0.009 0.02 0.009 0.055 0.134 0.028 0.011 0.047
Test 6 4 30 106 60 4 106 1 16 6 52
Test 7 41 232 188 115 41 232 21 137 48 345
Test 8 0.2 0.2 0.29 0.34 0.2 0.34 0.19 0.34 0.19 0.28
Test 9 0.13 0.1 0.11 0.17 0.5 1.53 0.03 0.07 0.08 0.15
Test 10 0.18 0.136 0.12 0.19 0.12 0.19 0.09 0.13 0.12 0.18
Test 11 1.88 1.98 2.01 1.91 0.1 0.17 2.77 4.09 1.73 2.69
Test 12 2.14 2.36 2.38 2.14 1.88 2.01 3.39 5.1 2.12 2.95
Test 13 1.53 1.49 0.5 0.68 2.14 2.38 0.51 1.46 0.76 1.69

15. Verification (3)
30.06.2011 ISMIS 2011 15
Chat Reviewer Application Modified application score
Chat 11 9.627 5.24 9.6
Chat 12 7.574 4.76 9.12
Chat 13 8.777 5.39 9.75
• Second verification method: 3 chats consisting of
911 replies, from the same set with the ones used
for learning and validation (4 participants debating
about chat, forum, blog and wiki), were manually
evaluated by another teacher of the HCI class. The
results were compared against the application’s
ones.
• The correlation between the reviewer and the
application’s grades: 0.7933.

16. Conclusions
• We have presented an application that evaluates the quality of a
chat according to a number of predefined conversation topics and
to the personal involvement.
• The good correlation between the application and the domain
experts obtained at both the validation and verification stages
recommends it as a reliable application.
• The models that are used to evaluate the chats are dependent on
the number of participants (2-3, 4-5,6-8. 8+ participants).
• The large number of tests, gives a lot of flexibility to the user,
allowing him/her to give more or less importance to some of the
tests and therefore to evaluate exactly the aspects considered to be
important.
30.06.2011 ISMIS 2011 16

17. Q&A
Thank you for your time!
30.06.2011 ISMIS 2011 17