An invited talk about potential multimodal indicators for assessing idea improvement in group knowledge building. Trieste, Italy. Early thoughts; comments welcomed.

1. Bodong Chen (@bodong_c)
Assistant Professor, University of Minnesota
September 9, 2015, KBSI2015, Trieste, Italy
Towards Multimodal Indicators
of Idea Improvement: A Review
of Possibilities

2. Idea Improvement in KB
▷ Why important?
▷ What is idea improvement?

3. Existing indicators
▷ Scientific sophistication (Zhang 2007)
▷ Levels of questioning (Law 2013)
▷ Lexical measures (Zhang & Sun 2011)
▷ Connectedness of concepts (Oshima 2013)
▷ ...

4. Multimodality
Multimodal interaction
▷ Human-Computer Interaction
▷ Multimodal Learning Analytics
Multiple modalities mediating
meaning-making
▷ Communication
▷ Linguistics
Goal of this presentation: Explore emerging possibilities
from other communities, with attention to both conceptions.

5. Two directions
Mining multimodal data in
existing KB environments
▷ Lexical
▷ Syntactic
▷ Graphic
Designing next-generation
multimodal KB environments
▷ Sociometrics
▷ Mobile sensing

6. 1. Mining multimodal data
in existing environments

7. 1.1. Lexical
▷ Text 1 – by a Grade 6 student: “My theory is solar eclipses are when the moon,
earth and sun are in a perfect line. Why solar eclipses don't happen every month is
because the moon's axis around the earth is slightly tilted. The moon needs to be so
close to the earth so that it covers the space of the sun. You have to be in a certain
spot in the earth to see a full solar eclipse but there is an arc the sun and moon make
across the earth but from other spots on the arc you would only see a partial eclipse.”
▷ Text 2 – from Wikipedia: “As seen from the Earth, a solar eclipse is a type of
eclipse that occurs when the Moon passes between the Sun and Earth, and the Moon
fully or partially blocks the Sun. This can happen only at new moon, when the Sun
and the Moon are in conjunction as seen from Earth in an alignment referred to as
syzygy. In a total eclipse, the disk of the Sun is fully obscured by the Moon. In partial
and annular eclipses, only part of the Sun is obscured.”

8. Text 1 Text 2
Sentence count, number of sentences 4 4
Word count, number of words 100 87
Sentence length, number of words, mean 25 21.75
Word length, number of letters, mean 3.75 4
Text Easability PC Narrativity, z score 0.54 0.08
WordNet verb overlap 0.61 0.30
LSA overlap, adjacent sentences, mean 0.56 0.63
Causal verb incidence 10 34.48

9. 1.2. Syntactic
Text 1 Text 2
Words before main verb, mean 3.5 7
Sentence syntax similarity, adjacent sentences, mean 0.09 0.24
Noun phrase density, incidence 320 333.33
Verb phrase density, incidence 220 172.41
Adverbial phrase density, incidence 10 45.98
Preposition phrase density, incidence 100 172.41

10. Xerox Incremental Parser
Source: CIC, UTS

11. 1.3. Graphic
▷ Layout of note
network / view
structure
low centralization; long path length
vs.
high centralization; short path length

12. x
y
1.3. Graphic
▷ View “landscape”
▷ Organizing
behaviors by
students

13. Other possibilities
▷ Affective states reflected in logs
(Baker 2010; McDaniel 2007; Pardos 2013)
▷ Certainty and identity in text
(Blikstein 2013)
▷ Sentiment in text (Chamlertwat 2012)

14. 2. Designing multimodal
knowledge building
environments

15. Motivations
▷ Pervasive knowledge building
▷ “One discourse, multiple entry points”
▷ Environments mediate knowledge
representation and meaning-making

16. 2.1. Sociometrics
(Onnela et al., 2014)
Physical proximity
(radio signal)
Speech time
(microphone)
Body movement
(accelerometer)
Social intimacy
Talkativeness
Turn-taking
Personality
Energy level

17. 2.2. Mobile & mobile sensing
▷ Multimedia
○ Citizen science (Riesch, 2014)
▷ Multimodal sensors
▷ Context awareness
○ Location
○ Surroundings
○ Proximity
▷ Augmented reality

18. Towards a holistic view
▷ “The Shadow Rabbit”
▷ Bridge different modalities
(Shum, 2015)

19. Scatterplot of reading and writing activities in six classes

20. Towards a holistic view
▷ “The Shadow Rabbit”
▷ Bridge different modalities
▷ Support multimodal
representations of ideas
(Shum, 2015)

21. “Thank you!
Slides: https://goo.gl/pRxuV9
Email: chenbd@umn.edu
Twitter: @bodong_c