Successfully reported this slideshow.
Università degli studi di Bari “Aldo Moro”                  Dipartimento di Informatica Recognising the Social Attitude in...
Overview           ●    Introduction           ●    Objective           ●    The proposed model           ●    The propose...
Introduction   Pedagogical Conversational Agent (PCA)   ●   fulfil pedagogical goals   ●   interact with the user through ...
Objective   Aim: building a multimodal framework for the recognition   of the social response of users to a PCA.   In part...
Objective                                  The intuition    The combination of these different input modalities may    imp...
The proposed model    Dynamic Belief Network (DBN):    ●   handling uncertainty and incompleteness of data    ●   represen...
The proposed approach  ●   The model is initialized  ●   At every dialog step:      ●   Knowledge about the evidence is pr...
Signs of social attitude                        in the language   ●   Sense of intimacy (use of common jargon)        ●   ...
Signs of social attitude                         in the prosody    Praat functions to extract features related to:    ●   ...
Signs of social attitude                         in the gestures                      Gesture recognition performed using ...
Signs of social attitude                         in the gestures                                    Signal              Po...
Evaluation                        Collecting a corpus    We collected moltimodal dialog moves, consisting in    linguistic...
Evaluation                        Collecting a corpusRecognising the Social Attitude in Natural Interaction with Pedagogic...
Evaluation                                          Results                       Move U6                                 ...
Conclusions    Existing model for recognising social attitude enriched    with the analysis of signals regarding non-verba...
Future works    ●   Improving gesture recognition since the new Kinect        should allow for a better hand recognition  ...
Upcoming SlideShare
Loading in …5
×

Recognising the Social Attitude in Natural Interaction with Pedagogical Agents

481 views

Published on

Pedagogical Conversational Agents (PCAs) have the advantage of offering to students not only task-oriented support but also the possibility to interact with the computer media at a social level. This form of intelligence is particularly important when the character is employed in an educational setting. This paper reports our initial results on the recognition of users' social response to a pedagogical agent from the linguistic, acoustic and gestural analysis of the student communicative act.

Published in: Technology, Education
  • Be the first to comment

  • Be the first to like this

Recognising the Social Attitude in Natural Interaction with Pedagogical Agents

  1. 1. Università degli studi di Bari “Aldo Moro” Dipartimento di Informatica Recognising the Social Attitude in Natural Interaction with Pedagogical AgentsB. De Carolis, S. Ferilli, N. Novielli, F. Leuzzi, F. Rotella {decarolis, ferilli, novielli}@di.uniba.it, {fabio.leuzzi, fulvio.rotella}@uniba.it DIDAMATICA, Informatica per la Didattica Taranto, Italy, May 14-16, 2012
  2. 2. Overview ● Introduction ● Objective ● The proposed model ● The proposed approach ● Signs of social attitude ● Evaluation ● Conclusions ● Future worksRecognising the Social Attitude in Natural Interaction with Pedagogical AgentsB. De Carolis, S. Ferilli, N. Novielli, F. Leuzzi, F. Rotella 2
  3. 3. Introduction Pedagogical Conversational Agent (PCA) ● fulfil pedagogical goals ● interact with the user through a natural dialog by appropriately mixing verbal and non verbal expressions: ● recognize verbal and non-verbal inputs ● generate verbal and non-verbal outputs ● handle typical functions of human conversations, with particular emphasis on social aspectsRecognising the Social Attitude in Natural Interaction with Pedagogical AgentsB. De Carolis, S. Ferilli, N. Novielli, F. Leuzzi, F. Rotella 3
  4. 4. Objective Aim: building a multimodal framework for the recognition of the social response of users to a PCA. In particular: building a framework that integrates the analysis of the linguistic component of the users communicative act with the analysis of the acoustic features of the spoken sentence and of the gestures.Recognising the Social Attitude in Natural Interaction with Pedagogical AgentsB. De Carolis, S. Ferilli, N. Novielli, F. Leuzzi, F. Rotella 4
  5. 5. Objective The intuition The combination of these different input modalities may improve the recognition of multimodal behaviours that may denote the openness attitude of the users towards the embodied agent. Steps: ● Recognize signs of social attitude ● Build a model to infer the user attitude toward the PCA ● Adapt the dialog strategies accordinglyRecognising the Social Attitude in Natural Interaction with Pedagogical AgentsB. De Carolis, S. Ferilli, N. Novielli, F. Leuzzi, F. Rotella 5
  6. 6. The proposed model Dynamic Belief Network (DBN): ● handling uncertainty and incompleteness of data ● representing situations which gradually evolve from a dialog step to the next oneRecognising the Social Attitude in Natural Interaction with Pedagogical AgentsB. De Carolis, S. Ferilli, N. Novielli, F. Leuzzi, F. Rotella 6
  7. 7. The proposed approach ● The model is initialized ● At every dialog step: ● Knowledge about the evidence is produced ● The produced knowledge is entered and propagated in the network ● The model revises the probabilities of the social attitude node ● The new probabilities of the signs of social attitude are used for planning the next agent move ● The probability of the social attitude node supports revising high-level planning of the agent behaviourRecognising the Social Attitude in Natural Interaction with Pedagogical AgentsB. De Carolis, S. Ferilli, N. Novielli, F. Leuzzi, F. Rotella 7
  8. 8. Signs of social attitude in the language ● Sense of intimacy (use of common jargon) ● Friendly self-introduction ● Familiar style ● Attempt to establish a common ground ● Talk about self ● Personal questions about the agent ● Irony and humour ● Benevolent/polemic attitude towards the system failures ● Favourable/negative comments ● Interest to protract interaction ● Friendly farewellRecognising the Social Attitude in Natural Interaction with Pedagogical AgentsB. De Carolis, S. Ferilli, N. Novielli, F. Leuzzi, F. Rotella 8
  9. 9. Signs of social attitude in the prosody Praat functions to extract features related to: ● variation of the fundamental frequency ● variation of energy ● variation of harmonicity ● Spectrum Central Moment, Standard Deviation, Gravity centre, Skeweness and Kurtosis ● speech rate Classification (using NNge algorithm) of users spoken sentence into 3 classes: positive, negative and neutral.Recognising the Social Attitude in Natural Interaction with Pedagogical AgentsB. De Carolis, S. Ferilli, N. Novielli, F. Leuzzi, F. Rotella 9
  10. 10. Signs of social attitude in the gestures Gesture recognition performed using Microsoft Kinect + KinectDTW need to consider only a subset of gestures compatible with the nodes in the skeleton that the Kinect SDK can detect.Recognising the Social Attitude in Natural Interaction with Pedagogical AgentsB. De Carolis, S. Ferilli, N. Novielli, F. Leuzzi, F. Rotella 10
  11. 11. Signs of social attitude in the gestures Signal Possible meaning(s) Crossed arms Defensiveness, closure Gripping own upper Insecurity, closure arms Adjusting cuff, Nervouseness, watchstrap, tic, using negative attitude an arm across the body touching or scratching Nervouseness, shoulder using arm negative attitude across bodyRecognising the Social Attitude in Natural Interaction with Pedagogical AgentsB. De Carolis, S. Ferilli, N. Novielli, F. Leuzzi, F. Rotella 11
  12. 12. Evaluation Collecting a corpus We collected moltimodal dialog moves, consisting in linguistic, acoustic and gesture data. Participants: 2 groups of 5 italian students aged between 16 and 25 (equally distributed by gender). Goal: getting information about a correct diet in order to stay in shape. PCA role: nutrition expert. Collected: about 300 moves.Recognising the Social Attitude in Natural Interaction with Pedagogical AgentsB. De Carolis, S. Ferilli, N. Novielli, F. Leuzzi, F. Rotella 12
  13. 13. Evaluation Collecting a corpusRecognising the Social Attitude in Natural Interaction with Pedagogical AgentsB. De Carolis, S. Ferilli, N. Novielli, F. Leuzzi, F. Rotella 13
  14. 14. Evaluation Results Move U6 Move U7 Move U8Recognising the Social Attitude in Natural Interaction with Pedagogical AgentsB. De Carolis, S. Ferilli, N. Novielli, F. Leuzzi, F. Rotella 14
  15. 15. Conclusions Existing model for recognising social attitude enriched with the analysis of signals regarding non-verbal communication: prosody and gesture. We propose an extension of the multimodal analysis to gesture modeling, according to the meanings that psycholinguistic researchers attach to gestures in conversations. Preliminary experiments show promising results.Recognising the Social Attitude in Natural Interaction with Pedagogical AgentsB. De Carolis, S. Ferilli, N. Novielli, F. Leuzzi, F. Rotella 15
  16. 16. Future works ● Improving gesture recognition since the new Kinect should allow for a better hand recognition ● extending the social attitude analysis with facial expressions Carrying out more evaluation studies in order to test the robustness of our framework: ● for social attitude recognition in different scenarios ● with respect to different interaction modalities with both ECAs and RobotsRecognising the Social Attitude in Natural Interaction with Pedagogical AgentsB. De Carolis, S. Ferilli, N. Novielli, F. Leuzzi, F. Rotella 16

×