• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Kinect-taped communication: Using motion sensing to study gesture use and similarity in face-to-face and computer-mediated brainstorming
 

Kinect-taped communication: Using motion sensing to study gesture use and similarity in face-to-face and computer-mediated brainstorming

on

  • 156 views

Presentation at ACM SIGCHI Conference on Human Factors in Computing Systems (CHI) 2014. See http://dl.acm.org/citation.cfm?doid=2556288.2557060 for the full paper.

Presentation at ACM SIGCHI Conference on Human Factors in Computing Systems (CHI) 2014. See http://dl.acm.org/citation.cfm?doid=2556288.2557060 for the full paper.

Statistics

Views

Total Views
156
Views on SlideShare
156
Embed Views
0

Actions

Likes
2
Downloads
2
Comments
0

0 Embeds 0

No embeds

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Kinect-taped communication: Using motion sensing to study gesture use and similarity in face-to-face and computer-mediated brainstorming Kinect-taped communication: Using motion sensing to study gesture use and similarity in face-to-face and computer-mediated brainstorming Presentation Transcript

    • Kinect-taped Communication:  Using Motion Sensing to Study Gesture Use  and Similarity in Face-to-Face and  Computer-Mediated Brainstorming Hao-Chuan Wang, Chien-Tung Lai National Tsing Hua University, Taiwan
    • [cf.  Bos  et  al.,  2002;  Setlock  et  al.,  2004;  Scissors  et  al.,  2008,  Wang  et  al.,  2009] Computer-mediated communication (CMC) tools are prevalent, but are they all equal? •  Ex. Video vs. Audio Media properties influence aspects of communication differently •  Task performance, grounding, styles, similarity of language patterns, social processes and outcomes etc. How media influence communication?
    • Communication could be more than speaking. Both verbal and non-verbal channels are active during conversations. Facial  expression Gesture [cf.  Goldin-­‐Meadow,  1999;  Giles  &  Coupland,  1991  ] The (missing) non-verbal aspect in CMC research
    • Studying gesture use in communication Current methods: •  Videotaping with manual coding. •  Giving specific instructions to participants (e.g., to gesture or not). •  Using confederates etc. Problems to solve: •  High cost. Labor-intensiveness. •  Resolution of manual analysis- Hard to recognize and reliably label small movements. •  Scalability- Hard to study arbitrary communication in the wild.
    • “Kinect-taping”method Like videotaping, we use motion sensing devices, such as Microsoft Kinect, to record hand and body movements during conversations. •  Detailed, easier-to-process representations. •  Behavioral science instrument (“microscope”) to study non-verbal communication in ad hoc groups. •  Low cost if automatic measures are satisfactory.
    • Re-appropriating motion sensors in HCI: Sensing-aided user research for  future designs From sensors as design elements to sensors as research instruments to help future designs. ! (a)!Face(to(face!(F2F)!communication! (b [cf.  Mark  et  al.,  2014]
    • A media comparison study Investigate how people use gestures during face-to- face and computer-mediated brainstorming Compare three communication media •  Face-to-Face •  Video •  Audio ! (a)!Face(to(face!(F2F)!communication! ! (b)!Video(mediated!communication! Figure'1.'A'sample'study'setting'that'compares'(a)'F2F'to'(b)'video<mediated'communication' by'using'Kinect'as'a'behavioral'science'instrument.'
    • Hypotheses H1. Visibility increases gesture use Proportion of gesture Face-to-Face > Video > Audio H2. Visibility increases accommodation Similarity between group members’ gestures Face-to-Face > Video > Audio Also explore how gesture use, level of understanding, and ideation productivity correlate. [cf.  Clark  &  Brennan,  1991] [cf.  Giles  &  Coupland,  1991]
    • Experimental design 36 individuals, 18 two-person groups Kinect-taped group brainstorming sessions Face-to-Face Video Audio Three  trials  (15  min  each)     in  counterbalanced  order   Data analysis Amount and similarity of gestures, Level of understanding, Productivity
    • How to quantify gestures? How many gestures are there in a 15 min talk?
    • moving not moving
    • Two unit motions with speed threshold 0
    • Three unit motions with speed threshold 2
    • Choose the thresholds (m/s)
    • Choose the thresholds Too  few  signals Almost  everything Data  points  of  interest (m/s)
    • How to measure similarity between unit motions?
    • Feature extraction and representation Unit motions are represented as feature vectors •  Time length, path length, displacement, velocity, speed, angular movement etc. •  Features extracted for both hands and both elbows. 73 features extracted for each unit motion. Similarity between unit motions: Cosine value between the two vectors.
    • Validating the similarity metric 1 2 3 Machine Ranking Human Ranking 1 2 3 Randomly select motion queries Retrieve similar and dissimilar motions Kinect-taped motion database
    • Count Human Rank R1 R2 R3 Machine Rank R1 29 2 5 R2 7 27 2 R3 0 7 29 x2=107.97,  p<.001 Validating the similarity metric Contingency analysis
    • H1: Amount of gesture use H2: Similarity between group members Associations •  Amount of gesture and understanding •  Amount of gesture and ideation productivity •  Gesture similarity and ideation productivity Key Results
    • Visibility on proportion of gesture use 0 2 4 6 8 10 12 14 16 Face-to-face Video Audio ProportionofGestureUse(%) H1 not supported. Media did not influence percentage of gesture. People gesture as much in Audio as in F2F and Video.
    • Association between self-gesture and level of understanding Model&Predicted,UnderstandingModel&Predicted,Num Propor9on,of,Individual’s,Own,Gesture,Use,(%) Audio F2F Video Individual’s Own Gesture Use (%) Non-communicative function of gesture. Understanding correlates with self-gesture but not partner-gesture Stronger correlation with reduced or no visibility.
    • Similarity between group members 0.46 0.47 0.48 0.49 0.5 0.51 0.52 0.53 0.54 0.55 Face-to-face Video Audio Between-participantGestural Similarity H2 supported. Similarity F2F > Video > Audio. People gesture more similarly when they can see each other.
    • Summary and implications   Media   Comparison     Study Kinect- taping Method
    • Motion sensing for studying non-verbal behaviors in CMC. Summary and implications   Media   Comparison     Study Kinect- taping Method Visibility influences similarity but not amount of gesture. Only self-gesture correlates with understanding. Gesture doesn’t seem to convey much meaning to the partner. Seeing the partner is not crucial to understanding.
    • Study communication of ad hoc groups in the wild. Distributed deployment study of CMC tools. Cross-lingual and cross-cultural communication. Summary and implications (cont.)   Media   Comparison     Study Kinect- taping Method The value of video may be relatively limited to the social and collaborative aspect (similarity etc.). Feedback that promotes self- gesturing may help understanding.
    • Microsoft Research Asia (UR FY13-RES-OPP-027) Ministry of Science and Technology, Taiwan (NSC 102-2221-E-007-073-MY3) Contact: Hao-Chuan Wang ⺩王浩全 haochan@cs.nthu.edu.tw Acknowledgement