Do We Click? Speaker-listener
        neural coupling underlies
       successful communication
 Lauren Silbert1, Greg Ste...
Verbal communication is a joint activity through
      which individuals share information




              Listener     ...
Communication is integral to the functioning of
          our complex societies




 Globalization and the rapidly changin...
Verbal communication is a joint activity through
      which individuals share information




              Listener     ...
Neurolinguistic studies constrained by
   boundaries of individual brains




         Listener        Speaker
     (compr...
Neurolinguistic studies constrained by
   boundaries of individual brains




         Listener        Speaker
     (compr...
Assessing an interactive process:
laboratory verse real-world stimuli

          Control simplified stimuli

             ...
Assessing an interactive process:
laboratory verse real-world stimuli

          Control simplified stimuli

             ...
Design

     Speaker



  15 min monologue




               (real life unrehearsed story)
FOMRI™ II Dual Channel MRI Microphone System
                           Mic 1

                          Mic 2


         ...
Design

     Speaker                                   Listener



  15 min monologue
                                    ...
Brain activity during production as a model for
                comprehension
 model




                                 ...
Brain activity during production as a model for
                comprehension
 model




                                 ...
Brain activity during production as a model for
                comprehension
 model                       Listener’s resp...
Brain activity during production as a model for
                comprehension


•  Bypass the need to specify a priori any...
Adding Temporal
Dynamics to the Model:




                         (n=11)
Adding Temporal
Dynamics to the Model:




                         (n=11)
Adding Temporal
Dynamics to the Model:




                         (n=11)
Adding Temporal
Dynamics to the Model:




                         (n=11)
Adding Temporal
Dynamics to the Model:




                         (n=11)
Adding Temporal
Dynamics to the Model:




                         (n=11)
Adding Temporal
Dynamics to the Model:




                         (n=11)
Adding Temporal
Dynamics to the Model:




                         (n=11)
Adding Temporal
Dynamics to the Model:




                         (n=11)
Conservative Statistics: Parametric and Non-parametric
                                  Correct for
  Analytic F-test app...
Predictions




1.  Are neural responses during speech production
    and speech comprehension coupled?

2.  Can the exten...
Predictions




1.  Are neural responses during speech production
    and speech comprehension coupled?

2.  Can the exten...
Brain responses are reliably shared across all
                  listeners during comprehension
response




             ...
Speaker’s brain responses during production are
coupled to listener’s responses during comprehension
Areas involved in comprehension overlap with areas
          coupled during communication
Necessary controls


I.    Is the speaker-listener coupling tied to the
       content of the story?
         •  Block Com...
Speaker-Listener neural coupling is absent in the
    absence of successful communication




                            ...
Speaker-Listener neural coupling is absent when
        communication is incongruous




                                 ...
Necessary controls


I.    Is the speaker-listener coupling tied to the
       content of the story?
         •  Block Com...
Brain responses among listeners are time-locked to
            the moment of vocalization

   Average beta weights
Listener’s brain responses mirror the speaker’s
          brain responses with a delay

 Average beta weights
Brain responses shared among listeners are
temporally aligned to the moment of vocalization
Temporal dynamics of speaker-listener neural
       coupling varies across areas
Importance of temporal coupling results:

•  Dynamics of coupling between a speaker and a
listener are fundamentally diffe...
Predictions




1.  Are neural responses during speech production
    and speech comprehension coupled?

2.  Can the exten...
Level of understanding varies across listeners, as
    measured by reliable independent raters
The degree of neural coupling predicts the success
                of communication




        Speaker-listener Neural Co...
Areas where the listener’s responses precede the
speaker’s show strongest correlation with behavior
Summary


I.    In the course of communication the listener’s brain responses
       become coupled with the speaker’s bra...
Summary


I.     In the course of communication the listener’s brain responses
        become coupled with the speaker’s b...
Summary


I.     In the course of communication the listener’s brain responses
        become coupled with the speaker’s b...
Summary


I.     In the course of communication the listener’s brain responses
        become coupled with the speaker’s b...
Thank You
Christopher Honey   David Heeger
Yulia Lerner        Bruno Galantucci
Chris Thompson      Simon Garrod
Mina Cika...
mirror neurons
Do We Click? - Laurent Silbert - H+ Summit @ Harvard
Upcoming SlideShare
Loading in …5
×

Do We Click? - Laurent Silbert - H+ Summit @ Harvard

3,759 views

Published on

Do we click?
Speaker-listener neural coupling underlies successful communication

Verbal communication enables us to directly convey information across brains, independent of the actual external state of affairs (e.g. telling a story of past events). Such phenomenon may be reflected in the ability of the speaker to directly induce similar brain patterns in another individual, via speech, in the absence of any other stimulation. The recording of the neural responses from both the speaker brain and the listener brain opens a new window into the neural basis of interpersonal communication, and may be used to assess verbal and non-verbal forms of interaction in both human and other model systems. Further understanding of the neural processes that facilitate neural coupling across interlocutors may shed light on the mechanisms by which our brains interact and bind to form societies.

The capacity to communicate internal thoughts from one person to another is at the foundation of human society. Communication naturally requires an interaction between at least two people. Existing neurolinguistic studies are concerned, however, either with speech production or with the comprehension of isolated words or sentences. Little is known, therefore, about the underlying neuronal mechanism that facilitates the transfer of information between two brains during communication.

Understanding the interaction between a speaker’s brain and a listener’s brain in the context of real-world communication requires the development of new experimental paradigms. Using function Magnetic Resonance Imaging (fMRI), we measured neural signals from two brains (a speaker and a listener) during a complex everyday communication. We then built a simple, interpretable model that leverages the dynamics of fMRI and uses the speaker’s brain responses as a model for predicting the brain responses within the listener. Our model reveals that during successful communication, the speaker and listener’s brains exhibit joint, temporally coupled, response patterns. Such speaker-listener neural coupling vanishes when participants fail to communicate (such as with different languages). The temporal nature of this speaker-listener coupling suggests that an ability to evoke similar brain patterns in another individual via speech may gate our communication abilities. Moreover, while in most areas the listeners’ brain responses mirror the speaker’s responses with a delay, some areas in the listeners’ brain exhibit predictive anticipatory responses. Finally, we found that the extent of the anticipatory neuronal coupling between interlocutors is predictive of communicative success.

Currently a PhD candidate in Neuroscience from Princeton University, Silbert also has a Bachelors from University of Pennsylvania (Biology and Photography), Masters in Neuroscience from Mt. Sinai School of Medicine, and Masters in Psychology from NYU.

  • Be the first to comment

Do We Click? - Laurent Silbert - H+ Summit @ Harvard

  1. 1. Do We Click? Speaker-listener neural coupling underlies successful communication Lauren Silbert1, Greg Stephens2,3, Uri Hasson1 1PrincetonNeuroscience Institute and Psychology Department, Princeton University, 2Lewis-Sigler Institute for Integrative Genomics, and 3 Joseph Henry Laboratories of Physics, Princeton University, Princeton, NJ
  2. 2. Verbal communication is a joint activity through which individuals share information Listener Speaker (comprehension) (production)
  3. 3. Communication is integral to the functioning of our complex societies Globalization and the rapidly changing world demand we increase our understanding of how communication succeeds
  4. 4. Verbal communication is a joint activity through which individuals share information Listener Speaker (comprehension) (production)
  5. 5. Neurolinguistic studies constrained by boundaries of individual brains Listener Speaker (comprehension) (production)
  6. 6. Neurolinguistic studies constrained by boundaries of individual brains Listener Speaker (comprehension) (production)
  7. 7. Assessing an interactive process: laboratory verse real-world stimuli Control simplified stimuli red cat bat Isolation from the environment
  8. 8. Assessing an interactive process: laboratory verse real-world stimuli Control simplified stimuli red Complex natural stimuli cat bat Two brains interacting Isolation from the environment
  9. 9. Design Speaker 15 min monologue (real life unrehearsed story)
  10. 10. FOMRI™ II Dual Channel MRI Microphone System Mic 1 Mic 2 Two orthogonal pressure gradient optical microphones Mic 1: Signal + noise Mic 2: noise Extracted signal
  11. 11. Design Speaker Listener 15 min monologue (n=11) (real life unrehearsed story)
  12. 12. Brain activity during production as a model for comprehension model (n=11)
  13. 13. Brain activity during production as a model for comprehension model (n=11)
  14. 14. Brain activity during production as a model for comprehension model Listener’s response (n=11)
  15. 15. Brain activity during production as a model for comprehension •  Bypass the need to specify a priori any formal model of linguistic processes in any given brain area •  Ability to look at similarity of responses during speech production and speech comprehension in the same area
  16. 16. Adding Temporal Dynamics to the Model: (n=11)
  17. 17. Adding Temporal Dynamics to the Model: (n=11)
  18. 18. Adding Temporal Dynamics to the Model: (n=11)
  19. 19. Adding Temporal Dynamics to the Model: (n=11)
  20. 20. Adding Temporal Dynamics to the Model: (n=11)
  21. 21. Adding Temporal Dynamics to the Model: (n=11)
  22. 22. Adding Temporal Dynamics to the Model: (n=11)
  23. 23. Adding Temporal Dynamics to the Model: (n=11)
  24. 24. Adding Temporal Dynamics to the Model: (n=11)
  25. 25. Conservative Statistics: Parametric and Non-parametric Correct for Analytic F-test applied multiple to overall model fit comparisons using FDR g=0.05 1000 Actual correlation coefficient (n=11)
  26. 26. Predictions 1.  Are neural responses during speech production and speech comprehension coupled? 2.  Can the extent of speaker-listener neural coupling predict the success of communication?
  27. 27. Predictions 1.  Are neural responses during speech production and speech comprehension coupled? 2.  Can the extent of speaker-listener neural coupling predict the success of communication?
  28. 28. Brain responses are reliably shared across all listeners during comprehension response subject 4 f MRI subject 3 subject 2 subject 1 time
  29. 29. Speaker’s brain responses during production are coupled to listener’s responses during comprehension
  30. 30. Areas involved in comprehension overlap with areas coupled during communication
  31. 31. Necessary controls I.  Is the speaker-listener coupling tied to the content of the story? •  Block Communication •  Misaligned Communication
  32. 32. Speaker-Listener neural coupling is absent in the absence of successful communication ? X Russian Speaker Non-Russian speaking Listeners
  33. 33. Speaker-Listener neural coupling is absent when communication is incongruous ? X Speaker telling story 2 Listeners to story 1
  34. 34. Necessary controls I.  Is the speaker-listener coupling tied to the content of the story? •  Block Communication •  Shuffle Communication II. Is the speaker a listener of herself? •  Temporal dynamics
  35. 35. Brain responses among listeners are time-locked to the moment of vocalization Average beta weights
  36. 36. Listener’s brain responses mirror the speaker’s brain responses with a delay Average beta weights
  37. 37. Brain responses shared among listeners are temporally aligned to the moment of vocalization
  38. 38. Temporal dynamics of speaker-listener neural coupling varies across areas
  39. 39. Importance of temporal coupling results: •  Dynamics of coupling between a speaker and a listener are fundamentally different from dynamics shared among all listeners. vs • Spatial specificity of temporal coupling demonstrates effect cannot be attributed to non- specific, spatially global effects like arousal.
  40. 40. Predictions 1.  Are neural responses during speech production and speech comprehension coupled? 2.  Can the extent of speaker-listener neural coupling predict the success of communication?
  41. 41. Level of understanding varies across listeners, as measured by reliable independent raters
  42. 42. The degree of neural coupling predicts the success of communication Speaker-listener Neural Coupling
  43. 43. Areas where the listener’s responses precede the speaker’s show strongest correlation with behavior
  44. 44. Summary I.  In the course of communication the listener’s brain responses become coupled with the speaker’s brain responses.
  45. 45. Summary I.  In the course of communication the listener’s brain responses become coupled with the speaker’s brain responses. II.  The extent of speaker-listener neural coupling is indicative of the success of the communication.
  46. 46. Summary I.  In the course of communication the listener’s brain responses become coupled with the speaker’s brain responses. II.  The extent of speaker-listener neural coupling is indicative of the success of the communication. III.  An ability to evoke similar brain patterns in another individual via speech may gate our communication abilities.
  47. 47. Summary I.  In the course of communication the listener’s brain responses become coupled with the speaker’s brain responses. II.  The extent of speaker-listener neural coupling is indicative of the success of the communication. III.  An ability to evoke similar brain patterns in another individual via speech may gate our communication abilities. IV.  The recording of neural responses from both the speaker brain and the listener brain may be used to assess verbal and non- verbal forms of interaction in both human and other model systems.
  48. 48. Thank You Christopher Honey David Heeger Yulia Lerner Bruno Galantucci Chris Thompson Simon Garrod Mina Cikara Alana D'Alfonso
  49. 49. mirror neurons

×