Music and the BrainStructured neural responses to natural stimuliPrinceton UniversityUri HassonPsychology department and t...
David HeegerMorwaread FarboodGary MarcusLab members CollaboratorsYulia LernerChris HoneyGreg StephensJanice ChenErez Simon...
Complex natural stimuliControl simplified stimuliSensory Coding and the Natural EnvironmentFew well characterized dimensio...
Complex natural stimuliControl simplified stimuliReal lifeComplexStructuredMovies/Stories/MusicSensory Coding and the Natu...
Memory formation of real world eventsHasson et al. Neuron 2008Neural responses to natural complex stimuliHasson et al. Sci...
SubjectsExperiment / measurements
Inter-subject correlationsTalairach coordinatesSubject 1/Run 1 Subject 2/Run 2
Idiosyncratic responsesSharedsignal Idiosyncratic signalIntra-SCInter-SC
Sharedsignal Idiosyncratic signalShared responsesIntra-SCInter-SC
Inter-subject correlationsHasson et al, Science, 2004
So I’m banging out my story and I know it’s good, and then I start to make it betterby adding an element of embellishment....
Early Auditory Cortexr = 0.55r = 0.72Individual subjectsAverage subjectBrain Responses to Real-Life Story“Pie Man”Story
PrecuneusEarly Auditory CortexAngular GyrusInferior Frontal Gyrusr = 0.66r = 0.6r = 0.72r = 0.41r = 0.56r = - 0.18r = 0.1B...
medial viewLeftHemisphereRightHemispherelateral view lateral viewA P P Ar=0.110.45N=11Brain Responses to Real-Life Story“P...
BrahmsPiano Concerto No.1 in Dminor	Brain Responses to Real-Life MusicA1M1M1STG STGA1BG/ThalamusBG/Thalamus
The extent of neural overlap between language-related and music-related processes
Intermediate summaryReal life stimuli, as movies, stories and music can exert considerable control overthe responses of ma...
How does the brain process such complex and richtemporal structures?
So I’m banging out my story and I know it’s good, and then I start to make it betterby adding an element of embellishment....
medial viewLeftHemisphereRightHemispherelateral view lateral viewA P P Ar=0.110.45N=11“Pie Man”StoryDifferent Processing T...
Scrambled pastWord WordpresentCoherent pastScrambled pastExtent of past information needed to evoke reliable responses in ...
WordsParametric variation of the temporal structure of a verbalmonologueParagraphsBackwardSentencesIntact storyTemporal ra...
A P PReverselateralmedialAP PLH RHq<0.05 (FDR)N = 11AInter-subject Correlation During a Narrated StoryWordsSentencesParagr...
medial viewLeftHemisphereRightHemispherelateral view lateral viewA P P AAP PN = 11sent paragrev wordsA Hierarchy of Proces...
A1+ TPJTemporo-Parietal AxisFS P S W Rcorrelationsent (S) parag (P)rev (R) words (W)FS P S W R FS P S W RN = 1112345FS P S...
auditory story+silent movieshort mid long overlap=Processing Time-Varying Information About the World
Bars PhrasesSectionsIntactShort temporal scales Mid temporal scales Long temporal scalesReversedParametric	  varia+on	  of...
Musical Temporal Receptive WindowsA1+
Musical vs. Linguistic based Temporal Receptive Windows
Musical vs. Linguistic based Temporal Receptive Windows
Word/barpresentShort temporal integration windowExtent of past information needed to evoke reliable responses in the prese...
Spatial Scale Temporal ScaleHubel & Wiesel (1959)J PhysiolGross et al (1972)J NeurophysiolUngerleider & Mishkin (1982)Anal...
Accumulating information over spaceand timeSpaceTimeElectrode in IT cortexTemporal Receptive Window in IT cortexA Hierarch...
Integrate Real-World InformationBaddeley & Hitch (1974)ClassicalWorking MemoryModelHierarchy of Temporal Receptive Windows...
The end
Is Integration Temporal or Ordinal?Essential Role of Time Essential Role of Semantic UnitsOr Both?See also Howard & Eichen...
Temporal units and the information units are easily dissociated in real-life speech100%75%50%150%200%Time
Yulia LernerOur speech perception is invariant to changes in rate
Is Integration Really Temporal or Just Ordinal?Essential Role of Time Essential Role of Semantic UnitsSpeech intelligibili...
Time NOWAccessibilitylow highTimescales of Information
Integrate Real-World InformationCowan (1999)EmbeddedProcesses ModelWorking Memory =“Activated”Memory TracesActivatedMemory...
Time vs. accumulation of information over time
Integrate Real-World InformationPersistent Neuronal ActivityActivatedMemoryCentralExecutiveLong Term MemoryAttendedFunahas...
HypothesisA Hierarchy of Timescales in Brain DynamicsHierarchy of Temporal Receptive Windows
Open QuestionsIs the hippocampus required to sustain the long temporal receptive windows?What happens in the hierarchy at ...
Lab QuestionsShould I shorten the title?Present larger questions at the beginning or at the end?Emphasize /time/ or mental...
Different Processing Timescales in Different Regions?Criterion Oneminimum prior duration of coherentinformation required f...
responses invariant to changesbeyond a maximum durationCriterion OneCriterion Twominimum amount of coherentinformation req...
Study of Naturalistic PerceptionCosts BenefitsPoorer experimental controlPoorer experimental control
Why unrelated materials?Why under conditions of distraction?Working Memory Experiments
Memory systems are organized to represent the real world.We may look into that window on the mind as through a glass darkl...
Stimulus locked Circuit Dynamics using BOLD
Mary Potterfast semanticsstabilization ideaISC?Van Dijk & Kintschunavaoidable semanticsthe log was on the tutrlteMcLelland...
Inter-subject Correlation during Movie ViewingSingle Subjects (N=9)Mean Timecoursemedial viewLeftHemisphereRightHemisphere...
Single Subjects (N=9)Mean Timecoursemedial viewLeftHemisphereRightHemispherelateral view lateral viewA P P Ar=0.150.55Inte...
Processing Time Scales and Working MemoryWorking memory is the“ability to keep a representation active,particularly in the...
Youssef Ezzyat, Lila Davachi (NYU): Neural mechanisms supporting the temporal organization of episodic long-termmemoryDisc...
“whole” “scrambled”“grid”Scrambling Objects in Spacecompletion no completionLerner 2003
“whole” “scrambled”“grid”12212211 2 1Lerner 2003Scrambling Objects in Space12
Uri Hasson - Neurocinematics: The Neuroscience of Film
Uri Hasson - Neurocinematics: The Neuroscience of Film
Uri Hasson - Neurocinematics: The Neuroscience of Film
Upcoming SlideShare
Loading in …5
×

Uri Hasson - Neurocinematics: The Neuroscience of Film

2,579 views

Published on

Published in: Technology, Health & Medicine

Uri Hasson - Neurocinematics: The Neuroscience of Film

  1. 1. Music and the BrainStructured neural responses to natural stimuliPrinceton UniversityUri HassonPsychology department and the Neuroscience Institute
  2. 2. David HeegerMorwaread FarboodGary MarcusLab members CollaboratorsYulia LernerChris HoneyGreg StephensJanice ChenErez SimonyLauren SilbertMor RegevWho did the work
  3. 3. Complex natural stimuliControl simplified stimuliSensory Coding and the Natural EnvironmentFew well characterized dimensionslinear properties, simple mathMultidimensionaland MessyParameterized Scientific investigation
  4. 4. Complex natural stimuliControl simplified stimuliReal lifeComplexStructuredMovies/Stories/MusicSensory Coding and the Natural Environment
  5. 5. Memory formation of real world eventsHasson et al. Neuron 2008Neural responses to natural complex stimuliHasson et al. Science 2004Scan paths of real life eventsHasson et al. In-pressDisruption of brain responses in autismHasson et al. Autism Research 2009Time scale of processingHasson et al. J neuroscience 2008Using movies as a research tool incognitive neuroscienceSocial communicationStephan et al. PNAS 2010
  6. 6. SubjectsExperiment / measurements
  7. 7. Inter-subject correlationsTalairach coordinatesSubject 1/Run 1 Subject 2/Run 2
  8. 8. Idiosyncratic responsesSharedsignal Idiosyncratic signalIntra-SCInter-SC
  9. 9. Sharedsignal Idiosyncratic signalShared responsesIntra-SCInter-SC
  10. 10. Inter-subject correlationsHasson et al, Science, 2004
  11. 11. So I’m banging out my story and I know it’s good, and then I start to make it betterby adding an element of embellishment. Reporters call this“making shit up”.And they recommend against crossing that line.But I had just seen the line crossed between a high-powered dean and an assault with apastry, and I kind of liked it.Real-Life Story Stimulus“Pie Man”Story
  12. 12. Early Auditory Cortexr = 0.55r = 0.72Individual subjectsAverage subjectBrain Responses to Real-Life Story“Pie Man”Story
  13. 13. PrecuneusEarly Auditory CortexAngular GyrusInferior Frontal Gyrusr = 0.66r = 0.6r = 0.72r = 0.41r = 0.56r = - 0.18r = 0.1Brain Responses to Real-Life Story“Pie Man”Story
  14. 14. medial viewLeftHemisphereRightHemispherelateral view lateral viewA P P Ar=0.110.45N=11Brain Responses to Real-Life Story“Pie Man”Story
  15. 15. BrahmsPiano Concerto No.1 in Dminor Brain Responses to Real-Life MusicA1M1M1STG STGA1BG/ThalamusBG/Thalamus
  16. 16. The extent of neural overlap between language-related and music-related processes
  17. 17. Intermediate summaryReal life stimuli, as movies, stories and music can exert considerable control overthe responses of many brain areas, evoking a similar time course of activity acrossall viewers.
  18. 18. How does the brain process such complex and richtemporal structures?
  19. 19. So I’m banging out my story and I know it’s good, and then I start to make it betterby adding an element of embellishment. Reporters call this“making shit up”.And they recommend against crossing that line.But I had just seen the line crossed between a high-powered dean and an assault with apastry, and I kind of liked it.1:43 1:45 1:471:50 1:541:582:050:55Time-scales of Information in a Narrated StorySo
  20. 20. medial viewLeftHemisphereRightHemispherelateral view lateral viewA P P Ar=0.110.45N=11“Pie Man”StoryDifferent Processing Timescales in Different Regions?
  21. 21. Scrambled pastWord WordpresentCoherent pastScrambled pastExtent of past information needed to evoke reliable responses in the presentEntire storyNeural responsesat the presentCoherent pastParagraph ParagraphpresentCoherent pastScrambled pastSentence SentencepresentCoherent pastScrambled pastpresentLongmemoryNomemoryIntermediate
  22. 22. WordsParametric variation of the temporal structure of a verbalmonologueParagraphsBackwardSentencesIntact storyTemporal rate is fixedEach 7 minutes condition iscomposed of the exact same basicunits
  23. 23. A P PReverselateralmedialAP PLH RHq<0.05 (FDR)N = 11AInter-subject Correlation During a Narrated StoryWordsSentencesParagraphs
  24. 24. medial viewLeftHemisphereRightHemispherelateral view lateral viewA P P AAP PN = 11sent paragrev wordsA Hierarchy of Processing Timescales
  25. 25. A1+ TPJTemporo-Parietal AxisFS P S W Rcorrelationsent (S) parag (P)rev (R) words (W)FS P S W R FS P S W RN = 1112345FS P S W RA Hierarchy of Processing Timescales
  26. 26. auditory story+silent movieshort mid long overlap=Processing Time-Varying Information About the World
  27. 27. Bars PhrasesSectionsIntactShort temporal scales Mid temporal scales Long temporal scalesReversedParametric  varia+on  of  the  coherent    temporal  structure  within  a  music  piece    Brahms    Piano  Concerto  No.1  in  D  minor
  28. 28. Musical Temporal Receptive WindowsA1+
  29. 29. Musical vs. Linguistic based Temporal Receptive Windows
  30. 30. Musical vs. Linguistic based Temporal Receptive Windows
  31. 31. Word/barpresentShort temporal integration windowExtent of past information needed to evoke reliable responses in the presentEntire story/musical piecepresentLong temporal integration windowParagraph/sectionpresentIntermediate temporal integration windowSentence/phrasepresentIntermediate temporal integration windowpresentLongmemoryNomemoryIntermediate
  32. 32. Spatial Scale Temporal ScaleHubel & Wiesel (1959)J PhysiolGross et al (1972)J NeurophysiolUngerleider & Mishkin (1982)Analysis of BehaviorFunctional HierarchyA Hierarchy of Temporal Receptive Windows
  33. 33. Accumulating information over spaceand timeSpaceTimeElectrode in IT cortexTemporal Receptive Window in IT cortexA Hierarchy of Temporal Receptive Windows
  34. 34. Integrate Real-World InformationBaddeley & Hitch (1974)ClassicalWorking MemoryModelHierarchy of Temporal Receptive WindowsLimited Capacity BottleneckProcessing Time Scales and Working MemoryMaintain Discrete Units of Information
  35. 35. The end
  36. 36. Is Integration Temporal or Ordinal?Essential Role of Time Essential Role of Semantic UnitsOr Both?See also Howard & Eichenbaum (in press) JEP Generalversus
  37. 37. Temporal units and the information units are easily dissociated in real-life speech100%75%50%150%200%Time
  38. 38. Yulia LernerOur speech perception is invariant to changes in rate
  39. 39. Is Integration Really Temporal or Just Ordinal?Essential Role of Time Essential Role of Semantic UnitsSpeech intelligibility recoveredby insertion of pausesRescaling of neural responsesthroughout the brainMemory for absolute tempoin musical sequencesIntegration of information over timemuch easier for meaningful speechNeurophysiology has intrinsic timescalesOr Both?Ghitza & Greenberg (2009)Lerner et al (submitted)See also Howard & Eichenbaum (in press) JEP General“Time”cells in hippocampusNaya & Suzuki (2011)Macdonald et al (2011)Behavioral invarianceto moderate changes in stimulus rateLevitin & Cook (1996)
  40. 40. Time NOWAccessibilitylow highTimescales of Information
  41. 41. Integrate Real-World InformationCowan (1999)EmbeddedProcesses ModelWorking Memory =“Activated”Memory TracesActivatedMemoryCentralExecutiveLong Term MemoryAttendedProcessing Time Scales and Working MemoryHebb (1949)ActivatedCell AssembliesHierarchy of Temporal Receptive Windows
  42. 42. Time vs. accumulation of information over time
  43. 43. Integrate Real-World InformationPersistent Neuronal ActivityActivatedMemoryCentralExecutiveLong Term MemoryAttendedFunahashi et al (1989)Gnadt & Anderson (1988)Goldman-Rakic (1996)Processing Time Scales and Working MemoryHierarchy of Temporal Receptive Windows
  44. 44. HypothesisA Hierarchy of Timescales in Brain DynamicsHierarchy of Temporal Receptive Windows
  45. 45. Open QuestionsIs the hippocampus required to sustain the long temporal receptive windows?What happens in the hierarchy at (macro & micro) event boundaries?c.f. Ranganath & Ritchie (2012) Nat Rev Neurosci
  46. 46. Lab QuestionsShould I shorten the title?Present larger questions at the beginning or at the end?Emphasize /time/ or mental contextNarrative style or argument style?
  47. 47. Different Processing Timescales in Different Regions?Criterion Oneminimum prior duration of coherentinformation required for a response
  48. 48. responses invariant to changesbeyond a maximum durationCriterion OneCriterion Twominimum amount of coherentinformation required for a responseDifferent Processing Timescales in Different Regions?equals
  49. 49. Study of Naturalistic PerceptionCosts BenefitsPoorer experimental controlPoorer experimental control
  50. 50. Why unrelated materials?Why under conditions of distraction?Working Memory Experiments
  51. 51. Memory systems are organized to represent the real world.We may look into that window on the mind as through a glass darkly,but what we are beginning to discern there looks very much like areflection of the world.Roger Shepard (1990) Mind SightsPerceptual systems are organized to represent the real world.Anderson & Schooler (1991) Psych ScienceBartlett (1932) RememberingNeisser (1978) Practical Aspects of Memory
  52. 52. Stimulus locked Circuit Dynamics using BOLD
  53. 53. Mary Potterfast semanticsstabilization ideaISC?Van Dijk & Kintschunavaoidable semanticsthe log was on the tutrlteMcLelland and RumelhartBransford and Johnsoneffects on memoryStabiliza
  54. 54. Inter-subject Correlation during Movie ViewingSingle Subjects (N=9)Mean Timecoursemedial viewLeftHemisphereRightHemispherelateral view lateral viewA P P Ar=0.150.55
  55. 55. Single Subjects (N=9)Mean Timecoursemedial viewLeftHemisphereRightHemispherelateral view lateral viewA P P Ar=0.150.55Inter-subject Correlation during Movie Viewing
  56. 56. Processing Time Scales and Working MemoryWorking memory is the“ability to keep a representation active,particularly in the face of interference and distraction”.Engle et al (1999) JEP:General
  57. 57. Youssef Ezzyat, Lila Davachi (NYU): Neural mechanisms supporting the temporal organization of episodic long-termmemoryDiscussant: Per Sederberg (Ohio State)Christopher J. Honey, Janice Chen, Erez Simony, Olga Lositsky, Daniel Toker, Kenneth A. Norman, UriHasson (Princeton): Temporal receptive windows in natural perception: a topographic map of mental contextDiscussant: Ryan Canolty (UC Berkeley)Gregory J. Koop, Amy H. Criss (Syracuse): Response dynamics as a measure of bias and strength in recognitionmemoryDiscussant: Adam Osth (Ohio State)Isabel A. Muzzio (Penn): Effects of emotion on hippocampal contextual representationsDiscussant: Sam Gershman (MIT)Robert M. Nosofsky, Christopher Donkin, Jason M. Gold, Richard M. Shiffrin (Indiana University): Discrete-slotsmodels of visual working memory response timesDiscussant: Michael Lee (UC Irvine)Sean M. Polyn (Vanderbilt): Incorporating neural signals into computational models of memory searchDiscussant: Jeremy Manning (Princeton)Alison R. Preston (University of Texas): Building new knowledge through memory integrationDiscussant: Marc Howard (Boston University)Maureen Ritchey, Andrew P. Yonelinas, Charan Ranganath (UC Davis): Medial temporal lobe subregions interactwith functionally distinct systemsDiscussant: Ken Norman (Princeton)Karthik Shankar, Marc W. Howard (Boston University): Optimally fuzzy memoryDiscussant: Sue Becker (McMaster University)Geoff Ward, Cathleen Cortis, Rachel Grenfell-Essam, Jessica Spurgeon, Lydia Tan (University of Essex): Why doparticipants initiate their immediate free recall of short lists of words with the first list item?Discussant:Karl Healey (Penn)26 minutes for primary speaker; 13 minutes for discussant; 6 for questions
  58. 58. “whole” “scrambled”“grid”Scrambling Objects in Spacecompletion no completionLerner 2003
  59. 59. “whole” “scrambled”“grid”12212211 2 1Lerner 2003Scrambling Objects in Space12

×