Successfully reported this slideshow.
Your SlideShare is downloading. ×

Uri Hasson - Neurocinematics: The Neuroscience of Film

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Loading in …3
×

Check these out next

1 of 62 Ad
Advertisement

More Related Content

Similar to Uri Hasson - Neurocinematics: The Neuroscience of Film (20)

More from swissnex San Francisco (20)

Advertisement

Recently uploaded (20)

Uri Hasson - Neurocinematics: The Neuroscience of Film

  1. 1. Music and the Brain Structured neural responses to natural stimuli Princeton University Uri Hasson Psychology department and the Neuroscience Institute
  2. 2. David Heeger Morwaread Farbood Gary Marcus Lab members Collaborators Yulia Lerner Chris Honey Greg Stephens Janice Chen Erez Simony Lauren Silbert Mor Regev Who did the work
  3. 3. Complex natural stimuliControl simplified stimuli Sensory Coding and the Natural Environment Few well characterized dimensions linear properties, simple math Multidimensional and Messy Parameterized Scientific investigation
  4. 4. Complex natural stimuliControl simplified stimuli Real life Complex Structured Movies/Stories/Music Sensory Coding and the Natural Environment
  5. 5. Memory formation of real world events Hasson et al. Neuron 2008 Neural responses to natural complex stimuli Hasson et al. Science 2004 Scan paths of real life events Hasson et al. In-press Disruption of brain responses in autism Hasson et al. Autism Research 2009 Time scale of processing Hasson et al. J neuroscience 2008 Using movies as a research tool in cognitive neuroscience Social communication Stephan et al. PNAS 2010
  6. 6. Subjects Experiment / measurements
  7. 7. Inter-subject correlations Talairach coordinates Subject 1/Run 1 Subject 2/Run 2
  8. 8. Idiosyncratic responsesSharedsignal Idiosyncratic signal Intra-SC Inter-SC
  9. 9. Sharedsignal Idiosyncratic signal Shared responses Intra-SC Inter-SC
  10. 10. Inter-subject correlations Hasson et al, Science, 2004
  11. 11. So I’m banging out my story and I know it’s good, and then I start to make it better by adding an element of embellishment. Reporters call this“making shit up”. And they recommend against crossing that line. But I had just seen the line crossed between a high-powered dean and an assault with a pastry, and I kind of liked it. Real-Life Story Stimulus “Pie Man” Story
  12. 12. Early Auditory Cortex r = 0.55 r = 0.72 Individual subjects Average subject Brain Responses to Real-Life Story “Pie Man” Story
  13. 13. PrecuneusEarly Auditory Cortex Angular GyrusInferior Frontal Gyrus r = 0.66 r = 0.6 r = 0.72 r = 0.41 r = 0.56r = - 0.18 r = 0.1 Brain Responses to Real-Life Story “Pie Man” Story
  14. 14. medial view Left Hemisphere Right Hemisphere lateral view lateral view A P P A r=0.11 0.45 N=11 Brain Responses to Real-Life Story “Pie Man” Story
  15. 15. Brahms Piano Concerto No.1 in D minor Brain Responses to Real-Life Music A1 M1 M1 STG STG A1 BG/ThalamusBG/Thalamus
  16. 16. The extent of neural overlap between language-related and music- related processes
  17. 17. Intermediate summary Real life stimuli, as movies, stories and music can exert considerable control over the responses of many brain areas, evoking a similar time course of activity across all viewers.
  18. 18. How does the brain process such complex and rich temporal structures?
  19. 19. So I’m banging out my story and I know it’s good, and then I start to make it better by adding an element of embellishment. Reporters call this“making shit up”. And they recommend against crossing that line. But I had just seen the line crossed between a high-powered dean and an assault with a pastry, and I kind of liked it. 1:43 1:45 1:47 1:50 1:54 1:58 2:05 0:55 Time-scales of Information in a Narrated Story So
  20. 20. medial view Left Hemisphere Right Hemisphere lateral view lateral view A P P A r=0.11 0.45 N=11 “Pie Man” Story Different Processing Timescales in Different Regions?
  21. 21. Scrambled past Word Word present Coherent pastScrambled past Extent of past information needed to evoke reliable responses in the present Entire story Neural responses at the present Coherent past Paragraph Paragraph present Coherent pastScrambled past Sentence Sentence present Coherent pastScrambled past present Long memory No memory Intermediate
  22. 22. Words Parametric variation of the temporal structure of a verbal monologue Paragraphs Backward Sentences Intact story Temporal rate is fixed Each 7 minutes condition is composed of the exact same basic units
  23. 23. A P P Reverselateral medial AP P LH RH q<0.05 (FDR) N = 11 A Inter-subject Correlation During a Narrated Story Words Sentences Paragraphs
  24. 24. medial view Left Hemisphere Right Hemisphere lateral view lateral view A P P A AP P N = 11 sent paragrev words A Hierarchy of Processing Timescales
  25. 25. A1+ TPJ Temporo-Parietal Axis FS P S W R correlation sent (S) parag (P)rev (R) words (W) FS P S W R FS P S W R N = 11 1 2 3 4 5 FS P S W R A Hierarchy of Processing Timescales
  26. 26. auditory story + silent movie short mid long overlap = Processing Time-Varying Information About the World
  27. 27. Bars Phrases Sections Intact Short temporal scales Mid temporal scales Long temporal scales Reversed Parametric  varia+on  of  the  coherent     temporal  structure  within  a  music  piece     Brahms     Piano  Concerto  No.1  in  D   minor
  28. 28. Musical Temporal Receptive Windows A1+
  29. 29. Musical vs. Linguistic based Temporal Receptive Windows
  30. 30. Musical vs. Linguistic based Temporal Receptive Windows
  31. 31. Word/bar present Short temporal integration window Extent of past information needed to evoke reliable responses in the present Entire story/musical piece present Long temporal integration window Paragraph/section present Intermediate temporal integration window Sentence/phrase present Intermediate temporal integration window present Long memory No memory Intermediate
  32. 32. Spatial Scale Temporal Scale Hubel & Wiesel (1959) J Physiol Gross et al (1972) J Neurophysiol Ungerleider & Mishkin (1982) Analysis of Behavior Functional Hierarchy A Hierarchy of Temporal Receptive Windows
  33. 33. Accumulating information over space and time Space Time Electrode in IT cortex Temporal Receptive Window in IT cortex A Hierarchy of Temporal Receptive Windows
  34. 34. Integrate Real-World Information Baddeley & Hitch (1974) Classical Working Memory Model Hierarchy of Temporal Receptive Windows Limited Capacity Bottleneck Processing Time Scales and Working Memory Maintain Discrete Units of Information
  35. 35. The end
  36. 36. Is Integration Temporal or Ordinal? Essential Role of Time Essential Role of Semantic Units Or Both? See also Howard & Eichenbaum (in press) JEP General versus
  37. 37. Temporal units and the information units are easily dissociated in real-life speech 100% 75% 50% 150% 200% Time
  38. 38. Yulia Lerner Our speech perception is invariant to changes in rate
  39. 39. Is Integration Really Temporal or Just Ordinal? Essential Role of Time Essential Role of Semantic Units Speech intelligibility recovered by insertion of pauses Rescaling of neural responses throughout the brain Memory for absolute tempo in musical sequences Integration of information over time much easier for meaningful speech Neurophysiology has intrinsic timescales Or Both? Ghitza & Greenberg (2009) Lerner et al (submitted) See also Howard & Eichenbaum (in press) JEP General “Time”cells in hippocampus Naya & Suzuki (2011) Macdonald et al (2011) Behavioral invariance to moderate changes in stimulus rate Levitin & Cook (1996)
  40. 40. Time NOWAccessibility low high Timescales of Information
  41. 41. Integrate Real-World Information Cowan (1999) Embedded Processes Model Working Memory =“Activated”Memory Traces Activated Memory Central Executive Long Term Memory Attended Processing Time Scales and Working Memory Hebb (1949) Activated Cell Assemblies Hierarchy of Temporal Receptive Windows
  42. 42. Time vs. accumulation of information over time
  43. 43. Integrate Real-World Information Persistent Neuronal Activity Activated Memory Central Executive Long Term Memory Attended Funahashi et al (1989) Gnadt & Anderson (1988) Goldman-Rakic (1996) Processing Time Scales and Working Memory Hierarchy of Temporal Receptive Windows
  44. 44. Hypothesis A Hierarchy of Timescales in Brain Dynamics Hierarchy of Temporal Receptive Windows
  45. 45. Open Questions Is the hippocampus required to sustain the long temporal receptive windows? What happens in the hierarchy at (macro & micro) event boundaries? c.f. Ranganath & Ritchie (2012) Nat Rev Neurosci
  46. 46. Lab Questions Should I shorten the title? Present larger questions at the beginning or at the end? Emphasize /time/ or mental context Narrative style or argument style?
  47. 47. Different Processing Timescales in Different Regions? Criterion One minimum prior duration of coherent information required for a response
  48. 48. responses invariant to changes beyond a maximum duration Criterion One Criterion Two minimum amount of coherent information required for a response Different Processing Timescales in Different Regions? equals
  49. 49. Study of Naturalistic Perception Costs Benefits Poorer experimental control Poorer experimental control
  50. 50. Why unrelated materials? Why under conditions of distraction? Working Memory Experiments
  51. 51. Memory systems are organized to represent the real world. We may look into that window on the mind as through a glass darkly, but what we are beginning to discern there looks very much like a reflection of the world. Roger Shepard (1990) Mind Sights Perceptual systems are organized to represent the real world. Anderson & Schooler (1991) Psych Science Bartlett (1932) Remembering Neisser (1978) Practical Aspects of Memory
  52. 52. Stimulus locked Circuit Dynamics using BOLD
  53. 53. Mary Potter fast semantics stabilization idea ISC? Van Dijk & Kintsch unavaoidable semantics the log was on the tutrlte McLelland and Rumelhart Bransford and Johnson effects on memory Stabiliza
  54. 54. Inter-subject Correlation during Movie Viewing Single Subjects (N=9) Mean Timecourse medial view Left Hemisphere Right Hemisphere lateral view lateral view A P P A r=0.15 0.55
  55. 55. Single Subjects (N=9) Mean Timecourse medial view Left Hemisphere Right Hemisphere lateral view lateral view A P P A r=0.15 0.55 Inter-subject Correlation during Movie Viewing
  56. 56. Processing Time Scales and Working Memory Working memory is the “ability to keep a representation active, particularly in the face of interference and distraction”. Engle et al (1999) JEP:General
  57. 57. Youssef Ezzyat, Lila Davachi (NYU): Neural mechanisms supporting the temporal organization of episodic long-term memory Discussant: Per Sederberg (Ohio State) Christopher J. Honey, Janice Chen, Erez Simony, Olga Lositsky, Daniel Toker, Kenneth A. Norman, Uri Hasson (Princeton): Temporal receptive windows in natural perception: a topographic map of mental context Discussant: Ryan Canolty (UC Berkeley) Gregory J. Koop, Amy H. Criss (Syracuse): Response dynamics as a measure of bias and strength in recognition memory Discussant: Adam Osth (Ohio State) Isabel A. Muzzio (Penn): Effects of emotion on hippocampal contextual representations Discussant: Sam Gershman (MIT) Robert M. Nosofsky, Christopher Donkin, Jason M. Gold, Richard M. Shiffrin (Indiana University): Discrete-slots models of visual working memory response times Discussant: Michael Lee (UC Irvine) Sean M. Polyn (Vanderbilt): Incorporating neural signals into computational models of memory search Discussant: Jeremy Manning (Princeton) Alison R. Preston (University of Texas): Building new knowledge through memory integration Discussant: Marc Howard (Boston University) Maureen Ritchey, Andrew P. Yonelinas, Charan Ranganath (UC Davis): Medial temporal lobe subregions interact with functionally distinct systems Discussant: Ken Norman (Princeton) Karthik Shankar, Marc W. Howard (Boston University): Optimally fuzzy memory Discussant: Sue Becker (McMaster University) Geoff Ward, Cathleen Cortis, Rachel Grenfell-Essam, Jessica Spurgeon, Lydia Tan (University of Essex): Why do participants initiate their immediate free recall of short lists of words with the first list item? Discussant:Karl Healey (Penn) 26 minutes for primary speaker; 13 minutes for discussant; 6 for questions
  58. 58. “whole” “scrambled”“grid” Scrambling Objects in Space completion no completion Lerner 2003
  59. 59. “whole” “scrambled”“grid” 1 2 212 2 1 1 2 1 Lerner 2003 Scrambling Objects in Space 1 2

×