Mind Reading: Neuroscience Methods and Technology

Mind Reading: Neuroscience Methods and Technology David J. Heeger New York University (http://www.cns.nyu.edu/~david/)

No Lie MRI, Inc. • Neuroscience-based lie detector technology • Commercial venture (http://www.noliemri.com/) • Application of functional magnetic resonance imaging (fMRI) • Claim 88% accuracy www.elsevier.com/locate/ynimg NeuroImage 28 (2005) 663 – 668 Classifying spatial patterns of brain activity with machine learning methods: Application to lie detection C. Davatzikos, a,* K. Ruparel, b Y. Fan, a D.G. Shen, a M. Acharyya, a J.W. Loughead, b R.C. Gur, b and D.D. Langleben b,c a Department of Radiology, University of Pennsylvania, 3600 Market Street, Suite 380, Philadelphia, PA 19104, USA b Department of Psychiatry, University of Pennsylvania, 3400 Spruce Street, Philadelphia, PA 19104, USA c Treatment Research Center, University of Pennsylvania, 3900 Chestnut Street, Philadelphia, PA 19104, USA• This is one of a series of recent brain imaging studies on lie detection, the most impressive to date.• The technique that they 1 March 2005; revised 15 June statistical analysis of brain imaging measurements. Received use relies on sophisticated 2005; accepted 4 August 2005• They conclude 88% accuracy (90% sensitivity, 86% specificity). This corresponds to 14% false alarm rate and 10% miss rate. Put another way, about 1 person Available online 5 October 2005 in 7 will be incorrectly identified as lying.• This approach is very different from, and potentially much more powerful than, traditional polygraph. Polygraph measures physiological responses that Patterns of brain activity during deception have recently been imaging data. The important distinction between a voxel-based indirectly reflectcharacterizedof the fMRI on the multi-subject average group pressure, sweat (electrical analysis of a spatio-temporal pattern is the same as the activity with autonomic nervous system: pulse, blood level. analysis and the conductance of the skin). The autonomic nervous system regulates essential value of fMRI in (heart, breathing, digestion). This the The clinical bodily functions lie detection will be determined by happens automatically and subconsciously. Changes in the polygraph measures the distinction between (mass) uni-variate and multi-variate can reflect arousal during deception but also general anxiety. fMRI liethan groupand EEG "brain fingerprinting" record neural activity in the brain including ability to detect deception in individual subjects, rather detection neural activity that is related to conscious non-linear pattern classification methods the polygraph which is an2004). Specifically, a pattern of brainautomatic and averages. High-dimensional mental state. This is very different from analysis (Davatzikos, indirect physiological measures of activity unconscious regulation of bodily function. is not only a collection of active voxels, but carries with it applied to functional magnetic resonance (fMRI) images were used to discriminate between the spatial patterns of brain activity associated correlations among different voxels. Notable efforts towards the with lie and truth. In 22 participants performing a forced-choice functional activity pattern analysis have been made (Strother et al., deception task, 99% of the true and false responses were discriminated 1995; McIntosh et al., 1996), some of which, attempt to use these correctly. Predictive accuracy, assessed by cross-validation in partic- methods to classify complex activation patterns using machine ipants not included in training, was 88%. The results demonstrate the learning methods (Cox and Savoy, 2003; LaConte et al., 2005). potential of non-linear machine learning techniques in lie detection and In this paper, we present an approach to the problem of other possible clinical applications of fMRI in individual subjects, and identifying patterns of functional activity, by using a high- indicate that accurate clinical tests could be based on measurements of dimensional non-linear pattern classification method. We apply brain function with fMRI. this approach to one of the long-standing challenges in applied D 2005 Elsevier Inc. All rights reserved. psychophysiology, namely lie detection. Deception is a socially and legally important behavior. The limitations of the specificity of the currently available physiological methods of lie detection Introduction prompted the exploration of alternative methods based on the correlates of the central nervous system activity, such as EEG and A large body of functional neuroimaging literature has fMRI (Rosenfeld, 2001; Spence et al., 2001; Langleben et al.,

Functional magnetic resonance imaging Revolution in psychology and neuroscience: ~1000 papers published per month over the past 5 years!• fMRI has revolutionized neuroscience over the past decade.• It is like regular MRI that is used clinically to take pictures of the anatomy of the body, but is used to take pictures of brain function in addition to brain anatomy.• New era of research into the function and dysfunction of the human brain.• fMRI bridges the gap between brain and behavior. Most of what we know about human behavior (perception, cognition, emotion) originates with experimental psychology. Most of what we know about how the brain works comes from neurobiology.Briefly, here’s how it works:1) The brain controls the flow of oxygenated blood to where it is needed (discovered over 100 years ago).2) There is iron in blood which is a magnetic metal. Oxygenated and deoxygenated blood have different magnetic properties (discovered by Linas Pauling in the1930s).3) MRI was invented in the 1970s, based on the physics of magnetic resonance which was discovered in the 1940s.4) The MRI scanner can be reprogrammed to pick up differences in magnetization that take place when the brain ships oxygenated blood to where it is needed.

Setting up an fMRI facility • Vendors: Siemens, GE, Phillips • Capital investment (scanner plus renovation): $2-3 million • Annual operating expenses (staff plus service contract): $250,000-$500,000

Mobile MRIGenesis Medical Imaging, Inc.(http://www.genesismedicalimaging.com/)

It’s all in the brain Perception Attention Memory Language Emotion Personality Decision making Arts & aesthetics Moral reasoning Social interaction Conjecture: The human mind – perceptions, emotions, memories, thoughts – can be completely explained by the electrical and chemical activity of the brain.Neuroscience research has broad ramifications, not only in biomedical science but for our society as a whole. It’s all in the brain - everything about human behavior, humannature, and human society is controlled by the human brain. Neuroscientists have focused in the past on a handful of basic, core topics (perception, memory, emotion). But weare now beginning new interdisciplinary collaborations to explore, from a neuroscientific point of view, topics that used to be outside the purview of science.If this conjecture is correct, then it will be possible with neuroscience methods like fMRI to measure any and all aspects of an individual’s mental state (conscious andunconscious). The current techniques for non-invasive measurement of human brain function are limited but both the techniques themselves and our ability to use themeffectively will continue to improve as we understand more about brain function.

Outline • Neurosci 101 (how the brain works) • How neurons work • A little bit of neuroanatomy • Functional specialization & computational theory • Case study: visual motion perception • Techniques for measuring human brain activity • Some examples of mind reading with fMRITo understand how these neuroscience techniques and applications (like neuroscience based lie detection) work and what their limitations are, we need totake a step back and learn some of the basics of neuroscience.

Outline • Neurosci 101 (how the brain works) • How neurons work • A little bit of neuroanatomy • Functional specialization & computational theory • Case study: visual motion perception • Techniques for measuring human brain activity • Some examples of mind reading with fMRI

Schematic of a neuron• Dendrites: Inputs from other neurons arrive here, they are combined in various ways (input signals might be added, subtracted, divided by one another).• Axon: Result of this computation propagates along the axon to serve as inputs to other neurons. Some axons are very short (local connections between neurons in one part of the brain). Some are very long (more than a meter) down the spinal cord to innervate a muscle.• Soma: cell body (in between dendrites and axon)

Electrophysiology (action potential) microelectrode amplifier Voltage (mV) Time (msec) Cortical pyramidal cell (Golgi stain)Neurons represent and transmit information electrically. If you place an electrode close to a neurons soma then you can measure and record these electricalsignals. When stimulated, a neuron fires an action potential during which the voltage jumps way up, then comes back down again. It all happens very fast (notethe time scale), all in a matter of a few thousandths of a second. The action potential starts at a neurons soma and travels all the way along its axon. The shapeof the action potential is basically the same anywhere along the length of the axon. Hodgkin and Huxley won the Nobel prize for work they did in the 1950s tounderstand what makes action potentials happen and how they propagate along axons. Action potentials are the “currency” of the brain.

Firing rate: visual neurons and contrast V1Firing rate: the number of action potentials that occur per unit of time. For example, the responses of neurons in visual areas of the brain depend on the contrastof the test light. For a low contrast we would get only a few action potentials. For a high contrast, we would get many more action potentials. Any given actionpotential looks exactly like all the others. When we increase the contrast, the individual action potentials do not get bigger.This graph plots the average firing rate (averaged across a large number of individual neurons) in response to series of stimuli with different stimulus contrasts.The graph also shows that fMRI measurements of human brain activity increase in the same way with contrast. This is important because it shows that the fMRImeasurements are tightly linked with the underlying firing rates of the neurons.

Neurons in MT are selective for motion direction Newsome et al.How is it that we know that a particular neural signal is due to a stimulus that originated as a sound, versus knowing that another neural signal arose from someother type of physical stimulus, such as a light source? Suppose that we initiate a neural response in our visual system by electrically stimulating neurons in youreye rather than by absorbing some light. When you excite the visual neurons in any way whatsoever, the response that is evoked in the nervous system is thatof a visual response. This holds true no matter what the physical cause of the excitation. The determining factor is which neuron is being excited. Likewise, nomatter how it is you excite an auditory neuron, the resulting sensation will be one of hearing (cochlear implants, for example, work by direct electrical stimulationof the auditory nerve).Neurons in different parts of the brain are selective for different things. Just keeping to neurons involved with vision, there are neurons that are selective for allkinds of properties: stimulus position, color, pattern, direction and speed of motion. There are even neurons that respond selectively to faces.Visual area MT is one of the most studied regions of the brain. MT neurons are velocity selective; each responds best to a preferred velocity (speed anddirection) of motion within particular spatial location, pretty much independent of stimulus pattern, shape, or color. Motion is the only thing that matters. Thisvideo shows a sequence of moving stimuli. A monkey watched these stimuli while neuroscientists recorded the electrical activity of a single neuron in area MT.The electrical signal from the neuron was amplified and recorded as the sound track. The popping sounds are action potentials. The neuron responded moststrongly to one direction (down-left) and less to other directions (least to up-right).

Neural communicationHow do neurons communicate? The axon of one neuron comes very close to (about 20 nanometers; a nanometer is a billionth of a meter) but doesnt quitetouch the dendrite of its target. The narrow space between the two neurons is called a synapse.

Synapse: electron microscope pictureHighly magnified slice through the brain showing the axon terminal, dendrite, and synapse. The little circles in the axon terminal are neurotransmittervesicles.

Synaptic transmission • Direction of action potential Neurotransmitter • Presynaptic vesicle axon • Synapse • Postsynaptic dendriteThe electrical signal from the (pre-synaptic neurons) axon is not transmitted directly across the synapse. Rather, when the action potential comes along, itcauses the release of certain chemicals called neurotransmitters. The neurotransmitters diffuse across the synapse and bind to receptor molecules in themembrane of the (postsynaptic neurons) dendrite. This, in turn, causes a change in the electrical properties of the postsynaptic neuron. If the postsynapticneuron is stimulated strongly enough, then it too will fire action potentials. Bernard Katz won a Nobel prize for figuring out how synapses work.There are a variety of different neurotransmitter molecules including: dopamine, acetylcholine, serotonin, GABA, glutamate. Some neurotransmitters areexcitatory, causing the postsynaptic neuron to respond. Some are inhibitory, shutting down the responses. A single neuron receives many syanpses, someexcitatory and some inhibitory. The combinations of excitatory and inhibitory inputs are what allow the neuron to perform computations. Some neurotransmittersact very quickly (just a few msec) and some more slowly so that neurons can combine information over short or long time intervals.

Cerebral cortex Frontal lobe Parietal lobe Occipital lobe Temporal lobeLobes: occipital in back, frontal in front, parietal on top, temporal on side.Sulcus: infoldingGyrus: outfoldingEach major sulcus and gyrus has a name (e.g., inferior temporal sulcus or inferior frontal gyrus). These anatomical landmarks are identifiable in every human brain and, as a general rule, the function and physiology of the neurons in a particular region (e.g., posterior end of the inferior temporal sulcus) will be the same in every human brain. More on this later.

Medial view cerebral cortex corpus collosum cerebral cortex thalamus optic nerve brain stem cerebellum

Gray matter and white matter Gray matter (stained purple): folded sheet containing cell bodies, dendrites, local axons collaterals. White matter: axons, long range connections.Right: Slab of cortical tissue removed from the back of a monkey brain.Left: This tissue is stained so that the cell bodies appear purple. All the cell bodies are in a layer near the surface of the brain, called the gray matter of the brainbecause it has grayish color when its not stained. The longer axons reach out of the gray matter into the central part of the brain, called the white matter of thebrain because the axons altogether look white. The white matter is like a massive tangle of wires - axons connecting from one part of the cortex to another.Gray matter is about 3-4 mm thick, filled with cell bodies (somas), dendrites, and axon terminals (synapsing on the dendrites). Gray matter is where all theinteresting stuff happens, where a neuron receives inputs (via synapses) from a whole bunch of other neurons, and then computes a new output. The axons inthe white matter just transmit that output to other parts of the brain.

Neural circuits perform computations ~50,000 neurons per cubic mm ~6,000 synapses per neuron ~10 billion neurons & ~60 trillion synapses in cortexNeurons perform computations. For example, a given neuron’s response might depend on the sum of the responses from two of its inputs. The function of aneuron in a particular brain area are best understood in terms of the computations that they perform. There are many different types of neurons in each bit ofbrain tissue, connected together to form circuits. It is not unlike a computer, but neural circuits are extremely complex.How can we possibly hope to understand such a complex process? There are two guiding principles.

Two guiding principles Functional specialization Computational theory

Functional specialization: primary sensory areasFunctional specialization: The brain is subdivided into a number of separate and distinct brain areas, each of which performs a different function. Particular brainareas are specialized for sensation and perception: vision, touch, olfaction, hearing.

Visual maps in the brain IPS2 IPS1 V7 V3 V2 V3A/B LO1 LO2 MT+ V4 Each visual brain area contains a map of the visual world and performs a different function.Within each sensory modality, there is a further subdivision. The visual cortex, for example, is subdivided into a number of distinct visual cortical areas. Eachvisual brain area contains a map of the visual world. The neurons at one location in the brain respond to visual features at one location in the world. The neuronsat a neighboring location respond to neighboring features. We routinely measure these visual maps in the brian using fMRI to identify each of the visual brainareas. In about an hour, I could measure your visual maps and identify each of your visual brain areas.Each visual brain area performs a specific function. Some examples follow.

Functional specialization: shape perception LO Malach et al.This video demonstrates that visual area LO is functionally specialized for the perception ofshape.

Functional specialization: motion perception MTThis video demonstrates that visual area MT is functionally specialized for motionperception.

How do neurons compute motion?This video illustrates a simplified theory of how a neuron can compute visual motion. The top two ovals represent photoreceptors in the retina of the eye(specialized neurons that convert light into electrical neural responses). The photoreceptor on the left send its action potentials to another neuron that delays thetransmission before reaching the output neuron at the bottom (the circle), whereas the photoreceptor on the right sends its action potentials directly withoutdelay. When a spot of light moves at a particular speed from left to right, the delayed signal from the first photoreceptor and the non-delayed signal from thesecond photoreceptor both reach the output neuron (circle) at the same time. The output neuron itself responds if, and only if, it receives coincident inputs.This is oversimplified, but illustrates the basic idea. A full-blown theory of MT physiology includes a series of neural computations along a well-defined pathwayof neural connections, beginning with a particular class of neurons in the retina, and a specialized subclass of neurons in primary visual cortex, before reachingMT.

Functional specialized brain areas are connected in pathways and networks Action optic nerve LGN primary visual cortex (V1) Recognition Ventral view showing visual Lateral view showing visual pathways from the eye to pathways for recognition the back of the brain. and visually-guided action.Neural computations are performed by local circuits within a brain area (e.g., MT) and then passed on to other brain areas. The brain areas are interconnectedin well-defined pathways and networks.The diagram on the left shows the pathway from the eye to primary visual cortex (also called V1). The diagram on the right shows the two main pathways invisual cortex. One goes to the parietal lobe and the other goes to the temporal lobe. What do these two pathways do? Clinical observations have provided uswith most of the information on this. One finds different deficits in patients who have lesions in these two different areas. The deficits are very different. Parietallesions lead to deficits in spatial orientation, attention, and visually-guided behavior (this is the "where" pathway or the “action” pathway). Temporal lobe lesionslead to deficits in object recognition (thus, this is the "what" pathway or the “recognition” pathway).

Maps of the body in the brainThe notion of functional specialization is not restricted to sensation and perception. There are, for example, areas of the brain that are functionally specializedfor movement. These areas of the brain are also organized into maps, in this case maps of the body. Each location in the brain is responsible for controllingmovement of a different part of the body.

Functional areas in speech and language Arcuate Fasciculus Broca’s area bundle of fibers connecting Broca’s and Wernicke’s areas speech production Wernicke’s Area speech perceptionLeft: Auditory pathways from the ear to auditory cortex.Right: Functional specialization for speech and language. Analogous to sensory brain areas, these language areas are also interconnected in well-definedpathway. These are the brain areas, the activity of which would be most closely associated with “verbal testimony”.

The frontal lobe and higher cognitive function prefrontal cortex anterior cingulate cortex• Purple: somatosensory cortex (contains a map of the body for touch sensation).• Orange: primary motor cortex (contains a map of the body for movement control)• Pink: supplementary motor cortex (contains a number of subregions involved in movement planning and control)• Prefrontal cortex: large region in the front of the brain that is involved in higher cognitive function. Prefrontal cortex is the least well-understood region of the brain, but it is believed that it might be organized into maps of cognitive function.• Anterior cingulate: a subregion of medial prefrontal cortex that has been implicated in deception (a critical part of the interpretation of a many lie detection studies). There is some evidence that this brain area is important for cognitive control (response selection, response inhibition, conflict resolution), which might be important for deception. However, we have a lot to learn about what this part of the brain really does and it is an active area of research. More on this later.

Vision is an unconscious inferenceWe all have the notion that vision is automatic and effortless. But in fact it is an amazing accomplishment of your brain. The light from this picture reaches youreye. It is only light, of course, that comes into your eye. The light is absorbed by specialized neurons in the retina and transformed into electrical signals. Thosesignals are processed by a series of neurons in the retina and then action potentials are transmitted to the visual part of the brain (at the back) where furtherprocessing is done. About a third of your brain is devoted to visual processing. The end result is your full visual experience of the world: recognition of objects,people, faces, facial expressions, color, distance, motion, and so on. All this is inferred from the changing pattern of light reaching your eye. These inferences areperformed by neural computations.

Visual inference: motion perceptionVisual inferences are usually quite accurate but not always. This video shows an example of a visual illusion of motion that I made a few years ago. When thevideo is playing, the figure appears to move while staying in the same location. It is based on Duchamp’s famous painting “Nude descending a staircase” whichis all about freezing motion in time. The changing pattern of light causes the visual brain to make a mistake. Normally, such changing patterns of light wouldcorrespond to an object that is moving in the world. In this case, however, there is no motion and the inferred/perceived motion is wrong.

Another visual illusion of motion, again demonstrating that visual motion perception is an unconsciousinference.

Cortical area MT is specialized for visual motion perception • Neurons in MT are selective for motion direction. MT • Neural responses in MT are correlated with the perception of motion. • Damage to MT causes deficits in visual motion perception. • Electrical stimulation in MT causes changes in visual motion perception. • Computational theory quantitatively explains both the responses of MT neurons and the perception of visual motion. • Well-defined pathway of brain areas (cascade of neural computations) underlying motion specialization in MT.Visual motion perception is accomplished by neural computations. Cortical area MT has been shown to play a particularly important role. Importantly, it has beenshown that MT is not only correlated with motion perception but also that activity in MT causes the perception of motion. We have rigorous, quantitative theoriesthat explain both the physiology of MT neurons and the psychology motion perception (e.g., in the visual illusions shown earlier). Correlational evidence, by itself,is not enough.• Confounding factor: some hidden variable that might be driving a correlation. Example: moving stimuli are more interesting to look at than stationary. If all we had was a correlation between between MT activity and motion perception, then it might be due to attention/engagement, not motion perception per se.• Generalization: without a theory, can’t generalize beyond the specific circumstances of the experiments that have already been done.

Functional magnetic resonance imagingNow that we have learned some of the basics of how the brain works, let’s revisitfMRI.

Indirect measure of neural activityIn an fMRI experiment, we acquire a time-series of images. Changes in blood oxygenation (following metabolic demand of neural activity) appear as changes inimage intensity. There are numerous demonstrations (including some of the above slides) that the fMRI measurement is tightly linked with underlying neural firingrates. fMRI is, however, an indirect measure of the underlying neural activityfMRI takes advantage of the coupling between neuronal activity and hemodynamics (the local control of blood flow and oxygenation) in the brain to allow the non-invasive localization and measurement of brain activity. The ultimate success of fMRI as a measurement of brain function depends on the relationship betweenthe fMRI signal and the underlying neuronal activity. The vascular source of the fMRI signal places important limits on the usefulness of the technique. Althoughwe know that the fMRI signal is triggered by the metabolic demands of increased neuronal activity, the details of this process are only partially understood.Consequently, this issue has emerged as one of the most important areas in neuroscience. There are two parts to the story: 1) The physics is very wellunderstood, but 2) The physiology is not so well understood.Physics. The fundamental signal for blood oxygen level dependent (BOLD) functional magnetic resonance imaging (fMRI) comes from hydrogen atoms,which areabundant in the water molecules of the brain. In the presence of a magnetic field,these hydrogen atoms absorb energy that is applied at a characteristic radiofrequency (~64 MHz for a standard, clinical 1.5-Tesla MRI scanner). After this step of applying radio-frequency excitation, the hydrogen atoms emit energy at thesame radio frequency until they gradually return to their equilibrium state. The MRI scanner measures the sum total of the emitted radio-frequency energy. Themeasured radio-frequency signal decays over time, owing to various factors, including the presence of inhomogeneities in the magnetic field. Greaterinhomogeneity results in decreased image intensity, because each hydrogen atom experiences a slightly different magnetic field strength, and after a short timehas passed (commonly called T2*), their radio-frequency emissions cancel one another out. BOLD fMRI techniques are designed to measure primarily changesin the inhomogeneity of the magnetic field,within each small volume of tissue, that result from changes in blood oxygenation. Deoxy- and oxyhemoglobin havedifferent magnetic properties; deoxyhemoglobin is paramagnetic and introduces an inhomogeneity into the nearby magnetic field,whereas oxyhemoglobin isweakly diamagnetic and has little effect. Hence, an increase in the concentration of deoxyhemoglobin causes a decrease in image intensity, and a decrease indeoxyhemoglobin causes an increase in image intensity.Physiology. The brain increases the local blood flow in reaction to the demand for glucose and oxygen. The details of this process are not fully understood butone theory posits that blood flow follows directly from increased synaptic activity. Astrocytes (specialize glial cells, not neurons, in the brain) surround bothsynapses and capillaries, and they are responsible for neurotransmitter recycling (taking the glutamate out of the synapse quickly to stop its action on the post-synaptic membrane, causing a chemical change in the glutamate molecules to deactivate it, and handing it back to the nearby neurons for reuse). All this takes alot of energy.

fMRI spatial and temporal resolution Time series of fMRI images Typical frame time: 2 sec Typical pixel size: 3mm x 3mm x 3mm TimeSpatial and temporal resolution of fMRI is limited by blood flow. fMRI response depends on the average activity of neurons in a little chunk of brain, andaveraged over time. Best case spatial resolution is about 1mm x 1mm x 1mm. Typical spatial resolution is 3mm x 3mm x 3mm. Recall that there are about50,000 neurons per cubic mm. That’s a lot of neurons and they could be doing different things which would get confounded with one another in the fMRImeasurement. Best case temporal resolution is about 100 msec, and typical temporal resolution is 2 sec. That’s a long time compared to the time-scale of neuralprocessing (recall that action potentials take place in just a couple msec). A series of neural processes could take place during that time period which would getconfounded with one another during that time period.

Neurovascular coupling limits spatial specificity I II III IV V VI Reina de la Torre et al Anatomical Record (1998) Spatial specificity (spread and mislocalization) depends on vein size.Veins that drain from large regions of the brain will reflect the metabolic demand of a large number of neurons whereas small veins that drain locally will reflectthe activity in a smaller population of neurons, with better spatial resolution and spatial accuracy.

Sluggishness of hemodynamics limits temporal specificity (% change image intensity) 0.5 0.4 fMRI response 0.3 0.2 0.1 0 -0.1 -0.2 0 2 4 6 8 10 12 14 16 Time (s) Brief pulse of neural activityHemodynamics: control of blood flow and oxygenation is sluggish. A brief pulse of neural activity evokes a change in blood oxygenation that evolves over manyseconds.In spite of sluggish hemodynamics, can make fine discriminations about timing of a neural processes. I.e., can discriminate the timing of a single neuralprocessing event. But cannot distinguish a series of neural processing events that all take place during a short time period.

PET (positron emission tomography)A patient lies in the scanner after being injected with radioactive-labeled water. Because more blood goes to active brain regions, there is more radioactivity inthose brain regions. The PET scanner detects and localizes these pockets of higher radioactivity.PET based upon the unique radioactive decay of positrons, the anitmatter equivalent of electrons. These tiny, positively charged, radioactive particles emergefrom the radioactive water. After being emitted they are attracted to nearby negatively charged electrons. When positrons and electrons come together, they areannihilated, and energy is released in the form of two photons (particles of light, no charge) that leave the point of annihilation in exactly opposite directions. ThePET scanner is set up to detect the coincidental arrival of pairs of photons. The location of the positron-electron annihilation is determined by which pair ofdetectors are simultaneously active.Both fMRI and PET depend on regional control of blood flow in the brain. But, current fMRI methods go way beyond the conventional PET subtractionmethodology. Critically, fMRI is non-invasive - no radioactive stuff needs be injected in your blood stream.

EEG (electroencephalography) Event-related potentials (ERP)EEG (electroencephalograph) is also being used to study brain function. At each electrode, the electrical activity is recorded at fixed intervals following astimulus presentation - say, every 4 milliseconds.With EEG, you get millisecond time resolution which is good. But a huge disadvantage is localization. Dont really know exactly where these electrical signalsare coming from. The EEG signal recorded with each electrode on the scalp reflects the pooled (average) electrical responses of large numbers of neuronsthroughout the brain. The strength of the EEG signals depend on how far the electrode is from the neural source so the array of electrodes on the scalp can beused to infer roughly where the neural source is located. This works well if there are only a few neural sources. But if there are a large number of brain regionsactive simultaneously then the locations of the different neural sources get confused with one another and there is no way to pull them apart.

MEG (magnetoencephalography)MEG measures the tiny magnetic fields evoked by electrical currents in neurons. The magnetic fields are generated when electrical currents travel along thedendrites of a large enough number of adjacent neurons simultaneously. Like EEG, it has the advantage of millisecond time resolution. But like EEG, it haslimited spatial resolution.Given the importance of spatial localization of activity in the brain, fMRI has become the method of choice.

TMS (transcranial magnetic stimulation)Transcranial magnetic stimulation (TMS) is the use of powerful rapidly changing magnetic fields to induce electric fields in the brain by electromagnetic inductionwithout the need for surgery or external electrodes. Repetitive transcranial magnetic stimulation is known as rTMS. TMS is a powerful tool in research formapping out how the brain functions, and has shown promise for noninvasive treatment of a host of disorders, including depression and auditory hallucinations.One reason TMS is important in neuroscience is that it can demonstrate causality. A noninvasive mapping technique such as fMRI allows researchers to seewhat regions of the brain are activated when a subject performs a certain task, but this is not proof that those regions are actually used for the task; it merelyshows that a region is associated with a task. If activity in the associated region is suppressed with TMS stimulation and a subject then performs worse on atask, this is much stronger evidence that the region is used in performing the task. For example, it has been shown the TMS in MT interferes with motionperception.

Outline • Neurosci 101 (how the brain works) • How neurons work • A little bit of neuroanatomy • Functional specialization & computational theory • Case study: visual motion perception • Techniques for measuring human brain activity •Some examples of mind reading with fMRI

Mind reading: attention Gandhi, Heeger, & Boynton, PNAS (1999)The study of attention has a very long history within psychology, dating back more than 100 years. We know that when people attend (without moving their eyes)to certain visual stimuli to perform a task, the responses of some visual neurons are enhanced compared to when attention is directed elsewhere.This video shows a demonstration of how attention modulates brain activity in visual cortex. In the experiments, subjects fixated the center of a display whileshifting their attention to the right or the left half of the display. Stimuli on the right are processed by neurons in the left hemisphere and vice versa. The stimuliwere always the same and the subjects did not move their eyes, nor did they perform any other overt behavioral response to indicate where they were attending.Only their internal mental state (attention to one side or the other) varied. Attending left increased brain activity in the right hemisphere. Attending right increaseactivity in the left hemisphere.

Functional specialization for recognition Parahippocampal place area (PPA) Fusiform face area (FFA)A series of experiments have shown that there are brain regions that are functionally specialized for recognizing different categories of objects. Here, forexample, are two particularly well-studied areas. The PPA responds more strongly to pictures of houses, buildings, and indoor or outdoor scenes. The FFAresponds selectively to pictures of faces.

Mind reading: imagined faces and places “Woody Allen” “Great Count” “Bill Clinton” “Cary Grant” “Media Lab” F F P P P F F P F Stimulus type FFA 2.0 presented PPA fMRI response 1.5 1.0 0.5 85% of trials were 0 correctly identified. O’Craven & Kanwisher, J Cogn Neurosci (2000)This experiment capitalized on the functional specializations of PPA and FFA to study what happens in the brain when you imagine something. Once every 12seconds, subject heard the name of a person or a familiar place (a building on the MIT campus). They were instructed to imagine it. The investigator recordedfMRI signals in the "face" area (that responds strongly to pictures of faces) and the "place" area (that responds strongly to pictures of familiar places, buildings,etc.). The red curve shows the time-course of response in the "face area". The blue curve shows the time-course of response in the the "place area". fMRIresponse is bigger in the "face" area when imagining a face. Response is bigger in the "place" area when imagining a place.

Classifying perceived motion direction Right vs left? 90-95% accuracyUp until now, I have used examples in which the different states of mind (attend right vs attend left, imagine a face vs imagine a place) correspond to brainactivity in widely separated locations. Often, the neural representation of a particular state of mind is interleaved with that of another state of mind. Visual motionperception is, again, a good example. The neurons (e.g., in area MT) that prefer leftward motion are interleaved with those preferring rightward motion. Hence,rightward motion evokes a distributed pattern of brain activity (blue) and leftward motion evokes a complementary distributed pattern (red). The perceived motiondirection can be “classified” on any given trial by measuring the brain activity and deciding which pattern it most resembles.

How the human brain interacts with the world in real life The full complexity of real life: Simple sensory stimuli:Neuroscience research has tended to follow a reductionistic, deductive line of reasoning. For decades, neuroscientists have worked toward simplification, usingsimple stimuli and behavioral tasks, precisely parameterized and in highly controlled laboratory settings. This approach has obvious advantages and has servedus well, as evidenced by the tremendous amount of knowledge amassed about brain structure and function. These conditions, however, are removed fromnatural real life situations.Recent research has started employing empirical protocols in which we measure brain activity using fMRI during free viewing of engaging, natural sensorystimuli (e.g., movie).

How the human brain interacts with the world in real life Inter-subject correlation Hasson et al., Science (2004)Requires developing new data analysis methods that do rely on predetermined stimulation or behavioral protocols. A simple example of this is an analysis of theinter-subject correlations in the timecourse of brain activity. Using the activity at each location of one brain to predict the activity in other brains, Hasson et al(Science, 2004) found that brain activity is highly correlated (across individuals) when watching the same movie. About 30-40% of your brain does the same thingas my brain as we share the same audiovisual and emotional experience.This characterizes the component of brain activity that is common/shared across individuals during “realistic” situations. Extensions of this approach will be ableto characterize differences between groups in how their brain react to complex stimuli, and eventually individual differences as well.

Brain activity interpretation competition “Mind reading” competition at Human Brain Mapping 2006 (http://www.ebc.pitt.edu/competition.html) fMRI responses to video from 3 segments of the Home Improvement TV series, rated for a variety of features (e.g., faces, emotions). Utilize data from segments 1 and 2 to train classifier, then generate predicted behavior ratings for segment 3.Combining the classifier analysis (above) with the use of complex, realistic stimuli, neuroscientists are now able to “read out” from the brain what the subject wasviewing as well as some other cognitive and emotional aspects of what they were experiencing while viewing. A competition was held at a recent brain imagingconference.

No Lie MRI, Inc. www.elsevier.com/locate/ynimg NeuroImage 28 (2005) 663 – 668 Classifying spatial patterns of brain activity with machine learning methods: Application to lie detection C. Davatzikos, a,* K. Ruparel, b Y. Fan, a D.G. Shen, a M. Acharyya, a J.W. Loughead, b R.C. Gur, b and D.D. Langleben b,c a Department of Radiology, University of Pennsylvania, 3600 Market Street, Suite 380, Philadelphia, PA 19104, USA b Department of Psychiatry, University of Pennsylvania, 3400 Spruce Street, Philadelphia, PA 19104, USA c Treatment Research Center, University of Pennsylvania, 3900 Chestnut Street, Philadelphia, PA 19104, USA Received 1 March 2005; revised 15 June 2005; accepted 4 August 2005 Available online 5 October 2005 Patterns of brain activity during deception have recently been imaging data. The important distinction between a voxel-based characterized with fMRI on the multi-subject average group level. analysis and the analysis of a spatio-temporal pattern is the same as The clinical value of fMRI in lie detection will be determined by the the distinction between (mass) uni-variate and multi-variate ability to detect deception in individual subjects, rather than group analysis (Davatzikos, 2004). Specifically, a pattern of brain activity averages. High-dimensional non-linear pattern classification methods is not only a collection of active voxels, but carries with it applied to functional magnetic resonance (fMRI) images were used to discriminate between the spatial patterns of brain activity associated correlations among different voxels. Notable efforts towards the with lie and truth. In 22 participants performing a forced-choice functional activity pattern analysis have been made (Strother et al., deception task, 99% of the true and false responses were discriminated 1995; McIntosh et al., 1996), some of which, attempt to use theseThis brings us backPredictive accuracy, assessed by methods described in this paper (and the basis for the lie detection technology that is being adopted by No correctly. full circle to lie detection. The cross-validation in partic- methods to classify complex activation patterns using machineLie MRI) is based on a classifier just like the one The results demonstrate the person’s motion perception. and Savoy, 2003; LaConte et al., 2005). ipants not included in training, was 88%. I illustrated for “reading” a learning methods (Cox potential of non-linear machine learning techniques in lie detection and In this paper, we present an approach to the problem ofIn this study, participants were first given two cards and they were told to lie about one of them during the experiment. Then, in the MRI scanner, they were other possible clinicalthey were instructed in individual subjects, buttons in identifying each card of indicate whether or not it using a of the cards thatshown a series of cards, and applications of fMRI to press one of two and reaction to patterns to functional activity, by was one high-they were indicateOn some trials, the participants were presented with one of the dimensional non-linear pattern initially and onmethod. Wethey were shown given. that accurate clinical tests could be based on measurements of two cards that they were given classification other trials applyother cards from the deck. The investigators measured the activity throughout the brain on each individualthe long-standing and applied in classifier to brain function with fMRI. this approach to one of trial (each card) challenges a applieddistinguishing “truth” fromInc. All rights reserved. D 2005 Elsevier “lie” trials. psychophysiology, namely lie detection. Deception is a socially and legally important behavior. The limitations of the specificity of the currently available physiological methods of lie detection Introduction prompted the exploration of alternative methods based on the correlates of the central nervous system activity, such as EEG and A large body of functional neuroimaging literature has fMRI (Rosenfeld, 2001; Spence et al., 2001; Langleben et al., elucidated relationships between structure and function, as well 2002). Using SPM-based analyses of multi-subject average group as functional activity patterns during a variety of functional data, several recent fMRI studies demonstrated differences in activation paradigms. Statistical parametric mapping (SPM) brain activation between truthful and non-truthful responses in (Friston et al., 1995) has played a fundamental role in these various experimental paradigms (Langleben et al., 2002; Langle- studies, by departing from the conventional biased ROI- and ben et al., in press; Ganis et al., 2003; Kozel et al., 2004a,b; Lee hypothesis-based methods of data analysis and enabling unbiased et al., 2002). In order to translate these data into a clinically voxel-by-voxel examination of all brain regions. While a great deal relevant application, discrimination between lie and truth has to of knowledge has been gained during the past decade regarding be achieved at the level of single participants and single trials brain regions that are activated during various tasks using voxel- (Kozel et al., 2004b), not just via group analysis. The potential of based SPM analysis, the quantitative characterization of entire the SPM-based approach to achieve this goal is limited due to the spatio-temporal patterns of brain activity, as opposed to voxel by between-subject variability of regional brain activity. In the

Generalizability of fMRI-based lie detection • Correlation only (no causation, no theory): many possible confounds. • Contrived scenario (instructed to lie for $50) versus real- world scenario (risk of severe societal, emotional, monetary damages). • Individual differences: experimental subject population (college students) versus real-world population (socioeconomic groups, cultural backgrounds, sociopaths, etc.). • Deception vs truth: false memory, imagination, mental illness. • External validity and verifiability.• Confounds. An example of a possible confound is the following. It turns out that in many of the lie detection studies, participants were slower in providing their response (”yes” or “no” by pressing one of two buttons) when they were lying than when they were telling the truth. There is certainly a neural correlate of this difference in behavioral response time. Might it be that some of the studies are basing their “lie detection” simply on this?• Theory. The theoretical claims in the literature are weak at best. The interpretation of the lie detection studies rests on previous reports that specific brain areas (e.g., anterior cingulate cortex or ACC) have particular functional specializations (e.g., cognitive control, conflict resolution, response inhibition). But those previous reports are themselves just correlations between brain activity and behavior. So it is like a house of cards: a correlation, the interpretation of which rests on a correlation, all the way down. Prefrontal cortex (including ACC) is the least well-understood region of the brain; we have a lot to learn about what this part of the brain really does and it is an active area of research. For example, close to 600 papers in 2005 about ACC. In addition to cognitive control, the topics of these papers are quite diverse: stress, depression, drug abuse, bipolar disorder, pain, verbal fluency, psychosis, schizophrenia, reward encoding, motivation and reward anticipation, PTSD, fear and fear memory, borderline personality disorder, vision, disgust, social phobia, delirious state, conscious effort, social anxiety disorder, and attention. Moreover, neuropsychology studies of patients with damage to ACC have shown that there are no performance deficits for tasks that require cognitive control (Fellows & Farah, Is anterior cingulate cortex necessary for cognitive control? Brain 128, 788-96, 2005). Hence, there is no causal link between ACC activity and cognitive control.• Individual differences. In all of the studies of lie detection, the results have been inconsistent across individual subjects’ brains. This might reflect different “strategies” for lying or it might reflect different confounding factors in different individuals. This is within a highly constrained population (typically, undergraduate college students). Indiv diffs will be even greater across different segments of the population.• Detecting conscious deception is not the same as finding the truth. Memory can be inaccurate. Can we distinguish true memories from false memories? Can we distinguish memories for real events from memories for imagined or fantasized events? Mental illness can make all this particularly complicated. More on this from Dan Schacter tomorrow.• Verifiability. For visual motion perception, there is an objectively correct answer for the classification on each trial and we know what that answer is. For lie detection, it is not clear that it would be possible to design a study to validate a lie detection technology. It would be relatively easy to show that it doesnt work using contrived situations. To show convincingly that it does work, however, you need to use real-world situations. But then, you dont know if someone is lying or telling the truth so there is no external validity.

Neuro-evidence checklist Visual motion perception Lie detectionClassification accuracy 90-95% 88%Correlational evidence yes yesCausal evidence yes noQuantitative theory yes noWell-defined neural pathway yes noConsistency across individuals yes noExternal validity & verifiability yes noPotential impact on law & society no yes

Mind reading and cognitive freedom Davatzikos et al., 2005: "Moreover, since the classification method is not specific to lie detection, it could ultimately be used to a very broad range of applications in which the state of mind is to be inferred from spatio-temporal patterns of brain activity." •Is mind reading a new kind of evidence, is it physical evidence (the physical state of the brain), or is it like verbal testimony? •To the extent that a brain measurement yields information about the state of the brain that is highly correlated with state-of-mind, should it be protected based on privacy considerations? •To the extent that a brain measurement is not highly correlated with state-of-mind, it will not be reliable.Many of the concerns about reliability might be mitigated by improving our understanding of how lying works in the brain. Even the issue of individual differencesand different strategies for lying might become well understood. But that leaves us with another dilemma: should state-of-mind be considered private?On the one hand, we should be concerned about misapplication and preadoption of a technique that is not reliable. On the other hand, we should be concernedabout invading the privacy of someone’s mind. Does a person have a right to keep his or her subjective thoughts private, the right to cognitive freedom?

Selected reading • Davatzikos C, Ruparel K, Fan Y, Shen DG, Acharyya M, Loughead JW, Gur RC, Langleben DD (2005) Classifying spatial patterns of brain activity with machine learning methods: application to lie detection. Neuroimage 28:663-668. • Dubin M (2002) How the Brain Works. Blackwell Science. • Gazzaniga MS, Ivry RB, Mangun GR (2002) Cognitive Neuroscience. W. W. Norton & Company. • Hasson U, Nir Y, Levy I, Fuhrmann G, Malach R (2004) Intersubject synchronization of cortical activity during natural vision. Science 303:1634-1640. • Heeger DJ, Ress D (2002) What does fMRI tell us about neuronal activity? Nature Reviews Neuroscince 3:142-151. • Wolpe PR, Foster KR, Langleben DD (2005) Emerging neurotechnologies for lie- detection: promises and perils. Am J Bioeth 5:39-49. • Cephos Corporation (http://www.cephoscorp.com/index.html) • No Lie MRI, Inc. (http://www.noliemri.com/) • “Mind Reading” competition (http://www.ebc.pitt.edu/competition.html)

Mind Reading: Neuroscience Methods and Technology

by marina761 on Nov 24, 2010

Accessibility

Categories

Upload Details

Usage Rights

2 Embeds 13

Statistics

Mind Reading: Neuroscience Methods and Technology Document Transcript

http://psychickeobtezovani.webnode.cz	11
http://cms.psychickeobtezovani.webnode.cz	2