Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
You thought what?! The promise of real-time
brain decoding
Ted Willke
Intel Labs
2
Alvarez&Oliva,2006
BUILDINGS PEOPLE
What is attention?
“Everyoneknowswhatattentionis. Itisthetaking
possessionbythemind,inclearandvividform,of
oneoutofwhatsee...
The great controller
Perception
MemoryLearning
Attention 4CourtesyofNickTurk-Browne,Princeton
Perception
5
The brain: The black box at the end of our necks
• Facts:
 Only 2% of body weight but
uses up to 20% of energy
 ~200B ...
stimulus
(task)
mind brain dataset?
what is present in
the mind as the task
is performed?
AdaptedfromFranciscoPereira,Botv...
Non-invasive neuroimaging
7
Electrical phenomena Metabolic phenomena
Positron
Emission
Tomography
Functional
Magnetic
Reso...
Real-Time Functional MRI (rtfMRI)
8
metabolic brain
anatomical brain
Adapted from graphic by JeremyManning,Princeton
stimulus
(task)
mind brain rtfMRI
classifier
conclusions from
structure of the
learnt model
conclusions from
feature choic...
Studying attention | dueling categories
%BOLDchange
Time
Face attention
Scene (place)
attention
Fusiformface
area(FFA)
Par...
Studying attention | coupling hypothesis
Occipital cortex Ventral temporal cortex
V4
FFA
PPA
r
Al-Aidroosetal.,2012,ProcNa...
Studying attention | coupling hypothesis
Al-Aidroosetal.,2012,ProcNatlAcadSci
Faceattention
Sceneattention
N = 7, *p < .05...
13
Standard types of fMRI analysis. (A) Univariate activation refers to the average
amplitude of BOLD activity evoked by e...
14
Standard types of fMRI analysis. (A) Univariate activation refers to the average
amplitude of BOLD activity evoked by e...
Offline fMRI image analysis experiment
data acquisition preprocessing
classifier testinganalyze results
Processing time …
...
16
real-time brain decoding for
causal experimentation
Studying attention | real-time neurofeedback
Attendtoscene
MORE
sceneevidence
LESS
sceneevidence
Rewarded with stronger
st...
data acquisition real-time preprocessing
classifier testingupdate stimulus display
Processing time …
6 to 55 hours
real-ti...
Studying attention | training and scoring
Neurofeedback
Use multivariate pattern analysis (MVPA) over whole-brain
activity...
20
Subject
Scanner
Scoring sequence – your brain on scenes?
21
22
This was done with MVPA. We’d also like to try
FCMA to include connectivity information, but...
A Big Data/HPC challenge
Some facts:
 To keep up with the rtfMRI scanner, must
process full brain scan and provide feedba...
Machine Learning Workload Convergence
24
Education
Health
Banking
Manufacturing
Usages Workloads
Machine
Learning
Algorith...
25
Intel® Math Kernel Library (Intel® MKL)
Random Number Gen.
• Congruential
• Wichmann-Hill
• Mersenne Twister
• Sobol
• ...
Unveiling Details of Knights Landing
(Next Generation Intel® Xeon Phi™ Products)
2nd half ’15
1st commercial systems
3+ TF...
FCMA Correlation Computation
27
voxels
voxels
scan
data
scan
data
Correlations
Need Pearson’s correlation coefficient for
...
FCMA Z-Score Computation
28
Correlations
Need to complete Z-score procedure across all
correlation matrices produced by a ...
Putting it all together: FCMA Z-score example
29
#pragma omp parallel for
for(int v = 0 ; v < step*nSubs ; v++)
{
int s = ...
FCMA SVM
30
Correlationwithvoxelvi Subjects, trials
Key is to find the most predictive voxels in the
correlation matrix
• ...
FCMA – Effect of Optimizations
31
0
1
2
3
4
5
6
7
Correlation
Z-score
SVM
Total
Correlation
Z-score
SVM
Total
Xeon Xeon Ph...
32
Model-based approaches
33
stimulus
(task)
mind brain rtfMRI
classifier
conclusions from
structure of the
learnt model
conclusions from
feature ch...
34
stimulus
(task)
mind brain rtfMRI
classifier
Adapted from Francisco Pereira, Botvinick Lab, Princeton
35
stimulus
(task)
mind brain rtfMRI
model
Adapted from Francisco Pereira, Botvinick Lab, Princeton
predicted
stimulus
or ...
36
stimulus
(task)
mind brain rtfMRI
model
Adapted from Francisco Pereira, Botvinick Lab, Princeton
predicted
rtfMRI
data
37
Modeling | Topographic Factor Analysis
Manning JR, Ranganath R, Norman KA, Blei DM (2014) Topographic Factor Analysis: ...
38
Modeling | Topographic Factor Analysis
Manning JR, Ranganath R, Norman KA, Blei DM (2014) Topographic Factor Analysis: ...
39
Modeling | Topographic Factor Analysis
Manning JR, Ranganath R, Norman KA, Blei DM (2014) Topographic Factor Analysis: ...
40
Modeling | Topographic Factor Analysis
Manning JR, Ranganath R, Norman KA, Blei DM (2014) Topographic Factor Analysis: ...
41
Modeling | Topographic Factor Analysis
Manning JR, Ranganath R, Norman KA, Blei DM (2014) Topographic Factor Analysis: ...
42
... is a work in progress....
 more basic neuroscience
research
 more machine learning
speed and accuracy
 a look at...
43
Conclusions
 Closed-loop rtfMRI amplifies and
externalizes internal states that are difficult
to access
 Holds promis...
Thanks Princeton Neuroscience Institute!
Jon Cohen — PNI Co-Founder, Professor of Neuroscience and Psychology
Matt Botvini...
Upcoming SlideShare
Loading in …5
×

Ted Willke, Senior Principal Engineer, Intel Labs at MLconf NYC

1,781 views

Published on

You Thought What?! The Promise of Real-Time Brain Decoding: What can faster machine learning and new model-based approaches tell us about what someone is really thinking? Recently, Intel joined up with some of the pioneers of brain decoding to understand exactly that. Using functional MRI as our microscope, we began analyzing large amounts of high-dimensional 4-D image data to uncover brain networks that support cognitive processes. But existing image preprocessing, feature selection, and classification techniques are too slow and inaccurate to facilitate the most exciting breakthroughs. In this talk, we’ll discuss the promise of accurate real-time brain decoding and the computational headwinds. And we’ll look at some of the approaches to algorithms and optimization that Intel Labs and its partners are taking to reduce the barriers.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Ted Willke, Senior Principal Engineer, Intel Labs at MLconf NYC

  1. 1. You thought what?! The promise of real-time brain decoding Ted Willke Intel Labs
  2. 2. 2 Alvarez&Oliva,2006 BUILDINGS PEOPLE
  3. 3. What is attention? “Everyoneknowswhatattentionis. Itisthetaking possessionbythemind,inclearandvividform,of oneoutofwhatseemseveralsimultaneously possibleobjectsortrainsofthought... Itimplies withdrawalfromsomethingsinordertodeal effectivelywithothers...” –WilliamJames(1890) Asimplebutimportantdistinction: • Overtattention:movingyoureyes • Covertattention:movingyourmind’seye CourtesyofNickTurk-Browne,Princeton 3
  4. 4. The great controller Perception MemoryLearning Attention 4CourtesyofNickTurk-Browne,Princeton Perception
  5. 5. 5 The brain: The black box at the end of our necks • Facts:  Only 2% of body weight but uses up to 20% of energy  ~200B neurons  Neurons fire up to ~10 kHz  1K to 10K connections per neuron • The cerebral neocortex (the “mammalian brain” associated with higher reasoning):  ~20B neurons  ~125 trillion synapses There are more ways to organize the neocortex’s ~125 trillion synapses than stars in the known universe.
  6. 6. stimulus (task) mind brain dataset? what is present in the mind as the task is performed? AdaptedfromFranciscoPereira,BotvinickLab,Princeton computational model? what is attended to in the mind as the task is performed? 6
  7. 7. Non-invasive neuroimaging 7 Electrical phenomena Metabolic phenomena Positron Emission Tomography Functional Magnetic Resonance Imaging (fMRI) Magneto- Encephalography (MEG) Consumer EEG (<sensors) Near-Infrared Spectroscopy (fNIRs) Betterspatial resolution Lab/Medical EEG (>sensors) Varying portability, temporal & spatial resolution. fMRI is the workhorse of brain research despite disadvantages of non-portability & expense
  8. 8. Real-Time Functional MRI (rtfMRI) 8 metabolic brain anatomical brain Adapted from graphic by JeremyManning,Princeton
  9. 9. stimulus (task) mind brain rtfMRI classifier conclusions from structure of the learnt model conclusions from feature choice  weights on features  hidden layers  voxel location  voxel behavior  time within trial dependent on prediction model dependent on experiment AdaptedfromFranciscoPereira,BotvinickLab,Princeton 9
  10. 10. Studying attention | dueling categories %BOLDchange Time Face attention Scene (place) attention Fusiformface area(FFA) Parahippocampal placearea(PPA) e.g.,O’Cravenetal.,1999,Nature 10
  11. 11. Studying attention | coupling hypothesis Occipital cortex Ventral temporal cortex V4 FFA PPA r Al-Aidroosetal.,2012,ProcNatlAcadSci 11
  12. 12. Studying attention | coupling hypothesis Al-Aidroosetal.,2012,ProcNatlAcadSci Faceattention Sceneattention N = 7, *p < .05,**p < .01 12
  13. 13. 13 Standard types of fMRI analysis. (A) Univariate activation refers to the average amplitude of BOLD activity evoked by events of an experimental condition. N B Turk-Browne Science 2013;342:580-584 *BOLD: blood oxygenation level–dependent (BOLD) contrast imaging
  14. 14. 14 Standard types of fMRI analysis. (A) Univariate activation refers to the average amplitude of BOLD activity evoked by events of an experimental condition. N B Turk-Browne Science 2013;342:580-584 *MVPA: Multivariate Pattern Analysis *FCMA: Full Correlation Matrix Analysis, Advanced Analysis MVPA FCMA Basic (i.e. common) Analysis
  15. 15. Offline fMRI image analysis experiment data acquisition preprocessing classifier testinganalyze results Processing time … 6 to 55 hours voxel analysis 15CourtesyofNickTurk-Browne,Princeton
  16. 16. 16 real-time brain decoding for causal experimentation
  17. 17. Studying attention | real-time neurofeedback Attendtoscene MORE sceneevidence LESS sceneevidence Rewarded with stronger stimulus and easier task Punished with degraded stimulus and harder task Starting stimulus 17CourtesyofNickTurk-Browne,Princeton
  18. 18. data acquisition real-time preprocessing classifier testingupdate stimulus display Processing time … 6 to 55 hours real-time voxel analysis Closed-loop rtfMRI neurofeedback system 18
  19. 19. Studying attention | training and scoring Neurofeedback Use multivariate pattern analysis (MVPA) over whole-brain activity to decode attention to faces vs. scenes Mean cross-validation accuracy = 78% *** Norman etal.(2006),LaConte (2011)Regularizedlogistic regression (penalty=1),***p<0.001 19
  20. 20. 20 Subject Scanner
  21. 21. Scoring sequence – your brain on scenes? 21
  22. 22. 22 This was done with MVPA. We’d also like to try FCMA to include connectivity information, but...
  23. 23. A Big Data/HPC challenge Some facts:  To keep up with the rtfMRI scanner, must process full brain scan and provide feedback in <1sec (for a 2sec TR)  Raw image data for 1 subject, ~480 Gbytes  Some studies train on 100’s of subjects  If we run correlations across all subjects involves a lot of data movement  Processing is expensive:  N~100K voxels per time slice  O(N2) for basic preprocessing (minutes today)  O(N3) to process the full correlation matrix (hours today) Raw fMRI Data Patterns of correlated voxels Image Sources: Princeton Neuroscience Institute and Wikipedia “Train classifier on 100’s of subjects during coffee break, classify a subject’s patterns in <1sec.” 23
  24. 24. Machine Learning Workload Convergence 24 Education Health Banking Manufacturing Usages Workloads Machine Learning Algorithms High-level Libraries Primitives Low-level Libraries Hardware Platforms Xeon Xeon Phi Xeon FPGA Xeon Gfx Add-in card New ISATransportation Building Blocks Intel can help accelerate a wide range of machine learning through a focus on key building blocks.
  25. 25. 25 Intel® Math Kernel Library (Intel® MKL) Random Number Gen. • Congruential • Wichmann-Hill • Mersenne Twister • Sobol • Neiderreiter • Non-deterministic Summary Statistics • Kurtosis • Variation coefficient • Quantiles • Order statistics • Min/max • Variance-covariance Data Fitting • Spline-based • Interpolation • Cell search Linear Algebra • BLAS, Sparse BLAS • LAPACK solvers • Sparse Solvers (DSS, PARADISO) • Iterative solver (RCI) • ScaLAPACK, PBLAS Fast Fourier Transforms • Multidimensional • FFTW interfaces • Cluster FFT • Trig. Transforms • Poisson solver • Convolution via VSL Vector Math • Trigonometric • Hyperbolic • Exponential, Logarithmic • Power / Root
  26. 26. Unveiling Details of Knights Landing (Next Generation Intel® Xeon Phi™ Products) 2nd half ’15 1st commercial systems 3+ TFLOPS In One Package Parallel Performance & Density On-Package Memory:  up to 16GB at launch  5X Bandwidth vs DDR4 Compute: Energy-efficient IA cores  Microarchitecture enhanced for HPC  3X Single Thread Performance vs Knights Corner  Intel Xeon Processor Binary Compatible  1/3X the Space  5X Power Efficiency . . . . . . Integrated Fabric Intel® Silvermont Arch. Enhanced for HPC Processor Package Conceptual—Not Actual Package Layout … Platform Memory: DDR4 Bandwidth and Capacity Comparable to Intel® Xeon® Processors Jointly Developed with Micron Technology 26
  27. 27. FCMA Correlation Computation 27 voxels voxels scan data scan data Correlations Need Pearson’s correlation coefficient for each pair of voxels  34,470 voxels => over 500 million pairs Functionality provided by Intel’s libraries  If scan data is normalized (mean-centered and unit norm) then Pearson correlation becomes matrix multiplication  Can use single-precision general matrix multiplication (SGEMM) built into Intel Math Kernel Library (MKL)  Current work is to improve SGEMM performance when computing with small numbers of scans (e.g. 12) ThankstoMikeAnderson,IntelLabs
  28. 28. FCMA Z-Score Computation 28 Correlations Need to complete Z-score procedure across all correlation matrices produced by a single subject  Fisher transformation of each correlation coefficient => 0.5* ln((1+x)/(1-x))  Then , at each location in correlation matrix, subtract mean and divide by standard deviation across all correlation matrices Acceleration using Single Instruction Multiple Data (SIMD) instructions  Correlation coefficients are grouped into contiguous vectors and processed using SIMD instructions to exploit data parallelism  Loop annotated with #pragma simd  Natural logarithm can also be vectorised using Intel Short Vector Math Library (SVML) to accelerate Fisher transformation voxels voxels ThankstoMikeAnderson,IntelLabs
  29. 29. Putting it all together: FCMA Z-score example 29 #pragma omp parallel for for(int v = 0 ; v < step*nSubs ; v++) { int s = v % nSubs; // subject id int i = v / nSubs; // voxel id float (*mat)[row] = (float(*)[row])&(voxels->corr_vecs[i*nTrials*row]); #pragma simd for(int j = 0 ; j < row ; j++) { float mean = 0.0f; float std_dev = 0.0f; for(int b = s*nPerSub; b < (s+1)*nPerSub; b++) { _mm_prefetch((char*)&(mat[b][j+32]), _MM_HINT_ET1); _mm_prefetch((char*)&(mat[b][j+16]), _MM_HINT_T0); float num = 1.0f + mat[b][j]; float den = 1.0f - mat[b][j]; num = (num <= 0.0f) ? 1e-4 : num; den = (den <= 0.0f) ? 1e-4 : den; mat[b][j] = 0.5f * logf(num/den); mean += mat[b][j]; std_dev += mat[b][j] * mat[b][j]; } mean = mean / (float)nPerSub; std_dev = std_dev / (float)nPerSub - mean*mean; float inv_std_dev = (std_dev <= 0.0f) ? 0.0f : 1.0f / sqrt(std_dev); for(int b = s*nPerSub; b < (s+1)*nPerSub; b++) { mat[b][j] = (mat[b][j] - mean) * inv_std_dev; } } } }  Several MPI processes running the above code  OpenMP divides independent voxels (dim1) and subjects across 60 Xeon Phi Cores  #pragma simd directive assigns consecutive voxels (dim2) to vector lanes voxels voxels ThankstoMikeAnderson,IntelLabs
  30. 30. FCMA SVM 30 Correlationwithvoxelvi Subjects, trials Key is to find the most predictive voxels in the correlation matrix • Rows of the correlation matrix are the feature vectors Very large number of SVMs are trained • One for each voxel - O(35000) • Each trained SVM is cross validated and the top few voxels are chosen for predictive analyses Acceleration using custom SVM code • Kernel matrix precomputed as #dimensions << #data points • Ported parallel GPUSVM code to run on Xeon and Xeon Phi platforms • Uses thread level and SIMD parallelism • Faster than libSVM ThankstoNarayananSundaram,IntelLabs
  31. 31. FCMA – Effect of Optimizations 31 0 1 2 3 4 5 6 7 Correlation Z-score SVM Total Correlation Z-score SVM Total Xeon Xeon Phi Runtimeinseconds(for17subjects) Before optimizations After optimizations 1.7X speedup on Xeon 5.8X speedup on Xeon Phi Xeon Phi 2.1X faster than Xeon ThankstoYidaWang,Princeton,andNarayananSundaram
  32. 32. 32 Model-based approaches
  33. 33. 33 stimulus (task) mind brain rtfMRI classifier conclusions from structure of the learnt model conclusions from feature choice  weights on features  hidden layers  voxel location  voxel behavior  time within trial dependent on prediction model dependent on experiment Adapted from Francisco Pereira, Botvinick Lab, Princeton
  34. 34. 34 stimulus (task) mind brain rtfMRI classifier Adapted from Francisco Pereira, Botvinick Lab, Princeton
  35. 35. 35 stimulus (task) mind brain rtfMRI model Adapted from Francisco Pereira, Botvinick Lab, Princeton predicted stimulus or task
  36. 36. 36 stimulus (task) mind brain rtfMRI model Adapted from Francisco Pereira, Botvinick Lab, Princeton predicted rtfMRI data
  37. 37. 37 Modeling | Topographic Factor Analysis Manning JR, Ranganath R, Norman KA, Blei DM (2014) Topographic Factor Analysis: A Bayesian Model for Inferring Brain Networks from Neural Data. PLoS ONE 9(5): e94914. doi:10.1371/journal.pone.0094914 http://127.0.0.1:8081/plosone/article?id=info:doi/10.1371/journal.pone.0094914
  38. 38. 38 Modeling | Topographic Factor Analysis Manning JR, Ranganath R, Norman KA, Blei DM (2014) Topographic Factor Analysis: A Bayesian Model for Inferring Brain Networks from Neural Data. PLoS ONE 9(5): e94914. doi:10.1371/journal.pone.0094914 http://127.0.0.1:8081/plosone/article?id=info:doi/10.1371/journal.pone.0094914
  39. 39. 39 Modeling | Topographic Factor Analysis Manning JR, Ranganath R, Norman KA, Blei DM (2014) Topographic Factor Analysis: A Bayesian Model for Inferring Brain Networks from Neural Data. PLoS ONE 9(5): e94914. doi:10.1371/journal.pone.0094914 http://127.0.0.1:8081/plosone/article?id=info:doi/10.1371/journal.pone.0094914  N trials  V voxels  voxel activations y  K shared sources (µ, )  weights w
  40. 40. 40 Modeling | Topographic Factor Analysis Manning JR, Ranganath R, Norman KA, Blei DM (2014) Topographic Factor Analysis: A Bayesian Model for Inferring Brain Networks from Neural Data. PLoS ONE 9(5): e94914. doi:10.1371/journal.pone.0094914 http://127.0.0.1:8081/plosone/article?id=info:doi/10.1371/journal.pone.0094914  number of sources?  specification of sources?  hyperparameter values?  initialization of sources?
  41. 41. 41 Modeling | Topographic Factor Analysis Manning JR, Ranganath R, Norman KA, Blei DM (2014) Topographic Factor Analysis: A Bayesian Model for Inferring Brain Networks from Neural Data. PLoS ONE 9(5): e94914. doi:10.1371/journal.pone.0094914 http://127.0.0.1:8081/plosone/article?id=info:doi/10.1371/journal.pone.0094914 “mental state” mn during nth trial gives rise to behavioral data bn and neural data yn
  42. 42. 42 ... is a work in progress....  more basic neuroscience research  more machine learning speed and accuracy  a look at other model- based methods Decoding your thoughts...
  43. 43. 43 Conclusions  Closed-loop rtfMRI amplifies and externalizes internal states that are difficult to access  Holds promise for people that suffer from mental disorders or simply want to improve brain performance  Intel is helping put the rt into rtfMRI and unlock the potential of this research
  44. 44. Thanks Princeton Neuroscience Institute! Jon Cohen — PNI Co-Founder, Professor of Neuroscience and Psychology Matt Botvinick — Professor of Neuroscience and Psychology Ken Norman — Professor of Neuroscience and Psychology Nick Turk-Browne — Professor of Neuroscience and Psychology Kai Li — Professor of Computer Science and Co-Founder of Data Domain Corporation 44

×