Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
1
WHISPERS
OF
SPECKLES
DEBDOOT SHEET
LAUNCHING
THIS MONSOON
Venue: IIT Mandi
Date: Thursday, 25 June 2015
Time: 11 am – 12...
Whispers of Speckle
Part I: Building Computational Imaging
Frameworks for Acoustic and Optical
Speckle Imaging
Dr. Debdoot...
Inspiration
“A wonderful fact to reflect
upon, that every human
creature is constituted to be
that profound secret and
mys...
Motivation
Whispers of Speckles [Debdoot Sheet] - WMLMIA 4
D. Sheet (2014), PhD Thesis
25 June 2015
Text books
R. K. Das (...
Introduction
• Human body consists of organs and
systems made up of different tissues.
• Pathological conditions and
abnor...
ACHIEVING IN SITU HISTOLOGY OF
VASCULAR PLAQUES
625 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA
• Atherosclerosis
– Plaque builds up in arteries
– Forms anywhere in the vascular system
• Cardiovascular diseases (CVD)
•...
Backdrop
8
White light source
350nm 750nm
Power
Stained tissue section
Tissue specific spectrum
350nm 750nm
Power
Calcifie...
9
White light source
350nm 750nm
Power
Stained tissue section
Tissue specific spectrum
350nm 750nm
Power
Calcified
Fibroti...
10
Computation Modelling of Tissue Energy
Interaction for In situ Histopathology
Computed histology
 : Probing energy (Ac...
Limited Resolution Challenge
11
r1
r2
r3
P. M. Shankar, “A general statistical model for ultrasonic backscattering from ti...
Statistical Physics in Acoustic Imaging
r1
r2
r3
r1
r2
m=0.5
Ω1
Ω2
r
P(r)
m=1.0
Ω1
Ω2
Ω3
r
P(r)
P. M. Shankar, “A general ...
Statistical physics of ultrasonic backscattering
Lipidic
r
P(r)
Fibrotic
r
P(r)
Calcified
r
P(r)
V. Dumane and P. M. Shank...
   
 
     
 






















32121
221121
,,, 1
,
1
,,
1
,
111
)(,|,|
...
Proposed Solution
Statistical physics model of ultrasonic backscattering
Set of signal received by the transducer
Training...
HOW TO DEAL WITH THIS AS A
MACHINE LEARNING CHALLENGE?
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 16
Learning?
A computer program is said to learn from
experience E with respect to some class of
tasks T and performance meas...
Demystifying Learning
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 18
Man 1 Man 2 Man 3Man 4
Great Wall logo...
How was it Learning?
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 19
Man 1 Man 2 Man 3Man 4
Great Wall logo
...
GETTING MACHINES TO LEARN
TISSUE – ENERGY INTERACTION
FOR IN SITU HISTOLOGY
25 June 2015 Whispers of Speckles [Debdoot She...
IVUS Tissue Characterization
21
Background
Lipidic
Fibrotic
Calcified
Necrosis
Iterative self-organizing
atherosclerotic t...
Ultrasound Signal Confidence
• An ultrasonic pulse as well as backscattered
echo travel along the same path through a
hete...
Transfer Learning
Framework
23
Ultrasound RF data
(i) Signal confidence
(ii) Speckle statistics
Tissue labels
 f
Learnt...
Random Forests for Learning
25 June 2015 24Whispers of Speckles [Debdoot Sheet] - WMLMIA
A. Criminisi and J. Shotton, Deci...
Experiment Design
• Data Collection:
– Columbia University, New York City, NY, USA
– Interventional Cardiologist: Dr. Step...
Ultrasonic Histology of
Atherosclerotic Plaques
• Characterization based on ultrasonic statistical physics.
• Superior mac...
Coronary Plaque Characterization
and Staging
25 June 2015 27Whispers of Speckles [Debdoot Sheet] - WMLMIA
SOME (AWESOME SCORES ON)
METRICS
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 28
Performance Evaluation
25 June 2015 29
Inter-observer variability
Intra-observer variability
Whispers of Speckles [Debdoot...
ANALYZING COMPLEXITY OF
MACHINE LEARNING
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 30
Computational Complexity
Training Complexity Testing Complexity
25 June 2015 31Whispers of Speckles [Debdoot Sheet] - WMLM...
Roles of Source Knowledge
(Features)
Feature 1
Feature2
25 June 2015 32Whispers of Speckles [Debdoot Sheet] - WMLMIA
Importance of Source Knowledge
Genuer, R., Poggi, J.-M.,
Tuleau-Malot, C., (2010).
Variable selection using
random forests...
Learning from Approximately
Labeled Minimum Samples
25 June 2015 34
Donomez, P., Lebanon, G., Balasubramanian, K., (2010)....
Variation of Residual Error in
Learning
25 June 2015 35
Learning example
Test results
Whispers of Speckles [Debdoot Sheet]...
END NOTE
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 36
Collaboration and Network
25 June 2015 37Whispers of Speckles [Debdoot Sheet] - WMLMIA
Take home message
• Different types of soft tissues have characteristic response
when interacting with acoustic energy.
• ...
Whispers of Speckle
Part II: Enlightenment from Shallow to
Complex Reasoning with Deep Learning
Dr. Debdoot Sheet
Assistan...
DOES THIS METHOD OF TRANSFER
LEARNING APPLY ONLY TO
ULTRASONIC IMAGING?
25 June 2015 Whispers of Speckles [Debdoot Sheet] ...
Skin• Skin forms the general covering of the
body protecting us from external
influences.
• Functions
– Thermoregulation
–...
Skin
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 42
• In situ investigation
– Optical Coherence Tomography ...
Tissue Photon Interaction
Whispers of Speckles [Debdoot Sheet] - WMLMIA 43
Incident
radiation
Regular
reflection Diffuse
r...
Optical Coherence Tomography
Whispers of Speckles [Debdoot Sheet] - WMLMIA 44
Low time-coherence
light source
Depth scan m...
TPI in Swept Source OCT
Whispers of Speckles [Debdoot Sheet] - WMLMIA 45
Source
Ballistic
backscattering
Non-ballistic
bac...
COMPUTATIONAL OPTICAL
COHERENCE HISTOLOGY
THROUGH TRANSFER LEARNING
4625 June 2015 Whispers of Speckles [Debdoot Sheet] - ...
Framework
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 47
Learn TPI Model
Training Image Ground Truth Labels...
Computational Histology of Skin
• Solution through a transfer learning
approach
• Performance benchmark (Accuracy)
– Epith...
In situ Histology
of Skin
OCT
Histo
Epithelium
Epithelium
Papillary dermis
Dermis
Adipose tissue
25 June 2015 49
Papillary...
In vitro
validation
towards
In vivo
translation
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 50
Transfer Lea...
Computational Histology of Retina
• Transfer learning approach
– Retinal OCT tissue labeling
• Performance benchmark (Accu...
DOES SOMETHING LOOK
FISHY?
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 52
State of the Art
• In situ Histology with OCT
– G. van Soest et al., (2010), G.
J. Ughi et al., (2013) –
Cardiovascular OC...
Heuristics in State of Art
Whispers of Speckles [Debdoot Sheet] - WMLMIA 5425 June 2015
(RE)EXPLORING THE CONCEPTS OF
HIERARCHY IN LEARNING
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 55
How was it Learning?
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 56
Man 1 Man 2 Man 3Man 4
Great Wall logo
...
Challenges
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 57
Salient Segments
Objectify
Detect
humans
Recogniz...
Challenges
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 58
Salient Segments
Objectify
Recognize
inanimate
De...
FROM SHALLOW TO COMPLEX
REASONING
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 59
Heuristics in State of Art
Whispers of Speckles [Debdoot Sheet] - WMLMIA 6025 June 2015
The Solution
Whispers of Speckles [Debdoot Sheet] - WMLMIA 61
DenoisingAutoEncoder
DenoisingAutoEncoder
LogisticReg.
25 Ju...
Using a Deep Network
Whispers of Speckles [Debdoot Sheet] - WMLMIA 6225 June 2015
COMPLEX REASONING AND
ITS DEEP LEARNING
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 63
Challenges
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 64
Salient Segments
Objectify
Detect
humans
Recogniz...
Challenges
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 65
Salient Segments
Objectify
Recognize
inanimate
De...
How to tackle this dilemma?
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 66
Great Wall
behind
Great Wall
log...
Multilayer Perceptron (MLP)
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 67
Hiddenlayers
Hiddenlayers
Hidden...
MLP Learning, troubles thereof
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 68
P
T1
T2
MLP Learning troubles, why so?
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 69
P
T1
T2
LBP
Wavelets
HoG
Body...
HOW TO DEEP LEARN?
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 70
Deep Learning, origin and growth
• Around 1950 – NN age
– Neural Nets (McCulloch and Pitts,
1943)
– Unsupervised Learn. (H...
Deep Learning, origin and growth
• 1980-2000 – Search for simple,
low-complexity, problem-solvers
– Recurrent Neural Netwo...
Deep Learning, origin and growth
• 2000 – Era of Deep Learning
– NIPS 2003 Feature Selection
Challenge (Neal and Zhang, 20...
DEEP LEARNING OF COMPLEX
REASONING FOR OCT TISSUE
CHARACTERIZATION
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WML...
Exploring Deep Architecture
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 75
Multi-scale
modeling of
OCT spec...
Auto Encoder for Deep Learning
Whispers of Speckles [Debdoot Sheet] - WMLMIA 7625 June 2015
Results in Wounds
Whispers of Speckles [Debdoot Sheet] - WMLMIA 77
(a) OCT image of wound (b) Ground truth (c) In situ his...
Experiment Design
• Data Collection
– School of Medical Science
and Technology, Indian
Institute of Technology
Kharagpur
–...
Learning of Representations
Whispers of Speckles [Debdoot Sheet] - WMLMIA 79
Representation of speckle
appearance models l...
END NOTE
25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 80
Messages for Human Learning
• Photons interact characteristically with different tissues.
– Stochastic similarity exists i...
About Deep Learning
“It’s like in quantum physics at the beginning of the
20th century” Trishul Chilimbi (MSR, DNN, Adam)
...
Take home message
“We’ve humanized the scientist;
now we must scientize the
humanist. We didn’t try to cover
physics... we...
Take home message
• Challenges
– Architectures
• Neural Nets vs. Others
– Implementation
• CPU vs. GPU vs. Cloud
– GPU (VL...
Upcoming SlideShare
Loading in …5
×

Whispers of Speckles ( Part I: Building Computational Imaging Frameworks for Acoustic and Optical Speckle Imaging) + (Part II: Enlightenment from Shallow to Complex Reasoning with Deep Learning)

7,945 views

Published on

Whispers of Speckles

( Part I: Building Computational Imaging Frameworks for Acoustic and Optical Speckle Imaging)

(Part II: Enlightenment from Shallow to Complex Reasoning with Deep Learning)

Presented at the Workshop on Machine Learning for Medical Image Analysis (WMLMIA), IIT Mandi, 25 June 2015.

Published in: Education
  • Be the first to comment

  • Be the first to like this

Whispers of Speckles ( Part I: Building Computational Imaging Frameworks for Acoustic and Optical Speckle Imaging) + (Part II: Enlightenment from Shallow to Complex Reasoning with Deep Learning)

  1. 1. 1 WHISPERS OF SPECKLES DEBDOOT SHEET LAUNCHING THIS MONSOON Venue: IIT Mandi Date: Thursday, 25 June 2015 Time: 11 am – 12:30 pm
  2. 2. Whispers of Speckle Part I: Building Computational Imaging Frameworks for Acoustic and Optical Speckle Imaging Dr. Debdoot Sheet Assistant Professor Department of Electrical Engineering Indian Institute of Technology Kharagpur www.facweb.iitkgp.ernet.in/~debdoot/
  3. 3. Inspiration “A wonderful fact to reflect upon, that every human creature is constituted to be that profound secret and mystery to every other.” - Charles Dickens (A Tale of Two Cities) “If you want to find the secrets of the universe, think in terms of energy, frequency and vibration.” - Nikola Tesla 25 June 2015 3Whispers of Speckles [Debdoot Sheet] - WMLMIA
  4. 4. Motivation Whispers of Speckles [Debdoot Sheet] - WMLMIA 4 D. Sheet (2014), PhD Thesis 25 June 2015 Text books R. K. Das (2012), PhD Thesis A. Barui (2011), PhD Thesis
  5. 5. Introduction • Human body consists of organs and systems made up of different tissues. • Pathological conditions and abnormalities affect their normal functioning. • Critical soft tissue abnormalities include – Plaque formation in the blood vascular system. – Lesions in the breast. – Degeneration of the retina. – Wounds in the skin. • Traditional practice of Histopathological diagnosis requires invasive Biopsy / Excision for tissue collection – Not possible in vessels in living Humans – Improper sampling from Breast lesion affects diagnostic outcome – Not possible in retina in living Humans – Not possible in healing wounds. 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 5
  6. 6. ACHIEVING IN SITU HISTOLOGY OF VASCULAR PLAQUES 625 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA
  7. 7. • Atherosclerosis – Plaque builds up in arteries – Forms anywhere in the vascular system • Cardiovascular diseases (CVD) • In vivo Imaging of Plaques – CT Angiography (CTA) – MR Angiography (MRA) – Intravascular Ultrasound (IVUS) – Intravascular OCT (IV-OCT) – Intravascular Near-Infrared Spectroscopy (NIR) • Plaque Vulnerability Assessment – Calcification, fibrosis identification – Lipid pool and Necrosis burden estimation Source: NIH – National Heart, Lung, and Blood Institute Blood Vascular System 25 June 2015 7Whispers of Speckles [Debdoot Sheet] - WMLMIA • Spectral analysis of received ultrasonic echo signal – Lizzi et al., 1983 – Nair et al., 2001 – Kawasaki et al., 2002 – Virtual Histology (Volcano Corp.) – iMap (BostonScientific) • Texture analysis of B-mode image/signal – Katouzian et al., 2008, 2010, 2012 (Prog. Hist. / PH) – Esclara et al., 2009 – Seabra et al., 2011 • Limitations – Unable to identify heterogeneous tissue composition – Cannot discriminate between dense fibrous tissue and calcification – Fails to discriminate true necrosis from shadows
  8. 8. Backdrop 8 White light source 350nm 750nm Power Stained tissue section Tissue specific spectrum 350nm 750nm Power Calcified Fibrotic 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA
  9. 9. 9 White light source 350nm 750nm Power Stained tissue section Tissue specific spectrum 350nm 750nm Power Calcified Fibrotic  : Probing energy (Light)  : Physiological property (Tissue type)   f   1  f Inferring tissue type based on color 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA
  10. 10. 10 Computation Modelling of Tissue Energy Interaction for In situ Histopathology Computed histology  : Probing energy (Acoustic)  : Tissue type (Backscatterer density)   f   1  f Inferring tissue type based on backscattering response 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA
  11. 11. Limited Resolution Challenge 11 r1 r2 r3 P. M. Shankar, “A general statistical model for ultrasonic backscattering from tissues”, IEEE Trans. Ultrasonics, Ferroelectrics, Freq. Control., vol. 47, no. 3, pp. 727-736, May 2000.  11 rr f    22 rr f    33 rr f   Ultrasound signal backscattered within a resolution cell     i i r r fE E     Signal sensed by the transducer    irfEf  1 ˆ   Estimated functional ensemble of backscatterer density  ˆ Improper estimation of tissue type in inhomogeneous media 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA
  12. 12. Statistical Physics in Acoustic Imaging r1 r2 r3 r1 r2 m=0.5 Ω1 Ω2 r P(r) m=1.0 Ω1 Ω2 Ω3 r P(r) P. M. Shankar, “A general statistical model for ultrasonic backscattering from tissues”, IEEE Trans. Ultrasonics, Ferroelectrics, Freq. Control., vol. 47, no. 3, pp. 727-736, May 2000.                2 12 exp 2 ,| r m m rm mr m mm N 25 June 2015 12Whispers of Speckles [Debdoot Sheet] - WMLMIA
  13. 13. Statistical physics of ultrasonic backscattering Lipidic r P(r) Fibrotic r P(r) Calcified r P(r) V. Dumane and P. M. Shankar, “Use of frequency diversity and Nakagami statistics in ultrasonic tissue characterization”, IEEE Trans. Ultrasonics, Ferroelectrics, Freq. Control, vol. 48, no. 4, pp. 1139-1146, Jul. 2001 F. Destrempes, J. Meunier, M. . F. Giroux, G. Soulez, G. Cloutier, “Segmentation in ultrasonic b- mode images of healthy carotid arteries using mixture of Nakagami distributions and stochastic optimization”, IEEE Trans. Med. Imaging, vol. 28, no. 2, pp. 215-229, Feb. 2009. 25 June 2015 13Whispers of Speckles [Debdoot Sheet] - WMLMIA
  14. 14.                                     32121 221121 ,,, 1 , 1 ,, 1 , 111 )(,|,| ,| ;),(,),,,,,(||        L l lll L l lll L l lll mrpmrpp mrpp ymprfyrp NN N Mathematical intractability, the problem )( )( )|( )|( yP rp yrp ryp  The probabilistic decision making framework Scales unknown Correlation among scales unknown No. components unknown Prior probab. of each comp. unknown 25 June 2015 14Whispers of Speckles [Debdoot Sheet] - WMLMIA
  15. 15. Proposed Solution Statistical physics model of ultrasonic backscattering Set of signal received by the transducer Training set of annotated examples to be used for supervised learning Supervised learner for learning tissue specific statistical physics model   train;,|)( ),( )|,( ),|( RR    yHyP rp yrp ryp Solution through Transfer Learning Framework25 June 2015 15Whispers of Speckles [Debdoot Sheet] - WMLMIA
  16. 16. HOW TO DEAL WITH THIS AS A MACHINE LEARNING CHALLENGE? 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 16
  17. 17. Learning? A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E -Tom Mitchell 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 17
  18. 18. Demystifying Learning 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 18 Man 1 Man 2 Man 3Man 4 Great Wall logo Great Wall tower Kim Jung WangDebdoot Experience (E) Performance(P) Debdoot, Kim, Jung and Wang are standing near the Great Wall logo and the Great Wall tower is behind them.
  19. 19. How was it Learning? 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 19 Man 1 Man 2 Man 3Man 4 Great Wall logo Great Wall tower Kim Jung WangDebdoot Salient Segments Objectify Detect humans Recognize inanimate Describe Scene Debdoot, Kim, Jung and Wang are standing near the Great Wall logo and the Great Wall tower is behind them. Recognize humans
  20. 20. GETTING MACHINES TO LEARN TISSUE – ENERGY INTERACTION FOR IN SITU HISTOLOGY 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 20
  21. 21. IVUS Tissue Characterization 21 Background Lipidic Fibrotic Calcified Necrosis Iterative self-organizing atherosclerotic tissue labeling in intravascular ultrasound images and comparison with virtual histology, IEEE TBME, 59(11), 2012 Hunting for necrosis in the shadows of intravascular ultrasound, CMIG, 38(2), 2014 Joint learning of ultrasonic backscattering statistical physics and signal confidence primal for characterizing atherosclerotic plaques using intravascular ultrasound, Med. Image Anal,18(1), 2014 Nakagami parameter and signal confidence estimate Random forest learning 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA
  22. 22. Ultrasound Signal Confidence • An ultrasonic pulse as well as backscattered echo travel along the same path through a heterogeneous media. • They are subjected to the same attenuation. • Confidence of the received signal is a reflection of fidelity of samples received by the transducer. • It can be estimated by treating its propagation as a random walk along an ultrasonic scan-line. • A random walker starting at a point on the scan-line reaches the virtual transducer element placed at the origin of each scan- line. • This random walk is solved using the electric network equivalent and solving it in the paradigm of graph theory. 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 22 A. Karamalis, W. Wein, T. Klein, N. Navab (2012) Ultrasound confidence maps using random walks, Medical Image Analysis, 16:1101–1112.
  23. 23. Transfer Learning Framework 23 Ultrasound RF data (i) Signal confidence (ii) Speckle statistics Tissue labels  f Learnt random forest Learning phase (offline) Tissue labels Prediction (online)  f Whispers of Speckles [Debdoot Sheet] - WMLMIA25 June 2015
  24. 24. Random Forests for Learning 25 June 2015 24Whispers of Speckles [Debdoot Sheet] - WMLMIA A. Criminisi and J. Shotton, Decision Forests for Computer Vision and Medical Image Analysis, Springer, 2013.
  25. 25. Experiment Design • Data Collection: – Columbia University, New York City, NY, USA – Interventional Cardiologist: Dr. Stephane G. Carlier – Cardiovascular Histopathologist: Dr. Renu Virmani, CV Path Institute, Gaithersburg, USA – Cases # 13 – Tissue Sections # 53 – Atlantis, 40 MHz IVUS, Boston Scientific,CA, USA – Sampling freq: 400 MHz – Sampling geometry: 256 scan lines per rotation, 2048 samples per scan line • Learning – Source task: {Ω,m} estimated at 28 scales + Ultrasonic Confidence (A. Karamalis, et al. (2012)) – Target task: Random forest 50 decision trees • Cross validation – 53 fold cross validation – Learn with 52, test on the remaining 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 25
  26. 26. Ultrasonic Histology of Atherosclerotic Plaques • Characterization based on ultrasonic statistical physics. • Superior machine learning algorithm. • Reliability measure for estimation of tissues. Probability of Calcified tissues Probability of Fibrotic tissues Probability of Lipidic tissues Probability of Necrotic tissues Calcified Lipidic Fibrotic Necrotic 25 June 2015 26Whispers of Speckles [Debdoot Sheet] - WMLMIA
  27. 27. Coronary Plaque Characterization and Staging 25 June 2015 27Whispers of Speckles [Debdoot Sheet] - WMLMIA
  28. 28. SOME (AWESOME SCORES ON) METRICS 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 28
  29. 29. Performance Evaluation 25 June 2015 29 Inter-observer variability Intra-observer variability Whispers of Speckles [Debdoot Sheet] - WMLMIA
  30. 30. ANALYZING COMPLEXITY OF MACHINE LEARNING 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 30
  31. 31. Computational Complexity Training Complexity Testing Complexity 25 June 2015 31Whispers of Speckles [Debdoot Sheet] - WMLMIA
  32. 32. Roles of Source Knowledge (Features) Feature 1 Feature2 25 June 2015 32Whispers of Speckles [Debdoot Sheet] - WMLMIA
  33. 33. Importance of Source Knowledge Genuer, R., Poggi, J.-M., Tuleau-Malot, C., (2010). Variable selection using random forests. Pat. Recog. Letters. 31(14):2225-2236 25 June 2015 33Whispers of Speckles [Debdoot Sheet] - WMLMIA
  34. 34. Learning from Approximately Labeled Minimum Samples 25 June 2015 34 Donomez, P., Lebanon, G., Balasubramanian, K., (2010). Unsupervised supervised learning I: Estimating Classification and Regression Errors without Labels, J. Mach. Learn. Res. 11:1323-1351 Whispers of Speckles [Debdoot Sheet] - WMLMIA
  35. 35. Variation of Residual Error in Learning 25 June 2015 35 Learning example Test results Whispers of Speckles [Debdoot Sheet] - WMLMIA
  36. 36. END NOTE 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 36
  37. 37. Collaboration and Network 25 June 2015 37Whispers of Speckles [Debdoot Sheet] - WMLMIA
  38. 38. Take home message • Different types of soft tissues have characteristic response when interacting with acoustic energy. • Heterogeneous tissues can be identified by learning of tissue specific energy interaction response using statistical physics models. • Transfer Learning can be employed for efficiently solving tissue characterization problems modeled as tissue-energy interaction problems. – CPU/GPU handshaking can be used for fast implementation of such tasks • Explore possibility of Functional Histopathology In situ 25 June 2015 38Whispers of Speckles [Debdoot Sheet] - WMLMIA
  39. 39. Whispers of Speckle Part II: Enlightenment from Shallow to Complex Reasoning with Deep Learning Dr. Debdoot Sheet Assistant Professor Department of Electrical Engineering Indian Institute of Technology Kharagpur www.facweb.iitkgp.ernet.in/~debdoot/
  40. 40. DOES THIS METHOD OF TRANSFER LEARNING APPLY ONLY TO ULTRASONIC IMAGING? 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 40
  41. 41. Skin• Skin forms the general covering of the body protecting us from external influences. • Functions – Thermoregulation – Sweat secretion – Tactile, pressure, temperature sensing • Stratified organization – Epidermis – Papillary dermis – Dermis – Adipose tissue • Wound – Major pathological injury – Skin is torn, cut, punctured • Clinical challenge in management – Healing in person specific – Patient specific intervention – In situ investigation of healing is challenge 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 41
  42. 42. Skin 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 42 • In situ investigation – Optical Coherence Tomography (OCT) • Cobb et al., (2006) • Barui et al., (2011) – Optical photography • Cross-sectional information about healing wound is not available – NIR imaging • Cross-section histological information not present • In situ Histology with OCT – G. van Soest et al., (2010) – Cardiovascular OCT – A. Barui et al., (2011) – Cutaneous wound beds. • Challenges – Identify co-located tissue heterogeneity – Identify and discriminate Inter-digitated structures
  43. 43. Tissue Photon Interaction Whispers of Speckles [Debdoot Sheet] - WMLMIA 43 Incident radiation Regular reflection Diffuse reflection Scattering Absorption Multispectral optical imageOCT B. Saleh, Introduction to Subsurface Imaging, Cambridge, 2011. 0.5 mm 0.5 mm 25 June 2015
  44. 44. Optical Coherence Tomography Whispers of Speckles [Debdoot Sheet] - WMLMIA 44 Low time-coherence light source Depth scan mirror Sample Detector Source beam Reference beam Sample beam Detector beam x z z OCT Image Michelson interferometer 25 June 2015
  45. 45. TPI in Swept Source OCT Whispers of Speckles [Debdoot Sheet] - WMLMIA 45 Source Ballistic backscattering Non-ballistic backscattering Reference Detector A. F. Fercher, et al, Optical coherence tomography — principles and applications, Rep. Prog. Phys. 66 (2003) 239–303 Epithelium Papillary dermis Dermis Adipose Speckle intensity Probability density 25 June 2015          S S S S I Ip  exp 1
  46. 46. COMPUTATIONAL OPTICAL COHERENCE HISTOLOGY THROUGH TRANSFER LEARNING 4625 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA
  47. 47. Framework 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 47 Learn TPI Model Training Image Ground Truth Labels Test Image Learn TPI Model Characterized tissue   train;,| II, xH
  48. 48. Computational Histology of Skin • Solution through a transfer learning approach • Performance benchmark (Accuracy) – Epithelium = 99% – Papilary dermis = 95% – Dermis = 99% – Adipose = 98% • D. Sheet, et al, “In situ histology of mice skin through transfer learning of tissue energy interaction in optical coherence tomography”, J. Biomed. Optics, 18(9), 2013. 25 June 2015 48 Multi-scale modeling of OCT speckles Training image set Ground truth Random forest learning Multi-scale modeling of OCT speckles Test image Labeled tissue Whispers of Speckles [Debdoot Sheet] - WMLMIA
  49. 49. In situ Histology of Skin OCT Histo Epithelium Epithelium Papillary dermis Dermis Adipose tissue 25 June 2015 49 Papillary dermisDermisAdipose tissueAll tissues In situ histology of mice skin through transfer learning of tissue energy interaction in optical coherence tomography, J. Biomed. Optics, 18(9), 2013 Whispers of Speckles [Debdoot Sheet] - WMLMIA
  50. 50. In vitro validation towards In vivo translation 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 50 Transfer Learning of Tissue Photon Interaction in Optical Coherence Tomography towards In vivo Histology of the Oral Mucosa, Proc. ISBI, 2014.
  51. 51. Computational Histology of Retina • Transfer learning approach – Retinal OCT tissue labeling • Performance benchmark (Accuracy) – Anterior coat > 98% – RPE > 92% – Posterior coat > 99% • SPK Karri and D. Sheet, et al., “Computational Histology of Retina through Transfer Learning of Tissue Photon Interaction in Optical Coherence Tomography”, Proc. Int. Symp. Biomedical Imaging (ISBI), 2014. 25 June 2015 51 Multi-scale modeling of OCT speckles Training image set Ground truth Random forest learning Multi-scale modeling of OCT speckles Test image Labeled tissue Whispers of Speckles [Debdoot Sheet] - WMLMIA
  52. 52. DOES SOMETHING LOOK FISHY? 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 52
  53. 53. State of the Art • In situ Histology with OCT – G. van Soest et al., (2010), G. J. Ughi et al., (2013) – Cardiovascular OCT – D. Sheet et al., (2013, 2014) – Cutaneous wounds, oral • Challenges – Heuristic features • Texture • Intensity statistics – Heuristic computational models • Transfer learning of speckle occurrence models – Incomplete representation dictionary Whispers of Speckles [Debdoot Sheet] - WMLMIA 53 Multi-scale modeling of OCT speckles Training image set Ground truth Random forest learning Multi-scale modeling of OCT speckles Test image Labeled tissue 25 June 2015
  54. 54. Heuristics in State of Art Whispers of Speckles [Debdoot Sheet] - WMLMIA 5425 June 2015
  55. 55. (RE)EXPLORING THE CONCEPTS OF HIERARCHY IN LEARNING 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 55
  56. 56. How was it Learning? 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 56 Man 1 Man 2 Man 3Man 4 Great Wall logo Great Wall tower Kim Jung WangDebdoot Salient Segments Objectify Detect humans Recognize inanimate Describe Scene Debdoot, Kim, Jung and Wang are standing near the Great Wall logo and the Great Wall tower is behind them. Recognize humans
  57. 57. Challenges 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 57 Salient Segments Objectify Detect humans Recognize inanimate Describe Scene Recognize humans Salient Segments Objectify Detect humans Recognize inanimate Describe Scene Recognize humans Salient Segments Objectify Detect humans Recognize inanimate Describe Scene Recognize humans Salient Segments Describe Scene
  58. 58. Challenges 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 58 Salient Segments Objectify Recognize inanimate Describe Scene Recognize humans LBP Wavelets HoG Body part recognition Human appearance Chroma clustering Posture realign Silhouette matching Recognize human Detect humans
  59. 59. FROM SHALLOW TO COMPLEX REASONING 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 59
  60. 60. Heuristics in State of Art Whispers of Speckles [Debdoot Sheet] - WMLMIA 6025 June 2015
  61. 61. The Solution Whispers of Speckles [Debdoot Sheet] - WMLMIA 61 DenoisingAutoEncoder DenoisingAutoEncoder LogisticReg. 25 June 2015
  62. 62. Using a Deep Network Whispers of Speckles [Debdoot Sheet] - WMLMIA 6225 June 2015
  63. 63. COMPLEX REASONING AND ITS DEEP LEARNING 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 63
  64. 64. Challenges 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 64 Salient Segments Objectify Detect humans Recognize inanimate Describe Scene Recognize humans Salient Segments Objectify Detect humans Recognize inanimate Describe Scene Recognize humans Salient Segments Objectify Detect humans Recognize inanimate Describe Scene Recognize humans Salient Segments Describe Scene
  65. 65. Challenges 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 65 Salient Segments Objectify Recognize inanimate Describe Scene Recognize humans LBP Wavelets HoG Body part recognition Human appearance Chroma clustering Posture realign Silhouette matching Recognize human Detect humans
  66. 66. How to tackle this dilemma? 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 66 Great Wall behind Great Wall logo beside Debdoot, Kim, Jung, Wang
  67. 67. Multilayer Perceptron (MLP) 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 67 Hiddenlayers Hiddenlayers Hiddenlayers
  68. 68. MLP Learning, troubles thereof 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 68 P T1 T2
  69. 69. MLP Learning troubles, why so? 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 69 P T1 T2 LBP Wavelets HoG Body part recognition Human appearance Chroma clustering Posture realign Silhouette matching Recognize human?
  70. 70. HOW TO DEEP LEARN? 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 70
  71. 71. Deep Learning, origin and growth • Around 1950 – NN age – Neural Nets (McCulloch and Pitts, 1943) – Unsupervised Learn. (Hebb, 1949) – Supervised Learn. (Rosenblatt, 1958) – Associative Memory (Palm, 1980; Hopfield, 1982) • 1960 – Discovery of visual sensory cells that respond to Edges (Hubel and Wiesel, 1962) – Feed Forward Multi Layer Perceptron (FF-MLP) (Ivakhnenko, 1968) • 1980 – Neocognition – Convolution + WeightReplication + Subsampling (Fukushima, 1980) – Max Pooling – Back-propagation (Werbos, 1981; LeCunn, 1985, 1988) 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 71
  72. 72. Deep Learning, origin and growth • 1980-2000 – Search for simple, low-complexity, problem-solvers – Recurrent Neural Network (RNN) (Hochreiter and Schmidhuber, 1996) – Local learning Feed forward NN (Dayan and Hinton, 1996) – Advanced gradient descent (Schaback and Werner, 1992) – Sequential Network Construction (Honavar and Uhr, 1988) – Unsupervised Pre-training (Ritter and Kohonen, 1989) – Auto-Encoder (Hinton et al., 1989) – Back Propagating Convolutional Neural Networks (LeCun et al., 1989, 1990a, 1998) 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 72
  73. 73. Deep Learning, origin and growth • 2000 – Era of Deep Learning – NIPS 2003 Feature Selection Challenge (Neal and Zhang, 2006) – MNIST digit recognition (LeCun et al., 1989) – Deep Belief Network (DBN) / Restricted Boltzmann Machines (Hinton et al., 2006) – Auto Encoders (Bengio, 2009) • 2006 – GPU based CNN (Chellapilla et al., 2006) • 2009 – GPU DBN (Raina et al., 2009) • 2011 – Max-Pooling CNN on the GPU (Ciresan et al., 2011) • 2012 – Image Net (Krizhevsky et al., 2012) 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 73
  74. 74. DEEP LEARNING OF COMPLEX REASONING FOR OCT TISSUE CHARACTERIZATION 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 74
  75. 75. Exploring Deep Architecture 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 75 Multi-scale modeling of OCT speckles Training image set Ground truth Random forest learning Multi-scale modeling of OCT speckles Test image Labeled tissue Stacked Auto- Encoders, Logistic Regression Random Forest Training image set Ground truth http://www.facweb.iitkgp.ernet.in/~debdoot/current.html
  76. 76. Auto Encoder for Deep Learning Whispers of Speckles [Debdoot Sheet] - WMLMIA 7625 June 2015
  77. 77. Results in Wounds Whispers of Speckles [Debdoot Sheet] - WMLMIA 77 (a) OCT image of wound (b) Ground truth (c) In situ histology Epithelium, Papillary dermis, Dermis, Adipose Epithelium, Papillary dermis, Dermis, Adipose 25 June 2015
  78. 78. Experiment Design • Data Collection – School of Medical Science and Technology, Indian Institute of Technology Kharagpur – 1300 nm (HPBW 100 nm) Swept Source OCT System • OCS 1300 SS, ThorLabs, NJ, USA • 8 bit bitmap images – Histology for ground truth • HE stained • Samples – Mus musculus (small mice) – 16 healthy skin – 2 wounds on skin • DNN architecture – Patch size – 36 × 36 px – DAE1 – 400 nodes – DAE2 – 100 nodes – Target – Logistic Reg. • 5 outputs – Sparsity – 20% – Mini-batch training • In situ Histology Performance – Epithelium – 96% – Papillary dermis – 93% – Dermis – 99% – Adipose tissue – 98% Whispers of Speckles [Debdoot Sheet] - WMLMIA 7825 June 2015
  79. 79. Learning of Representations Whispers of Speckles [Debdoot Sheet] - WMLMIA 79 Representation of speckle appearance models learned by DAE1 Sparsity of representations learned by DAE2 25 June 2015
  80. 80. END NOTE 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 80
  81. 81. Messages for Human Learning • Photons interact characteristically with different tissues. – Stochastic similarity exists in speckle appearance. – Such representations are hard to heuristically encode. • Deep learning and auto-encoders for computational imaging – Speckle imaging application viz. OCT tissue characterization – Hierarchical learning • Locally embedded representations. • Sparsity is in learned (auto-encoded) representations. Whispers of Speckles [Debdoot Sheet] - WMLMIA 81 Queries: Debdoot Sheet (debdoot@ee.iitkgp.ernet.in) 25 June 2015
  82. 82. About Deep Learning “It’s like in quantum physics at the beginning of the 20th century” Trishul Chilimbi (MSR, DNN, Adam) “The experimentalists and practitioners were ahead of the theoreticians. They couldn’t explain the results. We appear to be at a similar stage with DNNs. We’re realizing the power and the capabilities, but we still don’t understand the fundamentals of exactly how they work.” 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 82
  83. 83. Take home message “We’ve humanized the scientist; now we must scientize the humanist. We didn’t try to cover physics... we uncovered it.” - Robert Resnick 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 83
  84. 84. Take home message • Challenges – Architectures • Neural Nets vs. Others – Implementation • CPU vs. GPU vs. Cloud – GPU (VLSI) architectures • Hierarchical Temporal Memory • Potential Causal Connection • Toolboxes – Theano (Python/SciPy) – Pylearn2 – Torch – Caffe – Matlab (Rasmus Berg Palm) • More information – www.deeplearning.net – Schmidhuber (2014). Deep Learning in Neural Networks: An Overview (arXiv:1404.7828) – Bengio (2009). Learning Deep Architectures for AI. – Deng and Yu (2013). Deep Learning: Methods and Applications. • Conferences – Int. Conf. Learning Representations (ICLR) 25 June 2015 Whispers of Speckles [Debdoot Sheet] - WMLMIA 84

×