One approach to computerized histopathology image analysis is to leverage the multi-scale texture information resulting from single nuclei appearance to entire cell populations. In this talk, we will introduce a novel framework for learning highly adaptive texture-based local models of biomedical tissue. I will discuss our initial experience with the differentiation of brain tumor types in digital histopathology.

1. TEXTURE-BASED COMPUTATIONAL MODELS OF
TISSUE IN BIOMEDICAL IMAGES:
INITIAL EXPERIENCE WITH DIGITAL HISTOPATHOLOGY
Adrien Depeursinge, PhD
Journées GdR ISIS “Analyse de Tissu Biologique et Histopathologie Numérique”
Paris, June 23rd 2014

2. BACKGROUND – RADIOMICS - HISTOPATHOLOMICS
• Personalized medicine aims at enhancing the patient’s
quality of life and prognosis
• Tailored treatment and medical decisions based
on the molecular composition of diseased tissue
• Current limitations [1]
• Molecular analysis of tissue composition
is invasive (biopsy), slow and costly
• Cannot capture molecular heterogeneity
2
[1] Intratumor heterogeneity and branched evolution revealed by multiregion sequencing, Gerlinger et al.,
N Engl J Med, 366(10):883-92, 2012.
Intratumor Heterogeneity Reveale
B Regional Distribution of Mutations
A Biopsy Sites
SOX9
CENPN
PSMD7
RIMBP2
GALNT11
ABHD11
UGT2A1
MTOR
PPP6R2
ZNF780A
WSCD2
CDKN1B
PPFIA1
TH
SSNA1
CASP2
PLRG1
SETD2
CCBL2
SESN2
MAGEB16
NLRP7
IGLON5
KLK4
WDR62
KIAA0355
CYP4F3
AKAP8
ZNF519
DDX52
ZC3H18
TCF12
NUSAP1
X4
KDM2B
MRPL51
C11orf68
ANO5
EIF4G2
MSRB2
RALGDS
EXT1
ZC3HC1
PTPRZ1
INTS1
CCR6
DOPEY1
ATXN1
WHSC1
CLCN2
SSR3
KLHL18
SGOL1
VHL
C2orf21
ALS2CR12
PLB1
FCAMR
IFI16
BCAS2
IL12RB2
Ubiquitous Shared prima
10 cm
R7 (G4)
R5 (G4)
R9
R3 (G4)
R1 (G3) R2 (G3)
R4 (G1)
R6 (G1)
Hilum
R8 (G4)
ution of Mutations
ationships of Tumor Regions D Ploidy Profiling
PreP
PreM
R1
R2
R3
R5
R8
R9
R4
M1
M2a
M2b
C2orf85
WDR7
SUPT6H
CDH19
LAMA3
DIXDC1
HPS5
NRAP
KIAA1524
SETD2
PLCL1
BCL11A
IFNAR1
DAMTS10
C3
KIAA1267
RT4
CD44
ANKRD26
TM7SF4
SLC2A1
DACH2
MMAB
ZNF521
HMG20A
DNMT3A
RLF
MAMLD1
MAP3K6
HDAC6
PHF21B
FAM129B
RPS8
CIB2
RAB27A
SLC2A12
DUSP12
ADAMTSL4
NAP1L3
USP51
KDM5C
SBF1
TOM1
MYH8
WDR24
ITIH5
AKAP9
FBXO1
LIAS
TNIK
SETD2
C3orf20
MR1
PIAS3
DIO1
ERCC5
KL
ALKBH8
DAPK1
DDX58
SPATA21
ZNF493
NGEF
DIRAS3
LATS2
ITGB3
FLNA
SATL1
KDM5C
KDM5C
RBFOX2
NPHS1
SOX9
CENPN
PSMD7
RIMBP2
GALNT11
ABHD11
UGT2A1
MTOR
PPP6R2
ZNF780A
WSCD2
CDKN1B
PPFIA1
TH
SSNA1
CASP2
PLRG1
SETD2
CCBL2
SESN2
MAGEB16
NLRP7
IGLON5
KLK4
WDR62
KIAA0355
CYP4F3
AKAP8
ZNF519
DDX52
ZC3H18
TCF12
NUSAP1
X4
KDM2B
MRPL51
C11orf68
ANO5
EIF4G2
MSRB2
RALGDS
EXT1
ZC3HC1
PTPRZ1
Privatebiquitous Shared primary Shared metastasis
Lung
metastases
Chest-wall
metastasis
Perinephric
metastasis
M1
10 cm
R2 (G3)
R4 (G1)
R6 (G1)
HilumR8 (G4)
Primary
tumor
M2b
M2a
ution of Mutations
PreP
PreM
R1
R2
R3
R5
R8
R9
R4
M1
M2a
M2b
C2orf85
WDR7
SUPT6H
CDH19
LAMA3
DIXDC1
HPS5
NRAP
KIAA1524
SETD2
PLCL1
BCL11A
IFNAR1
DAMTS10
C3
KIAA1267
RT4
CD44
ANKRD26
TM7SF4
SLC2A1
DACH2
MMAB
ZNF521
HMG20A
DNMT3A
RLF
MAMLD1
MAP3K6
HDAC6
PHF21B
FAM129B
RPS8
CIB2
RAB27A
SLC2A12
DUSP12
ADAMTSL4
NAP1L3
USP51
KDM5C
SBF1
TOM1
MYH8
WDR24
ITIH5
AKAP9
FBXO1
LIAS
TNIK
SETD2
C3orf20
MR1
PIAS3
DIO1
ERCC5
KL
ALKBH8
DAPK1
DDX58
SPATA21
ZNF493
NGEF
DIRAS3
LATS2
ITGB3
FLNA
SATL1
KDM5C
KDM5C
RBFOX2
NPHS1
SOX9
CENPN
PSMD7
RIMBP2
GALNT11
ABHD11
UGT2A1
MTOR
PPP6R2
ZNF780A
WSCD2
CDKN1B
PPFIA1
TH
SSNA1
CASP2
PLRG1
SETD2
CCBL2
SESN2
MAGEB16
NLRP7
IGLON5
KLK4
WDR62
KIAA0355
CYP4F3
AKAP8
ZNF519
DDX52
ZC3H18
TCF12
NUSAP1
X4
KDM2B
MRPL51
C11orf68
ANO5
EIF4G2
MSRB2
RALGDS
EXT1
ZC3HC1
PTPRZ1
Privatebiquitous Shared primary Shared metastasis
Lung
metastases
Chest-wall
metastasis
Perinephric
metastasis
M1
10 cm
R2 (G3)
R4 (G1)
R6 (G1)
Hilum
R8 (G4)
Primary
tumor
M2b
M2a

3. BACKGROUND – RADIOMICS - HISTOPATHOLOMICS
• Huge potential for computerized medical image analysis
• Create imaging biomarkers to predict diagnosis, prognosis,
treatment response [2]
3
[2] Imaging and genomics: is there a synergy?, Jaffe et al., Radiology, 264(2):329-31, 2012
[3] Radiomics: the process and the challenges, Kumar et al., Magn Reson Imaging, 30(9):1234-48, 2012
[4] Histopathological image analysis: a review, Gurcan et al., IEEE Reviews in Biomed Eng, 2:147-71, 2009
Radiomics [3] “Histopatholomics” [4]
Reuse existing
diagnostic images ✓ radiology data ✓ digital pathology
Capture tissue
heterogeneity
✓ 3D neighborhoods
(e.g., 512x512x512)
✓ large 2D regions
(e.g., 15,000x15,000)
Analytic power beyond
naked eyes
✓ complex 3D tissue
morphology
✓exhaustive characterization
of 2D tissue structures
Non-invasive ✓ x

4. BACKGROUND – RADIOMICS - HISTOPATHOLOMICS
• Huge potential for computerized medical image analysis
• Create imaging biomarkers to predict diagnosis, prognosis,
treatment response
• Local quantitative image feature extraction
• Supervised machine learning
4
malignant, nonresponder
malignant, responder
benign
pre-malignant
undefined
quant. feat. #1
quant.feat.#2
Supervised learning,
big data
This could include an additional step of
studying the spatial relationships
between local image properties (e.g.,
using image graphs)

5. • Shape, intensity, margin, texture, …
• Shape and margin features often
require prior image segmentation
• 2D and 3D texture analysis can quantify micro- and
macro- structures in biomedical images [4,6]
IMAGE FEATURES
5
GURCANetal.:HISTOPATHOLOGICALIMAGEANALYSIS:AREVIEW
Fig.5.Resultsoftheautomaticsegmentationalgorithm(bluecontours:lumen
boundary,blackcontours:innerboundaryofthenucleioftheepithelialcells
surroundingthegland).Shownfromlefttorightareexampleimagesofbenign
epithelium,intermediate-,andhigh-gradecancer.
andare2-DCartesiancoordinatesof.Theevolutionof
isthendescribedbyalevel-setformulationadoptedfrom[78]
(4.1)
[4] Histopathological image analysis: a review, Gurcan et al., IEEE Reviews in Biomed Eng, 2:147-71, 2009
[5] Quantifying the margin sharpness of lesions on radiological images for content-based image retrieval, Xu et al., Med Phys, 39(9):5405-18, 2012
[6] Three-dimensional solid texture analysis in biomedical imaging: review and opportunities, Depeursinge et al., Med Image Anal, 18(1):176-96, 2014
[7] Prediction of prognosis for prostatic adenocarcinoma by combined histological grading and clinical staging, Gleason et al., J Urol, 111(1):58-64, 1974
[8] HyMaP: A hybrid magnitude-phase approach to unsupervised segmentation of tumor areas in breast cancer histology images, Khan et al.,
J Pathol Inform, 4, 2013
[4]
entsdrawnontheborderof
formationtocharacterizethe
hescaleandwindowparam-
entsfromthisportionofthe
(d)
)byradiologist.(b)Automatically
Ioutline(inwhite).(c)Finallung
ewithlinesegments(inwhite)that
[5]
J Pathol Inform 2013, 1:1 http://www.jpath
the phase spectrum to repres
HyperCS regions in a breast
the recently established efficac
exhibiting randomness.
Let vi
(x,y) denote the ith
Gab
normalized and smoothened v
I(x,y), where i = ,1,2,...,Ng
, Ng
the number of orientations. We
v x y v x y j x yi i i( , ) ( , ) exp( ( , ))= φ
where |·| denotes the magnit
denotes the local phase. The g
and its magnitude can then be
φi
i
i
i
i
x y
v x y
v x y
v x y
v x y
′
′ ′
= −
⎡
⎢
⎢
⎤
⎥
⎥
( , )
( , )
( , )
( , )
( , )
Figure 1:A sample H & E–stained breast cancer histology image:(a)
Original image, and (b) Overlaid image, with four types of contents
shown in different colors.The tumor areas are shown in Red,HypoCS
in Purple,and HyperCS in Green.Areas containing background or fat
tissue are shown in white with black outline. Note the difference in
morphology of the Hypo- and Hypercellular stromal regions
ba
[Downloaded free from http://www.jpathinformatics.org on Tuesday, June 16, 2015, IP: 128.179.146.236]
[7] [8]

6. COMPUTERIZED TEXTURE ANALYSIS
directionsscale
6
• Image scales and directions are important for visual texture
discrimination
• Most approaches are leveraging these two properties
• Explicitly: Gray-level co-occurrence matrices (GLCMs), run-length matrices
(RLE), directional filterbanks and wavelets, Fourier, histograms of gradients
(HOG), local binary patterns (LBP)
• Implicitly: Convolutional neural networks (CNN), scattering transform,
topographic independant component analysis (TICA)

7. COMPUTERIZED TEXTURE ANALYSIS
7
• Texture invariances: computer vision VS biomedical imaging
Computer vision Biomedical image analysis
scale scale-invariant multi-scale
rotation rotation-invariant rotation-invariant
[4] Histopathological image analysis: a review, Gurcan et al., IEEE Reviews in Biomed Eng, 2:147-71, 2009
[9] A sparse texture representation using local affine regions, Lazebnik et al., IEEE Trans on Pattern Anal and
Mach Intel, 27(8):1265-78, 2005
160
Fig. 10. (a) A digitized histopathology image of Grade 4 CaP and different graph-based r
Diagram, and Minimum Spanning tree.
Fig. 11. Digitized histological image at successively higher scales (magnifica-
tions) yields incrementally more discriminatory information in order to detect
suspicious regions.
or resolution. For instance at low or coarse scales color or tex-
ture cues are commonly used and at medium scales architec-
tural arrangement of individual histological structures (glands
and nuclei) start to become resolvable. It is only at higher res-
olutions that morphology of specific histological structures can
be discerned.
In [93], [94], a multiresolution approach has been used for the
classification of high-resolution whole-slide histopathology im-
ages. The proposed multiresolution approach mimics the eval-
uation of a pathologist such that image analysis starts from the
lowest resolution, which corresponds to the lower magnification
levels in a microscope and uses the higher resolution represen-
Fig. 12
image
1, (c) r
as susp
show
three
scale
(scal
the n
dition
highe
tumo
At
is com
COMPUTERIZED TEXTURE ANALYSIS
7
• Invariances: computer vision versus biomedical imaging
Computer vision Biomedical image analysis
scale scale-invariant multi-scale
rotation rotation-invariant rotation-invariant
[4] Histopathological image analysis: a review, Gurcan et al., IEEE Reviews in Biomed Eng, 2:147-71, 2009
160 IE
Fig. 10. (a) A digitized histopathology image of Grade 4 CaP and different graph-based representation
Diagram, and Minimum Spanning tree.
Fig. 11. Digitized histological image at successively higher scales (magnifica-
tions) yields incrementally more discriminatory information in order to detect
suspicious regions.
or resolution. For instance at low or coarse scales color or tex-
ture cues are commonly used and at medium scales architec-
tural arrangement of individual histological structures (glands
and nuclei) start to become resolvable. It is only at higher res-
olutions that morphology of specific histological structures can
be discerned.
In [93], [94], a multiresolution approach has been used for the
classification of high-resolution whole-slide histopathology im-
ages. The proposed multiresolution approach mimics the eval-
uation of a pathologist such that image analysis starts from the
lowest resolution, which corresponds to the lower magnification
levels in a microscope and uses the higher resolution represen-
tations for the regions requiring more detailed information for
a classification decision. To achieve this, images were decom-
posed into multiresolution representations using the Gaussian
pyramid approach [95]. This is followed by color space con-
version and feature construction followed by feature extraction
and feature selection at each resolution level. Once the classifier
is confident enough at a particular resolution level, the system
assigns a classification label (e.g., stroma-rich, stroma-poor or
Fig. 12. Results fr
image with the tum
1, (c) results at scale
as suspicious at low
shows the origin
three columns s
scales. Pixels cl
(scale) are disc
the number of p
ditionally, the p
higher scales al
tumor and nont
At lower reso
is commonly us
pattern of gland
tized histologic
scenes can be
to every pixel i
dient, and Gab
the scale, orien
gion of interest
features within
[4][9]

8. • Averaging texture properties over the entire lesions or
images discards tissue heterogeneity [10]
• Exhaustive and non-specific features
• Limited characterization of directions [6]
BIOMEDICAL TEXTURE ANALYSIS: LIMITATIONS (1/4)
8
[6] Three-dimensional solid texture analysis in biomedical imaging: review and opportunities, Depeursinge et al., Med
Image Anal, 18(1):176-96, 2014
[10] Quantitative imaging in cancer evolution and ecology, Gatenby et al., Radiology, 269(1):8-15, 2013
feartures are not specific when not
learned from data

9. • A global characterization of directions is not enough [11]
• local organization of image directions:
• independently from their local orientation:
• Rotation-covariance
• (local) grouped steering
of the operators:
BIOMEDICAL TEXTURE ANALYSIS: LIMITATIONS (2/4)
image operators: grouped steering:
9
[11] Rotation–covariant texture learning using steerable
Riesz wavelets, Depeursinge et al., IEEE Trans Imag Proc,
23(2):898-908, 2014

10. BIOMEDICAL TEXTURE ANALYSIS: LIMITATIONS (3/4)
GLCM contrast [1 0]
GLCMcontrast[01]
GLCMs (unaligned) Riesz (unaligned) Riesz (aligned)
10

11. BIOMEDICAL TEXTURE ANALYSIS: LIMITATIONS (4/4)
GLCM contrast [1 0]
GLCMcontrast[01]
GLCMs (unaligned) Riesz (unaligned) Riesz (aligned)
11

12. OBJECTIVES
12
• Highly adaptive texture-based computational models of
biomedical tissue:
✓ Complete coverage of image scales and
directions in 2-D and 3-D
✓ Rotation-covariance
✓ Specificity: the models can be trained to
characterize specific tissue types
✓ Local characterization of tissue properties
✓ Locate tissue properties in organ anatomy
to create digital phenotypes
• Goal: predict survival, function, treatment response
and reveal subtypes
be averaged over the folds of the CV and used to build texture models for each location along dROI (see Fig 9).
The sum of the weights for all channels from each location will reveal the subregions that are specific to each
tumor subtypes, and most related to patient survival. A K–means clustering of the vectors w will be carried out
for each locations to evaluate the stability of the regional models over the folds of the CV and define homogeneous
groups among patients. A selection of the models based on stability and location importance will be carried out.
Fig. 9 Prototype tissue archi-
tecture of a GBM tumor [128].
In a second step, the selected models will be locally steered to maximize their
magnitude. The energies of the maximal magnitudes will be used to construct
a final feature space for (1) predicting the tumor subtypes and (2) performing
Kaplan–Meier survival analysis. The performance of the proposed approach for
predicting tumor subtypes and patient survival will be compared to (1) unaligned
wavelet energies and (2) average wavelet energies over the entire tumor. While
starting with a LOPO CV on the TCGA–TCIA dataset, the generalizability of the
approach will be further assessed by training with the TCGA–TCIA and testing
with the SU dataset.
Deliverable 2.1: Predicting tumor subtype and survival from localized of tissue
properties in GBM tumors.
Task 2.2: Digital lung tissue atlases of ILD diagnoses (8 months of the PI)
In this task, we will use a simple atlas of the lungs to locate texture properties and create prototype diagnosis
phenotypes of ILDs. In previous work, we developed a simple 3–D atlas of the lungs with 36 subregions
that will be used in this task [42]. In a first step, diagnosis–wise digital tissue atlases will be created by
learning 3–D texture models (i.e., the average of w over the folds of the LOPO CV) for each 36 regions of
the lungs (see Fig. 10). The regions for which the models are most distant11
from all other diagnoses will
be highlighted and compared to previously built models of tissue patterns [25] to create 3–D prototype tissue
atlases for each diagnosis. The obtained results will be validated using medical knowledge (e.g., Table 1 of [42]).
Fig. 10 Tissue atlas of the lungs.
For each diagnosis, K–means clustering of the vectors w will be carried out
for each locations to evaluate the stability of the regional models over the folds
of the CV and reveal homogeneous groups among patients (e.g., subtypes of
UIP). The most stable models will be kept for the further characterization of
lung tissue types. A hierarchical clustering of the models from all diagnoses
will be carried out to define a radiomics–based hierarchy of all diagnoses, which
will be compared to medical knowledge [140] (e.g., Fig. 1 of [40]). A large
feature space including the energies of the steered models from each of the 36
localizations will be used to predict the diagnoses with uncertainty assessment
(using e.g., pairwise coupling [141]). When a minimum amount of trust is not
achieved when predicting a given diagnosis, the parent group of ILD diagnoses
in the previously built hierarchy will be predicted instead [19].
Deliverable 2.2: Digital tissue atlases of ILD diagnoses and their subtypes.
Task 2.3: Digital tissue atlases of ILDs: correlation with PFTs and survival (6 months )
Digital tissue atlases will be constructed for poor versus normal/high (1) PFTs and (2) survival. Regions for
which the models significantly di↵er between poor versus normal/high will be revealed as being of primary
importance to evaluate pulmonary function. The links between these models and previously built models of
tissue patterns will be investigated to define the combination of regions and patterns that are most responsible
for lung function impairment. The feature space spanned by the energies of the steered models will be used to
predict (1) PFT values or (2) survival with a LOPO CV, which can be evaluated using ground truth.
Deliverable 2.3: Estimating pulmonary function from digital tissue atlases of ILD diagnoses in CT images.
5.3 WP3. Imaging genomics
8
(a) Original Image
(c) Voronoi Diagram
8
(a) Original Image (b) Simple Cell Graph
(c) Voronoi Diagram (d) Delaunay Triangulation
(e) ECM-Aware Cell-Graph
Fig. 2 A fractured bone tissue example is shown in 2(a). Note the fracture cells in the
middle of the original image. The simple-cell-graph representation, the Voronoi diagram
and the Delaunay triangulation for this sample tissue are depicted in 2(b), 2(c) and 2(d).
The corresponding ECM-aware cell-graph is drawn in 2(e). The interactions between fracture
cells are drawn with blue and the red cells with red color. Delaunay triangulation represents
the tissue as a single connected component and does not allow crossing of edges. Simple-cell-
graphs relaxes these restrictions and allows the tissue to non-planar graph and disconnected.
Likewise, ECM-aware cell-graphs do not put such restrictions on the tissue and moreover
can capture the structural organization of different cells in a tissue. Furthermore, Delaunay
triangulations are fixed representations whereas ECM-aware cell-graphs can be adjusted
with different linking thresholds.[12] Automated classification of usual interstitial pneumonia using regional volumetric texture analysis in high-
resolution CT, Depeursinge et al., Invest Radiol, 50(4):261-67, 2015
[13] ECM-Aware Cell-Graph Mining for Bone Tissue Modeling and Classification, Bilgin et al., Data Min Knowl Discov,
20(3):416-38, 2009
[12] [13]

13. Steerable texture models
Σ
associated texture
model
EXAMPLE: PATTERN = WIGGLED CHECKERBOARD
pattern to learn:
30
Steerable texture models: our
workhorse ….
13

14. • Multi-directional, multi-scale and rotation-covariant image
analysis is achieved using Riesz wavelets
• The components of the th-order Riesz transform of a
2-D signal are defined in the Fourier domain as [14]:
• Yields allpass filters: only phase (i.e., direction) is kept and is defined
by th-order partial derivatives
THE RIESZ TRANSFORM
14
[14] Wavelet Steerability and the Higher-Order Riesz Transform, Unser et al., IEEE Trans on Imag Proc,
19(3):636-52, 2010

15. • Riesz components
THE RIESZ TRANSFORM
input image
15

16. • Riesz components
THE RIESZ TRANSFORM
16

17. • Riesz components
THE RIESZ TRANSFORM
17

18. • Multi-scale filterbanks are obtained by combining the
Riesz transform with isotropic wavelets [15]
• E.g., dyadic ( )
• Systematic coverage
of image scales
THE RIESZ TRANSFORM
18
UNSER et al.: STEERABLE PYRAMIDS AND TIGHT WAVELET FRAMES IN 2711
of unity) guarantees that is dense in ; it is
essential for the -completeness of the wavelet decomposition.
We now proceed with the construction of orienta-
tion-free wavelets by projecting the multiresolution Riesz
basis onto some appropriate wavelet subspace
.
B. Isotropic Bandlimited Wavelet Frames
There are a number of constructions in the literature that
fall into this category [13], [27]–[29]. Before reviewing them,
we apply the aforementioned projection strategy to obtain a
straightforward design that is in direct correspondence with
Shannon’s sampling theorem, and that is the starting point for
the specification of Meyer-type wavelets.
1) Construction of Isotropic, Shannon-Type Wavelets: By
selecting in (19),
we specify the so-called Shannon multiresolution analysis of
, which consists of a sequence of embedded subspaces
that are bandlimited to
We then define some corresponding wavelet subspaces of radi-
ally bandpass functions
(20)
Since is a closed subspace of , we can apply Proposition
3 to its orthogonal sinc basis to obtain the tight wavelet frame
of with
(21)
where is the impulse response of the
ideal radial bandpass filter, whose frequency response is
Fig. 1. Tiling of the 2-D frequency domain using radial-bandpass filters. The
shaded area corresponds to the spectral support of the wavelet subspace ;
it is included in the spectral support of (enclosing square).
satisfy (22). This leads to the following extended definition of
the wavelet subspaces
which is equivalent to (20) if is the impulse response of
the ideal radial bandpass filter. Since can be written as
, there exists a sequence such that
where the wavelet functions are still given by (21). This
indicates that is a frame of , albeit not neces-
sarily a tight one. Yet, if Condition (22) is satisfied, then one
recovers the tight frame property over which is the
union of the wavelet subspaces , . The condition for
the wavelet frame to be isotropic is that the restriction of the fil-
tering function over be isotropic, i.e.,
.
[15] Steerable pyramids and tight wavelet frames in , Unser et al., IEEE Trans on Imag Proc,
20(10):2705-21, 2011

19. THE RIESZ TRANSFORM
19
• Steerability [16]:
• Example :
• Higher-order steering can be done fully analytical
✓Complete coverage of the image directions
✓Steerability enables rotation-covariance
[16] The design and use of steerable filters, Adelson et al., IEEE Trans on Pattern Anal and Mach Intel,
13(9):891-906, 1991

20. • A Riesz filterbank constitutes a dictionary of basic
textures:
• Higher-level texture models are built from linear
combinations of Riesz components
LEARNING TEXTURE MODELS
20

21.
Σ
EXAMPLE: PATTERN = WIGGLED CHECKERBOARD
pattern to learn:

22.
Σ
EXAMPLE: PATTERN = WIGGLED CHECKERBOARD
pattern to learn:

23.
Σ
EXAMPLE: PATTERN = WIGGLED CHECKERBOARD
pattern to learn:

24.
Σ
EXAMPLE: PATTERN = WIGGLED CHECKERBOARD
pattern to learn:

25.
Σ
EXAMPLE: PATTERN = WIGGLED CHECKERBOARD
pattern to learn:

26.
Σ
EXAMPLE: PATTERN = WIGGLED CHECKERBOARD
pattern to learn:

27.
Σ
EXAMPLE: PATTERN = WIGGLED CHECKERBOARD
pattern to learn:

28.
Σ
associated texture
model
EXAMPLE: PATTERN = WIGGLED CHECKERBOARD
pattern to learn:
28

29. • Support vector machines (SVM) were used to learn (multi-scale )
versus
ONE-VERSUS-ALL SUPERVISED MODEL LEARNING [11]:
texture
all others
[11] Rotation-covariant texture learning using steerable Riesz wavelets, Depeursinge et al., IEEE Trans Imag Proc,
23(2):898-908, 2014
29

30. • 2-D synthetic textures with noise ( )
✓ Captures specific local properties in terms of image
scales and directions
LEARNING TEXTURE MODELS
30

31. • 2-D tissue from interstitial lung diseases (ILD) in CT [17]
• ( , 32x32 patches)
LEARNING TEXTURE MODELS
healthy emphysema ground glass fibrosis micronodules
3011 patches,
7 patients.
407 patches,
6 patients.
2226 patches,
32 patients.
2962 patches,
37 patients.
5988 patches,
16 patients.
31
[17] Multiscale lung texture signature learning using the Riesz transform, Depeursinge et al., Med Image Comput
Comput Assist Interv (MICCAI), 15(3):517-24, 2012

32. • 3-D synthetic textures ( , based on [18])
• Vertical planes
• 3-D checkerboard
• 3-D wiggled
checkerboard
LEARNING TEXTURE MODELS
32
[18] 3D Steerable Wavelets and Monogenic Analysis for Bioimaging, Chenouard et al., IEEE 8th Int Symp on Biomed
Imag (ISBI), 2132-5, 2011

33. • Steering texture models:
STEERABLE TEXTURE MODELS [11]
The expression of the rotated texture model remains a
linear combination of the initial Riesz components
[11] Rotation-covariant texture learning using steerable Riesz wavelets, Depeursinge et al., IEEE Trans Imag Proc,
23(2):898-908, 2014

34. • Steering texture models:
• Rotation-covariant models: local steering of
• Keep the maximum magnitude of the model
• Local quantitative features: energies/abs values of the
magnitudes in the patch
STEERABLE TEXTURE MODELS [11]
34
The expression of the rotated texture model remains a
linear combination of the initial Riesz components
[11] Rotation-covariant texture learning using steerable Riesz wavelets, Depeursinge et al., IEEE Trans Imag Proc,
23(2):898-908, 2014

35. Classification of 2-D natural textures
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2013 5
1) canvas001 2) canvas002 3) canvas003 4) canvas005 5) canvas006 6) canvas009 7) canvas011 8) canvas021
9) canvas022 10) canvas023 11) canvas025 12) canvas026 13) canvas031 14) canvas032 15) canvas033 16) canvas035
17) canvas038 18) canvas039 19) tile005 20) tile006 21) carpet002 22) carpet004 23) carpet005 24) carpet009
Fig. 5. 128 ⇥ 128 blocks from the 24 texture classes of the Outex database.
1) canvas 2) cloth 3) cotton 4) grass 5) leather 6) matting 7) paper 8) pigskin
APPLICATIONS (1/2)
35

36. • 24 classes, 180 images/class, 9 rotation angles in
• A SVM classifier is trained with unrotated images only
• 98.4% best acc.
• aligned models
( )
• Literature: 90-99%
• E.g., scattering transform:
98.75% [18]
2-D TEXTURE CLASSIFICATION: OUTEX DATABASE [17]
Errors occur with most
stochastic textures
confusion matrix:
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2013
TABLE
AVERAGE Ac OBTAINED WITH THE LOCAL ORIENTATION MAXIMIZATION O
Outex TC 00010 Outex TC 0001
¯Ac for N = 2, 4, 6, 8, 10. 98 ± 0.7 97.2 ± 0
¯Ac for N = 1, 3, 5, 7, 9. 94.4 ± 0.7 93.6 ± 0
¯Ac for N = 1, . . . , 10. 96.2 ± 2 95.4 ±
0.4
0.5
0.6
0.7
0.8
0.9
1
1 2 3 4 5 6 7 8 9 10
classificationaccuracy
N
aligned
aligned
initial Riesz
N
c,1
R(0,N)
Fig. 7. Classification performance with the Outex TC 00010 test suite. The
two rotation covariant approaches are performing much better than using
the initial Riesz coefficients. The local orientation maximization of N
c,1
outperforms the local orientation of the first Riesz template R(0,N) as
proposed in [8]. A maximum Ac of 98.4% is reached with N = 8.
classes are balanced, the classification accuracy Ac is used as
a performance measure of the methods. All performances are
summarized in Table I.
1) Outex TC 00010: The classification performance for
orders N = 1, . . . , 10 is shown in Fig. 7. The performance
using the energy of the coefficients that are maximizing the
response of the first Riesz template (i.e., R(0,N)
) at the
smallest scale was also evaluated as a first rotation–covariant
approach [8]. An average Ac of 96.2 ± 2% is obtained with
N = 1 . . . 10 and the local orientation maximization of N
.
F
f
m
c
a
i
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2013 5
1) canvas001 2) canvas002 3) canvas003 4) canvas005 5) canvas006 6) canvas009 7) canvas011 8) canvas021
9) canvas022 10) canvas023 11) canvas025 12) canvas026 13) canvas031 14) canvas032 15) canvas033 16) canvas035
17) canvas038 18) canvas039 19) tile005 20) tile006 21) carpet002 22) carpet004 23) carpet005 24) carpet009
Fig. 5. 128 ⇥ 128 blocks from the 24 texture classes of the Outex database.
1) canvas 2) cloth 3) cotton 4) grass 5) leather 6) matting 7) paper 8) pigskin
9) raffia 10) rattan 11) reptile 12) sand 13) straw 14) weave 15) wood 16) wool
Fig. 6. 16 Brodatz texture classes of the Contrib TC 00000 test suite.
180 ⇥ 180 images from rotation angles 20 , 70 , 90 , 120 ,
135 and 150 of the other seven Brodatz images for each
class. The total number of images in the test set is 672.
G. Experimental setup
OVA SVM models using Gaussian kernels as K(xi, xj) =
exp(
||xi xj ||2
2 2
k
) are used both to learn texture signatures and
to classify the texture instances in the final feature space
obtained after k iterations. A number of scales J = 6
Inc., 2012. The computational complexity is dominated by the
local orientation of N
c in Eq. 11, which consists of finding the
roots of the polynomials defined by the steering matrix A✓
.
It is therefore NP–hard (Non–deterministic Polynomial–time
hard), where the order of the polynomials is controlled by the
order of the Riesz transform N.
III. RESULTS
The performance of our approach is demonstrated with
[17] Multiresolution gray-scale and rotation invariant texture classification with local binary patterns, Ojala et al., IEEE Trans on Pattern Anal and Mach Intel, 24:7(971-87), 2002
[18] Combined scattering for rotation invariant texture analysis, Sifre et al., European Symposium on Artificial Neural Networks, 2012
Importance of rotation-covariance
and learned models

37. Medulloblastoma tumor classification
APPLICATIONS (2/2)
ompute the UFL features learned by TICA and the supervised features learned
with Riesz wavelets for each image as described in Sections 2.1 and 2.2. Once
TICA and Riesz wavelets are computed a final step of supervised classification is
made using the combination of the computed features in a concatenated vector
as input for a standard softmax classifier as described in Section 2.3. Parameter
uning is presented in Section 3.2.
[2.4] Medulloblastoma
Image Cases
Fig. 2. Flowchart for MB feature extraction and classification for both learned repre-
entations: Riesz and TICA, the details of each stage are described in subsections.
steerable Riesz
texture models
topographic ICA
37

38. • Medulloblastoma (MB)
• Most common (i.e., 25%) pediatric brain tumor
• Major cause of death in pediatric oncology
• Histological subtypes of MB have
different prognosis and treatments
• Anaplastic subtype is the worst!
• “marked nuclear pleomorphism,
cell wrapping,
high mitotic count,
and abundant apoptotic bodies” [19]
MEDULLOBLASTOMA TUMORS [19]
38
[19] Childhood medulloblastoma: novel approaches to the classification of a heterogeneous disease, Ellison et al.,
Acta Neuropathol, 20(3):305-16, 2010
Pleomorphism (cytology): variability in
the size and shape of cells and/or their
nuclei

39. • Proposed approach [20]
• Anaplastic versus non-anaplastic
• Compare/combine steerable Riesz texture models with
unsupervised topographic independent component analysis (TICA)
MEDULLOBLASTOMA TUMOR CLASSIFICATION
39
[20] Combining Unsupervised Feature Learning and Riesz Wavelets for Histopathology Image Representation:
Application to Identifying Anaplastic Medulloblastoma, Otálora et al., Med Image Comput Comput Assist Interv
(MICCAI), 2015
The workflow of the proposed approach is summarized in Fig. 2. As first step, we
ompute the UFL features learned by TICA and the supervised features learned
with Riesz wavelets for each image as described in Sections 2.1 and 2.2. Once
TICA and Riesz wavelets are computed a final step of supervised classification is
made using the combination of the computed features in a concatenated vector
as input for a standard softmax classifier as described in Section 2.3. Parameter
uning is presented in Section 3.2.
[2.4] Medulloblastoma
Image Cases
Fig. 2. Flowchart for MB feature extraction and classification for both learned repre-
entations: Riesz and TICA, the details of each stage are described in subsections.
steerable Riesz
texture models
topographic ICA

40. • Unsupervised topographic independent component
analysis (TICA) [21]
• Learn while minimizing cost function :
MEDULLOBLASTOMA TUMOR CLASSIFICATION
40
[21] Energy correlations and topographic organization, Hyvärinen et al., Natural Image Statistics, 39:249-272, 2009
TICA is an unsupervised feature learning model, inspired by findings of the visual
cortex behaviour. It groups activations of units in order to discover features that
are rotation and translation invariant [1]. These are appropriate features for
histopathology image characterization since shapes and cell organizations can
be present regardless of the position or orientation of cells. Particularly, TICA
organizes feature detectors in a square matrix for l groups such that adjacent
feature detectors activate in a similar proportion to the same stimulus. To learn
such groups, we need to optimize the cost function:
JTICA(W) =
2
TX
i=1
WT
Wx(i)
x(i)
2
2
+
mX
i=1
lX
k=1
»
Hk(Wx(i))2 + ✏ (1)
where x(i)
2 Rm
is the i-th sample, T is the number of samples, W 2 Rn⇥m
is
the matrix that encodes the features in each row, and H 2 {0, 1}l⇥n
is the binary
topographic organization where H
(j)
k = 1, if the j-th feature detector, j-th row
of W , belongs to the k-th group, and 0 otherwise. This model sets H fixed while
learning W. In addition, TICA has two main computational advantages. First,
the only parameters to be tuned are the regularization hyperparameter and the
sparsity controller ✏. Second, it is an unconstrained optimization problem, which
can be solved e ciently by optimization techniques such as Limited memory-
Broyden-Fletcher-Goldfarb-Shanno (L-BFGS).
iriiiiéiiiiiiiiiiiiosiiióiiiiïiiiiiii1=11.e1111MMIIIM
111111111111111M
11=1
MMMMMMMMMMMMMMMMMMM
iiiMUMMMiMMMiiiiiii iíiíi iiiíiii
ii UWE:lMiiM
ME= M MEE
is i iiiiíiiiiMóóóii
iMMMMMlMiMMMMMMMiMMMiMMMMMMMMMiiMMiMMiMMM
iMMMiMMMiiMMMMiiiiiiiMiMMiMiMiiMMiiMiiiiiiiiMM
JTICA(W )
reconstruction penaltyL2 topographic constraint:
if the patch belongs
to the local group of patches
Hk = 1 x(i)
k
~scale inv. ~rotational inv.
~translational inv.
W

41. • Dataset:
• 5 anaplastic, 5 non-anaplastic patients
• 750 patches (200x200) / patient
• Leave-2-patients-out cross-validation
• Softmax classifier
• Comparison with state-of-the-art [22]
• Convolutional neural networks (CNN), sparse autoencoders (sAE),
bag of features (BOF), Haar, MR8, …
• Predictions over entire-slides:
MEDULLOBLASTOMA TUMOR CLASSIFICATION
41
[22] A comparative evaluation of supervised and unsupervised representation learning approaches for anaplastic
medulloblastoma differentiation, Cruz-Roa et al., Proc. SPIE 9287, Int Symp Med Inf Proc Anal, 92870G, 2015
Medulloblastoma Di↵erentiat
Table 1. MB classification performance (
averaged over the 20 test runs with standa
Method Accuracy Se
TICA + Riesz[N1
3 , N2
2 , N2
1 ] 0.997 ± 0.002 0.99
TICA [1] 0.972 ± 0.018 0.97
Riesz [N1
3 , N2
2 , N2
1 ] [3] 0.964 ± 0.038 0.99
Riesz [N1
3 ] [3] 0.958 ± 0.062 0.9
Riesz [N2
2 ] [3] 0.94 ± 0.02 0.9
2-Layer CNN [1] 0.90 ± 0.1 0.8
sAE [1] 0.90
BOF + A2NMF (Haar) [2] 0.87
Riesz [N2
1 ] [3] 0.85 ± 0.23 0.
BOF + K - NN (Haar) [7] 0.80
BOF + K - NN (MR8) [7] 0.62
4 Concluding Remarks
We present a feature fusion between u
vised Riesz wavelet representation that
[22]
[22]
[22]
[22]
[22]
TICA [1] 0.972 ± 0.018 0.977 ± 0.021 0.967 ± 0.031
Riesz [N1
3 , N2
2 , N2
1 ] [3] 0.964 ± 0.038 0.999 ± 0.001 0.932 ± 0.07
Riesz [N1
3 ] [3] 0.958 ± 0.062 0.963 ± 0.05 0.916 ± 0.125
Riesz [N2
2 ] [3] 0.94 ± 0.02 0.94 ± 0.02 0.3 ± 0.04
2-Layer CNN [1] 0.90 ± 0.1 0.89 ± 0.18 0.9 ± 0.0.3
sAE [1] 0.90 0.87 0.93
BOF + A2NMF (Haar) [2] 0.87 0.86 0.87
Riesz [N2
1 ] [3] 0.85 ± 0.23 0.9 ± 0.15 0.7 ± 0.47
BOF + K - NN (Haar) [7] 0.80 - -
BOF + K - NN (MR8) [7] 0.62 - -
4 Concluding Remarks
We present a feature fusion between unsupervised feature learning and super-
vised Riesz wavelet representation that captures subtle pattern of textures as
well as high level features, allowing to create a more separable feature space
where the di↵erentiation of medulloblastoma into anaplastic and non-anaplastic
can be made with high classification accuracy outperforming any other result
previously described in the literature. To our knowledge this is the first time
that a feature fusion method is presented between UFL and the Riesz wavelets
in the context of histopathology image analysis showing the complementarity
between these learned features for the challenging task of tumour di↵erentia-
tion, we are currently working on extending the method to other patch-based
histopathology image analysis problems with larger cohorts of patients.
Fig. 3. Predictions over two WSIs, non-anaplastic MB (left) and anaplastic (right).
non-anaplastic anaplastic
TICA: unsupervised
Riesz: rotation-covariant information
Small dataset

42. • Highly adaptive texture-based computational models of
biomedical tissue:
✓ Complete/systematic coverage of image
scales and directions in 2-D and 3-D
✓ Rotation-covariance
✓ Specificity: the models can be trained to
characterize specific tissue types
✓ Local characterization of tissue properties
• Locate tissue properties in organ anatomy
to create digital phenotypes
• localization systems,
quant. graph analysis
CONCLUSIONS
model learning
Multi-resolution,
steerability
patch-based analysis
42
be averaged over the folds of the CV and used to build texture models for each location along dROI (see Fig 9).
The sum of the weights for all channels from each location will reveal the subregions that are specific to each
tumor subtypes, and most related to patient survival. A K–means clustering of the vectors w will be carried out
for each locations to evaluate the stability of the regional models over the folds of the CV and define homogeneous
groups among patients. A selection of the models based on stability and location importance will be carried out.
Fig. 9 Prototype tissue archi-
tecture of a GBM tumor [128].
In a second step, the selected models will be locally steered to maximize their
magnitude. The energies of the maximal magnitudes will be used to construct
a final feature space for (1) predicting the tumor subtypes and (2) performing
Kaplan–Meier survival analysis. The performance of the proposed approach for
predicting tumor subtypes and patient survival will be compared to (1) unaligned
wavelet energies and (2) average wavelet energies over the entire tumor. While
starting with a LOPO CV on the TCGA–TCIA dataset, the generalizability of the
approach will be further assessed by training with the TCGA–TCIA and testing
with the SU dataset.
Deliverable 2.1: Predicting tumor subtype and survival from localized of tissue
properties in GBM tumors.
Task 2.2: Digital lung tissue atlases of ILD diagnoses (8 months of the PI)
In this task, we will use a simple atlas of the lungs to locate texture properties and create prototype diagnosis
phenotypes of ILDs. In previous work, we developed a simple 3–D atlas of the lungs with 36 subregions
that will be used in this task [42]. In a first step, diagnosis–wise digital tissue atlases will be created by
learning 3–D texture models (i.e., the average of w over the folds of the LOPO CV) for each 36 regions of
the lungs (see Fig. 10). The regions for which the models are most distant11
from all other diagnoses will
be highlighted and compared to previously built models of tissue patterns [25] to create 3–D prototype tissue
atlases for each diagnosis. The obtained results will be validated using medical knowledge (e.g., Table 1 of [42]).
Fig. 10 Tissue atlas of the lungs.
For each diagnosis, K–means clustering of the vectors w will be carried out
for each locations to evaluate the stability of the regional models over the folds
of the CV and reveal homogeneous groups among patients (e.g., subtypes of
UIP). The most stable models will be kept for the further characterization of
lung tissue types. A hierarchical clustering of the models from all diagnoses
will be carried out to define a radiomics–based hierarchy of all diagnoses, which
will be compared to medical knowledge [140] (e.g., Fig. 1 of [40]). A large
feature space including the energies of the steered models from each of the 36
localizations will be used to predict the diagnoses with uncertainty assessment
(using e.g., pairwise coupling [141]). When a minimum amount of trust is not
achieved when predicting a given diagnosis, the parent group of ILD diagnoses
in the previously built hierarchy will be predicted instead [19].
Deliverable 2.2: Digital tissue atlases of ILD diagnoses and their subtypes.
Task 2.3: Digital tissue atlases of ILDs: correlation with PFTs and survival (6 months )

43. CONCLUSIONS
• Importance of rotation-covariant information to model
biomedical textures
• i.e., the local organization of directions
• Rotational invariance is not enough
• Called “roto-translation invariance” in [23]
TextureQbased'biomarkers:'current'limitaGons'
x Assume'homogeneous'texture'properGes'over'the'
enGre'lesion'[5]'
'
x NonQspecific'features'
x Global'vs'local'characterizaGon'of'image'direcGons'[6]'
REVIEW: Quantitative Imaging in Cancer Evolution and Ecology Gatenby et al
with the mean signal value. By using just
two sequences, a contrast-enhanced T1
sequence and a fluid-attenuated inver-
sion-recovery sequence, we can define
four habitats: high or low postgadolini-
um T1 divided into high or low fluid-at-
tenuated inversion recovery. When these
voxel habitats are projected into the tu-
mor volume, we find they cluster into
spatially distinct regions. These habitats
can be evaluated both in terms of their
relative contributions to the total tumor
volume and in terms of their interactions
with each other, based on the imaging
characteristics at the interfaces between
regions. Similar spatially explicit analysis
can be performed with CT scans (Fig 5).
Analysis of spatial patterns in
cross-sectional images will ultimately re-
quire methods that bridge spatial scales
from microns to millimeters. One possi-
ble method is a general class of numeric
tools that is already widely used in ter-
restrial and marine ecology research to
link species occurrence or abundance
with environmental parameters. Species
distribution models (48–51) are used to
gain ecologic and evolutionary insights
and to predict distributions of species or
morphs across landscapes, sometimes
extrapolating in space and time. They
can easily be used to link the environ-
mental selection forces in MR imaging-
defined habitats to the evolutionary dy-
namics of cancer cells.
Summary
Imaging can have an enormous role in
the development and implementation of
patient-specific therapies in cancer. The
achievement of this goal will require new
methods that expand and ultimately re-
place the current subjective qualitative
assessments of tumor characteristics.
rise to local-regional phenotypic adap-
tations. Phenotypic alterations can re-
sult from epigenetic, genetic, or chro-
mosomal rearrangements, and these in
turn will affect prognosis and response
to therapy. Changes in habitats or the
relative abundance of specific ecologic
communities over time and in response
to therapy may be a valuable metric with
which to measure treatment efficacy and
emergence of resistant populations.
Emerging Strategies for Tumor Habitat
Characterization
A method for converting images to spa-
tially explicit tumor habitats is shown in
Figure 4. Here, three-dimensional MR
microenvironment can be rewarded by
increased proliferation. This evolution-
ary dynamic may contribute to distinct
differences between the tumor edges
and the tumor cores, which frequently
can be seen at analysis of cross-sec-
tional images (Fig 5).
Interpretation of the subsegmenta-
tion of tumors will require computa-
tional models to understand and predict
the complex nonlinear dynamics that
lead to heterogeneous combinations
of radiographic features. We have ex-
ploited ecologic methods and models to
investigate regional variations in cancer
environmental and cellular properties
that lead to specific imaging character-
istics. Conceptually, this approach as-
Figure 4
Figure 4: Left: Contrast-enhanced T1 image from subject TCGA-02-0034 in The Cancer Genome
Atlas–Glioblastoma Multiforme repository of MR volumes of glioblastoma multiforme cases. Right: Spatial
distribution of MR imaging–defined habitats within the tumor. The blue region (low T1 postgadolinium, low
fluid-attenuated inversion recovery) is particularly notable because it presumably represents a habitat with
low blood flow but high cell density, indicating a population presumably adapted to hypoxic acidic conditions.
rise to local-regional phenotypic adap-
tations. Phenotypic alterations can re-
sult from epigenetic, genetic, or chro-
mosomal rearrangements, and these in
turn will affect prognosis and response
to therapy. Changes in habitats or the
relative abundance of specific ecologic
communities over time and in response
to therapy may be a valuable metric with
which to measure treatment efficacy and
emergence of resistant populations.
Emerging Strategies for Tumor Habitat
Characterization
A method for converting images to spa-
tially explicit tumor habitats is shown in
microenvironment can be rewarded by
increased proliferation. This evolution-
ary dynamic may contribute to distinct
differences between the tumor edges
and the tumor cores, which frequently
can be seen at analysis of cross-sec-
tional images (Fig 5).
Interpretation of the subsegmenta-
tion of tumors will require computa-
tional models to understand and predict
the complex nonlinear dynamics that
lead to heterogeneous combinations
of radiographic features. We have ex-
ploited ecologic methods and models to
investigate regional variations in cancer
environmental and cellular properties
that lead to specific imaging character-
Figure 4: Left: Contrast-enhanced T1 image from subject TCGA-02-0034 in The Cancer Genome
Atlas–Glioblastoma Multiforme repository of MR volumes of glioblastoma multiforme cases. Right: Spatial
distribution of MR imaging–defined habitats within the tumor. The blue region (low T1 postgadolinium, low
fluid-attenuated inversion recovery) is particularly notable because it presumably represents a habitat with
low blood flow but high cell density, indicating a population presumably adapted to hypoxic acidic conditions.
[5]'QuanGtaGve'imaging'in'cancer'evoluGon'and'ecology,'Gatenby'et'al.,'Radiology,'269(1):8Q15,'2013'
5'
global'direcGonal'operators:' local'grouped'steering:'
[6]'RotaGonQcovariant'texture'learning'using'steerable'Riesz'wavelets,'Depeursinge'et'al.,'IEEE'Trans'Imag'Proc.,'23(2):898Q908,'2014.'
global directional operators local grouped steering
43
[11] Rotation-covariant texture learning using steerable Riesz wavelets, Depeursinge et al., IEEE Trans Imag Proc,
23(2):898-908, 2014
[23] Rotation, Scaling and Deformation Invariant Scattering for Texture Discrimination, Sifre et al., IEEE Conf Comp
Vis and Pat Rec (CVPR), 1233-40, 2013
with a second operator R2 which is invariant to the action
of G2. Indeed for all g1.g2 ∈ G1 G2 and all images x(u):
R2(R1(g1.g2.x)) = R2(g2.R1(x)) = R2(R1(x)).
However, such separable invariants do not capture the joint
property of the action of G2 relatively to G1, and may lose
important information. This is why two-dimensional trans-
lation invariant representations are not computed by cascad-
ing invariants to horizontal and vertical translations. It is
also important for rotations and translations. Let us consider
for example the two texture patches of Figure 1. A separa-
ble product of translation and rotation invariant operators
can represent the relative positions of the vertical patterns,
and the relative positions of the horizontal patterns, up to
global translations. However, it can not represent the po-
sitions of horizontal patterns relatively to vertical patterns,
because it is not sensitive to a relative shift between these
two sets of oriented structures. It loses the relative positions
of different orientations, which is needed to be sensitive to
curvature, crossings and corners. Such a separable invariant
thus can not discriminate the two textures of Figure 1.
Figure 1: The left and right textures are not discriminated
by a separable invariant along rotations and translations, but
can be discriminated by a joint roto-translation invariant.
Several authors [6, 7, 8] have proposed to take into ac-
count the joint structure of roto-translation operators in im-
age processing, particularly to implement diffusion oper-
ators. Computing a joint invariant between rotations and
translations also means taking into account the joint rela-
tive positions and orientations of image structures, so that
the textures of Figure 1 can be discriminated. Section 3
introduces a roto-translation scattering operator, which is
computed by cascading wavelet transforms on the roto-
translation group.
Calculating joint invariants on large non-commutative
groups may however become very complex. Keeping a sep-
information.
2.2. Hierarchical Architecture
We now explain how to build an affine invariant repre-
sentation, with a hierarchical architecture. We separate vari-
abilities of potentially large amplitudes such as translations,
rotations and scaling, from smaller amplitude variabilities,
but which may belong to much higher dimensional groups
such as shearing and general diffeomorphisms. These small
amplitude deformations are linearized to remove them with
linear projectors.
Image variabilities typically differ over domains of dif-
ferent sizes. Most image representations build localized in-
variants over small image patches, for example with SIFT
descriptors [15]. These invariant coefficients are then ag-
gregated into more invariant global image descriptors, for
example with bag of words [10] or multiple layers of deep
neural network [4, 5]. We follow a similar strategy by first
computing invariants over image patches and then aggregat-
ing them at the global image scale. This is illustrated by the
computational architecture of Figure 2.
x
roto-trans.
patch
scattering
log
global
space-scale
averaging
deformat.
invariant
linear proj.
Figure 2: An affine invariant scattering is computed by ap-
plying a roto-translation scattering on image patches, a log-
arithmic non-linearity and a global space-scale averaging.
Invariants to small shearing and deformations are computed
with linear projectors optimized by a supervised classifier.
Within image patches, as previously explained, one must
keep the joint information between positions and orienta-
tions. This is done by calculating a scattering invariant on
the joint roto-translation group. Scaling invariance is then
implemented with a global scale-space averaging between
patches, described in Section 4. A logarithmic non-linearity
is first applied to invariant scattering coefficients to linearize
their power law behavior across scales. This is similar to the
normalization strategies used by bag of words [10] and deep
neural networks [5].
Because of three dimensional surface curvature in the vi-
sual scene, the image patches are also deformed. A scat-
tering transform was proved to be stable to deformations
[23]
[11]

44. • Rotation-covariance
• Local orientation of the models is computationally intensive
• 2D:
• 3D:
• Use graphics processor units (GPUs) [24]: 60x speedup
• Explore other steerable wavelet representations [25,26]
LIMITATIONS AND FUTURE WORK
44
G ⇤ R(2,0,0)
G ⇤ R(0,2,0)
G ⇤ R(0,0,2)
G ⇤ R(1,1,0)
G ⇤ R(1,0,1)
G ⇤ R(0,1,1)
Fig. 1. Second–order Riesz kernels R(n1,n2,n3)
convolved with isotropic Gaussian kernels G(x).
Support vector machines (SVM) are then used to classify
between 9,347 normal and embolic cubic instances of lung
parenchyma from 19 patients with APE and 8 control cases.
2. MATERIAL AND METHODS
2.1. Rotation–covariant texture analysis
3D multiscale Riesz filterbanks are used to characterize the
texture of the lung parenchyma in 3D at a given CT energy
level. The N–th order Riesz transform R(N)
of a three–
dimensional signal f(x) is defined in the Fourier domain as:
¤R(n1,n2,n3)f(!) =
…
n1 + n2 + n3
n1!n2!n3!
( j!1)n1
( j!2)n2
( j!3)n3
||!||n1+n2+n3
ˆf(!),
(1)
Hospitals of Geneva with an inter–slice distance of 1mm, a
slice thickness of 1.25mm, and a sub–millimetric resolution
in the axial plane. 11 energy levels are used from 40keV to
140keV with a step of 10keV. All five lobes of each patient
have been manually segmented using the OsiriX software2
.
The perfusion levels of each lobe were quantified using the
Qanadli index (QI) on a lobe basis [14]. The QI is defined as
the sum of the scores of all arteries as: 0 if no occlusion is
visible, 1 if partially occluded, and 2 if totally obstructed.
DECT data of all energy levels are preprocessed to have
an isotropic voxel resolution, which is obtained by dividing
samples along the z axis. All lobes are divided into 323
overlapping blocks to constitute a local instance of the lung
parenchyma. A block is considered as valid when at least
95% of its voxels belong to it. To obtain a sufficient number
of blocks for the middle right lobe, this rule was changed to
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2013 10
RN
f✓
(x) =
0
B
B
B
B
B
B
@
R(0,N)
f✓
(x)
...
R(n,N n)
f✓
(x)
...
R(N,0)
f✓
(x)
1
C
C
C
C
C
C
A
| {z }
RN {f✓}(x)
=
0
B
B
B
B
B
B
@
A✓
0,0 . . . A✓
0,l . . . A✓
0,N
...
...
...
A✓
n,1 . . . A✓
n,l . . . A✓
n,N
...
...
...
A✓
N,1 . . . A✓
N,l . . . A✓
N,N
1
C
C
C
C
C
C
A
| {z }
A✓
0
B
B
B
B
B
B
@
R(0,N)
{f} (x)
...
R(l,N l)
{f} (x)
...
R(N,0)
{f} (x)
1
C
C
C
C
C
C
A
| {z }
RN {f}(x)
.
(12)
ˆR(n,N n)
=
N
n!(N n)!
( j(cos(✓)!1 + sin(✓)!2))n
( j( sin(✓)!1 + cos(✓)!2))N n
||!||N
=
1
||!||N
N
n!(N n)!
nX
k1=0
Ç
n
k1
å
(cos(✓))k1
( j!1)k1
(sin(✓))n k1
( j!2)n k1
N nX
k2=0
Ç
N n
k2
å
(cos(✓))k2
( j!1)k2
(sin(✓))N n k2
( j!2)N n k2
=
1
||!||N
N
n!(N n)!
nX
k1=0
N nX
k2=0
( j!1)k1+k2
( j!2)N k1 k2
( 1)k2
Ç
n
k1
åÇ
N n
k2
å
(cos(✓))N n k2+k1
(sin(✓))n k1+k2
=
nX
k1=0
N nX
k2=0
N
(k1 + k2)!(N k1 k2)!
( j!1)k1+k2
( j!2)N k1 k2
||!||N
| {z }
ˆR(k1+k2,N k1 k2)
(k1 + k2)!(N k1 k2)!
n!(N n)!
( 1)k2
n!
k1!(n k1)!
(N n)!
k2!(N n k2)!
(cos(✓))N n k2+k1
(sin(✓))n k1+k2
.
(13)
ˆR(n,N n)
(!) =
NX
l=0
N
l!(N l)!
( j!1)l
( j!2)N l
||!||N
| {z }
ˆR(l,N l)
min(l,n)
X
l1=max(0,l N+n)
( 1)l l1
l!(N l)!
l1!(n l1)!(l l1)!
(cos(✓))N n+2l1 l
(sin(✓))n 2l1+l
| {z }
A✓
n,l
.
(14)
[32] T. Leung and J. Malik. Representing and recognizing the visual
appearance of materials using three–dimensional textons. International
Journal of Computer Vision, 43(1):29–44, 2001.
invariant texture classification with local binary patterns. In Computer
Vision — ECCV 2000, volume 1842 of Lecture Notes in Computer
Science, pages 404–420. Springer Berlin Heidelberg, 2000.
[24] GPU-accelerated texture analysis using steerable Riesz wavelets, Vizitiu et al., 11th Int Conf Par Proc and App Math (PPAM),
2015 (submitted)
[25] A unifying parametric framework for 2D steerable wavelet transforms, Unser et al., SIAM Jour Imag Sci, 6(1):102-35, 2013
[26] Harmonic Singular Integrals and Steerable Wavelets in , Ward et al., App and Comp Harm Anal, 36(2):183-197, 2014

45. • Multi-scale
• Influence of surrounding objects: bandlimitedness VS compact support [27]
• Continuous band-limited scale characterization [28]
• Dyadic is not enough!
LIMITATIONS AND FUTURE WORK
45
Contact and more information: adrien.depeursinge@epfl.ch, http://bigww
References
[1] An Official ATS/ERS/JRS/ALAT Statement: Idiopathic Pulmonary Fibrosis: Evidence-based Guidelines for
Diagnosis and Management, G. Raghu et al., Am J Respir Crit Care Med 2011; 183(6):788-824
[2] VOW: Variance Optimal Wavelets for the Steerable Pyramid, P. Pad et al., IEEE ICIP 2014; 2973-2977
[3] 3D Steerable Wavelets and Monogenic Analysis for Bioimaging, N. Chenouard et al., IEEE ISBI 2011;
2132-2135
[4] A Parametric Texture Model Based on Joint Statistics of Complex Wavelet Coefficients, J. Portilla et al.,
Int Jour Comput Vision 2000; 1: 49-70
[5] Nonseparable radial frame multiresolution analysis in multidimensions and isotropic fast wavelet
algorithms, M. Papadakis et al., SPIE Wavelets 2003; 5207: 631-642
[6] Ten Lectures on Wavelets, I. Daubechies, SIAM 1992; 61
• bandwidth limited to
• generates tight frames
• Analytical approximation in Fourier:
Results
• The proposed texture
AUC and ACC for
isotropic wavelet pyra
Conclusions and
• New family of 3D isot
bandwidth to balance
operators and the infl
• Importance of rotation
wavelet bandwidth de
• Future work includes
higher orders of the R
. .
. .
. .
. .
. .
. .
[27] Optimized steerable wavelets for texture analysis of lung tissue in 3-D CT: classification of usual interstitial
pneumonia, Depeursinge et al., IEEE Int Symp on Biomed Imag (ISBI), 403-6, 2015
[28] Fast detection and refined scale estimation using complex isotropic wavelets, Püspöki et al., IEEE Int Symp on
Biomed Imag (ISBI), 512-5, 2015
spatial domain Fourier

46. (a) Synthetic image containing 3 visual concepts:
1) vertical lines (quadrants I and III),
2) checkerboard (quadrant II),
3) wiggled checkerboard (quadrant IV).
PCA 1
PCA2
10
1
10
2
10
3
(b) PCA visualization of 32⇥32 overlapping blocks and clus-
ters from the left image (N = 10, J = 4, K = 3). The tem-
plates 10
k corresponding to the respective visual concepts are
dislayed for scale j = 3.
10
• Model learning
• Limited performance for stochastic textures with no clear multi-scale
signature
• Reveal visual diversity with unsupervised learning [29,30]
LIMITATIONS AND FUTURE WORK
46
[29] Rotation-covariant visual concept detection using steerable Riesz wavelets and bags of visual words,
Depeursinge et al., SPIE Wavelets and Sparsity XV, 8858:885816-885816-11, 2013
[30] Unsupervised texture segmentation using monogenic curvelets and the Potts model, Storath et al., IEEE Int Conf
Imag Proc, 4348-52, 2014
(a) Synthetic image containing 3 visual concepts:
1) vertical lines (quadrants I and III),
2) checkerboard (quadrant II),
3) wiggled checkerboard (quadrant IV).
PCA 1
PCA2
10
1
10
2
10
3
(b) PCA visualization of 32⇥32 overlapping blocks and clus-
ters from the left image (N = 10, J = 4, K = 3). The tem-
plates 10
k corresponding to the respective visual concepts are
dislayed for scale j = 3.
Figure 5: Qualitative evaluation of the visual concepts 10
k found using K–means in the feature space spanned
20135
nvas0055)canvas0066)canvas0097)canvas0118)canvas021
anvas02613)canvas03114)canvas03215)canvas03316)canvas035
ile00621)carpet00222)carpet00423)carpet00524)carpet009
atabase.

47. • THANKS !
47
Matlab code available!
adrien.depeursinge@epfl.ch
MICCAI tutorial on Biomedical
Texture Analysis: Oct 5th in Münich
https://sites.google.com/site/btamiccai2015/

48. BIOMEDICAL TISSUE MODELING IN 2D AND 3D
• Interstitial lung diseases in CT
• Lung texture classification using locally-oriented Riesz components, Depeursinge A, Foncubierta-
Rodriguez A, Van de Ville D, Müller H, Med Image Comput Comput Assist Interv. (MICCAI)
2011;14(3):231-8.
• Multiscale lung texture signature learning using the Riesz transform, Depeursinge A, Foncubierta-
Rodriguez A, Van de Ville D, Müller H, Med Image Comput Comput Assist Interv. (MICCAI)
2012;15(3):517-24.
• Automated classification of usual interstitial pneumonia using regional volumetric texture analysis in
high-resolution CT, Depeursinge A, Chin A, Leung A, Terrone D, Bristow M, Rosen G, Rubin D, Invest
Radiol.,
in press.
• Pulmonary embolism in dual-energy CT
• Rotation-covariant texture analysis of 4D dual-energy CT as an indicator of local pulmonary
perfusion, Depeursinge A, Foncubierta-Rodriguez A, Vargas A, Van de Ville D, Platon A, Poletti PA,
48

49. BIOMEDICAL TISSUE MODELING IN 2D AND 3D
• Liver lesions in CT
• Predicting visual semantic descriptive terms from radiological image data: preliminary results with
liver lesions in CT, Depeursinge A, Kurtz C, Beaulieu C, Napel S, Rubin D, IEEE Trans Med Imag.
2014;33(8):1669-76.
• Brain epileptogenic lesions in MRI
• Epileptogenic lesion quantification in MRI using contralateral 3D texture comparisons, Jiménez del
Toro OA, Foncubierta-Rodríguez A, Vargas Gómez MI, Müller H, Depeursinge A, Med Image
Comput Comput Assist Interv. (MICCAI) 2013;16(2):353-60.
49

50. EVEN VS ODD ORDERS: 1-D
signal:
Heaviside
filter:
1st (dashed) and 2nd
order Gaussian
derivatives
convolution:
1st (dashed)
versus 2nd
50