We propose a novel imaging biomarker of lung cancer relapse from 3-D texture analysis of CT images. Three-dimensional morphological nodular tissue properties are described in terms of 3-D Riesz-wavelets. The responses of the latter are aggregated within nodular regions by means of feature covariances, which leverage rich intra- and inter-variations of the feature space dimensions. The obtained Riesz-covariance descriptors lie on a manifold governed by Riemannian geometry requiring specific geodesic metrics to locally approximate scalar products. The latter are used to construct a kernel for support vector machines (SVM). The effectiveness of the presented models is evaluated on a dataset of 92 patients with non-small cell lung carcinoma (NSCLC) and cancer recurrence information. Disease recurrence within a timeframe of 12 months could be predicted with an accuracy above 80, and highlighted the importance of covariance-based texture aggregation. At the end of the talk, computer tools will be presented to easily extract 3D radiomics quantitative features from PET-CT images.
Apidays New York 2024 - The value of a flexible API Management solution for O...
Ā
A 3-D Riesz-Covariance Texture Model for the Prediction of Nodule Recurrence in Lung CT
1. A 3āD RIESZāCOVARIANCE TEXTURE MODEL FOR THE
PREDICTION OF NODULE RECURRENCE IN LUNG CT
Pol Cirujeda, Yashin Dicente Cid, Henning MĆ¼ller, Daniel Rubin, Todd A. Aguilera,āØ
Billy W. Loo Jr., Maximilian Diehn, Xavier Binefa, Adrien Depeursinge
logY
TY
expY
Fig. 3: Mapping of points in a Sym+
d manifold to the tangent
space TY .
Due to the convexity of the Sym+
d manifold, the mean of a set
of covariance matrices Xi=1..N on a Riemannian manifold has
to be approximated in order to lay on the manifold ensuring:
Āµ({X}) = argmin
X02Sym+
JX
2
(Xn, X0
) , (10)
Riemannian distance, as depicted in Figure 4. Such a projec
tion demonstrates the following: a) the provided 3āD Rieszā
covariance descriptors are able to capture several class entities
b) the provided Riemannian metrics and mapping operators
are able to provide an adequate kernel for classiļ¬cation
and c) this classiļ¬cation separability correlates with clinica
knowledge on classes like recurrence locality of the nodules
and recurrence time annotations, as is analyzed in this article
ā5
ā4
ā3
ā2
ā1
0
2
4
6ā4
ā2
0
2
4
6
3D descriptor space embedding
Z
No failure
Local Failure
Regional failure
Distant methastasis
treatment failure
treatment success
quant. feat. #1
quant.feat.#2
2. OUTLINE
ā¢ Introduction
ā¢ Non-invasive personalized estimations of cancer treatment success
ā¢ Methods
ā¢ Texture operator: locally aligned 3-D Riesz wavelets
ā¢ Aggregation function: covariance matrices and Riemannian manifolds
ā¢ kernel for support vector machines
ā¢ Experiments
ā¢ Radiomics PET-CT computer tools
ā¢ Conclusions & future work
Sym+
P
3. OUTLINE
ā¢ Introduction
ā¢ Non-invasive personalized estimations of cancer treatment success
ā¢ Methods
ā¢ Texture operator: locally aligned 3-D Riesz wavelets
ā¢ Aggregation function: covariance matrices and Riemannian manifolds
ā¢ kernel for support vector machines
ā¢ Experiments
ā¢ Radiomics PET-CT computer tools
ā¢ Conclusions & future work
Sym+
P
4. ā¢ The structures of tumor tissue in CT reļ¬ects their nature
ā¢ E.g., active cancer cells, angiogenesis, necrosis [Aerts2014]
ā¢ Underlying cancer-related genomics [Gevaert2012]
ā¢ Cancer ecosystem is composed of micro-habitats [Gatenby2013]
ā¢ Relates to cancer subtype, patient survival, response to treatment
PREDICTING CANCER TREATMENT SUCCESS
5. ā¢ Goal: image-based personalized phenotyping
ā¢ Use 3-D texture analysis to predict response to stereotactic ablative
radiotherapy (SABR)
ā¢ Surrogate slow, costly and invasive molecular analysis
ā¢ Related work [Ganeshan2013, Ravanelli2013, Mattonen2014, Depeursinge2015]
ā¢ 2-D and suboptimal texture operators (isotropic, single scale)
ā¢ No separate analysis of nodule components
PERSONALIZED PHENOTYPING
treatment failure
treatment success
quant. feat. #1
quant.feat.#2
6. OUTLINE
ā¢ Introduction
ā¢ Non-invasive personalized estimations of cancer treatment success
ā¢ Methods
ā¢ Texture operator: locally aligned 3-D Riesz wavelets
ā¢ Aggregation function: covariance matrices and Riemannian manifolds
ā¢ kernel for support vector machines
ā¢ Experiments
ā¢ Radiomics PET-CT computer tools
ā¢ Conclusions & future work
Sym+
P
7. TEXTURE OPERATORS
7
ā¢ Texture operators [Depeursinge2014]
ā¢ A -dimensional texture analysis approach is characterized by a set of āØ
local operators centered at the position
ā¢ Each operator is local in the sense its response to an image only
depends on a subregion of
ā¢ The subregion is the support of the operator
N
d
L1 ā„ Ā· Ā· Ā· ā„ Ld
L1
L2
M1
M2
Ā·
m
m
L1 ā„ Ā· Ā· Ā· ā„ Ld
I(k)
k 2 M1 ā„ Ā· Ā· Ā· ā„ Md
M1
M2
L1
L2
gn
I(k)
gn(k, m)
8. TEXTURE OPERATORS
8
ā¢ Texture operators [Depeursinge2014]
ā¢ A -dimensional texture analysis approach is characterized by a set of āØ
local operators centered at the position
ā¢ Each operator is local in the sense its response to an image only
depends on a subregion of
ā¢ The subregion is the support of the operator
ā¢ For each position , the operator is applied (e.g., multiplied) to the image,
yielding response maps:
N
d
L1 ā„ Ā· Ā· Ā· ā„ Ld
L1
L2
M1
M2
Ā·
m
m
m
L1 ā„ Ā· Ā· Ā· ā„ Ld
I(k)
k 2 M1 ā„ Ā· Ā· Ā· ā„ Md
M1
M2
L1
L2
)gn
I(k)
response map
gn(k, m)
9. TEXTURE OPERATORS
9
ā¢ Texture operators
ā¢ Example: response maps ofāØ
multi-scale operators
ā¢ Multi-directional operators:
scale 1 scale 2 scale 3 scale 4
g1 g2 g3 g4
IA IB
XX 2013 2
otationā
ar pixels
ovariant
elatively
N = 1 G ā¤ R(0,1) G ā¤ R(1,0)
N = 2 G ā¤ R(0,2) G ā¤ R(1,1) G ā¤ R(2,0)
TIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2013 2
e operatorsā outputs over the instances. Rotationā
BPs are obtained by using āuniformā circular pixels
hat are rotationāinvariant [39]. Rotationācovariant
RIFT [31]) measures HOG orientations relatively
N = 1 G ā¤ R(0,1) G ā¤ R(1,0)
N = 2 G ā¤ R(0,2) G ā¤ R(1,1) G ā¤ R(2,0)
10. TEXTURE OPERATOR
10
ā¢ Locally-oriented 3-D steerable Riesz wavelets
ā¢ Rotation-invariant characterization of the local organization of image
directions (LOID) is important for characterizing local tissue architectures
[Depeursinge2014]
ael Unser
b)
reattentive texture segregation [3].
easily separated from L-shaped
patterns (left) are found to be more
can be distinguished by counting
11. TEXTURE OPERATOR
ā¢ Locally-oriented 3-D steerable Riesz wavelets
ā¢ th-order Riesz transform in 3-D in Fourier [Unser2011]āØ
āØ
āØ
āØ
āØ
yields for all combinations of
N
ā
N + 2
2
ā
n1 + n2 + n3 = N, n1,2,3 2 N
R(n1,n2,n3){f}(!) = ( j)N
r
N!
n1!n2!n3!
!n1
1 !n2
2 !n3
3
||!||n1+n2+n3
Ėf(!),
12. TEXTURE OPERATOR
ā¢ Locally-oriented 3-D steerable Riesz wavelets
ā¢ th-order Riesz transform in 3-D in Fourier [Unser2011]āØ
āØ
āØ
āØ
āØ
yields for all combinations of
ā¢ Example
N
ā
N + 2
2
ā
n1 + n2 + n3 = N, n1,2,3 2 N
R(n1,n2,n3){f}(!) = ( j)N
r
N!
n1!n2!n3!
!n1
1 !n2
2 !n3
3
||!||n1+n2+n3
Ėf(!),
2
ļ¬nition
of the
visual
ith the
expert
to ļ¬nd
ons of
k, and
s in a
G ā¤ R(2,0,0)
G ā¤ R(0,2,0)
G ā¤ R(0,0,2)
G ā¤ R(1,1,0)
G ā¤ R(1,0,1)
G ā¤ R(0,1,1)
N = 2
' ā¤ R(2,0,0)
' ā¤ R(0,2,0)
' ā¤ R(0,0,2)
' ā¤ R(0,1,1)
' ā¤ R(1,0,1)
' ā¤ R(1,1,0)
13. TEXTURE OPERATOR
13
ā¢ Locally-oriented 3-D steerable Riesz wavelets
ā¢ th-order Riesz transform in 3-D in Fourier [Unser2011]āØ
āØ
āØ
āØ
āØ
yields for all combinations of
ā¢ Steerability [Chenouard2012]āØ
āØ
āØ
is a rotation matrix and is the corresponding steering matrix
N
ā
N + 2
2
ā
n1 + n2 + n3 = N, n1,2,3 2 N
RN
{fR} = SRRN
{f}
R 3 ā„ 3 SR
R(n1,n2,n3){f}(!) = ( j)N
r
N!
n1!n2!n3!
!n1
1 !n2
2 !n3
3
||!||n1+n2+n3
Ėf(!),
14. TEXTURE OPERATOR
14
ā¢ Locally-oriented 3-D steerable Riesz wavelets
ā¢ th-order Riesz transform in 3-D in Fourier [Unser2011]āØ
āØ
āØ
āØ
āØ
yields for all combinations of
ā¢ Steerability [Chenouard2012]āØ
āØ
āØ
is a rotation matrix and is the corresponding steering matrixāØ
ā¢ Spatial support
ā¢ Isotropic dyadic wavelet frames
N
ā
N + 2
2
ā
n1 + n2 + n3 = N, n1,2,3 2 N
RN
{fR} = SRRN
{f}
R 3 ā„ 3 SR
R(n1,n2,n3){f}(!) = ( j)N
r
N!
n1!n2!n3!
!n1
1 !n2
2 !n3
3
||!||n1+n2+n3
Ėf(!),
of order ā1/2 (an isotropic smoothing operator) of f: Rf =
āāāā1/2
f. Letās indeed recall the Fourier-domain deļ¬nition of
these operators: ā
F
āā jĻ and āā1/2 F
āā ||Ļ||ā1
. Unlike the
usual gradient ā, the Riesz transform is self-reversible
Rā
Rf(Ļ) =
(jĻ)ā
(jĻ)
||Ļ||2
Ėf(Ļ) = Ėf(Ļ).
This allows us to deļ¬ne a self-invertible wavelet frame of L2(R3
)
(tight frame). We however see that there exists a singularity for the
frequency (0, 0, 0). This issue will be ļ¬xed later, thanks to the van-
ishing moments of the primary wavelet transform.
RN
{f ā¤ i}[k]
Ėi(!)
ā”
2i
L1 ā„ L2 ā„ L3
16. OUTLINE
ā¢ Introduction
ā¢ Non-invasive personalized estimations of cancer treatment success
ā¢ Methods
ā¢ Texture operator: locally aligned 3-D Riesz wavelets
ā¢ Aggregation function: covariance matrices and Riemannian manifolds
ā¢ kernel for support vector machines
ā¢ Experiments
ā¢ Radiomics PET-CT computer tools
ā¢ Conclusions & future work
Sym+
P
17. FEATURE MAPS AND AGGREGATION FUNCTIONS
ā¢ From texture operators to texture measurements
ā¢ The operator is typically applied to all positions of the image
by āslidingā its window over the image
ā¢ Yields feature maps (potentially concatenating outputs from several operators)
ā¢ Regional texture measurements can be obtained from the aggregation of
over a region of interest
ā¢ E.g., provide estimates of features statistics
L1
L2
M1
M2
L1 ā„ Ā· Ā· Ā· ā„ Ld
Ā·
m
M
M
m
gn[k, m]
gn[f[k], m]
Mmargin
Mtexture
18. ā¢ For instance, integration can be used to aggregate the vectors āØ
over
ā¢ Average
ā¢ The average of absolute values can be used for bandlimited operators
INTEGRATIVE AGGREGATION FUNCTIONS
18
M
ā¢ From texture operators to texture measurements
ā¢ The operator is typically applied to all positions
by āslidingā its window over the image
ā¢ Regional texture measurements can be obtained
aggregation of over a region of interest
ā¢ For instance, integration can be used to aggregate
ā¢ e.g., average:
L1
L2
M1
M2
L1 ā„ Ā· Ā· Ā· ā„ Ld
Ā·
gn(x, m)
Āµ 2 RP
gn(f(x), m) M
m
gn(f(x
Āµ =
0
B
@
Āµ1
...
ĀµP
1
C
A =
1
|M|
Z
M
gn(f(x), m) p=1,...,P
dm
M
'm = gn[f[k], m] 2 RP
Āµ =
0
B
@
Āµ1
...
ĀµP
1
C
A =
1
|M|
X
m2M
'm
19. INTEGRATIVE AGGREGATION FUNCTIONS
ā¢ How large must be the region of interest ?
ā¢ No more than enough to evaluate texture stationarity āØ
in terms of human perception / tissue biology
ā¢ Example with operator: undecimated isotropic Simoncelliās dyadic wavelets
[Portilla2000] applied to all image positions
ā¢ Operatorsā responses are averaged over
M
ā¢ The operator is typically applied to all position
by āslidingā its window over the image
ā¢ Regional texture measurements can be obtained
aggregation of over a region of interest
ā¢ For instance, integration can be used to aggregate
ā¢ e.g., average:
L1
L2
M1
M2
L1 ā„ Ā· Ā· Ā· ā„ Ld
Ā·
gn(x, m)
Āµ 2 RP
gn(f(x), m) M
m
gn(f(
Āµ =
0
B
@
Āµ1
...
ĀµP
1
C
A =
1
|M|
Z
M
gn(f(x), m) p=1,...,P
dm
M
f(x) g1(f(x), m)
m 2 RM1ā„M2
g2(f(x), m)
original image with
regions I
1
|M|
Z
M
|g1(f(x), m)|dm
M
feature space
1
|M|
Z
M
|g2(f(x),m)|dm
f(x)
Ma, Mb, Mc
The averaged responses
over the entire image
does not correspond āØ
to anything visually!
Ėg1(ā¢) =
ā¢
cos ā”
2 log2
2ā¢
ā” , ā”
4 < ā¢ ļ£æ ā”
0, otherwise.
Ėg2(ā¢) =
ā¢
cos ā”
2 log2
4ā¢
ā” , ā”
8 < ā¢ ļ£æ ā”
2
0, otherwise.
g1,2 f(ā¢, ) = Ėg1,2(ā¢, ) Ā· Ėf(ā¢, )
Nor biologically!
20. ā¢ For instance, integration can be used to aggregate the vectors āØ
over
ā¢ Average
ā¢ The average of absolute values can be used for bandlimited operators
ā¢ Covariance matrix
ā¢ Encodes pixelwise inter-feature variations [Cirujeda2015]
ā¢ Variance is a reasonable statistic for bandlimited operators
ā¢ Can be vectorized to keep unique elements as
INTEGRATIVE AGGREGATION FUNCTIONS
20
M
ā¢ From texture operators to texture measurements
ā¢ The operator is typically applied to all positions
by āslidingā its window over the image
ā¢ Regional texture measurements can be obtained
aggregation of over a region of interest
ā¢ For instance, integration can be used to aggregate
ā¢ e.g., average:
L1
L2
M1
M2
L1 ā„ Ā· Ā· Ā· ā„ Ld
Ā·
gn(x, m)
Āµ 2 RP
gn(f(x), m) M
m
gn(f(x
Āµ =
0
B
@
Āµ1
...
ĀµP
1
C
A =
1
|M|
Z
M
gn(f(x), m) p=1,...,P
dm
M
'm = gn[f[k], m] 2 RP
= vec(X) = X1,1,
p
2X1,2, . . . ,
p
2X1,P , X2,2,
p
2X2,3, . . . XP,P
X =
1
|M| 1
X
m2M
('m ĀµM )('m ĀµM )T
Āµ =
0
B
@
Āµ1
...
ĀµP
1
C
A =
1
|M|
X
m2M
'm
2 RP (P +1)/2
21. ā¢ Covariance matrices lie in Riemannian manifolds of real
symmetric positive deļ¬nite matrices [Pennec2006]
ā¢ Euclidean distance between different texture regions fails
RIEMANNIAN MANIFOLDS
Sym+
P
Sym+
P
1
2
3
Mj
21
22. ā¢ Covariance matrices lie in Riemannian manifolds of real
symmetric positive deļ¬nite matrices [Pennec2006]
ā¢ Euclidean distance between different texture regions fails
ā¢ Meaningful distances exist:
ā¢ e.g., [Fƶrstner2003]: āØ
āØ
āØ
where and are the elements of SVD of āØ
āØ
Therefore: āØ
āØ
āØ
where are the positive eigenvalues of
RIEMANNIAN MANIFOLDS
Sym+
P
Sym+
P
1
2
3
(X1, X2) =
s
trace
ā
log
ā£
X
1
2
1 X2X
1
2
1
ā2
ā
,
log(X) = Ulog(D)UT
,
SVD of X: X=UDV^T
other distances:
Jensen-Bregman divergence
U D X 2 Sym+
P
(X1, X2) =
v
u
u
t
PX
i=1
log( i)2,
X
1
2
1 X2X
1
2
1i
Mj
22
23. ā¢ What if we want to do more than measuring distances on the
manifold?
ā¢ e.g., computing scalar products?
RIEMANNIAN MANIFOLDS
23
24. ā¢ What if we want to do more than measuring distances on the
manifold?
ā¢ e.g., computing scalar products?
ā¢ Local estimations of the manifold can be obtained by
projecting in a tangent space at reference projection
point
RIEMANNIAN MANIFOLDS
logY
TY
expY
Fig. 3: Mapping of points in a Sym+
manifold to the tangent
Riemannian
tion demonst
covariance de
b) the provid
are able to
and c) this c
knowledge o
and recurrenc
Xi
Y 2 Sym+
P
TY
X2X1
x2x1
Sym+
P
24
25. ā¢ Projections are obtained by the point-dependent operationāØ
[Arsigny2006]āØ
āØ
āØ
and we can come back
RIEMANNIAN MANIFOLDS
logY
TY
expY
Fig. 3: Mapping of points in a Sym+
manifold to the tangent
Riemannian
tion demonst
covariance de
b) the provid
are able to
and c) this c
knowledge o
and recurrenc
X2X1
x2x1
Sym+
P
logY
expY
x = logY (X) = Y
1
2 log
ā£
Y
1
2 XY
1
2
ā
Y
1
2
X = expY (x) = Y
1
2 exp
ā£
Y
1
2 xY
1
2
ā
Y
1
2
25
26. ā¢ Now we can use the Euclidean metric on the tangent space
ā¢ Scalar product between two points and [Pennec2006]:
ā¢ It can be used to deļ¬ne a kernel for e.g., support vector machines (SVM)
RIEMANNIAN MANIFOLDS
logY
TY
expY
Fig. 3: Mapping of points in a Sym+
manifold to the tangent
Riemannian
tion demonst
covariance de
b) the provid
are able to
and c) this c
knowledge o
and recurrenc
X2X1
x2x1
Sym+
P
logY
expY
TY
x2x1
hx1, x2iY = trace x1Y 1
x2Y 1
26
27. ā¢ How to choose the projection point ?
ā¢ Reminder: is a local estimation of
ā¢ should be near to all of the dataset
RIEMANNIAN MANIFOLDS
logY
TY
expY
Fig. 3: Mapping of points in a Sym+
manifold to the tangent
Riemannian
tion demonst
covariance de
b) the provid
are able to
and c) this c
knowledge o
and recurrenc
X2X1
x2x1
Sym+
P
logY
expY
Y 2 Sym+
P
Sym+
PTY
Y Xj
27
28. ā¢ How to choose the projection point ?
ā¢ Reminder: is a local estimation of
ā¢ should be near to all of the dataset
ā¢ The mean of covariances is a natural choice [Pennec2006]:
ā¢ can be estimated with gradient descent and iterative re-projectionāØ
[Pennec2006, Karcher1977, Moakher2005]
ā¢ is convex
RIEMANNIAN MANIFOLDS
Y 2 Sym+
P
Sym+
PTY
Y Xj
XĀµ : argmin
XĀµ2Sym+
d
JX
j=1
2
(Xj, XĀµ)
Y = XĀµ
XĀµ
Sym+
P
28
29. OUTLINE
ā¢ Introduction
ā¢ Non-invasive personalized estimations of cancer treatment success
ā¢ Methods
ā¢ Texture operator: locally aligned 3-D Riesz wavelets
ā¢ Aggregation function: covariance matrices and Riemannian manifolds
ā¢ kernel for support vector machines
ā¢ Experiments
ā¢ Radiomics PET-CT computer tools
ā¢ Conclusions & future work
Sym+
P
31. ā¢ SVM kernel
ā¢ Scalar products on the tangent space can be used to deļ¬ne a SVM
kernel based on Riemannian metrics
SVM KERNEL
31
Sym+
P
TY
logY
TY
expY
Fig. 3: Mapping of points in a Sym+
manifold to the tangent
Riemannian d
tion demonstr
covariance des
b) the provide
are able to p
and c) this cl
knowledge on
and recurrence
hY ( i) = hw, iiY + b
32. OUTLINE
ā¢ Introduction
ā¢ Non-invasive personalized estimations of cancer treatment success
ā¢ Methods
ā¢ Texture operator: locally aligned 3-D Riesz wavelets
ā¢ Aggregation function: covariance matrices and Riemannian manifolds
ā¢ kernel for support vector machines
ā¢ Experiments
ā¢ Radiomics PET-CT computer tools
ā¢ Conclusions & future work
Sym+
P
33. ā¢ Patients
ā¢ 92 non-small cell lung carcinoma (NSCLC) from Stanford Hospital and Clinics
ā¢ Gross tumor volume, ground glass (GGO) and solid āØ
regions contoured in CT in 3-D
ā¢ Disease-free survival times available
ā¢ Estimation of the generalization performance with a 10-fold cross-validation (CV)
ā¢ Each training fold
ā¢ Computation of the projection point
ā¢ Training of SVMs with the kernel
ā¢ Each test fold
ā¢ Classify test patients and compute classiļ¬cation accuracy
EXPERIMENTS
33
0 5 10 15 20
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
months after SABR treatment
0 5 10 15 20
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
recurrence
M
XĀµ
(w; b) Sym+
P
34. ā¢ Results
ā¢ Accuracies (Riesz order , 3 dyadic scales, 5 Monte-Carlo CV repetitions)
ā¢ 12 months: 23 recurrences versus 62 remissions
ā¢ 24 months: 30 recurrences versus 62 remissions
ā¢ Observations
ā¢ Predicts treatment failure in ļ¬rst 12 months with accuracy > 80%
EXPERIMENTS
34
0 5 10 15 20
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
months after SABR treatment
0 5 10 15 20
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
recurrence
N = 2
9
TABLE I: Results for the binary classiļ¬cation of patient recurrence, using shortā (12 months) and longāterm (24 months)
binarization and several nodule region descriptors. Table A presents the performance evaluation of the presented kernelābased
SVM formulation for covariance-based descriptors. Table B shows the results of a linear SVM for plain vectorized covariance
descriptors. Finally, Table C assesses the performance of a linear SVM using the average of the 3āD Riesz ļ¬lter responses
within the delineated region as templates (e.g., corresponding to our approach in [18]). The shortāterm experiment involved
23 recurrences versus 62 remissions. The longāterm experiment involved 30 recurrences versus 62 remissions. Table values
are expressed in terms of CV repetition averages Ā± standard deviations.
A) 12 MONTHS ā SVM KERNEL 24 MONTHS ā SVM KERNEL
Features accuracy
sensitivity
(T P/T P +F N)
speciļ¬city
(T N/T N+F P ) F1-score accuracy
sensitivity
(T P/T P +F N)
speciļ¬city
(T N/T N+F P ) F1-score
GGO 81.33 Ā± 0.12 87.38 Ā± 0.05 78.33 Ā± 0.13 80.75 Ā± 0.12 54.94 Ā± 0.12 61.64 Ā± 0.14 58.51 Ā± 0.05 53.74 Ā± 0.07
Solid 82.00 Ā± 0.15 85.14 Ā± 0.13 76.67 Ā± 0.14 78.13 Ā± 0.14 57.33 Ā± 0.05 68.98 Ā± 0.08 50.89 Ā± 0.02 49.37 Ā± 0.03
GTV 82.67 Ā± 0.17 87.62 Ā± 0.05 78.33 Ā± 0.13 80.89 Ā± 0.12 44.69 Ā± 0.15 63.33 Ā± 0.22 47.80 Ā± 0.15 41.77 Ā± 0.10
B) 12 MONTHS ā LINEAR SVM VECT. COVARIANCES 24 MONTHS ā LINEAR SVM VECT. COVARIANCES
Features accuracy
sensitivity
(T P/T P +F N)
speciļ¬city
(T N/T N+F P ) F1-score accuracy
sensitivity
(T P/T P +F N)
speciļ¬city
(T N/T N+F P ) F1-score
GGO 78.67 Ā± 0.13 83.62 Ā± 0.07 75.67 Ā± 0.17 77.42 Ā± 0.17 49.63 Ā± 0.15 58.89 Ā± 0.05 56.92 Ā± 0.15 52.30 Ā± 0.16
Solid 80.67 Ā± 0.09 84.57 Ā± 0.05 74.32 Ā± 0.12 75.89 Ā± 0.12 57.33 Ā± 0.11 67.11 Ā± 0.06 58.01 Ā± 0.03 56.24 Ā± 0.09
GTV 81.32 Ā± 0.15 84.38 Ā± 0.09 75.72 Ā± 0.18 76.79 Ā± 0.18 44.87 Ā± 0.08 57.71 Ā± 0.11 48.76 Ā± 0.07 42.86 Ā± 0.09
C) 12 MONTHS ā LINEAR SVM FOR FEATURES AVERAGES 24 MONTHS ā LINEAR SVM FOR MEAN OF FEATURES TEMPLATE
Features accuracy
sensitivity
(T P/T P +F N)
speciļ¬city
(T N/T N+F P ) F1-score accuracy
sensitivity
(T P/T P +F N)
speciļ¬city
(T N/T N+F P ) F1-score
GGO 74.38 Ā± 0.08 77.65 Ā± 0.14 68.97 Ā± 0.07 69.58 Ā± 0.07 46.67 Ā± 0.25 50.00 Ā± 0.23 50.41 Ā± 0.23 46.41 Ā± 0.25
Solid 79.90 Ā± 0.14 85.96 Ā± 0.09 76.45 Ā± 0.11 76.20 Ā± 0.15 53.33 Ā± 0.20 55.90 Ā± 0.23 53.60 Ā± 0.18 52.04 Ā± 0.19
GTV 75.62 Ā± 0.17 83.17 Ā± 0.15 70.24 Ā± 0.17 68.97 Ā± 0.19 51.67 Ā± 0.15 53.62 Ā± 0.15 52.17 Ā± 0.16 46.80 Ā± 0.15
35. ā¢ Results
ā¢ Accuracies (Riesz order , 3 dyadic scales, 5 Monte-Carlo CV repetitions)
ā¢ 12 months: 23 recurrences āØ
versus 62 remissions
ā¢ Observations:
ā¢ A) vs B): kernel improves āØ
over plain vectorized SVMs
EXPERIMENTS
35
0 5 10 15 20
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
months after SABR treatment
0 5 10 15 20
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
recurrence
N = 2
TABLE I: Results for the binary classiļ¬cation of patient recurrence,
binarization and several nodule region descriptors. Table A presents the
SVM formulation for covariance-based descriptors. Table B shows the
descriptors. Finally, Table C assesses the performance of a linear SV
within the delineated region as templates (e.g., corresponding to our a
23 recurrences versus 62 remissions. The longāterm experiment invol
are expressed in terms of CV repetition averages Ā± standard deviation
A) 12 MONTHS ā SVM KERNEL
Features accuracy
sensitivity
(T P/T P +F N)
speciļ¬city
(T N/T N+F P ) F1-score
GGO 81.33 Ā± 0.12 87.38 Ā± 0.05 78.33 Ā± 0.13 80.75 Ā± 0.12 54
Solid 82.00 Ā± 0.15 85.14 Ā± 0.13 76.67 Ā± 0.14 78.13 Ā± 0.14 57
GTV 82.67 Ā± 0.17 87.62 Ā± 0.05 78.33 Ā± 0.13 80.89 Ā± 0.12 44
B) 12 MONTHS ā LINEAR SVM VECT. COVARIANCES
Features accuracy
sensitivity
(T P/T P +F N)
speciļ¬city
(T N/T N+F P ) F1-score
GGO 78.67 Ā± 0.13 83.62 Ā± 0.07 75.67 Ā± 0.17 77.42 Ā± 0.17 49
Solid 80.67 Ā± 0.09 84.57 Ā± 0.05 74.32 Ā± 0.12 75.89 Ā± 0.12 57
GTV 81.32 Ā± 0.15 84.38 Ā± 0.09 75.72 Ā± 0.18 76.79 Ā± 0.18 44
C) 12 MONTHS ā LINEAR SVM FOR FEATURES AVERAGES 2
Features accuracy
sensitivity
(T P/T P +F N)
speciļ¬city
(T N/T N+F P ) F1-score
GGO 74.38 Ā± 0.08 77.65 Ā± 0.14 68.97 Ā± 0.07 69.58 Ā± 0.07 46
Solid 79.90 Ā± 0.14 85.96 Ā± 0.09 76.45 Ā± 0.11 76.20 Ā± 0.15 53
GTV 75.62 Ā± 0.17 83.17 Ā± 0.15 70.24 Ā± 0.17 68.97 Ā± 0.19 51
TABLE II: Comparison with other studies predicting tumor recurren
Sym+
P
36. ā¢ Results
ā¢ Accuracies (Riesz order , 3 dyadic scales, 5 Monte-Carlo CV repetitions)
ā¢ 12 months: 23 recurrences āØ
versus 62 remissions
ā¢ Observations:
ā¢ A) vs B): kernel improves āØ
over plain vectorized SVMs
ā¢ C) vs A), B): Averaging damagesāØ
performance, especially when the āØ
tissue is non-stationary
ā¢ Covariance aggregation keeps āØ
pixelwise interaction between features
EXPERIMENTS
36
0 5 10 15 20
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
months after SABR treatment
0 5 10 15 20
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
recurrence
N = 2
TABLE I: Results for the binary classiļ¬cation of patient recurrence,
binarization and several nodule region descriptors. Table A presents th
SVM formulation for covariance-based descriptors. Table B shows the
descriptors. Finally, Table C assesses the performance of a linear SV
within the delineated region as templates (e.g., corresponding to our
23 recurrences versus 62 remissions. The longāterm experiment invo
are expressed in terms of CV repetition averages Ā± standard deviation
A) 12 MONTHS ā SVM KERNEL
Features accuracy
sensitivity
(T P/T P +F N)
speciļ¬city
(T N/T N+F P ) F1-score
GGO 81.33 Ā± 0.12 87.38 Ā± 0.05 78.33 Ā± 0.13 80.75 Ā± 0.12 5
Solid 82.00 Ā± 0.15 85.14 Ā± 0.13 76.67 Ā± 0.14 78.13 Ā± 0.14 5
GTV 82.67 Ā± 0.17 87.62 Ā± 0.05 78.33 Ā± 0.13 80.89 Ā± 0.12 4
B) 12 MONTHS ā LINEAR SVM VECT. COVARIANCES
Features accuracy
sensitivity
(T P/T P +F N)
speciļ¬city
(T N/T N+F P ) F1-score
GGO 78.67 Ā± 0.13 83.62 Ā± 0.07 75.67 Ā± 0.17 77.42 Ā± 0.17 4
Solid 80.67 Ā± 0.09 84.57 Ā± 0.05 74.32 Ā± 0.12 75.89 Ā± 0.12 5
GTV 81.32 Ā± 0.15 84.38 Ā± 0.09 75.72 Ā± 0.18 76.79 Ā± 0.18 4
C) 12 MONTHS ā LINEAR SVM FOR FEATURES AVERAGES 2
Features accuracy
sensitivity
(T P/T P +F N)
speciļ¬city
(T N/T N+F P ) F1-score
GGO 74.38 Ā± 0.08 77.65 Ā± 0.14 68.97 Ā± 0.07 69.58 Ā± 0.07 4
Solid 79.90 Ā± 0.14 85.96 Ā± 0.09 76.45 Ā± 0.11 76.20 Ā± 0.15 5
GTV 75.62 Ā± 0.17 83.17 Ā± 0.15 70.24 Ā± 0.17 68.97 Ā± 0.19 5
TABLE II: Comparison with other studies predicting tumor recurren
Sym+
P
37. OUTLINE
ā¢ Introduction
ā¢ Non-invasive personalized estimations of cancer treatment success
ā¢ Methods
ā¢ Texture operator: locally aligned 3-D Riesz wavelets
ā¢ Aggregation function: covariance matrices and Riemannian manifolds
ā¢ kernel for support vector machines
ā¢ Experiments
ā¢ Radiomics PET-CT computer tools
ā¢ Conclusions & future work
Sym+
P
38. ā¢ Web service for PET-CT image analysis in 3D
1. Upload a zip ļ¬le containing a ābatchā of patients:
ā¢ PET and CT image series āØ
( anonymized data)
ā¢ DICOM RT structure with āØ
gross tumor volume (GTV) āØ
delineated (e.g., āGTV Tā)
2. Wait (~5-10 min for upload and processing)
Structure of BatchPatients.zip:
COMPUTER TOOL: WEB SERVICE
38
2
6
4
(1)
...
(C)
3
7
5 = U
2
6
4
(0)
...
(N)
3
7
5
39. ā¢ Web service for PET-CT image analysis in 3D
1. Upload a zip ļ¬le containing a ābatchā of patients:
ā¢ PET and CT image series āØ
( anonymized data)
ā¢ DICOM RT structure with āØ
gross tumor volume (GTV) āØ
delineated (e.g., āGTV Tā)
2. Wait (~5-10 min for upload and processing)
3. Download a spreadsheet with a list of quantitative image features:
COMPUTER TOOL: WEB SERVICE
39
40. ā¢ Web service for PET-CT image analysis in 3D
ā¢ Preprocessing
ā¢ PET-CT alignment, scale normalization with mm voxel size
ā¢ Intensity features from PET
ā¢ SUVmax, tumorVolume
ā¢ SUVmean, SUVvariance, SUVskewness, SUVkurtosis, SUVpeak, MTV, TLGāØ
from multiple metabolic regions based on minimum SUV thresholds :
ā¢ Absolute (SUV):
ā¢ Relative to SUVmax (%):
ā¢ Intensity features from CT
ā¢ HUmean for , (SUV) et (SUVmax)
COMPUTER TOOL: WEB SERVICE
40
0.75 ā„ 0.75 ā„ 0.75
p
. . . . . .
2.5 5 8
p 2 [2.5 : 0.5 : 8]
p 2 [30 35 40 : 2 : 60 65 70]
Mp
M2.5 M5 M8
p = 3 p = 42%Mp
41. @
@x
@
@y
@
@z
ā¢ Web service for PET-CT image analysis in 3D
ā¢ 3D texture from PET and CT
ā¢ 3D LoG with scales
ā¢ 3D 1st-order Riesz (i.e., aligned gradients) with 4 dyadic scales
ā¢ 3D GLCMs with and averaged over all directionsāØ
(i.e., rotation-invariant)
ā¢ 11 GLCM features (see [Haralick1973, Soh1999, Clausi2002] for deļ¬nitions): āØ
Contrast, correlation, energy, homogeneity, entropy, InverseDiffMoment, SumAverage,
SumEntropy, SumVariance, DiffVariance, DiffEntropy
COMPUTER TOOL: WEB SERVICE
41
Table 3
Comparison of the various techniques used for 3-D biomedical texture analysis.
Technique Example of primitive Primitive neighborhood Illumination invariance Typical coverage of 3-D directions
GLCMs Voxel pairs No Incomplete for R > 1
RLE Linear No Incomplete for R > 1
scale 1 scale 2
LoG = 0.25 : 0.5 : 2.25
. . .
. . .
Mmargin
Mtexture
012,Depeursinge2015]
mage derivatives
een voxel values
41
terize the morphological properties of lung tissue associated with
tial lung diseases.16,17,20,21
They consist in counting the co-
ence of voxels with identical gray level values that are separated
stance d, which results in a co-occurrence matrix. Eleven statistics
xtracted from these matrices29
as texture attributes. The choices
d the number of gray levels were optimized by considering values
; 3} and {8, 16, 32}, respectively. The size of the vector of attri-
l was 540 for the gray-level histogram attributes (called HU there-
) and 396 for the GLCM attributes.
RESULTS
A leave-one-patient-out cross-validation evaluation was used to
te the performance of the proposed approach. The leave-one-
-out cross-validation consisted of using all patients but 1 to train
VM model and to measure the prediction performance on the re-
g test patient. The prediction performance was then averaged
ll possible combinations of training and test patients. Receiver
ng characteristic (ROC) curves of the system's performance in
ying between classic and atypical UIP are shown in Figure 4 for
nt feature groups and their combinations. The ROC curves were
ed by varying the decision threshold between the minimum and
ions X, Y, Z and 3 diagonals XY, XZ, and YZ. Figure 2 can be viewed
www.investigativeradiology.com 3
d reproduction of this article is prohibited.
y
@2
@x@z
@2
@y@z
scale 2
dGLCM = 1
42. ā¢ Web service for PET-CT image analysis in 3D
ā¢ 2 measures of metastasis spread [Fried2016]
ā¢ : distance between the primary tumor and the āØ
barycenter of the metastases (TNdistance)
ā¢ : sum of distances between each metastasis and the āØ
āØ
barycenter of the metastases (MetSpread)
COMPUTER TOOL: WEB SERVICE
42
kT
k ĀÆM
dT M
dTM = ||kT k ĀÆM ||
dMet =
X
i
||kMi
k ĀÆM ||
kM1
kM2
43. OUTLINE
ā¢ Introduction
ā¢ Non-invasive personalized estimations of cancer treatment success
ā¢ Methods
ā¢ Texture operator: locally aligned 3-D Riesz wavelets
ā¢ Aggregation function: covariance matrices and Riemannian manifolds
ā¢ kernel for support vector machines
ā¢ Experiments
ā¢ Radiomics PET-CT computer tools
ā¢ Conclusions & future work
Sym+
P
44. ā¢ Predicts treatment failure within 12 months (accuracy > 80%)
ā¢ Covariance manifolds provides an elegant framework for
aggregating texture feature maps
ā¢ Keeps pixelwise (local) interaction between features
ā¢ Can be used with any texture operator
ā¢ Riemannian metrics and estimated scalar products showed to improve āØ
over plain vectorized covariance matrices
ā¢ Radiomics computer tools are available
ā¢ Future work
ā¢ Further validation with more and multi-centric patients
ā¢ Estimate the impact of the choice of the projection point
CONCLUSIONS AND FUTURE WORK
44
TEXTURE OPERATORS AND PRIMITIVES
ā¢ From texture operators to texture measurements
ā¢ The operator is typically applied to all positions
by āslidingā its window over the image
ā¢ Regional texture measurements can be obtained
aggregation of over a region of interest
ā¢ For instance, integration can be used to aggregate
ā¢ e.g., average:
L1
L2
M1
M2
L1 ā„ Ā· Ā· Ā· ā„ Ld
Ā·
gn(x, m)
Āµ 2 RP
gn(f(x), m) M
m
gn(f(
Āµ =
0
B
@
Āµ1
...
ĀµP
1
C
A =
1
|M|
Z
M
gn(f(x), m) p=1,...,P
dm
M
COMPUTER TOOL: WEB SERVICE
45
45. REFERENCES (SORTING IN ALPHABETICAL ORDER)
45
[Aerts2014] Aerts, H. J. W. L.; Velazquez, E. R.; Leijenaar, R. T. H.; Parmar, C.; Grossmann, P.; Carvalho, S.; Bussink, J.; Monshouwer, R.; Haibe-Kains, B.;
Rietveld, D.; Hoebers, F.; Rietbergen, M. M.; Leemans, C. R.; Dekker, A.; Quackenbush, J.; Gillies, R. J. & Lambin, P.āØ
Decoding Tumour Phenotype by Noninvasive Imaging Using a Quantitative Radiomics ApproachāØ
Nature Communications, Nature Publishing Group, a division of Macmillan Publishers Limited. All Rights Reserved, 2014, 5
[Arsigny2006] Arsigny, V.; Fillard, P.; Pennec, X. & Ayache, N.
Log-Euclidean metrics for fast and simple calculus on diffusion tensors
Magnetic Resonance in Medicine, 2006, 56, 411-421
[Chenouard2011] Chenouard, N. & Unser, M.āØ
3D Steerable Wavelets and Monogenic Analysis for BioimagingāØ
2011 IEEE International Symposium on Biomedical Imaging: From Nano to Macro, 2011, 2132-2135
[Chenouard2012] Chenouard, N. & Unser, M.āØ
3D Steerable Wavelets in PracticeāØ
IEEE Transactions on Image Processing, 2012, 21, 4522-4533
[Cirujeda2015] Cirujeda, P.; Dicente Cid, Y.; Mateo, X. & Binefa, X.
A 3D Scene Registration Method via Covariance Descriptors and an Evolutionary Stable Strategy Game Theory Solver
International Journal of Computer Vision, 2015, 115, 306-329
[Cortes1995] Cortes, C. & Vapnik, V.
Support-vector networks
Machine learning, Springer, 1995, 20, 273-297
[Depeursinge2014] Depeursinge, A.; Foncubierta-Rodrguez, A.; Van De Ville, D. & MĆ¼ller, H.
Three-dimensional solid texture analysis and retrieval in biomedical imaging: review and opportunities
Medical Image Analysis, 2014, 18, 176-196
[Depeursinge2015] Depeursinge, A.; Yanagawa, M.; Leung, A. N. & Rubin, D. L.
Predicting Adenocarcinoma Recurrence Using Computational Texture Models of Nodule Components in Lung CT
Medical Physics, 2015, 42, 2054-2063
[Fƶrstner2003] Fƶrstner, W. & Moonen, B.
A metric for covariance matrices
Geodesy - The Challenge of the 3rd Millennium, Springer, 2003, 299-309
46. REFERENCES (SORTING IN ALPHABETICAL ORDER)
46
[Gatenby2013] Gatenby, R. A.; Grove, O. & Gillies, R. J.
Quantitative Imaging in Cancer Evolution and Ecology
Radiology, 2013, 269, 8-14
[Ganeshan2013] Ganeshan, B.; Goh, V.; Mandeville, H. C.; Ng, Q. S.; Hoskin, P. J. & Miles, K. A.
Non-Small Cell Lung Cancer: Histopathologic Correlates for Texture Parameters at CT
Radiology, 2013, 266, 326-336
[Gevaert2012] Gevaert, O.; Xu, J.; Hoang, C. D.; Leung, A. N.; Xu, Y.; Quon, A.; Rubin, D. L.; Napel, S. & Plevritis, S. K. Non--Small Cell Lung Cancer:
Identifying Prognostic Imaging Biomarkers by Leveraging Public Gene Expression Microarray Data --- Methods and Preliminary Results
Radiology, 2012, 264, 387-396
[Karcher1977 Karcher, H.
Riemannian center of mass and molliļ¬er smoothing
Communications on Pure and Applied Mathematics, 1977, 30, 509-541
[Mattonen2014] Mattonen, S. A.; Palma, D. A.; Haasbeek, C. J. A.; Senan, S. & Ward, A. D.
Early prediction of tumor recurrence based on CT texture changes after stereotactic ablative radiotherapy (SABR) for lung cancer
Medical Physics, 2014, 41, 1-14
[Moakher2005] Moakher, M.
A differential geometric approach to the geometric mean of symmetric positive-deļ¬nite matrices
SIAM Journal on Matrix Analysis and Applications, 2005, 26, 735-747
[Pennec2006] Pennec, X.; Fillard, P. & Ayache, N.
A Riemannian framework for tensor computing
International Journal of Computer Vision, Springer, 2006, 66, 41-66
[Ravanelli2013] Ravanelli, M.; Farina, D.; Morassi, M.; Roca, E.; Cavalleri, G.; Tassi, G. & Maroldi, R.
Texture analysis of advanced non-small cell lung cancer (NSCLC) on contrast-enhanced computed tomography: prediction of the response to
the ļ¬rst-line chemotherapy European
Radiology, 2013, 23, 3450-3455
[Skibbe2012] Skibbe, H.; Reisert, M.; Schmidt, T.; Brox, T.; Ronneberger, O. & Burkhardt, H.
Fast Rotation Invariant 3D Feature Computation Utilizing Efļ¬cient Local Neighborhood Operators
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 34, 1563-1575
[Unser2011] Unser, M.; Chenouard, N. & Van De Ville, D.āØ
Steerable Pyramids and Tight Wavelet Frames in āØ
IEEE Transactions on Image Processing, 2011, 20, 2705-2721
[Ward2014] Ward, J. & Unser, M.
Harmonic Singular Integrals and Steerable Wavelets in
Applied and Computational Harmonic Analysis, 2014, 36, 183-197
L2(Rd
)
L2(Rd
)