- 1. FUNDAMENTALS OF TEXTURE PROCESSING FOR BIOMEDICAL IMAGE ANALYSIS Adrien Depeursinge, PhD MICCAI 2015 Tutorial on Biomedical Texture Analysis (BTA), Munich, Oct 5 2015 nts for textures de- nates. Solid textures s that a number of Euclidean space is Foncubierta-Rodrí- textured surface in 004), or 2.5-dimen- et al. (2008), where jects and can be in- also used in Kajiya aindl (2009), where bjects to create real- s in videos can also analysis problem ‘‘dynamic texture’’ rowley (1999), and the literature pub- dical solid textures is text is on the fea- hniques, since only re analysis. undamentals of 3-D on 2. Section 3 de- atically retrieve pa- n and retrieval. The e literature are re- he various expecta- sulting application- d grouped together s of the various ap- given in Section 7, establish a general Julesz, 1981), it is atical model of tex- is problem (Mallat, s approaches based deﬁned by (x,y,z) 2 [iDx,(i + 1)Dx]; [jDy,(j + 1)Dy]; [kDz,(k + 1)Dz]) (Toriwaki and Yoshida, 2009). This cuboid is called a voxel. The three spherical coordinates (r,h,/) are unevenly sampled to (R,H,U) as shown in Fig. 3. 2.2. Texture primitives The notion of texture primitive has been widely used in 2-D tex- ture analysis and deﬁnes the elementary building block of a given texture class (Haralick, 1979; Jain et al., 1995; Lin et al., 1999). All texture processing approaches aim at modeling a given texture using sets of prototype primitives. The concept of texture primitive is naturally extended in 3-D as the geometry of the voxel sequence used by a given texture analysis method. We consider a primitive C(i,j,k) centered at a point (i,j,k) that lives on a neighborhood of this point. The primitive is constituted by a set of voxels with gray tone values that forms a 3-D structure. Typical C neighborhoods are voxel pairs, linear, planar, spherical or unconstrained. Signal assignment to the primitive can be either binary, categorical or continuous. Two example texture primitives are shown in Fig. 4. Texture primitives refer to local processing of 3-D images and local patterns (see Toriwaki and Yoshida, 2009). Fig. 2. 3-D digitized images and sampling in Cartesian coordinates. Fig. 3. 3-D digitized images and sampling in spherical coordinates. ized images and sampling an coordinates, a generic 3-D continuous image is de- nction of three variables f(x,y,z), where f represents a oint ðx; y; zÞ 2 R3 . A 3-D digital image F(i,j,k) of dimen- Â O is obtained from sampling f at points ði; j; kÞ 2 Z3 red array (see Fig. 2). Increments in (i,j,k), correspond displacements in R3 parametrized by the respective ,Dy,Dz). For every cell of the digitized array, the value ypically obtained by averaging f in the cuboid domain (x,y,z) 2 [iDx,(i + 1)Dx]; [jDy,(j + 1)Dy]; [kDz,(k + waki and Yoshida, 2009). This cuboid is called a voxel. herical coordinates (r,h,/) are unevenly sampled to hown in Fig. 3. primitives n of texture primitive has been widely used in 2-D tex- and deﬁnes the elementary building block of a given (Haralick, 1979; Jain et al., 1995; Lin et al., 1999). All essing approaches aim at modeling a given texture prototype primitives. The concept of texture primitive xtended in 3-D as the geometry of the voxel sequence ven texture analysis method. We consider a primitive ered at a point (i,j,k) that lives on a neighborhood of he primitive is constituted by a set of voxels with gray that forms a 3-D structure. Typical C neighborhoods irs, linear, planar, spherical or unconstrained. Signal to the primitive can be either binary, categorical or Two example texture primitives are shown in Fig. 4. itives refer to local processing of 3-D images and local Toriwaki and Yoshida, 2009). -D digitized images and sampling in Cartesian coordinates. lattices)isnotstraightforwardandraisesseveralchal- atedtotranslation,scalingandrotationinvariancesand esthatarebecomingmorecomplexin3-D. dworkandscopeofthissurvey dingonresearchcommunities,varioustaxonomiesare ferto3-Dtextureinformation.Aclariﬁcationofthetax- proposedinthissectiontoaccuratelydeﬁnethescopeof y.ItispartlybasedonToriwakiandYoshida,2009.Three- altextureandvolumetrictexturearebothgeneraland termsdesigningatexturedeﬁnedinR3 andinclude: umetrictexturesexistingin‘‘ﬁlled’’objects :x;y;z2Vx;y;z&R3 gthataregeneratedbyavolumetric aacquisitiondevice(e.g.,tomography,confocalimaging). Dtexturesexistingonsurfacesof‘‘hollow’’objectsas :x;y;z2Cu;v&R3 g, namictexturesintwo–dimensionaltimesequencesas x;y;t2Sx;y;t&R3 g, exturereferstocategory(1)andaccountsfortexturesde- volumeVx,y,zindexedbythreecoordinates.Solidtextures ntrinsicdimensionof3,whichmeansthatanumberof equaltothedimensionalityoftheEuclideanspaceis representthesignal(Bennett,1965;Foncubierta-Rodrí- .,2013a).Category(2)isdesignedastexturedsurfacein Nayar(1999)andCulaandDana(2004),or2.5-dimen- turesinLuetal.(2006)andAguetetal.(2008),where Careexistingonthesurfaceof3-Dobjectsandcanbein- quelybytwocoordinates(u,v).(2)isalsousedinKajiya 1989),Neyret(1995),andFilipandHaindl(2009),where etriesareaddedontothesurfaceofobjectstocreatereal- eringofvirtualscenes.Motionanalysisinvideoscanalso deredamulti-dimensionaltextureanalysisproblem tocategory(3)andisdesignedby‘‘dynamictexture’’ myandFablet(1998),ChomatandCrowley(1999),and al.(2000). survey,acomprehensivereviewoftheliteraturepub- classiﬁcationandretrievalofbiomedicalsolidtextures ory(1))iscarriedout.Thefocusofthistextisonthefea- actionandnotmachinelearningtechniques,sinceonly xtractionisspeciﬁcto3-Dsolidtextureanalysis. ureofthisarticle urveyisstructuredasfollows:Thefundamentalsof3-D xtureprocessingaredeﬁnedinSection2.Section3de- ereviewmethodologyusedtosystematicallyretrievepa- ngwith3-Dsolidtextureclassiﬁcationandretrieval.The modalitiesandorgansstudiedintheliteraturearere- Sections4and5,respectivelytolistthevariousexpecta- needsof3-Dimageprocessing.Theresultingapplication- chniquesaredescribed,organizedandgroupedtogether 6.Asynthesisofthetrendsandgapsofthevariousap- conclusionsandopportunitiesaregiveninSection7, ely. mentalsofsolidtextureprocessing ghseveralresearchersattemptedtoestablishageneral onthe3-Dgeometricalpropertiesoftheprimitivesused,i.e.,the elementarybuildingblockconsidered.Thesetofprimitivesused andtheirassumedinteractionsdeﬁnethepropertiesofthetexture analysisapproaches,fromstatisticaltostructuralmethods. InSection2.1,wedeﬁnethemathematicalframeworkand notationsconsideredtodescribethecontentof3-Ddigitalimages. Thenotionoftextureprimitivesaswellastheirscalesanddirec- tionsaredeﬁnedinSection2.2. 2.1.3-Ddigitizedimagesandsampling InCartesiancoordinates,ageneric3-Dcontinuousimageisde- ﬁnedbyafunctionofthreevariablesf(x,y,z),wherefrepresentsa scalaratapointðx;y;zÞ2R3 .A3-DdigitalimageF(i,j,k)ofdimen- sionsMÂNÂOisobtainedfromsamplingfatpointsði;j;kÞ2Z3 ofa3-Dorderedarray(seeFig.2).Incrementsin(i,j,k),correspond tophysicaldisplacementsinR3 parametrizedbytherespective spacings(Dx,Dy,Dz).Foreverycellofthedigitizedarray,thevalue ofF(i,j,k)istypicallyobtainedbyaveragingfinthecuboiddomain deﬁnedby(x,y,z)2[iDx,(i+1)Dx];[jDy,(j+1)Dy];[kDz,(k+ 1)Dz])(ToriwakiandYoshida,2009).Thiscuboidiscalledavoxel. Thethreesphericalcoordinates(r,h,/)areunevenlysampledto (R,H,U)asshowninFig.3. 2.2.Textureprimitives Thenotionoftextureprimitivehasbeenwidelyusedin2-Dtex- tureanalysisanddeﬁnestheelementarybuildingblockofagiven textureclass(Haralick,1979;Jainetal.,1995;Linetal.,1999).All textureprocessingapproachesaimatmodelingagiventexture usingsetsofprototypeprimitives.Theconceptoftextureprimitive isnaturallyextendedin3-Dasthegeometryofthevoxelsequence usedbyagiventextureanalysismethod.Weconsideraprimitive C(i,j,k)centeredatapoint(i,j,k)thatlivesonaneighborhoodof thispoint.Theprimitiveisconstitutedbyasetofvoxelswithgray tonevaluesthatformsa3-Dstructure.TypicalCneighborhoods arevoxelpairs,linear,planar,sphericalorunconstrained.Signal assignmenttotheprimitivecanbeeitherbinary,categoricalor continuous.TwoexampletextureprimitivesareshowninFig.4. Textureprimitivesrefertolocalprocessingof3-Dimagesandlocal patterns(seeToriwakiandYoshida,2009). Fig.2.3-DdigitizedimagesandsamplinginCartesiancoordinates. A.Depeursingeetal./MedicalImageAnalysis18(2014)176–196
- 2. OUTLINE • Biomedical texture analysis: background • Deﬁning texture processes • Notations, sampling and texture functions • Texture operators, primitives and invariances • Multiscale analysis • Operator scale and uncertainty principle • Region of interest and operator aggregation • Multidirectional analysis • Isotropic versus directional operators • Importance of the local organization of image directions • Conclusions • References L1 L2 M1 M2 M of1(x) : of2(x) : R X Ø Ø @f (x @x1 R X Ø Ø@f(x) @x2 Ø Ødx image gra Fig. 1. The joint responses of image gradients ≥ | @f (x) @x1 |,| @f (x @x2 able to discriminate between the two textures classes f1(x) when integrated over the image domain X . One circle in t representation corresponds to one realization (i.e., full image) · m
- 3. BACKGROUND – RADIOMICS - HISTOPATHOLOMICS • Personalized medicine aims at enhancing the patient’s quality of life and prognosis • Tailored treatment and medical decisions based on the molecular composition of diseased tissue • Current limitations [Gerlinger2012] • Molecular analysis of tissue composition is invasive (biopsy), slow and costly • Cannot capture molecular heterogeneity 3 Intratumor Heterogeneity Reveale B Regional Distribution of Mutations A Biopsy Sites SOX9 CENPN PSMD7 RIMBP2 GALNT11 ABHD11 UGT2A1 MTOR PPP6R2 ZNF780A WSCD2 CDKN1B PPFIA1 TH SSNA1 CASP2 PLRG1 SETD2 CCBL2 SESN2 MAGEB16 NLRP7 IGLON5 KLK4 WDR62 KIAA0355 CYP4F3 AKAP8 ZNF519 DDX52 ZC3H18 TCF12 NUSAP1 X4 KDM2B MRPL51 C11orf68 ANO5 EIF4G2 MSRB2 RALGDS EXT1 ZC3HC1 PTPRZ1 INTS1 CCR6 DOPEY1 ATXN1 WHSC1 CLCN2 SSR3 KLHL18 SGOL1 VHL C2orf21 ALS2CR12 PLB1 FCAMR IFI16 BCAS2 IL12RB2 Ubiquitous Shared prima 10 cm R7 (G4) R5 (G4) R9 R3 (G4) R1 (G3) R2 (G3) R4 (G1) R6 (G1) Hilum R8 (G4) ution of Mutations ationships of Tumor Regions D Ploidy Profiling PreP PreM R1 R2 R3 R5 R8 R9 R4 M1 M2a M2b C2orf85 WDR7 SUPT6H CDH19 LAMA3 DIXDC1 HPS5 NRAP KIAA1524 SETD2 PLCL1 BCL11A IFNAR1 DAMTS10 C3 KIAA1267 RT4 CD44 ANKRD26 TM7SF4 SLC2A1 DACH2 MMAB ZNF521 HMG20A DNMT3A RLF MAMLD1 MAP3K6 HDAC6 PHF21B FAM129B RPS8 CIB2 RAB27A SLC2A12 DUSP12 ADAMTSL4 NAP1L3 USP51 KDM5C SBF1 TOM1 MYH8 WDR24 ITIH5 AKAP9 FBXO1 LIAS TNIK SETD2 C3orf20 MR1 PIAS3 DIO1 ERCC5 KL ALKBH8 DAPK1 DDX58 SPATA21 ZNF493 NGEF DIRAS3 LATS2 ITGB3 FLNA SATL1 KDM5C KDM5C RBFOX2 NPHS1 SOX9 CENPN PSMD7 RIMBP2 GALNT11 ABHD11 UGT2A1 MTOR PPP6R2 ZNF780A WSCD2 CDKN1B PPFIA1 TH SSNA1 CASP2 PLRG1 SETD2 CCBL2 SESN2 MAGEB16 NLRP7 IGLON5 KLK4 WDR62 KIAA0355 CYP4F3 AKAP8 ZNF519 DDX52 ZC3H18 TCF12 NUSAP1 X4 KDM2B MRPL51 C11orf68 ANO5 EIF4G2 MSRB2 RALGDS EXT1 ZC3HC1 PTPRZ1 Privatebiquitous Shared primary Shared metastasis Lung metastases Chest-wall metastasis Perinephric metastasis M1 10 cm R2 (G3) R4 (G1) R6 (G1) HilumR8 (G4) Primary tumor M2b M2a ution of Mutations PreP PreM R1 R2 R3 R5 R8 R9 R4 M1 M2a M2b C2orf85 WDR7 SUPT6H CDH19 LAMA3 DIXDC1 HPS5 NRAP KIAA1524 SETD2 PLCL1 BCL11A IFNAR1 DAMTS10 C3 KIAA1267 RT4 CD44 ANKRD26 TM7SF4 SLC2A1 DACH2 MMAB ZNF521 HMG20A DNMT3A RLF MAMLD1 MAP3K6 HDAC6 PHF21B FAM129B RPS8 CIB2 RAB27A SLC2A12 DUSP12 ADAMTSL4 NAP1L3 USP51 KDM5C SBF1 TOM1 MYH8 WDR24 ITIH5 AKAP9 FBXO1 LIAS TNIK SETD2 C3orf20 MR1 PIAS3 DIO1 ERCC5 KL ALKBH8 DAPK1 DDX58 SPATA21 ZNF493 NGEF DIRAS3 LATS2 ITGB3 FLNA SATL1 KDM5C KDM5C RBFOX2 NPHS1 SOX9 CENPN PSMD7 RIMBP2 GALNT11 ABHD11 UGT2A1 MTOR PPP6R2 ZNF780A WSCD2 CDKN1B PPFIA1 TH SSNA1 CASP2 PLRG1 SETD2 CCBL2 SESN2 MAGEB16 NLRP7 IGLON5 KLK4 WDR62 KIAA0355 CYP4F3 AKAP8 ZNF519 DDX52 ZC3H18 TCF12 NUSAP1 X4 KDM2B MRPL51 C11orf68 ANO5 EIF4G2 MSRB2 RALGDS EXT1 ZC3HC1 PTPRZ1 Privatebiquitous Shared primary Shared metastasis Lung metastases Chest-wall metastasis Perinephric metastasis M1 10 cm R2 (G3) R4 (G1) R6 (G1) Hilum R8 (G4) Primary tumor M2b M2a
- 4. BACKGROUND – RADIOMICS - HISTOPATHOLOMICS • Huge potential for computerized medical image analysis • Explore and reveal tissue structures related to tissue composition, function, …. • Local quantitative image feature extraction • Supervised and unsupervised machine learning 4 malignant, nonresponder malignant, responder benign pre-malignant undeﬁned quant. feat. #1 quant.feat.#2 Supervised learning, big data
- 5. BACKGROUND – RADIOMICS - HISTOPATHOLOMICS • Huge potential for computerized medical image analysis • Create imaging biomarkers to predict diagnosis, prognosis, treatment response [Aerts2014] 5 Radiomics [Kumar2012] “Histopatholomics” [Gurcan2009] Reuse existing diagnostic images ✓ radiology data1 ✓ digital pathology Capture tissue heterogeneity ✓ 3D neighborhoods (e.g., 512x512x512) ✓ large 2D regions (e.g., 15,000x15,000) Analytic power beyond naked eyes ✓ complex 3D tissue morphology ✓exhaustive characterization of 2D tissue structures Non-invasive ✓ x 1e.g., X-ray, Ultrasound, CT, MRI, PET, OCT, …
- 6. BACKGROUND – RADIOMICS - HISTOPATHOLOMICS • Huge potential for computerized medical image analysis • Explore and reveal tissue structures related to tissue composition, function, …. • Local quantitative image feature extraction • Supervised and unsupervised machine learning 6 malignant, nonresponder malignant, responder benign pre-malignant undeﬁned quant. feat. #1 quant.feat.#2 Supervised learning, big data Speciﬁc to texture!
- 7. OUTLINE • Biomedical texture analysis: background • Deﬁning texture processes • Notations, sampling and texture functions • Texture operators, primitives and invariances • Multiscale analysis • Operator scale and uncertainty principle • Region of interest and operator aggregation • Multidirectional analysis • Isotropic versus directional operators • Importance of the local organization of image directions • Conclusions • References L1 L2 M1 M2 M of1(x) : of2(x) : R X Ø Ø @f (x @x1 R X Ø Ø@f(x) @x2 Ø Ødx image gra Fig. 1. The joint responses of image gradients ≥ | @f (x) @x1 |,| @f (x @x2 able to discriminate between the two textures classes f1(x) when integrated over the image domain X . One circle in t representation corresponds to one realization (i.e., full image) · m
- 8. • Deﬁnition of texture • Everybody agrees that nobody agrees on the deﬁnition of “texture” (context-dependent) • “coarse”, “edgy”, “directional”, “repetitive”, “random”, … • Oxford dictionary: “the feel, appearance, or consistency of a surface or a substance” • [Haidekker2011]: “Texture is a systematic local variation of the image values” • [Petrou2011]: “The most important characteristic of texture is that it is scale dependent. Different types of texture are visible at different scales” COMPUTERIZED TEXTURE ANALYSIS 8 A. Depeursinge et al. / Medical Image Analysis 18 (2014) 176–196 177 [Depeursinge2014a]
- 9. COMPUTERIZED TEXTURE ANALYSIS directions 9 scales • Spatial scales and directions in images are fundamental for visual texture discrimination [Blakemore1969, Romeny2011] • Relating to directional frequencies (shown in Fourier)
- 10. COMPUTERIZED TEXTURE ANALYSIS 10 directionsscales • Spatial scales and directions in images are fundamental for visual texture discrimination [Blakemore1969, Romeny2011] • Most approaches are leveraging these two properties • Explicitly: Gray-level co-occurrence matrices (GLCM), run-length matrices (RLE), directional ﬁlterbanks and wavelets, Fourier, histograms of gradients (HOG), local binary patterns (LBP) • Implicitly: Convolutional neural networks (CNN), scattering transform, topographic independant component analysis (TICA)
- 11. NOTATIONS, SAMPLING AND TEXTURE FUNCTIONS 11 • 2-D continuous texture functions in space and Fourier • Cartesian coordinates: • Polar coordinates: • 2-D digital texture functions • Cartesian coordinates: • Polar coordinates: • Sampling (Cartesian) • Increments in corresponds to physical displacements in as f(k), k = ✓ k1 k2 ◆ 2 Z2 f(R, ⇥), R 2 Z+ , ⇥ 2 [0, 2⇡) (k1, k2) R2 ✓ x1 x2 ◆ = ✓ x1 · k1 x2 · k2 ◆ x1 x2 k2 k1 ) x1 x2 R2 Z2 · f(x) f(k) f(x), x = ✓ x1 x2 ◆ 2 R2 , f(x) F ! ˆf(!) = Z R2 f(x)e jh!,xi dx, ! 2 R2 f(r, ✓), r 2 R+ , ✓ 2 [0, 2⇡), f(r, ✓) F ! ˆf(⇢, ), ⇢ 2 R+ , 2 [0, 2⇡)
- 12. NOTATIONS, SAMPLING AND TEXTURE FUNCTIONS 12 • 3-D continuous texture functions in space and Fourier • Cartesian coordinates: • Polar coordinates: • 3-D digital texture functions • Cartesian coordinates: • Polar coordinates: • Sampling (Cartesian) • Increments in corresponds to physical displacements in as x1 x2 k2 k1 ) x1x2 · f(x) f(k) f(k), k = 0 @ k1 k2 k3 1 A 2 Z3 f(r, ✓, ), r 2 R+ , ✓ 2 [0, 2⇡), 2 [0, 2⇡) f(R, ⇥, ), R 2 Z+ , ⇥ 2 [0, 2⇡), 2 [0, 2⇡) (k1, k2, k3) R3 0 @ x1 x2 x3 1 A = 0 @ x1 · k1 x2 · k2 x3 · k3 1 A x3 x3 k3 R3 Z3 f(x), x = 0 @ x1 x2 x3 1 A 2 R3 , f(x) F ! ˆf(!) = Z R3 f(x)e jh!,xi dx, ! 2 R3
- 13. • We consider a texture function as a realization of a spatial stochastic process of where is the value at the spatial position indexed by • The values of follow one or several probability density functions • Examples NOTATIONS, SAMPLING AND TEXTURE FUNCTIONS 13 f(x) Rd {Xm, m 2 RM1⇥···⇥Md } Xm m moving average Gaussian pointwise Poisson biomedical: lung ﬁbrosis in CT m 2 R128⇥128 m 2 R32⇥32 m 2 R84⇥84 Xm fXm (q)
- 14. NOTATIONS, SAMPLING AND TEXTURE FUNCTIONS 14 • Stationarity of spatial stochastic processes • A spatial process is stationary if the probability density functions are equivalent for all • Example: heteroscedastic moving average Gaussian process {Xm, m 2 RM1⇥···⇥Md } mfXm (q) stationary non-stationary (strict sense) fb,Xm (q) = 1 3 p 2⇡ e (q 0)2 2 32 fa fb fa,Xm (q) = 1 1 p 2⇡ e (q 0)2 2 12
- 15. NOTATIONS, SAMPLING AND TEXTURE FUNCTIONS 15 • Stationarity of textures and human perception / tissue biology • Strict/weak process stationarity and texture class deﬁnition is not equivalent • Image analysis tasks when textures are considered as • Stationary (wide sense): texture classiﬁcation • Non-stationary: texture segmentation Outex “canvas039”: stationary? brain glioblastoma in T1-MRI: stationary? ) [Storath2014]
- 16. OUTLINE • Biomedical texture analysis: background • Deﬁning texture processes • Notations, sampling and texture functions • Texture operators, primitives and invariances • Multiscale analysis • Operator scale and uncertainty principle • Region of interest and operator aggregation • Multidirectional analysis • Isotropic versus directional operators • Importance of the local organization of image directions • Conclusions • References L1 L2 M1 M2 M of1(x) : of2(x) : R X Ø Ø @f (x @x1 R X Ø Ø@f(x) @x2 Ø Ødx image gra Fig. 1. The joint responses of image gradients ≥ | @f (x) @x1 |,| @f (x @x2 able to discriminate between the two textures classes f1(x) when integrated over the image domain X . One circle in t representation corresponds to one realization (i.e., full image) · m
- 17. TEXTURE OPERATORS AND PRIMITIVES 17 • Texture operators • A -dimensional texture analysis approach is characterized by a set of local operators centered at the position • is local in the sense that each element only depends on a subregion of • The subregion is the support of • can be linear (e.g., wavelets) or non-linear (e.g., median, GLCMs, LBPs) • For each position , maps the texture function into a -dimensional space N d f(x) xL1 ⇥ · · · ⇥ Ld L1 L2 M1 M2 · m gn(x, m) : RM1⇥···⇥Md 7! RP , n = 1, . . . , N gn(x, m) p=1,...,P gn m gn m gn P gn(f(x), m) : RM1⇥···⇥Md 7! RP L1 ⇥ · · · ⇥ Ld gn
- 18. TEXTURE OPERATORS AND PRIMITIVES 18 • From texture operators to texture measurements (i.e., features) • The operator is typically applied to all positions of the image by “sliding” its window over the image • Regional texture measurements can be obtained from the aggregation of over a region of interest • For instance, integration can be used to aggregate over • e.g., average: L1 L2 M1 M2 L1 ⇥ · · · ⇥ Ld · gn(x, m) m µ 2 RP gn(f(x), m) M M m gn(f(x), m) M µ = 0 B @ µ1 ... µP 1 C A = 1 |M| Z M gn(f(x), m) p=1,...,P dm
- 19. TEXTURE OPERATORS AND PRIMITIVES • Texture primitives • A “texture primitive” (also called “texel”) is a fundamental elementary unit (i.e., a building block) of a texture class [Haralick1979, Petrou2006] • Intuitively, given a collection of texture functions , an appropriate set of texture operators must be able to: (i) Detect and quantify the presence of all distinct primitives in (ii) Characterize the spatial relationships between the primitives (e.g., geometric transformations, density) when aggregated texture primitives , 19 texture primitive , fj f1 f2 primitive , primitive , fj=1,...,J gn
- 20. TEXTURE OPERATORS AND PRIMITIVES • General-purpose texture operators • In general, the texture primitives are neither well-deﬁned, nor known in advance (e.g., biomedical tissue) • General-purpose operator sets are useful to estimate the primitives • How to build such operator sets? 20 patterns in the whole HRCT volume. Methods ataset used is part of an internal multimedia database of ILD cases con- HRCT images with annotated ROIs created in the Talisman project1 . OIs from healthy and ﬁve pathologic lung tissue patterns are selected for g and testing the classiﬁers selecting classes with suﬃciently high repre- on (see Table 1). e wavelet frame decompositions with dyadic and quincunx subsampling plemented in Java [11, 16] as well as optimization of SVMs. The basic mentation of the SVMs is taken from the open source Java library Weka2 . 1. Visual aspect and distribution of the ROIs per class of lung tissue pattern. al ect ss healthy emphysema ground glass ﬁbrosis micronodules macronodules ROIs 113 93 148 312 155 22 tients 11 6 14 28 5 5 esults sotropic Polyharmonic B–Spline Wavelets ntioned in Section 1.1, isotropic analysis is preferable for lung texture terization. The Laplacian operator plays an important role in image pro- and is clearly isotropic. Indeed, ∆ = ∂2 ∂x2 1 + ∂2 ∂x2 2 , is rotationally invariant. olyharmonic B–spline wavelets implement a multiscale smoothed version Laplacian [16]. This wavelet, at the ﬁrst decomposition level, can be char- ed as 2-D lung tissue in CT images [Depeursinge2012a] 3-D normal and osteoporotic bone in CT [Dumas2009]µ
- 21. • General-purpose texture operators • The exhaustive analysis of spatial scales and directions is computationally expensive when the support of the operators is large: choices are required • Directions (e.g., GLCMs, RLE, HOG) • Scales the classiﬁcation accuracy of RLE versus GLCMs for categorizing lung tissue patterns associated with diffuse lung disease in HRCT. Using identical choices of the directions for RLE and GLCMs, they found found no statistical differences between the classiﬁcation performance. Gao et al. (2010) and Qian et al. (2011) compared the performance of three-dimensional GLCM, LBP, Gabor ﬁlters and WT in retrieving similar MR images of the brain. They ob- served a small increase in retrieval performance for LBP and GLCM when compared to Gabor ﬁlters and WT. However, the database used is rather small and the results might not be statistically signiﬁcant. Several papers compared the performance of texture analysis algorithms in their 2-D versus 3-D forms. As expected, 2-D texture analysis is most often less discriminative than 3-D, which was ob- served for various applications and techniques, such as: GLCMs, RLE and fractal dimension for the classiﬁcation of lung tissue types in HRCT in Xu et al. (2005, 2006b). GLCMs for the classiﬁcation of brain tumors in MRI in Mah- moud-Ghoneim et al. (2003) and Allin Christe et al. (2012). GLCMs for the classiﬁcation of breast in contrast–enhanced MRI, where statistical signiﬁcance was assessed in Chen et al. (2007). GMRF for the segmentation of gray matter in MRI in Ranguelova and Quinn (1999). LBP for synthetic texture classiﬁcation in Paulhac et al. (2008). This demonstrates that 2-D slice-based discrimination of 3-D native texture does not allow fully exploiting the information available in 3-D datasets. An exception was observed with 2-D ver- sus 3-D WTs in Jafari-Khouzani et al. (2004), where the 2-D ap- proach showed a small increase in classiﬁcation performance of abnormal regions responsible for temporal lobe epilepsy. A separa- ble 3-D WT was used, which did not allow to adequately exploit the 3-D texture information available and may explain the ob- served results. 7. Discussion In the preceding sections, we have reviewed the current state- of-the-art in 3-D biomedical texture analysis. The papers were cat- egorized in terms of imaging modality used, organ studied and im- age processing techniques. The increasing number of papers over the past 32 years clearly shows a growing interest in computerized characterization of three-dimensional texture information (see Fig. 5). This is a consequence of increasingly available 3-D data acquisition devices that are reaching high spatial resolutions allowing to capture tissue properties in its natural space. The analysis of the medical applications in 100 papers in Sec- tion 5 shows the diversity of 3-D biomedical textures. The various geometrical properties of the textures are summarized in Table 2, which deﬁnes the multiple challenges of 3-D texture analysis. The need for methods able to characterize structural scales and Fig. 15. Amounts of pixels/voxels covered by the main 13 directions. The number N of discarded pixels/voxels (gray regions in (a) and (b)) according to the neighborhood size R is shown in (c). A. Depeursinge et al. / Medical Image Analysis 18 (2014) 176–196 191 GMRF for the segmentation of gray matter in MRI in Ranguelova and Quinn (1999). LBP for synthetic texture classiﬁcation in Paulhac et al. (2008). geometrical properties of the textures are summarized which deﬁnes the multiple challenges of 3-D textu The need for methods able to characterize structural Fig. 15. Amounts of pixels/voxels covered by the main 13 directions. The number N of discarded pixels/voxels (gray regions in (a) and (b)) according to the neig R is shown in (c). TEXTURE OPERATORS AND PRIMITIVES 21 2-D 3-D (one quadrant) such that numberofdiscarded pixels/voxelsof order −1/2 (an isotropic smoothing operator) of f: Rf = −∇∆−1/2 f. Let’s indeed recall the Fourier-domain deﬁnition of these operators: ∇ F ←→ jω and ∆−1/2 F ←→ ||ω||−1 . Unlike the usual gradient ∇, the Riesz transform is self-reversible R⋆ Rf(ω) = (jω)∗ (jω) ||ω||2 ˆf(ω) = ˆf(ω). This allows us to deﬁne a self-invertible wavelet frame of L2(R3 ) (tight frame). We however see that there exists a singularity for the frequency (0, 0, 0). This issue will be ﬁxed later, thanks to the van- ishing moments of the primary wavelet transform. 2-D GLCMs with various spatial distances [Haralick1979] 2-D isotropic dyadic wavelets in Fourier [Chenouard2011] d = 1 ( k1 = 1, k2 = 0) d = 2 ( k1 = 2, k2 = 0) d = 2 p 2 ( k1 = 2, k2 = 2) r L1 ⇥ · · · ⇥ Ld L1 = L2 = L3 = 2r + 1 d
- 22. g(f(x), m) = g(f(R✓0 x x0), m), 8x0 2 R2 , ✓0 2 [0, 2⇡) • Invariances of the texture operators to geometric transformations can be desirable • E.g., scaling, rotations and translations • Invariances of texture operators can be enforced • Example with 2-D Euclidean transforms (i.e., rotation and translation) with the rotation matrix INVARIANCE OF TEXTURE OPERATORS 22 R✓0 = ✓ cos ✓0 sin ✓0 sin ✓0 cos ✓0 ◆
- 23. INVARIANCE OF TEXTURE OPERATORS 23 • Computer vision versus biomedical imaging Computer vision Biomedical image analysis translation translation-invariant translation-invariant rotation rotation-invariant rotation-invariant scale scale-invariant multi-scale 160 Fig. 10. (a) A digitized histopathology image of Grade 4 CaP and different graph-based r Diagram, and Minimum Spanning tree. Fig. 11. Digitized histological image at successively higher scales (magniﬁca- tions) yields incrementally more discriminatory information in order to detect suspicious regions. or resolution. For instance at low or coarse scales color or tex- ture cues are commonly used and at medium scales architec- tural arrangement of individual histological structures (glands and nuclei) start to become resolvable. It is only at higher res- olutions that morphology of speciﬁc histological structures can be discerned. In [93], [94], a multiresolution approach has been used for the classiﬁcation of high-resolution whole-slide histopathology im- ages. The proposed multiresolution approach mimics the eval- uation of a pathologist such that image analysis starts from the lowest resolution, which corresponds to the lower magniﬁcation levels in a microscope and uses the higher resolution represen- Fig. 12 image 1, (c) r as susp show three scale (scal the n dition highe tumo At is com COMPUTERIZED TEXTURE ANALYSIS 7 • Invariances: computer vision versus biomedical imaging Computer vision Biomedical image analysis scale scale-invariant multi-scale rotation rotation-invariant rotation-invariant [4] Histopathological image analysis: a review, Gurcan et al., IEEE Reviews in Biomed Eng, 2:147-71, 2009 160 IE Fig. 10. (a) A digitized histopathology image of Grade 4 CaP and different graph-based representation Diagram, and Minimum Spanning tree. Fig. 11. Digitized histological image at successively higher scales (magniﬁca- tions) yields incrementally more discriminatory information in order to detect suspicious regions. or resolution. For instance at low or coarse scales color or tex- ture cues are commonly used and at medium scales architec- tural arrangement of individual histological structures (glands and nuclei) start to become resolvable. It is only at higher res- olutions that morphology of speciﬁc histological structures can be discerned. In [93], [94], a multiresolution approach has been used for the classiﬁcation of high-resolution whole-slide histopathology im- ages. The proposed multiresolution approach mimics the eval- uation of a pathologist such that image analysis starts from the lowest resolution, which corresponds to the lower magniﬁcation levels in a microscope and uses the higher resolution represen- tations for the regions requiring more detailed information for a classiﬁcation decision. To achieve this, images were decom- posed into multiresolution representations using the Gaussian pyramid approach [95]. This is followed by color space con- version and feature construction followed by feature extraction and feature selection at each resolution level. Once the classiﬁer is conﬁdent enough at a particular resolution level, the system assigns a classiﬁcation label (e.g., stroma-rich, stroma-poor or Fig. 12. Results fr image with the tum 1, (c) results at scale as suspicious at low shows the origin three columns s scales. Pixels cl (scale) are disc the number of p ditionally, the p higher scales al tumor and nont At lower reso is commonly us pattern of gland tized histologic scenes can be to every pixel i dient, and Gab the scale, orien gion of interest features within [Gurcan2009][Lazebnik2005]
- 24. OUTLINE • Biomedical texture analysis: background • Deﬁning texture processes • Notations, sampling and texture functions • Texture operators, primitives and invariances • Multiscale analysis • Operator scale and uncertainty principle • Region of interest and operator aggregation • Multidirectional analysis • Isotropic versus directional operators • Importance of the local organization of image directions • Conclusions • References L1 L2 M1 M2 M of1(x) : of2(x) : R X Ø Ø @f (x @x1 R X Ø Ø@f(x) @x2 Ø Ødx image gra Fig. 1. The joint responses of image gradients ≥ | @f (x) @x1 |,| @f (x @x2 able to discriminate between the two textures classes f1(x) when integrated over the image domain X . One circle in t representation corresponds to one realization (i.e., full image) · m
- 25. k2 k1 x1x2f(k) x3 k3 MULTISCALE TEXTURE OPERATORS 25 • Inter-patient and inter-dimension scale normalization • Most medical imaging protocols yield images with various sampling steps • Inter-patient scale normalization is required to ensure the correspondance of spatial frequencies • Inter-dimension scale normalization is required to ensure isotropic scale/directions deﬁnition ( x1, x2, x3) x1 = x2 = 0.4mm x1 = x2 = 1.6mm spatial Fourier Fourierspatial 0 0 0 0 ⇡ ⇡⇡ ⇡ ⇡ ⇡ x1x2 x3 ) x0 1x0 2 x0 3 d = 1 ( k1 = 1, k2 = 0) GLCMs 0.4mm 1.6mm
- 26. MULTISCALE TEXTURE OPERATORS 26 • Which scales for texture measurements? • Two aspects: A. How to deﬁne the size(s) of the operator(s) ? B. How to deﬁne the size of the region of interest ? TEXTURE OPERATORS AND PRIMITIVES • From texture operators to texture measurements • The operator is typically applied to all positions by “sliding” its window over the image • Regional texture measurements can be obtained from aggregation of over a region • For instance, integration can be used to aggregate • e.g., average: L1 L2 M1 M2 L1 ⇥ · · · ⇥ Ld · gn(x, m) m µ 2 RP gn(f(x), m) M M m gn(f(x), m µ = 0 B @ µ1 ... µP 1 C A = 1 |M| Z M gn(f(x), m) p=1,...,P dm M original image d increasingly small operator size L1 L2 · · · LN 0 B B B B B B B B B B B B B B B B B B B B B B B B @ µ1 1 ... µ1 P µ2 1 ... µ2 P ... µN 1 ... µN P 1 C C C C C C C C C C C C C C C C C C C C C C C C A = µM: concatenated measurements from multiscale operators aggregated over Ln 1 ⇥ · · · ⇥ Ln d gn=1,...,N (f(x), m) M f(x) M [Depeursinge2012b]
- 27. MULTISCALE TEXTURE OPERATORS 27 • Which scales for texture measurements? • Two aspects: A. How to deﬁne the size(s) of the operator(s) ? B. How to deﬁne the size of the region of interest ? TEXTURE OPERATORS AND PRIMITIVES • From texture operators to texture measurements • The operator is typically applied to all positions by “sliding” its window over the image • Regional texture measurements can be obtained from aggregation of over a region • For instance, integration can be used to aggregate • e.g., average: L1 L2 M1 M2 L1 ⇥ · · · ⇥ Ld · gn(x, m) m µ 2 RP gn(f(x), m) M M m gn(f(x), m µ = 0 B @ µ1 ... µP 1 C A = 1 |M| Z M gn(f(x), m) p=1,...,P dm M original image d increasingly small operator size L1 L2 · · · LN 0 B B B B B B B B B B B B B B B B B B B B B B B B @ µ1 1 ... µ1 P µ2 1 ... µ2 P ... µN 1 ... µN P 1 C C C C C C C C C C C C C C C C C C C C C C C C A = µM: concatenated measurements from multiscale operators aggregated over Ln 1 ⇥ · · · ⇥ Ln d gn=1,...,N (f(x), m) M f(x) M [Depeursinge2012b]
- 28. MULTISCALE TEXTURE OPERATORS 28 • Size and spectral coverage of operators • Uncertainty principle: operators cannot be well located both in space and Fourier [Petrou2006] • In 2D, the trade-off between the spatial support (i.e., ) and frequency support (i.e., ) of the operators is given by L1 ⇥ L2 ⌦1 ⇥ ⌦2 F ! ˆf(!)f(x) L2 1 ⌦2 1 L2 2 ⌦2 2 1 16
- 29. MULTISCALE TEXTURE OPERATORS 29 • Size and spectral coverage of operators • Becomes a problem in the case of non-stationary texture: ,accurate spatial localization poor spectrum characterization Gaussian window: = 3.2mm 0 ⇡|!1| F !
- 30. MULTISCALE TEXTURE OPERATORS 30 • Size and spectral coverage of operators • Becomes a problem in the case of non-stationary texture: ,accurate spatial localization poor spectrum characterization Gaussian window: = 3.2mm Gaussian window: = 38.4mm 0 ⇡|!1| F !
- 31. MULTISCALE TEXTURE OPERATORS 31 • Size and spectral coverage of operators • Becomes a problem in the case of non-stationary texture: ,accurate spatial localization poor spectrum characterization Gaussian window: = 3.2mm Gaussian window: = 38.4mm 0 ⇡|!1| F ! The spatial support should have the minimum size that allows rich enough texture-speciﬁc spectral characterization
- 32. MULTISCALE TEXTURE OPERATORS 32 • Other consequence: • Large inﬂuence of proximal objects when the support of operators is larger than the region of interest • Example with band-limited operators (2D isotropic wavelets) and lung boundary [Ward2015, Depeursinge2015a] • Tuning the shape/bandwidth was found to have a strong inﬂuence on lung tissue classiﬁcation accuracy M 0 0.05 0.1 0.15 0.2 0.25 0.3 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 0 5 10 15 20 25 30 35 0 0.05 0.1 0.15 0.2 0 0.5 1 1.5 2 0 ⇡|!1|0 0 ⇡|!1||!1| ⇡x1 x1 x1 ˆf(!)f(x) f(x) f(x)ˆf(!) ˆf(!) COMPUTERIZED TEXTURE ANALYSIS 31 • Other consequence: • Large inﬂuence of proximal objects when the support of operators is larger than the region of interest: • Example with band-limited operators (2D isotropic wavelets) and lung boundary [DPC2015,WPU2015] • Tuning the shape/bandwidth was found to have a strong inﬂuence on lung tissue classiﬁcation accuracy L1 L2 M1 M2 · M m M 0 0.05 0.1 0.15 0.2 0.25 0.3 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 0 5 10 15 20 25 30 35 0 0.05 0.1 0.15 0.2 0 0.5 1 1.5 2 0 ⇡|!1|0 0 ⇡|!1||!1| ⇡x1 x1 x1 ˆf(!)f(x) f(x) f(x)ˆf(!) ˆf(!) better spatial localization worse spectral localization
- 33. TEXTURE OPERATORS AND PRIMITIVES • From texture operators to texture measurements • The operator is typically applied to all positions by “sliding” its window over the image • Regional texture measurements can be obtained from aggregation of over a region of interest • For instance, integration can be used to aggregate • e.g., average: L1 L2 M1 M2 L1 ⇥ · · · ⇥ Ld · gn(x, m) m µ 2 RP gn(f(x), m) M m gn(f(x), m µ = 0 B @ µ1 ... µP 1 C A = 1 |M| Z M gn(f(x), m) p=1,...,P dm MULTISCALE TEXTURE OPERATORS 33 • Which scales for texture measurements? • Two aspects: A. How to deﬁne the size(s) of the operator(s) ? B. How to deﬁne the size of the region of interest ?M original image d increasingly small operator size L1 L2 · · · LN 0 B B B B B B B B B B B B B B B B B B B B B B B B @ µ1 1 ... µ1 P µ2 1 ... µ2 P ... µN 1 ... µN P 1 C C C C C C C C C C C C C C C C C C C C C C C C A = µM: concatenated measurements from multiscale operators aggregated over Ln 1 ⇥ · · · ⇥ Ln d gn=1,...,N (f(x), m) M f(x) M M [Depeursinge2012b]
- 34. MULTISCALE TEXTURE OPERATORS • How large must be the region of interest ? • No more than enough to evaluate texture stationarity in terms of human perception / tissue biology • Example with operator: undecimated isotropic Simoncelli’s dyadic wavelets [Portilla2000] applied to all image positions • Operators’ responses are averaged over M • The operator is typically applied to all position by “sliding” its window over the image • Regional texture measurements can be obtained aggregation of over a region of interest • For instance, integration can be used to aggregate • e.g., average: L1 L2 M1 M2 L1 ⇥ · · · ⇥ Ld · gn(x, m) µ 2 RP gn(f(x), m) M m gn(f( µ = 0 B @ µ1 ... µP 1 C A = 1 |M| Z M gn(f(x), m) p=1,...,P dm M f(x) g1(f(x), m) g2(f(x), m) original image with regions I 1 |M| Z M |g1(f(x), m)|dm M feature space 1 |M| Z M |g2(f(x),m)|dm f(x) Ma, Mb, Mc The averaged responses over the entire image does not correspond to anything visually! ˆg1(⇢) = ⇢ cos ⇡ 2 log2 2⇢ ⇡ , ⇡ 4 ⇢ ⇡ 0, otherwise. ˆg2(⇢) = ⇢ cos ⇡ 2 log2 4⇢ ⇡ , ⇡ 8 ⇢ ⇡ 2 0, otherwise. g1,2 f(⇢, ) = ˆg1,2(⇢, ) · ˆf(⇢, ) m 2 RM1⇥M2
- 35. MULTISCALE TEXTURE OPERATORS • How large must be the region of interest ? • No more than enough to evaluate texture stationarity in terms of human perception / tissue biology • Example with operator: undecimated isotropic Simoncelli’s dyadic wavelets [Portilla2000] applied to all image positions • Operators’ responses are averaged over M • The operator is typically applied to all position by “sliding” its window over the image • Regional texture measurements can be obtained aggregation of over a region of interest • For instance, integration can be used to aggregate • e.g., average: L1 L2 M1 M2 L1 ⇥ · · · ⇥ Ld · gn(x, m) µ 2 RP gn(f(x), m) M m gn(f( µ = 0 B @ µ1 ... µP 1 C A = 1 |M| Z M gn(f(x), m) p=1,...,P dm M f(x) g1(f(x), m) m 2 RM1⇥M2 g2(f(x), m) original image with regions I 1 |M| Z M |g1(f(x), m)|dm M feature space 1 |M| Z M |g2(f(x),m)|dm f(x) Ma, Mb, Mc The averaged responses over the entire image does not correspond to anything visually! ˆg1(⇢) = ⇢ cos ⇡ 2 log2 2⇢ ⇡ , ⇡ 4 ⇢ ⇡ 0, otherwise. ˆg2(⇢) = ⇢ cos ⇡ 2 log2 4⇢ ⇡ , ⇡ 8 ⇢ ⇡ 2 0, otherwise. g1,2 f(⇢, ) = ˆg1,2(⇢, ) · ˆf(⇢, ) Nor biologically!
- 36. MULTISCALE TEXTURE OPERATORS • How large must be the region of interest ? • No more than enough to evaluate texture stationarity in terms of human perception / tissue biology • Example with operator: undecimated isotropic Simoncelli’s dyadic wavelets [Portilla2000] applied to all image positions • Operators’ responses are averaged over M • The operator is typically applied to all position by “sliding” its window over the image • Regional texture measurements can be obtained aggregation of over a region of interest • For instance, integration can be used to aggregate • e.g., average: L1 L2 M1 M2 L1 ⇥ · · · ⇥ Ld · gn(x, m) µ 2 RP gn(f(x), m) M m gn(f( µ = 0 B @ µ1 ... µP 1 C A = 1 |M| Z M gn(f(x), m) p=1,...,P dm M f(x) g1(f(x), m) m 2 RM1⇥M2 g2(f(x), m) original image with regions I 1 |M| Z M |g1(f(x), m)|dm M feature space 1 |M| Z M |g2(f(x),m)|dm f(x) Ma, Mb, Mc The averaged responses over the entire image does not correspond to anything visually! ˆg1(⇢) = ⇢ cos ⇡ 2 log2 2⇢ ⇡ , ⇡ 4 ⇢ ⇡ 0, otherwise. ˆg2(⇢) = ⇢ cos ⇡ 2 log2 4⇢ ⇡ , ⇡ 8 ⇢ ⇡ 2 0, otherwise. g1,2 f(⇢, ) = ˆg1,2(⇢, ) · ˆf(⇢, ) Nor biologically! Deﬁne regions that are homogeneous in terms of operators’ responses (e.g., pixelwise clustering, graph cuts [Malik2001], Pott’s model [Storath2014])
- 37. MULTISCALE TEXTURE OPERATORS 37 • How large must be the region of interest ? • Example: supervised texture segmentation with undecimated isotropic Simoncelli’s wavelets and linear support vector machines (SVM) M feature space (training) predicted labels 1 |M| Z M |g1(f(x), m)|dm 1 |M| Z M |g2(f(x),m)|dm decision values -10 -8 -6 -4 -2 0 2 4 6 8 train class 1 (128x128) test image (256x256)train class 2 (128x128) segmentation error=0.05127
- 38. MULTISCALE TEXTURE OPERATORS 38 • How large must be the region of interest ? • Example: supervised texture segmentation with undecimated isotropic Simoncelli’s wavelets and linear support vector machines (SVM) M predicted labels decision values -10 -8 -6 -4 -2 0 2 4 6 8 train class 1 (128x128) test image (256x256)train class 2 (128x128) segmentation error=0.05127 patch radius 0 20 40 60 80 100 120 segmentationerror 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2 0.22 0.24 : circular patch with M r = 30
- 39. : circular patch with MULTISCALE TEXTURE OPERATORS 39 • How large must be the region of interest ? • Example: supervised texture segmentation with undecimated isotropic Simoncelli’s wavelets and linear support vector machines (SVM) M patch radius 0 20 40 60 80 100 120 segmentationerror 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2 0.22 0.24 train class 1 (128x128) train class 2 (128x128) test image (256x256) r = 8 decision values -2 -1 0 1 2 3 predicted labels segmentation error=0.23853 M
- 40. MULTISCALE TEXTURE OPERATORS 40 • How large must be the region of interest ? • Example: supervised texture segmentation with undecimated isotropic Simoncelli’s wavelets and linear support vector machines (SVM) M patch radius 0 20 40 60 80 100 120 segmentationerror 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2 0.22 0.24 : circular patch with r = 128 test image (256x256)train class 1 (128x128) train class 2 (128x128) predicted labels decision values -1.5 -1 -0.5 0 0.5 1 segmentation error=0.22839 M
- 41. MULTISCALE TEXTURE OPERATORS 41 • How large must be the region of interest ? • Tissue properties are not homogeneous (i.e., non-stationary) over the entire organ • Importance of building tissue atlases and digital phenotypes [Depeursinge2015b] M FIGURE 3. The 36 subregions of the lungs localized the prototype regional distributions of the texture properties. Figure 3 can be viewed online in color Depeursinge et al Investigative Radiology • Volume 00, Number 00, Month 2015 M1 M2 = µM1,...,S
- 42. OUTLINE • Biomedical texture analysis: background • Deﬁning texture processes • Notations, sampling and texture functions • Texture operators, primitives and invariances • Multiscale analysis • Operator scale and uncertainty principle • Region of interest and operator aggregation • Multidirectional analysis • Isotropic versus directional operators • Importance of the local organization of image directions • Conclusions • References L1 L2 M1 M2 M of1(x) : of2(x) : R X Ø Ø @f (x @x1 R X Ø Ø@f(x) @x2 Ø Ødx image gra Fig. 1. The joint responses of image gradients ≥ | @f (x) @x1 |,| @f (x @x2 able to discriminate between the two textures classes f1(x) when integrated over the image domain X . One circle in t representation corresponds to one realization (i.e., full image) · m
- 43. 43 • Which directions for texture measurements? • Isotropic operators: insensitive to image directions • Linear: • Non-linear: e.g., median ﬁlter • Directional operators • Linear: Other: e.g., Fourier, circular and spherical harmonics [Unser2013, Ward2014], MR8 [Varma2005], HOG [Dalal2005], Simoncelli’s pyramid [Simoncelli1995], curvelets [Candes2000] • Non-linear: Other: e.g., RLE [Galloway1975] 2D Gaussian ﬁlter 2D isotropic wavelets [Portilla2000] MULTIDIRECTIONAL TEXTURE OPERATORS 2D Gaussian derivatives (e.g., Riesz wavelets [Unser2011]) @ @x1 @ @x2 @2 @x2 2 @2 @x2 1 @2 @x1@x2 d = 2 ( k1 = 2, k2 = 0) 2D GLCMs d = 2 p 2 ( k1 = 2, k2 = 2) 2D LBPs [Ojala2002] spaced pixels on a circle of radius R …R 0† that form a circularly symmetric neighbor set. T % t…s…g0 À gc†; s…g1 À gc†; . . . ; s…gPÀ1 À gc††; …5† where OJALA ET AL.: MULTIRESOLUTION GRAY-SCALE AND ROTATION INVARIANT TEXTURE CLASSIFICATION WITH LOCAL BINARY PATTERNS 973 Fig. 1. Circularly symmetric neighbor sets for different (P; R). spaced pixels on a circle of radius R …R 0† that form a circularly symmetric neighbor set. T % t…s…g0 À gc†; s…g1 À gc†; . . . ; s…gPÀ1 À gc††; …5† where OJALA ET AL.: MULTIRESOLUTION GRAY-SCALE AND ROTATION INVARIANT TEXTURE CLASSIFICATION WITH LOCAL BINARY PATTERNS 973 Fig. 1. Circularly symmetric neighbor sets for different (P; R). [?] Texture in Biomedical Images, Petrou M., L1 L2 M1 M2 Mass- in a able om- ures. tion rom use ping iven ciﬁc nt of the tally for of1(x) : of2(x) : R X Ø Ø @f (x) @x1 R X Ø Ø@f(x) @x2 Ø Ødx image grad Fig. 1. The joint responses of image gradients ≥ | @f (x) @x1 |,| @f (x) @x2 able to discriminate between the two textures classes f1(x) when integrated over the image domain X . One circle in th representation corresponds to one realization (i.e., full image) · m gn(r, ✓, m) 7! gn(r, m)x
- 44. MULTIDIRECTIONAL TEXTURE OPERATORS 44 • Which directions for texture measurements? • Is directional information important for texture discrimination? F ! ˆf(!)f(x) F ! ˆf(!)f(x)
- 45. MULTIDIRECTIONAL TEXTURE OPERATORS 45 • Which directions for texture measurements? • Importance of the local organization of image directions (LOID) • i.e., how directional structures intersect
- 46. MULTIDIRECTIONAL TEXTURE OPERATORS 46 • Which directions for texture measurements? • Isotropic and unidirectional operators can hardly characterize the LOIDs, especially when aggregated over a region [Sifre2014, Depeursinge2014b] • Example of feature representation when integrated over entire image M isotropic Simoncelli wavelets scale 1 scale2 o o GLCM contrast GLCMcontrast d = 1 ( k1 = 1, k2 = 0) d=1(k1=0,k2=1) GLCMsgradients along and 1 |M| Z M ✓ @f(x) @x1 ◆2 dx 1 |M| Z M ✓ @f(x) @x2 ◆2 dx x1 x2 M ⌘ L1 L2 M1 M2 · M m L1 L2 M1 M2 · M m
- 47. MULTIDIRECTIONAL TEXTURE OPERATORS 47 • Which directions for texture measurements? • Isotropic and unidirectional operators can hardly characterize the LOIDs, especially when aggregated over a region [Sifre2014, Depeursinge2014b] • Example of feature representation when integrated over entire image M isotropic Simoncelli wavelets scale 1 scale2 o o GLCM contrast GLCMcontrast d = 1 ( k1 = 1, k2 = 0) d=1(k1=0,k2=1) GLCMsgradients along and 1 |M| Z M ✓ @f(x) @x1 ◆2 dx 1 |M| Z M ✓ @f(x) @x2 ◆2 dx x1 x2 M ⌘ L1 L2 M1 M2 · M m L1 L2 M1 M2 · M m Very poor discrimination! Solutions proposed in a few slides…
- 48. MULTIDIRECTIONAL TEXTURE OPERATORS 48 • Locally rotation-invariant operators over • Isotropic operators: • By deﬁnition • Directional: • Averaging operators’ responses over all directions: 2D GLCMs ¯µ1 (e.g., GLCM contrast) No characterization of image directions! µ ⇡/2 1µ ⇡/4 1 µ 3⇡/4 1 µ0 1 L1 ⇥ L2 g(f(x), m) = g(f(R✓0 x), m), 8✓0 2 [0, 2⇡), 8x 2 RL1⇥L2
- 49. REFERENCES [?] Texture in Biomedical Images, Petrou M., L1 L2 M1 M2 M MULTIDIRECTIONAL TEXTURE OPERATORS 49 • Locally rotation-invariant operators over • Locally “aligning” directional operators • MR8 ﬁlterbank [Varma2005] • Rotation-invariant LBP [Ojala2002, Ahonen2009] • Steerable Riesz wavelets [Unser2013, Depeursinge2014b] L1 ⇥ L2 g(f(x), m) = g(f(R✓0 x), m), 8✓0 2 [0, 2⇡), 8x 2 RL1⇥L2
- 50. MULTIDIRECTIONAL TEXTURE OPERATORS 50 • Locally rotation-invariant operators over • Locally “aligning” directional operators • Maximum response 8 (MR8) ﬁlterbank [Varma2005] • Filter responses are obtained for each pixel from the convolution of the ﬁlter and the image • For each position , only the maximum responses among gradient and Laplacian ﬁlters are kept isotropic mutliscale oriented gradients multiscale oriented Laplacians m L1 ⇥ L2 g(f(x), m) = g(f(R✓0 x), m), 8✓0 2 [0, 2⇡), 8x 2 RL1⇥L2
- 51. MULTIDIRECTIONAL TEXTURE OPERATORS 51 • Locally rotation-invariant operators over • Locally “aligning” directional operators • Maximum response 8 (MR8) ﬁlterbank [Varma2005] • Filter responses are obtained for each pixel from the convolution of the ﬁlter and the image • For each position , only the maximum responses among gradient and Laplacian ﬁlters are kept isotropic mutliscale oriented gradients multiscale oriented Laplacians m L1 ⇥ L2 Yields approximate local rotation invariance Poor characterization of the LOIDs g(f(x), m) = g(f(R✓0 x), m), 8✓0 2 [0, 2⇡), 8x 2 RL1⇥L2
- 52. MULTIDIRECTIONAL TEXTURE OPERATORS 52 • Locally rotation-invariant operators over • Locally “aligning” directional operators • Local binary patterns (LBP) [Ojala2002] • Rotation-invariant LBP [Ahonen2009] L1 ⇥ L2 1) deﬁne a circular neighborhood 2) Binarize and build a number that encode the LOIDs ) 3) Aggregate over the entire image and count code occurrences 0 170 ⌘ 4) make codes invariant to circular shifts U8(1, 0) = 10101010 = 170 U8(1, 0) = 10101010 U8(1, 1) = 01010101 m rotation r discrete Fourier transform The new measures are independent of the rotation ) r H8(1, u) = 7X r=0 hI(U8(1, r))e j2⇡ur/8 µp = |H8(1, u)| µ0,p = hI (U8(1, 0)) g(f(x), m) = g(f(R✓0 x), m), 8✓0 2 [0, 2⇡), 8x 2 RL1⇥L2
- 53. MULTIDIRECTIONAL TEXTURE OPERATORS 53 • Locally rotation-invariant operators over • Locally “aligning” directional operators • Local binary patterns (LBP) [Ojala2002] • Rotation-invariant LBP [Ahonen2009] L1 ⇥ L2 1) deﬁne a circular neighborhood 2) Binarize and build a number that encode the LOIDs ) 3) Aggregate over the entire image and count code occurrences 0 170 ⌘ 4) make codes invariant to circular shifts U8(1, 0) = 10101010 = 170 U8(1, 0) = 10101010 U8(1, 1) = 01010101 m rotation r discrete Fourier transform The new measures are independent of the rotation ) r H8(1, u) = 7X r=0 hI(U8(1, r))e j2⇡ur/8 µp = |H8(1, u)| µ0,p = hI (U8(1, 0)) Encodes the LOIDs independently from their local orientations! Requires binarization… Spherical sequences are undeﬁned in 3D… g(f(x), m) = g(f(R✓0 x), m), 8✓0 2 [0, 2⇡), 8x 2 RL1⇥L2
- 54. MULTIDIRECTIONAL TEXTURE OPERATORS 54 • Locally rotation-invariant operators over • Locally “aligning” directional operators • Steerable Riesz wavelets [Unser2013, Depeursinge2014b] • Operators: th-order multi-scale image derivatives L1 ⇥ L2 input imageN4 A. Depeursinge et al. N = 1 N = 2 N = 3 Fig. 1. Templates corresponding to the Riesz kernels convolved with a Gaussian smoother for N=1,2,3. N = 1 g(1,0)(x, m) f(x) g(1,0)(f(x), m) g(0,1)(x, m) g(0,1)(f(x), m) g(f(x), m) = g(f(R✓0 x), m), 8✓0 2 [0, 2⇡), 8x 2 RL1⇥L2
- 55. MULTIDIRECTIONAL TEXTURE OPERATORS 55 • Locally rotation-invariant operators over • Locally “aligning” directional operators • Steerable Riesz wavelets [Unser2013, Depeursinge2014b] • Operators: th-order multi-scale image derivatives L1 ⇥ L2 N4 A. Depeursinge et al. N = 1 N = 2 N = 3 Fig. 1. Templates corresponding to the Riesz kernels convolved with a Gaussian smoother for N=1,2,3. N = 1 N = 2 N = 3 g(1,0)(x, m) g(2,0)(x, m)g(0,1)(x, m) g(0,2)(x, m) g(0,3)(x, m) g(1,1)(x, m) g(3,0)(x, m) g(2,1)(x, m) g(1,2)(x, m) g(f(x), m) = g(f(R✓0 x), m), 8✓0 2 [0, 2⇡), 8x 2 RL1⇥L2
- 56. MULTIDIRECTIONAL TEXTURE OPERATORS 56 • Locally rotation-invariant operators over • Locally “aligning” directional operators • Steerable Riesz wavelets [Unser2013, Depeursinge2014b] • Steerability: L1 ⇥ L2 g(1,0)(R✓0 x, 0) = cos ✓0 g(1,0)(x, 0) + sin ✓0 g(0,1)(x, 0) g(f(x), m) = g(f(R✓0 x), m), 8✓0 2 [0, 2⇡), 8x 2 RL1⇥L2
- 57. MULTIDIRECTIONAL TEXTURE OPERATORS 57 • Locally rotation-invariant operators over • Locally “aligning” directional operators • Steerable Riesz wavelets [Unser2013, Depeursinge2014b] • Local rotation-invariance: L1 ⇥ L2 ✓max(m) := arg max ✓02[0,2⇡) ✓ cos ✓0 g(1,0)(f(x), m) + sin ✓0 g(0,1)(f(x), m) ◆ µM = 1 |M| Z M ✓ g(1,0)(f(R✓max(m) x), m) ◆2 dm) g(f(x), m) = g(f(R✓0 x), m), 8✓0 2 [0, 2⇡), 8x 2 RL1⇥L2
- 58. MULTIDIRECTIONAL TEXTURE OPERATORS 58 • Locally rotation-invariant operators over • Locally “aligning” directional operators • Steerable Riesz wavelets [Depeursinge2014b, Unser2013] • Local rotation-invariance: L1 ⇥ L2 ✓max(m) := arg max ✓02[0,2⇡) ✓ cos ✓0 g(1,0)(f(x), m) + sin ✓0 g(0,1)(f(x), m) ◆ µM = 1 |M| Z M ✓ g(1,0)(f(R✓max(m) x), m) ◆2 dm) Encodes the LOIDs independently from their local orientations! No binarization required! Available in 3D [Chenouard2012, Depeursinge2015a], and combined with feature learning [Depeursinge2014b]. g(f(x), m) = g(f(R✓0 x), m), 8✓0 2 [0, 2⇡), 8x 2 RL1⇥L2
- 59. • Operators characterizing the LOIDs MULTIDIRECTIONAL TEXTURE OPERATORS GLCMs Riesz wavelets ( ) 59 GLCM contrast GLCMcontrast d = 1 ( k1 = 1, k2 = 0) d=1(k1=0,k2=1) N = 2 1 |M| Z M ✓ g(2,0)(f(x),m) ◆2 dm 1 |M| Z M ✓ g(0,2)(f(x), m) ◆2 dm aligned Riesz wavelets ( )N = 2 1 |M| Z M ✓ g(0,2)(f(R✓max(m) x), m) ◆2 dm 1 |M| Z M ✓ g(2,0)(f(R✓max(m)x),m) ◆2 dm
- 60. • Operators characterizing the LOIDs MULTIDIRECTIONAL TEXTURE OPERATORS 60 GLCMs GLCM contrast GLCMcontrast d = 1 ( k1 = 1, k2 = 0) d=1(k1=0,k2=1) Riesz wavelets ( ) 1 |M| Z M ✓ g(2,0)(f(x),m) ◆2 dm 1 |M| Z M ✓ g(0,2)(f(x), m) ◆2 dm N = 2 aligned Riesz wavelets ( )N = 2 1 |M| Z M ✓ g(0,2)(f(R✓max(m) x), m) ◆2 dm 1 |M| Z M ✓ g(2,0)(f(R✓max(m)x),m) ◆2 dm
- 61. MULTIDIRECTIONAL TEXTURE OPERATORS 61 • Isotropic or directional analysis? [Depeursinge2014b] • Outex [Ojala2002]: 24 classes, 180 images/class, 9 rotation angles in • Texture classiﬁcation: linear SVMs trained with unrotated images only IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2013 5 1) canvas001 2) canvas002 3) canvas003 4) canvas005 5) canvas006 6) canvas009 7) canvas011 8) canvas021 9) canvas022 10) canvas023 11) canvas025 12) canvas026 13) canvas031 14) canvas032 15) canvas033 16) canvas035 17) canvas038 18) canvas039 19) tile005 20) tile006 21) carpet002 22) carpet004 23) carpet005 24) carpet009 Fig. 5. 128 ⇥ 128 blocks from the 24 texture classes of the Outex database. 1) canvas 2) cloth 3) cotton 4) grass 5) leather 6) matting 7) paper 8) pigskin 9) rafﬁa 10) rattan 11) reptile 12) sand 13) straw 14) weave 15) wood 16) wool Fig. 6. 16 Brodatz texture classes of the Contrib TC 00000 test suite. 180 ⇥ 180 images from rotation angles 20 , 70 , 90 , 120 , 135 and 150 of the other seven Brodatz images for each class. The total number of images in the test set is 672. G. Experimental setup OVA SVM models using Gaussian kernels as K(xi, xj) = Inc., 2012. The computational complexity is dominated by the local orientation of N c in Eq. 11, which consists of ﬁnding the roots of the polynomials deﬁned by the steering matrix A✓ . It is therefore NP–hard (Non–deterministic Polynomial–time hard), where the order of the polynomials is controlled by the order of the Riesz transform N. 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 1 2 3 4 5 6 7 8 9 10 ini/alcoeﬀs aligned1sttemplate order of the Riesz transformN classiﬁcationaccuracy Riesz wavelets aligned Riesz wavelets
- 62. MULTIDIRECTIONAL TEXTURE OPERATORS 62 • Isotropic or directional analysis? [Depeursinge2014b] • Outex [Ojala2002]: 24 classes, 180 images/class, 9 rotation angles in • Texture classiﬁcation: linear SVMs trained with unrotated images only IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2013 5 1) canvas001 2) canvas002 3) canvas003 4) canvas005 5) canvas006 6) canvas009 7) canvas011 8) canvas021 9) canvas022 10) canvas023 11) canvas025 12) canvas026 13) canvas031 14) canvas032 15) canvas033 16) canvas035 17) canvas038 18) canvas039 19) tile005 20) tile006 21) carpet002 22) carpet004 23) carpet005 24) carpet009 Fig. 5. 128 ⇥ 128 blocks from the 24 texture classes of the Outex database. 1) canvas 2) cloth 3) cotton 4) grass 5) leather 6) matting 7) paper 8) pigskin 9) rafﬁa 10) rattan 11) reptile 12) sand 13) straw 14) weave 15) wood 16) wool Fig. 6. 16 Brodatz texture classes of the Contrib TC 00000 test suite. 180 ⇥ 180 images from rotation angles 20 , 70 , 90 , 120 , 135 and 150 of the other seven Brodatz images for each class. The total number of images in the test set is 672. G. Experimental setup OVA SVM models using Gaussian kernels as K(xi, xj) = Inc., 2012. The computational complexity is dominated by the local orientation of N c in Eq. 11, which consists of ﬁnding the roots of the polynomials deﬁned by the steering matrix A✓ . It is therefore NP–hard (Non–deterministic Polynomial–time hard), where the order of the polynomials is controlled by the order of the Riesz transform N. 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 1 2 3 4 5 6 7 8 9 10 ini/alcoeﬀs aligned1sttemplate order of the Riesz transformN classiﬁcationaccuracy Riesz wavelets aligned Riesz wavelets Isotropic operators (i.e., ) perform best when not aligned! N = 0
- 63. OUTLINE • Biomedical texture analysis: background • Deﬁning texture processes • Notations, sampling and texture functions • Texture operators, primitives and invariances • Multiscale analysis • Operator scale and uncertainty principle • Region of interest and operator aggregation • Multidirectional analysis • Isotropic versus directional operators • Importance of the local organization of image directions • Conclusions • References L1 L2 M1 M2 M of1(x) : of2(x) : R X Ø Ø @f (x @x1 R X Ø Ø@f(x) @x2 Ø Ødx image gra Fig. 1. The joint responses of image gradients ≥ | @f (x) @x1 |,| @f (x @x2 able to discriminate between the two textures classes f1(x) when integrated over the image domain X . One circle in t representation corresponds to one realization (i.e., full image) · m
- 64. CONCLUSIONS • We presented a general framework to describe and analyse texture information in 2D and 3D • Tissue structures in 2D/3D medical images contain extremely rich and valuable information to optimize personalized medicine in a non-invasive way • Invisible to the naked eye! 64 L1 L2 M1 M2 M of1(x) : of2(x) : R X Ø Ø @f @x R X Ø Ø@f(x) @x2 Ø Ødx image gr . 1. The joint responses of image gradients ≥ | @f (x) @x1 |,| @f @x e to discriminate between the two textures classes f1(x en integrated over the image domain X . One circle in · m malignant, nonresponder malignant, responder benign pre-malignant undeﬁned quant. feat. #1 quant.feat.#2
- 65. MULTISCALE TEXTURE OPERATORS • How large must be the region of interest ? • Enough to evaluate texture stationarity in terms of human perception / tissue biology • Example with operator: undecimated isotropic Simoncelli’s dyadi [PoS2000] applied to all image positions • Operators’ responses are averaged over M TEXTURE OPERATORS AND P • From texture operators to • The operator is typ by “sliding” its window • Regional texture measuremen aggregation of • For instance, integration can b • e.g., average: M1 L1 ⇥ · gn(x, m) gn(f(x), m) µ = 0 B @ µ1 ... µP 1 C A = |Mf(x) g1(f(x), m) m 2 RM1⇥M2 g2 original image with regions I 1 |M| Z M |g1(f(x), m)|dm M feature space 1 |M| Z M |g2(f(x),m)|dm f(x) Ma, Mb, Mc The average over the en does not c to anythin ˆg1(⇢) = ⇢ cos ⇡ 2 log2 2⇢ ⇡ , ⇡ 4 ⇢ ⇡ 0, otherwise. ˆg2(⇢) = ⇢ cos ⇡ 2 log2 4⇢ ⇡ , ⇡ 8 ⇢ ⇡ 2 0, otherwise. g1,2 f(⇢, ) = ˆg1,2(⇢, ) · ˆf(⇢, ) Nor biolo BoVW can be used to ﬁrst reveal the intra-class visual diversity texture operators region of interest and aggregation scales uncertainty principle averaging operators’ responses directions isotropic versus directional importance of LOIDs CONCLUSIONS • Biomedical textures are realizations of complex non-stationary spatial stochastic processes • General-purpose image operators are necessary to identify data-speciﬁc discriminative scales and directions 65 MULTISCALE TEXTURE OPERATORS • How large must be the region of interest ? • Enough to evaluate texture stationarity in terms of human perception / tissue biology • Example with operator: undecimated isotropic Simoncelli’s dyad [PoS2000] applied to all image positions • Operators’ responses are averaged over M TEXTURE OPERATORS AND P • From texture operators to • The operator is ty by “sliding” its window • Regional texture measureme aggregation of • For instance, integration can • e.g., average: M1 L1 ⇥ gn(x, m) gn(f(x), m) µ = 0 B @ µ1 ... µP 1 C A = f(x) g1(f(x), m) m 2 RM1⇥M2 g original image with regions I 1 |M| Z M |g1(f(x), m)|dm M feature space 1 |M| Z M |g2(f(x),m)|dm f(x) Ma, Mb, Mc The average over the e does not c to anythin ˆg1(⇢) = ⇢ cos ⇡ 2 log2 2⇢ ⇡ , ⇡ 4 ⇢ ⇡ 0, otherwise. ˆg2(⇢) = ⇢ cos ⇡ 2 log2 4⇢ ⇡ , ⇡ 8 ⇢ ⇡ 2 0, otherwise. g1,2 f(⇢, ) = ˆg1,2(⇢, ) · ˆf(⇢, ) Nor bio BoVW can be used to ﬁrst reveal the intra-class visual diversity F ! ˆf(!)f(x) )TextureQbased'biomarkers:'current'limitaGons' x Assume'homogeneous'texture'properGes'over'the' enGre'lesion'[5]' ' x NonQspeciﬁc'features' x Global'vs'local'characterizaGon'of'image'direcGons'[6]' REVIEW: Quantitative Imaging in Cancer Evolution and Ecology Gatenby et al with the mean signal value. By using just two sequences, a contrast-enhanced T1 sequence and a ﬂuid-attenuated inver- sion-recovery sequence, we can deﬁne four habitats: high or low postgadolini- um T1 divided into high or low ﬂuid-at- tenuated inversion recovery. When these voxel habitats are projected into the tu- mor volume, we ﬁnd they cluster into spatially distinct regions. These habitats can be evaluated both in terms of their relative contributions to the total tumor volume and in terms of their interactions with each other, based on the imaging characteristics at the interfaces between regions. Similar spatially explicit analysis can be performed with CT scans (Fig 5). Analysis of spatial patterns in cross-sectional images will ultimately re- quire methods that bridge spatial scales from microns to millimeters. One possi- ble method is a general class of numeric tools that is already widely used in ter- restrial and marine ecology research to link species occurrence or abundance with environmental parameters. Species distribution models (48–51) are used to gain ecologic and evolutionary insights and to predict distributions of species or morphs across landscapes, sometimes extrapolating in space and time. They can easily be used to link the environ- mental selection forces in MR imaging- deﬁned habitats to the evolutionary dy- namics of cancer cells. Summary Imaging can have an enormous role in the development and implementation of rise to local-regional phenotypic adap- tations. Phenotypic alterations can re- sult from epigenetic, genetic, or chro- mosomal rearrangements, and these in turn will affect prognosis and response to therapy. Changes in habitats or the relative abundance of speciﬁc ecologic communities over time and in response to therapy may be a valuable metric with which to measure treatment efﬁcacy and emergence of resistant populations. microenvironment can be rewarded by increased proliferation. This evolution- ary dynamic may contribute to distinct differences between the tumor edges and the tumor cores, which frequently can be seen at analysis of cross-sec- tional images (Fig 5). Interpretation of the subsegmenta- tion of tumors will require computa- tional models to understand and predict the complex nonlinear dynamics that lead to heterogeneous combinations of radiographic features. We have ex- Figure 4 Figure 4: Left: Contrast-enhanced T1 image from subject TCGA-02-0034 in The Cancer Genome Atlas–Glioblastoma Multiforme repository of MR volumes of glioblastoma multiforme cases. Right: Spatial distribution of MR imaging–deﬁned habitats within the tumor. The blue region (low T1 postgadolinium, low ﬂuid-attenuated inversion recovery) is particularly notable because it presumably represents a habitat with low blood ﬂow but high cell density, indicating a population presumably adapted to hypoxic acidic conditions. REVIEW: Quantitative Imaging in Cancer Evolution and Ecology Gatenby et a with the mean signal value. By using jus two sequences, a contrast-enhanced T1 sequence and a ﬂuid-attenuated inver sion-recovery sequence, we can deﬁne four habitats: high or low postgadolini um T1 divided into high or low ﬂuid-at tenuated inversion recovery. When these voxel habitats are projected into the tu mor volume, we ﬁnd they cluster into spatially distinct regions. These habitat can be evaluated both in terms of thei relative contributions to the total tumo volume and in terms of their interaction with each other, based on the imaging characteristics at the interfaces between regions. Similar spatially explicit analysi can be performed with CT scans (Fig 5) Analysis of spatial patterns in cross-sectional images will ultimately re quire methods that bridge spatial scale from microns to millimeters. One possi ble method is a general class of numeric tools that is already widely used in ter restrial and marine ecology research to link species occurrence or abundance with environmental parameters. Specie distribution models (48–51) are used to gain ecologic and evolutionary insight and to predict distributions of species o morphs across landscapes, sometime extrapolating in space and time. They can easily be used to link the environ mental selection forces in MR imaging deﬁned habitats to the evolutionary dy namics of cancer cells. Summary Imaging can have an enormous role in the development and implementation o patient-speciﬁc therapies in cancer. The rise to local-regional phenotypic adap- tations. Phenotypic alterations can re- sult from epigenetic, genetic, or chro- mosomal rearrangements, and these in turn will affect prognosis and response to therapy. Changes in habitats or the relative abundance of speciﬁc ecologic communities over time and in response to therapy may be a valuable metric with which to measure treatment efﬁcacy and emergence of resistant populations. Emerging Strategies for Tumor Habitat microenvironment can be rewarded by increased proliferation. This evolution- ary dynamic may contribute to distinct differences between the tumor edges and the tumor cores, which frequently can be seen at analysis of cross-sec- tional images (Fig 5). Interpretation of the subsegmenta- tion of tumors will require computa- tional models to understand and predict the complex nonlinear dynamics that lead to heterogeneous combinations of radiographic features. We have ex- ploited ecologic methods and models to Figure 4 Figure 4: Left: Contrast-enhanced T1 image from subject TCGA-02-0034 in The Cancer Genome Atlas–Glioblastoma Multiforme repository of MR volumes of glioblastoma multiforme cases. Right: Spatial distribution of MR imaging–deﬁned habitats within the tumor. The blue region (low T1 postgadolinium, low ﬂuid-attenuated inversion recovery) is particularly notable because it presumably represents a habitat with low blood ﬂow but high cell density, indicating a population presumably adapted to hypoxic acidic conditions. [5]'QuanGtaGve'imaging'in'cancer'evoluGon'and'ecology,'Gatenby'et'al.,'Radiology,'269(1):8Q15,'2013' 5' global'direcGonal'operators:' local'grouped'steering:' [6]'RotaGonQcovariant'texture'learning'using'steerable'Riesz'wavelets,'Depeursinge'et'al.,'IEEE'Trans'Imag'Proc.,'23(2):898Q908,'2014.' TextureQbased'biomarkers:'current'limita x Assume'homogeneous'texture'properGes'over't enGre'lesion'[5]' ' x NonQspeciﬁc'features' x Global'vs'local'characterizaGon'of'image'direcGo REVIEW: Quantitative Imaging in Cancer Evolution and Ecology with th two se sequen sion-re four h um T1 tenuat voxel mor v spatia can be relativ volum with e charac region can be An cross- quire from m ble me tools t restria link s with e distrib gain e and to morph extrap can ea menta deﬁne namic Summ Imagin the de patien achiev metho place assess The n been c Cance mation work. consor ducibl extrac rise to local-regional phenotypic adap- tations. Phenotypic alterations can re- sult from epigenetic, genetic, or chro- mosomal rearrangements, and these in turn will affect prognosis and response to therapy. Changes in habitats or the relative abundance of speciﬁc ecologic communities over time and in response to therapy may be a valuable metric with which to measure treatment efﬁcacy and emergence of resistant populations. Emerging Strategies for Tumor Habitat Characterization A method for converting images to spa- tially explicit tumor habitats is shown in Figure 4. Here, three-dimensional MR imaging data sets from a glioblastoma are segmented. Each voxel in the tumor is deﬁned by a scale that includes its image intensity in different sequences. In this case, the imaging sets are from (a) a contrast-enhanced T1 sequence, (b) a fast spin-echo T2 sequence, and (c) a ﬂuid-attenuated inversion-recov- microenvironment can be rewarded by increased proliferation. This evolution- ary dynamic may contribute to distinct differences between the tumor edges and the tumor cores, which frequently can be seen at analysis of cross-sec- tional images (Fig 5). Interpretation of the subsegmenta- tion of tumors will require computa- tional models to understand and predict the complex nonlinear dynamics that lead to heterogeneous combinations of radiographic features. We have ex- ploited ecologic methods and models to investigate regional variations in cancer environmental and cellular properties that lead to speciﬁc imaging character- istics. Conceptually, this approach as- sumes that regional variations in tumors can be viewed as a coalition of distinct ecologic communities or habitats of cells in which the environment is governed, at least to ﬁrst order, by variations in vascular density and blood ﬂow. The environmental conditions that result from alterations in blood ﬂow, such as Figure 4 Figure 4: Left: Contrast-enhanced T1 image from subject TCGA-02-0034 in The Cancer Genome Atlas–Glioblastoma Multiforme repository of MR volumes of glioblastoma multiforme cases. Right: Spatial distribution of MR imaging–deﬁned habitats within the tumor. The blue region (low T1 postgadolinium, low ﬂuid-attenuated inversion recovery) is particularly notable because it presumably represents a habitat with low blood ﬂow but high cell density, indicating a population presumably adapted to hypoxic acidic conditions. REVIEW: Quantitative Imaging in Cancer Evolution and Ecology rise to local-re tations. Pheno sult from epig mosomal rearr turn will affect to therapy. Ch relative abunda communities ov to therapy may which to measu emergence of r Emerging Stra Characterizati A method for c tially explicit tu Figure 4. Here imaging data s are segmented. is deﬁned by image intensity In this case, th microenvironment can be rewarded by increased proliferation. This evolution- ary dynamic may contribute to distinct differences between the tumor edges and the tumor cores, which frequently can be seen at analysis of cross-sec- tional images (Fig 5). Interpretation of the subsegmenta- tion of tumors will require computa- tional models to understand and predict the complex nonlinear dynamics that lead to heterogeneous combinations of radiographic features. We have ex- ploited ecologic methods and models to investigate regional variations in cancer environmental and cellular properties that lead to speciﬁc imaging character- istics. Conceptually, this approach as- sumes that regional variations in tumors can be viewed as a coalition of distinct ecologic communities or habitats of cells in which the environment is governed, at least to ﬁrst order, by variations in Figure 4 Figure 4: Left: Contrast-enhanced T1 image from subject TCGA-02-0034 Atlas–Glioblastoma Multiforme repository of MR volumes of glioblastoma mu distribution of MR imaging–deﬁned habitats within the tumor. The blue regio ﬂuid-attenuated inversion recovery) is particularly notable because it presum low blood ﬂow but high cell density, indicating a population presumably adap [5]'QuanGtaGve'imaging'in'cancer'evoluGon'and'ecology,'Gatenby'et'al.,'Radiology,'269(1):8Q15,'2013' 5' global'direcGonal'operators:' local'groupe [6]'RotaGonQcovariant'texture'learning'using'steerable'Riesz'wavelets,'Depeursinge'et'al.,'IEEE'Trans'Imag'Proc.,'23(2):898 MULTIDIRECTIONAL TEXTURE OPERATORS 58 • Isotropic or directional analysis? [DFV2014] • Outex [OPM2002]: 24 classes, 180 images/class, 9 rotation angles in • Texture classiﬁcation: linear SVMs trained with unrotated images only IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2013 5 1) canvas001 2) canvas002 3) canvas003 4) canvas005 5) canvas006 6) canvas009 7) canvas011 8) canvas021 9) canvas022 10) canvas023 11) canvas025 12) canvas026 13) canvas031 14) canvas032 15) canvas033 16) canvas035 17) canvas038 18) canvas039 19) tile005 20) tile006 21) carpet002 22) carpet004 23) carpet005 24) carpet009 Fig. 5. 128 ⇥ 128 blocks from the 24 texture classes of the Outex database. 1) canvas 2) cloth 3) cotton 4) grass 5) leather 6) matting 7) paper 8) pigskin 9) rafﬁa 10) rattan 11) reptile 12) sand 13) straw 14) weave 15) wood 16) wool Fig. 6. 16 Brodatz texture classes of the Contrib TC 00000 test suite. 180 ⇥ 180 images from rotation angles 20 , 70 , 90 , 120 , 135 and 150 of the other seven Brodatz images for each class. The total number of images in the test set is 672. G. Experimental setup OVA SVM models using Gaussian kernels as K(xi, xj) = ||xi xj ||2 Inc., 2012. The computational complexity is dominated by the local orientation of N c in Eq. 11, which consists of ﬁnding the roots of the polynomials deﬁned by the steering matrix A✓ . It is therefore NP–hard (Non–deterministic Polynomial–time hard), where the order of the polynomials is controlled by the order of the Riesz transform N. 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 1 2 3 4 5 6 7 8 9 10 ini/alcoeﬀs aligned1sttemplate order of the Riesz transformN classiﬁcationaccuracy Riesz wavelets aligned Riesz wavelets Isotropic operators (i.e., ) perform best when not aligned! N = 0 ULTIDIRECTIONAL TEXTURE OPERATORS 58 • Isotropic or directional analysis? [DFV2014] • Outex [OPM2002]: 24 classes, 180 images/class, 9 rotation angles in • Texture classiﬁcation: linear SVMs trained with unrotated images only IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2013 5 1) canvas001 2) canvas002 3) canvas003 4) canvas005 5) canvas006 6) canvas009 7) canvas011 8) canvas021 9) canvas022 10) canvas023 11) canvas025 12) canvas026 13) canvas031 14) canvas032 15) canvas033 16) canvas035 17) canvas038 18) canvas039 19) tile005 20) tile006 21) carpet002 22) carpet004 23) carpet005 24) carpet009 Fig. 5. 128 ⇥ 128 blocks from the 24 texture classes of the Outex database. 1) canvas 2) cloth 3) cotton 4) grass 5) leather 6) matting 7) paper 8) pigskin 9) rafﬁa 10) rattan 11) reptile 12) sand 13) straw 14) weave 15) wood 16) wool Fig. 6. 16 Brodatz texture classes of the Contrib TC 00000 test suite. 180 ⇥ 180 images from rotation angles 20 , 70 , 90 , 120 , 135 and 150 of the other seven Brodatz images for each class. The total number of images in the test set is 672. G. Experimental setup OVA SVM models using Gaussian kernels as K(xi, xj) = exp( ||xi xj ||2 2 2 k ) are used both to learn texture signatures and to classify the texture instances in the ﬁnal feature space obtained after k iterations. A number of scales J = 6 Inc., 2012. The computational complexity is dominated by the local orientation of N c in Eq. 11, which consists of ﬁnding the roots of the polynomials deﬁned by the steering matrix A✓ . It is therefore NP–hard (Non–deterministic Polynomial–time hard), where the order of the polynomials is controlled by the order of the Riesz transform N. III. RESULTS The performance of our approach is demonstrated with 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 1 2 3 4 5 6 7 8 9 10 ini/alcoeﬀs aligned1sttemplate order of the Riesz transformN classiﬁcationaccuracy Riesz wavelets aligned Riesz wavelets Isotropic operators (i.e., ) perform best when not aligned! N = 0
- 66. THANKS ! BIG @ EPFL Michael Unser Julien Fageot Arash Amini and all members MedGIFT @ HES-SO Henning Müller Yashin Dicente Roger Schaer Ranveer Joyseree Oscar Jimenez Manfredo Atzori 66 Source code and data available! https://sites.google.com/site/btamiccai2015/ adrien.depeursinge@epﬂ.ch PROCESSING FOR BIOMEDICAL IMAGE AN Adrien Depeursinge, PhD MICCAI 2015 Tutorial on Biomedical Texture Analysis (BTA), Munich, O 09), where create real- os can also s problem ic texture’’ 1999), and ature pub- id textures on the fea- since only sis. tals of 3-D ction 3 de- etrieve pa- trieval. The ure are re- us expecta- pplication- d together various ap- Section 7, h a general 1981), it is del of tex- m (Mallat, ches based using sets of prototype primitives. The concept of texture primitive is naturally extended in 3-D as the geometry of the voxel sequence used by a given texture analysis method. We consider a primitive C(i,j,k) centered at a point (i,j,k) that lives on a neighborhood of this point. The primitive is constituted by a set of voxels with gray tone values that forms a 3-D structure. Typical C neighborhoods are voxel pairs, linear, planar, spherical or unconstrained. Signal assignment to the primitive can be either binary, categorical or continuous. Two example texture primitives are shown in Fig. 4. Texture primitives refer to local processing of 3-D images and local patterns (see Toriwaki and Yoshida, 2009). Fig. 2. 3-D digitized images and sampling in Cartesian coordinates. Fig. 3. 3-D digitized images and sampling in spherical coordinates. 2 [iDx,(i + 1)Dx]; [jDy,(j + 1)Dy]; [kDz,(k + Yoshida, 2009). This cuboid is called a voxel. coordinates (r,h,/) are unevenly sampled to Fig. 3. ure primitive has been widely used in 2-D tex- nes the elementary building block of a given k, 1979; Jain et al., 1995; Lin et al., 1999). All pproaches aim at modeling a given texture e primitives. The concept of texture primitive in 3-D as the geometry of the voxel sequence ure analysis method. We consider a primitive point (i,j,k) that lives on a neighborhood of ive is constituted by a set of voxels with gray ms a 3-D structure. Typical C neighborhoods r, planar, spherical or unconstrained. Signal rimitive can be either binary, categorical or mple texture primitives are shown in Fig. 4. fer to local processing of 3-D images and local ki and Yoshida, 2009). d images and sampling in Cartesian coordinates. plexin3-D. varioustaxonomiesare Aclariﬁcationofthetax- ratelydeﬁnethescopeof andYoshida,2009.Three- rearebothgeneraland edinR3 andinclude: in‘‘ﬁlled’’objects neratedbyavolumetric raphy,confocalimaging). sof‘‘hollow’’objectsas ionaltimesequencesas accountsfortexturesde- oordinates.Solidtextures meansthatanumberof ftheEuclideanspaceis 1965;Foncubierta-Rodrí- nedastexturedsurfacein na(2004),or2.5-dimen- guetetal.(2008),where -Dobjectsandcanbein- (2)isalsousedinKajiya andHaindl(2009),where eofobjectstocreatereal- nalysisinvideoscanalso xtureanalysisproblem edby‘‘dynamictexture’’ andCrowley(1999),and ewoftheliteraturepub- biomedicalsolidtextures ofthistextisonthefea- gtechniques,sinceonly textureanalysis. Thefundamentalsof3-D Section2.Section3de- ystematicallyretrievepa- ﬁcationandretrieval.The intheliteraturearere- olistthevariousexpecta- Theresultingapplication- edandgroupedtogether dgapsofthevariousap- aregiveninSection7, andtheirassumedinteractionsdeﬁnethepropertiesofthetexture analysisapproaches,fromstatisticaltostructuralmethods. InSection2.1,wedeﬁnethemathematicalframeworkand notationsconsideredtodescribethecontentof3-Ddigitalimages. Thenotionoftextureprimitivesaswellastheirscalesanddirec- tionsaredeﬁnedinSection2.2. 2.1.3-Ddigitizedimagesandsampling InCartesiancoordinates,ageneric3-Dcontinuousimageisde- ﬁnedbyafunctionofthreevariablesf(x,y,z),wherefrepresentsa scalaratapointðx;y;zÞ2R3 .A3-DdigitalimageF(i,j,k)ofdimen- sionsMÂNÂOisobtainedfromsamplingfatpointsði;j;kÞ2Z3 ofa3-Dorderedarray(seeFig.2).Incrementsin(i,j,k),correspond tophysicaldisplacementsinR3 parametrizedbytherespective spacings(Dx,Dy,Dz).Foreverycellofthedigitizedarray,thevalue ofF(i,j,k)istypicallyobtainedbyaveragingfinthecuboiddomain deﬁnedby(x,y,z)2[iDx,(i+1)Dx];[jDy,(j+1)Dy];[kDz,(k+ 1)Dz])(ToriwakiandYoshida,2009).Thiscuboidiscalledavoxel. Thethreesphericalcoordinates(r,h,/)areunevenlysampledto (R,H,U)asshowninFig.3. 2.2.Textureprimitives Thenotionoftextureprimitivehasbeenwidelyusedin2-Dtex- tureanalysisanddeﬁnestheelementarybuildingblockofagiven textureclass(Haralick,1979;Jainetal.,1995;Linetal.,1999).All textureprocessingapproachesaimatmodelingagiventexture usingsetsofprototypeprimitives.Theconceptoftextureprimitive isnaturallyextendedin3-Dasthegeometryofthevoxelsequence usedbyagiventextureanalysismethod.Weconsideraprimitive C(i,j,k)centeredatapoint(i,j,k)thatlivesonaneighborhoodof thispoint.Theprimitiveisconstitutedbyasetofvoxelswithgray tonevaluesthatformsa3-Dstructure.TypicalCneighborhoods arevoxelpairs,linear,planar,sphericalorunconstrained.Signal assignmenttotheprimitivecanbeeitherbinary,categoricalor continuous.TwoexampletextureprimitivesareshowninFig.4. Textureprimitivesrefertolocalprocessingof3-Dimagesandlocal patterns(seeToriwakiandYoshida,2009). Fig.2.3-DdigitizedimagesandsamplinginCartesiancoordinates. Stanford University Daniel Rubin Olivier Gevaert Ann Leung Dimitri Van de Ville, UNIGE Camille Kurtz, Paris Descartes Pierre-Alexandre Poletti, HUG John-Paul Ward, UCF
- 67. REFERENCES (SORTING IN ALPHABETICAL ORDER) 67 [Ahonen2009] Ahonen, T.; Matas, J.; He, C. Pietikäinen, M. Salberg, A.-B.; Hardeberg, J. Jenssen, R. (Eds.) Rotation Invariant Image Description with Local Binary Pattern Histogram Fourier Features Image Analysis, Springer Berlin Heidelberg, 2009, 5575, 61-70 [Aerts2014] Aerts, H. J. W. L.; Velazquez, E. R.; Leijenaar, R. T. H.; Parmar, C.; Grossmann, P.; Carvalho, S.; Bussink, J.; Monshouwer, R.; Haibe-Kains, B.; Rietveld, D.; Hoebers, F.; Rietbergen, M. M.; Leemans, C. R.; Dekker, A.; Quackenbush, J.; Gillies, R. J. Lambin, P. Decoding Tumour Phenotype by Noninvasive Imaging Using a Quantitative Radiomics Approach Nature Communications, Nature Publishing Group, a division of Macmillan Publishers Limited. All Rights Reserved, 2014, 5 [Blakemore1969] Blakemore, C. Campbell, F. W. On the Existence of Neurones in the Human Visual System Selectively Sensitive to the Orientation and Size of Retinal Images The Journal of Physiology, 1969, 203, 237-260 [Candes2000] Candès, E. J. Donoho, D. L. Curvelets - A Surprisingly Effective Nonadaptive Representation For Objects with Edges Curves and Surface Fitting, Vanderbilt University Press, 2000, 105-120 [Chenouard2011] Chenouard, N. Unser, M. 3D Steerable Wavelets and Monogenic Analysis for Bioimaging 2011 IEEE International Symposium on Biomedical Imaging: From Nano to Macro, 2011, 2132-2135 [Chenouard2012] Chenouard, N. Unser, M. 3D Steerable Wavelets in Practice IEEE Transactions on Image Processing, 2012, 21, 4522-4533
- 68. REFERENCES (SORTING IN ALPHABETICAL ORDER) 68 [Dalal2005] Dalal, N. Triggs, B. Histograms of Oriented Gradients for Human Detection Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), IEEE Computer Society, 2005, 1, 886-893 [Depeursinge2012a] Depeursinge, A.; Foncubierta-Rodrguez, A.; Van De Ville, D. Müller, H. Multiscale Lung Texture Signature Learning Using The Riesz Transform Medical Image Computing and Computer-Assisted Intervention MICCAI 2012, Springer Berlin / Heidelberg, 2012, 7512, 517-524 [Depeursinge2012b] Depeursinge, A.; Van De Ville, D.; Platon, A.; Geissbuhler, A.; Poletti, P.-A. Müller, H. Near-Afﬁne-Invariant Texture Learning for Lung Tissue Analysis Using Isotropic Wavelet Frames IEEE Transactions on Information Technology in BioMedicine, 2012, 16, 665-675 [Depeursinge2014a] Depeursinge, A.; Foncubierta-Rodrguez, A.; Van De Ville, D. Müller, H. Three-Dimensional Solid Texture Analysis and Retrieval in Biomedical Imaging: Review and Opportunities Medical Image Analysis, 2014, 18, 176-196 [Depeursinge2014b] Depeursinge, A.; Foncubierta-Rodrguez, A.; Van De Ville, D. Müller, H. Rotation-Covariant Texture Learning Using Steerable Riesz Wavelets IEEE Transactions on Image Processing, 2014, 23, 898-908 [Depeursinge2015a] Depeursinge, A.; Pad, P.; Chin, A. C.; Leung, A. N.; Rubin, D. L.; Müller, H. Unser, M. Optimized Steerable Wavelets for Texture Analysis of Lung Tissue in 3-D CT: Classiﬁcation of Usual Interstitial Pneumonia IEEE 12th International Symposium on Biomedical Imaging, IEEE, 2015, 403-406 [Depeursinge2015b] Depeursinge, A.; Chin, A. C.; Leung, A. N.; Terrone, D.; Bristow, M.; Rosen, G. Rubin, D. L. Automated Classiﬁcation of Usual Interstitial Pneumonia Using Regional Volumetric Texture Analysis in High-Resolution CT Investigative Radiology, 2015, 50, 261-267
- 69. REFERENCES (SORTING IN ALPHABETICAL ORDER) 69 [Dumas2009] Dumas, A.; Brigitte, M.; Moreau, M.; Chrétien, F.; Baslé, M. Chappard, D. Bone Mass and Microarchitecture of Irradiated and Bone Marrow-Transplanted Mice: Inﬂuences of the Donor Strain Osteoporosis International, Springer-Verlag, 2009, 20, 435-443 [Galloway1975] Galloway, M. M. Texture Analysis Using Gray Level Run Lengths Computer Graphics and Image Processing, 1975, 4, 172-179 (Gal1975) [Gerlinger2012] Gerlinger, M.; Rowan, A. J.; Horswell, S.; Larkin, J.; Endesfelder, D.; Gronroos, E.; Martinez, P.; Matthews, N.; Stewart, A.; Tarpey, P.; Varela, I.; Phillimore, B.; Begum, S.; McDonald, N. Q.; Butler, A.; Jones, D.; Raine, K.; Latimer, C.; Santos, C. R.; Nohadani, M.; Eklund, A. C.; Spencer-Dene, B.; Clark, G.; Pickering, L.; Stamp, G.; Gore, M.; Szallasi, Z.; Downward, J.; Futreal, P. A. Swanton, C. Intratumor Heterogeneity and Branched Evolution Revealed by Multiregion Sequencing New England Journal of Medicine, 2012, 366, 883-892 [Gurcan2009] Gurcan, M. N.; Boucheron, L. E.; Can, A.; Madabhushi, A.; Rajpoot, N. M. Yener, B. Histopathological Image Analysis: A Review IEEE Reviews in Biomedical Engineering, 2009, 2, 147-171 [Haidekker2011] Haidekker, M. A. Texture Analysis Advanced Biomedical Image Analysis, John Wiley Sons, Inc., 2010, 236-275 [Haralick1979] Haralick, R. M. Statistical and Structural Approaches to Texture Proceedings of the IEEE, 1979, 67, 786-804
- 70. REFERENCES (SORTING IN ALPHABETICAL ORDER) 70 [Khan2013] Khan, A. M.; El-Daly, H.; Simmons, E. Rajpoot, N. M. HyMaP: A Hybrid Magnitude-Phase Approach to Unsupervised Segmentation of Tumor Areas in Breast Cancer Histology Images Journal of pathology informatics, Medknow Publications, 2013, 4 [Kumar2012] Kumar, V.; Gu, Y.; Basu, S.; Berglund, A.; Eschrich, S. A.; Schabath, M. B.; Forster, K.; Aerts, H. J. W. L.; Dekker, A.; Fenstermacher, D.; Goldgof, D. B.; Hall, L. O.; Lambin, P.; Balagurunathan, Y.; Gatenby, R. A. Gillies, R. J. Radiomics: The Process and the Challenges Magnetic Resonance Imaging, 2012, 30, 1234-1248 [Lazebnik2005] Lazebnik, S.; Schmid, C. Ponce, J. A Sparse Texture Representation Using Local Afﬁne Regions IEEE Transactions on Pattern Analysis and Machine Intelligence, IEEE Computer Society, 2005, 27, 1265-1278 [Malik2001] Malik, J.; Belongie, S.; Leung, T. Shi, J. Contour and Texture Analysis for Image Segmentation International Journal of Computer Vision, Kluwer Academic Publishers, 2001, 43, 7-27 [Mosquera2015] Mosquera-Lopez, C.; Agaian, S.; Velez-Hoyos, A. Thompson, I. Computer-Aided Prostate Cancer Diagnosis From Digitized Histopathology: A Review on Texture-Based Systems IEEE Reviews in Biomedical Engineering, 2015, 8, 98-113 [Ojala2002] Ojala, T.; Pietikäinen, M. Mäenpää, T. Multiresolution Gray-Scale and Rotation Invariant Texture Classiﬁcation with Local Binary Patterns IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002, 24, 971-987
- 71. REFERENCES (SORTING IN ALPHABETICAL ORDER) 71 [Petrou2006] Petrou, M. Garcia Sevilla, P. Image Processing: Dealing with Texture Wiley, 2006 [Petrou2011] Petrou, M. Deserno, T. M. (Ed.) Texture in Biomedical Images Biomedical Image Processing, Springer-Verlag Berlin Heidelberg, 2011, 157-176 [Portilla2000] Portilla, J. Simoncelli, E. P. A Parametric Texture Model Based on Joint Statistics of Complex Wavelet Coefﬁcients International Journal of Computer Vision, Kluwer Academic Publishers, 2000, 40, 49-70 [Romeny2011] ter Haar Romeny, B. M. Deserno, T. M. (Ed.) Multi-Scale and Multi-Orientation Medical Image Analysis Biomedical Image Processing, Springer-Verlag Berlin Heidelberg, 2011, 177-196 [Sifre2014] Sifre, L. Mallat, S. Rigid-Motion Scattering for Texture Classiﬁcation Submitted to International Journal of Computer Vision, 2014, abs/1403.1687, 1-19 [Simoncelli1995] Simoncelli, E. P. Freeman, W. T. The Steerable Pyramid: A Flexible Architecture for Multi-Scale Derivative Computation Proceedings of International Conference on Image Processing, 1995, 3, 444-447 [Storath2014] Storath, M.; Weinmann, A. Unser, M. Unsupervised Texture Segmentation Using Monogenic Curvelets and the Potts Model IEEE International Conference on Image Processing (ICIP), 2014, 4348-4352
- 72. REFERENCES (SORTING IN ALPHABETICAL ORDER) 72 [Unser2011] Unser, M.; Chenouard, N. Van De Ville, D. Steerable Pyramids and Tight Wavelet Frames in IEEE Transactions on Image Processing, 2011, 20, 2705-2721 [Unser2013] Unser, M. Chenouard, N. A Unifying Parametric Framework for 2D Steerable Wavelet Transforms SIAM Journal on Imaging Sciences, 2013, 6, 102-135 [Varma2005] Varma, M. Zisserman, A. A Statistical Approach to Texture Classiﬁcation from Single Images International Journal of Computer Vision, Kluwer Academic Publishers, 2005, 62, 61-81 [Ward2014] Ward, J. Unser, M. Harmonic Singular Integrals and Steerable Wavelets in Applied and Computational Harmonic Analysis, 2014, 36, 183-197 [Ward2015] Ward, J.; Pad, P. Unser, M. Optimal Isotropic Wavelets for Localized Tight Frame Representations IEEE Signal Processing Letters, 2015, 22, 1918-1921 [Xu2012] Xu, J.; Napel, S.; Greenspan, H.; Beaulieu, C. F.; Agrawal, N. Rubin, D. Quantifying the Margin Sharpness of Lesions on Radiological Images for Content-Based Image Retrieval Medical Physics, 2012, 39, 5405-5418 L2(Rd ) L2(Rd )