Computationally Efficient NMRF model based Texture Synthesis

Outline Ph.D. Research Work Conclusion and Possible Future Directions Fast NMRF based texture synthesis algorithms Arnab Sinha arnab@iitk.ac.in April 16, 2009 Thesis Supervisor: Dr. Sumana Gupta ACES-205, Dept. of EE, Indian Institute of Technology Kanpur, India Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Outline Earlier Methods Ph.D. Research Work Research problems in NMRF-tex-syn algorithms Conclusion and Possible Future Directions 1 Outline Earlier Methods Research problems in NMRF-tex-syn algorithms 2 Ph.D. Research Work Order Estimation from Fourier Domain Reduction of Computational complexity Order Estimation : Revisited Inverse Texture Synthesis 3 Conclusion and Possible Future Directions Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Outline Earlier Methods Ph.D. Research Work Research problems in NMRF-tex-syn algorithms Conclusion and Possible Future Directions The significance of texture synthesis • What defines texture ? • Locally varying intensities and/or color values • The local variations can be found perceptually similar within the total region • Texture Synthesis: Original D104 Texture Given a small texture exempler, synthesize an arbitrary sample of texture, so that the synthesized texture is visually similar to the original sample. Synthesized texture should look alike the original texture • Application of texture synthesis in - • Image segmentation, classification, synthesis, etc. • Content-based image retrieval • Development of high-level computer vision algorithms • Animation of real scenes • Perceptual analysis • Computationally fast and efficient handling of objects Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Outline Earlier Methods Ph.D. Research Work Research problems in NMRF-tex-syn algorithms Conclusion and Possible Future Directions Texture synthesis: Difﬁculty Figure: Spectrum of Natural Textures Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Outline Earlier Methods Ph.D. Research Work Research problems in NMRF-tex-syn algorithms Conclusion and Possible Future Directions Brief History of Models Texture Synthesis Algorithm Image Domain Model Mixed Domain Model Transformed Domain Model Non−Linear Models Linear Non−Linear Hidden Markov Tree Non−Linearity Introduced by Histogram Equalization Fan and Xia (2003) 2D−NCAR, Hard−limited Gaussian Process Chellappa and Kashyap (1985) Jacovitti et al. (1998) 2D−Wold Francos et al. (1993) Zhang et al. (1998) Wavelet + AR 2D− MA 1. Zhu et al. (1997) Eom (1998) Circular Harmonic Func 2. Zhu et al. (2000) + Hard−limited Gaussian Campasi and Scarano (2002) 3. Portilla and Simoncelli (2000) NNMRF Charalampidis (2006) Paget and Longstaff (1998) Mathematical Models Intuitive Models Heeger and Bergen (1995) Efros and Leung (1999) Wei and Levoy (2000) Pixel−based Ashikhmin (2001) We are working within this Framework Sampling Process Tonietto et al. (2005) Kwatra et al. (2003) Patch−based Patch−based sampling with wavelet transformation Wu et al. (2004) as a feature set for graph−cut algorithm Popular Methods Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Outline Earlier Methods Ph.D. Research Work Research problems in NMRF-tex-syn algorithms Conclusion and Possible Future Directions Description of N-MRF model • S is the lattice • Ys is the random variable at site s ∈ S • Concept of Neighborhood system: ℵs •s s = (i,j), site • r ∈ ℵs ⇔ s ∈ ℵr 1st order neighbors 2 2 • Circular neighborhood: ℵs = {r; s.t., |r − s| ≤ o } { } 2nd order neighbors • say, Xs = {Yr ; r ∈ ℵs } • Say, Y(s) = {Yr ; r s}, r, s ∈ S • Deﬁnition of MRF: P(Ys |Y(s) ) = P(Ys |{Yr ; r ∈ ℵs }) • parameteric model for P(Ys |Xs ) • semi-parameteric model for P(Ys |Xs ) • non-parameteric model for P(Ys |Xs ) Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Outline Earlier Methods Ph.D. Research Work Research problems in NMRF-tex-syn algorithms Conclusion and Possible Future Directions Description of N-MRF model: Kernel Density Estimation Deﬁnition of KDE, [Scott(1992)] N 1 • single dimensional: P(x) = N Kh (x − xi ) i=1 Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Outline Earlier Methods Ph.D. Research Work Research problems in NMRF-tex-syn algorithms Conclusion and Possible Future Directions Description of N-MRF model: Kernel Density Estimation Deﬁnition of KDE, [Scott(1992)] N 1 • single dimensional: P(x) = N Kh (x − xi ) i=1 N d 1 • multi-dimensional: P(X) = Khj (X (j) − Xi (j)) i=1 j=1 N (X (j)−Xi (j))2 • where, in case of Gaussian kernel, Khj (X (j) − Xi (j)) = √1 } exp{− 2hj2 N 2πhj • and, hj = σj N −1/(d+4) Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Outline Earlier Methods Ph.D. Research Work Research problems in NMRF-tex-syn algorithms Conclusion and Possible Future Directions Texture Synthesis Algorithm Some definitions • Input texture field: {Ys }, where, s ∈ Sin • Output texture field: {Yq }, where, q ∈ Sout Kh (Ys −Yq )Kh (Xs −Xq ) s∈Sin Y X • Definition of LCPDF: P(Yq |Xq ) = Kh (Xs −Xq ) s∈Sin X Iterative Conditional Mode (ICM) algorithm • Evaluate P(Yq = y|Xq ), for y = 0, 1, . . . , 255 gray values. • Assign Yq = y, for which the above conditional probability is maximum Local Simulated Annealing • Define a Confidence field, Cq ; q ∈ Sout , and a matrix Φq = DIAG{Cr ; r ∈ ℵq } • KhX (Xs − Xq ; Φq ) = exp{−(Xs − Xq )T Φh,q (Xs − Xq )}, and Φh,q = Φq HX ≈ hΦq • Updation rule for the confidence field 1 • Cq = min{1, |r∈ℵ | r∈ℵ Cr + u × e} q q • where, u is a random number and e is a constant scale factor Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Outline Earlier Methods Ph.D. Research Work Research problems in NMRF-tex-syn algorithms Conclusion and Possible Future Directions Texture Synthesis Algorithm Approximate Independent Conditional Mode (ICM) algorithm • Ds,q = (Xs − Xq )T Φq (Xs − Xq ) • Deﬁne Sq = {r ∈ Sin } ⊂ Sin , s.t., ∀r ∈ Sq , Dr,q = constant. • Assign Yq = yr , where r is sampled from the set Sq randomly. S in S out C q q Input texture Output Texture Confidence Field Output Neighborhood Output Confidence Vector X q Vector Wq Matrix Input Neighborhood Vectors Similarity Measure t {X s} N−MRF : ( X q − X s) ( X q − X s) t WL alg : ( X q − X s) ( X q − X s) Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Outline Earlier Methods Ph.D. Research Work Research problems in NMRF-tex-syn algorithms Conclusion and Possible Future Directions Research Problems • Order estimation • Large Computational Complexity • Computational complexity ∝ d, the dimension of the neighborhood vector. • Computational complexity ∝ (M × N), the input image size • Computational complexity ∝ I, the number of iterations required to attain global convergence Original Texture Neighborhood vector dimension ’d’ 8000 7000 computational complexity of 6000 texture synthesis algorithm is proportional to ’d’ 5000 4000 3000 2000 1000 0 0 10 20 30 40 50 Model order ’o’ Order 4 Order 8 Order 14 Figure: Effect of order on the synthesis results and computational complexity Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Order estimation from two fundamental frequencies Y c d o b a X Yj Xj Yi Xi Figure: Points a, b, c, d are the four corners of texton deﬁned by the fundamental spatial period vectors [Xi Yi ] and [Xj Yj ]. The major diagonal o gives the order of causal circular neighborhood and o/2 gives the order of non-causal circular neighborhood. Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Extraction of the parameters • Dimitri’s algorithm • estimate the two fundamental frequecies from the two-dimensional DFT of the texture sample. • computational complexity is of the order of image size. • Hays’s algorithm • it estimates the two fundamental spatial vectors from the correlation function • the algorithm is iterative • computationally expensive with respect to Dimitri’s algorithm Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis D21 D20 order = 9 order = 23 order = 4 order = 18 D52 D35 order = 48 order = 8 order = 22 order = 21 Figure: Comparison of estimated order through Dimitrios and Hays’s methods with (NR) texture synthesis results Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis A new neighborhood system Y proposed Non−causal neighborhood X Yj Xj Yi circular Non−causal neighborhood Xi Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Results: Proposed neighborhood system Circular Neighborhood Proposed Neighborhood Proposed Neighborhood Circular Neighborhood D104 D65 D95 D64 D67 D3 Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Two approaches Computational complexity affected by the neighbourhood dimension, d, and 1 the number of input pixels, N 2 Reduction methodologies Dimensionality reduction methodologies, e.g., Principal Component Analysis 1 (PCA) – to reduce the effect of d A data structure for fast search 2 Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis With Dimensionality reduction methods How the distance metric Ds,q looks after projection (Xq − Xs )T Φq (Xq − Xs ) (X q − X s )T Φ q (X q − X s ) ≈ [PrT Pr (Xq − Xs )]T Φq [PrT Pr (Xq − Xs )] = [Pr (Xq − Xs )]T Pr Φq PrT [Pr (Xq − Xs )] = (Zq − Zs )T Ψq (Zq − Zs ) = T • What is Ψq = Pr Φq Pr ? • Is it reducing the computational complexity ? Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Simulated Annealing for Principal components Original Ds,q (Xq − Xs )T Φq (Xq − Xs ) = Ds,q where, Φq = DIAG{W1 , W2 , . . . , Wd } ˆ Proposed Ds,q ˆ ˆ (Zq − Zs )T Φq (Zq − Zs ) = Ds,q ˆ where, Φq = DIAG{W1 , W2 , . . . , Wk } WHY ? Because we need only • a steady increase in the value of conﬁdence, and • the starting value has to be ”0” and ending value has to be ”1” Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Results Table: Comparison of dimensionality; Original dimension, |ℵs | = d; Reduced dimension, k (<< d), η is the ratio of computational complexities between earlier and proposed one Texture Type Texture order d k η NR D20 20 1516 60 21.9405 NR D3 30 2820 580 3.7926 NR D21 25 1960 56 29.2642 NR D22 20 1256 177 6.3042 NR D35 28 2452 287 6.6612 NR D36 22 1516 258 5.1024 ST D7 27 2288 511 3.6438 ST D13 24 1792 131 11.6006 NR+ST D18 32 3208 95 25.5666 NR+ST D4 29 2628 465 4.4755 NR+ST D5 29 2628 179 11.6262 IN/STR D15 23 1652 167 8.4897 IN/STR D42 26 2120 293 5.9699 Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Results continued ... NNMRF Proposed Algorithm Proposed Algorithm NNMRF D20 D7 k = 511 d = 2288 d = 1516 k=60 D13 D21 k=131 d=1792 d=1960 k=56 D22 D42 k = 293 d=2120 k = 177 d=1256 Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis With Fast Kernel Density Estimation Assumption: The h parameters along all the directions are equal. • Let Rn = {ts : ||ts − tn || ≤ R} N 1 • P(tn ) = N s=1 KH (ts − tn ) Y 1 • P(tn ) = N s∈Rn KH (ts − tn ) • Let Nn = |{s Rn }| R=100 1 = KH (ts − tn ) Err(tn , R) N s Rn Nn K (R) ≤ To calculate KDE NH at this target point X we only need KH (R) ≤ Source data vector these two points Target data vector max(Err) • Rel err(R) = max(Probability) Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Earlier FKDE algorithms • Improved Fast Gaussian Transform (IFGT) [Yang et al.(2003)Yang, Duraiswami, Gumerov, and Davis] • kd-tree based FKDE [Gray and Moore(2003)] • Reconstructionhistogram [Zhang et al.(2005)Zhang, Tang, and Kwok] Reconstructionhistogram • Clustering: {Clusti ; i = 1 . . . M} • Let ni as the number of source data vectors within i th cluster M 1 • P(tn ) = N i=1 KH (tn − Clusti )ni • KDE of tn given the source data points at cluster centroids with a weight factor ni /N • ﬂexibility ? Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Improved Fast Gaussian Transform • P(tn ) = KH (tn − Clusti )f (tn , Clusti ) ||tn −Clusti ||≤RIFGT 1 • P(tn ) = N KH (tn − ts ) ||tn −Clusti ||≤RIFGT s∈Clusti Source data clusters Target data vectors Due to this overlap we need to consider this source cluster R IFGT R To consider the source cluster In effect it can include some source cluster R the radius threshold has to be IFGT which was not needed at all The R IFGT can vary with the overlap size and cluster shape Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis kd-tree-based FKDE Build up the kd-tree and Search according to the radius R. Y 6 σ X > σY Hyper−Sphere Y -6 - Partition Span in Y direction e 6 e ee R max_rec_err Hyper−Rectangle e ee e e e e e e e Eigen vectors e e e e e e e e Centroid e e - e ee Sub-spaces e e - e ? e X e Reconstruction X Error R rec_err,n - R tn > R+ R max_rec_err - ? Span in X direction Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Why do we need another algorithm for FKDE Table: Why do we need another algorithm for FKDE ? C-FKDE KD-FKDE Advantage Clustering algorithm provides more Due to the hyper-plane boundary, one can use original radius R compact representation of the data space for strict error bound optimal RIFGT has to estimated Disadvantage kd-tree is not a good clustering for every tn , maximum RIFGT algorithm, therefore it does not can increase computational provide compact representation complexity of the data space Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Principal Directive Divisive Partitioning (PDDP)[Boley(1998)] • project each source data point within the present space onto the ﬁrst principal direction (eigen vector corresponding to the largest eigen value). • partition the present space into two sub-spaces with respect to the mean (or median) of the projected values. Hieararchical Boundaries Source data Tree structure of the nodes 6th IV I 1st VI 7th 2nd II III I III II VII 5th V VI VII 4th IV 8th V 8th 6th 7th 2nd 3rd 4th 5th 1st 3rd Leaf nodes Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis FKDE based on PDDP If the present node is a leaf then evaluate KDE. 1 Source data vectors Projection of source data Target data vector (t n) Projection of terget point D rec_err 2nd Direction T is within left child D If R < D rec_err T Return 0 Drec_err Else If D < R process both children Process its children D 2nd Direction Dlb (Boundary for Partition) 1st Direction mr vn D For the right child lb If Dlb R > (Which child to process) (Process child) Return 0 If Dlb > R => return 0 if D > R => process left child Else If R < D rec_err Else if Drec_err > R => return 0 Else process both children Return 0 Else process Else Process its children vn Projected target data vector mr Projected mean vector 1st Direction (a) Target point is outside (b) Target point is inside Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis FKDE based on PDDP If the present node is a leaf then evaluate KDE. 1 Is there any need to go further for the children of the present node ? 2 Source data vectors Projection of source data Target data vector (t n) Projection of terget point D rec_err 2nd Direction T is within left child D If R < D rec_err T Return 0 Drec_err Else If D < R process both children Process its children D 2nd Direction Dlb (Boundary for Partition) 1st Direction mr vn D For the right child lb If Dlb R > (Which child to process) (Process child) Return 0 If Dlb > R => return 0 if D > R => process left child Else If R < D rec_err Else if Drec_err > R => return 0 Else process both children Return 0 Else process Else Process its children vn Projected target data vector mr Projected mean vector 1st Direction (c) Target point is outside (d) Target point is inside Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis FKDE based on PDDP If the present node is a leaf then evaluate KDE. 1 Is there any need to go further for the children of the present node ? 2 Which child node (left or right or both) of the present node to process further ? 3 Source data vectors Projection of source data Target data vector (t n) Projection of terget point D rec_err 2nd Direction T is within left child D If R < D rec_err T Return 0 Drec_err Else If D < R process both children Process its children D 2nd Direction Dlb (Boundary for Partition) 1st Direction mr vn D For the right child lb If Dlb R > (Which child to process) (Process child) Return 0 If Dlb > R => return 0 if D > R => process left child Else If R < D rec_err Else if Drec_err > R => return 0 Else process both children Return 0 Else process Else Process its children vn Projected target data vector mr Projected mean vector 1st Direction (e) Target point is outside (f) Target point is inside Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Comparison between the FKDE algorithms (i−1,j) 2 (i,j+1) 3 (i,j) 1 4 (i,j+1) {1} X RGB ==> 3 dimensions {1,2} X RGB ==> 6 dimensions {1,2,3} X RGB ==> 9 dimensions {1,2,3,4} X RGB ==> 12 dimensions (g) Image considered (h) Creation of the data for creating the data set space Figure: Data set creation for FKDE analysis Table: Time comparison Dimension 3 6 9 12 KDE: Time (sec) 981.78 1204.77 2529.22 2668.58 PDDP-FKDE: Time (sec) 11.3 23.39 52.55 65.79 KD-FKDE: Time (sec) 22.55 33.88 110.62 228.44 IFGT-FKDE: Time (sec) 399.79 425.45 209.33 384.08 Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Comparison between the FKDE algorithms Table: Comparative analysis of FKDE algorithms FKDE Dimension Maximum Maximum Relative Radius algorithms probability Error Error threshold 4.33531e − 05 0.0003e − 04 PDDP-FKDE 3 0.0007 14.5445 1.32977e − 10 0.0000e − 10 6 0.0000 28.1622 9.64769e − 18 0.0014e − 18 9 0.0001 47.5693 1.71367e − 23 0.0000e − 25 12 0.0000 59.9263 4.33531e − 05 0.0005e − 04 KD-FKDE 3 0.0011 14.5445 1.32977e − 10 0.0001e − 10 6 0.0000 28.1622 9.64769e − 18 0.0014e − 18 9 0.0001 47.5693 1.71367e − 23 0.0000e − 25 12 0.0000 59.9263 4.33531e − 05 0.2618e − 04 IFGT-FKDE 3 0.6040 18.1807 1.32977e − 10 0.5389e − 10 6 0.4053 35.2028 9.64769e − 18 0.4183e − 18 9 0.0434 59.4616 1.71367e − 23 0.132e − 25 12 0.0007703 74.9079 Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Computationally efficient Texture synthesis algorithm with FKDE algorithms Problems how to include the effect of Wq (the temperature field) within the PDDP-based tree 1 structure for the implementation of FKDE, and there are two joint densities corresponding to {Yq , Xq } and Xq ; therefore, it 2 requires two FKDE structure, which is not computationally efficient. Inclusion of Wq • Starting State: {Wq,i = 0} ⇒ P(Xq ; Wq ) = constant ⇒ P(Xq ) is uniform ⇒ Each Xs has equal effect upon Xq ⇒ Every Xs should be considered in the KDE ⇒ Rnew is very large Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Computationally efficient Texture synthesis algorithm with FKDE algorithms Problems how to include the effect of Wq (the temperature field) within the PDDP-based tree 1 structure for the implementation of FKDE, and there are two joint densities corresponding to {Yq , Xq } and Xq ; therefore, it 2 requires two FKDE structure, which is not computationally efficient. Inclusion of Wq • Starting State: {Wq,i = 0} ⇒ P(Xq ; Wq ) = constant ⇒ P(Xq ) is uniform ⇒ Each Xs has equal effect upon Xq ⇒ Every Xs should be considered in the KDE ⇒ Rnew is very large • Ending State: {Wq,i = 1} ⇒ P(Xq ; Wq ) = P(Xq ) ⇒ Rnew = R Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Computationally efficient Texture synthesis algorithm with FKDE algorithms Problems how to include the effect of Wq (the temperature field) within the PDDP-based tree 1 structure for the implementation of FKDE, and there are two joint densities corresponding to {Yq , Xq } and Xq ; therefore, it 2 requires two FKDE structure, which is not computationally efficient. Inclusion of Wq • Starting State: {Wq,i = 0} ⇒ P(Xq ; Wq ) = constant R = Rnew ⇒ P(Xq ) is uniform cq ⇒ Each Xs has equal effect upon Xq abs(vn − mr ) ≤ Rnew ⇒ Every Xs should be considered in the KDE R ⇒ Rnew is very large ⇒ abs(vn − mr ) ≤ cq • Ending State: ⇒ abs(vn − mr )cq ≤ R {Wq,i = 1} ⇒ P(Xq ; Wq ) = P(Xq ) ⇒ Rnew = R Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Results with comparisons Original NNMRF kd−tree IFGT Proposed D102 D49 D20 Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Results with comparisons Original kd−tree NNMRF IFGT Proposed D53 D104 D4 D82 Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Results with comparisons Original NNMRF IFGT kd−tree Proposed D110 D60 D93 D97 Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Results with comparisons Table: Time taken in texture synthesis: input texture size 128 × 128 and output texture size 256 × 256 NNMRF C-FKDE KD-FKDE PDDP-FKDE hours 8 5 8 6 minutes 7 55 34 0 seconds 39 12 56 41 Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Maximum Log-Pseudo-likelihood LPL = log[P(Ys |Xs )] s∈Sin For 1st order neighborhood system For 2nd order neighborhood system Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis How to estimate MLPL ? • parametric MRF model • non-parametric MRF model: what should be the kernel ? • Gaussian kernel: as used in [Paget and Longstaff(1998)] • Dirac-delta kernel • Some other solution Effect of kernel upon the MLPL estimate nd LPL is getting saturated before 2 order LPL is not saturating rather it is increasing 0 −10000 −50 −20000 −100 −30000 −150 −40000 LPL −200 LPL −250 −50000 D102: Near regular −300 D104: Near regular −60000 −350 D110: Stochastic −70000 −400 D60: Stochastic D93: Stochastic −450 −80000 0 1 2 3 4 0 5 10 15 20 25 30 35 40 Order Order Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Why does the original LPL measure, not saturate ? Why ? Kh (Ys −Yq )Kh (Xs −Xq ) s∈S • original LCPDF: p(Yq |Xq ) = q∈S Kh (Xs −Xq ) • Changing terms with order: • hy = σy N −1/(d+4) : changes due to change in d and N, with order √ • In case of LCPDF the normalizing term becomes: 2πhy ; • Moreover, hy also affect the argument within the exponential term. • One can not neglect this term. Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis A new deﬁnition for LCPDF δ(Ys − Yq )Kh (Xs − Xq ) q∈S p(Ys |Xs ) = Kh (Xs − Xq ) q∈S Two reasons in the support for this new deﬁnition • From the texture synthesis algorithm point of view • From a numerical point of view 0.018 0.016 Probability calculated with Gaussian kernel D104 Near Regular Texture 0.014 D110 Stochastic Texture 0.012 0.01 probaility = 0.003544 0.008 0.006 Probability = 0.0002963 0.004 0.002 0 0 50 100 150 200 250 300 Gray Levels Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Results: D104 D104 8 2 6 4 10 14 12 16 22 24 18 20 Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Results: D9 D9 5 1 3 7 9 11 13 15 19 17 21 23 Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Results for Near-regular textures Original Original NNMRF Our Synthesis Algorithm NNMRF Our Synthesis Algorithm D104 D20 o = 12 o = 18 D22 D34 o = 17 0 = 13 Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Results for Stochastic textures Our Synthesis Algorithm Original NNMRF Original NNMRF Our Synthesis Algorithm D4 D9 O=9 O = 10 D93 D97 O = 16 O=9 Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Results for Some other textures Original NNMRF Our Synthesis Algorithm Original NNMRF Our Synthesis Algorithm D53 D55 O = 14 O = 17 D82 D80 O = 14 O = 12 Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Problem Deﬁnition Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Problem Deﬁnition Texture synthesis HOW ? Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Motivation Applications of Inverse Texture Synthesis • Understanding of Textures • Content-based image/video retrieval • Perceptual Image/Video compression • Computer Vision Tasks • Perceptual understanding of textures within the image • Creation of animation – Collecting information from natural images/sequences • Perceptual Understanding of temporal texture – such as, dance sequence, walk sequence, music sequence etc. Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Deﬁnition of Objective Functions According to N-MRF model • Distance between two LCPDF’s evaluated w.r.t. both input and output texture patches. Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Deﬁnition of Objective Functions According to N-MRF model • Distance between two LCPDF’s evaluated w.r.t. both input and output texture patches. • What distance function ? Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Deﬁnition of Objective Functions According to N-MRF model • Distance between two LCPDF’s evaluated w.r.t. both input and output texture patches. • What distance function ? • Computationally expensive Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Deﬁnition of Objective Functions According to N-MRF model • Distance between two LCPDF’s evaluated w.r.t. both input and output texture patches. • What distance function ? • Computationally expensive According to N-MRF model: Intuitively • M×N • Size of the output patch 1 min{||Xs − Xq ||2 , where • |S | • Do the input neighborhood vectors s∈Sin in exist within output patch ? q ∈ Sout } Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Problem with these two objective functions Scaled up versions of solutions A B approximately same Neighborhood Deformation/variation within \"A\" difficult to find within \"B\" Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Three objectives • Say Sout = {s ∈ Sin , s.t., (si − i)2 ≤ M 2 and (sj − j)2 ≤ N 2 } {i,j,M,N} = S − Sout • Define S in • First objective finds neighborhood from input texture within the output texture • Second objective finds neighborhood from output texture within the input texture, excluding the part of Sout 1 min{||Xs − Xq ||2 ; q ∈ Sout } = F1 |Sin | s∈Sin 1 {i,j,M,N} min{||Xq − Xs ||2 ; s ∈ Sin = } F2 |Sout | q∈Sout = M×N F3 Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Multi-objective Framework    f1 (x)      f2 (x)   inequality constraints: gj (x) ≥ 0, j = 1, 2, ..., J       such that   minx  .    equality constraints: hk (x) = 0, k = 1, 2, ..., K .     .     solution space: xiL ≤xi ≤ xiU , i = 1, 2, ..., N fm (x) Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Multi-objective Framework    f1 (x)      f2 (x)   inequality constraints: gj (x) ≥ 0, j = 1, 2, ..., J       such that   minx  .    equality constraints: hk (x) = 0, k = 1, 2, ..., K .     .     solution space: xiL ≤xi ≤ xiU , i = 1, 2, ..., N fm (x) Domination A vector x ∈ RN is said to dominate y ∈ RN if both the conditions stated below hold true: fi (x) ≤ fi (y), ∀i ∈ [1 . . . m] ∃ j ∈ [1 . . . m], such that, fj (x) < fj (y) Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Pareto-optimal Front Best in F 1 2nd objective function F 2 Worst in F 2 All are optimal solutions Best in F 2 Worst in F 1 1st objective function F1 Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Genetic Algorithm Why not classical optimization algorithms 1 min{||Xs − Zt ||2 ; t ∈ Sout } = F1 |Sin | s∈Sin 1 {i,j,M,N} min{||Zt − Xs ||2 ; s ∈ Sin = } F2 |Sout | t∈Sout = M×N F3 Genetic Algorithm • It is an intuitive algorithm, biologically inspired, • Based upon the phylosophy of survival of the ﬁttest • Crossover, mutation operators • There are many algorithms, we choose NSGA [Srinivas and Deb(1994)] Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Redundancy of objectives F1 F2 F3 + − − F1 + + − F2 + + − F3 Table: Conflict matrix: In each case of Fabric.0014, D22, Fabric.0009, and Grass textures the same trait has been observed. 300 900 300 250 850 250 200 800 200 F2 F1 150 750 150 F2 700 100 100 650 50 50 600 0 0 0 2000 4000 6000 8000 0 2000 4000 6000 8000 600 700 800 900 F1 F3 F3 (a) F1 conflicts F2 (b) non-conflicting F2 and F3 (c) F1 conflicts F3 Figure: Two-dimensional views of Pareto-optimal front for the Fabric.0014 texture: the positivity or negativity of the cross correlation between the objective functions can be understood Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Results: D22 400 Synthesis Result 1 Synthesis Extracted Result 300 Original Textured Exempler Region F2 200 Extracted 2 Exempler 100 3 5 4 0 2500 3000 F1 3500 4000 Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Results: Fabric.0009 3000 Original Textured Region 2500 1st solution 2000 F2 1500 2nd 1000 3rd 500 1900 1950 2000 2050 2100 2150 2200 2250 F1 Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Order Estimation from Fourier Domain Outline Reduction of Computational complexity Ph.D. Research Work Order Estimation : Revisited Conclusion and Possible Future Directions Inverse Texture Synthesis Results: Grass Synthesized textures from the extracted solutions 200 180 1st solution 160 Original Textured Region 140 120 F2 100 80 60 2nd solution 40 20 0 260 280 300 320 340 360 380 400 420 3rd solution F1 Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Outline Ph.D. Research Work Conclusion and Possible Future Directions Conclusion • The problem of Order estimation • The problem of computational complexity reduction • With the incorporation of Dimensionality Reduction methodology, e.g., PCA • With fast estimation of Kernel Density Estimation with an improvised data structure • An inverse application of texture synthesis with NMRF model • Objective functions • Analysis of objective functions • Multi-objective framework Possible Future Directions • Order estimation for in-homogeneous textures or globally varying textures • three-dimansional variation of surface • structural variation • time speciﬁc variation • How to incorporate a control ﬁeld within the texture analysis • How to choose a particular solution from the multi-objective framework, depending upon the application in hand Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Outline Ph.D. Research Work Conclusion and Possible Future Directions D L Boley. Principal direction divisive partitioning. Data Mining and Knowledge Discovery, 2(4):325–344, 1998. Alexander G. Gray and Andrew W. Moore. Nonparametric density estimation: toward computational tractability. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(8): 1344–1348, January 2003. R. Paget and I. D. Longstaff. Texture synthesis via a noncausal nonparametric multiscale markov random ﬁeld. IEEE Transactions on Image Processing, 7(6):925–931, June 1998. David W Scott. Multivariate density estimation - theory, practice and visualization. Wiley interscience, 1992. N. Srinivas and Kalyanmoy Deb. Multiobjective optimization using nondominated sorting in genetic algorithms. Evolutionary Computation, 2:221–248, 1994. Changjiang Yang, Ramani Duraiswami, Nail A Gumerov, and Larry Davis. Improved fast gauss transform and efﬁcient kernel density estimation. In Proceedings. Ninth IEEE International Conference on Computer Vision, 2003. Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Outline Ph.D. Research Work Conclusion and Possible Future Directions Kai Zhang, Ming Tang, and James T Kwok. Applying neighborhood consistency for fast clustering and kernel density estimation. In Proceedings of the 2005 Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), volume 2, 2005. Arnab Sinha arnab@iitk.ac.in Fast NMRF based texture synthesis algorithms

Computationally Efficient NMRF model based Texture Synthesis

0 comments

Notes on slide 1

Favorites, Groups & Events