Your SlideShare is downloading. ×
0
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
NIPS2008: tutorial: statistical models of visual images
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

NIPS2008: tutorial: statistical models of visual images

403

Published on

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
403
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
8
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Statistical Image Models Eero Simoncelli Howard Hughes Medical Institute, Center for Neural Science, andCourant Institute of Mathematical Sciences New York University
  • 2. Photographic ImagesDiverse specialized structures:• edges/lines/contours• shadows/highlights• smooth regions• textured regions
  • 3. Photographic Images Diverse specialized structures: • edges/lines/contours • shadows/highlights • smooth regions • textured regionsOccupy a small region of the full space
  • 4. spa ce o f all ima ges typical imagesOne could describe this set as adeterministic manifold....
  • 5. • Step edges are rare (lighting, junctions, texture, noise)
  • 6. • Step edges are rare (lighting, junctions, texture, noise)• One scale’s texture is another scale’s edge
  • 7. • Step edges are rare (lighting, junctions, texture, noise)• One scale’s texture is another scale’s edge• Need seamless transitions from isolated features to dense textures
  • 8. spa ce o f all ima ges typical imagesOne could describe this set as adeterministic manifold....
  • 9. spa ce o f all ima ges typical imagesOne could describe this set as adeterministic manifold....But seems more natural to use probability
  • 10. spa ce o f all ima ges typical imagesOne could describe this set as a P(x)deterministic manifold....But seems more natural to use probability
  • 11. “Applications”• Engineering: compression, denoising, restoration, enhancement/modification, synthesis, manipulation [Hubel ‘95]
  • 12. “Applications”• Engineering: compression, denoising, restoration, enhancement/modification, synthesis, manipulation• Science: optimality principles for neurobiology (evolution, development, learning, adaptation) [Hubel ‘95]
  • 13. Density modelsnonparametric parametric/ constrained
  • 14. Density modelsnonparametric parametric/ constrainedbuild a histogramfrom lots ofobservations...
  • 15. Density modelsnonparametric parametric/ constrainedbuild a histogram use “natural constraints”from lots of (geometry/photometryobservations... of image formation, computation, maxEnt)
  • 16. Density modelsnonparametric parametric/ historical trend constrained (technology driven)build a histogram use “natural constraints”from lots of (geometry/photometryobservations... of image formation, computation, maxEnt)
  • 17. histogramOriginal imageRange: [0, 237]Dims: [256, 256] 0 50 100 150 200 250
  • 18. histogramOriginal image Range: [0, 237] Dims: [256, 256] 0 50 100 150 200 250 histogramEqualized imageRange: [1.99, 238] Dims: [256, 256] 0 50 100 150 200 250
  • 19. histogramOriginal image Range: [0, 237] Dims: [256, 256] 0 50 100 150 200 250 histogramEqualized imageRange: [1.99, 238] Dims: [256, 256] 0 50 100 150 200 250
  • 20. General methodologyObserve “interesting” Transform to Joint Statistics Optimal Representation
  • 21. General methodologyObserve “interesting” Transform to Joint Statistics Optimal Representation
  • 22. General methodologyObserve “interesting” Transform to Joint Statistics Optimal Representation“Onion peeling”
  • 23. Evolution of image modelsI. (1950’s): Fourier + GaussianII. (mid 80’s - late 90’s): Wavelets + kurtotic marginalsIII. (mid 90’s - present): Wavelets + local context • local amplitude (contrast) • local orientationIV. (last 5 years): Hierarchical models
  • 24. a. Pixel correlation b. 1 Correlation I(x+2,y) I(x+4,y)I(x+1,y) I(x,y) I(x,y) I(x,y) 0 10 Spatia
  • 25. a. Pixel correlation b. 1 Correlation I(x+2,y) I(x+4,y) I(x+1,y) I(x,y) I(x,y) I(x,y) 0 10 b. Spatia 1 CorrelationI(x+4,y) I(x,y) 0 10 20 30 40 Spatial separation (pixels)
  • 26. Translation invarianceAssuming translation invariance,
  • 27. Translation invarianceAssuming translation invariance, => covariance matrix is Toeplitz (convolutional)
  • 28. Translation invarianceAssuming translation invariance, => covariance matrix is Toeplitz (convolutional) => eigenvectors are sinusoids
  • 29. Translation invarianceAssuming translation invariance, => covariance matrix is Toeplitz (convolutional) => eigenvectors are sinusoids => can diagonalize (decorrelate) with F.T.
  • 30. Translation invarianceAssuming translation invariance, => covariance matrix is Toeplitz (convolutional) => eigenvectors are sinusoids => can diagonalize (decorrelate) with F.T.Power spectrum captures full covariance structure
  • 31. Spectral powerStructural:Assume scale-invariance: F (sω) = s F (ω) p then: 1 F (ω) ∝ p ω [Ritterman 52; DeRiugin 56; Field 87; Tolhurst 92; Ruderman/Bialek 94; ...]
  • 32. Spectral powerStructural: Empirical: 6Assume scale-invariance: 5 F (sω) = s F (ω) p 4 Log power 3 10 then: 2 1 F (ω) ∝ p ω 1 0 0 1 2 3 Log spatialfrequency (cycles/image) 10 [Ritterman 52; DeRiugin 56; Field 87; Tolhurst 92; Ruderman/Bialek 94; ...]
  • 33. Principal Components Analysis (PCA) + whitening a. b. c.20 20 4-20 -20 -4 -20 20 -20 20 -4 4
  • 34. PCA basis for image blocks
  • 35. PCA basis for image blocks PCA is not unique
  • 36. Maximum entropy (maxEnt) The density with maximal entropy satisfying E (f (x)) = c is of the form pME (x) ∝ exp (−λf (x)) where λ depends on cExamples: f (x) = x 2 f (x) = |x|
  • 37. Model I (Fourier/Gaussian) Coefficient density: Basis set: Image: : : : :
  • 38. Gaussian model is weak ω −2 −1 1/f2 F F -1P(x) P(c)
  • 39. Gaussian model is weak ω −2 −1 1/f2 F F -1P(x) P(c) a. b. F 2 −1 ω F
  • 40. Gaussian model is weak ω −2 −1 1/f2 F F -1 P(x) P(c) a. b. F 2 −1 ω F a. b. c.20 20 4-20 -20 -4 -20 20 -20 20 -4 4
  • 41. Bandpass Filter Responses 0 10 Response histogram Gaussian density Probability -2 10 -4 10 500 0 500 Filter Response [Burt&Adelson 82; Field 87; Mallat 89; Daugman 89, ...]
  • 42. “Independent” Components Analysis (ICA) a. b. c. d.20 20 4 4-20 -20 -4 -4 -20 20 -20 20 -4 4 -4 4 For Linearly Transformed Factorial (LTF) sources: guaranteed independence (with some minor caveats) [Comon 94; Cardoso 96; Bell/Sejnowski 97; ...]
  • 43. ICA on image blocks [Olshausen/Field ’96; Bell/Sejnowski ’97] [example obtained with FastICA, Hyvarinen]
  • 44. Marginal densities log(Probability) log(Probability) log(Probability) log(Probability) p = 0.46 p = 0.58 p = 0.48 !H/H = 0.0031 !H/H = 0.0011 !H/H = 0.0014 Wavelet coefficient value Wavelet coefficient value Wavelet coefficient value Fig. 4. Log histograms of a single wavelet subband of four example images (see Fig. 1 for image histogram, tails are truncated so as to show 99.8% of the distribution. Also shown (dashed lines) are corresponding to equation (3). Text indicates the maximum-likelihood value of p used for the fitte Well-fit by a generalized Gaussian: the relative entropy (Kullback-Leibler divergence) of the model and histogram, as a fraction of th histogram. P (x) ∝ exp −|x/s| pnon-Gaussian than others. By the mid 1990s, a numberof authors had developed methods of optimizing a ba-sis of filters in order to to maximize the non-Gaussianityof the responses [e.g., 36, 4]. Often these methods oper- [Mallat 89; Simoncelli&Adelson 96; Moulin&Liu 99; ...]ate by optimizing a higher-order statistic such as kurto-
  • 45. Kurtosis vs. bandwidth 16 14 12 Sample Kurtosis 10 8 6 4 0 0.5 1 1.5 2 2.5 3 Filter Bandwidth (octaves)Note: Bandwidth matters much more than orientation[see Bethge 06] [after Field 87]
  • 46. Octave-bandwidth representations SpatialFrequencySelectivity: Filter:
  • 47. Model II (LTF)Coefficient density: Basis set: Image: : : :
  • 48. LTF also a weak model... Sample Gaussianized Sample ICA-transformed and Gaussianized
  • 49. Trouble in paradise
  • 50. Trouble in paradise• Biology: Visual system uses a cascade - Where’s the retina? The LGN? - What happens after V1? Why don’t responses get sparser? [Baddeley etal 97; Chechik etal 06]
  • 51. Trouble in paradise• Biology: Visual system uses a cascade - Where’s the retina? The LGN? - What happens after V1? Why don’t responses get sparser? [Baddeley etal 97; Chechik etal 06]• Statistics: Images don’t obey ICA source model - Any bandpass filter gives sparse marginals [Baddeley 96] => Shallow optimum [Bethge 06; Lyu & Simoncelli 08] - The responses of ICA filters are highly dependent [Wegmann & Zetzsche 90, Simoncelli 97]
  • 52. Conditional densities 1 1 0.6 0.6 0.2 0.2 -40 0 40 50 -40 0 40 40 0 -40 -40 0 40Linear responses are not independent, even for optimized filters! [Simoncelli 97; Schwartz&Simoncelli 01]CSH-02
  • 53. [Schwartz&Simoncelli 01]
  • 54. • Large-magnitude subband coefficients are found at neighboring positions, orientations, and scales.
  • 55. Modeling heteroscedasticity (i.e., variable variance) Method 1: Conditional GaussianP (xn |{xk }) ∼ N 0; wnk |xk | + σ 2 2 k [Simoncelli 97; Buccigrossi&Simoncelli 99; see also ARCH models in econometrics!]
  • 56. Joint densities adjacent near far other scale other ori 150 150 150 150 150 100 100 100 100 100 50 50 50 50 50 0 0 0 0 0!50 !50 !50 !50 !50!100 !100 !100 !100 !100!150 !150 !150 !150 !150 !100 0 100 !100 0 100 !100 0 100 !500 0 500 !100 0 100 150 150 150 150 150 100 100 100 100 100 50 50 50 50 50 0 0 0 0 0!50 !50 !50 !50 !50!100 !100 !100 !100 !100!150 !150 !150 !150 !150 !100 0 100 !100 0 100 !100 0 100 !500 0 500 !100 0 100 • Nearby: densities are approximately circular/ellipticalFig. 8. Empirical joint distributions of wavelet coefficients associated with different pairs of basis functions, for a singleimage of a New York City street scene (see Fig. 1 for image description). The top row shows joint distributions as contourplots, with lines drawn at equal intervals of log probability. The three leftmost examples correspond to pairs of basis func- • Distant: densities are approximately factorialtions at the same scale and orientation, but separated by different spatial offsets. The next corresponds to a pair at adjacentscales (but the same orientation, and nearly the same position), and the rightmost corresponds to a pair at orthogonal orien-tations (but the same scale and nearly the same position). The bottom row shows corresponding conditional distributions:brightness corresponds to frequency of occurance, except that each column has been independently rescaled to fill the fullrange of intensities. [Simoncelli, ‘97; Wainwright&Simoncelli, ‘99]
  • 57. ICA-transformed joint densities d=2 d=16 d=32 12 12 12 10 10 10kurtosis 8 8 8 6 6 6 4 4 4 0 !/4 !/2 3!/4 ! 0 !/4 !/2 3!/4 ! 0 !/4 !/2 3!/4 ! orientation data (ICA’d): sphericalized: factorialized:
  • 58. ICA-transformed joint densities d=2 d=16 d=32 12 12 12 10 10 10kurtosis • Local densities are elliptical (but non-Gaussian) 8 8 8 6 6 6 • Distant densities are factorial 4 4 4 0 !/4 !/2 3!/4 ! 0 !/4 !/2 3!/4 ! 0 !/4 !/2 3!/4 ! orientation data (ICA’d): [Wegmann&Zetzsche ‘90; Simoncelli ’97; + many recent models] sphericalized: factorialized:
  • 59. Spherical vs LTF 0.2 blk blk 0.4 blk blk size = 3x3 0.2 blk size = 7x7 blk size = 11x11 spherical spherical spherical factorial factorial 0.35 factorial 0.15 0.3 0.15 0.25 0.1 0.2 0.1 0.15 0.05 0.1 0.05 0.05 0 0 0 3 6 9 12 15 18 20 3 6 9 12 15 18 20 3 6 9 12 15 18 20 kurtosis kurtosis kurtosis 3x3 7x7 15x15 data (ICA’d): sphericalized: factorialized:• Histograms, kurtosis of projections of image blocks onto randomunit-norm basis functions.• These imply data are closer to spherical than factorial [Lyu & Simoncelli 08]
  • 60. non-Gaussian elliptical observationsand models of natural images: - Zetzsche & Krieger, 1999; - Huang & Mumford, 1999; - Wainwright & Simoncelli, 2000; - Hyvärinen and Hoyer, 2000; - Parra et al., 2001; - Srivastava et al., 2002; - Sendur & Selesnick, 2002; - Teh et al., 2003; - Gehler and Welling, 2006 - Lyu & Simoncelli, 2008 - etc.
  • 61. Modeling heteroscedasticityMethod 2: Hidden scaling variable for each patchGaussian scale mixture (GSM)[Andrews & Mallows 74]: √ x= zu• u is Gaussian, z > 0• z and u are independent• x is elliptically symmetric, with covariance ∝ Cu• marginals of x are leptokurtotic [Wainwright&Simoncelli 99]
  • 62. GSM - prior on z• Empirically, z is approximately lognormal [Portilla etal, icip-01] exp (−(log z − µl )2 /(2σl )) 2 pz (z) = 2 )1/2 z(2πσl• Alternatively, can use Jeffrey’s noninformative prior [Figueiredo&Nowak, ‘01; Portilla etal, ‘03] pz (z) ∝ 1/z
  • 63. GSM simulation Image data GSM simulation ! !#" #" " "#" #" !!" " !" !!" " !" [Wainwright & Simoncelli, NIPS*99]
  • 64. Model III (GSM)Coefficient density: Basis set: Image: X X X Xsqrt(z) X X u
  • 65. √ Original coefficients Normalized by z !2 !4 !4 !5marginal Log probability Log probability !6 !6 !7 [Ruderman&Bialek 94] !8 !8 !9 !10 !500 0 500 !10 !5 0 5 100 8 50 6 joint 0 4 [Schwartz&Simoncelli 01] !50 2 !100 0 !100 !50 0 50 100 0 2 4 6 8subband
  • 66. 6Model Encoding Cost (bits/coeff) Model Encoding cost (bits/coeff) 5.5 5 5 4.5 4 4 3 3.5 3 2 Gaussian Model First Order Ideal 2.5 Generalized Laplacian Conditional Model 1 3 4 5 1 2 3 4 5 6 Empirical First Order Entropy (bits/coeff) Empirical Conditional Entropy [Buccigrossi & Simoncelli 99]
  • 67. Bayesian denoising• Additive Gaussian noise: y =x+w 2 2 P (y|x) ∝ exp[−(y − x) /2σw ]• Bayes’ least squares solution is conditional mean: x(y) = IE(x|y) ˆ = dxP(y|x)P(x)x/P(y)
  • 68. I. ClassicalIf signal is Gaussian, BLS estimator is linear: denoised (ˆ) x 2 σxx(y) =ˆ 2 2 ·y σx + σn=> suppress fine scales, noisy (y) retain coarse scales
  • 69. Non-Gaussian coefficients " #" -*./01.*,6.)07+48 94:..41,;*1.)5,, 2+0343()5 !% #" !$ #" !!"" " !"" &()*+,-*./01.* [Burt&Adelson ‘81; Field ‘87; Mallat ‘89; Daugman ‘89; etc]
  • 70. II. BLS for non-Gaussian prior• Assume marginal distribution [Mallat ‘89]: P (x) ∝ exp −|x/s| p• Then Bayes estimator is generally nonlinear: p = 2.0 p = 1.0 p = 0.5 [Simoncelli & Adelson, ‘96]
  • 71. MAP shrinkagep=2.0 p=1.0 p=0.5 [Simoncelli 99]
  • 72. Denoising: Joint IE(x|y) = dz P(z|y) IE(x|y, z) −1y    = dz P(z|y)  zCu(zCu + Cw ) ctrwhere P(y|z) P(z) exp(−y T (zCu + Cw )−1y/2)P(z|y) = , P(y|z) = Py (2π)N |zCu + Cw |Numerical computation of solution is reasonably efficient ifone jointly diagonalizes Cu and Cw ...[Portilla, Strela, Wainwright, Simoncelli, ’03]IPAM, 9/04 20
  • 73. ESTIMATED COEFF. Example estimators !" +1&2/1+3)*%+,,- " !w !!" #" !" " " NOISY COEFF. !#" $%&()./0+$1 !!" $%&()*%+,,-Estimators for the scalar and single-neighbor cases [Portilla etal 03]
  • 74. Comparison to other methods "& "& " " ,456748+91:;<=/456,(74-.)/-0123 !"& !"& :>6965#8*>?6< !! !! !!& !!& !# !# !#& !#& )89!:1:;<=>? !$ .=9@A?-9?=@BC8D !$ !$& !$& !" #" $" %" &" !" #" $" %" &" ()*+,-*.)/-0123 ()*+,-*.)/-0123 Results averaged over 3 images [Portilla etal 03]
  • 75. NoisyOriginal (22.1 dB)Matlab’s BLS-GSMwiener2 (30.5 dB)(28 dB)
  • 76. Noisy Original (8.1 dB)UndWvlt BLS-GSM Thresh (21.2 dB)(19.0 dB)
  • 77. Real sensor noise400 ISO denoised
  • 78. GSM summary• GSM captures local variance• Underlying Gaussian leads to simple computation• Excellent denoising results• What’s missing? • Global model of z variables [Wainwright etal 99; Romberg etal ‘99; Hyvarinen/Hoyer ‘02; Karklin/ Lewicki ‘02; Lyu/Simoncelli 08] • Explicit geometry: phase and orientation
  • 79. Global models for z• Non-overlapping neighborhoods, tree-structured z [Wainwright etal 99; Romberg etal ’99] z u Coarse scale Fine scale• Field of GSMs: z is an exponentiated GMRF, u is a GMRF, subband is the product [Lyu&Simoncelli 08]
  • 80. D MACHINE INTELLIGENCE, VOL. X, NO. X, XX 200X 9 State-of-the-art denoising Lena Boats " " # # !" !" ∆()*+, ∆()*+, !$ !$ ! ! !& !&"## ! "#"! $! !# %! "## ! "#"! $! !# %! "## σ σ FoGSM BM3D kSVDthods for three different images. Plotted are differences in PSNR for different input noise levels (σ) between BLS-GSM [17], kSVD [39] and FoE [27]). The PSNR values for these methods were taken from GSM FoE [Lyu&Simoncelli, PAMI 08]
  • 81. Measuring Orientation2-band steerable pyramid: Image decomposition interms of multi-scale gradient measurements [Simoncelli et.al., 1992; Simoncelli & Freeman 1995]
  • 82. Multi-scale gradient basis
  • 83. Multi-scale gradient basis• Multi-scale bases: efficient representation
  • 84. Multi-scale gradient basis• Multi-scale bases: efficient representation• Derivatives: good for analysis • Local Taylor expansion of image structures • Explicit geometry (orientation)
  • 85. Multi-scale gradient basis• Multi-scale bases: efficient representation• Derivatives: good for analysis • Local Taylor expansion of image structures • Explicit geometry (orientation)• Combination: • Explicit incorporation of geometry in basis • Bridge between PDE / harmonic analysis approaches
  • 86. orientation magnitude orientation[Hammond&Simoncelli 06; cf. Oppenheim and Lim 81]
  • 87. Importance of local orientation Randomized orientation Randomized magnitude [Hammond&Simoncelli 05]
  • 88. Reconstruction from orientation Original Quantized to 2 bits • Reconstruction by projections onto convex sets • Resilient to quantization [Hammond&Simoncelli 06]
  • 89. Image patches related by rotation two-band steerable[Hammond&Simoncelli 06] pyramid coefficients
  • 90. raw rotatedpatches patches PCA of normalized gradient patches --- Raw Patches Rotated Patches [Hammond&Simoncelli 06]
  • 91. Orientation-Adaptive GSM modelModel a vectorized patch of wavelet coefficients as: patch rotation operator hidden magnitude/orientation variables [Hammond&Simoncelli 06]
  • 92. Orientation-Adaptive GSM modelModel a vectorized patch of wavelet coefficients as: patch rotation operator hidden magnitude/orientation variablesConditioned on ; is zero mean gaussian with covariance [Hammond&Simoncelli 06]
  • 93. Estimation of C(θ) from noisy data noisy patch unknown, approximate by measured from noisy data.Assuming independent and noise rotationally invariant (assuming w.l.o.g. E[z] =1 ) [Hammond&Simoncelli 06]
  • 94. Bayesian MMSE Estimator [Hammond&Simoncelli 06]
  • 95. Bayesian MMSE Estimator condition on and integrate over hidden variables [Hammond&Simoncelli 06]
  • 96. Bayesian MMSE Estimator condition on and integrate over hidden variables [Hammond&Simoncelli 06]
  • 97. Bayesian MMSE Estimator condition on and integrate over hidden variables Wiener estimate [Hammond&Simoncelli 06]
  • 98. Bayesian MMSE Estimator condition on and integrate over hidden variables Wiener estimate has covarianceseparable prior for hidden variables [Hammond&Simoncelli 06]
  • 99. Bayesian MMSE Estimator condition on and integrate over hidden variables Wiener estimate has covarianceseparable prior for hidden variables [Hammond&Simoncelli 06]
  • 100. σ = 40 noisy 2.81 dB gsm2 oagsm12.4 dB 13.1 dB
  • 101. Locally adaptive covariance • Karklin & Lewicki 08: Each patch is Gaussian, with covariance constructed from a weighted outer- product of fixed vectors: p(x) = G (x; C(y)) log C(y) = yn Bn n Tp(y) = exp(−|yn |) Bn = wnk bk bk n k • Guerrero-Colon, Simoncelli & Portilla 08: Each patch is a mixture of GSMs (MGSMs): p(x) = Pk p(zk ) G(x; zk Ck ) dzk k
  • 102. MGSMs generative model √ √ √Patch x chosen from { z1 u1 , z2 u2 , ... zK uK } with probabilities {P1 , P2 , ..., PK }Parameters: • Covariances Ck • Scale densities pk (zk ) • Component probabilities Pk • Number of components KParameters can be fit to data of one or more imagesby maximizing likelihood (EM-like) [Guerrero-Colon, Simoncelli, Portilla 08]
  • 103. MGSM “segmentation” image 1 2 4 First sixeigenvectors of GSM covariance matrices [Guerrero-Colon, Simoncelli, Portilla 08]
  • 104. MGSM“segmentation”Eigenvectors of GSMcomponents represent invariant subspaces:“generalized complex cells”
  • 105. Potential of local homogeneous models?Consider an implicit model: maxEnt subject to constraints on subband coefficients: • marginal statistics [var,skew,kurtosis] • local raw correlations • local variance correlations • local phase correlations [Portilla & Simoncelli 00; cf. Zhu, Wu & Mumford 97]
  • 106. Visual texture
  • 107. Visual textureHomogeneous, with repeated structures
  • 108. Visual textureHomogeneous, with repeated structures “You know it when you see it”
  • 109. All Images Texture Images Equivalence class (visually indistinguishable)
  • 110. Iterative synthesis algorithm AnalysisExample Transform Measure Texture Statistics Synthesis Measure StatisticsRandom Transform Inverse Synthesized Adjust Seed Transform Texture [Portilla&Simoncelli 00; cf. Heeger&Bergen ‘95]
  • 111. Examples: Artificial
  • 112. Photographic, quasi-periodic
  • 113. Photographic, aperiodic
  • 114. Photographic, structured
  • 115. Photographic, color
  • 116. Non-textures?
  • 117. Texture mixtures
  • 118. Texture mixturesConvex combinations in parameter space
  • 119. Texture mixturesConvex combinations in parameter space=> Parameter space includes non-textures
  • 120. Summary• Fusion of empirical data with structural principles• Statistical models have led to state-of-the-art image processing, and are relevant for biological vision• Local adaptation to {variance, orientation, phase, ...} gives improvement, but makes learning harder• Cascaded representations emerge naturally• There’s still much room for improvement!
  • 121. Cast• Local GSM model: Martin Wainwright, Javier Portilla• GSM Denoising: Javier Portilla, Martin Wainwright, Vasily Strela• Variance-adaptive compression: Robert Buccigrossi• Local orientation and OAGSM: David Hammond• Field of GSMs: Siwei Lyu• Mixture of GSMs: Jose-Antonio Guerrero-Colón, Javier Portilla• Texture representation/synthesis: Javier Portilla

×