Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 21, NO. 1, JANUARY 2012                                                       ...
380                                                                               IEEE TRANSACTIONS ON IMAGE PROCESSING, V...
[1] s;t = j .erty, different informative distributions (for different parameter positivevalues of p and c) can be found or...
[2] s;t values. In the same way, this model can be easilytion (or a normalized edge histogram possibly estimated from a ta...
[l] s;t = j .in the first nearest eight neighbors of s) is considered as an indepen-                                       ...
[1] s;t 0 (Is 0 It )2          (1) images is straightforward as follows.                        I                         ...
[1] s;t are the values                                                                            have integer luminance v...
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 21, NO. 1, JANUARY 2012                                                       ...
Upcoming SlideShare
Loading in …5

An energy based model for the image edge-histogram specification problem.bak


Published on

  • Be the first to comment

  • Be the first to like this

An energy based model for the image edge-histogram specification problem.bak

  1. 1. IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 21, NO. 1, JANUARY 2012 379Correspondence An Energy-Based Model for the Image Edge-Histogram and these latter cannot be separated (they can only be merged together) Specification Problem in order to approximate the different bins of a uniform histogram [2]. It has been finally realized that the key to achieve a discrete exact his- Max Mignotte togram specification method was to find a strict ordering relation sepa- rating each pixel of the original image with the same intensity into sev- eral subsets (in order to approximate the different bins of the desired Abstract—In this correspondence, we present an original energy-based output distribution). Practically speaking, let I be a discrete image with L gray levels and N 2M pixels I (xi ; yj ) with coordinates (xi ; yj ) rep-model that achieves the edge-histogram specification of a real input imageand thus extends the exact specification method of the image luminance(or gray level) distribution recently proposed by Coltuc et al. Our edge- resenting the discrete pixel locations. Let also H = fh0 ; h1 ; . . . hL01 ghistogram specification approach is stated as an optimization problem in be the nonnormalized target histogram (i.e., the desired output inten- sity-level distribution), and let be a strict ordering relation on the setwhich each edge of a real input image will tend iteratively toward some of pixels of I , which is defined as I (x1 ; y1 ) I (x2 ; y2 ) if the grayspecified gradient magnitude values given by a target edge distribution (or anormalized edge histogram possibly estimated from a target image). To thisend, a hybrid optimization scheme combining a global and deterministic level (or the intensity value) of pixel I (x1 ; y1 ) is lower or equal thanconjugate-gradient-based procedure and a local stochastic search using the the gray level of pixel I (x2 ; y2 ) with respect to the lexicographic order.Metropolis criterion is proposed herein to find a reliable solution to our en- Then, the exact specification proceeds simply as follows [5] (i.e., algo-ergy-based model. Experimental results are presented, and several appli- rithm A). 1) Order pixels are I (x1 ; y1 ) I (x2 ; y2 ) . . . I (x ; y )cations follow from this procedure. Index Terms—Conjugate gradient, edge-histogram specification, en-ergy-based model, gradient magnitude, local stochastic search, Metropolis 2) Split this pixel ordering relation from left to right in L groups,algorithm. such as group j has hj pixels. 3) For all the pixels in group j , assign the gray level j . In this context, the structure of the image is thus distorted by en- I. INTRODUCTION forcing the target histogram, and it yields exact results if a strict or- dering relation is found. In practice, several ordering relation strate- An image histogram, by its ability to represent the intensity-level dis- gies can be used. The simplest one consist of preprocessing the orig-tribution of the image pixels, remains a useful and popular statistical inal image by adding a small amount of uniform noise to each pixeltool that enables information about the visual appearance of an image intensity value [6], [7] or separating randomly each pixel of the orig- http://ieeexploreprojects.blogspot.comintensity level [8]–[10]. Another solution,to be obtained quickly and easily or histogram-based features (such as inal image with the samethe mode, mean, variance, entropy, energy, kurtosis, etc.) widely used i.e., avoiding noise, consists of separating pixels of the same intensityin region-based image segmentation, and indexing or local enhance- group either according to their local mean on the four horizontal andment techniques to be computed . Among the classical algorithms ex- vertical neighbors [11] or to the average intensity (of the surroundingploiting this intensity-level distribution, histogram specification (also pixels) at their location [5] or, finally, by taking into account not onlycalled histogram matching) refers to a class of image transforms, which the local mean intensity but also the local edge information [12] viachanges the histogram of a given image to another desired one. It is an a wavelet transform (which preserves edge information and producesimportant and well-known technique that can be used, e.g., to water- sharper image enhancement results compared with the classical localmark an image [1], to enhance the contrast in only some specific re- mean model [5], [11]).gions (of interest) of the image (by modifying the dynamic range of Edges are also important features of an image because they containthe pixel values) [2]–[4], or to normalize two images (e.g., for fusion, significant information; indeed, edges may correspond to object bound-mosaicing, registration, etc.). aries or to changes in the surface orientation, discontinuities in depth, or Although the histogram specification algorithm has an exact solu- material properties, to name a few. Edges also help to extract useful in-tion for a continuous image (thus yielding to a perfect match between formation and characteristics of an image. For example, the edge-basedthe input and the desired intensity-level distribution), it is generally an features of shape and texture are important for image retrieval and in-ill-posed problem that does not admit an exact solution in the discrete dexing. Consequently, an edge histogram may be important to obtaincase. For example, in the case where the output distribution is uniform, information about the visual appearance of an image (i.e., coarse orthe resulted histogram after specification (or so-called equalization) is highly detailed image, the structure in the image spatial configuration,flattened but may be far from being uniform. This comes from the fact spatial resolution, spatial detail statistics, and fractal dimension of anthat, since the number of pixels is usually much larger that the number image1) or its content (with naturally uneven or perfectly geometricallyof intensity levels, there are many pixels with the same intensity level, shaped or man-made objects). In the light of the discussion above, it is fair to think that the edge-histogram specification of an image may be of interest for several computer vision and image processing appli- Manuscript received August 28, 2010; revised December 14, 2010 and March cations. If the statistical distribution of the intensity value of any real09, 2011; accepted June 09, 2011. Date of publication June 20, 2011; date of images varies, the statistical distribution of edges or the gradient mag-current version December 16, 2011. The associate editor coordinating the reviewof this manuscript and approving it for publication was Prof. Rafael Molina. nitude of an image follows a (well known in the denoising community The author is with the Département d’Informatique et de Recherche Opéra- [14]) long-tail distribution mathematically expressed by a two-param-tionnelle, Université de Montréal, Faculté des Arts et des Sciences, Montréal, eter density function of form H (z ) / exp(0jz=cjp ). This is due to theQC H3C 3J7, Canada (e-mail: Color versions of one or more of the figures in this paper are available online 1The fractal dimension of an image surface corresponds to the human percep-at tion of image roughness [13]. Digital Object Identifier 10.1109/TIP.2011.2159804 1057-7149/$26.00 © 2011 IEEE
  2. 2. 380 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 21, NO. 1, JANUARY 2012intrinsic stationary property of real-world images, containing smooth edge-histogram specification histogram method is thus a two-stepareas interspersed with occasional sharp transitions, i.e., edges. The procedure which proceeds as follows (i.e., algorithm B2).smooth regions produce small-amplitude gradient magnitude values, 1) Ordering relation;and the transitions produce sparse large-amplitude gradient magnitude a) Normalize H in order that W = k=0 01 hk k =Zvalues [14]. Due to this intrinsic stationary property of any real-world b) Order the W = 81 N 1 M pairwise pixel absolute differences:images, the edge histogram will be associated with a decreasing func- jI s 0 I t j j I u 0 I v j . . . j I x 0 I y jtion with a unique mode (the value that occurs the most frequently) at c) Split this pixel-absolute-difference ordering relation from(amplitude gradient magnitude) zero, corresponding to the numerous left to right in Z groups, such as group j has hj elements, i.e., hj couples of pixels. d) For all pairs of pixels or pair of sites (s; t) whose the absolutesmooth regions existing in any real-world images. Except for this prop- difference is in group j , assign
  3. 3. [1] s;t = j .erty, different informative distributions (for different parameter positivevalues of p and c) can be found or specified for a given input image. 2) Optimize (1) (see Section III). In this paper, an approach for the edge-histogram specification of a This model can be easily generalized in order to ensure an edge spec- ification histogram, e.g., in the n = 2-order sense (i.e., with gradientreal image is proposed. This approach combines the ordering relationdescribed above but applied to the set of the gradient magnitude values magnitude using the second-order derivative). To this end, the sum-of an input image (and related to each pair of pixels separated from a mation of (1) should be all the pair of sites (s; t) with t 2 Ns and 2given distance). It allows us to obtain first the set of increasing gradient Ns2 designating the 16-pixel neighborhood of s separated by distancemagnitude values of an input image and then to assign to each of them d = 2 pixels (d = maxfjxs 0xt j; jys 0yt jg) and W = 161N 1M pairsa specified gradient magnitude value given by a target edge distribu- of pixels or
  4. 4. [2] s;t values. In the same way, this model can be easilytion (or a normalized edge histogram possibly estimated from a target generalized in order to ensure simultaneously an edge-histogram spec-image). A hybrid optimization scheme combining a global and de- ification following nt different distributions for, respectively, the set ofterministic conjugate-gradient-based procedure and a local stochastic nt gradient magnitude values in nt different order senses. To this end,search allow each pair of pixel values to tend (iteratively) toward these let H[n ] be the vector associated to the nt nonnormalized target distri-specified gradient magnitude levels. The remainder of this paper is or- butions H[l] = fh[l];0 ; h[l];1 ; . . . h[l];Z 01 g with l 2 [0 . . . nt [ (possiblyganized as follows: Sections II and III describe, respectively, the pro- a priori imposed or estimated from a target image), let Ns represents lposed model and the optimization strategy. Finally, Section IV presents the W[l] neighbors of s separated by distance d = l pixels, the proce-the set of experimental results and the applications of this edge-his- dure will commence as follows (i.e., algorithm C):togram specification method. 1) Specification with ordering relation; a) For l = 1 to nt ; I) W[l] = 8 1 l 1 N 1 M II. PROPOSED MODEL II) Normalize H[l] such that, W[l] = k=0 01 h[l];kk =Z http://ieeexploreprojects.blogspot.comW[l] pairwise pixel absolute differences: Let us first consider the case of an edge-histogram specification pro- III) Order thecedure in the first-order sense, i.e., using the absolute value of the gra- jI s 0 I t j j I u 0 I v j . . . j I x 0 I y jdient magnitude with the first-order derivative. To this end, let I bean input discrete image with N 2 M pixels Is located at discrete IV) Split this pixel-absolute-difference ordering relationlocations s = (xs ; ys ). Our edge-histogram specification procedure from left to right in Z groups, such as group j has ^aims at finding the new luminance mapping I in which each jIs 0 It j^ ^ h[l];j elements, i.e., h[l];j couples of this input image with a pair of sites (s; t) separated by a distance V) For all pairs of pixels or sites (s; t) whose absoluted = maxfjxs 0 xt j; jys 0 yt jg = 1 pixel (i.e., with site t located difference is in a group j , assign
  5. 5. [l] s;t = j .in the first nearest eight neighbors of s) is considered as an indepen- 1 2 2) Optimize (1) for t 2 Ns [ Ns [ . . . [ Ns (see Section III). ldent random variable whose distribution follows a target distribution or Finally, this model can be easily generalized in order to ensure si-the normalized histogram H with a desired shape (possibly estimated multaneously nt edge-histogram specifications (following nt different ^from a target image). If this mapping I is estimated in the minimal given distributions) and an exact histogram specification of the lumi- ^mean square sense, then I is the solution image that should minimize nance (or intensity) level. The method (i.e., algorithm D) simply con-the following objective function E (I ): sists of alternating algorithms C and A until a stability criterion is reached (i.e., the output image does not change too much between two NM iterations). We would like to add that extending our approach to color ^ 2 I = arg min 2
  6. 6. [1] s;t 0 (Is 0 It )2 (1) images is straightforward as follows. I s=1 t2N 1) In the case where the input image is specified directly from a target distribution law, it consists first of representing the input image E (I ) (originally expressed in the RGB color space) in a color space where one coordinate is the intensity or luminance value, such as 1where Ns represents the eight nearest neighbors of s and, conse- the perceptual LAB color space, and processing only on the lumi-quently, the summation is over all the pair of sites (i.e., for all sites ^ nance value. After treatment, let L be the output (edge specified)s and for all the pair of sites including s with t belonging to the 2Algorithm B corresponds to the cases where the input and target imageseight nearest neighbors of s). In this case,
  7. 7. [1] s;t are the values have integer luminance values ranging from [0 : 255], and we also considergiven by an edge-histogram specification method (of the first-order Z = 256 bins for the luminance histogram and for the target histogram ofmagnitude gradient) with the nonnormalized target distribution the absolute value of the first-order difference (first-order gradient magnitude).H = fh0 ; h1 ; . . . hZ 01 g (possibly a priori imposed or estimated from Consequently, if the target image has the same size of the input image, h isa target image) with Z as its number of bins. Practically speaking, let necessarily a natural number (ranging from [0 : 8NM ]). Nevertheless, if theW = 8 1 N 1 M be the number of absolute values of the first-order target image does not have the same size of the input image, each value h ofdifference jIs 0 It j in the original image, and let be a strict ordering distribution H , i.e., after the first step of Algorithm B (i.e., the step ensuring that this histogram integrates to 8 2 N 2 M ), must be rounded to the nearestrelation, which is defined among jIs 0 It j (as jIs 0 It j jIu 0 Iv j if integer, and this then ensures that h remains a natural number. Let us also notethe first-order difference jIs 0 It j is lower or equal than the first-order that, after rounding up to the nearest integer, the new h do not add up W , butdifference jIu 0 Iv j with respect to the lexicographic order). Our this is not a problem in practice.
  8. 8. IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 21, NO. 1, JANUARY 2012 381 luminance map; then, it continues by converting back the LAB ^ proportional to this distance). In order to avoid being trapped in a local into the classical RGB color space. minimum and to be closer to the global minimum, a strategy (that was 2) In the case where the input image is specified from a target image empirically tested and relatively efficient) consists of alternating the for which we want to keep its color palette, there are two different specification procedure (ordering relation), and this hybrid optimiza- ways. tion method until a given criterion is reached such as when the value of a) The histogram of the components L, A, and B of the input the energy cost function E and/or the similarity between the target and image is specified (algorithm A) from the components L, A, output edge histograms (e.g., estimated by a Bhattacharya distance) is and B , of the target image. not too high. Let us note that a global minimum is not ensured by this b) As proposed in [5], one has to define a strict ordering relation strategy. For certain images, the image structure and its properties do among color image pixels, and a possible solution is to use not allow a perfect match between the input and the desired distribution (i.e., thus inducing an error E 6= 0) to be arrived at every time. Con- the luminance or the gray value for that. In this case, the color-histogram specification procedure proceeds as follows: i) Order color pixels of the input image (I ) from their sequently, our strategy that consists of alternating the ordering relation luminance or gray value as follows: and the proposed hybrid optimization method has to be stopped when a maximal number of iterations is reached. I (x1 ; y1 ) I (x2 ; y2 ) . . . I (x ; y ): ii) Order color pixels of the target image (T ) from their IV. EXPERIMENTAL RESULTS luminance or gray value as follows: A. Setup T (u1 ; v1 ) T (u2 ; v2 ) . . . T (u ; v ): In all the experiments, the input image is assumed to be toroidal (i.e., wrapping around at the borders; this property only simplifies the im- Note that if the size of the target image is different from the size of the input image, an up-sampling or a subsam- plementation, but we can also replicate the border pixels or use a dif- pling procedure should be used. ferent strategy) with colors or luminance values or magnitude gradient iii) Assign to I (xs ; ys ) the color value T (us ; vs ) for all ranging from [0.0:1.0]. We have used 256 bins for the histogram of the s NM . luminance values and for the histogram of the magnitude gradient. Since two luminance values can be identical for different color For the conjugate gradient, the step is set to = 0:5. The maximalvalues, the first strategy thus seems to be more appropriate if we number of iterations is set to L D = 20. For the local explorationwant to preserve the different hues of an image to be specified in the search, using the Metropolis criteria, the initial temperature and thecolor-histogram sense. Nevertheless, the second strategy seems also final temperature are set to T0 = 3 1 1005 and Tf = 5 1 10010 , respec-well suited if the target image has a dominant hue as is the case in a tively. The radius of exploration is r = 0:04, and the maximal number = 100.3 Finally, in order to obtain a finaltexture transfer procedure such as that presented in Section IV-C. of iterations is set to LS edge-specified image, which will be close enough to a reliable solution, III. OPTIMIZATION STRATEGY we have respectively set E = 0:1, DB = 0:1, and LH = 6. The objective function to be minimized E may be more or less com-plex according to both the shape of the target edge distribution and the B. Edge-Histogram Specificationedge structure of the input image (i.e., the edge distribution shape of Our initial experiment with algorithm B was with a target distribu-the input image). This cost function may be sometimes nearly convex tion (for the edge histogram using the first-order derivative) with a de-if the two edge histograms are close or very complex with several local sired shape. For this experiment, it is worth recalling that the set ofextrema, if the shape of the two edge histograms are different, or if one possible shapes for the edge histogram of an image (see Section I) areof these two edge histograms exhibits some discontinuities or an un- the set of decreasing functions with a mode at (amplitude gradient mag-usual shape (i.e., a shape far away from the classical density function nitude) zero (due to the numerous smooth regions that necessarily existof the form H (z ) / exp(0jz=cjp ) put forward in [14]). In order to in any real-world images and that induce, statistically and more gen-ensure a good minimization and, thus, an accurate edge specification erally, an original edge histogram with a density function of the formprocess in all cases, we have proposed the following hybrid and adap- H (z ) / exp(0jz=cjp ) [14]). We have thus considered the followingtive optimization strategy: three unimodal (at 0) decreasing (envelope) distributions (H denoting 1) Since an analytical expression of the derivative of this function the Heaviside step function). E to be optimized is easily available, we first use a conjugate- 1) First, the semi-Gaussian function is given as gradient procedure initialized with the input original image. For 1 the conjugate gradient, the step size is fixed to and adaptively H (z ) = exp(020z )2 H(z ): decreased by a factor of 2 if the energy to be minimized increases Zh between two iterations. We stop the optimization procedure if a 2) Second, the semitriangle function is given as fixed number of iterations LD or the convergence is reached. 1 2) In order to refine the estimation given by the aforementioned de- terministic optimization method, we use the previous optimiza- H (z ) = Zh (1 0 3z ) H(1 0 3z ) H(z ): tion result as the initialization of a stochastic local search. To this 3) Third, the (nonunimodal at zero) shifted Gaussian function is end, we use a local exploration around the current solution using given as the Metropolis criteria [15] and a small radius of exploration (see algorithm 1). H (z ) = N (256 3 z ; mean = 0:1; var = 0:0001): After this hybrid optimization procedure, it is possible that a localminimum of the energy function E is reached (in this case, at conver- 3Due to the small radius of exploration, the computational complexity of this optimization step (in fact, a simple local search around the gradient estimation)gence, E 0, and practically speaking, the histogram to be specified is considerably reduced, and this explains why a low number of iterations isis yet far from being the target histogram, and with the value of E being herein performed.
  9. 9. 382 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 21, NO. 1, JANUARY 2012Fig. 1. Algorithm B. Edge-histogram specification procedure with a target distribution model. (from left to right) Original input image and four edge-histogramspecification results. (bottom right) Four output edge histograms (with the target and original histogram superimposed on the output edge histogram). The Bhat-tacharya distanceD is respectively for these four experiments 0.312, 0.257, 0.783, and 0.264 before the edge specification process and 0.091, 0.061, 0.544 and0.190 after the edge specification process. of our algorithm are then achieved qualitatively by visually comparing the output and the desired edge histogram shapes and quantitatively by estimating the Bhattacharya distance (ranging from 0 to 1) as follows: Z 01 1= 2 DB [H z ; H (z)] = ( ) 1 0 H z H (z ) ( ) (2) z =0 between the two (normalized) edge histograms before and after the specification process. Fig. 1 (and Fig. 3) shows the obtained results. We can notice that our edge-histogram specification procedure is not exact since the output histogram shape is not a perfect match with the target histogram shape. This may derive from the fact that the edge image structure could not be geometrically more distorted in order to better match the desired target edge histogram (or equivalently, the gradient descent procedure has reached the global minimum of the energy func- E E tions and 6= 0 in this case). Another possibility is that the gradient procedure is stuck in a local minimum. Nevertheless, in all tested cases, the estimation of the Bhattacharya distance shows us that the simi- larity between these two histogram shapes noticeably increases (or the Bhattacharya distance decreases) after our edge specification process, except for the nonunimodal at zero shifted Gaussian distribution for which the output histogram remains far away from the target distribu- D tion and the Bhattacharya distance remains high ( B = 0 544). We : can notice that the resulting output images are, in these three cases, visually different with different edge statistic properties (and this will also be confirmed in the following experiments). Our edge specification process may also be efficiently used for 4) Fourth, a decreasing exponential function is given as detail enhancement of an input image or even as an original detail exaggeration procedure that goes much further than the results usually obtained with classical high-boost filters for which artifacts due to H z ( )= 1 Zh 0 z H (z ) exp( 8 ) the noise amplification of the high-pass filter (in the case of high value of the boosting parameter) quickly appear and may degrade the Zwith h , i.e., a normalizing factor ensuring that these functions inte- image quality. In our case, this detail exaggeration procedure simplygrate to one (these aforementioned target distributions are graphically consists of the use of algorithm B in order to realize an edge-histogramshown at the bottom right in Fig. 1). The validation and the efficiency equalization technique (i.e., by considering the target distribution
  10. 10. IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 21, NO. 1, JANUARY 2012 383Fig. 2. Algorithm B. Detail enhancement and detail exaggeration procedure on the input image shown at top and bottom left. (from top to bottom) Cathedral(Notre Dame, Lyon, France) and statue images (Berkeley database) and results for two different decreasing values of the Bhattacharya distance (respectively, 20:85 D 2 and 0:78 D as the stopping criterion of algorithm B with H being the uniform distribution), i.e., cathedral image D = 0:57 D = 0:41.(original image), D = 0:48, and D = 0:44. Statue image D = 0:54 (original image), D = 0:44, andH simply equal to the uniform distribution). Our iterative mini-mization-based edge-histogram specification procedure then will aimat flattening, as much as possible, the gradient magnitude distributionof the input image. In this latter procedure, the desired level of detailin the output image can also be easily controlled, e.g., by estimatingat each iteration the Bhattacharya distance between the output andthe desired uniform distribution and simply by stopping our iterativeprocedure when this parameter reaches a given similarity value. In thissupervised procedure, the desired level of detail in the output imagewill increase as the user increases the value for this, i.e., the Bhat-tacharya-distance-based similarity measure between the output edgehistogram and the uniform edge distribution. Figs. 2 and 3 show theobtained results for two different increasing values of this aforemen-tioned Bhattacharya (similarity) value as a stopping criterion. Our procedure of edge-histogram specification may also be used in Fig. 3. Algorithm B. Edge-histogram specification procedure with a target dis- tribution model. (from left to right) Magnified regions in Figs. 1 and 2.order to render an input image with different detail levels or, more gen-erally, into a specified number of separate levels of detail depths. Thisrendering is possible if one specifies the output edge histogram with a as a motion or focal blur) between two images of (possibly) the samemultimodal (edge) distribution. This allows us to render an image with scene. This correction can be useful in order to normalize an imagedifferent classes of edge magnitude values or to enhance a specified set (e.g., for mosaicing generation, fusion, registration, lighting correc-class of detail. Fig. 4 shows the obtained image results for one, two, tion, indexing, retrieval systems, or other applications). Fig. 5 showsand three different classes of detail accuracy levels, respectively (thus different views and icons of the cathedral church of Notre Dame deby specifying the output edge histogram to be respectively unimodal, Fourviere (Lyon, France) taken by different cameras at different timesbimodal, and trimodal). (thus with different resolution levels and color palettes). One of these images is the cathedral image already used in the preceding experi-C. Specification of Multiple-Edge Histograms ments and considered in this test as the target high-resolution color Our edge-histogram specification model can also be used to some- image on which we desire to normalize the other images in the color-what eliminate an effect of unequal resolution (i.e., loss of accuracy, and resolution-degree senses. The results of our edge-and-color-his-contrast, or details) possibly created by a blurring degradation (such n togram specification method (algorithm D with t = 2, i.e., in the two
  11. 11. 384 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 21, NO. 1, JANUARY 2012 any specification method), the image after our edge-and-color-his- togram specification method and after a classical color-histogram specification, respectively. For our algorithm, the similarity of the edge-histogram shapes of the resulting and target image thus notice- ably increases, demonstrating that our algorithm allows to transfer not only the color information but also the edge geometric properties of the target image (more precisely, the nt shapes of its edge distributions). Another consequence of our algorithm is that it does not distribute the different colors of the target image in the same way of our edge-and color-histogram specification method since our algorithm D seems to find a compromise between the similarity between the distribution of color levels and also the distribution of gradient magnitude values of the target image. These remarks can also be confirmed in the case of a texture transfer technique only using a single color-histogram specification strategy (second algorithm of Section II), which do not allow to copy the edge textural property of a given drawing style. This is particularly visible in the case of the pointillist-style transfer technique for which its edge distributions are specific and far away from those of a natural image. D. Sensitivity to Internal Parameters First, it is worth mentioning that our algorithm is relatively insensi- tive to high values of the step size because of our adaptive decreasing schedule which adaptively adjusts and reduces this value in the conju- gate-gradient procedure if this parameter is set mistakenly too high. Second, it is also worth mentioning that our overall minimization procedure is relatively insensitive to the three parameters LD , LS , and LH , which are related to the different number of iterations of the minimization procedures since the final stopping criterion (E and DB ) will ultimately check if the final solution is close enough to a reliable solution. Third, E = 0:1 and DB = 0:1 (except for algorithm B, which is used as a detail enhancement or exaggeration procedure, for whichFig. 4. Algorithm B. Image rendering procedure with different classes of detail DB has to be set by the user) must not be considered as two internalaccuracy levels. (from top to bottom and left to right) Images and edge-his- parameters of our algorithm but rather as a criterion (e.g., required bytogram specification results (with the target histogram superimposed on the the schedule of conditions) for the expected estimation accuracy of theoutput edge histogram) for one, two, and three different classes of detail ac- final result.curacy levels, respectively (by specifying the output edge histogram to be re-spectively unimodal, bimodal, and trimodal). The original images are shown at Fourth, Tf is easily findable in our case since a good final tempera-Figs. 1 and 2. ture for a simulated annealing-like minimization procedure has to en- sure that, at the end of the stochastic search, very few sites change their luminance values between two complete image sweeps. In our algo-first-order senses and exploiting the first algorithm, which is presented rithm, this parameter has been easily found after a few trials. We haveat the end of Section II, to ensure a specification of the color histogram) found that Tf = 5 1 10010 was appropriate for all the experiments pre-on the three original images are shown in Fig. 5. sented in this paper. The proposed edge specification method can also be used to trans- Finally, two internal parameters are sensitive and crucial for our al-form one input image into another with different edge geometric and gorithm, i.e., the radius of exploration r and, in a least measure, thetextural properties. To this end, we have applied the algorithm D on a starting temperature T0 of the local stochastic search. The first one wasportrait image exploiting the three first edge distributions (i.e., nt = 3 set in order to locally explore a solution whose luminance values arein algorithm D along with the second specification method for the close to the initial solution given by the gradient minimization proce-color histogram) estimated from a target image representing a certain dure (value r = 0:04 ensures that the final solution will exhibit outputdrawing style. The resulting images are shown in Fig. 6. luminance values, which are centered around the gradient estimation, We have compared our nt -edge-and-color-histogram specification with 60:04 3 255 = 610 luminance values for a final luminancemethod (Algorithm D) with a classical method that exploits only color image whose luminance values are comprised in [0 : 255]). T0 is setinformation. Fig. 7 shows a magnified region of the cathedral image in order to ensure that, at the beginning of the stochastic search, ap-[see Fig. 5(a)] that a single color-histogram specification strategy (first proximately 50% of sites change their luminance values between twoalgorithm of Section II, i.e., the same as that used in our algorithm complete image sweeps.D) does not allow to get an output image with the same statisticaledge geometric properties and level of detail of the target image E. Algorithm(which is more detailed that the original image). This “detail-level The computational times of our procedure vary greatly dependingspecification” can also be quantified with the Bhattacharya distance on the shape of the input and target edge histograms (i.e., between 10DB (between the edge histograms of the output and the target image), and 300 s) for an AMD Athlon 64 Processor 3500+ 2.2-GHz 2010.17which is 0.310, 0.058, and 0.534 for the original image (i.e., before bogomips and for a nonoptimized code running on Linux. In addition,
  12. 12. IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 21, NO. 1, JANUARY 2012 385Fig. 5. Algorithm D. Image (resolution and color) normalization procedure. (from left to right) Two original images, the target image, and output edge-and-color-normalized result images obtained by our algorithm (with n =2 ). Fig. 7. (Top to bottom) Magnified region of the image shown at Fig. 5(a) for a single color-histogram specification strategy and our edge-and-color-histogram specification method. The texture transfer technique using the input imageFig. 6. Algorithm D. Transfer procedure of the edge textural properties be- shown at top of Fig. 6 and a drawing style, and exploiting only a singlelonging to an image to another. (from top to bottom and left to right) Original color-histogram specification strategy (to be compared with the results shownimage and a set of pairs of images including drawing styles (i.e., the ink painting, in the last row in Fig. 6).sanguine, pointillist, and painting styles, respectively) and the obtained transfer V. CONCLUSIONresult with algorithm D (with n =3 ). In this paper, we have presented an original edge-histogram specifi- cation model. Our approach is based both on a strict ordering relation between each pair of pixels (existing in the input image and separatedit must be noted than our energy minimization can be efficiently im- by a given distance) followed by a hybrid optimization process (i.e.,plemented by using the parallel abilities of a graphic processor unit a deterministic global gradient followed by a stochastic local search),(GPU) (embedded on most graphics hardware currently available on particularly well suited to our energy-based edge-histogram specifica-the market) and can be greatly accelerated (up to a factor of 200) with tion model. Concretely, this energy-based model iteratively and geo-a standard NVIDIA GPU (2004), as indicated in [16]. The source code metrically distorts the edge structure of the input image during the min-(in C++ language) and the pseudocode of our algorithm with the set imization process in order to transform its edge histogram, as muchof original and presented images (and some additional images) are as possible, to another desired edge histogram. Several applications ofpublicly available at the following web address this model, such as a detail exaggeration procedure, an edge high-boost~mignotte/ResearchMaterial/obehs in order to make possible eventual or enhancement filter, and a texture transfer technique, have been pre-comparisons with future algorithms and visual comparisons. sented and discussed.
  13. 13. 386 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 21, NO. 1, JANUARY 2012 ACKNOWLEDGMENT PSF Estimation via Gradient Domain Correlation The author would like to thank the anonymous reviewers for their Wei Hu, Jianru Xue, Member, IEEE, andmany valuable comments and suggestions that helped to improve both Nanning Zheng, Fellow, IEEEthe technical content and the presentation quality of this paper. Abstract—This paper proposes an efficient method to estimate the point REFERENCES spread function (PSF) of a blurred image using image gradients spatial correlation. A patch-based image degradation model is proposed for esti- mating the sample covariance matrix of the gradient domain natural image. [1] D. Coltuc and P. Bolon, “Color image watermaking in HSI space,” Based on the fact that the gradients of clean natural images are approxi- in Proc. IEEE ICIP, Vancouver, BC, Canada, Sep. 2000, vol. 3, pp. mately uncorrelated to each other, we estimated the autocorrelation func- 698–701. tion of the PSF from the covariance matrix of gradient domain blurred [2] R. C. Gonzales and R. W. Woods, Digital Image Processing. Upper image using the proposed patch-based image degradation model. The PSF Saddle River, NJ: Prentice-Hall, 1992. is computed using a phase retrieval technique to remove the ambiguity in- [3] V. Caselles, J. Lisani, J. Morel, and G. Sapiro, “Shape preserving local troduced by the absence of the phase. Experimental results show that the histogram modification,” IEEE Trans. Signal Process., vol. 8, no. 2, pp. proposed method significantly reduces the computational burden in PSF 220–230, Feb. 1999. estimation, compared with existing methods, while giving comparable blur- [4] J. Stark, “Adaptive image contrast enhancement using generalization ring kernel. of histogram equalization,” IEEE Trans. Signal Process., vol. 9, no. 5, pp. 889–896, May 2000. Index Terms—Deblur, phase retrieval (PR), point spread function (PSF) [5] D. Coltuc, P. Bolon, and J. M. Chassery, “Exact histogram specifica- estimation. tion,” IEEE Trans. Image Process., vol. 15, no. 5, pp. 1143–1152, May 2006. [6] A. Rosenfeld and K. Kak, Digital Picture Processing. Upper Saddle River, NJ: Prentice-Hall, 1982. I. INTRODUCTION [7] L. Shapiro and G. Stockman, Computer Vision. Englewood Cliffs, NJ: Prenctice-Hall, 2001. [8] Y. Zhang, “Improving the accuracy of direct histogram specification,” The problem of blur kernel estimation, and more generally blind de- Electron. Lett., vol. 28, no. 3, pp. 213–214, Jan. 1992. convolution, is a long-standing problem in computer vision and image [9] J. Rolland, V. Vo, B. Bloss, C. Abbey, and G. Young, “Fast algorithm processing. Recovering the point spread function (PSF) from a single for histogram matching applications to texture synthesis,” J. Electron. Imag., vol. 9, no. 1, pp. 39–45, Jan. 2000. blurred image is an inherently ill-posed problem due to the loss of in- [10] A. Bevilacqua and P. Azzari, “A high performance exact histogram formation during blurring. The observed blurred image provides only specification algorithm,” in Proc. 14th IEEE ICIAP, Modena, Italy, a partial constraint on the solution as there are many combinations of Sep. 2007, pp. 623–628. can be convolved to match the observed PSFs and “sharp” images that [11] E. L. Hall, “Almost uniform distribution for computer image enhance- ment,” IEEE Trans. Comput., vol. C-23, no. 2, pp. 207–208, Feb. 1974. blurred image. There are numerous papers on this subject in the signal [12] Y. Wang and D. Shi, “Joint exact histogram specification and image and image processing literature. We refer the reader to a survey paper enhancement through the wavelet transform,” IEEE Trans. Image [1] for the earlier works. Recently, state-of-the-art PSF estimation al- Process., vol. 16, no. 9, pp. 2245–2250, Sep. 2007. gorithms have been proposed based on natural image statistics [2], [3] [13] B. B. Mandelbrot, The Fractal Geometry of Nature. San Francisco, using a variational Bayesian method [2] or an advance optimization CA: Freeman, 1982. [14] E. Simoncelli, “Statistical models for images: Compression, restora- technique [4]. In [5], Levin et al. provide a good summary for these tion and synthesis,” in Conf. Rec. 31st Asilomar Conf. Signals, Syst. methods. In this paper, we propose a new simple and efficient method Comput., Pacific Grove, CA, Feb. 1997, pp. 673–678. to extract a blur kernel using a gradient domain correlation technique. [15] N. Metropolis, A. W. Rosenbluth, M. N. Rosenbluth, A. H. Teller, and It has been proven in various fields of image processing that the gradi- E. Teller, “Equation of state calculations by fast computing machines,” J. Chem. Phys., vol. 21, no. 6, pp. 1087–1092, Jun. 1953. ents of the natural images are approximately independent to each other [16] P.-M. Jodoin and M. Mignotte, “Markovian segmentation and param- [2]. Many existing techniques for blind deconvolution take advantage eter estimation on graphics hardware,” J. Electron. Imag., vol. 15, no. of this property to simplify the inference [2], [4]. In particular, when 3, pp. 033 015–1–033 015–15, Sep. 2006. the problem is formulated into the Bayesian framework, this assump- tion usually substantially simplifies the prior term [2]. These sophis- ticated algorithms [2], [4] give good deblurred results. However, they are usually very slow since most of them involve a complex alternating optimization process, which is very time consuming. In this paper, we have observed that the correlation property of the image gradients is Manuscript received May 10, 2010; revised January 30, 2011 and May 17, 2011; accepted May 27, 2011. Date of publication June 20, 2011; date of current version December 16, 2011. This work was supported in part by the NSFC under Project 60875008 and Project 60805044 and in part by the 973 Program under Project 2010CB327902. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Farhan A. Baqai. The authors are with the Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Xi’an 710049, China (e-mail:;; Color versions of one or more of the figures in this paper are available online at Digital Object Identifier 10.1109/TIP.2011.2160073 1057-7149/$26.00 © 2011 IEEE