WU: A LINEAR PROGRAMMING APPROACH FOR OPTIMAL CONTRAST-TONE MAPPING 1263of . The in-between histogram is computed by minimizing aweighted distance . The authors showedthat undesirable side effects of histogramequalization can besuppressed via choosing the Lagrangian multiplier . Thislatest paper also gave a good synopses of existing contrastenhancement techniques. We refer the reader to  for a surveyof previous works, instead of reparaphrasing them here.Compared with the aforementioned works on histogram-based contrast enhancement techniques, this paper presents amore rigorous study of the problem. We reexamine contrast en-hancement in a new perspective of optimal allocation of outputdynamic range constrained by tune continuity. This brings abouta more principled approach of image enhancement. Our critiqueof the current practice is that directly manipulating histogramsfor contrast enhancement was ill conceived. The histogram is anunwieldy, obscure proxy for contrast. The wide use of histogramequalization as a means of context-free contrast enhancement isapparently due to the lack of a proper mathematical formulationof the problem. To ﬁll this void we deﬁne an expected (con-text-free) contrast gain of a transfer function. This relative mea-sure of contrast takes on its base value of one if the input imageis left unchanged (i.e., identity transfer function), and increasesif a skewed histogram is made more uniform. However, percep-tual image quality is more than the single aspect of high contrast.If the output dynamic range is less than that of the human visualsystem, which is the case for most display and printing tech-nologies, context-free contrast enhancement will inevitably dis-tort subtle tones. To balance between tone subtlety and contrastenhancement we introduce a counter measure of tone distortion.Based upon the said measures of contrast gain and tone distor-tion, we formulate the problem of optimal contrast-tone map-ping (OCTM) that aims to achieve high contrast and subtle tonereproduction at the same time, and propose a linear program-ming strategy to solve the underlying constrained optimizationproblem. In the OCTM formulation, the optimal transfer func-tion for images of uniform histogram is the identify function.Although an image of uniform histogram cannot be further en-hanced, histogram equalization does not produce OCTM so-lutions in general for arbitrary input histograms. Instead, theproposed linear programming-based OCTM algorithm can op-timize the transfer function such that sharp contrast and subtletone are best balanced according to application requirementsand user preferences. The OCTM technique offers a greaterand more precise control of visual effects than existing tech-niques of contrast enhancement. Common side effects of con-trast enhancement, such as contours, shift of average intensity,over exaggerated gradient, etc., can be effectively suppressedby imposing appropriate constraints in the linear programmingframework.In addition, in the OCTM framework, input gray levels canbe mapped to an arbitrary number L of output gray levels, al-lowing L to be equal, less or greater than . The OCTM tech-nique is, therefore, suited to output conventional images on highdynamic range displays or high dynamic range images on con-ventional displays, with perceptual quality optimized for de-vice characteristics and image contents. As such, OCTM can beuseful tool in high dynamic range imaging. Moreover, OCTMcan be uniﬁed with Gamma correction.Analogously to global and local histogram equalization,OCTM can be performed based upon either global or local sta-tistics. However, in order to make our technical developmentsin what follows concrete and focused, we will only discuss theproblem of contrast enhancement over an entire image insteadof adapting to local statistics of different subimages. All theresults and observations can be readily extended to locallyadaptive contrast enhancement.The remainder of the paper is organized as follows. In thenext section, we introduce some new deﬁnitions related to theintuitive notions of contrast and tone, and propose the OCTMapproach of image enhancement. In Section III, we developa linear programming algorithm to solve the OCTM problem.In Section IV, we discuss how to ﬁne tune output images ac-cording to application requirements or users’ preferences withinthe proposed linear programming framework. Experimental re-sults are reported in Section V, and they demonstrate the ver-satility and superior visual quality of the new contrast enhance-ment technique.II. CONTRAST AND TONEConsider a gray scale image of bits with a histogram ofnonzero entries, .Let be the probability of gray level . Wedeﬁne the expected context-free contrast of by(1)By the deﬁnition, the maximum contrast and itis achieved by a binary black-and-white image; the minimum contrast when the image is aconstant. As long as the histogram of is full without holes,i.e., regardlessthe intensity distribution . Likewise, if, then .Contrast enhancement is to increase the difference betweentwo adjacent gray levels and it is achieved by a remapping ofinput gray levels to output gray levels. Such a remapping is alsonecessary when reproducing a digital image of gray levels bya device of L gray levels, L. This process is an integer-to-integer transfer functionL (2)In order not to violate physical and psychovisual common sense,the transfer function should be monotonically nondecreasingsuch that does not reverse the order of intensities.1 In otherwords, we must have if and, hence, anytransfer function has the formLL (3)1This restriction may be relaxed in locally adaptive contrast enhancement.But in each locality the monotonicity should still be imposed.
1264 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 20, NO. 5, MAY 2011Fig. 1. (a) Original. (b) Output of histogram equalization. (c) Output of the proposed OCTM method. (d) Transfer functions and the original histogram.(e) Histograms of the original image (left), the output image of histogram equalization (middle), and the output image of OCTM.where is the increment in output intensity versus a unit stepup in input level (i.e., ), and the last inequalityensures the output dynamic range not exceeded by .In (3), can be interpreted as context-free contrast at level, which is the rate of change in output intensity withoutconsidering the pixel context. Note that a transfer function iscompletely determined by the vector ,namely the set of contrasts at all input gray levels. Havingassociated the transfer function with context-free contrasts’s at different levels, we induce from (3) and deﬁnition (1) anatural measure of expected contrast gain made by(4)where is the probability that a pixel in has input gray level .The previous measure conveys the colloquial meaning of con-trast enhancement. To see this let us examine some special cases.Proposition 1: The maximum contract gain is achievedby L such that , and.Proof: Assume for a contradiction that, would achieve higher contrast gain. Due to the con-straint L equals at most L . ButL L , refuting the previousassumption.Proposition 1 reﬂects our intuition that the highest contrastis achieved when achieves a single step (thresholding) blackto white transition, converting the input image from gray scaleto binary. The binary threshold is set at level such thatto maximize contrast gain.One can preserve the average intensity while maximizing thecontrast gain. The average-preserving maximum contrast gain isachieved by L , such that. Namely, is the binary thresholding functionat the average gray level.
WU: A LINEAR PROGRAMMING APPROACH FOR OPTIMAL CONTRAST-TONE MAPPING 1265Fig. 2. (a) Original. (b) Output of histogram equalization. (c) Output of the proposed OCTM method. (d) Transfer functions and the original histogram.(e) Histograms of the original image (left), the output image of histogram equalization (middle), and the output image of OCTM.Fig. 3. (a) Original. (b) Output of histogram equalization. (c) Output of the proposed OCTM method. (d) Transfer functions and the original histogram.If L (i.e., when the input and output dynamic rangesare the same), the identity transfer function , namely,, makes regardless the gray leveldistribution of the input image. In our deﬁnition, the unit con-
1266 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 20, NO. 5, MAY 2011Fig. 4. (a) Original image before Gamma correction. (b) After Gamma correction. (c) Gamma correction followed by histogram equalization. (d) Joint Gammacorrection and contrast-tone optimization by the proposed OCTM method.Fig. 5. Comparison of different methods on image Pollen. (a) Original image. (b) HE. (c) CLAHE. (d) OCTM.trast gain means a neutral contrast level without any enhance-ment. The notion of neutral contrast can be generalized to thecases when L. We call L the tone scale. In general,the transfer functionL(5)or equivalently , corresponds to the state ofneutral contrast .High contrast by itself does not equate high image quality.Another important aspect of image ﬁdelity is the tone conti-nuity. A single-minded approach of maximizing wouldlikely produce over-exaggerated, unnatural visual effects, as re-
WU: A LINEAR PROGRAMMING APPROACH FOR OPTIMAL CONTRAST-TONE MAPPING 1267Fig. 6. Comparison of different methods on image Rocks. (a) Original image. (b) HE. (c) CLAHE. (d) OCTM.vealed by Proposition 1. The resulting degenerates a con-tinuous-tone image to a binary image. This maximizes the con-trast of a particular gray level but completely ignores accuratetone reproduction. We begin our discussions on the tradeoff be-tween contrast and tone by stating the following simple and yetinformative observation.Proposition 2: The isachieved if and only if , or .As stated previously, the simple linear transfer function, i.e.,doing nothing in the traditional sense of contrast enhancement,actually maximizes the minimum of context-free contrasts ofdifferent levels , and the neutral contrast gain largestis possible when satisfying this maxmin criterion.In terms of visual effects, the reproduction of continuoustones demands the transfer function to meet the maxmin crite-rion of proposition 2. The collapse of distinct gray levels intoone tends to create contours or banding artifacts. In this con-sideration, we deﬁne the tone distortion of a transfer functionby(6)In the deﬁnition we account for the fact that the transfer functionis not a one-to-one mapping in general. The smaller thetone distortion the smoother the tone reproduced by .It is immediate from the deﬁnition that the smallest achievabletone distortion isHowever, since the dynamic range Lof the output device is ﬁ-nite, the two visual quality criteria of high contrast and tonecontinuity are in mutual conﬂict. Therefore, the mitigation ofsuch an inherent conﬂict is a critical issue in designing contrastenhancement algorithms, which is seemingly overlooked in theexisting literature on the subject.Following the previous discussions, the problem of contrastenhancement manifests itself as the following optimizationproblem(7)The OCTM objective function (7) aims for sharpness of highfrequency details and tone subtlety of smooth shades at the sametime, using the Lagrangian multiplier to regulate the relativeimportance of the two mutually conﬂicting ﬁdelity metrics.Interestingly, the OCTM solution of (7) is if the inputhistogram of an image is uniform. It is easy to verify thatfor all but when for. In other words, no other transferfunctions can make any contrast gain over the identity transferfunction (or ), and at the same time the identitytransfer function achieves the minimum tone distortion. This concludes that an image of uniform his-togram cannot be further enhanced in OCTM, lending a supportfor histogram equalization as a contrast enhancement technique.For a general input histogram, however, the transfer function ofhistogram equalization is not necessarily the OCTM solution,as we will appreciate in the following sections.
1268 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 20, NO. 5, MAY 2011Fig. 7. Comparison of different methods on image Tree. (a) Original image. (b) HE. (c) CLAHE. (d) OCTM.III. CONTRAST-TONE OPTIMIZATION BYLINEAR PROGRAMMINGTo motivate the development of an algorithm for solving (7),it is useful to view contrast enhancement as an optimal resourceallocationproblemwithconstraint.Theresourceistheoutputdy-namic range and the constraint is tone distortion. The achievablecontrast gain and tone distortion are physically con-ﬁned by the output dynamic range L of the output device. In (4)the optimization variables represent an alloca-tion of L available output intensity levels, each competing for alarger piece of dynamic range. While contrast enhancement nec-essarily invokes a competition for dynamic range (an insufﬁcientresource),ahighlyskewedallocationof Loutputlevelsto inputlevels can deprive some input gray levels of necessary represen-tations, incurring tone distortion. This causes unwanted side ef-fects, such as ﬂattened subtle shades, unnatural contour bands,shifted average intensity, and etc. Such artifacts were noticed byother researchers as drawbacks of the original histogram equal-ization algorithm, and they proposed a number of ad hoc. tech-niques to alleviate these artifacts by reshaping the original his-togram prior to the equalization process. In OCTM, however, thecontrol of undesired side effects of contrast enhancement is re-alized by the use of constraints when maximizing contrast gain.Since the tone distortion function is not linear in , di-rectly solving (7) is difﬁcult. Instead, we rewrite (7) as the fol-lowing constrained optimization problem:subject to L(8)In (8), constraint (a) is to conﬁne the output intensity level to theavailable dynamic range; constraints (b) ensure that the transferfunction be monotonically nondecreasing; constraints (c)specify the maximum tone distortion allowed, where is anupper bound . The objective function and all the con-straints are linear in . The choice of depends upon user’s re-
WU: A LINEAR PROGRAMMING APPROACH FOR OPTIMAL CONTRAST-TONE MAPPING 1269Fig. 8. Comparison of different methods on image Notre Dame. (a) Original image. (b) HE. (c) CLAHE. (d) OCTM.quirement on tone continuity. In our experiments, pleasing vi-sual appearance is typically achieved by setting to 2 or 3.Computationally, the OCTM problem formulated in (8) is oneof integer programming. This is because the transfer functionis an integer-to-integer mapping, i.e., all components ofare integers. But we relax the integer constraints on andconvert (8) to a linear programming problem. By the relax-ation any solver of linear programming can be used to solvethe real version of (8). The resulting real-valued solutioncan be easily converted to an integer-valuedtransfer function(9)For all practical considerations the proposed relaxation solu-tion does not materially compromise the optimality. As a ben-eﬁcial side effect, the linear programming relaxation simpliﬁesconstraint (c) in (8), and allows the contrast-tone optimizationproblem to be stated assubject to L(10)IV. FINE TUNING OF VISUAL EFFECTSThe proposed OCTM technique is general and it can achievedesired visual effects by including additional constraints in (10).We demonstrate the generality and ﬂexibility of the proposedlinear programming framework for OCTM by some of manypossible applications.The ﬁrst example is the integration of Gamma correction intocontrast-tone optimization. The optimized transfer functioncan be made close to the Gamma transfer function byadding to (10) the following constraint:(11)where is the Gamma parameter and is the degree of close-ness between the resulting and the Gamma mapping.In applications when the enhancement process cannot changethe average intensity of the input image by certain amount ,the user can impose this restriction easily in (10) by adding an-other linear constraintL(12)
1270 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 20, NO. 5, MAY 2011Fig. 9. Results of different methods on image Rocks of noise, to be compared with those in Fig. 6. (a) Noisy image. (b) HE. (c) CLAHE. (d) OCTM.Besides the use of constraints in the linear programmingframework, we can incorporate context-based or seman-tics-based ﬁdelity criteria directly into the OCTM objectivefunction. The contrast gain depends onlyupon the intensity distribution of the input image. We canaugment by weighing in the semantic or perceptualimportance of increasing the contrast at different gray levelsby . In general, can be set up to reﬂectspeciﬁc requirements of different applications. In medicalimaging, for example, the physician can read an image ofgray levels on an L-level monitor, L , with a certain rangeof gray levels enhanced. Such a weightingfunction presents itself naturally if there is a preknowledgethat the interested anatomy or lesion falls into the intensityrange for given imaging modality. In combining imagestatistics and domain knowledge or/and user preference, weintroduce a weighted contrast gain function(13)where is a Lagrangian multiplier to factor in user-prioritizedcontrast into the objective function.In summarizing all of the previous discussions, we ﬁnallypresent the following general linear programming frameworkfor visual quality enhancementsubject to LLL(14)In this paper, we focus on global contrast-tone optimization.The OCTM technique can be applied separately to differentimage regions and, hence, made adaptive to local image sta-tistics. The idea is similar to that of local histogram equaliza-tion. However, in locally adaptive histogram equalization ,, each region is processed independently of others. A linear
WU: A LINEAR PROGRAMMING APPROACH FOR OPTIMAL CONTRAST-TONE MAPPING 1271Fig. 10. Results of different methods on image Notre Dame of noise, to be compared with those in Fig. 8. (a) Noisy image. (b) HE. (c) CLAHE. (d) OCTM.weighting scheme is typically used to fuse the results of neigh-boring blocks to prevent block effects. In contrast, the proposedlinear programming approach can optimize the contrasts andtones of adjacent regions jointly while limiting the divergenceof the transfer functions of these regions. The only drawbackis the increase in complexity. Further investigations in locallyadaptive OCTM are underway.V. EXPERIMENTAL RESULTSFigs. 1–4 present some sample images that are enhanced bythe OCTM technique in comparison with those produced byconventional histogram equalization (HE). The transfer func-tions of both enhancement techniques are also plotted in accom-pany with the corresponding input histograms to show differentbehaviors of the two techniques in different image statistics.In image Beach (Fig. 1), the output of histogram equalizationis too dark in overall appearance because the original histogramis skewed toward the bright range. But the OCTM method en-hances the original image without introducing unacceptable dis-tortion in average intensity. This is partially because of the con-straint in linear programming that bounds the relative difference( % in this instance) between the average intensities of theinput and output images.Fig. 2 compares the results of histogram equalization and theOCTM method when they are applied to a common portraitimage. In this example histogram equalization overexposes theinput image, causing an opposite side effect as in image Beach,whereas the OCTM method obtains high contrast, tone conti-nuity and small distortion in average intensity at the same time.Fig. 3 shows an example when the user assigns higher weightsin (14) to gray levels , whereis an intensity range of interest (brain matters in the head image).The improvement of OCTM over histogram equalization in thistypical scenario of medical imaging is very signiﬁcant.In Fig. 4, the result of joint Gamma correction and contrast-tone optimization by the OCTM technique is shown, and com-pared with those in difference stages of the separate Gamma cor-rection and histogram equalization process. The image qualityof OCTM is clearly superior to that of the separation method.The new OCTM approach is also compared with thewell-known contrast-limited adaptive histogram equalization(CLAHE)  in visual quality. CLAHE is considered to be oneof the best contrast enhancement techniques, and it alleviatesmany of the problems of histogram equalization, such as over-or under-exposures, tone discontinuities, and etc. Figs. 5–8are side-by-side comparisons of OCTM, CLAHE, and HE.CLAHE is clearly superior to HE in perceptual quality, as well
1272 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 20, NO. 5, MAY 2011recognized in the existing literature and among practitioners,but it is somewhat inferior to OCTM in overall image quality,particularly in the balance of sharp details and subtle tones.In fact, the OCTM technique was assigned and implementedas a course project in one of the author’s classes. There was aconsensus on the superior subjective quality of OCTM over HEand its variants among more than one hundred students.Finally, in Figs. 9 and 10, we empirically assess the sensitivityof OCTM to noises in comparison with histogram-based con-trast enhancement methods. Since contrast enhancement tendsto boost high frequency signal components, it is difﬁcult buthighly desirable for a contrast enhancement algorithm to resistnoises. By inspecting the output images of different algorithmsin Figs. 9 and 10, it should be apparent that OCTM is more re-sistant to noises than HE and CLAHE. This is because OCTMcan better balance the increase in contrast and the smoothnessin tone.VI. CONCLUSIONA new, general image enhancement technique of optimal con-trast-tone mapping is proposed. The resulting OCTM problemcan be solved efﬁciently by linear programming. The OCTMsolution can increase image contrast while preserving tone con-tinuity, two conﬂicting quality criteria that were not handled andbalanced as well in the past.REFERENCES E. Reinhard, G. Ward, S. Pattanaik, and P. Debevec, High DynamicRange Imaging: Acquisition, Display, and Image-Based Lighting.San Mateo, CA: Morgan Kaufmann, 2005. Y. T. Kim, “Enhancement using brightness preserving bihistogramequalization,” IEEE Trans. Consum. Electron.s, vol. 43, no. 1, pp. 1–8,1997. Y. Wang, Q. Chen, and B. Zhang, “Image enhancement based on equalarea dualistic sub-image histogram equalization method,” IEEE Trans.Consum. Electron., vol. 45, no. 1, pp. 68–75, Feb. 1999. J. M. Gauch, “Investigations of image contrast space deﬁned by vari-ations on histogram equalization,” in Proc. CVGIP: Graph. ModelsImage Process., 1992, pp. 269–280. J. A. Stark, “Adaptive image contrast enhancement using generaliza-tions of histogram equalization,” IEEE Trans. Image Process., vol. 9,no. 5, pp. 889–896, May 2000. Z. Y. Chen, B. R. Abidi, D. L. Page, and M. A. Abidi, “Gray-levelgrouping(glg): An automatic method for optimized image contrast en-hancement-part i: The basic method,” IEEE Trans. Image Process., vol.15, no. 8, pp. 2290–2302, Aug. 2006. Z. Y. Chen, B. R. Abidi, D. L. Page, and M. A. Abidi, “Gray-levelgrouping(glg): An automatic method for optimized image contrast en-hancement-part ii: The variations,” IEEE Trans. Image Process., vol.15, no. 8, pp. 2303–2314, Aug., 2006. E. D. Pisano, S. Zong, B. Hemminger, M. Deluca, R. E. Johnston, K.Muller, M. P. Braeuning, and S. Pizer, “Contrast limited adaptive his-togram image processing to improve the detection of simulated spic-ulations in dense mammograms,” J. Dig. Imag., vol. 11, no. 4, pp.193–200, 1998. T. Arici, S. Dikbas, and Y. Altunbasak, “A histogram modiﬁcationframework and its application for image contrast enhancement,” IEEETrans. Image Process., vol. 18, no. 9, pp. 1921–1935, Sep. 2009. S. M. Pizer, E. P. Amburn, J. D. Austin, R. Cromartie, A. Geselowitz,T. Greer, B. H. Romeny, J. Zimmerman, and K. Zuiderveld, “Adaptivehistogram equalizatin and its variations,” Comput. Vis. Graph. ImageProcess., vol. 39, pp. 355–368, 1987.Xiaolin Wu (SM’96–F’11) received the B.Sc. degreein computer science from Wuhan University, Wuhan,China, in 1982, and the Ph.D. degree in computer sci-ence from the University of Calgary, Calgary, AB,Canada, in 1988.He started his academic career in 1988, and hassince been on the faculty of University of WesternOntario, New York Polytechnic University, andcurrently McMaster University, Hamilton, ON,Canada, where he is a Professor in the Departmentof Electrical and Computer Engineering and holdsthe NSERC senior industrial research chair in Digital Cinema. His researchinterests include multimedia signal compression, joint source-channel coding,multiple description coding, network-aware visual communication and imageprocessing. He has published over two hundred research papers and holds twopatents in these ﬁelds.Dr. Wu is an associate editor of IEEE TRANSACTIONS ON IMAGE PROCESSING.