Red Eye Removal

Red Eye Removal - Presentation Transcript

1/26 Red Eye Removal y Tony Meccio Red eye removal 17/04/2008

Overview 2/26 • I t d ti Introduction t the Red Eye problem to th R d E bl • Red Eye prevention • Red Eye detection • Semi-automatic methods • Automatic methods •R dE Red Eye correction ti • Desaturation • Inpainting techniques • False positives and unnatural corrections a po a du au a o o • Red Eye removal examples Red eye removal 17/04/2008

The Red Eye problem 3/26 The Red E Th R d Eye phenomenon is a well- h i ll known problem which happens when taking flash-lighted p g g pictures of people. The pupils in the picture appear red instead of black. This h Thi happens more often when ft h using compact, consumer-oriented cameras. a a Red eye removal 17/04/2008

Why the Red Eye? 4/26 Red Eye Cone Eye α β • The red eye cone shines from the flashed eye back at the flash with an angle α; g ; • Its red color is caused by the reflection of the flash off the blood vessels o the retina; a o b ood of a; • The camera will record this red hue if the angle β be ee between the flash and the camera is not greater e as a d e ca e a s o g ea e than α. Red eye removal 17/04/2008

Red Eye prevention 5/26 Eye Red Eye Cone α β • If the equipment allows it, the flash can be spaced further away from the sensor in order to increase y β (not possible on compact devices); • One o more additional flashes before picture O or o add o a a b o p u acquisition make the iris tighten and decrease α; • This methods reduce the probability of the Red s e ods educe e p obab y o e ed Eye phenomenon but don’t remove it entirely. Red eye removal 17/04/2008

Red Eye detection 6/26 Most f th ti M t of the times, red eyes must be removed d tb d during post-processing. For d F red eyes to be successfully removed, they must t b f ll d th t be first detected then corrected. Methods are classified according to the detection phase: h • Semi-automatic methods ask the user to manually l ll localize th red eyes; li the d • Automatic methods detect the red eyes themselves. th l Red eye removal 17/04/2008

Semi-automatic methods 7/26 The eyes are manually selected using a visual interface ( g (Adobe Photoshop ®, Corel Paint Shop Pro ®, ACD See ®, etc.) Pros: • Eyes are easy to localize for y y men. Cons: • It may be difficult to have such an interface on a mobile device; • Automatic methods are easier uo a od a a to use and more appealing. Red eye removal 17/04/2008

Automatic methods 8/26 Automatic methods attempt to find red eyes on A t ti th d tt t t fi d d their own. The task is harder than it may seem: No “ N “perfect” R d E f t” Red Eye d t ti detection method has th d h been developed yet. Automatic methods extract features from images in A t ti th d t tf t f i i order to identify red eyes. Different methods work on different features: • Face detection • Skin detection • Eye detection • Flash-noFlash comparison Red eye removal 17/04/2008

Face-based methods 9/26 •F Faces are looked for using a multiple feature l k df i lti l f t object based approach; •O Once th f the face features have been found then the f t h b f d th th research is restricted to red pixels. Face Extraction Extracted Multiple Masks Area Red-Eye Red Eye Research Red eye removal 17/04/2008

Eye-based methods 10/26 Similar t face detection, but more complex Si il to f d t ti b t l because the features are less evident: •EEyes are seeked, matching fi d t k d t hi fixed templates at l t t different resolutions with regular eyes present into the images, or looking for red pixels using g , g p g computed color LUTs. Initial Single Eye g y Pairingg Candidate Verification Verification Detection Red eye removal 17/04/2008

Skin-based methods 11/26 • Ski i d t t d first by pixel colors; Skin is detected fi t b i l l • Red circular patches near the skin are then looked for. l k df This approach is simpler and does not take into account th presence of more complex features. t the f l f t Skin Detection Red Circular Pairingg Patches Verification Research Optional Red eye removal 17/04/2008

Flash-noFlash methods 12/26 • Two different pictures, one with flash and one ith fl h d without, are acquired one after the other;; • Red eyes are detected as patches whose color is p red in the “flash” image and black in the “nonflash” i “ fl h” image. This approach has several drawbacks: • The dimension of the buffer must double; • The two images may be mis-aligned; e o ages ay s a g ed; • The subject(s) may move between acquisitions. Red eye removal 17/04/2008

An algorithm in detail 13/26 Red Color Input Skin Morphological Detection in Detection Operators Skin Area Image Morphological Operators Corrected Red-Eye Pairing Single Eye Verification Verification Image g Correction The Algorithm is Skin Feature Extraction based. • First the skin is extracted and morphologically modified to blob the enhanced areas; • Then a successive red color detection is performed to find red eyes in the skin areas; • The red regions are then dilated, eroded and analyzed to identify the Red-Eyes pairs. Red eye removal 17/04/2008

Example 14/26 Input Image Skin Detection Morphological Red Color Detection Operators in Skin Area Morphological Pairing Corrected Operators Verification Image Red eye removal 17/04/2008

Input/Output comparison 15/26 Red eye removal 17/04/2008

Algorithm drawbacks 16/26 • U bl t perform single red eye detection; Unable to f i l d d t ti • The skin detection is performed over the whole image (slow); • Big ( g (slow) morphological operators p ) p g p permit to g get good results only on small images (less than 1 Mpixels): it would require even bigger (and slower) operators t operate on larger images. t to t l i Red eye removal 17/04/2008

Red eye correction 17/26 Once red eyes have been detected, they must be O d h b d t t d th tb corrected. Red eye correction is quite simpler than detection, but there are more diffi lt cases than others. b t th difficult th th Correction may vary from simple desaturation to complete reconstruction of iris and pupil. Red eye removal 17/04/2008

Desaturation 18/26 Desaturation means lowering/zeroing the chrominance components while mantaining the luminance component component. It is the best way to correct “easy” red eyes. Red eye removal 17/04/2008

Washed-out irises 19/26 Washed-out irises Wrong correction Sometimes irises are totally washed out by reflected light. In these cases a simple desaturation or color correction is not enough. It is necessary to use a more complex method to reconstruct a realistic image of the eye. Red eye removal 17/04/2008

Inpainting techniques 20/26 Some tools completely reconstruct the irises and the pupils to t replace the red eye (Jasc Paint Shop Pro ®). l th d (J P i t Sh P ®) The results, however, are often unrealistic and look like glass eyes. Red eye removal 17/04/2008

False positives 21/26 One of the biggest issue in red eye removal are false p positives in the detection phase. p Unwanted corrections are much less desirable than missing ones ones. Red eye removal 17/04/2008

Unnatural corrections 22/26 • Unnatural corrections are another important issue. • The most common ones are: • Partial correction: only a Partial correction portion of the red pupil has f been corrected; • Noisy correction: the presence of heavy noise or jpeg compression can introduce false red pixels around the pupil and Noisy correction thus a strange correction is made over the iris; • Wrong luminance correction: in this case the disk has been correctly found but the correction is unnatural due to Wrong luminance wrong luminance distribution. correction Red eye removal 17/04/2008

Red Eye removal examples 23/26 StopRedeyes® AutoRemover® Red eye removal 17/04/2008

References 26/26 • J.Y. Hardeberg, “Red Eye Removal using Digital Color Image Processing”, Conexant Systems, Inc. , Redmond, Washington, USA • M. Gaubatz, R. Ulichney, “Automatic Red-Eye Detection and Correction”, HP Lab, Cornell University, University ICIP 2002 • S. Ioffe, “Red Eye Detection With Machine Learning”, Fujifilm Software, California, ICIP 2003. • F Gasparini, R. Schettini “Automatic Redeye Removal for Smart Enhancement of Photos of F. Gasparini R Schettini, Automatic Unknown Origin”, DISCO University of Milano-Bicocca, VIS 2005. • H. Luo, J. Yen, D. Tretter, “An Efficient Redeye Detection and Correction Algorithm”, HP labs, Palo Alto, California, ICPR 2004. • A. Patti, K. Konstantinides, D. Tretter, Q. Lin, “Automatic Digital Redeye Reduction”, HP Labs, Palo Alto California, ICIP 1998. • L. Zhang, Y. Sun, M. Li, H. Zhang, “Automated Red-Eye Detection and Correction in Digital Photographs”, Photographs” Microsoft Research Asia China ICIP 2004 Asia, China, • B. Smolka, K. Czubin, J.Y. Hardeberg, K.N. Palataniotis, M. Szczepanski, K. Wojciechowski, “Towars Automatic Redeye Effect Removal”, Pattern Recognition Letter 2003. • J S Schildkraut R T Gray “A Fully Automatic Redeye Detection and Correction Algorithm”, J.S. Schildkraut, R. T. Gray, A Algorithm Kodak Company, NY, ICIP 2002. • R. Schettini, F. Gasparini, F. Chazli, ”A Modular Procedure for Automatic Redeye Correction in Digital Photos”, DISCO University of Milano-Bicocca, SPIE 2004 • PETSCHNIGG, G., AGRAWALA, M., HOPPE, H., SZELISKI, R., COHEN, M., and TOYAMA, K. 2004. Digital photography with flash and no-flash image pairs. ACM Transactions on Graphics 23, 3 (Aug.), 664–672. Red eye removal 17/04/2008