Image Quality Assessment: From Error Visibility to Structural Similarity
Upcoming SlideShare
Loading in...5
×
 

Image Quality Assessment: From Error Visibility to Structural Similarity

on

  • 726 views

 

Statistics

Views

Total Views
726
Views on SlideShare
726
Embed Views
0

Actions

Likes
0
Downloads
12
Comments
0

0 Embeds 0

No embeds

Accessibility

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Image Quality Assessment: From Error Visibility to Structural Similarity Image Quality Assessment: From Error Visibility to Structural Similarity Presentation Transcript

  • Image Quality Assessment: From ErrorVisibility to Structural SimilaritySarbjeet SinghNITTTRSector 26 CHD.
  • MSE=0, MSSIM=1 MSE=225, MSSIM=0.949 MSE=225, MSSIM=0.989MSE=215, MSSIM=0.671 MSE=225, MSSIM=0.688 MSE=225, MSSIM=0.723Motivationoriginal Image
  • Define PerceptualIQA MeasuresOptimize IP Systems &Algorithms “Perceptually”PERCEPTUAL IMAGE PROCESSINGApplication Scope: essentially all IP applicationsimage/video compression, restoration, enhancement,watermarking, displaying, printing …Perceptual Image ProcessingStandard measure (MSE) does not agreewith human visual perceptionWhy?
  • • Goal—Automatically predict perceived image quality• Classification— Full-reference (FR); No-reference (NR); Reduced-reference (RR)• Widely Used Methods—FR: MSE and PSNR—NR & RR: wide open research topic• IQA is DifficultImage Quality AssessmentMSELPSNR210log10=
  • • VQEG (video quality experts group)1. Goal: recommend video quality assessment standards(TV, telecommunication, multimedia industries)2. Hundreds of experts(Intel, Philips, Sarnoff, Tektronix, AT&T, NHK, NASA,Mitsubishi, NTIA, NIST, Nortel ……)• Testing methodology1. Provide test video sequences2.  Subjective evaluation3.  Objective evaluation by VQEG proponents4. Compare subjective/objective results, find winnerVQEG (1)
  • • Current Status1. Phase I test (2000): Diverse types of distortions 10 proponents including PSNR no winner, 8~9 proponents statistically equivalent,including PSNR!2. Phase II test (2003): Restricted types of distortions (MPEG) Result: A few models slightly better than PSNR3. VQEG is extending their directions: FR/RR/NR, Low Bit Rate Multimedia: video, audio and speech …VQEG (2)
  • • Representative work– Pioneering work [Mannos & Sakrison ’74]– Sarnoff model [Lubin ’93]– Visible difference predictor [Daly ’93]– Perceptual image distortion [Teo & Heeger ’94]– DCT-based method [Watson ’93]– Wavelet-based method [Safranek ’89, Watson et al. ’97]Philosophydistorted signal = reference signal + error signalAssume reference signal has perfect qualityQuantify perceptual error visibilityStandard IQA Model: Error Visibility (1)
  • • MotivationSimulate relevant early HVS componentsStandard IQA Model: Error Visibility (2)R e f e r e n c es ig n a lD is t o r t e ds ig n a lQ u a lit y /D is t o r t io nM e a s u r eC h a n n e lD e c o m p o s it io nE r r o rN o r m a liz a t io n...E r r o rP o o lin gP r e -p r o c e s s in g...• Key featuresChannel decomposition  linear frequency/orientation transformsFrequency weighting  contrast sensitivity functionMasking  intra/inter channel interactionββ/1, = ∑∑l kkleE
  • Standard IQA Model: Error Visibility (3)• Quality definition problem– Error visibility = quality ?• The suprathreshold problem– Based on threshold psychophysics– Generalize to suprathreshold range?• The natural image complexity problem– Based on simple-pattern psychophysics– Generalize to complex natural images?[Wang, et al., “Why is image quality assessment so difficult?” ICASSP ’02][Wang, et al., IEEE Trans. Image Processing, ’04]
  • New Paradigm: Structural Similarity• How to define structural information?• How to separate structural/nonstructural information?PhilosophyPurpose of human vision: extract structural informationHVS is highly adapted for this purposeEstimate structural information changeClassical philosophy New philosophyBottom-up Top-downPredict Error Visibility Predict Structural Distortion
  • ++_d i s t o r t e di m a g eo r i g i n a li m a g eSeparation of Structural/nonstructural Distortion
  • ++_s t r u c t u r a ld i s t o r t i o nd i s t o r t e di m a g eo r i g i n a li m a g en o n s t r u c t u r a ld i s t o r t i o nSeparation of Structural/nonstructural Distortion
  • Separation of Structural/nonstructural Distortion++_s t r u c t u r a ld i s t o r t i o n+d i s t o r t e di m a g eo r i g i n a li m a g en o n s t r u c t u r a ld i s t o r t i o n
  • ++_s t r u c t u r a ld i s t o r t i o n+d i s t o r t e di m a g eo r i g i n a li m a g e+n o n s t r u c t u r a ld i s t o r t i o nSeparation of Structural/nonstructural Distortion
  • ++_s t r u c t u r a ld i s t o r t i o n+d i s t o r t e di m a g eo r i g i n a li m a g e +n o n s t r u c t u r a ld i s t o r t i o nAdaptive Linear System
  • ++_= + +. . .. . .s t r u c t u r a ld i s t o r t i o n+d i s t o r t e di m a g eo r i g i n a li m a g e= + ++n o n s t r u c t u r a ld i s t o r t i o nc K + 1.c 1.c K + 2.c 2.c M.c K.++n o n s t r u c t u r a l d i s t o r t i o nc o m p o n e n t ss t r u c t u r a l d i s t o r t i o nc o m p o n e n t sAdaptive Linear System
  • ++_+ +. . .. . .d i s t o r t e di m a g eo r i g i n a li m a g e+ + +c K + 1.c 1.c K + 2.c 2.c M.c K.++n o n s t r u c t u r a l d i s t o r t i o nc o m p o n e n t ss t r u c t u r a l d i s t o r t i o nc o m p o n e n t sAdaptive Linear System=overcomplete, adaptive basis in the space of all images[Wang & Simoncelli, ICIP ’05, submitted]
  • ikjxx i + x j + x k = 0x - xOlu m in a n c ec h a n g ec o n t r a s tc h a n g es t r u c t u r a lc h a n g ex i = x j = x k),(),(),(),( yxyxyxyx sclSSIM ⋅⋅=12212),(CClyxyx+++=µµµµyx22222),(CCcyxyx+++=σσσσyx33),(CCsyxxy++=σσσyxStructural Similarity (SSIM) Index in Image Space[Wang & Bovik, IEEE Signal Processing Letters, ’02][Wang et al., IEEE Trans. Image Processing, ’04]
  • O O OO OMinkowski (MSE) component-weighted magnitude-weightedmagnitude and component-weighted SSIMModel Comparison
  • originalimageJPEG2000compressedimageabsoluteerror mapSSIM indexmap
  • originalimageGaussiannoisecorruptedimageabsoluteerror mapSSIM indexmap
  • originalimageJPEGcompressedimageabsoluteerror mapSSIM indexmap
  • MSE=0, MSSIM=1 MSE=225, MSSIM=0.949 MSE=225, MSSIM=0.989MSE=215, MSSIM=0.671 MSE=225, MSSIM=0.688 MSE=225, MSSIM=0.723Demo Images
  • Validation LIVE DatabasePSNR MSSIM0.4 0.5 0.6 0.7 0.8 0.9 10102030405060708090100MSSIM (Gaussian window, K1 = 0.01, K2 = 0.03)MOSJPEG imagesJPEG2000 imagesFitting with Logistic Function15 20 25 30 35 40 45 500102030405060708090100PSNRMOSJPEG imagesJPEG2000 imagesFitting with Logistic FunctionDataset JP2(1) JP2(2) JPG(1) JPG(2) Noise Blur Error# of images 87 82 87 88 145 145 145PSNR 0.934 0.895 0.902 0.914 0.987 0.774 0.881SSIM 0.968 0.967 0.965 0.986 0.971 0.936 0.944
  • o r ig in a l im a g ein it ia l im a g ein it ia l d is t o r t io nMAD Competition: MSE vs. SSIM (1)[Wang & Simoncelli, Human Vision and Electronic Imaging, ’04]
  • w o r s t S S I M f o rf ix e d M S Eb e s t S S I M f o rf ix e d M S Eo r ig in a l im a g eMAD Competition: MSE vs. SSIM (2)[Wang & Simoncelli, Human Vision and Electronic Imaging, ’04]
  • w o r s t M S E f o rf ix e d S S I Mb e s t M S E f o rf ix e d S S I Mo r ig in a l im a g eMAD Competition: MSE vs. SSIM (3)[Wang & Simoncelli, Human Vision and Electronic Imaging, ’04]
  • w o r s t S S I M f o rf ix e d M S Eb e s t S S I M f o rf ix e d M S Ew o r s t M S E f o rf ix e d S S I Mb e s t M S E f o rf ix e d S S I Mo r ig in a l im a g eMAD Competition: MSE vs. SSIM (4)[Wang & Simoncelli, Human Vision and Electronic Imaging, ’04]
  • o r ig in a l im a g ein it ia l d is t o r t e dim a g eb e s t S S I M f o rf ix e d M S Ew o r s t S S I M f o rf ix e d M S Eb e s t M S E f o rf ix e d S S I Mw o r s t M S E f o rf ix e d S S I M
  • o r ig in a l im a g ein it ia l d is t o r t e dim a g eb e s t S S I M f o rf ix e d M S Ew o r s t S S I M f o rf ix e d M S Eb e s t M S E f o rf ix e d S S I Mw o r s t M S E f o rf ix e d S S I M
  • • Color image quality assessment• Video quality assessment• Multi-scale SSIM• Complex wavelet SSIMExtensions of SSIM (1)[Wang, et al., Signal Processing: Image Communication, ’04][Wang, et al., Invited Paper, IEEE Asilomar Conf. ’03][Wang & Simoncelli, ICASSP ’05][Toet & Lucassen., Displays, ’03]
  • Extensions of SSIM (2)CccCccyxyx+++⋅=∑∑∑22*2),(SSIM yx: complex waveletcoefficients in imagesx and y• Complex wavelet SSIM– Motivation: robust totranslation, rotation and scaling[Wang & Simoncelli, ICASSP ’05]yx cc ,c o m p l e x w a v e l e tt r a n s f o r m
  • Correct Recognition Rate:MSE: 59.6%; SSIM: 46.9%; Complex wavelet SSIM: 97.7%Database: 2430 imagesStandard patterns: 10 imagesImage Matching without Registration[Wang & Simoncelli, ICASSP ’05]
  • Using SSIMWeb site: www.cns.nyu.edu/~lcv/ssim/SSIM Paper: 11,000+ downloads; Matlab code: 2400+ downloadsIndustrial implementation: http://perso.wanadoo.fr/reservoir/• Image/video coding and communications– Image/video transmission, streaming & robustness [Kim & Kaveh ’02,Halbach & Olsen ’04, Lin et al. ’04, Leontaris & Reibman ’05]– Image/video compression [Blanch et al. ’04, Dikici et al. ’04 , Ho et al. ‘03,Militzer et al. ’03]– High dynamic range video coding [Mantiuk et al. ’04]– Motion estimation/compensation [Monmarthe ’04]• Biomedical image processing– Microarray image processing for bioinformatics [Wang et al. ’03]– Image fusion of CT and MRI images [Piella & Heijmans ’03, Piella ‘04]– Molecular image processing [Ling et al. ’02]– Medical image quality analysis [Chen et al. ’04]
  • Using SSIM (continued)• Watermarking/data hiding [Alattar ’03, Noore et al. ’04, Macq et al. ‘04Zhang & Wang ’05, Kumsawat et al. ‘04]• Image denoising [Park & Lee ’04, Yang & Fox ’04 , Huang et al. ’05Roth & Black ’05, Hirakawa & Parks ’05]• Image enhancement [Battiato et al. ’03]• Image/video hashing [Coskun & Sankur ’04, Hsu & Lu ‘04]• Image rendering [Bornik et al. ’03]• Image fusion [Zheng et al. ’04, Tsai ’04, Gonzalez-Audicana et al. ’05]• Texture reconstruction [Toth ’04]• Image halftoning [Evans & Monga ’03, Neelamani ‘03]• Radar imaging [Bentabet ’03]• Infrared imaging [Torres ’03, Pezoa et al. ‘04]• Ultrasound imaging [Loizou et al. ’04]• Vision processor design [Cembrano et al., ’04]• Wearable display design [von Waldkirch et al. ’04]• Contrast equalization for LCD [Iranli et al. ’05]• Airborne hyperspectral imaging [Christophe et al. ’05]• Superresolution for remote sensing [Rubert et al. ’05]
  • THE ENDThank you!