Image Quality Assessment: From Error Visibility to Structural Similarity
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
833
On Slideshare
833
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
14
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Image Quality Assessment: From ErrorVisibility to Structural SimilaritySarbjeet SinghNITTTRSector 26 CHD.
  • 2. MSE=0, MSSIM=1 MSE=225, MSSIM=0.949 MSE=225, MSSIM=0.989MSE=215, MSSIM=0.671 MSE=225, MSSIM=0.688 MSE=225, MSSIM=0.723Motivationoriginal Image
  • 3. Define PerceptualIQA MeasuresOptimize IP Systems &Algorithms “Perceptually”PERCEPTUAL IMAGE PROCESSINGApplication Scope: essentially all IP applicationsimage/video compression, restoration, enhancement,watermarking, displaying, printing …Perceptual Image ProcessingStandard measure (MSE) does not agreewith human visual perceptionWhy?
  • 4. • Goal—Automatically predict perceived image quality• Classification— Full-reference (FR); No-reference (NR); Reduced-reference (RR)• Widely Used Methods—FR: MSE and PSNR—NR & RR: wide open research topic• IQA is DifficultImage Quality AssessmentMSELPSNR210log10=
  • 5. • VQEG (video quality experts group)1. Goal: recommend video quality assessment standards(TV, telecommunication, multimedia industries)2. Hundreds of experts(Intel, Philips, Sarnoff, Tektronix, AT&T, NHK, NASA,Mitsubishi, NTIA, NIST, Nortel ……)• Testing methodology1. Provide test video sequences2.  Subjective evaluation3.  Objective evaluation by VQEG proponents4. Compare subjective/objective results, find winnerVQEG (1)
  • 6. • Current Status1. Phase I test (2000): Diverse types of distortions 10 proponents including PSNR no winner, 8~9 proponents statistically equivalent,including PSNR!2. Phase II test (2003): Restricted types of distortions (MPEG) Result: A few models slightly better than PSNR3. VQEG is extending their directions: FR/RR/NR, Low Bit Rate Multimedia: video, audio and speech …VQEG (2)
  • 7. • Representative work– Pioneering work [Mannos & Sakrison ’74]– Sarnoff model [Lubin ’93]– Visible difference predictor [Daly ’93]– Perceptual image distortion [Teo & Heeger ’94]– DCT-based method [Watson ’93]– Wavelet-based method [Safranek ’89, Watson et al. ’97]Philosophydistorted signal = reference signal + error signalAssume reference signal has perfect qualityQuantify perceptual error visibilityStandard IQA Model: Error Visibility (1)
  • 8. • MotivationSimulate relevant early HVS componentsStandard IQA Model: Error Visibility (2)R e f e r e n c es ig n a lD is t o r t e ds ig n a lQ u a lit y /D is t o r t io nM e a s u r eC h a n n e lD e c o m p o s it io nE r r o rN o r m a liz a t io n...E r r o rP o o lin gP r e -p r o c e s s in g...• Key featuresChannel decomposition  linear frequency/orientation transformsFrequency weighting  contrast sensitivity functionMasking  intra/inter channel interactionββ/1, = ∑∑l kkleE
  • 9. Standard IQA Model: Error Visibility (3)• Quality definition problem– Error visibility = quality ?• The suprathreshold problem– Based on threshold psychophysics– Generalize to suprathreshold range?• The natural image complexity problem– Based on simple-pattern psychophysics– Generalize to complex natural images?[Wang, et al., “Why is image quality assessment so difficult?” ICASSP ’02][Wang, et al., IEEE Trans. Image Processing, ’04]
  • 10. New Paradigm: Structural Similarity• How to define structural information?• How to separate structural/nonstructural information?PhilosophyPurpose of human vision: extract structural informationHVS is highly adapted for this purposeEstimate structural information changeClassical philosophy New philosophyBottom-up Top-downPredict Error Visibility Predict Structural Distortion
  • 11. ++_d i s t o r t e di m a g eo r i g i n a li m a g eSeparation of Structural/nonstructural Distortion
  • 12. ++_s t r u c t u r a ld i s t o r t i o nd i s t o r t e di m a g eo r i g i n a li m a g en o n s t r u c t u r a ld i s t o r t i o nSeparation of Structural/nonstructural Distortion
  • 13. Separation of Structural/nonstructural Distortion++_s t r u c t u r a ld i s t o r t i o n+d i s t o r t e di m a g eo r i g i n a li m a g en o n s t r u c t u r a ld i s t o r t i o n
  • 14. ++_s t r u c t u r a ld i s t o r t i o n+d i s t o r t e di m a g eo r i g i n a li m a g e+n o n s t r u c t u r a ld i s t o r t i o nSeparation of Structural/nonstructural Distortion
  • 15. ++_s t r u c t u r a ld i s t o r t i o n+d i s t o r t e di m a g eo r i g i n a li m a g e +n o n s t r u c t u r a ld i s t o r t i o nAdaptive Linear System
  • 16. ++_= + +. . .. . .s t r u c t u r a ld i s t o r t i o n+d i s t o r t e di m a g eo r i g i n a li m a g e= + ++n o n s t r u c t u r a ld i s t o r t i o nc K + 1.c 1.c K + 2.c 2.c M.c K.++n o n s t r u c t u r a l d i s t o r t i o nc o m p o n e n t ss t r u c t u r a l d i s t o r t i o nc o m p o n e n t sAdaptive Linear System
  • 17. ++_+ +. . .. . .d i s t o r t e di m a g eo r i g i n a li m a g e+ + +c K + 1.c 1.c K + 2.c 2.c M.c K.++n o n s t r u c t u r a l d i s t o r t i o nc o m p o n e n t ss t r u c t u r a l d i s t o r t i o nc o m p o n e n t sAdaptive Linear System=overcomplete, adaptive basis in the space of all images[Wang & Simoncelli, ICIP ’05, submitted]
  • 18. ikjxx i + x j + x k = 0x - xOlu m in a n c ec h a n g ec o n t r a s tc h a n g es t r u c t u r a lc h a n g ex i = x j = x k),(),(),(),( yxyxyxyx sclSSIM ⋅⋅=12212),(CClyxyx+++=µµµµyx22222),(CCcyxyx+++=σσσσyx33),(CCsyxxy++=σσσyxStructural Similarity (SSIM) Index in Image Space[Wang & Bovik, IEEE Signal Processing Letters, ’02][Wang et al., IEEE Trans. Image Processing, ’04]
  • 19. O O OO OMinkowski (MSE) component-weighted magnitude-weightedmagnitude and component-weighted SSIMModel Comparison
  • 20. originalimageJPEG2000compressedimageabsoluteerror mapSSIM indexmap
  • 21. originalimageGaussiannoisecorruptedimageabsoluteerror mapSSIM indexmap
  • 22. originalimageJPEGcompressedimageabsoluteerror mapSSIM indexmap
  • 23. MSE=0, MSSIM=1 MSE=225, MSSIM=0.949 MSE=225, MSSIM=0.989MSE=215, MSSIM=0.671 MSE=225, MSSIM=0.688 MSE=225, MSSIM=0.723Demo Images
  • 24. Validation LIVE DatabasePSNR MSSIM0.4 0.5 0.6 0.7 0.8 0.9 10102030405060708090100MSSIM (Gaussian window, K1 = 0.01, K2 = 0.03)MOSJPEG imagesJPEG2000 imagesFitting with Logistic Function15 20 25 30 35 40 45 500102030405060708090100PSNRMOSJPEG imagesJPEG2000 imagesFitting with Logistic FunctionDataset JP2(1) JP2(2) JPG(1) JPG(2) Noise Blur Error# of images 87 82 87 88 145 145 145PSNR 0.934 0.895 0.902 0.914 0.987 0.774 0.881SSIM 0.968 0.967 0.965 0.986 0.971 0.936 0.944
  • 25. o r ig in a l im a g ein it ia l im a g ein it ia l d is t o r t io nMAD Competition: MSE vs. SSIM (1)[Wang & Simoncelli, Human Vision and Electronic Imaging, ’04]
  • 26. w o r s t S S I M f o rf ix e d M S Eb e s t S S I M f o rf ix e d M S Eo r ig in a l im a g eMAD Competition: MSE vs. SSIM (2)[Wang & Simoncelli, Human Vision and Electronic Imaging, ’04]
  • 27. w o r s t M S E f o rf ix e d S S I Mb e s t M S E f o rf ix e d S S I Mo r ig in a l im a g eMAD Competition: MSE vs. SSIM (3)[Wang & Simoncelli, Human Vision and Electronic Imaging, ’04]
  • 28. w o r s t S S I M f o rf ix e d M S Eb e s t S S I M f o rf ix e d M S Ew o r s t M S E f o rf ix e d S S I Mb e s t M S E f o rf ix e d S S I Mo r ig in a l im a g eMAD Competition: MSE vs. SSIM (4)[Wang & Simoncelli, Human Vision and Electronic Imaging, ’04]
  • 29. o r ig in a l im a g ein it ia l d is t o r t e dim a g eb e s t S S I M f o rf ix e d M S Ew o r s t S S I M f o rf ix e d M S Eb e s t M S E f o rf ix e d S S I Mw o r s t M S E f o rf ix e d S S I M
  • 30. o r ig in a l im a g ein it ia l d is t o r t e dim a g eb e s t S S I M f o rf ix e d M S Ew o r s t S S I M f o rf ix e d M S Eb e s t M S E f o rf ix e d S S I Mw o r s t M S E f o rf ix e d S S I M
  • 31. • Color image quality assessment• Video quality assessment• Multi-scale SSIM• Complex wavelet SSIMExtensions of SSIM (1)[Wang, et al., Signal Processing: Image Communication, ’04][Wang, et al., Invited Paper, IEEE Asilomar Conf. ’03][Wang & Simoncelli, ICASSP ’05][Toet & Lucassen., Displays, ’03]
  • 32. Extensions of SSIM (2)CccCccyxyx+++⋅=∑∑∑22*2),(SSIM yx: complex waveletcoefficients in imagesx and y• Complex wavelet SSIM– Motivation: robust totranslation, rotation and scaling[Wang & Simoncelli, ICASSP ’05]yx cc ,c o m p l e x w a v e l e tt r a n s f o r m
  • 33. Correct Recognition Rate:MSE: 59.6%; SSIM: 46.9%; Complex wavelet SSIM: 97.7%Database: 2430 imagesStandard patterns: 10 imagesImage Matching without Registration[Wang & Simoncelli, ICASSP ’05]
  • 34. Using SSIMWeb site: www.cns.nyu.edu/~lcv/ssim/SSIM Paper: 11,000+ downloads; Matlab code: 2400+ downloadsIndustrial implementation: http://perso.wanadoo.fr/reservoir/• Image/video coding and communications– Image/video transmission, streaming & robustness [Kim & Kaveh ’02,Halbach & Olsen ’04, Lin et al. ’04, Leontaris & Reibman ’05]– Image/video compression [Blanch et al. ’04, Dikici et al. ’04 , Ho et al. ‘03,Militzer et al. ’03]– High dynamic range video coding [Mantiuk et al. ’04]– Motion estimation/compensation [Monmarthe ’04]• Biomedical image processing– Microarray image processing for bioinformatics [Wang et al. ’03]– Image fusion of CT and MRI images [Piella & Heijmans ’03, Piella ‘04]– Molecular image processing [Ling et al. ’02]– Medical image quality analysis [Chen et al. ’04]
  • 35. Using SSIM (continued)• Watermarking/data hiding [Alattar ’03, Noore et al. ’04, Macq et al. ‘04Zhang & Wang ’05, Kumsawat et al. ‘04]• Image denoising [Park & Lee ’04, Yang & Fox ’04 , Huang et al. ’05Roth & Black ’05, Hirakawa & Parks ’05]• Image enhancement [Battiato et al. ’03]• Image/video hashing [Coskun & Sankur ’04, Hsu & Lu ‘04]• Image rendering [Bornik et al. ’03]• Image fusion [Zheng et al. ’04, Tsai ’04, Gonzalez-Audicana et al. ’05]• Texture reconstruction [Toth ’04]• Image halftoning [Evans & Monga ’03, Neelamani ‘03]• Radar imaging [Bentabet ’03]• Infrared imaging [Torres ’03, Pezoa et al. ‘04]• Ultrasound imaging [Loizou et al. ’04]• Vision processor design [Cembrano et al., ’04]• Wearable display design [von Waldkirch et al. ’04]• Contrast equalization for LCD [Iranli et al. ’05]• Airborne hyperspectral imaging [Christophe et al. ’05]• Superresolution for remote sensing [Rubert et al. ’05]
  • 36. THE ENDThank you!