Activity report on Deep-learning based compression
•Download as PPTX, PDF•
0 likes•1,459 views
This presentation illustrates the activity of the Multimedia team in deep-learning-aided image and video compression.
Recommended
More Related Content
What's hot
What's hot (10)
Similar to Activity report on Deep-learning based compression
Similar to Activity report on Deep-learning based compression (20)
Recently uploaded
Recently uploaded (20)
Activity report on Deep-learning based compression
- 1. Institut Mines-Télécom Compression meets Deep Learning: Breakthrough or breakdown? Report on research in DL+Compression @ Multimedia group Marco Cagnazzo, Attilio Fiandrotti, Andrei Purica
- 2. Institut Mines-Télécom Context DL and compression Connexion with compression problems ■ Learn the best choices for classical encoders • E.g., fast mode decision, rate allocation ■ Improve classical tasks of compression algorithms • Probability models • Block prediction • Segmentation for object-based coding • MPEG contributions ■ Paradigm shift in signal representation • Autoencoders, GAN’s 2 IMPACT Decisive Incremental Disruptive
- 3. Institut Mines-Télécom Outline ■ On-going works @ MM group • Airplane screen content video compression • Virtual viewpoint synthesis and super-resolution • Subjective quality comparison of DL-based compression algorithms • Other possible applications ■ The ML-Compression working group ■ Conclusions 3
- 4. Institut Mines-Télécom Airplane screen content video compression ■ Airplane screen content: critical text information embedded over natural image or synthetic background • Sensors, navigation and positioning information, etc • No direct access to these data only to captured screens ■ Compression is required for several use cases ■ Semantic (or object-based) video coding • Text is recognized and encoded as such • Perfect text reconstruction at the decoder side 4
- 5. Institut Mines-Télécom Semantic coding ■ Deep learning is a key component of such schemes since it allows to obtain a reliable detection of the semantic information ■ Three NN architectures tested (complexity vs. accuracy trade-off) ■ First results: up to -90% rate reduction wrt the state of the art for the same quality, or +4.6 dB PSNR improvement ■ Example : HEVC-SCC at 0.018 bpp, 33.1 dB Proposed at 0.007 bpp, 38.2 dB 5
- 6. Institut Mines-Télécom Outline ■ On-going works @ MM group • Airplane screen content video compression • Virtual viewpoint synthesis and super-resolution • Subjective quality comparison of DL-based compression algorithms • Other possible applications ■ The ML-Compression working group ■ Conclusions 6
- 7. Institut Mines-Télécom Virtual viewpoint synthesis 7 x y z Real cameras Virtual viewpoints
- 8. Institut Mines-Télécom View Synthesis Reference Software (MPEG) 8 3D back- projection 3D back- projection Merging Filling holes Reference Homography Matrix Reference Homography Matrix Synthesis Homography Matrix
- 9. Institut Mines-Télécom Proposed scheme 9 3D back- projection 3D back- projection CNN-based merge Reference Homography Matrix Reference Homography Matrix Synthesis Homography Matrix
- 10. Institut Mines-Télécom CNN-based merge Architecture derived from a video super-resolution technique 10 Concatenate Convolutional Layer 1 Convolutional Layer 2
- 11. Institut Mines-Télécom CNN-based view synthesis: results 11 VSRS
- 12. Institut Mines-Télécom CNN-based view synthesis: results 12 Ground truth
- 13. Institut Mines-Télécom CNN-based view synthesis: results 13 Proposed
- 14. Institut Mines-Télécom Outline ■ On-going works • Airplane screen content video compression • Virtual viewpoint synthesis and super-resolution • Subjective quality comparison of DL-based compression algorithms • Other possible applications ■ The ML-Compr working group ■ Conclusions 14
- 15. Institut Mines-Télécom Subjective quality evaluation of DL-compression methods ■ Deep generative models try to learn the latent distribution generating images ■ A typical architecture is based on auto-encoders, i.e. networks trained to reproduce their input ■ Autoencoders include an information bottleneck, achieving compression ■ Very low-bitrate compression could also be obtained with GANs ─ Training process stability? ─ Naturaliness vs. fidelity 15 Encoder Code Decoder 𝑥 L=||𝑥-𝑦||2 𝑦
- 16. Institut Mines-Télécom Subjective quality evaluation of DL-compression methods ■ Subjective quality evaluation (PSNR is not reliable enough) ■ 6 images, 113 compressed stimuli (uniform span of the impairment scale) ■ 23 participants ■ Double stimulus impairment scale ■ Four compression methods: 1. Ballé et al.: 3-layers autoencoder with biologically-inspired non- linearity and an approximation of rate-distortion optimization 2. Toderici et al.: Progressive RNN-based encoder working on 32x32 pixels patches 3. JP2K: Wavelet Transform, RDO, arithmetic coding 4. BPG: Spatial prediction, variable size prediction and transform units, DCT and arithmetic coding 16
- 17. Institut Mines-Télécom Subjective quality evaluation of DL-compression methods 17 Image 1 Image 2
- 18. Institut Mines-Télécom Subjective quality evaluation of DL-compression methods - Image 1 18 Ballé, 0.38 bpp JP2K, 0.43 bpp
- 19. Institut Mines-Télécom Subjective quality evaluation of DL-compression methods - Image 2 19 Toderici, 0.125 bpp JP2K, 0.1 bpp
- 20. Institut Mines-Télécom Outline ■ On-going works • Airplane screen content video compression • Virtual viewpoint synthesis and super-resolution • Subjective quality comparison of DL-based compression algorithms • Other possible applications ■ The ML-Compression working group ■ Conclusions 20
- 21. Institut Mines-Télécom Other applications ■ The flexibility of learning methods make them suitable for several other problems in the field of compression and streaming • Spatial image prediction • Probability distribution estimation for lossless coding • Digital Hologram Compression • HTTP Adaptive streaming (Q-learning) Digicosme post-doc BCOM PhD HUAWEI (CIFRE) PhD
- 22. Institut Mines-Télécom Outline ■ On-going works • Airplane screen content video compression • Virtual viewpoint synthesis and super-resolution • Subjective quality comparison of DL-based compression algorithms • Other possible applications ■ The ML-Compression working group ■ Conclusions 22
- 23. Institut Mines-Télécom ML-Compression working group http://mlcompr.wp.imt.fr ■ Started in 4 months ago ■ People • 2 MdC, 2 Post-doc, 3 PhD, interns @ MM • Possible recruiting of 1-3 PhDs in the next months (CIFRE) • Researchers from L2S (with PhDs and one post-doc) • Contributions from other groups (talks, discussions, …) ■ Regular seminars with contributions from ─ IMAGES and S2A groups ─ Former IDS members • Paris 5, L2S ─ Other universities (Paris13, Poitiers, CentraleSupéléc …) ─ Companies (Orange, Zodiac, …) ■ Make it an “official research topic”(aka “theme”)? ■ Connection with the Learning theme? 23
- 24. Institut Mines-Télécom Outline ■ On-going works • Airplane screen content video compression • Virtual viewpoint synthesis and super-resolution • Subjective quality comparison of DL-based compression algorithms • Other possible applications ■ The ML-Compr working group ■ Conclusions 24
- 25. Institut Mines-Télécom Conclusions ■ Deep learning has triggered a revolution in many fields: will it be the same for compression? • Possible, if we consider the impact on close fields (computer vision) • But not sure: traditional methods still have very important properties that cannot (yet) be guaranteed by DL-based methods (robustness, progressivity, rate-control, low-complexity decoders, …) ■ Will DL provide decisive gains inside traditional architecture? • Possible, but many difficulties have to be faced ■ Or will DL just be used for incremental improvements in traditional architectures? • Almost sure that this minimal target can be achieved 25 IMPACT Decisive Incremental Disruptive
- 26. Institut Mines-Télécom Perspectives ■ Increasing activity of the ML-Compression group • Seminars from AI experts • Growing network of collaborations ■ Industrial activity • 2 or 3 PhD Cifre proposals for next autumn ■ Critical mass? • Intra-department? NewUni? New recruitment? 26
- 27. Institut Mines-Télécom Thank you! Working group: http://mlcompr.wp.imt.fr Next seminar on July 19th Subscribe to the mailing list: https://listes.telecom-paristech.fr/mailman/listinfo/mlcompr 27