Every day millions of digital images are edited to be displayed on different devices (e.g. desktop, tablet, mobile). The most common way for adapting images to different aspect ratios is to crop and scale such images. Automatically get optimal crops is a challenging problem due to the uniqueness of each image. Smart cropping algorithms are demanded to automatically choose the best crop that preserves the content and aesthetics of the image. While content can be analysed using saliency maps, aesthetics rating is a highly subjective task that makes difficult identifying a set of aesthetics attributes that can be expressed algorithmically. However, recently major advances have taken place in state-of-the-art aesthetics prediction algorithms with Convolutional Neural Networks (CNNs). Based on these advances we present the smart cropping service developed in Abraia Software to provide automatic art-directed responsive images on modern web designs. Our solution combines saliency and aesthetics models to provide a cloud service able to smartly crop and resize images and photos on-the-fly.
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Automatic Image Cropping based on Aesthetics Prediction using CNNs
1. Automatic Image Cropping based
on Aesthetics Prediction using
CNNs
Perception-inspired tools for the web
Abraia Software SL
https://abraia.me
2. 2
▫ > 1.25 billion websites
▫ >51% mobile web visits
▫ 69% mobile web traffic is images
Some figures
3. 3
The goal is to provide an optimal viewing experience for any
display format.
▫ Adapts the layout to the viewing environment
▫ Multiple layouts for a range of devices
▫ Different aspect ratios
▪ Lots of images must be transformed
Problem: Adapting pictures to different formats is hard
Responsive web design
4. 4
Cloud services to transform and optimize images on-the-fly using
simple URLs.
▫ Smart image retargeting driven by perception
▫ Easy-to-use API to make a site responsive
▫ Retarget image to the many screen formats that enter your
site
▪ On the fly
https://abraia.me/images?w=400&h=200&url=https://images.pexels.com/photos/376
464/pexels-photo-376464.jpeg
▪ In your stock
▫ A perfect fit to media and ecommerce sites
Abraia cloud services
5. 5
Problem
▫ Adapting images to different displays
▫ How to crop lots of images to different sizes?
Automatic image cropping
Solution
▫ Crop and resize
▫ Smart cropping algorithms
6. Attention-based approaches
Saliency prediction to detect
the most relevant content.
Image Content
Automatic cropping schemes
Aesthetics-based approaches
Photographic rules to balance
the composition.
Image Composition
6
Content & Aesthetics
7. 7
1. Visual attention
analysis
A saliency heatmap is
generated
2. Cropping regions
proposal
3. Content preservation
ranking
Best regions are selected
by content
Content
8. 8
1. Aesthetics scoring
Sorts crops by their
aesthetics prediction
2. Final selection and
resize
Aesthetics6.68
6.09
6.31
Best score
Intermediate score
Worst score
9. ▫ Regression model
▪ Predict the average aesthetic score
▪ Based on MobileNet architecture
▫ WMSE as loss function:
▪ Weighted Mean Squared Error (WMSE)
▪ To reduce the bias introduced by the unbalanced
training dataset
▫ AVA dataset
▪ 70% train
▪ 10% validation
▪ 20% test
▫ Optimizer
▪ RMSprop
MSE: 0.45 RMSE: 0.67
9
Aesthetics prediction with CNNsfirst
last
second
11. 11
▫ Aesthetics is a highly subjective attribute
▫ Datasets available are heavily unbalanced and biased
▫ Even though…
▪ Aesthetics regression models improve the quality of
automatic image cropping
Next steps
▫ Prediction of aesthetics ratings regarding specific attributes
▪ e.g. symmetry, lighting, vividness
▫ Automatic image enhancement guided by aesthetics
Conclusions