This document discusses a method for integrating contextual knowledge from knowledge graphs with visual features from images to improve fine art classification. It presents ArtGraph, a knowledge graph created from WikiArt and DBpedia containing contextual information about artworks. A multi-task multi-modal classification model is developed that extracts graph embeddings from ArtGraph to provide contextual information about artworks, which is combined with visual features extracted from images using ResNet50 to improve classification accuracy. Experimental results on artwork images show that incorporating contextual knowledge from the knowledge graph leads to better classification performance compared to using visual features alone.

1. Integrating Contextual Knowledge to
Visual Features for Fine Art Classification
Giovanna Castellano, Giovanni Sansaro, Gennaro Vessio
Department of Computer Science, University of Bari, Italy
gennaro.vessio@uniba.it

2. Context
❏ A large-scale digitization effort has been
made in recent years, which has led to the
increasing availability of large collections
of digitized artworks
❏ This availability, coupled with recent
advances in Deep Learning, has opened up
new opportunities in the field of automatic
art analysis
❏ Among other benefits, support for the use
and study of fine arts with automatic tools
can help art historians and promote the
dissemination of culture
2

3. Motivations
❏ Most existing solutions rely
solely on the “visual features” a
CNN can automatically extract
from digital artwork images
❏ This has been used
successfully for several tasks,
but has led to the neglect of an
enormous amount of
knowledge related to the
“context” of each artwork
❏ Goal: encode this knowledge
into a KG, which can then be
used in conjunction with DL
models to improve the
effectiveness of current
systems
3

4. ArtGraph
❏ We propose an artistic KG
❏ It is based on two sources:
❏ WikiArt
❏ DBpedia
❏ And it has been implemented in
Neo4j
4

5. Web interface
A web interface, written in JavaScript, allows for easy navigation of the graph and
can display the results of queries, written in Cypher, to support art historians
5

6. Multi-Task
Multi-Modal
Classification
6
❏ ArtGraph encodes a valuable
source of contextual knowledge
to integrate with visual features
automatically learned by deep
neural networks
❏ Multi-modal learning: graph
embeddings are extracted from
ArtGraph to provide the
“context” information of the
artwork; this information is
intended to improve the
accuracy of “visual” features
extracted from the artwork
using ResNet50

7. Results
❏ Experiments conducted on Google Colab
❏ Artwork images resized to 224✕224
❏ node2vec of size 128
❏ Graph embeddings were not learned on the entire graph, otherwise a bias
would have been introduced..!
7

8. Conclusion
8
❏ An artistic KG has been proposed primarily intended to provide art historians
with a rich and easy-to-use tool to perform art analysis
❏ Future work: expand the proposed learning model by leveraging the GCN
framework to improve performance

9. Thanks for the
attention!
9