jean-philippe moreux

1. Image Retrieval:
State of the Art at the BnF
Jean-Philippe Moreux
DSR/DCP

2. CONTENT
TECHNICS
Fouille
USE CASES
SERVICES
• Gallica
• Mandragore
• Banque
d’image
…

3. Which content?
• Diverses document genres, artistic/print technics
• All time periods and fields of knowledge
• Heterogeneous metadata

4. Which content?

5. Use cases?
General audience Researchers Information Retrieval

6. In-house Use cases?
Iconographic service (DIP) Digital mediation

7. In-house Use cases?
Catalogs curation, digitization workflow
btv1b90055240
btv1b9005519p
btv1b9003068t
btv1b8413056n
btv1b84130520
Catalog
curation

8. In-house Use cases?
Digitization
workflow
No metadata available!

9. Which technics?
1. Where are the images?
(segmentation)
2. What is their content? (visual recognition)
3. How to look into this content?
(information retrieval)

10. 1. Use the segmentation contained in the OCR (when
available: monographs, newspapers)
2. Segment the image (e.g. with supervised learning)
1. Segment

11. Image classification
(1/n)
2. Index
Multiclasses image
classification (m/n)
Object detection Instance
detection
Semantic
segmentation
Digital signature Classification (0/1)

12. Today, deep learning makes the feature extraction phase unnecessary
(the artificial neural network takes care of it)
Which tools for indexing?
Yesterday, visual feature models and learning algorithms....

13. Generic pretrained models Transfert learning
Software as a Service Deep learning
frameworks
Ad hoc models:
(« dog » ou « griffo » ?)
Applications
?
Which tools for indexing?
Deep learning approaches, with…
and tools

14. • In the digital signatures space: [12,5,87,54…]
• In the (textual) labels of classes: "aircraft", "pilot", "biplane"...
• In the visual characteristics: blue, large size, contrast....
• With several indexing (multimodal search): "fish" + "red“ + “drawing”
3. Search
Similarity search Semantic search

15. Select Segment Index QA Use
Search
Disseminate
APIs,
datasets
Access
The workflow

16. http://patrimeph.ensea.fr/retin/
ETIS/BnF, projet Patrimeph (2018)
3,000 images (Gallica twits)
1. Visual similarity search
2. Human in the loop for accessing
the results relevance

17. • Picard, D., Gosselin, P.-H., Gaspard, M.-C., “Challenged in Content-Based Image Indexing of
Cultural Heritage Collections”. IEEE Signal Processing Magazine, Institute of Electrical and
Electronics Engineers, 2015, 32 (4), pp. 95-102
• Évaluation de descripteurs visuels pour l’annotation automatique d’images patrimoniales,
David PICARD, Philippe-Henri GOSSELIN, ETIS UMR 8051
Collaboration BnF/DRE and ETIS lab (2015)
3,000 annotated images
(400 concepts, 6 categories :
visual, physical, historical…)
1. Global Descriptors (e.g. color and textures
histograms)
2. Aggregation of local descriptors
3. Deep learning (CNN 8-19 layers)
• Best results : deep learning
• Too much concepts, too much diversity, too few images

18. • Indexing by visual and structural similarities
• Search by sample image
• Search API (e.g. by color)
GallicaSimilitudes (BnF, 2017)
http://gallicastudio.bnf.fr/gallicaimages/
770k images
(coll. Spécialisées)
• Global descriptors computed on the dataset
• FCTH (Fuzzy Color and Texture Histogram) footprint
calculation and Tanimoto similarity measurement
• Good results on color and overall structure criteria
• No consideration of shapes
• Complementary to "semantic" approaches

19. • Iconographic retrieval in encyclopedic collections
• Multimodal search (metadata, text, image)
• Creation of iconographic datasets for visual studies
• IIIF driven
GallicaPix (BnF, 2017)
http://demo14-18.bnf.fr:8984/rest?run=findIllustrations-form.xq
220k images from 500k pages
(print, images, maps…).
Topic: WW1
• Data mining of the collections (metadata, OCR)
• Visual recognition (IBM, Google, open source models)
• Classification of 12 genres (picture, engraving, drawing,
map, text…) with a CNN trained on 12k BnF illustrations

20. GallicaPix and Numapresse research project
http://demo14-18.bnf.fr:8984/rest?run=findIllustrations-form.xq
• Face and genre recognition on Excelsior and Paris-Match front pages (1910-1920)
• Human correction and annotation (named people/anonymous)
• Statistical analysis of representation
Genres in newspapers illustrations: GallicaPix for the Digital Humanities

21. GallicaPix and DH 2019
http://demo14-18.bnf.fr:8984/rest?run=findIllustrations-form.xq
• DH Hackathon, May 2019, Helsinki : “Newspapers and Capitalism”, focus on ads
• Object detection (Yolo v3) on 60k illustrated ads
• Means of transport analysis (WW1)
Illustrated ads: GallicaPix for the Digital Humanities

22. (à compléter)
• Segmentation for heritage documents: DhSegment (EPFL)
Ecole polytechnique fédérale de Lausanne (2017-2019)
• Creation of learning datasets
(binary mask + source image)
• Convolutional neural networks
(CNN, Resnet 50)
https://drive.google.com/file/d/1QYVRYRW4_chgtatz9cTmxgJg3--LcQYV/view

23. (à compléter)
• Two INRIA teams: ZENITH (Montpellier), LinkedMedia (Rennes)
• Datasets: manuscripts (Mandragore) and WW1
•
INRIA, BnF research project (2018-2020)
• SNOOP (INRIA, INA), Pl@ntnet:
collaborative recognition of plants
• Search in large visual indexes
https://interstices.info/plantnet-un-reseau-et-des-outils-pour-une-recherche-participative
Objectives :
• Human in the loop :
• Relevance evaluation
• Object detection on small datasets
• Visual signatures for heritage content
search in large indexes
This is not a lion

24. 24
Initial
Query
User
selection New query
leveraging the
user inputs
Selection…
INRIA/Zenith
• Tests on 1M images (WW1)
• Iterative retrain of the model
• New classes are created by the user
Good/Wrong
Human in the loop: iterative queries

25. 25
Iterative query to build a « German helmet » detector

26. 26
INRIA/LinkedMedia
• Tests on Mandragore database (manuscripts): zoologie
• Small datasets, visual heterogeneity (1000 BC to XXth c.)
Heritage content, small datasets

27. 27
INRIA/LinkedMedia
• Activation maps
Tests on unsegmented documents

28. 28
INRIA/LinkedMedia
• Faster RCNN
• Manual annotations
Object Detection

29. Object detection
Results

30. 30
Image Retrieval
app
Indexation workflow and
Gallica Pix DB
Others BnF apps
APIs
Datasets
Gallica Digital
repositories
What’s next?