1. FIRE 2015 track on the
Automatic illustration of children’s short stories
Iacer Calixto1 and Debasis Ganguly1
1Dublin City University
ADAPT Centre
icalixto@computing.dcu.ie
November 27, 2015
Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 1 / 17
2. Overview
1 Corpus
2 FIRE’15 track
3 Conclusions
Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 2 / 17
3. Corpus creation (i)
27 total children’s short stories.
All stories are Aesop’s fables.
Human annotations with important entities, actions and events in
each story.
entity: noun or noun phrase;
main character, important place.
action: verb or verb phrase;
event: combination of entities and actions;
most important happenings and facts in story.
Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 3 / 17
4. Corpus creation (ii)
(i) Wikipedia ImageCLEF 2010 dataset [1].
(ii) Built text index using (i)’s image captions in
English, French and German.
(iii) Built image pool for each story built from baseline retrieval against
(ii) using entities, actions and events.
(iv) Built relevance judgements for (iii).
Each image mapped to discrete interval [0, 1, 2, 3, 4].
0: image is completely irrelevant to story and
4: image perfectly illustrates story.
Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 4 / 17
5. Once upon a time...
The Fox and the Crow
A Fox once saw a Crow fly off with a piece of cheese in its beak and settle
on a branch of a tree.
”That’s for me, as I am a Fox,” said Master Reynard, and he walked up to
the foot of the tree.
(...)
Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 5 / 17
6. Example :: entities
The Fox and the Crow
A Fox once saw a Crow fly off with a piece of cheese in its beak and
settle on a branch of a tree.
”That’s for me, as I am a Fox,” said Master Reynard, and he walked up to
the foot of the tree.
(...)
Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 6 / 17
7. Example :: actions
The Fox and the Crow
A Fox once saw a Crow fly off with a piece of cheese in its beak and settle
on a branch of a tree.
”That’s for me, as I am a Fox,” said Master Reynard, and he walked up
to the foot of the tree.
(...)
Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 7 / 17
8. Example :: events
The Fox and the Crow
A Fox once saw a Crow fly off with a piece of cheese in its beak and
settle on a branch of a tree.
”That’s for me, as I am a Fox,” said Master Reynard, and he walked up to
the foot of the tree.
(...)
Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 8 / 17
9. Example :: illustration
IN a field one summer’s day a Grasshopper was hopping about, chirping and
singing to its heart’s content. An Ant passed by, bearing along with great toil
an ear of corn he was taking to the nest. “Why not come and chat with me, said
the Grasshopper, “instead of toiling and moiling in that way?” “I am helping
to lay up food for the winter,” said the Ant, “and recommend you to do the
same.” “Why bother about winter?” said the Grasshopper; “we have got plenty
of food at present.” But the Ant went on its way and continued its toil. When
the winter came the Grasshopper had no food, and found itself dying of hunger,
while it saw the ants distributing every day corn and grain from the stores they
had collected in the summer. Then the Grasshopper knew: “IT IS BEST TO
PREPARE FOR THE DAYS OF NECESSITY.”
Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 9 / 17
10. FIRE’15 track
Goal: given a story and an image dataset,
automatically illustrate the story using images in the dataset.
Motivation: Increased readability and understandability.
Several domains can make use of such technologies
(e.g., news articles).
Two subtasks of increasing difficulty:
(i) Input: story text with human annotations
(entities, actions and events)
and image dataset;
Output: ranked list of images for story illustration.
(ii) Input: story text (no human annotations)
and image dataset;
Output: ranked list of images for story illustration.
Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 10 / 17
11. FIRE’15 track
Evaluation metrics: Precision-at-K, Recall-at-K, Mean Average Precision
Baselines
whole story as query
whole story as query weighted by TF-IDF
Approach MAP P@5 P@10
Unweighted qry terms 0.0275 0.1048 0.0905
TF-IDF weighted qry terms 0.0529 0.1714 0.1238
Table : Retrieval effectiveness of 2 baselines averaged over 22 stories.
Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 11 / 17
12. FIRE’15 participants
Two groups submitted runs to the track in total.
Group Affiliation #members
1 Amrita Vishwa Vidyapeetham, Coimbatore, India 3
2 i) Charotar University of Science and Technology,
Anand, India;
4
ii) L.D.R.P. College, Gandhinagar, India;
iii) Gujarat University, Ahmedabad, India.
Table : Participating groups – FIRE’15 Automated Story Illustration track.
Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 12 / 17
13. FIRE’15 participants
Group 1 used Terrier1 to index image dataset and a
Divergence From Randomness (DFR) retrieval algorithm.
Queries were built from story event annotations only.
Group 2 used Gensim2 to build document vectors for image dataset
and use TF-IDF to measure similarity between queries and images.
Queries are human annotations (entities and events only).
No groups outperformed both baselines.
1
http://terrier.org/
2
https://en.wikipedia.org/wiki/Gensim
Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 13 / 17
14. FIRE’15 participants
Grp Run Evaluation Metrics
Id Id MAP MRR B-pref P@5
1 1 0.0107† 0.1245† 0.1241 0.0636
2 1 0.0047 0.3708 0.0074 0.1273
2 2 0.0053 0.2997 0.0095 0.1545†
2 3 0.0030 0.2504 0.0065 0.0909
B13 – 0.0275 – – 0.1048
B24 – 0.0529 – – 0.1714
Table : Official results of the FIRE Automated Story Illustration task 2015.
Best overall results in bold, best participants submissions (†).
3
Baseline 1 (query contains all words in story).
4
Baseline 2 (query contains all words in story scaled by TF-IDF).
Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 14 / 17
15. Conclusions
Corpus of children short stories.
FIRE’15 track on automatic illustration of children’s short stories
2 participants, no results improved on the baseline
Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 15 / 17
16. References
Adrian Popescu, Theodora Tsikrika, and Jana Kludas.
Overview of the wikipedia retrieval task at imageclef 2010.
In Martin Braschler, Donna Harman, and Emanuele Pianta, editors,
CLEF (Notebook Papers/LABs/Workshops), 2010.
Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 16 / 17