SlideShare a Scribd company logo
1 of 17
Download to read offline
FIRE 2015 track on the
Automatic illustration of children’s short stories
Iacer Calixto1 and Debasis Ganguly1
1Dublin City University
ADAPT Centre
icalixto@computing.dcu.ie
November 27, 2015
Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 1 / 17
Overview
1 Corpus
2 FIRE’15 track
3 Conclusions
Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 2 / 17
Corpus creation (i)
27 total children’s short stories.
All stories are Aesop’s fables.
Human annotations with important entities, actions and events in
each story.
entity: noun or noun phrase;
main character, important place.
action: verb or verb phrase;
event: combination of entities and actions;
most important happenings and facts in story.
Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 3 / 17
Corpus creation (ii)
(i) Wikipedia ImageCLEF 2010 dataset [1].
(ii) Built text index using (i)’s image captions in
English, French and German.
(iii) Built image pool for each story built from baseline retrieval against
(ii) using entities, actions and events.
(iv) Built relevance judgements for (iii).
Each image mapped to discrete interval [0, 1, 2, 3, 4].
0: image is completely irrelevant to story and
4: image perfectly illustrates story.
Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 4 / 17
Once upon a time...
The Fox and the Crow
A Fox once saw a Crow fly off with a piece of cheese in its beak and settle
on a branch of a tree.
”That’s for me, as I am a Fox,” said Master Reynard, and he walked up to
the foot of the tree.
(...)
Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 5 / 17
Example :: entities
The Fox and the Crow
A Fox once saw a Crow fly off with a piece of cheese in its beak and
settle on a branch of a tree.
”That’s for me, as I am a Fox,” said Master Reynard, and he walked up to
the foot of the tree.
(...)
Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 6 / 17
Example :: actions
The Fox and the Crow
A Fox once saw a Crow fly off with a piece of cheese in its beak and settle
on a branch of a tree.
”That’s for me, as I am a Fox,” said Master Reynard, and he walked up
to the foot of the tree.
(...)
Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 7 / 17
Example :: events
The Fox and the Crow
A Fox once saw a Crow fly off with a piece of cheese in its beak and
settle on a branch of a tree.
”That’s for me, as I am a Fox,” said Master Reynard, and he walked up to
the foot of the tree.
(...)
Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 8 / 17
Example :: illustration
IN a field one summer’s day a Grasshopper was hopping about, chirping and
singing to its heart’s content. An Ant passed by, bearing along with great toil
an ear of corn he was taking to the nest. “Why not come and chat with me, said
the Grasshopper, “instead of toiling and moiling in that way?” “I am helping
to lay up food for the winter,” said the Ant, “and recommend you to do the
same.” “Why bother about winter?” said the Grasshopper; “we have got plenty
of food at present.” But the Ant went on its way and continued its toil. When
the winter came the Grasshopper had no food, and found itself dying of hunger,
while it saw the ants distributing every day corn and grain from the stores they
had collected in the summer. Then the Grasshopper knew: “IT IS BEST TO
PREPARE FOR THE DAYS OF NECESSITY.”
Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 9 / 17
FIRE’15 track
Goal: given a story and an image dataset,
automatically illustrate the story using images in the dataset.
Motivation: Increased readability and understandability.
Several domains can make use of such technologies
(e.g., news articles).
Two subtasks of increasing difficulty:
(i) Input: story text with human annotations
(entities, actions and events)
and image dataset;
Output: ranked list of images for story illustration.
(ii) Input: story text (no human annotations)
and image dataset;
Output: ranked list of images for story illustration.
Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 10 / 17
FIRE’15 track
Evaluation metrics: Precision-at-K, Recall-at-K, Mean Average Precision
Baselines
whole story as query
whole story as query weighted by TF-IDF
Approach MAP P@5 P@10
Unweighted qry terms 0.0275 0.1048 0.0905
TF-IDF weighted qry terms 0.0529 0.1714 0.1238
Table : Retrieval effectiveness of 2 baselines averaged over 22 stories.
Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 11 / 17
FIRE’15 participants
Two groups submitted runs to the track in total.
Group Affiliation #members
1 Amrita Vishwa Vidyapeetham, Coimbatore, India 3
2 i) Charotar University of Science and Technology,
Anand, India;
4
ii) L.D.R.P. College, Gandhinagar, India;
iii) Gujarat University, Ahmedabad, India.
Table : Participating groups – FIRE’15 Automated Story Illustration track.
Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 12 / 17
FIRE’15 participants
Group 1 used Terrier1 to index image dataset and a
Divergence From Randomness (DFR) retrieval algorithm.
Queries were built from story event annotations only.
Group 2 used Gensim2 to build document vectors for image dataset
and use TF-IDF to measure similarity between queries and images.
Queries are human annotations (entities and events only).
No groups outperformed both baselines.
1
http://terrier.org/
2
https://en.wikipedia.org/wiki/Gensim
Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 13 / 17
FIRE’15 participants
Grp Run Evaluation Metrics
Id Id MAP MRR B-pref P@5
1 1 0.0107† 0.1245† 0.1241 0.0636
2 1 0.0047 0.3708 0.0074 0.1273
2 2 0.0053 0.2997 0.0095 0.1545†
2 3 0.0030 0.2504 0.0065 0.0909
B13 – 0.0275 – – 0.1048
B24 – 0.0529 – – 0.1714
Table : Official results of the FIRE Automated Story Illustration task 2015.
Best overall results in bold, best participants submissions (†).
3
Baseline 1 (query contains all words in story).
4
Baseline 2 (query contains all words in story scaled by TF-IDF).
Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 14 / 17
Conclusions
Corpus of children short stories.
FIRE’15 track on automatic illustration of children’s short stories
2 participants, no results improved on the baseline
Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 15 / 17
References
Adrian Popescu, Theodora Tsikrika, and Jana Kludas.
Overview of the wikipedia retrieval task at imageclef 2010.
In Martin Braschler, Donna Harman, and Emanuele Pianta, editors,
CLEF (Notebook Papers/LABs/Workshops), 2010.
Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 16 / 17
Thank you!
Questions?
Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 17 / 17

More Related Content

Viewers also liked

Presentacion Comportamiento Organizacional1.
Presentacion Comportamiento Organizacional1.Presentacion Comportamiento Organizacional1.
Presentacion Comportamiento Organizacional1.Guadalupe_esca
 
Design and Development Of Automated Examination System.
Design and Development Of Automated Examination System.Design and Development Of Automated Examination System.
Design and Development Of Automated Examination System.Shivakant Dubey
 
Motosiklet neden tehlikeli değildir
Motosiklet neden tehlikeli değildirMotosiklet neden tehlikeli değildir
Motosiklet neden tehlikeli değildirAliAytacBulut
 
Community and Social Service Projects
Community and Social Service ProjectsCommunity and Social Service Projects
Community and Social Service ProjectsShreyas G S
 
Remediación Ambiental de Texaco en Ecuador
Remediación Ambiental de Texaco en EcuadorRemediación Ambiental de Texaco en Ecuador
Remediación Ambiental de Texaco en EcuadorJuicioCrudo
 

Viewers also liked (12)

Presentación FCO
Presentación FCOPresentación FCO
Presentación FCO
 
Business Briefing
Business BriefingBusiness Briefing
Business Briefing
 
Martin pêcheur
Martin pêcheurMartin pêcheur
Martin pêcheur
 
Faucon pèlerin
Faucon pèlerinFaucon pèlerin
Faucon pèlerin
 
Lézard ocellé
Lézard ocelléLézard ocellé
Lézard ocellé
 
Presentacion Comportamiento Organizacional1.
Presentacion Comportamiento Organizacional1.Presentacion Comportamiento Organizacional1.
Presentacion Comportamiento Organizacional1.
 
La perdrix rouge
La perdrix rougeLa perdrix rouge
La perdrix rouge
 
Aigle royal
Aigle royalAigle royal
Aigle royal
 
Design and Development Of Automated Examination System.
Design and Development Of Automated Examination System.Design and Development Of Automated Examination System.
Design and Development Of Automated Examination System.
 
Motosiklet neden tehlikeli değildir
Motosiklet neden tehlikeli değildirMotosiklet neden tehlikeli değildir
Motosiklet neden tehlikeli değildir
 
Community and Social Service Projects
Community and Social Service ProjectsCommunity and Social Service Projects
Community and Social Service Projects
 
Remediación Ambiental de Texaco en Ecuador
Remediación Ambiental de Texaco en EcuadorRemediación Ambiental de Texaco en Ecuador
Remediación Ambiental de Texaco en Ecuador
 

FIRE_StoryIllustration_Iacer_Draft_1

  • 1. FIRE 2015 track on the Automatic illustration of children’s short stories Iacer Calixto1 and Debasis Ganguly1 1Dublin City University ADAPT Centre icalixto@computing.dcu.ie November 27, 2015 Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 1 / 17
  • 2. Overview 1 Corpus 2 FIRE’15 track 3 Conclusions Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 2 / 17
  • 3. Corpus creation (i) 27 total children’s short stories. All stories are Aesop’s fables. Human annotations with important entities, actions and events in each story. entity: noun or noun phrase; main character, important place. action: verb or verb phrase; event: combination of entities and actions; most important happenings and facts in story. Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 3 / 17
  • 4. Corpus creation (ii) (i) Wikipedia ImageCLEF 2010 dataset [1]. (ii) Built text index using (i)’s image captions in English, French and German. (iii) Built image pool for each story built from baseline retrieval against (ii) using entities, actions and events. (iv) Built relevance judgements for (iii). Each image mapped to discrete interval [0, 1, 2, 3, 4]. 0: image is completely irrelevant to story and 4: image perfectly illustrates story. Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 4 / 17
  • 5. Once upon a time... The Fox and the Crow A Fox once saw a Crow fly off with a piece of cheese in its beak and settle on a branch of a tree. ”That’s for me, as I am a Fox,” said Master Reynard, and he walked up to the foot of the tree. (...) Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 5 / 17
  • 6. Example :: entities The Fox and the Crow A Fox once saw a Crow fly off with a piece of cheese in its beak and settle on a branch of a tree. ”That’s for me, as I am a Fox,” said Master Reynard, and he walked up to the foot of the tree. (...) Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 6 / 17
  • 7. Example :: actions The Fox and the Crow A Fox once saw a Crow fly off with a piece of cheese in its beak and settle on a branch of a tree. ”That’s for me, as I am a Fox,” said Master Reynard, and he walked up to the foot of the tree. (...) Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 7 / 17
  • 8. Example :: events The Fox and the Crow A Fox once saw a Crow fly off with a piece of cheese in its beak and settle on a branch of a tree. ”That’s for me, as I am a Fox,” said Master Reynard, and he walked up to the foot of the tree. (...) Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 8 / 17
  • 9. Example :: illustration IN a field one summer’s day a Grasshopper was hopping about, chirping and singing to its heart’s content. An Ant passed by, bearing along with great toil an ear of corn he was taking to the nest. “Why not come and chat with me, said the Grasshopper, “instead of toiling and moiling in that way?” “I am helping to lay up food for the winter,” said the Ant, “and recommend you to do the same.” “Why bother about winter?” said the Grasshopper; “we have got plenty of food at present.” But the Ant went on its way and continued its toil. When the winter came the Grasshopper had no food, and found itself dying of hunger, while it saw the ants distributing every day corn and grain from the stores they had collected in the summer. Then the Grasshopper knew: “IT IS BEST TO PREPARE FOR THE DAYS OF NECESSITY.” Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 9 / 17
  • 10. FIRE’15 track Goal: given a story and an image dataset, automatically illustrate the story using images in the dataset. Motivation: Increased readability and understandability. Several domains can make use of such technologies (e.g., news articles). Two subtasks of increasing difficulty: (i) Input: story text with human annotations (entities, actions and events) and image dataset; Output: ranked list of images for story illustration. (ii) Input: story text (no human annotations) and image dataset; Output: ranked list of images for story illustration. Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 10 / 17
  • 11. FIRE’15 track Evaluation metrics: Precision-at-K, Recall-at-K, Mean Average Precision Baselines whole story as query whole story as query weighted by TF-IDF Approach MAP P@5 P@10 Unweighted qry terms 0.0275 0.1048 0.0905 TF-IDF weighted qry terms 0.0529 0.1714 0.1238 Table : Retrieval effectiveness of 2 baselines averaged over 22 stories. Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 11 / 17
  • 12. FIRE’15 participants Two groups submitted runs to the track in total. Group Affiliation #members 1 Amrita Vishwa Vidyapeetham, Coimbatore, India 3 2 i) Charotar University of Science and Technology, Anand, India; 4 ii) L.D.R.P. College, Gandhinagar, India; iii) Gujarat University, Ahmedabad, India. Table : Participating groups – FIRE’15 Automated Story Illustration track. Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 12 / 17
  • 13. FIRE’15 participants Group 1 used Terrier1 to index image dataset and a Divergence From Randomness (DFR) retrieval algorithm. Queries were built from story event annotations only. Group 2 used Gensim2 to build document vectors for image dataset and use TF-IDF to measure similarity between queries and images. Queries are human annotations (entities and events only). No groups outperformed both baselines. 1 http://terrier.org/ 2 https://en.wikipedia.org/wiki/Gensim Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 13 / 17
  • 14. FIRE’15 participants Grp Run Evaluation Metrics Id Id MAP MRR B-pref P@5 1 1 0.0107† 0.1245† 0.1241 0.0636 2 1 0.0047 0.3708 0.0074 0.1273 2 2 0.0053 0.2997 0.0095 0.1545† 2 3 0.0030 0.2504 0.0065 0.0909 B13 – 0.0275 – – 0.1048 B24 – 0.0529 – – 0.1714 Table : Official results of the FIRE Automated Story Illustration task 2015. Best overall results in bold, best participants submissions (†). 3 Baseline 1 (query contains all words in story). 4 Baseline 2 (query contains all words in story scaled by TF-IDF). Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 14 / 17
  • 15. Conclusions Corpus of children short stories. FIRE’15 track on automatic illustration of children’s short stories 2 participants, no results improved on the baseline Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 15 / 17
  • 16. References Adrian Popescu, Theodora Tsikrika, and Jana Kludas. Overview of the wikipedia retrieval task at imageclef 2010. In Martin Braschler, Donna Harman, and Emanuele Pianta, editors, CLEF (Notebook Papers/LABs/Workshops), 2010. Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 16 / 17
  • 17. Thank you! Questions? Calixto and Ganguly (DCU, ADAPT Centre) November 27, 2015 17 / 17