As microblogs have become commonplace, recommending relevant hashtags for microblog posts has become increasingly important. However, recommending appropriate hashtags for a post is challenging because it requires a high-level understanding of the context and relationships of the information in the post. In this paper, we propose a novel hashtag recommendation framework that incorporates external knowledge to enrich the context of posts. Using an image of the post, we obtain the hierarchical external knowledge extracted by the Open Directory Project (ODP)-based classifier. Experimental results show that our framework performs better than the baselines on a multimodal hashtag recommendation benchmark dataset. It outperformed the existing state-of-the-art model by providing a 39.86% increase in the average F1-score.

1. 1
EXTRA : Integrating External Knowledge into Multimodal
Hashtag Recommendation System
1 Department of Artificial Intelligence, The Catholic University of Korea, Bucheon, South Korea
2 Department of Data Science, The Catholic University of Korea, Bucheon, South Korea
3 Department of Computer Science and Engineering, Korea University, Seoul, South Korea
Hyun-Sik Won1∗, Su-Min Roh1∗, Dohyun Kim3, Min-Ji Kim1, Huiju Kim3, and Kang-Min Kim12
* Equal contribution

2. Background
2
• Microblogging Platforms
Hashtags allow people to promote their own posts or categorize posts into specific topics.

3. Introduction
• Hashtag Recommendation Task
: Recommend appropriate hashtags related to the post.
• Previous Works
: Recommend hashtags by utilizing explicit information in the post (i.e., text, image, etc.).
: Recently, each user’s hashtag habits1 from the microblog are also utilized to recommend personalized hashtags.
3
1. Chen, Yu-Chi, et al. "Tagnet: triplet-attention graph networks for hashtag recommendation." IEEE Transactions on Circuits and Systems for Video Technology 32.3 (2021): 1148-1159.

4. Introduction
• Hashtag Recommendation Task
: Recommend appropriate hashtags related to the post.
• Previous Works
: Recommend hashtags by utilizing explicit information in the post (i.e., text, image, etc.).
: Recently, each user’s hashtag habits1 from the microblog are also utilized to recommend personalized hashtags.
However, these methods only utilize the information found in the microblog.
 This method may have difficulty understanding the contextual information of the post.
4
1. Chen, Yu-Chi, et al. "Tagnet: triplet-attention graph networks for hashtag recommendation." IEEE Transactions on Circuits and Systems for Video Technology 32.3 (2021): 1148-1159.

5. Introduction
• External Knowledge in Hashtag Recommendation
5

6. Introduction
6
• External Knowledge in Hashtag Recommendation
Does not appear directly
in the text or image.

7. Introduction
7
However, #dogtraining can be predicted based on external knowledge that dogs are typically trained.
• External Knowledge in Hashtag Recommendation
Does not appear directly
in the text or image.

8. Introduction
8
However, #dogtraining can be predicted based on external knowledge that dogs are typically trained.
• External Knowledge in Hashtag Recommendation
Does not appear directly
in the text or image.
How can we integrate the external knowledge to recommend
more relevant hashtags for the post?

9. Related Work
• Open Directory Project (ODP)
: This is a comprehensive directory of the World Wide Web, constructed and maintained by a community of
volunteer editors.
: It organizes web pages into the most related categories and subcategories.
(e.g., Health/Animal/Mammals/Dogs)
• There have been previous studies utilizing ODP categories as an external knowledge.
: The study1 trained a classifier to predict the category based on the description of the webpage.
9
1. Kim et al., “meChat: In-device Personal Assistant for Conversational Photo Sharing” IEEE Internet Computing 2019

10. Methodology
10
• Overview of the proposed model

11. Methodology
11
• Overview of the proposed model
① External Knowledge
Feature Extraction Model

12. Methodology
12
① External Knowledge
Feature Extraction Model
② Integrating External
Knowledge into
Multimodal Hashtag
Recommendation
System (EXTRA)
• Overview of the proposed model

13. Methodology
① External Knowledge Feature Extraction Model
13

14. Methodology
① External Knowledge Feature Extraction Model
14
Caption
Generator
A woman is holding a
certificate with a dog

15. Methodology
① External Knowledge Feature Extraction Model
15
ODP-based Classifier
(ODP-I)
Health/Animal/Mammals/Dogs
Recreation/Pets/Dogs/Training
Recreation/Pets/Dogs/Breeds/Working_Group
Caption
Generator
A woman is holding a
certificate with a dog

16. Methodology
① External Knowledge Feature Extraction Model
16
ODP-based Classifier
(ODP-I)
Health/Animal/Mammals/Dogs
Recreation/Pets/Dogs/Training
Recreation/Pets/Dogs/Breeds/Working_Group
Caption
Generator
A woman is holding a
certificate with a dog

17. Methodology
② Integrating External Knowledge into Multimodal Hashtag Recommendation System (EXTRA)
17
Image of the post Text of the post Relevant Category of the Image
Algy is our other graduate. Today, Algy aced the see-saw and
weave poles so impressively with this obstacle awareness..
Recreation/Pets/Dogs/Breeds/Working_Group
Health/Animal/Mammals/Dogs
Recreation/Pets/Dogs/Training

18. Methodology
② Integrating External Knowledge into Multimodal Hashtag Recommendation System (EXTRA)
18
Image of the post Text of the post Relevant Category of the Image
…
Image Encoder
Algy is our other graduate. Today, Algy aced the see-saw and
weave poles so impressively with this obstacle awareness..
Recreation/Pets/Dogs/Breeds/Working_Group
Health/Animal/Mammals/Dogs
Recreation/Pets/Dogs/Training

19. Methodology
② Integrating External Knowledge into Multimodal Hashtag Recommendation System (EXTRA)
19
Image of the post Text of the post Relevant Category of the Image
Dogs Training Working Group
awareness
algy other …
is our obstacle
…
Image Encoder Text Encoder
Algy is our other graduate. Today, Algy aced the see-saw and
weave poles so impressively with this obstacle awareness..
Recreation/Pets/Dogs/Breeds/Working_Group
Health/Animal/Mammals/Dogs
Recreation/Pets/Dogs/Training

20. Methodology
② Integrating External Knowledge into Multimodal Hashtag Recommendation System (EXTRA)
20
Image of the post Text of the post Relevant Category of the Image
Dogs Training Working Group
awareness
algy other …
is our obstacle
…
Image Encoder Text Encoder
Multimodal Encoder
Algy is our other graduate. Today, Algy aced the see-saw and
weave poles so impressively with this obstacle awareness..
Recreation/Pets/Dogs/Breeds/Working_Group
Health/Animal/Mammals/Dogs
Recreation/Pets/Dogs/Training

21. Dataset
21
• Open Directory Project Dataset (for extracting external knowledge)
- # Web pages : 60,710
- # Categories : 2,531
• MaCon Dataset (for hashtag recommendation)
- # Posts : 624,520
- # Hashtags : 3,896
- # Average of hashtags per post : 9.3

22. Baselines
22
• Co-Attention (Co-AT)
: the model that sequentially generates text and image feature vectors from a post using a co-attention mechanism.
• MaCon
: the model that applies a parallel co-attention mechanism to combine the text and image features with users’
tagging habits.
• TAGNet
: the model that integrates text, visual, and user habit features through a triplet attention module by constructing
visual similarity graphs.
SOTA

23. Settings
23
• Implementation Details
- Model : Pre-trained FLAVA1
- Optimizer : AdamW
- Learning Rate : 1e-5 (10% warmup, cosine decay)
- Batch Size : 64
• Evaluation Metric
: Top-k Precision, Recall, F1-score
1. Singh, Amanpreet, et al. "Flava: A foundational language and vision alignment model." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

24. Experiments
24
• Comparison with Baselines
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
Top-1 Top-5 Top-9
F1-score
Co-AT MaCon TAGNet EXTRA (Ours)
 EXTRA achieved a 39.78% improvement over the state-of-the-art model TAGNet in terms of the Top-5 F1-Score.

25. Experiments
25
• Comparison Between External Knowledge From Each Modality
 The models that have integrated external knowledge perform better than those that have not.

26. Conclusions
26
• We propose EXTRA, a novel multimodal hashtag recommendation system that combines text and image with
external knowledge in a Transformer-based architecture.
• We employ the most relevant categories extracted from the image of the post as external knowledge using the
ODP-based classifier.
• Our approach outperformed the existing state-of-the-art methods on the MaCon dataset, demonstrating its
effectiveness in handling multimodal information for the hashtag recommendation task.

27. 27

28. Experiments
28
• Illustrations of Recommended Hashtags
Predicted
Hashtag
(w/o ODP-I)
#lax #makeportraits #mountainlife
#plasticpollution #pursuitofportraits #spa
#greenery #travel #detail #blackandwhite
Predicted
Hashtag
(w/ ODP-I)
#makeportraits #plasticpollution
#mountainlife #adventure #lax
#pursuitofportraits #travel
#greatnorthcollective #nature

29. Experiments
29
• Comparison between Each Modality

30. Experiments
30
• Comparison between The Number of Subcategories