This paper proposes using a user's favorite images and their associated tags for personalized tag recommendation of new images. It finds that favorite image context outperforms personal and collective context in recommending tags, achieving higher precision and success rates. Tags are recommended by linearly combining relevance scores from tag statistics and visual similarity to favorite images. Experiments on over 387,000 Flickr images show favorite image context improves tag recommendation accuracy.

1. IMPROVING IMAGE TAG RECOMMENDATION
USING FAVORITE IMAGE CONTEXT
Wonyong Eom, Sihyoung Lee, Wesley De Neve, and Yong Man Ro
Image and Video Systems Lab
Korea Advanced Institute of Science and Technology (KAIST)
Daejeon, South Korea
e-mail: ymro@ee.kaist.ac.kr website: http://ivylab.kaist.ac.kr
I. INTRODUCTION III. EXPERIMENTS
- Observation 1. Experimental setup
- the number of images shared on online social network services - We collected images from Flickr users meeting the following
keeps growing at a fast rate requirements: 1) uploaded at least 100 images, 2) assigned at least
- Problem 500 tags, and 3) bookmarked at least 500 favorite images
- manual tagging of images is labor intensive and time consuming, - Consequently, using the Flickr API, we retrieved a total of 387,397
making it difficult to facilitate effective image retrieval images from 27 users (on September 30, 2010)
- Novel solution - the images retrieved are either favorite images or images owned by
- personalized tag recommendation using favorite image context the 27 users, and are annotated with 4,657,288 tags by 46,686 users
• source of collective knowledge that consists of images and - To calculate visual distance, we used global and local image features
associated tags that have been bookmarked by a particular user
- assumptions made 2. Effectiveness of using favorite image context
• favorite images and their associated tags are indicative of the - Recommending tags using tag statistics: Rtag(t, q)
Context P@5 S@5 P@1
visual and topical interests of a user Personal 0.158 0.609 0.318
• people actively bookmark favorite images Collective 0.208 0.612 0.373
Favorite 0.247 0.729 0.457
II. PROPOSED TAG RECOMMENDATION METHOD - Recommending tags using visual similarity: SCD-based Rimg(t, q)
1. Number of favorite images on Flickr for users of MIRFLICKR-25000 Context P@5 S@5 P@1
Personal 0.187 0.629 0.384
1000000
Collective 0.208 0.611 0.324
Favorite 0.294 0.813 0.446
Number of favorite images
100000
- Recommending tags using visual similarity: BoVW-based Rimg(t, q)
10000
Context P@5 S@5 P@1
1000 Personal 0.206 0.697 0.367
Collective 0.309 0.767 0.523
100 Favorite 0.317 0.813 0.513
10
3. Influence of linear fusion and bookmarking activity
1
1 9861 0.4
187 2538 6872 8855 9861
User
0.3
Fig. 1. Number of favorite images per MIRFLICKR-25000 user
2. Personal, collective, and favorite image context
P@5
0.2
tag statistics
visual similarity (SCD)
0.1 visual similarity (BoVW)
tag statistics + visual similarity (SCD)
tag statistics + visual similarity (BoVW)
personal context
0
Level 1 Level 2 Level 3 Level 4
Type of user group
favorite image context
Fig. 3. P@5 for users with different levels of bookmarking activity
...
4. Example query images
collective context
... tag statistics +
query image tag statistics visual similarity
visual similarity
Fig. 2. Relation between personal, collective, and favorite image context, visualized nature, africa, photo, nature, wildlife,
from the point-of-view of a user who uploaded a new image that is to be annotated nature, wildlife,
image, macro, birds, africa,
macro, birds, africa,
moth, bird, wildlife, flower, animal,
3. Mathematical modeling animal, bird, flower,
birds, flowers, safari,
-The relevance of a set of tags Tq to the content of a query image q safari, flowers
 
macro, Australia butterfly
Tq  t t  T and R(t , q)   tag
-R(t, q) is modeled by linearly fusing the output of two relevance functions sardegna, mare, italy, bw, red, italy, bw, red, green,
donna, fitness, street, milano, green, street, street, milano,
R(t , q)    Rtag (t , q)  (1   )  Rimg (t , q) red, luce, bw, green, silhouette, people, silhouette, light,
-Rtag(t, q) is modeled by making use of tag statistics light paris, canon sardegna, shadow
P(t | v), if P(t | v)  0
R tag (t , q)  P(t ) 
vV

 , otherwise Fig. 4. Example images with tags recommended using favorite image context
-Rimg(t, q) is modeled using a MAP-based method IV. CONCLUSIONS
P(q | t , Q ) P(t | Q ) - Tag recommendation using favorite image context is, for the users
R img (t , q)  P(t | q, Q )  , selected, more effective than the use of personal and collective context
P(q | Q ) - Linearly fusing tag statistics and visual similarity allows for a higher
effectiveness in terms of P@5, compared to their separate usage
IEEE International Conference on Image Processing (ICIP), September 2011, Brussels (Belgium)