This document proposes a data augmentation method for image sentiment analysis using hashtags. It involves collecting a small set of manually labeled images and their hashtags, learning to predict sentiment labels from the hashtags using machine learning, and using this model to automatically label more images. Preliminary results show the hashtag-predicted labels match human labels with 83-95% accuracy. However, more testing is needed on a general set of images to fully evaluate the method's effectiveness.

1. Data Augmentation Method for the
Image Sentiment
Analysis
Alexander Rakovsky1, Arseny Moskvichev2, Andrey Filchenkov1
1ITMO University
Saint Petersburg, Russia
2Saint Petersburg State University
Saint Petersburg, Russia
str1t3r@gmail.com, arseny.moskvichev@gmail.com, afilchenkov@corp.mail.ru

2. Image sentiment analysis
Positiveness: 0.9
Positiveness: 0.01

3. Why is it important?
Two words:
Social networks

4. How do we approach it?
1.Collect lots of labeled images
2.Train a convolutional neural network
3.???
4.Profit

5. How do we approach it?
1.Collect lots of labeled images
2.Train a convolutional neural network
3.???
4.Profit
Problem!

6. Solution
Data augmentation.
1.Get a few manually labeled images with
corresponding hashtags
2.Learn to reconstruct labels from hashtags
3.Collect as much labeled data as you need!

7. Details
• Collecting data through FLICKR API (using
keywords)
• Assessors evaluate the emotional colouring
(positiveness) of each image
• Converting hashtags to vector representation
(word2vec), and averaging them
• Using machine learning to predict assessors’
estimation

8. (Preliminary!) Results
• kNN accuracy on classification task: 0.95
• Average correlation between assessors: 0.86
• Between the kNN regression and assessors:
0.83
• Using this algorithm is almost as good as
hiring one more assessor!
• Suspiciously good...

9. Details
• Collecting data through FLICKR API (using
keywords)
• Assessors evaluate the emotional colouring
(positiveness) of each image
• Converting hashtags to vector representation
(word2vec), and averaging them
• Using machine learning to predict assessors’
estimation
Nonrepresentative sample!

10. Pros
• Easy to use (no word preprocessing)
• Good results* (compared to dictionary -
based solutions)

11. Cons
• Needs pre-training and an initial manually
labeled sample

12. Conclusions
• The proposed method affords a simple and
efficient hashtag-based data augmentation
solution for image sentiment analysis.
• More work is to be done to estimate the
method’s performance on a general set of
images.

13. Thank you!