Proud to have graduated from the University of Amsterdam MSc Information Studies program in Data Science, this was my final msc thesis presentation on my defense last July. The thesis answers to the contemporary complex problem of popularity prediction, which is highly discussed but not yet solved. An effort for optimizing feature engineering, using several regression models and answering to the question of "what makes a post popular" according to its content e.g. action, scene, people, animals etc., taking advantage in the meantime both computer vision techniques and text processing, is the summary of this work. My MSc Thesis was marked unanimously with 8/10 - one of the highest grades in UvA Informatics department - and was submitted for a publication in MMM2018 24th International Conference on Multimedia Modeling (Thailand, Bangkok).

1. RECOMMENDATIONS FOR POST POPULARITY
PREDICTION IN SOCIAL MEDIA
Iliana Pappi - MSc Information Studies: Data Science track, University of Amsterdam (UvA)
MSc Thesis Supervisor: Dr. Masoud Mazloom
MSc Thesis Duration: 1/4/2017 – 30/6/2017 (3 months)
1

2. PROBLEM STATEMENT
• Huge amount of content every minute in
the social media:
• Text
• Image
• Videos
• Audio
• Some posts receive thousands of likes,
positive comments etc. while others are
completely ignored
• What could make a post popular on the
web?
2

3. WHY DOES IT MATTER?
3
• Advertising – Marketing
• Political Campaigns
... and more
• Understanding user behavior
• Modifying popularity
• Make recommendations
• Video summarization
• etc.

4. A CHALLENGING PROBLEM TO SOLVE
• Machine learning Popularity prediction
• Data from social media Various features
• Feature extraction from text and image
• Multimodal framework: image in the post,
image caption or user’s comments, hashtags
• Which are useful features to predict post
popularity?
• …..
• It has been proven a challenging problem to
solve.
• How is popularity expressed in social
media?
4

5. RESEARCH QUESTIONS
• RQ1: How can we define which features affect post popularity in social media, in order to
make recommendations to the users?
• RQ2: What is the role of low and high-level visual features for popularity prediction? Eg.
content like action, scene, people, pets, brand.
• RQ3: How visual centrality in a user’s post can be combined with textual and numerical data
for post popularity?
• RQ4: Outline a multimodal model exploiting different features? Make recommendations for
popularity prediction?
5

6. OUR PROPOSED METHODOLOGY –
MULTIMODAL POST POPULARITY PREDICTION
FOR CONTENT CATEGORIES
6

7. POST POPULARITY PREDICTION
• K features: both visual and textual
• Each post: K feature vectors
• Construct sample – feature matrices for
each subset of the data and each feature
type among the K extracted
• Define y-vector as the log-normalized
number of likes in a post
• Prediction for y
• Average/Max Pooling between
different features
• Evaluation:
• Spearman’s Rank Correlation
Coefficient:
• Denotes the monotonic
relationship between the
prediction and the ground
truth (1: perfect correlation)
7
Regression Model

8. THE DATASET
• ~40k of Instagram posts
• 13k – Human actions
#kiss, #dance, #horseriding
• 15k – Places/ Sceneries
#forest, #urban, #kitchen
• 2k – People/Pets
#selfie, #pets
• 9k – Brand – Related posts
#Wendys
• The data crawled by #hashtag from
Instagram API
• Dataset Preprocessing:
• Python Pandas Data Analysis Library
• Python Natural Language Processing Library
(NLTK)
• Remove duplicate posts – bad image files –
posts without textual content
• Remove non-ASCII words, strip out
symbols, keep only English words
• Pool the textual content of each post
coming from the user
8

9. VISUAL FEATURE EXTRACTION
• High-level/Low level visual features:
• Keras/Tensorflow Deep Learning Python Library
• GoogleNet Inception V3 deep network – trained on ImageNet 1000-concepts
• High-level: 1x1000 feature vectors – probability of appearance of each
Imagenet concept in the image
• Low-level: 1x2048 feature vectors – Convolutional Pool Layer 8x8
(Max Pooling)
• Visual Sentiment visual features:
• SentiBank detectors on Visual Sentiment Ontology (VSO) - MATLAB
• Visual Sentiment: 1x1200 feature vectors: probability of appearance of each adjective-noun pair
(ANP) in the visual sentiment ontology, eg. ‘clean_pool’, ’happy_mother’ etc.
9

10. TEXTUAL FEATURE EXTRACTION
• Word-to-Vec (W2V) textual features:
• Python Gensim Library – W2V implementation trained on a part of Google News dataset (100 billion
words)
• Extraction of 1x300 feature vectors for each word  average pooling of all the words in the textual
content of each post
• Bag-of-Words (BoW) textual features:
• Count Vectorizer – Python Scikit-Learn – preconstructed vocabulary of all the sorted list of unique
words in the dataset
• Extraction of 1x19166 feature vectors for each post  sparse frequency representation for words in
post
• Textual sentiment features:
• TextBlob Python Library – Naïve Bayes Analyzer based on NLTK
• Extraction of 1x2 feature vectors – scores for positive and negative sentiment in the textual content of
each post
10

11. EXPERIMENT 1: CATEGORY-MIX ANALYSIS
• Support Vector Regression (SVR) – Radial Basis
Function (RBF) kernel – Scikit-Learn
• Tuning over C=[0.01,0.1,1,10,100,1000] in a 5-
fold cross validation, l1-normalization
• Compared with l2-normalization of linear SVR,
Random Forrest(RF) regression (100
estimators), Multi-Layer Perceptron (MLP)
Regression (default) – Scikit-Learn
• Results for every subset category:
• Action, Scene, People-Pets, Brand
• And every feature type:
• 3 visual features – 3 textual features
• Evaluation: Spearman’s Rank Correlation
Coefficient (SCRR)
• Storing linear SVR model weights – for high-
level visual features, visual sentiment
features and bag-of-words textual features
• Rank the weights to make semantic
recommendations about the top-10
concepts, ANP’s or words that affect most
popularity prediction for each subset
category
• Late fusion over visual features or textual
features for each model and category
• Late fusion over all features for each model
and category
11

12. EXPERIMENT 1- RESULTS
12
Tb1:Post Popularity prediction with visual features Tb2:Post Popularity prediction with textual features

13. EXPERIMENT 1- RESULTS
13
Top-10 ANP’s : (a)action, (b)scene, (c) people-pets, (d) brand Top-10 words - BoW : (a)action, (b)scene, (c) people-pets, (d) brand

14. EXPERIMENT 1- RESULTS
14
Top-10 ImageNet concepts per category
Tb3: Multimodal Fusion
RQ2: What is the role of low and high-level visual features? Eg.
content like action, scene, people, pets, brand.
• Visual features, especially low-level are more correlated
with action
RQ3: What is the role of textual features when combined with
visual features ?
• Textual features, especially BoW are more correlated with
scene
• Textual features are necessary to increase the predictability
of the model along with visual features

15. EXPERIMENT 2 – CATEGORY-SPECIFIC
(WITH VISUAL FEATURES) – HEATMAP
15
• Subsets
categorized
by hashtag
• SVR- RBF
• Report on
SCRR

16. EXPERIMENT 2 – CATEGORY-SPECIFIC
(WITH TEXTUAL FEATURES) - HEATMAP
16
• Subsets
categorized
by hashtag
• SVR- RBF
• Report on
SCRR

17. EXPERIMENT 3: CONCEPT SPECIFIC
• Label 1000-concepts
of imageNet
• Run SVR-RBF for
each category mix
subset for all
different feature
categories of
concepts
• Report SCRR
• 50 concepts –
action
• 151 concepts –
scene
• 8 concepts –
people
• 406 concepts –
animals
• 525 concepts –
objects(general)
17

18. CONCLUSIONS
• A number of features were tested  have been proven adequate to lead to recommendations
(RQ1).
• Hashtag subsets indicated human joyful activities and fun places that could make a post popular.
• Visual features, especially low-level  action content.
• Textual features, especially bag-of-words  scene content.
• High-level concepts related to scene  highest correlation with popularity prediction in scenery
datasets.
• In general, more concepts taken under account are better.
• Visual - textual complementarity  multimodal framework (RQ4)  Imagenet concepts, ANPs,
BoW  Bridge the semantic gap!  recommendations for users.S
• Future work- Reflection: select the best features for each category to fuse them in a multimodal
framework, try early fusion, explore the semantics in hashtag concept –specific analysis.
18

19. Thank you for your attention!
19

20. RECOMMENDATIONS FOR POST POPULARITY
PREDICTION IN SOCIAL MEDIA
Iliana Pappi - MSc Information Studies: Data Science track, University of Amsterdam (UvA)
MSc Thesis Supervisor: Dr. Masoud Mazloom
MSc Thesis Duration: 1/4/2017 – 30/6/2017 (3 months)
20