Master defence 2020 - Andrew Kurochkin - Meme Generation for Social Media Audience Engagement

Meme Generation
for Social Media
Audience Engagement
author supervisor
Andrew Kurochkin
andrewkurochkin.com
PhD. Kostiantyn Bokhan

Agenda
1. Introduction
2. Research Objectives
3. Approach
4. Dataset
5. Solution
6. Evaluation
7. Summary
2

Deﬁnition #1
Meme - a unit of cultural information spread by imitation.
1976, The Selﬁsh Gene, Richard Dawkins.
4

Deﬁnition #1
Image macro - an image superimposed with text.
Image macro is one of the most common form of the
internet meme.
5

Deﬁnition #1
Image macro - an image superimposed with text.
Image macro is one of the most common form of the
internet meme.
*Image macro = Meme
6

Deﬁnition #1
Image macro consists of:
1. Top and bottom captions.
Font: Impact.
2. Background image.
7

Deﬁnition #2
Meme template - is the background
image, which is common across many
meme instances.
Each template has its cultural context.
8

Deﬁnition #3
Engage [verb] - occupy, attract, or involve someone's
interest or attention.
9

Deﬁnition #3
Engage [verb] - occupy, attract, or involve someone's
interest or attention.
4 levels of engagement [2019, Kholoud Khalil Aldous et al]:
1. View
2. Like
3. Comment, shares
4. External post
10

Motivation
11
3.5 billion people use social media in 2019
2h 20m people spend using social media every day

Motivation
12
~80% of marketers use visual assets in their social media marketing
~30% of marketers say visual images are the most important form of content
for their business
~70% of marketers state that creating more engaging content is the most
important task

Motivation
13
Memes can be used to manipulate public opinion, for political or
ideological propaganda
~80% of marketers use visual assets in their social media marketing
~30% of marketers say visual images are the most important form of content
for their business
~70% of marketers state that creating more engaging content is the most
important task

Research Objectives
1. To investigate whether the current state-of-the-art
neural network can produce memes that engage
audience.
2. To create a dataset with an image macros,
meta-information, and related comments.
3. To create a pipeline in order to evaluate the degree of
engagement which the image macros induces in the
audience.
15

Approach
A. Get actual information occasion (event)
17

Approach
B. Embed an event as a vector
18

Approach
C. Choose a meme template based on the event-vector
19

Approach
D. Generate image macro captions based on the meme
template context and small event-vector
20

Approach
D. Generate image macro captions based on the meme
template context and small event-vector
E. Combine the results from C and D
21

(A) Get actual information occasion
22

(B) Embed event as a vector
23

(C) Meme template selection
24

(D) Generate meme captions and post title
25

(D) Generate meme captions and post title
GPT-2 model [2019, OpenAI, Alec Radford et al.]
26
1. Trained on a dataset of 8 million web pages, 40GB of Internet text, including
Reddit.
2. GPT-2 is based on multi-layer Transformer decoder. This model gives structured
memory for handling long-term dependencies in text.
3. GPT-2 beat state-of-the-art solutions on various tasks without ﬁnetuning.
We used the smallest GPT-2 117M which has 117 million of parameters.

(E) Combine the results from C and D
27

Engaging
content
creation
pipeline
28

Dataset requirements
1. Post title
2. Post comments
3. Submitted meme
4. Meme template
5. Meme captions (top and bottom)
6. Collected engagement (amount of scores/likes)
30

Dataset collection
1. Download Reddit submissions and comments data for the
subreddit AdviceAnimals.
32

Dataset collection
2. Download images from the submissions where it was possible.
33

Dataset collection
3. Recognize a meme template [2018, Savvas Zannettou et al]:
34

Dataset collection
a. Embed each image as a vector with 64 elements based on
its Perceptual Hash (pHash)
35

Dataset collection
b. Calculate Hamming distances between pHashes
36

Dataset collection
c. Cluster images using DBSCAN
37

Dataset collection
c. Cluster images using DBSCAN
4. Optical character recognition (OCR) to extract top and bottom
pieces of text from the meme image (Azure)
38

Volumes of engagement in dataset
39

Final dataset size
650K memes
350K with comments (keywords)
40

Data preparation
43
Class balancing
memes templates prior
resampling (imbalanced)
memes templates after
resampling (balanced)

Data preparation
46
Input data:
<|startoftext|>~`
001~^
photographer catdog mediocre said maybe~@
As a photography student...~}
just because you own a camera~{
it does not make you a. photographer
<|endoftext|>

Data preparation
47
Input data:
start token
template id
5 keywords
submission title
top caption (lowercased)
bottom caption (lowercased)
end token
<|startoftext|>~`
001~^
photographer catdog mediocre said maybe~@
As a photography student...~}
just because you own a camera~{
it does not make you a. photographer
<|endoftext|>

Generative model training, ~20h
48
Training loss Validation loss

49
Training loss, 1.63 Validation loss, 2.35
Generative model training, ~20h

Evaluation
1. Post in social network (Reddit).
Challenges:
a. must ﬁlter “bad” content
b. time consuming
c. other factors (posting time, etc).
2. Use crowdsourcing to evaluate engagement.
Compare engagement from the machine and human
memes groups.
51

Evaluation
1. Post in social network (Reddit).
Challenges:
a. must ﬁlter “bad” content
b. time consuming
c. other factors (posting time, etc).
2. Use crowdsourcing to evaluate engagement.
Compare engagement from the machine and human
memes groups.
52

Evaluation pipeline
Machine-generated memes Human-created memes
53

Evaluation pipeline
54
Task to collect engagement (MTurk)

Evaluation pipeline
1. Crowdsourcing (MTurk).
2. Contingency table.
3. Chi-square test.
Compare if two groups are similar (statistical hypothesis test).
4. Cohen effect size w.
55

MTurk justiﬁcation
56
Like Not like
good memes (A) 41 59
bad memes (B) 21 79

57
Like Not like
bad memes (B) 21 79
observations p-value w α power
200 0.2 8.44 0.003 0.21 0.05 0.82

58
Like Not like
bad memes (B) 21 79
observations p-value w α power
200 0.2 8.44 0.003 0.21 0.05 0.82

Chi-square test prior calculations
59
w 0.1 (small)
α 0.05
df (degrees of freedom) 1
test power 0.9

Chi-square test prior calculations
60
w 0.1 (small)
α 0.05
df (degrees of freedom) 1
test power 0.9
total observations ∼1000
observations in sample ~500

Evaluation setting
We targeted workers with the following characteristics:
● Hit approval rate >= 95%
● Number of HITs approved >= 5000
● Location is the United States
● Worker is MTurk Master
We collected 900 observations per sample.
Removed 5% of HITs that were done extremely fast or slow.
61

Memes samples
62
* “bad” human memes are not always bad.
Description
Memes
num.
Observations
num.
Liked (%)
human, random 90 846 0.35
human, bad 90 881 0.27
machine 90 876 0.24

#1 Random (A) vs bad human (B) memes
63
observations 1727
0.08
11.06
p-value <0.01
w 0.08
α 0.05
power 0.91

#1 Random (A) vs bad human (B) memes
64
observations 1727
0.08
11.06
p-value <0.01
w 0.08
α 0.05
power 0.91

#2 Random (A) vs machine generated (B)
65
observations 1722
0.09
21.29
p-value <0.01
w 0.11
α 0.05
power 0.99

#2 Random (A) vs machine generated (B)
66
observations 1722
0.09
21.29
p-value <0.01
w 0.11
α 0.05
power 0.99

#3 Bad human (A) vs machine generated (B)
67
observations 1757
0.03
1.40
p-value 0.27
w 0.02
α 0.05
power 0.22

#3 Bad human (A) vs machine generated (B)
68
observations 1757
0.03
1.40
p-value 0.27
w 0.02
α 0.05
power 0.22

Contributions
1. Created a unique dataset.
2. Justiﬁed that MTurk can be used to approximate measure engagement
from memes.
3. Found out that current SOTA generates memes which collect similar
amount of engagement to the least popular human memes.
4. Proposed approach which can be adapted to generate more complex
content.
71

Future work
1. Detect and ﬁlter out offensive machine-generated content.
2. Create model to detect good memes.
3. Publish generated submissions in the big communities to
evaluate engagement in the wild.
4. Advance training data selection.
5. Improve template selection.
72

Highlights from the review
1. “Data collection section should deﬁnitely outline all the data scesiﬁc counts”
a. users - 8.1M
b. posts - 3.9M
2. “some of the data cleaning details were missing from the report”.
a. remove special characters and urls
b. word2num
c. tokenize
d. lemmatize
3. “it would be interested to see if there are any features that distinguish
unpopular memes from the popular ones, e.g., time when the meme was posted”.
“What’s in a name? Understanding the Interplay between Titles, Content, and
Communities in Social Media”
2013, Himabindu Lakkaraju et al, cseweb.ucsd.edu/~jmcauley/pdfs/icwsm13.pdf
74

Master defence 2020 - Andrew Kurochkin - Meme Generation for Social Media Audience Engagement

Recommended

Recommended

More Related Content

What's hot

What's hot (6)

Similar to Master defence 2020 - Andrew Kurochkin - Meme Generation for Social Media Audience Engagement

Similar to Master defence 2020 - Andrew Kurochkin - Meme Generation for Social Media Audience Engagement (20)

More from Lviv Data Science Summer School

More from Lviv Data Science Summer School (20)

Recently uploaded

Recently uploaded (20)

Master defence 2020 - Andrew Kurochkin - Meme Generation for Social Media Audience Engagement