This document discusses using deep learning techniques to analyze visual and textual content on social media platforms like Tumblr in order to understand what drives consumer engagement. It collected data on posts and engagements from 183 company Tumblr blogs. Using deep convolutional neural networks, it defined a measure of "semantic complexity" to capture the high-level meaning of images. It also analyzed other visual features like aesthetics and complexity, as well as textual features from posts. Finally, it used these content features to build a model for predicting consumer engagement with posts, as measured by likes and reblogs. The findings could help firms optimize their social media strategies and content design.

1. Content Complexity, Similarity, and
Consistency in Social Media:
A Deep Learning Approach
Gene Moo Lee
University of Texas at Arlington
Joint work with
Donghyuk Shin (UT Austin/Amazon), Shu He (UConn),
Andrew B. Whinston (UT Austin)
DSI 2016, Austin TX

2. Social media: More users
2

3. Social media: More spending
3

4. Challenges and opportunities: 78% photos
4
Source: Chang et al. 2014

5. Research questions
• How can firms optimize social media strategies by
incorporating visual content?
• Specifically, what are the determinants of consumer
engagement in terms of “likes” and “reblogs” (sharing)
actions?
• How visual and textual contents play role?
• Operationally, how to construct measures on these
unstructured data sources?
5

6. Tumblr data
• Tumblr: microblogging platform (acquired by Yahoo!)
• 35,651 posts by 183 companies (May - Oct 2014)
• Automobile, Entertainment, Food, Fashion,
Finance, Leisure, Retail, Tech
• 89.7% photo & text, 6.3% pure text, 4% videos
• Collected “likes” and “reblogs” until Apr 2015
6

7. Company blogs in Tumblr
7
BMW USA Vogue IBM

8. Data: blog post and engagement
8
Post = Visual Info (Image) + Textual Info (Text, Tags)
Customer engagement = Notes (Likes + Reblogs)

9. Visual features
• Aesthetics (beautiful photos)
• Adult-contents
• Celebrity
• Feature complexity (low-level, flashy images)
• Semantic complexity (high-level, complex meaning)
• Number of salient objects
9

10. Feature complexity (low level)
• Visual complexity theory [Donderi 2006a, Pieters et al. 2010]
• Visually complex (flashy) images (colors, luminance,
shape) gets more attention
• This feature complexity can be captured by the
image’s compressed file size [Donderi 2006a; Donderi
2006b; Machado et al. 2015; Forsythe et al. 2011]
• However, this complexity can only capture low-level
complexity based on “pixel” values
10

11. Semantic complexity (high level)
• Recognition-By-Components theory [Biederman 1987]
• Human object recognition is invariant to feature
factors (colors, brightness, edges, positions, etc.)
• Vessel and Rubin (2010) show that visual preferences
are influenced by semantic content in the image
• We posit that semantic complexity matters!
• Operational question: How do we calculate semantics
from unstructured images?
11

12. Deep learning
• A branch of machine learning, inspired by human brain
• Algorithms to model high-level abstractions with multiple processing
layers of non-linear transformations
• (1) theoretical breakthroughs, (2) Big Data, (3) powerful computation
• Successfully applied in image/video/voice recognition, AlphaGo, etc.
12

13. Semantic complexity via deep learning
• Deep convolutional neural network (CNN) [Jia et al. 2014]
• Model trained with 1.2 million images with tags (ImageNet, Flickr)
• Tested on 53,417 images from brand-generated Tumblr posts
• Each image is represented by a 1,700 dimensional vector, where each
value is the confidence score w.r.t. an object (tag)
• We define semantic complexity as the Shannon Diversity Index (entropy)
on the 1,700-dimensional vector
• max = log(d), if p is uniformly distributed
• min = 0, if p_i = 1 for some i
13

14. ImageNet: Image DB with tree-structure tags
14
Source: ImageNet

15. More visual features
• 7th-layer output = robust representation of the image for “computer vision” tasks
• Aesthetic/beauty score [Dhar et al. 2011 (CVPR, Vision)]
• Adult-content score [Sengamedu et al. 2011 (MM, Vision)]
• Celebrity (450 celebrities) [Parhki et al. 2015 (BMV, Vision)]
• Number of salient objects [Zhang et al. 2015 (CVPR, Vision)]
15

16. Examples: Visual features
• Visual complexity theory (Attneave 1994,
Donderi 2006, Pieters et al. 2010)
• Visual stimuli are a composite of
colors,luminance, shape, number of
objects/patterns
16

17. Textual features
• Two textual sources: text and tags
• Length: # of words, # of tags
• Topic complexity: LDA topic model (text, tags)
• Order complexity: word2vec (for text only)
17

18. Examples: Textual features
• Topics
• Word clusters
18

19. Visual-Textual Content Similarity
• Image: pixels, Text/Tags: characters
— Need a common representation!
1. Represent each image as a collection of the predicted labels
obtained from deep learning — “image corpus”
2. Train LDA with both image and text/tags corpora — topic
distribution for images and text/tags
3. Cosine similarity between the two corresponding topic
distribution
19

20. Examples: Content similarity
• Topics
• Word clusters
20

21. 21

22. Empirical Model
• Linear fixed effects model
• DV (likes/reblogs): take log transformation due to their
skewed distributions
• Capture blog (firm) heterogeneity
• Capture time effects (day of week, month)
• Other models
• Identical results with random effects
• Consistent results with negative binomial model
22

23. 23

24. Summary and implications
1. Large-scale analysis on visual content in social
media
2. New visual semantic complexity via deep learning
• Able to relate visual and textual content
Visual content analysis can be used to optimize
content design for social media marketing
24

25. Thank you!
Contact Info: Gene Moo Lee
gene.lee@uta.edu