Social Media and Deep Learning

1. Content Complexity, Similarity, and
Consistency in Social Media:
A Deep Learning Approach
Gene Moo Lee
University of Texas at Arlington
Joint work with
Donghyuk Shin (UT Austin/Amazon), Shu He (UConn),
Andrew B. Whinston (UT Austin)
DSI 2016, Austin TX

2. Social media: More users
2

3. Social media: More spending
3

4. Challenges and opportunities: 78% photos
4
Source: Chang et al. 2014

5. Research questions
• How can firms optimize social media strategies by
incorporating visual content?
• Specifically, what are the determinants of consumer
engagement in terms of “likes” and “reblogs” (sharing)
actions?
• How visual and textual contents play role?
• Operationally, how to construct measures on these
unstructured data sources?
5

6. Tumblr data
• Tumblr: microblogging platform (acquired by Yahoo!)
• 35,651 posts by 183 companies (May - Oct 2014)
• Automobile, Entertainment, Food, Fashion,
Finance, Leisure, Retail, Tech
• 89.7% photo & text, 6.3% pure text, 4% videos
• Collected “likes” and “reblogs” until Apr 2015
6

7. Company blogs in Tumblr
7
BMW USA Vogue IBM

8. Data: blog post and engagement
8
Post = Visual Info (Image) + Textual Info (Text, Tags)
Customer engagement = Notes (Likes + Reblogs)

9. Visual features
• Aesthetics (beautiful photos)
• Adult-contents
• Celebrity
• Feature complexity (low-level, flashy images)
• Semantic complexity (high-level, complex meaning)
• Number of salient objects
9

10. Feature complexity (low level)
• Visual complexity theory [Donderi 2006a, Pieters et al. 2010]
• Visually complex (flashy) images (colors, luminance,
shape) gets more attention
• This feature complexity can be captured by the
image’s compressed file size [Donderi 2006a; Donderi
2006b; Machado et al. 2015; Forsythe et al. 2011]
• However, this complexity can only capture low-level
complexity based on “pixel” values
10

11. Semantic complexity (high level)
• Recognition-By-Components theory [Biederman 1987]
• Human object recognition is invariant to feature
factors (colors, brightness, edges, positions, etc.)
• Vessel and Rubin (2010) show that visual preferences
are influenced by semantic content in the image
• We posit that semantic complexity matters!
• Operational question: How do we calculate semantics
from unstructured images?
11

12. Deep learning
• A branch of machine learning, inspired by human brain
• Algorithms to model high-level abstractions with multiple processing
layers of non-linear transformations
• (1) theoretical breakthroughs, (2) Big Data, (3) powerful computation
• Successfully applied in image/video/voice recognition, AlphaGo, etc.
12

13. Semantic complexity via deep learning
• Deep convolutional neural network (CNN) [Jia et al. 2014]
• Model trained with 1.2 million images with tags (ImageNet, Flickr)
• Tested on 53,417 images from brand-generated Tumblr posts
• Each image is represented by a 1,700 dimensional vector, where each
value is the confidence score w.r.t. an object (tag)
• We define semantic complexity as the Shannon Diversity Index (entropy)
on the 1,700-dimensional vector
• max = log(d), if p is uniformly distributed
• min = 0, if p_i = 1 for some i
13

14. ImageNet: Image DB with tree-structure tags
14
Source: ImageNet

15. More visual features
• 7th-layer output = robust representation of the image for “computer vision” tasks
• Aesthetic/beauty score [Dhar et al. 2011 (CVPR, Vision)]
• Adult-content score [Sengamedu et al. 2011 (MM, Vision)]
• Celebrity (450 celebrities) [Parhki et al. 2015 (BMV, Vision)]
• Number of salient objects [Zhang et al. 2015 (CVPR, Vision)]
15

16. Examples: Visual features
• Visual complexity theory (Attneave 1994,
Donderi 2006, Pieters et al. 2010)
• Visual stimuli are a composite of
colors,luminance, shape, number of
objects/patterns
16

17. Textual features
• Two textual sources: text and tags
• Length: # of words, # of tags
• Topic complexity: LDA topic model (text, tags)
• Order complexity: word2vec (for text only)
17

18. Examples: Textual features
• Topics
• Word clusters
18

19. Visual-Textual Content Similarity
• Image: pixels, Text/Tags: characters
— Need a common representation!
1. Represent each image as a collection of the predicted labels
obtained from deep learning — “image corpus”
2. Train LDA with both image and text/tags corpora — topic
distribution for images and text/tags
3. Cosine similarity between the two corresponding topic
distribution
19

20. Examples: Content similarity
• Topics
• Word clusters
20

21. 21

22. Empirical Model
• Linear fixed effects model
• DV (likes/reblogs): take log transformation due to their
skewed distributions
• Capture blog (firm) heterogeneity
• Capture time effects (day of week, month)
• Other models
• Identical results with random effects
• Consistent results with negative binomial model
22

23. 23

24. Summary and implications
1. Large-scale analysis on visual content in social
media
2. New visual semantic complexity via deep learning
• Able to relate visual and textual content
Visual content analysis can be used to optimize
content design for social media marketing
24

25. Thank you!
Contact Info: Gene Moo Lee
gene.lee@uta.edu