1. Visible
PartisanshipPolmeth XXXIII, Rice University, July 22, 2016
Convolutional Neural Networks for the Analysis of Political Images
L. Jason Anastasopoulos ljanastas@uga.edu (University of Georgia, Public Admin +
Policy, Political Science, Georgia Informatics Institute)
Dhruvil Badani (UC Berkeley, EECS)
Crystal Lee (UC Berkeley, EECS)
Shiry Ginosar (UC Berkeley, EECS)

2. Outline
▪ Background
▪ Image experiment - “the people you pose with”– how race and gender of
people politicians pose with affect perceptions.
▪ Race classifier for images using convolutional neural networks.
▪ Analysis of race in US House of Representative Facebook profile
photos.

3. Visual semantics: image elements
▪ Symbols
▪ Objects
▪ People
▪ Poses

4. Images convey political meaning: symbols

5. Images convey political meaning: objects
Source - usa4palin.com: Still from Sarah Palin’s Amazing America

6. Images convey political meaning: people
Source - haaretz.com: Netanyahu (left), Obama (middle), Abbas (right)

7. Images convey political meaning: poses
Source - google.com: Image search for “John Boehner”

8. Political functions of images
For politicians…
▪ Signaling
▪ Partisanship/ideology.
▪ Policy positions.
▪ Homestyle (Fenno 1978)
▪ Qualification – competence.
▪ Identification – “I am one of you.”
▪ Empathy – “I care about your needs.”

9. Political functions of images
For politicians…
▪ Signaling
▪ Partisanship/ideology.
▪ Policy positions.
▪ Homestyle (Fenno 1978)
▪ Qualification – competence.
▪ Identification – “I am one of you.”
▪ Empathy – “I care about your needs.”

10. Political functions of images
For news media…
▪ Issue framing.
▪ Persuasion.

11. Hardware and software limitations
▪ Hardware
▪ Even small images are “big data.”
▪ One 200 x 200 image = 3 200x200
matrices or 1 vector of length
120,000.

12. Hardware and software limitations
▪ Software
▪ High dimensional statistical theory developed more recently.
▪ Asymptotics deals with properties of estimators as with a fixed number of
parameters, p.
▪ In modern machine learning applications,
Introduction
classical asymptotic theory: sample size n → +∞ with number of
parameters p fixed
modern applications in science and engineering:
large-scale problems: both p and n may be large (possibly p ≫ n)
need for high-dimensional theory that allows (n, p) → +∞
Introduction
classical asymptotic theory: sample size n → +∞ with number of
parameters p fixed
modern applications in science and engineering:
large-scale problems: both p and n may be large (possibly p ≫ n)
need for high-dimensional theory that allows (n, p) → +∞

13. Image analysis renaissance in social science
Hardware: Powerful CPUs
and now GPUs in desktop
computers (thanks gamers!)

14. Image analysis renaissance in social science
▪ Software
▪ Statistical theory for computing in high
dimensions.
▪ Advances in numerical computing.
▪ Deep-learning frameworks: Torch,
Tensorflow, Theano, Caffe.

15. Signaling and image features
▪ Symbols
▪ Objects
▪ People
▪ Poses

16. Signaling and image features
▪ Symbols
▪ Objects
▪ People
▪ Poses

17. Questions
▪ How do the group characteristics (gender, race, age, etc.) of people that
Members of Congress pose with affect how they are perceived?
▪ Social media “homestyle”
▪ Is there evidence that Members of Congress use social media images to signal
identification and empathy with constituents using group characteristics?

18. Questions
▪ How do the group characteristics (gender, race, age, etc.) of people that
Members of Congress pose with affect how they are perceived?
▪ Social media “homestyle”
▪ Is there evidence that Members of Congress use social media images to signal
identification and empathy with constituents using group characteristics?

19. “The people you pose with” experiment: Lou
Barletta
▪ Lou Barletta (R-PA, 11) chosen for initial experiment because of relative obscurity
and similar pictures with different groups of people.
▪ MTurk respondents randomly assigned one of 7 images with Barletta
▪ Alone – Barletta by himself.
▪ Woman – Barletta with a woman.
▪ Man – Barletta with a white man.
▪ Black – Barletta with African-American men.
▪ Asked a series of questions based only on the image.

20. Image treatments
Alone Man Woman Af. American

21. What is your best guess of the political party
that this politician belongs to?
Beliefs about Barletta’s party ID
vary significantly by image shown.
Alone: 39% guessed Democrat
61% guessed Republican
Black: 58% guessed Democrat
42% guessed Republican
Man: 42% guessed Democrat
58% guessed Republican
Woman: 43% guessed Democrat
57% guessed Republican

22. What is your best guess of this politician’s
ideological orientation?
Average by treatment groups
Alone: Moderate.
Black: Liberal.
Man: Moderate.
Woman: Moderate.

23. Does this politician seem…honest and
trustworthy?
Perceived to be more trustworthy
when pictured next to a woman.

24. Does the politician seem…like a strong and
decisive leader?
Perceived to be a stronger leader
when pictured next to a woman.

25. Does the politician seem…knowledgeable about
the issues?
Perceived to be less
knowledgeable when pictured
next to an older white man.

26. Does the politician seem…like someone who
shares my values? (non-white respondents)
Perceived by non-white
respondents to share their
values when pictured next to
African-American men.

27. Barletta experiment conclusions
▪ Opinion of Barletta affected by group identity of individuals included in images.
▪ Race affected beliefs about partisanship/ideology and implied “shared
values.”
▪ Gender affected beliefs in trustworthiness, honesty and decisiveness.
▪ Survey experiment expanding – goal is to test which aspects of photos most
strongly tied to perceptions of candidate ideology and party.

28. Questions
▪ How do the group characteristics (gender, race, age, etc.) of people that
Members of Congress pose with affect how they are perceived?
▪ Social media “homestyle”
▪ Is there evidence that Members of Congress use social media images to signal
identification and empathy with constituents using group characteristics?

29. Questions
▪ How do the group characteristics (gender, race, age, etc.) of people that
Members of Congress pose with affect how they are perceived?
▪ Social media “homestyle”
▪ Is there evidence that Members of Congress use social media images to signal
identification and empathy with constituents using group characteristics?

30. Data
300,000+ Facebook
images with text
posts for accounts of:
300 US House
members.
56 US Senate
members.

31. Goals and Methods
▪ Identify race of individuals pictured in Facebook profiles of House Members.
▪ Viola-Jones Algorithm
▪ Train a convolutional neural network race classifier.
▪ Explore how distribution of racial groups in photos compare to congressional
district demographics, partisanship and ideology.
▪ Compare Facebook profile “demographics” with district demographics, party id
and DW-Nominate scores.

32. Results
▪ Democrats and Republicans in the US House of Representatives have very
different social media styles.
▪ Evidence that Democrats use Facebook images to elicit racial identification
and empathy among constituents.

33. How you see an image…
▪ Image as data.
▪ Eg 620x412
pixel image.
You see:
{Donald
Trump, blue
tie, black suit,
blue
background,
anger}

34. How a computer sees an image...
▪ 640x412 pixel
image.
▪ 3 Channels:
Red, Green,
Blue
▪ 3 640x412
matrices of
pixel intensity
values.
▪ -or- 3x640x412
= 791,040 x 1
vector

35. Human image classification is robust

36. Machine image classification is error prone...
Image Credit: Andrej Karpathy

37. Theoretical means of image feature extraction
limited mostly to faces...
Viola-Jones Object Detection Framework

38. Data driven approach/supervised machine learning
approach – train models utilizing pixel intensity
data...
Collect labeled
images.
Train a machine
learning
classifier.
Test classifier
accuracy.
CIFAR-10 library
of 32x32 labels
images
benchmark
performance.

39. One layer neural network
X1
X2
X3
▪ Inputs multiplied by weights
and added create “hidden”
layer.
▪ Hidden layer passed
through “activation function”
multiplied by another set of
weights to generate class
probabilities/scores.
▪ Simplest model discussed
by psychologist Rosenblatt
(1958)

40. Neural network activation functions
▪ Most common activation
functions are sigmoid and
tangent.
▪ Optimizing predictions
requires
▪ Choice of activation
functions and;
▪ Choice of weights.

41. Backpropagation
▪ Rumelhart, Hinton and Williams (1986)
▪ Selection of weights involves:
▪ Forward pass - Calculation of loss function.
▪ Backward pass – Use of chain rule and stochastic gradient descent to iteratively
calculate new weights.

42. Convolutional neural networks for image feature
classification
Multi-layer neural network involving series of activation functions on chunks of pixel data.

43. Convolutional neural networks for image feature
classification
• Equivalent of
passing pixel data
through a number
of “filters.”
• Discover which
“filter” is activated
by which labeled
image category.
• Output is highest
probability category
given filter
responses
Image credit: Andrej Karpathy

44. Convolutional neural network: building a race
classifier
• Labeled image data
• 60,000 high school yearbook images, 1960-2013
• 6,000 images sampled from Congressional Facebook dataset we collected.
• Categories: White, African-American, East Asian, Hispanic.
• 16-layer CNN model for large-scale image recognition from CNN Model Zoo by
Simonyan and Zisserman (2015): http://arxiv.org/pdf/1409.1556.pdf and
https://gist.github.com/ksimonyan/211839e770f7b538e2d8#file-readme-md

45. Convolutional neural network: building a race
classifier
• Step 1: Identify
faces using Viola-
Jones algorithm.
• Image on the right is
a Facebook photo
from Representative
Tammy Duckworth ‘s
(D-IL) profile.

46. Convolutional neural network: building a race
classifier
• Step 2: Label race of
faces.

47. Convolutional neural network: building a race
classifier
• Step 3: Train CNN
on labeled data.

48. Convolutional neural network: building a race
classifier
• Step 4: Test classifier accuracy
• Avg. cross-validated accuracy rates of 90% for whites, 85% for African-
American, 75% for Asian, 65% for HIspanic.

49. Convolutional neural network: building a race
classifier
• Step 5: Estimate race of individuals in Congressional Facebook image set using
trained model.

50. Race and partisanship in House Facebook
image posts (white House members)
White Democrats post Facebook
photos of …
African-Americans at 4x the rate of
white Republicans
Hispanics at 1.2x the rate of white
Republicans.
Asians at 2x the rate of white
Republicans.

51. Race and partisanship in House Facebook
image posts (white House members)
Even conditional on relevant district
demographics, state and region
fixed effects, evidence of conscious
efforts by partisans to
include/exclude racial groups in
Facebook image posts.
Democrats (white): +6% more
African-Americans in posts.
Republicans (white): +6% more
whites in posts.

52. Identification and empathy: district
demographics and Facebook image posts
Do MCs strategically post photos of
racial groups to engender
identification and empathy from
constituents?
Overall strong evidence that they do.
Strong relationship between % of
racial group in district and % of racial
group posted in Facebook profiles.

53. Identification and empathy: district demographics
and Facebook image posts by party
Strategic use of race in image posts much more evident among Democrats than Republicans

54. Identification and empathy: district demographics
and Facebook image posts by party
Y = % white in Facebook profile photos
White Democrats more
“race conscious” when
posting FB photos.
After conditioning on
state and region fixed effects and
district demographics, Democrats
Facebook photos more likely to reflect
racial/ethnic mix of district.

55. Identification and empathy: district demographics
and Facebook image posts by party
Representation =
% White in Facebook profile photos –
% White in Congressional District
Whites over-represented in
Facebook photos
of white Democrats and
Republicans…

56. Identification and empathy: district demographics
and Facebook image posts by party
Representation =
% Black in Facebook profile photos –
% Black in Congressional District
African-Americans under-represented
in Facebook photos of Republican MCs
by an average of about 3.8%

57. Identification and empathy: district demographics
and Facebook image posts by party
Representation =
% Hispanic in Facebook profile photos
% Hispanic in Congressional District
Hispanics under-represented
In Facebook photos of both parties, more
so among Democrats

58. Identification and empathy: district demographics
and Facebook image posts by party
Representation =
% Asian in Facebook profile photos
% Asian in Congressional District
Asians under-represented in Facebook
photos of white Democrats.

59. Discussion
• Modern computational methods allow for the large scale analysis of
images.
• Here we build a race classifier for images using convolutional neural
networks.

60. Discussion
▪ Characteristics of people that politicians pose with shape
perceptions.
▪ Democrats and Republicans in the US House of Representatives
have very different social media styles.
▪ Evidence that Democrats use Facebook images to elicit racial
identification and empathy among constituents.