This document summarizes a study that analyzed millions of news photos through the GDELT database and deep learning vision APIs to understand patterns in how people and topics are portrayed. Key findings include that news photos commonly feature people, who are often portrayed as neutral or smiling. The analysis found unequal gender representation and stereotyping, with women smiling more and looking younger than men. A case study on politician photos from CNN found Hillary Clinton smiling more than Bernie Sanders in some media coverage. The researchers propose deeper analysis of the relationships between photos, headlines, topics and gender representation to build tools for computational journalism.

1. Revealing the Hidden Patterns of
News Photos:
Analysis of Millions of News Photos through
GDELT and Deep Learning-based Vision APIs
Haewoon Kwak Jisun An
Qatar Computing Research Institute
Hamad Bin Khalifa University

2. 2

3. 3

4. 4

5. Roles of News Photos
● Influence people’s perception
● Enhance reader’s memory
● Deliver emotion otherwise hard to be
conveyed
5

6. 6

7. 7
Why was this photo not picked?
(source: https://www.donaldjtrump.com)

8. 8

9. To Understand Messages of Photos
● We need to know
○ What are shown in the photos
○ How they are portrayed
9

10. Challenges in News Photo Analysis
● Text mining has been a useful tool for
analyzing news text
→ What is the appropriate tool for
examining news photos?
10

11. Conventional Tool for Photo Analysis
● Manual coding … hard to scale
11

12. Deep Learning for Image Recognition
12

13. Object Recognition
13

14. Emotion Detection
https://www.microsoft.com/cognitive-services/en-us/emotion-api 14

15. Deep learning enables us to study
news photos in large-scale
15

16. Goal of This Work
● To offer a general understanding of
news photos
○ What are shown in the photos?
○ How are people portrayed?
■ From the perspective of emotion
■ From the perspective of gender
● Case study: Portrayal of politicians
16

17. ● We can crawl photos from news
websites and analyze them
● But, setting the deep learning
framework and training it take
time/money/...
Data Collection
17

18. 18

19. GDELT Visual GKG (VGKG)
● Collects news articles around the world
● Extract photos from the articles
● Calls Google Cloud Vision API to analyze
photos
● VGKG is available since 1 Jan 2016 http:
//blog.gdeltproject.org/announcing-the-new-gdelt-visual-global-
knowledge-graph-vgkg/
19

20. Example of VGKG
20160101004500 http://www.bbc.co.uk/news/uk-35205943 http://ichef.bbci.co.
uk/news/1024/cpsprodpb/B89F/production/_87436274_87436273.jpg profession<FIELD>0.
95780987<FIELD>/m/063km<RECORD>person<FIELD>0.
85714287<FIELD>/m/01g317<RECORD>close up<FIELD>0.
82379222<FIELD>/m/02cqfm<RECORD>bishop<FIELD>0.
78259438<FIELD>/m/01b7b<RECORD>bishop<FIELD>0.
71334475<FIELD>/m/027k49j<RECORD>diocesan bishop<FIELD>0.
64282793<FIELD>/m/09sgrf<RECORD>auxiliary bishop<FIELD>0.
57118613<FIELD>/m/05mx3n<RECORD>clergy<FIELD>0.57113737<FIELD>/m/0db79 -2
<FIELD>-2<FIELD>-2<FIELD>-2 0.95443642<FIELD>3.199043<FIELD>12.419704<FIELD>-
7.179338<FIELD>0.621747<FIELD>433,215;575,215;575,357;433,357<FIELD>-2<FIELD>-
2<FIELD>-2<FIELD>2<FIELD>-2<FIELD>-2<FIELD>-2
20
Date, Document identifier (URL), Image URL, Labels (description,
confidence score, unique id), Geographic Landmarks, Logos, Safe
Search, Faces (Angle, Emotion, etc.), OCR

21. (Potential) Limitations of GDELT
● List of news sources is not explicitly
announced (also, growing) - coverage
bias might exist
● Our work of comparing GDELT with
another news dataset will be presented
in the poster session
Two Tales of the World: Comparison of Widely Used World News Datasets - GDELT and EventRegistry
Haewoon Kwak and Jisun An
ICWSM'16: The 10th International Conference on Web and Social Media (poster), 2016 21

22. Our Dataset - Full
● GKG and VGKG in January 2016
● Popularity measured by Alexa.com
22

23. Our Dataset - 7 Popular News Media
● Top 30 & > 1K records
23

24. Data Preprocessing
● Keep labels whose confidence score ≥ .8
http://i2.cdn.turner.com/cnnnext/dam/assets/160116174054-kerry-handshake-zarif-large-169.jpg
Person 0.84957772
Business 0.59667766
24

25. What Are Shown in the Photos?
Common Objects in News Photos
25

26. News Topics and Relevant Photos
● News photos should relate with topics
of news articles
→ Common objects might be different
across topics
● CNN has ‘section’ info. in its URL
http://edition.cnn.com/2016/04/07/travel/japan-
best-of-wakayama/index.html
http://edition.cnn.com/2016/05/05/politics/paul-
ryan-donald-trump-republican-resistance/index.html 26

27. Person is the Most Common Object
27

28. But, in Travel, Person is Uncommon
28

29. Region-related Sections
29
● Why does this matter?

30. Western Media and the Third World
● Golan reports that western mass media
strengthen the portrayal of the third
world by reporting war, poverty, famine,
conflicts, violence and conflicts and lead
to negative perception (Golan 2008).
30

31. How CNN deals with MENA region?
31

32. How Are People Portrayed?
From the Perspective of Emotion
32

33. Classification of Emotions
33https://articulation360.wordpress.com/2011/08/26/emotions-memory-game/

34. Google API Can Detect 4 Emotions
34https://articulation360.wordpress.com/2011/08/26/emotions-memory-game/
SURPRISE
SORROW ANGER
JOY

35. Neutral (75%) or Joy (24%)
● Among 11,127 faces (in 7 popular media),
2,740 faces (24.6%) have one of emotions
● Most of them (2,665 faces) express joy
35

36. Nonverbal & Verbal Communication
● Happy faces accelerate the cognitive
processing of positive words and slow
down that of negative words (Stenberg,
Wiking, and Dahl 1998)
36

37. We Use Microsoft Face API
● Measures smiling intensity (0.0~1.0)
37
0.998 0.0 (baby)
https://www.microsoft.com/cognitive-services/en-us/face-api

38. Smile Comes with Positive Text
● Positive correlation between smile
intensity and tone (sentiment) of the
text
⍴=0.225
38

39. How Are People Portrayed?
From the Perspective of Gender
39

40. Previous Studies on News Media
1. Men outnumber women
2. Men and women are associated with
particular roles
3. More women than men were depicted
as happy and calm.
→ We’ll verify this in large-scale
40

41. Again, We Use Microsoft Face API
41https://www.microsoft.com/cognitive-services/en-us/face-api
● Measures Gender and Age

42. Unequal Gender Representation
0.5
42

43. Stereotyping: Women in “Living”
0.5
43

44. Women Smile More Than Men
44

45. Younger Women, Older Men
45

46. Case Study
Portrayal of Politicians
46

47. Smiling Politicians
● Goodnow (2010) found that Obama
smiles more than Clinton in photos in
Time magazine
● Smile gives a positive, non-threatening
impression to viewers (Goffman 1979)
47

48. Bias of CNN Toward Sanders?
(Smiling faces / All faces)
* CNN even uses “Sorrow” faces for Sanders
48

49. Pro-Clinton Media Behave Similarly
49

50. Summary and Future Work
50

51. Key Findings
● What are shown in the news photos
○ People commonly appear (≥ 40.5% @top500)
● How they are portrayed
○ People are neutral (75%) or smiling (24%)
○ Gender representation is unequal
○ Gender role stereotyping is found
○ Women smile more and look younger than men
● Clinton smiles more than Sanders in some media
→We demonstrate the great potential of deep
learning
for computational journalism
51

52. Deeper Analysis on Text and Photos
● Headline and photos?
● Topic and photos?
● Keywords and photos?
52

53. Building PhotoBiasMeter.org
● Showing the preference of media
outlets toward candidates over time
● Challenges
○ Modeling complex dimension of
preference - “Smile” is only one
dimension
53

54. @haewoon
Full paper is available via
http://arxiv.org/abs/1603.04531
54