Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Revealing the Hidden Patterns of
News Photos:
Analysis of Millions of News Photos through
GDELT and Deep Learning-based Vi...
2
3
4
Roles of News Photos
● Influence people’s perception
● Enhance reader’s memory
● Deliver emotion otherwise hard to be
conv...
6
7
Why was this photo not picked?
(source: https://www.donaldjtrump.com)
8
To Understand Messages of Photos
● We need to know
○ What are shown in the photos
○ How they are portrayed
9
Challenges in News Photo Analysis
● Text mining has been a useful tool for
analyzing news text
→ What is the appropriate t...
Conventional Tool for Photo Analysis
● Manual coding … hard to scale
11
Deep Learning for Image Recognition
12
Object Recognition
13
Emotion Detection
https://www.microsoft.com/cognitive-services/en-us/emotion-api 14
Deep learning enables us to study
news photos in large-scale
15
Goal of This Work
● To offer a general understanding of
news photos
○ What are shown in the photos?
○ How are people portr...
● We can crawl photos from news
websites and analyze them
● But, setting the deep learning
framework and training it take
...
18
GDELT Visual GKG (VGKG)
● Collects news articles around the world
● Extract photos from the articles
● Calls Google Cloud ...
Example of VGKG
20160101004500 http://www.bbc.co.uk/news/uk-35205943 http://ichef.bbci.co.
uk/news/1024/cpsprodpb/B89F/pro...
(Potential) Limitations of GDELT
● List of news sources is not explicitly
announced (also, growing) - coverage
bias might ...
Our Dataset - Full
● GKG and VGKG in January 2016
● Popularity measured by Alexa.com
22
Our Dataset - 7 Popular News Media
● Top 30 & > 1K records
23
Data Preprocessing
● Keep labels whose confidence score ≥ .8
http://i2.cdn.turner.com/cnnnext/dam/assets/160116174054-kerr...
What Are Shown in the Photos?
Common Objects in News Photos
25
News Topics and Relevant Photos
● News photos should relate with topics
of news articles
→ Common objects might be differe...
Person is the Most Common Object
27
But, in Travel, Person is Uncommon
28
Region-related Sections
29
● Why does this matter?
Western Media and the Third World
● Golan reports that western mass media
strengthen the portrayal of the third
world by r...
How CNN deals with MENA region?
31
How Are People Portrayed?
From the Perspective of Emotion
32
Classification of Emotions
33https://articulation360.wordpress.com/2011/08/26/emotions-memory-game/
Google API Can Detect 4 Emotions
34https://articulation360.wordpress.com/2011/08/26/emotions-memory-game/
SURPRISE
SORROW ...
Neutral (75%) or Joy (24%)
● Among 11,127 faces (in 7 popular media),
2,740 faces (24.6%) have one of emotions
● Most of t...
Nonverbal & Verbal Communication
● Happy faces accelerate the cognitive
processing of positive words and slow
down that of...
We Use Microsoft Face API
● Measures smiling intensity (0.0~1.0)
37
0.998 0.0 (baby)
https://www.microsoft.com/cognitive-s...
Smile Comes with Positive Text
● Positive correlation between smile
intensity and tone (sentiment) of the
text
⍴=0.225
38
How Are People Portrayed?
From the Perspective of Gender
39
Previous Studies on News Media
1. Men outnumber women
2. Men and women are associated with
particular roles
3. More women ...
Again, We Use Microsoft Face API
41https://www.microsoft.com/cognitive-services/en-us/face-api
● Measures Gender and Age
Unequal Gender Representation
0.5
42
Stereotyping: Women in “Living”
0.5
43
Women Smile More Than Men
44
Younger Women, Older Men
45
Case Study
Portrayal of Politicians
46
Smiling Politicians
● Goodnow (2010) found that Obama
smiles more than Clinton in photos in
Time magazine
● Smile gives a ...
Bias of CNN Toward Sanders?
(Smiling faces / All faces)
* CNN even uses “Sorrow” faces for Sanders
48
Pro-Clinton Media Behave Similarly
49
Summary and Future Work
50
Key Findings
● What are shown in the news photos
○ People commonly appear (≥ 40.5% @top500)
● How they are portrayed
○ Peo...
Deeper Analysis on Text and Photos
● Headline and photos?
● Topic and photos?
● Keywords and photos?
52
Building PhotoBiasMeter.org
● Showing the preference of media
outlets toward candidates over time
● Challenges
○ Modeling ...
@haewoon
Full paper is available via
http://arxiv.org/abs/1603.04531
54
Upcoming SlideShare
Loading in …5
×

Revealing the Hidden Patterns of News Photos: Analysis of Millions of News Photos through GDELT and Deep Learning-based Vision APIs

745 views

Published on

Presentation @ ICWSM Workshop on NEws and publiC Opinion (NECO), 2016

Published in: Data & Analytics
  • Be the first to comment

Revealing the Hidden Patterns of News Photos: Analysis of Millions of News Photos through GDELT and Deep Learning-based Vision APIs

  1. 1. Revealing the Hidden Patterns of News Photos: Analysis of Millions of News Photos through GDELT and Deep Learning-based Vision APIs Haewoon Kwak Jisun An Qatar Computing Research Institute Hamad Bin Khalifa University
  2. 2. 2
  3. 3. 3
  4. 4. 4
  5. 5. Roles of News Photos ● Influence people’s perception ● Enhance reader’s memory ● Deliver emotion otherwise hard to be conveyed 5
  6. 6. 6
  7. 7. 7 Why was this photo not picked? (source: https://www.donaldjtrump.com)
  8. 8. 8
  9. 9. To Understand Messages of Photos ● We need to know ○ What are shown in the photos ○ How they are portrayed 9
  10. 10. Challenges in News Photo Analysis ● Text mining has been a useful tool for analyzing news text → What is the appropriate tool for examining news photos? 10
  11. 11. Conventional Tool for Photo Analysis ● Manual coding … hard to scale 11
  12. 12. Deep Learning for Image Recognition 12
  13. 13. Object Recognition 13
  14. 14. Emotion Detection https://www.microsoft.com/cognitive-services/en-us/emotion-api 14
  15. 15. Deep learning enables us to study news photos in large-scale 15
  16. 16. Goal of This Work ● To offer a general understanding of news photos ○ What are shown in the photos? ○ How are people portrayed? ■ From the perspective of emotion ■ From the perspective of gender ● Case study: Portrayal of politicians 16
  17. 17. ● We can crawl photos from news websites and analyze them ● But, setting the deep learning framework and training it take time/money/... Data Collection 17
  18. 18. 18
  19. 19. GDELT Visual GKG (VGKG) ● Collects news articles around the world ● Extract photos from the articles ● Calls Google Cloud Vision API to analyze photos ● VGKG is available since 1 Jan 2016 http: //blog.gdeltproject.org/announcing-the-new-gdelt-visual-global- knowledge-graph-vgkg/ 19
  20. 20. Example of VGKG 20160101004500 http://www.bbc.co.uk/news/uk-35205943 http://ichef.bbci.co. uk/news/1024/cpsprodpb/B89F/production/_87436274_87436273.jpg profession<FIELD>0. 95780987<FIELD>/m/063km<RECORD>person<FIELD>0. 85714287<FIELD>/m/01g317<RECORD>close up<FIELD>0. 82379222<FIELD>/m/02cqfm<RECORD>bishop<FIELD>0. 78259438<FIELD>/m/01b7b<RECORD>bishop<FIELD>0. 71334475<FIELD>/m/027k49j<RECORD>diocesan bishop<FIELD>0. 64282793<FIELD>/m/09sgrf<RECORD>auxiliary bishop<FIELD>0. 57118613<FIELD>/m/05mx3n<RECORD>clergy<FIELD>0.57113737<FIELD>/m/0db79 -2 <FIELD>-2<FIELD>-2<FIELD>-2 0.95443642<FIELD>3.199043<FIELD>12.419704<FIELD>- 7.179338<FIELD>0.621747<FIELD>433,215;575,215;575,357;433,357<FIELD>-2<FIELD>- 2<FIELD>-2<FIELD>2<FIELD>-2<FIELD>-2<FIELD>-2 20 Date, Document identifier (URL), Image URL, Labels (description, confidence score, unique id), Geographic Landmarks, Logos, Safe Search, Faces (Angle, Emotion, etc.), OCR
  21. 21. (Potential) Limitations of GDELT ● List of news sources is not explicitly announced (also, growing) - coverage bias might exist ● Our work of comparing GDELT with another news dataset will be presented in the poster session Two Tales of the World: Comparison of Widely Used World News Datasets - GDELT and EventRegistry Haewoon Kwak and Jisun An ICWSM'16: The 10th International Conference on Web and Social Media (poster), 2016 21
  22. 22. Our Dataset - Full ● GKG and VGKG in January 2016 ● Popularity measured by Alexa.com 22
  23. 23. Our Dataset - 7 Popular News Media ● Top 30 & > 1K records 23
  24. 24. Data Preprocessing ● Keep labels whose confidence score ≥ .8 http://i2.cdn.turner.com/cnnnext/dam/assets/160116174054-kerry-handshake-zarif-large-169.jpg Person 0.84957772 Business 0.59667766 24
  25. 25. What Are Shown in the Photos? Common Objects in News Photos 25
  26. 26. News Topics and Relevant Photos ● News photos should relate with topics of news articles → Common objects might be different across topics ● CNN has ‘section’ info. in its URL http://edition.cnn.com/2016/04/07/travel/japan- best-of-wakayama/index.html http://edition.cnn.com/2016/05/05/politics/paul- ryan-donald-trump-republican-resistance/index.html 26
  27. 27. Person is the Most Common Object 27
  28. 28. But, in Travel, Person is Uncommon 28
  29. 29. Region-related Sections 29 ● Why does this matter?
  30. 30. Western Media and the Third World ● Golan reports that western mass media strengthen the portrayal of the third world by reporting war, poverty, famine, conflicts, violence and conflicts and lead to negative perception (Golan 2008). 30
  31. 31. How CNN deals with MENA region? 31
  32. 32. How Are People Portrayed? From the Perspective of Emotion 32
  33. 33. Classification of Emotions 33https://articulation360.wordpress.com/2011/08/26/emotions-memory-game/
  34. 34. Google API Can Detect 4 Emotions 34https://articulation360.wordpress.com/2011/08/26/emotions-memory-game/ SURPRISE SORROW ANGER JOY
  35. 35. Neutral (75%) or Joy (24%) ● Among 11,127 faces (in 7 popular media), 2,740 faces (24.6%) have one of emotions ● Most of them (2,665 faces) express joy 35
  36. 36. Nonverbal & Verbal Communication ● Happy faces accelerate the cognitive processing of positive words and slow down that of negative words (Stenberg, Wiking, and Dahl 1998) 36
  37. 37. We Use Microsoft Face API ● Measures smiling intensity (0.0~1.0) 37 0.998 0.0 (baby) https://www.microsoft.com/cognitive-services/en-us/face-api
  38. 38. Smile Comes with Positive Text ● Positive correlation between smile intensity and tone (sentiment) of the text ⍴=0.225 38
  39. 39. How Are People Portrayed? From the Perspective of Gender 39
  40. 40. Previous Studies on News Media 1. Men outnumber women 2. Men and women are associated with particular roles 3. More women than men were depicted as happy and calm. → We’ll verify this in large-scale 40
  41. 41. Again, We Use Microsoft Face API 41https://www.microsoft.com/cognitive-services/en-us/face-api ● Measures Gender and Age
  42. 42. Unequal Gender Representation 0.5 42
  43. 43. Stereotyping: Women in “Living” 0.5 43
  44. 44. Women Smile More Than Men 44
  45. 45. Younger Women, Older Men 45
  46. 46. Case Study Portrayal of Politicians 46
  47. 47. Smiling Politicians ● Goodnow (2010) found that Obama smiles more than Clinton in photos in Time magazine ● Smile gives a positive, non-threatening impression to viewers (Goffman 1979) 47
  48. 48. Bias of CNN Toward Sanders? (Smiling faces / All faces) * CNN even uses “Sorrow” faces for Sanders 48
  49. 49. Pro-Clinton Media Behave Similarly 49
  50. 50. Summary and Future Work 50
  51. 51. Key Findings ● What are shown in the news photos ○ People commonly appear (≥ 40.5% @top500) ● How they are portrayed ○ People are neutral (75%) or smiling (24%) ○ Gender representation is unequal ○ Gender role stereotyping is found ○ Women smile more and look younger than men ● Clinton smiles more than Sanders in some media →We demonstrate the great potential of deep learning for computational journalism 51
  52. 52. Deeper Analysis on Text and Photos ● Headline and photos? ● Topic and photos? ● Keywords and photos? 52
  53. 53. Building PhotoBiasMeter.org ● Showing the preference of media outlets toward candidates over time ● Challenges ○ Modeling complex dimension of preference - “Smile” is only one dimension 53
  54. 54. @haewoon Full paper is available via http://arxiv.org/abs/1603.04531 54

×