softwarestudies.com
The Science of Culture?

Theory and examples of computational analysis
of visual culture
Dr. Lev Manov...
softwarestudies.com 2
manovich.lev@gmail.com
@manovich
instagram: levmanovich
facebook: lev.manovich
our lab:
www.software...
softwarestudies.com 3
2004 - present:

new types / cheaper urban sensors; new
ways to capture human behavior / new
forms o...
softwarestudies.com 4
softwarestudies.com 7
2006 - present:
new research fields that use big cultural
data (including social media, user
generate...
softwarestudies.com 8
-social computing

-computational social science

-other CS fields: computer vision, media
computing,...
softwarestudies.com 9
softwarestudies.com 10
We have to always remember that not everybody is using social media.

Example: our map of 100 milli...
softwarestudies.com 11
Key characteristics of social media
relevant for the study of cultural and
social patterns:





softwarestudies.com 12
1 

very high spatial and temporal resolution 

in cities (time and location metadata)
2 

automati...
softwarestudies.com 13


4

engagement: likes, comments, shares, 

web navigation, gameplay, etc. -
for the first time, we ...
softwarestudies.com 14
data allows us to qualitatively study 

interactions between -



a) people (online and physically)...
softwarestudies.com 15
softwarestudies.com 16
softwarestudies.com 17
softwarestudies.com 18
softwarestudies.com 19
softwarestudies.com 20
social networks as medium
and message 



(using Instagram as the
example)


softwarestudies.com 21
message 1 (“surface”): what people who
use social networks say, do, capture



message 2 (“depth”):...
softwarestudies.com 22
softwarestudies.com 24
softwarestudies.com 25
softwarestudies.com 26
softwarestudies.com 27
softwarestudies.com 28
softwarestudies.com 29
softwarestudies.com 30
softwarestudies.com 31
data analysis and visualization as
“medium”
Features extracted from content, the
available metadata...
softwarestudies.com 32
the general and the particular:
18th-20th centuries: 

social statistics, social science and
data v...
softwarestudies.com 35
softwarestudies.com 36
examples of cultural
analysis using large
visual data



all examples in the following slides are f...
softwarestudies.com 37
Visual evolution of news 

media
(front covers of a single
newspapers)
softwarestudies.com 38
softwarestudies.com 39
Visual evolution of news 

media - longer period
(4535 Time magazine 

covers, 1923-2009)
softwarestudies.com 40


4535 Time covers
1923-2009.
Organized by date,
left to right, top to
bottom.
Every pattern we
obs...
softwarestudies.com 41


4535 Time covers
1923-2009.
closeup: 1920s
softwarestudies.com 42


4535 Time covers
1923-2009.
closeup: 1990s-2000s
softwarestudies.com 43


4535 Time covers 1923-2009 (left to right). Each cover is represented by a single vertical line.
softwarestudies.com
Image plots of 4535 Time covers, 1923-2009. X-axis = date; Y-axis = saturation mean.

44
softwarestudies.com 45closeup

softwarestudies.com 46covers that have highest saturation (1960s)
softwarestudies.com 47
History of a cultural 

medium (photography) 

as represented by a single

institution (MoMA)
softwarestudies.com 48
In 2013 we were invited by MoMA to analyze their whole
photo collection and contribute to the OBJEC...
softwarestudies.com 49
Seeing the museum
collection: 20,000 photographs
from MoMA,1844-1989.

Organized by year 

(top to ...
softwarestudies.com 50
closeup, 1925-1929
softwarestudies.com 51
Mapping an artistic 

movement



(French Impressionists)
softwarestudies.com 52
Visualization of 5000 paintings
of French Impressionist
artists

x and y - first two dimensions
of P...
softwarestudies.com 53closeup
softwarestudies.com 54
Mapping a cultural field
using a large sample 



(883 manga publications
containing 1,074,790 

pag...
softwarestudies.com
1 million manga pages

x - standard deviation
y - entropy
softwarestudies.com 56
Closeups of the bottom left corner and top corner (previous slide). Entropy feature sorts all pages...
softwarestudies.com 57
1 million manga pages plotted as points

x - standard deviation
y - entropy


Some plot areas are d...
softwarestudies.com 58
Visual signatures of cities:
sampling and aggregating
images from a
social media platform
(Instagra...
softwarestudies.com 59
Comparing San Francisco and Tokyo using 50K image samples. 

Photos are organized by average bright...
softwarestudies.com 60
softwarestudies.com 61
Comparing NYC and Tokyo using 50K image samples shared over few days (organized by upload date/time...
softwarestudies.com 62closeup of the visualization from previous slide
softwarestudies.com 63
Example of data
aggregation - reducing 2.3M
photos to 13 data points
(one point per city)
softwarestudies.com 64
Another plot of
cities differences
(using only color
features)
softwarestudies.com
65
How people represent
themselves? How they
construct themselves in 

social media? 



softwarestudies.com 66
Selfiecity project, 2014: analysis of 3200 Instagram single selfie photos from 5 global cities.
http:...
softwarestudies.com 67One of the visualizations from Selfiecity
softwarestudies.com 68
Screenshot from interactive app selfiexploratory: 

http://selfiecity.net/selfiexploratory/
softwarestudies.com 69
softwarestudies.com 70
SelfieSaoPaolo project, 2014. Views of the animated projection.
softwarestudies.com 71
Interfaces for people to
interact with urban social
media data + census and
government data;
Combin...
softwarestudies.com 72
On Broadway project, 2014. http://http://on-broadway.nyc/
softwarestudies.com 73
On Broadway is an interactive installation shown at New York Public Library, 12/2004-1/2016
softwarestudies.com 74Artists team in front of On Broadway installation
softwarestudies.com 77
Interface uses familiar multi-touch gestures to navigate Broadway street in Manhattan (21 km, 30M d...
softwarestudies.com 78
Video showing interaction with On Broadway interactive application on a 46-inch touch screen
softwarestudies.com 79
How exceptional events
are represented in visual
social media? The
exceptional vs the
everyday 



...
softwarestudies.com 80The Exceptional & The Everyday project, 2014
softwarestudies.com 81
The Exceptional & The Everyday 

project, 2014
http://www.the-everyday.net/



The visualization sh...
softwarestudies.com 82
softwarestudies.com 83
softwarestudies.com 84
softwarestudies.com 85
using visual social 

media to predict socio-
economic indicators
softwarestudies.com 86
Mehrdad Yazdani and Lev Manovich.

“Predicting Social Trends from Non-
photographic Images on Twitt...
softwarestudies.com 87
softwarestudies.com 88
softwarestudies.com 91
our lab: free tools, publications, projects, news:
www.softwarestudies.com




articles, books, pro...
Science of culture? Computational analysis and visualization of cultural image collections and datasets
Science of culture? Computational analysis and visualization of cultural image collections and datasets
Science of culture? Computational analysis and visualization of cultural image collections and datasets
Science of culture? Computational analysis and visualization of cultural image collections and datasets
Science of culture? Computational analysis and visualization of cultural image collections and datasets
Science of culture? Computational analysis and visualization of cultural image collections and datasets
Science of culture? Computational analysis and visualization of cultural image collections and datasets
Science of culture? Computational analysis and visualization of cultural image collections and datasets
Science of culture? Computational analysis and visualization of cultural image collections and datasets
Upcoming SlideShare
Loading in …5
×

Science of culture? Computational analysis and visualization of cultural image collections and datasets

13,841 views

Published on

Concepts, research questions and examples of computational analysis and visualizations of cultural image collections from our research lab (softwarestudies.com) created between 2009 and 2015. Visualized datasets include 20,000 images from MoMA photo collection, 773 Vincent van Gogh paintings, and 2.3 million Instagram images from 13 cities worldwide. (Note that the original presentation has a few videos that are not part of this PDF document.)

Published in: Data & Analytics

Science of culture? Computational analysis and visualization of cultural image collections and datasets

  1. 1. softwarestudies.com The Science of Culture?
 Theory and examples of computational analysis of visual culture Dr. Lev Manovich
 Professor of Computer Science, The Graduate Center, City University of New York | Director, Software Studies Initiative, California Institute for Telecommunication and Information 03/2016
 

  2. 2. softwarestudies.com 2 manovich.lev@gmail.com @manovich instagram: levmanovich facebook: lev.manovich our lab: www.softwarestudies.com
 

  3. 3. softwarestudies.com 3 2004 - present:
 new types / cheaper urban sensors; new ways to capture human behavior / new forms of digital culture - social media + user generated content - higher resolution satellite photography - location + movement data (phones) - Arduino for interfacing with sensors - data from city bike programs - open data movement - etc.
  4. 4. softwarestudies.com 4
  5. 5. softwarestudies.com 7 2006 - present: new research fields that use big cultural data (including social media, user generated content and digitized cultural heritage) to study social and cultural patterns and cultural histories
  6. 6. softwarestudies.com 8 -social computing
 -computational social science
 -other CS fields: computer vision, media computing, web science, NLP
 -science of cities, urban analytics 
 -digital humanities, digital history, digital art history
  7. 7. softwarestudies.com 9
  8. 8. softwarestudies.com 10 We have to always remember that not everybody is using social media.
 Example: our map of 100 million tweets with images (sampled from 265 million tweets, 2011-2014)
  9. 9. softwarestudies.com 11 Key characteristics of social media relevant for the study of cultural and social patterns:
 
 

  10. 10. softwarestudies.com 12 1 
 very high spatial and temporal resolution 
 in cities (time and location metadata) 2 
 automatic detection of subjects + styles + sentiment 3
 connectivity (content propagation, 
 influences, groups, structure of networks)
  11. 11. softwarestudies.com 13 
 4
 engagement: likes, comments, shares, 
 web navigation, gameplay, etc. - for the first time, we can study cultural reception on mass scale

  12. 12. softwarestudies.com 14 data allows us to qualitatively study 
 interactions between -
 
 a) people (online and physically)
 b) people and spaces
 c) people and cultural software tools d) people and cultural artifacts (“reception,” “engagement”)
  13. 13. softwarestudies.com 15
  14. 14. softwarestudies.com 16
  15. 15. softwarestudies.com 17
  16. 16. softwarestudies.com 18
  17. 17. softwarestudies.com 19
  18. 18. softwarestudies.com 20 social networks as medium and message 
 
 (using Instagram as the example) 

  19. 19. softwarestudies.com 21 message 1 (“surface”): what people who use social networks say, do, capture
 
 message 2 (“depth”): what they “really” say, do, and how they live medium 1: digital vernacular photography as it exists on Instagram
 
 medium 2: Instagram as its own visual, narrative and networked media platform

  20. 20. softwarestudies.com 22
  21. 21. softwarestudies.com 24
  22. 22. softwarestudies.com 25
  23. 23. softwarestudies.com 26
  24. 24. softwarestudies.com 27
  25. 25. softwarestudies.com 28
  26. 26. softwarestudies.com 29
  27. 27. softwarestudies.com 30
  28. 28. softwarestudies.com 31 data analysis and visualization as “medium” Features extracted from content, the available metadata, and data mining techniques determine what we can learn from social media Visualization layouts and options further influences what patterns we can see, the meanings, and interpretations
 

  29. 29. softwarestudies.com 32 the general and the particular: 18th-20th centuries: 
 social statistics, social science and data visualization - aggregation, summarization, reduction; focusing on the regular (that can be modeled and predicted) 21st century: from summarization to individualization; from general to focusing on variability and individual
 

  30. 30. softwarestudies.com 35
  31. 31. softwarestudies.com 36 examples of cultural analysis using large visual data
 
 all examples in the following slides are from 
 - projects created in our lab, 
 - collaborative projects with external collaborators
 - student work from Manovich’s classes 
 2009 - 2015
  32. 32. softwarestudies.com 37 Visual evolution of news 
 media (front covers of a single newspapers)
  33. 33. softwarestudies.com 38
  34. 34. softwarestudies.com 39 Visual evolution of news 
 media - longer period (4535 Time magazine 
 covers, 1923-2009)
  35. 35. softwarestudies.com 40 
 4535 Time covers 1923-2009. Organized by date, left to right, top to bottom. Every pattern we observe is continuous, with changes taking places over years or decades.
  36. 36. softwarestudies.com 41 
 4535 Time covers 1923-2009. closeup: 1920s
  37. 37. softwarestudies.com 42 
 4535 Time covers 1923-2009. closeup: 1990s-2000s
  38. 38. softwarestudies.com 43 
 4535 Time covers 1923-2009 (left to right). Each cover is represented by a single vertical line.
  39. 39. softwarestudies.com Image plots of 4535 Time covers, 1923-2009. X-axis = date; Y-axis = saturation mean.
 44
  40. 40. softwarestudies.com 45closeup

  41. 41. softwarestudies.com 46covers that have highest saturation (1960s)
  42. 42. softwarestudies.com 47 History of a cultural 
 medium (photography) 
 as represented by a single
 institution (MoMA)
  43. 43. softwarestudies.com 48 In 2013 we were invited by MoMA to analyze their whole photo collection and contribute to the OBJECT : PHOTO exhibition website
  44. 44. softwarestudies.com 49 Seeing the museum collection: 20,000 photographs from MoMA,1844-1989.
 Organized by year 
 (top to bottom). Each bar shows photographs from a particular year.
  45. 45. softwarestudies.com 50 closeup, 1925-1929
  46. 46. softwarestudies.com 51 Mapping an artistic 
 movement
 
 (French Impressionists)
  47. 47. softwarestudies.com 52 Visualization of 5000 paintings of French Impressionist artists
 x and y - first two dimensions of PCA using 200 features The familiar paintings of French impressionists (see closeup on next slide) turn to be only %20- %30 of their whole creative output
  48. 48. softwarestudies.com 53closeup
  49. 49. softwarestudies.com 54 Mapping a cultural field using a large sample 
 
 (883 manga publications containing 1,074,790 
 pages)
  50. 50. softwarestudies.com 1 million manga pages
 x - standard deviation y - entropy
  51. 51. softwarestudies.com 56 Closeups of the bottom left corner and top corner (previous slide). Entropy feature sorts all pages according to low detail/no texture/flat - high detail/texture/3D dimension. Visualization reveals continuos variation on this dimension. This example suggests that our standard concept of "style" may not be appropriate when looking at particular characteristics of big cultural samples (because "style assumes presence of distinct characteristics, not continuos variation across a whole dimension).
  52. 52. softwarestudies.com 57 1 million manga pages plotted as points
 x - standard deviation y - entropy 
 Some plot areas are densely filled in, while others are almost empty. Why manga visual language developed in this way? Visualization of a large number of samples allows us to map a cultural fields to see what is typical and what is rare, and what kind of clusters (if they exist) are present in this field.
  53. 53. softwarestudies.com 58 Visual signatures of cities: sampling and aggregating images from a social media platform (Instagram)
  54. 54. softwarestudies.com 59 Comparing San Francisco and Tokyo using 50K image samples. 
 Photos are organized by average brightness (distance to plot center) and average hue (angle).
  55. 55. softwarestudies.com 60
  56. 56. softwarestudies.com 61 Comparing NYC and Tokyo using 50K image samples shared over few days (organized by upload date/time.)
  57. 57. softwarestudies.com 62closeup of the visualization from previous slide
  58. 58. softwarestudies.com 63 Example of data aggregation - reducing 2.3M photos to 13 data points (one point per city)
  59. 59. softwarestudies.com 64 Another plot of cities differences (using only color features)
  60. 60. softwarestudies.com 65 How people represent themselves? How they construct themselves in 
 social media? 
 

  61. 61. softwarestudies.com 66 Selfiecity project, 2014: analysis of 3200 Instagram single selfie photos from 5 global cities. http://selfiecity.net

  62. 62. softwarestudies.com 67One of the visualizations from Selfiecity
  63. 63. softwarestudies.com 68 Screenshot from interactive app selfiexploratory: 
 http://selfiecity.net/selfiexploratory/
  64. 64. softwarestudies.com 69
  65. 65. softwarestudies.com 70 SelfieSaoPaolo project, 2014. Views of the animated projection.
  66. 66. softwarestudies.com 71 Interfaces for people to interact with urban social media data + census and government data; Combining multiple types + resolution of data
  67. 67. softwarestudies.com 72 On Broadway project, 2014. http://http://on-broadway.nyc/
  68. 68. softwarestudies.com 73 On Broadway is an interactive installation shown at New York Public Library, 12/2004-1/2016
  69. 69. softwarestudies.com 74Artists team in front of On Broadway installation
  70. 70. softwarestudies.com 77 Interface uses familiar multi-touch gestures to navigate Broadway street in Manhattan (21 km, 30M data points)
  71. 71. softwarestudies.com 78 Video showing interaction with On Broadway interactive application on a 46-inch touch screen
  72. 72. softwarestudies.com 79 How exceptional events are represented in visual social media? The exceptional vs the everyday 
 
 Social media sensors vs. pop media coverage
  73. 73. softwarestudies.com 80The Exceptional & The Everyday project, 2014
  74. 74. softwarestudies.com 81 The Exceptional & The Everyday 
 project, 2014 http://www.the-everyday.net/
 
 The visualization shows 13,208 
 Instagram images shared by 
 6,165 people in the center of 
 Kiev during 2014 Ukrainian revolution (
 February 17 - February 22, 2014). The photos are organized 
 chronologically (left to right, top to 
 bottom). The right column shows 
 summary of the events from Wikipedia page about the revolution. A single condensed narrative history (Wikipedia text) vs. 
 visual experiences of thousands of people 
 (Instagram)? The second is potentially richer - but also more difficult to interpet. Can we narrate history without aggregation 
 and summarization? History as timelines of million of people?
  75. 75. softwarestudies.com 82
  76. 76. softwarestudies.com 83
  77. 77. softwarestudies.com 84
  78. 78. softwarestudies.com 85 using visual social 
 media to predict socio- economic indicators
  79. 79. softwarestudies.com 86 Mehrdad Yazdani and Lev Manovich.
 “Predicting Social Trends from Non- photographic Images on Twitter.” Big Data and the Humanities workshop, IEEE 2015 Big Data conference. We classified 1 million twitter images shared in 20 US cities in 2013 using Google free Deep Learning Model available to all researchers

  80. 80. softwarestudies.com 87
  81. 81. softwarestudies.com 88
  82. 82. softwarestudies.com 91 our lab: free tools, publications, projects, news: www.softwarestudies.com 
 
 articles, books, projects:
 www.manovich.net academia.edu
 
 
 contact:
 www.facebook.com/lev.manovich
 twitter.com/manovich 
 manovich.lev@gmail.com
 
 


×