This document discusses experimental categorization and deep visualization as approaches to analyzing culture using techniques from visual computing and data analysis. It describes cultural analytics as using heterogeneous data and visualization tools to analyze relationships and patterns in global culture. Experimental categorization involves developing innovative and provisional dimensions for describing visual images based on characteristics like color, shape, and texture. Deep visualization makes the underlying technical processes behind images and visualizations more transparent. Examples of projects applying these approaches are provided.

1. Experimental categorization and
deep visualization
AISV Lund | August 22 | 2019
Everardo Reyes
Université Paris 8
Lev Manovich
City University of New York
http://lab.culturalanalytics.info/

2. Cultural Analytics

3. Cultural Analytics
● It’s an approach to analysis of culture defined by Manovich in 2005
● Using techniques and technologies in visual computing
● Using data methods to see contemporary global culture
● Some research questions:
○ What are the themes, styles, behaviors, and their patterns in contemporary global culture?
○ Where are they active? (Spatial distribution)
○ When they emerge? How they diffuse, change over time?

4. Cultural Analytics
● A cultural analyst designs methods, datasets, visualization models, and
exploratory tools that allow us to see relationships among
heterogeneous data
● Our contribution in this talk:
○ Experimental Categorization
○ Deep Visualization

5. Examples of projects
http://lab.culturalanalytics.info/p/projects.html

6. The Plastic Semiotics Approach

7. The Plastic Semiotics Approach
We focus on three plastic categories: colors, shapes, textures
Such categories are helpful for segmenting an image, naming its parts, and
establishing syntactic, semantic, and pragmatic correlations of meaning
These dimensions can be measured in all images regardless of the content
and type. Such measurements were commonly used in Computer Vision
since early 1990s for content-based image retrieval

8. Analysis of “La fille à la montre et au chapeau”
Göran Sonesson, 1988

9. Plastic Categories as Low-Level Image Descriptors
Color Descriptors Tone extraction using color models (RGB, HSV, HSL)
Shape Descriptors Geometrical features (aspect ratio, rectangularity,
circularity, elongatedness)
Texture Descriptors Numerical vectors that determine if points in a ROI are
lesser, greater or approximate to a central reference
point

10. Mid-Level
Spatial e.g. top, center, right, left...
Structural e.g. Has-parts, has-eyes,
has-legs...
Holistic e.g. Furry, shiny, metallic...
High-Level
“Animal”, “human”, “house”, “advertising”...
Moods and sentiments (happy, angry, sad...)

11. Microsoft Azure's Computer Vision
https://azure.microsoft.com/en-us/services/cognitive-services/computer-vision/
Nowadays CV services use different kinds of neural networks. In this example we see high-level
descriptors based on combinations of lower levels. It is important to note that they are not
obtained manually

12. Experimental Categorization

13. Developed by pho.to, based on AI algorithms for image processing
Consider software like PhotoLab...

14. PhotoLab has more than 600 effects, 30 categories, and templates that allow users to
create their own “combos”

15. PhotoLab Combos
https://photolab.me/d/5021744

17. Pho.to API
http://developers.pho.to/documentation/methods/filters/color

18. Experimental Categorization
● A term to describe innovative, speculative, unstable, and work-in-progress
measurements and dimensions to describe visual images
● Although they are built on top of other low and mid-level descriptors, they are unique
in the sense that they are based on visual characteristics, not on linguistic terms, e.g.
colors, sharpness, blur, gradients, texture patterns, degree of skewness, circularity,
rectangularity of shapes
● An experimental category may thus receive any name that stands for a combination
of descriptors in a determined context of use (a tool for photo retouching,
information retrieval systems, etc.)

19. How to implement an experimental categorization?
● Let’s see two cases:
○ Case 01: As pre-defined routines that are used at the moment of
generating data (while extracting, segmenting, naming)
○ Case 02: As a method for visual analytics, i.e. as a combination
of user-interfaces to explore data by using obtained
measurements

20. Case 01: As pre-defined routines that are used at the moment of generating
data (while extracting, segmenting, naming). For example, creating new shape
descriptors such as Russ (2011):

21. Case 02: As a method for visual analytics, i.e. as a combination of
user-interfaces to explore data by using obtained measurements.
While one of the most common methods to extract and determine
frequent colors in an image is to measure separately R, G, B channels...
Most dominant color
Red: 67.52
Green: 67.07
Blue: 70.05

22. We can also calculate the difference between percentage amounts of
colors in an image based on k-means clusters
-30.43 3.88 = 26.55
The difference can be used as a threshold to filter images in a large collection
e.g. “Plot only those images whose % distance is higher than 25”

23. From the following dataset (Albums released in 1988)

24. This is a Radial Plot representation of the same data, along the chromatic circle

25. Radial Plots Using Percentage Difference
Percentage Diff. = 0 Percentage Diff. = 25 Percentage Diff. = 50
Albums with high/less variety of colors can be filtered

26. L’homme à la guitare
George Braque, 1914
An application in painting

27. Experimental Categories:
● Differences in ROIs
● Texture Shadows
● Infrequent Plastic Signs
● High Variability: most varied
categories
● Colored Shapes: particles that have
similar color and shape
Interactive 3D Surface Plot
Kai Uwe Barthel, 2004
https://imagej.nih.gov/ij/plugins/surface-plot-3d.html

28. Deep Visualization

29. Blink: The Ising Model (2004-2008)
https://vimeo.com/257843869
George Legrady
Let’s consider this artwork

30. Blink: The Ising Model (2004-2008)
George Legrady
polypTelic: The Ising Model (2004-2008)
George Legrady
And this is the process occuring behind the scenes

31. Deep Visualization
A method to make evident the underlying (and sometimes simultaneous)
technical processes that occur behind the scenes and are often taken for
granted by common users

32. Venice, Italy, May 2019 Mount San Jancinto, August 2019
Some inspiration from culture-nature, “seeing” historical traces in layers

33. In our interactive
visualization, the
user can slide
a bar to reveal
other technical layers
that could be
potentially seen

34. Converting between polar and
Cartesian coordinates:
x = r cos φ
y = r sin φ
var radius = 7;
var polarX = radius * Math.sin( a.c01_h_hsl * Math.PI/
var polarY = radius * Math.cos( a.c01_h_hsl * Math.PI/
These other layers that could be seen, put together, and modify each other in real time

35. One of the theoretical and practical tasks of Cultural Analytics is the
development of the appropriate measures of cultural diversity, structure
(relations, networks), dynamics (temporal changes), and variability (themes
and deviations) for different types of media and cultural fields
For Pictorial Semiotics the main obstacle before developing a full
computer-aided analysis of pictures may not be technological, but rather
phenomenological. We need to understand the nature of those holistic,
topological, and physiognomic properties of perception on which human
beings focus in order to make sense of pictures from a plastic point of view

36. Tack för din uppmärksamhet
AISV | Lund | August 22 | 2019
Everardo Reyes & Lev Manovich
http://lab.culturalanalytics.info/