Images in Social Media Research: Digital Tools and Methodological Challenges Online-Workshop, 10th February 2023. https://tu-dresden.de/gsw/phil/ikm/kuge/forschung/aktuelle-projekte/bildproteste/news/online-workshop-images-in-social-media-research-digital-tools-and-methodological-challenges

1. Analysing image collections with
the computer vision network approach
Janna Joceli Omena
Centre for Interdisciplinary Methodologies,
University of Warwick
Richard Rogers
Media Studies,
University of Amsterdam
Richard Rogers
Media Studies,
University of Amsterdam
Images in Social Media Research:
Digital Tools and Methodological Challenges
Online-Workshop, 10th February 2023.

2. 1. Introduce the computer vision network approach
2. Demonstrate how to make image collections
and do network vision analysis
a. network building, visualisation and analysis
b. visually interpret associations between
cross-platform meme collections and web entities
c. what should we pay attention to?
3. Methodological considerations
Network of web entities associated with cross-platform Covid
meme collections. Source: Authors.
Situating this talk

3. The computer vision
network approach
● Interdisciplinary profile I Software Studies, Digital Methods, AI
techniques and web technologies
● Designed to repurpose the analytics provided by
Web-based vision APIs for research purposes.
● The aggregation of computer vision outputs, web technologies,
online images and their metadata as networks.
● Different forms of interpreting the same image collection (s):
○ Content of the image itself (what we see in the image)
○ Image web cultural-social-political contexts and content
○ Sites of image circulation Three types of computer vision networks.

4. Making image collections
Algorithmic techniques
Network building &
Visualisation
Network vision analysis
Staging the main findings
WHAT WE DO, KNOW &
ENGAGE WITH
e.i. Google Vision AI > Label & Web Detection Methods

5. Google Vision three different outputs for the same Tumblr blog post associated with 2018 Brazilian Presidential Elections. Image URL:
https://78.media.tumblr.com/d9618eb8e17c368bdccff52cacd98e92/tumblr_pfv04zIrLV1xesyy9o1_1280.jpg

6. Google Vision three different outputs for the same Tumblr blog post associated with 2018 Brazilian Presidential Elections. Image URL:
https://78.media.tumblr.com/d9618eb8e17c368bdccff52cacd98e92/tumblr_pfv04zIrLV1xesyy9o1_1280.jpg
A literal description of what the machine sees > LABEL DETECTION
What is in a collection of images?

7. Google Vision three different outputs for the same Tumblr blog post associated with 2018 Brazilian Presidential Elections. Image URL:
https://78.media.tumblr.com/d9618eb8e17c368bdccff52cacd98e92/tumblr_pfv04zIrLV1xesyy9o1_1280.jpg
A contextual web-based description > WEB DETECTION
What are the contexts of the image collection?

8. Google Vision three different outputs for the same Tumblr blog post associated with 2018 Brazilian Presidential Elections. Image URL:
https://78.media.tumblr.com/d9618eb8e17c368bdccff52cacd98e92/tumblr_pfv04zIrLV1xesyy9o1_1280.jpg
Image circulation > WEB DETECTION
What are the sites of image circulation
and who are associated with these?

9. The computer vision
network approach
One or multiple image collections
Algorithmic outputs
Web environment where
images come from
networks built upon:
Researcher’s subject expertise
relevant to the topic area under
investigation
1) multiple forms of interpreting the same image collection (s),
cross-platform image collections
○ Content of the image itself (what we see in the image)
○ Images cultural-social-political contexts and content
○ Sites of image circulation
○ (Cross-)Platforms visual vernaculars
2) visual methods to understand/critique algorithmic outputs
○ Cross vision-API studies
○ Temporal vision analysis

10. Networks of computer vision outputs and images or link domains. Source: Authors, 2022.
Network with images = imgs & CV outputs as nodes Network without images = platforms where imgs come
from & CV outputs as nodes

11. 1. Introduce the computer vision network approach
2. Demonstrate how to make image collections
and do network vision analysis
a. network building, visualisation and analysis
b. visually interpret associations between
cross-platform meme collections and web entities
c. what should we pay attention to?
3. Methodological considerations
Network of web entities associated with cross-platform Covid
meme collections. Source: Authors.
Situating this talk

12. Image collection (s)
1 2 Software or script to invoke vision APIs
Vision API
3
Basic research tools
4
ForceAtlas2
5 A force-directed algorithm
Gephi ❣✨
6 7 Do not rush, be patient. Be curious
and take your time to read the network.
Time for practical work & analysis
Network building (step-by-step)

13. Image collection (s)
1 2 Software or script to invoke vision APIs
Vision API
3
Basic research tools
4
ForceAtlas2
5 A force-directed algorithm
Gephi ❣✨
6 7 Do not rush, be patient. Be curious
and take your time to read the network.
Time for practical work & analysis
Network building and interpretation
Image collection (s)
1

14. Making image collections
1. Scraping and API calling methods provide images unique resource locators (URLs) and identifiers
2. The entry points to make image collections vary according to the environment where images
come from. Common entry points are: keywords, hashtags, account names, list of web pages or media ids
3. Image URLs can have a short lifespan (e.i. TikTok, Instagram)

15. 1. Making meme collections
Facebook
(CrowdTangle)
“covid meme”
Know Your
Meme
1005 image 📁
1000 image 📁
Imgur
(Instant Data Scraper)
1126 image 📁
1000 image 📁
Query Meme environments
(Data collection software)
Meme collections
Instagram
(CrowdTangle) Google Vision AI
Web detection
Computer vision
Adapted from Rogers, Omena, Giorgi et al, 2022.
(Instant Data Scraper)
DownThemAll (Maier, 2019)
to download the images

16. Image collection (s)
1 2 Software or script to invoke vision APIs
Vision API
3
Basic research tools
4
ForceAtlas2
5 A force-directed algorithm
Gephi ❣✨
6 7 Do not rush, be patient. Be curious
and take your time to read the network.
Time for practical work & analysis
Network building and interpretation
Image collection (s)
1 2 Software or script to invoke vision APIs
https:/
/github.com/jason-chao/memespector-gui Vision API
3

17. 1.Save the image collections
on your computer
4. Receive analysis results
Google Vision API
3. Send image files (or URLs)
Full results in
JSON format
2. Insert the vision API credential files, opt for Google
Vision and choose web detection. Upload one image
folder at a time. Run.
Your computer
Simplified & flattened results in
CSV format
5. Output the results
Source: Adapted from Chao & Omena (2022).
2. Memespector-GUI to invoke Google Vision API
6. Download the CSV file. Repeat this
procedure with the other image folders.
Chao (2021)
Know Your Meme, Imgur,
Instagram & Facebook.

18. Image collection (s)
1 2 Software or script to invoke vision APIs
Vision API
3
Basic research tools
4
ForceAtlas2
5 A force-directed algorithm
Gephi ❣✨
6 7 Do not rush, be patient. Be curious
and take your time to read the network.
Time for practical work & analysis
Web detection methods
(web entities)
Vision API
3

19. Web entities
Meme;
Wojak;
Know Your Meme;
Pepe the Frog;
Internet meme;
4chan;
Doge;
Humor
Covid Meme scraped from Know Your Meme.
Source: Rogers, Omena, Giorgi et al, 2022.
A web entity is digital representation of a
real-world object or concept, such as
people, things, places, organizations, etc.

21. Google Vision API
1. The power of Google Image Search
Two indicators of what informs and
defines a web entity for Google Vision:

22. Webpage content surrounding an image
Web entities
Meme;
Wojak;
Know Your Meme;
Pepe the Frog;
Internet meme;
4chan;
Doge;
Humor
(Omena, et. al 2021, see also Li et al. 2017; Google Cloud, 2017; Google user Content, 2020; Sullivan, 2020).

23. Google Vision API
1. The power of Google Image Search
2. Google’s Knowledge Graph
Two indicators of what informs and
defines a web entity for Google Vision:

24. 2. Google’s Knowledge Graph
Web entities
Meme;
Wojak;
Know Your Meme;
Pepe the Frog;
Internet meme;
4chan;
Doge;
Humor

25. Image collection (s)
1 2 Software or script to invoke vision APIs
Vision API
3
Basic research tools
4
ForceAtlas2
5 A force-directed algorithm
Gephi ❣✨
6 7 Do not rush, be patient. Be curious
and take your time to read the network.
Time for practical work & analysis
Network building and interpretation
Image collection (s)
1 2 Software or script to invoke vision APIs
Basic research tools
4
Vision API
3

26. 4
Source: Authors, 2022.

27. Image collection (s)
1 2 Software or script to invoke vision APIs
Vision API
3
Basic research tools
4
ForceAtlas2
5 A force-directed algorithm
Gephi ❣✨
6 7 Do not rush, be patient. Be curious
and take your time to read the network.
Time for practical work & analysis
Basic research tools
4
Network building and interpretation
ForceAtlas2
5 A force-directed algorithm
Image collection (s)
1 2 Software or script to invoke vision APIs
Vision API
3

28. which points to the origin of our dataset and how connections are made
[the importance of degree centrality]
the position of the nodes
respond to attraction
force vs. repulsion by
degree
ForceAtlas2
Jacomy, Venturinim Heymann & Bastian (2014)
5

29. Image collection (s)
1 2 Software or script to invoke vision APIs
Vision API
3
Basic research tools
4
ForceAtlas2
5 A force-directed algorithm
Gephi ❣✨
6 7 Do not rush, be patient. Be curious
and take your time to read the network.
Time for practical work & analysis
Basic research tools
4
Network building and interpretation
ForceAtlas2
5 A force-directed algorithm
Image collection (s)
1 2 Software or script to invoke vision APIs
Vision API
3
Gephi ❣✨
6

30. 6
Gephi ❣✨
Bastian, Heymann & Jacomy (2009)
Source: Authors, 2022.

31. Image collection (s)
1 2 Software or script to invoke vision APIs
Vision API
3
Basic research tools
4
ForceAtlas2
5 A force-directed algorithm
Gephi ❣✨
6 7 Do not rush, be patient. Be curious
and take your time to read the network.
Time for practical work & analysis
Basic research tools
4
Network building and interpretation
ForceAtlas2
5 A force-directed algorithm
Image collection (s)
1 2 Software or script to invoke vision APIs
Vision API
3
Gephi ❣✨
6 7 Do not rush, be patient. Be curious
and take your time to read the network.
Time for practical work & analysis

32. How to visually interpret
associations between
cross-platform meme
collections and web
entities?

33. Source: Authors, 2022.

34. Different network zones,
different research questions
● [centre] Dominant web
entities and the ontological
structure of computer vision
What web entities constitute a
meme? What are Covid memes
to Google Vision AI?
● [mid-zones] What is common
or absent between platforms?
● [periphery] Cross-platform
meme vernaculars
What are Covid meme cultures
characterised across
platforms?
Source: Authors, 2022.

35. Slidedeck by participants of the 2022 DMI Winter School

36. Slidedeck by participants of the 2022 DMI Winter School

37. Giorgi, G., Rogers, R., & Omena, J. J., (2022). How to Make Meme
Collections [How to Guide]. SAGE Research Methods: Doing
Research Online. https:/
/dx.doi.org/10.4135/9781529611267
https:/
/methods.sagepub.com/how-to-guide/how-to-make-me
me-collections
2022 DMI Winter School
https:/
/wiki.digitalmethods.net/Dmi/WinterSchool2022Wha
tIsAMeme
Findings of this study:
Omena, J.J. (2021). A digital methodology for building and
reading computer vision networks (second draft version).
http:/
/bit.ly/ComputerVisionNetworks-method-recipe
Digital methods research:

38. 1. Introduce the computer vision network approach
2. Demonstrate how to make image collections
and do network vision analysis
a. network building, visualisation and analysis
b. visually interpret associations between
cross-platform meme collections and web entities
c. what should we pay attention to?
3. Methodological considerations
Network of web entities associated with cross-platform Covid
meme collections. Source: Authors.
Situating this talk

39. Inquisitive and iterative attitude
before method design & implementation
Technical knowledge about
computational mediums in use
Technical practices and
empirical awareness
Making image collections
Algorithmic techniques
Network building &
Visualisation
Network vision analysis
Staging the main findings
WHAT WE DO, KNOW &
ENGAGE WITH
One or multiple image collections
Algorithmic outputs
METHOD REASONING
Defining the nodes and how
connections are made in the network
Descriptive and interpretive tasks
Conceptually, technically and empirically
translating the newly created arrangements
Methodological considerations
The computer vision network approach
Omena & Rogers, 2023

40. Analysing image collections with
the computer vision network approach
Janna Joceli Omena
Centre for Interdisciplinary Methodologies,
University of Warwick
Richard Rogers
Media Studies,
University of Amsterdam
Images in Social Media Research:
Digital Tools and Methodological Challenges
Online-Workshop, 10th February 2023.
THANK YOU.