Research project on building and interpreting computer vision networks with the purpose to develop visual digital methods for social and media research. Project diary: https://thesocialplatforms.wordpress.com/2020/09/10/computer-vision-networks/

1. Making methods with vision APIs,
online data & network building
(lessons learnt
)
Dr Janna Joceli Omena
26 January 2022 I CAIS fellowship closing presentation I Computer Vision Networks

2. 2
Janna Joceli Omena I CAIS fellowship closing presentation I Computer Vision Networks
PERSONAL
• Living in Bochu
m
J.J.Omena@fcsh.unl.pt
https://thesocialplatforms.wordpress.com
/
RESEARCH INTERESTS
Digital method
s
Software studies
Visual network analysi
s
Methodological innovation
EDUCATION
Universidade Nova de Lisboa I UT Austin Portugal Progra
m
PhD thesis in Digital Media studies:
Digital methods and technicity-of-the-mediums.
PROFESSIONAL
Invited associate professor in digital media and methods,
NOVA University Lisbo
n
iNOVA Media Lab & Public Data La
b

3. 3
Janna Joceli Omena I CAIS fellowship closing presentation I Computer Vision Networks
1. Situating the computer vision network
approach to study image collections

4. Developing digital visual methods for social and medium research
Janna Joceli Omena I CAIS fellowship closing presentation I Computer Vision Networks
computer vision networks as an ensemble of computational mediums, data,
methods, research, and technical practices orchestrated by the researcher(s)
.
a computer vision network approach offers three different forms of interpreting
the same image collection
:
1. the content of image itself & its web cultural -social-political contexts
2. the site of image audiencing & to whom the images matte
r
3. the site of image circulation

5. 5

6. 6
Computer Vision Networks
Developing digital visual methods for social and medium research
Janna Joceli Omena I CAIS fellowship closing presentation I Computer Vision Networks
Aims Objectives Results
1. Interrogate the potentials and
limitations of computer vision APIs
for social and medium research
• Organise data sprints with field experts to try
and test the computer vision network approach,
while mapping its potentials/limitations.
• Develop research software to facilitate the
processes of network building with vision APIs
outputs. (in collaboration w/ Jason Chao)
• Create a method recipe to explain the approach,
testing and trying it in different contexts
Research software to invoke multiple vision APIs and query image
collections in collaboration with Jason Chao
2. Develop accessible and reproducible
visual methodologies with digital
methods

7. 7
Chao, T. H. J. (2021). Memespector GUI: Graphical User
Interface Client for Computer Vision APIs (Version 0.2)
[Software]. Available from https://github.com/jason-chao/
memespector-gui.
Memespector GUI

8. Offline Image Query and Extraction Tool
8
Chao, T. H. J. & Omena, J. J. (2021). Of
fl
ine Image Query and Extraction Tool
(Version 0.1) [Software]. Available from https://github.com/jason-chao/of
fl
ine-
image-query.

9. 9
Computer Vision Networks
Developing digital visual methods for social and medium research
Janna Joceli Omena I CAIS fellowship closing presentation I Computer Vision Networks
Aims Objectives Results
1. Interrogate the potentials and
limitations of computer vision APIs
for social and medium research
• Organise data sprints with field experts to try
and test the computer vision network approach,
while mapping its potentials/limitations.
• Develop research software to facilitate the
processes of network building with vision APIs
outputs. (in collaboration w/ Jason Chao)
• Create a method recipe to explain the approach,
testing and trying it in different contexts
Research software to invoke multiple vision APIs and query image
collections in collaboration with Jason Chao
A method recipe to build and interpret computer vision networks
(first step to propose a conceptual-methodological model)
2. Develop accessible and reproducible
visual methodologies with digital
methods

10. 10
The method recipe
The method protocol
analyse
images
QUERY OR QUERIES DESIGN
RESEARCH QUESTION
DATASET
DESIGN
PROCESS
VISUALISATION
PROCESS
VISUAL
NETWORK
ANALYSIS
images URL, engagement metrics, timestamps .csv
EXTRACT IMAGE METADATA
IMAGES METADATA
alike Image Tagnet Explorer, Tumbrl Tool, Google Image Extractor
according to analysed the digital platform
DIGITAL PLATFORM
 Alike Instragam, Tumbrl, Google Image ]
IMAGES URL
.csv, .tsv, .txt file with URLs
ORGANISE AND CLEAN IMAGE METADATA
alike Excel, Google Spreadsheet
DOWNLOAD IMAGES FROM URL
DownThemAll or similar
RESIZE IMAGES
BulkResize or similar]
FOLDER OF IMAGES
[.native format]
FOLDER OF RESIZED IMAGES
[.native format]
alike Google Vision API
USE THE WEB VISION API SERVICE
CREATE AN API KEY
BUILD  OR USE AN ALREADY EXSISTING  SCRIPT
script (.py or .php) es: “from google.cloud import vision (...)
NETWORK
[.gexf]
.csv
VISION API METADATA
image url label web entitites image domain
BUILD THE EDGES/NODES TABLE
BUILD THE NETWORK
Table2Net or manually]
VISUALISE THE NETWORK
Gephi]
Gephi image preview plugin]
*Image Network Plotter Script works as well on Pyhton]
INSTALL THE GEPHI
IMAGE PREVIEW PLUGIN
Gephi + domain knowledge]
[es: ForceAtlas2 Spatialisation]
VISUALISE AND SPATIALIZE IT ROUGHLY
CHANGE NETWORK APPEREANCE
ACCORDING TO DATA
ADD PICTURES TO THE NETWORK
label
VISUALLY ANALYSE THE NETWORK
PRESENT/STAGE THE NETWORK
label
GO BACK TO THE
RESEARCH QUESTIONS
label
label
*size
*color
[attributes]
[in-degree, degree,out-degree]
EXPORT THE NETWORK IN AN VECTOR FORMAT
EDIT
ADD
ANNOTATIONS
ADD
A TITLE
ADD
KEY
REFINE
COLORS
ADJUST
LABEL SIZE
[.svg, .eps]
Adobe Illustrator, Inkscape]
mediums / software
technical practice
output
researcher intervention
VISUALISATION
PRESENTATION
label
label
annotation
Title
color 1
color 2
GEPHI DATA LAB SPREADSHEET
GEPHI OVERVIEW PRINTED NETWORK
BIG SCREEN
RUN IT
notebook, terminal
label
label
image url label web entitites image domain
label
label
*size
*color
[attributes]
[in-degree, degree,out-degree]
label
label
annotation
Title
color 1
color 2
Design by Beatrice Gobbo I Concept by Janna Joceli Omena

11. 11
Prerequisites I method recipe
📣No coding skills are demanded
• be willing to practice new methods
• get familiar with a range of software
• bring your own computers to the classroom
• work with spreadsheets and a list of research software

12. 12
Computer Vision Networks
Developing digital visual methods for social and medium research
Janna Joceli Omena I CAIS fellowship closing presentation I Computer Vision Networks
Aims Objectives Results
1. Interrogate the potentials and
limitations of computer vision APIs
for social and medium research
• Organise data sprints with field experts to try
and test the computer vision network approach,
while mapping its potentials/limitations.
• Develop research software to facilitate the
processes of network building with vision APIs
outputs. (in collaboration w/ Jason Chao)
• Create a method recipe to explain the approach,
testing and trying it in different contexts
Research software to invoke multiple vision APIs and query image
collections in collaboration with Jason Chao
A method recipe to build and interpret computer vision networks
(first step to propose a conceptual-methodological model)
2. Develop accessible and reproducible
visual methodologies with digital
methods
Peer-review article + data sprint reports +
tutorials Digital Methods Initiative (Summer/Winter Schools)

13. 13
Trying-and-testing the method recipe
I Digital Methods Summer School 2021 I Projects
Teaching methods and software-using
I Digital Methods Summer School 2021 I Tutorials

14. 14
Diseña // No. 19 (2021): Visual Methods for Online Images: Collection, Circulation, and Machine Co-Creation
Methodological proposal
I Article: The potentials of Google Vision API-based Networks to Study Natively Digital Images

15. 15
Computer Vision Networks
Developing digital visual methods for social and medium research
Janna Joceli Omena I CAIS fellowship closing presentation I Computer Vision Networks
Aims Objectives Results
1. Interrogate the potentials and
limitations of computer vision APIs
for social and medium research
• Organise data sprints with field experts to try
and test the computer vision network approach,
while mapping its potentials/limitations.
• Develop research software to facilitate the
processes of network building with vision APIs
outputs. (in collaboration w/ Jason Chao)
• Create a method recipe to explain the approach,
testing and trying it in different contexts
Research software to invoke multiple vision APIs and query image
collections in collaboration with Jason Chao
A method recipe to build and interpret computer vision networks
(first step to propose a conceptual-methodological model)
2. Develop accessible and reproducible
visual methodologies with digital
methods
Peer-review article + data sprint reports +
tutorials Digital Methods Initiative (Summer/Winter Schools)
Innovative network building and reading techniques

16. Network building without image
s
nodes as computer vision outputs and
_vision apis service
s
_web environments where images come from (e.g. social media, meme generator platforms)
_platform data (e.g. location based-data pointing to countries, images posted by public social media accounts)
16
Reading network visualisation
through fixed layers of interpretatio
n
_centre: what the actors have in common
_periphery: the unique characteristics of the actors
_mid-zone: specific aspects shared among actors or the shadow of a particular acto
r

17. 17
Network building
nodes as web entities and meme environment
s

18. 18
Reading network visualisatio
n
Meme project I DMI Winter School 202
2
I Method recipe, network building and reading techniques: Janna Joceli Omena I Network visualisation: Marco Valli

19. 19
Reading network visualisatio
n
Meme project I DMI Winter School 202
2

20. 20
Reading network visualisatio
n
Meme project I DMI Winter School 202
2

21. 21
Reading network visualisatio
n
Meme project I DMI Winter School 202
2

22. 22
This technique enables
image
cross-platform
analysis
Janna Joceli Omena I CAIS fellowship closing presentation I Computer Vision Networks

23. 23
Comparing
vision APIs’ labels with
networks

24. 24
nodes as vision APIs and their outputs (labels)

25. 25
Cross-platform
image analysis
making sense of image collections without rendering the images
in the network: an alternative solution for the lack of computing
capacity to plot large collections of images

26. 26
Mapping Deepfakes project I DMI Summer School 202
1
Technique conceptualisation: Janna Joceli Omen
a
Network building and analysis: Giulia Tucci
nodes as countries and web entities associated with
the image dataset (Tweets using #deepfakes and imgs)

27. 27
nodes as populists political leaders (Instagram
accounts) and web entities associated
with their recent publications (images)
Political leaders on Instagram I Project by TISE master students
Course: Introduction to Digital Method
s
Technique conceptualisation and recipe: Janna Joceli Omena
Network building and analysis: Ilya Lavrov, Chiara Miozzo,
Marie Palaffre, Franziska Schranz

28. 28
Meme project I DMI Winter School 2022 I Method recipe, network building and reading techniques: Janna Joceli Omena

29. Developing digital visual methods for social and medium research
Janna Joceli Omena I CAIS fellowship closing presentation I Computer Vision Networks
Aims Objectives Results
1. Interrogate the potentials and
limitations of computer vision APIs
for social and medium research
• Organise data sprints with field experts to try
and test the computer vision network approach,
while mapping its potentials/limitations.
• Develop research software to facilitate the
processes of network building with vision APIs
outputs. (in collaboration w/ Jason Chao)
• Create a method recipe to explain the approach,
testing and trying it in different contexts
Research software to invoke multiple vision APIs and query image
collections
A method recipe to build and interpret computer vision networks
(first step to propose a conceptual-methodological model)
2. Develop accessible and reproducible
visual methodologies with digital
methods
Peer-review article & data sprint reports
Tutorials Digital Methods Initiative (Summer/Winter Schools
Innovative network building and reading techniques

30. 30
Three lessons from making, testing & validatin
g
the computer vision network approach
Janna Joceli Omena I CAIS fellowship closing presentation I Computer Vision Networks

31. 31
1 a technicity perspective
is not optional
but a crucial task
Janna Joceli Omena I CAIS fellowship closing presentation I Computer Vision Networks

32. 32
2 to mechanise methodolog
y
does not exclude human
decisions, intervention and
engagement (on the contrary!)
Janna Joceli Omena I CAIS fellowship closing presentation I Computer Vision Networks

33. 33
3 what is still unknown about vision APIs
functioning or its limitations are not
methodological bias bu
t
aspects to be taken into account
through method implementation
Janna Joceli Omena I CAIS fellowship closing presentation I Computer Vision Networks

34. 34
What’s next?

35. 35
Warren Pearce Carlo De Gaetano José Moreno
Rita Sepúlveda
Jason Chao
Amazing group of Erasmus Mundus Master Students I
The seminal class of the course introduction to digital
methods at NOVA University Lisbon
CAIS team & fellows
Thank you
=)
Beatrice Gobbo I Johannes Breuer I Richard Rogers
Giulia Tucci I Francisco Kerche

36. CAIS visiting
NOVA University Lisbon