This document summarizes a DeepLabCut AI Residency program held in September 2022. The residency aimed to democratize AI-based behavioral analysis and increase diversity, equity, and inclusion among code contributors and users. Six residents from diverse backgrounds were fully funded. Main mentors included professors Mackenzie and Alexander Mathis who lead labs on neural mechanisms and complex behavior. Residents collaborated at the EPFL campus in Switzerland, which hosts numerous neuroscience and biotechnology institutes. Projects involved using DeepLabCut and machine learning to analyze mouse sexual behavior, identify behavior in wildlife camera trap images, and develop tools to classify and summarize results. The residency helped build an inclusive community around open-source behavioral analysis tools.

1. DeepLabCut AI Residency
September 2022
Vic Shao-Chih Chiang (蔣紹志)

2. Introduction to DeepLabCut (downloaded > 375,000 times)

3. Introduction to DeepLabCut

4. Limitations of traditional approaches
AV Devineni & KM Scalpen (2022) Front Behav Neurosci
CT Miller et al. CM Niell (2022) Curr Biol

5. Limitations of traditional approaches
CL Ebbesen & RC Froemke (2021) Curr Opin Neurobiol
S Ocklenburg, J Packheiser, GH Gadea (2022) Curr Psychol

6. How machine learning helps
MH McCullough & GJ Goodhill (2021) Curr Opin Neurobiol
L Wade et al. J Bilzon (2022) PeerJ
L Jiang et al. S Ostadabbas (2022) Comput Vis Image Underst

7. DeepLabCut
T Nath et al. MW Mathis (2019) Nat Protoc

8. Pose estimation examples
J Stenum et al. RT Roemmich (2021) Sensors
ZY Chiu et al. MC Yip (2022) arXiv
S Jasek et al. G Jekely (2021) Biorxiv A Corver et al. A Gordus (2021) Curr Biol

9. Pose estimation
examples
BY Hayden, HS Park, J Zimmermann (2021) Am J Primatol
RJ Cotton (2022) arXiv
JKY Ma & AA Wrench (2022) Int J Lang Commun Disord

10. Pose estimation examples
NL Henriksen et al. S Pankratova (2022) Physiol Rep
L Wade et al. J Bilzon (2022) PeerJ
A Annenerino et al. PI Gouma (2022) Plos One
J Lauer et al. MW Mathis (2022) Nat Methods

11. Better for biodiversity, ethics & beyond
EB Contreras et al. IQ Whishaw (2022) Neurosci Biobehav Rev
D Tuia et al. TB Wolf (2022) Nat Commun
JT Schultz et al. CJ Clemente (2021) Proc R Soc Lond B Biol Sci
HM Luo, JY Yu (2022) ACM Trans Graph

12. Examples for Mental Health
N Grujic et al. R Polania (2022) Sci Adv
RJ Li et al. J Alty (2022) Comput Biol Med
LY Liu et al. JY Chen (2021) Sensors
H Abbasi et al. AJC McMorland (2022) Biorxiv

13. DeepLabCut AI Residency
• democratize AI-based behavioral analysis to explicitly increase DEI in code contributors and
users
• empower users to modify code, develop new add-ons, and become leaders within the
DeepLabCut consortium, also in the broader open source community
• building a diverse community of contributors, that more users will organically be welcomed into
the community
• Six residents from across the globe fully funded by Chan Zuckerberg Initiatives, Center For
Imaging at EPFL, NVIDIA, and ELLIS Unit & Center For Intelligent Systems (including costs for
travel, housing, etc covered)

14. DeepLabCut AI Residency Main Mentors
Mackenzie Mathis
• Assistant Professor at EPFL
• Fellowships: NSF, Rowland Institute at Harvard
• Scholars: Vallee Foundation, European Lab for
Learning & Intelligent Systems
• Federation of European Neuroscience Societies Young
Investigator Prize
• Bertarelli Foundation Chair of Integrative Neuroscience
• Lab: mechanisms underlying adaptive behavior in
intelligent systems.
Alexander Mathis
• Assistant Professor at EPFL
• Lab: How the brain creates
complex behavior
Steffen Schneider
• Google PhD Fellow
• Research: build machine learning
models capable of approaching
the performance of biological
brains

15. The Residents
• 1st time working in small
close-knit group with
people from such diverse
background, identity and
lived experience:
• Disability, religion,
upbringing
• Nationalities & ethnicities:
Sweden, Kenya, Turkey,
Argentina
• Research: bat
aerodynamics, hawk gaze,
pigeon cognition, place
cells, wildlife conservation

16. The Research Institution
• Founded in 2013, the Campus functions like a giant
incubator and focuses on pure science and its translation
into real solutions and practical outcomes that have an
impact on society and the world.
• Water from the lake is the main energy source for the
building and it significantly reduces the environmental
impact of this unique global research center.
• hosting research institutes and biotechnology companies:
Center for Neuroprosthetics, Human Brain Project, Blue
Brain Project, Wyss Center for Bio and
Neuroengineering, Foundation for Innovative New
Diagnostics, Health 2030 Genome Center
• the former Merck Serono building, part of the Swiss
Innovation Park, bought by Ernesto Bertarelli and
Hansjörg Wyss

17. The Research Institution

18. Geneva, Switzerland
• Full of world organisation headquarters: Red Cross, United Nations, World Trade Organisation, World
Meteorological Organization
• Switzerland is very humanitarian-based, since 19th century is a neutral territory: high equity & quality in healthcare
& education, low crime rate, world’s highest minimum wage
• Drinking water from the 7000 Swiss lakes (Lac Léman in Geneva) & breathing mountain air from the Northern Alps
• First time to do daily commute to cross country (live in France, and go to Geneva for work daily)

19. Faugères, France
• One of my closest friend’s
wedding!
• Small town in South France
(population ~500)
• Held at a meteor impact site
(~10,000 years ago)
• Now is a vineyard, which
has special terroir from the
schist due to the meteor
impact

20. Featured on
Nature Methods

21. B6D2F1 sexual behaviour: 9 cage setup
• When dealing with so
many mice & long
recordings (>180 min)
• Hard to sync hormonal
priming (48h P, 4-6h E)

22. B6D2F1 sexual behaviour: Cropping individual videos
• Night vision
• One cage of 9
• Very low resolution
• Even if choosing selected bodyparts (a
priori assumption for the parameters we
are measuring
• Difficult to accurately label bodyparts –
lethal to training a model
• Also blurring & close interactions
• 95% training fraction
• Neural Network: ResNet 50
• Trained for 200,000 (~1 day on GPU)
• High pixel error & very jittery poses

23. B6D2F1 sexual behaviour: alternative strategies
• Label on high resolution
videos
• Train with multiple data
augmentation that could
mimick conditions of the 9
cage setup

24. B6D2F1 sexual behaviour: alternative strategy
• Neural Network: ResNet 50
• Tried single and different combinations
of data augmentation types
• Include: random order, add random
pixels to image, Gaussian noise,
cartoon, different blend, different blur,
white color space, contrast, Contrast
Limited Adaptive Histogram Equalization
(CLAHE), convolution, sharpen, canny
edge, flipping, affine, scaling, rotate,
elastic transformation, jigsaw, corrupt,
frost, brightness, emboss, pooling,
weather
• Trained for 1,000,000 (~2 days on GPU)
• Acceptable pixel error & pose jitteriness

25. B6D2F1 sexual behaviour: Behavioural Classification
SimBA (Simple Behavioral Analysis)
• Labelled mounting,
intromission, ejaculation,
anogenital sniffing in >
10,000 frames
• Trained various random
forest classifier for each
behaviour

26. B6D2F1 sexual behaviour: Behavioural Classification
SimBA (Simple Behavioral Analysis)
• Used different behaviour
classifiers and generate
the prediction of the
behaviour (individual
videos)
• Model can still improve
in accuracy, but this is
further dependent on the
DLC model.

27. B6D2F1 sexual behaviour: complete by November in
time for SFN (San Diego, CA)
• Python script to obtain MSB parameters
• Proof of principle on 180min recording of
one cropped cage of 9 cage setup
• Generate ethogram (bodyparts coord &
behaviour across time)
R Heijkoop, PT Huijgens, EMS Snoeren
(2018) Behav Brain Res V Susoy et al. ADT Samuel (2021) Cell

28. B6D2F1 sexual behaviour: remaining to do
• Crop individual videos from 9 cage
setup
• ~35 mice
• >180 minutes (~4 videos)
• Across 11 tests
• Run DLC models on all videos
• Run SimBA models on all videos
• Obtain sexual parameters on all videos
• Aggregate all sexual parameters per
mice, across time.
• Unsupervised clustering of each mice
according these parameters
JR Knoedler et al. NM Shah (2022) Cell

29. Featured on
DeepLabCut
Medium

30. London, UK
• I used to study & work at University College London
(UCL) – went back to catch up with some old
friends
• Reunion with the cohort of my program, those that
are still in the area
• First time living by myself in a foreign country,
renting houses with various types of people.
• First time being surrounded by so many
intellectuals – radically shaped my world views and
values
• First time learning about the diversity of
neuroscience from elite research at UCL (Faculty of
Brain Sciences with schools of: neurodegeneration,
mental health, language, ear, ophthalmology, prion
etc.)
• Mostly stayed in London last time I lived in Europe
though, so this time I got to know more about the
European culture

31. Liechtenstein
• Established in 1719 (now ~300 years)
• Human activity since the Paleolithic
period
• The only two countries in the world that
are doubly land-locked
• Sixth smaller country in the world (only
around the same size as Falmouth, MA)
• Army, defence and diplomatic are all
outsourced to Switzerland
• Become a citizen: live here for >30y, or
majority votes from the referendum

32. Guest – Merve Noyan from Hugging Face
• Hugging Face is a community
and data science platform that
provides: Tools that enable
users to build, train and deploy
ML models based on open
source (OS) code and
technologies.
• Merve Noyan is a Google
Developer Expert in Machine
Learning, a former machine
learning engineer and currently
working as a developer
advocate at Hugging Face.
She's the host of The Inference
podcast where she hosts
machine learning engineers,
data scientists and researchers
in machine learning.

33. Guest – Sara Beery on MegaDetector
• Conservation biologists invest a huge amount of time reviewing camera trap images, and a huge
fraction of that time is spent reviewing images they aren't interested in. This primarily includes empty
images, but for many projects, images of people and vehicles are also "noise", or at least need to be
handled separately from animals.
• Machine learning can accelerate this process, letting biologists spend their time on the images that
matter – Megadetector detect animals, people, and vehicles
• Assistant professor at MIT EECS' Faculty of AI and Decision Making and CSAIL, and a Visiting
Researcher at Google working on Auto Arborist. research focuses on building computer vision
methods that enable global-scale environmental and biodiversity monitoring across data modalities

34. Identifying sexual behaviour in the wild
JD Monk et al. CE McDonough (2019) Nat Ecol Evol

35. Neuroscience research has
moved from studies conducted
across a spectrum of animals to
reliance on a few species
V Marx (2021) Nat Methods
AS Mathuru et al. S Teseo (2020) Neurosci Biobehav Rev

36. Sexual behaviour occurs in a collective behaviour context
J Sliwa (2021) Science

37. Evolution & sexual diversity
CEC Campos, MRD Souza, A Fouquet
(2021) Herpetol Notes
S Hasan et al. T Ahmed (2021) Bangladesh J Zool
SGQ Lebron et al. M Kuntner (2016)
Sci Rep
JJ Falk, MS Webster, DR Rubenstein (2021)
Curr Biol

38. Camera traps are
pivotal to understand
sexual diversity in
the wild
NCM Sun, KJC Pei, LY Wu (2021) Sci Rep F Coppola & A Felicioli (2021) Sci Rep
RD Fuentes et al. A Batista (2021) Therya Note
AH Dalziell & JA Welbergen (2022) Int J Avian Sci

39. Megadetector

40. Megadetector combines
with DeepLabCut
Contributions from other DLC AI Residents

41. Pose information to infer behaviour
Contributions from other DLC AI Residents

42. Pose information to infer behaviour
Contributions from other DLC AI Residents

43. Cookbook
Contributions from other DLC AI Residents

44. Behaviorual models &
species identification JW Bubnicki, M Churski, DPJ Kuijper (2016) Methods Ecol Evol
MA Tabak et al. RS Miller (2022) Biorxiv

45. Highly portable, cloud-enabled, non-invasive
neuroscience measures
LA Hawkes, A Fahlman, K Sato (2021) Philos Trans R Soc Lond B Biol Sci

46. Important for wildlife conservation… but also ethical decisions
NR Sommer & KM Ferraro (2021) Conserv Sci Pract

47. Sexual behaviour
could be
paramount for the
welfare of some
organisms
E Edelblutte, R Krithivasan, MN Hayek (2021) Conservation Biol

48. Human Sexual Diversity

49. Identifying Sexual
Behaviour in the Wild
• Batch processing camera trap dataset:
270,450 camera trap images from 187
camera locations in Wellington, New
Zealand
• Sort out those that are physically
interacting
• Visually inspect for possible sexual
behaviour
• Identify the species picked up
• Good tool for future neuroscientists to
study sexual behaviour in the wild, to
generate hypotheses
• Batch process google COLAB available
open source on DeepLabCut github
• Hope to also present this as part of SFN

50. Featured on DeepLabCut Twitter

51. Annecy, France
• Referred to as “Pearl of the Alps” and “Venice of the Alps”
• Occupied by the Celtics since 4th Century BC, then occupied by Ancient Romans in 1st Century AD,
became part of France in 1860
• Le Palais de I’Île built in 12th century that was a penitentiary, money factory, and asylum
• Annecy has the most transparent lake in Europe, with transparency ~10 meters beneath

52. Switzerland – Oeschinensee, Lausanne
• Lake Oeschinensee is 1500 meters above sea level
• Caused by an ancient landslide that gave rise to this natural dam
• Visited EPFL main campus in Lausanne
• Lausanne is the Olympic Capital since 1915
• Switerzerland: Law enforce most activities on Sundays are prohibited – most things
close on Sundays
• Law also enforce people with pets must be in pairs, to minimise the pets being lonely
• The Swiss culture is generally more punctual, more formal, and more reserved

53. Guest – Devis Tuia on wildlife conservation
• Professor at EPFL: leads the Environmental
Computational Science and Earth Observation
laboratory (ECEO) in Sion
• The climate and biodiversity crises are leaving scars on
our planet. To find solutions to these challenges, we
need technologies enabling environmental monitoring
which is spatially and temporally resolved. The good
news is that we now have access to an impressive
amount of data acquired by satellites, airplanes, drones
and citizen sensors. There are some recent learning
algorithms to make sense of this remote sensing data,
as well as the interactive approaches to make them
available to the public.
• Making remote sensing accessible to everyone!
Developing algorithms for human machine interaction
• Open the black box: interpretable deep learning and
uncertainties in environmental modeling
• Digital wildlife conservation: using imaging to
automatize censuses and conservation efforts

54. Guest – Chris Holdgraf on 2i2c and Project Jupyter
• Project Jupyter is a project with goals to develop open-source
software, open standards, and services for interactive computing
across multiple programming languages.
• 2i2c: non-profit dedicated to - developing and hosting open
infrastructure for interactive computing in research and education
- supporting open communities that underlie this infrastructure
• Executive Director of 2i2c & core member of Project Jupyter

55. Using ModelZoo models on your own datasets

56. Improving network
performance on
unbalanced data
via augmentation

57. Exploring active
learning with
DeepLabCut

58. Clustering in the napari-DeepLabCut GUI

59. Featured on Nature Portfolio

60. Paris, France
• My partner was studying in Paris – visited his past favourite hangouts & landmarks e.g. Montmartre,
Place de la Concorde, Grande Arche
• Paris is known as the City of Lights, with catacombs under the city, and metro operational since 1900s
• Tried various French cuisine: bone marrow, snails, Calf’s head, roast veal, floating island
• Paris Syndrome: those that fantasised about Paris, but the disappointments when actually seeing it,
causing many mental health issues

61. Croatia – Split & Trogir
• Built from 3rd Century BC by ancient Greek
• Replete with Roman & Venice culture
• Went through Ottoman Empire occupation,
Nazi invasion, founding of the Yugoslavia
• Cuisine influenced by Italian and
Mediterranean: seafood, spaghetti, olive oil
• Place with the most discovery of
Neantherdal remains
• Where they created the first pen, tie, and
Dalmatians
• Shootings for Games of Throne

62. Guest – Jonny Saunders on Autopilot
• Systems neuroscientist that studies the computational mechanisms of complex sounds in the
auditory cortex, and released Auto-Pi-Lot
• Auto-Pi-Lot is a Python framework for performing complex, hardware-intensive behavioral
experiments with swarms of networked Raspberry Pis. As a tool, it provides researchers with a
toolkit of flexible modules to design experiments without rigid programming & API limitations.

63. Collaborations with Albert Kao – Golden Shiners
AS Peshkin et al. ID Couzin (2013) Curr Biol
• Obligate schooling fish species, typically less than 10cm in length, minnow, native to
eastern North America
• Hundreds of millions bred annually for use as bait & forage
• Key studies in the field of collective animal behaviour have used golden shiners as their
study species (including few previous studies by Albert)
• No studies to date on copulation / mating / sexual behaviour of golden shiners –
especially interesting as an understanding of this in the context of collective behaviour.

64. Injustices in AI
• AI excludes marginalised
minorities, serves
predominantly the WEIRD
population that are
heterosexual, cisgender,
white males
• pervaded with biases as
observed in the “gorilla”
incident pointed out by Jacky
Alciné.

65. Injustices in AI
• Job, university, politics, crimes all use mathematical models
• A good model should be transparent to know how calculations
are done, and equal
• From the 80s, US started using models to calculate university
rankings – universities then reduced admission rate and
tuition fees to boost their rankings
• Poorer families do not have time or money for extracurricular
activities and tutoring to fulfil these admission criteria, nor the
money to pay for elite universities
• Other models include personality tests for jobs to screen out
candidates, and social media algorithms to determine what
content you see, that influenced elections
• To improve models: need to rethink the value behind these
models – what makes a good society? And admit that some
things cannot be quantified e.g. beauty, equity, charity

66. Injustices in AI
• You did not own the operating system itself, and
in fact you entered into a legally binding
agreement indicating the very precise ways in
which you would be allowed to use the software
before you could actually use your computer.
• Just like you do not truly have an alternative to
your landlord users and businesses often have
few alternatives to the most egregious software
license terms and audits. How many businesses,
large or small, can operate without Microsoft,
Google, Adobe, and social media today?
• Open source software makes it more
economically accessible to marginalised
communities

67. Diversifying AI

68. Diversifying AI

69. Palma, Spain
• Spain began to colonise way before the British: the Americas, and the Philippines
• Now Spanish is the 4th most spoken language: after English, Mandarin, Hindi
• Colonial history means the language mix with various colonies’ language e.g. Nahuatl of Aztec,
Quechua of Inca
• 7th Century AD onwards for 700 years, the North African Arabs (the Moors) occupied Spain, so
Spanish contains a lot of Arabic.
• Like British, is a constitutional monarch: now reigned by King Felipe VI
• The people eats lunch & dinner very late, and enjoys the Flamenco music

70. Switzerland – Bern, Spiez
• Bern was established in the 10th Century AD
• Legend has it, the name comes from the first animal the Count hunted, which is the Bear
• Filled with cobblestone alleys, and sandstone buildings – World UNESCO heritage
• Spiez is next to Lake Thun, with a castle built in the 15th century AD by the Burgandandy Kind Rufus II
• Switzerland: First time independently living in a country that speaks languages I am not familiar in: French,
German, Romansch, Italian
• Switzerland: The only country that practices direct democracy: the electorate decides on policy initiatives
without elected representatives as proxies

71. DeepLabCut AI Residency – Application Video

72. DeepLabCut AI Residency – Application Video
https://youtu.be/3FE2HhQnFh0