This document describes a proposed solution using machine learning and artificial intelligence to help create a safer stadium experience. The solution involves two parts: 1) linking access to stadiums to a verified identity through a fan app for preregistration, and 2) using AI/ML to help detect unwanted behaviors or events early. The rest of the document provides more details on the proposed smart video review framework, including using computer vision and audio analysis techniques to help identify issues like flares, flags, banners, chants including monkey chants. The goal is to help reviewers more efficiently identify potential problems but with privacy, ethics and human oversight.

1. July 4 - 6, 2022
2 n d E d i t i o n

2. BigML, Inc #DutchMLSchool
Smart Safe Stadiums
A nice and safe stadium experience for everyone
Ramon van Ingen
CTIO, Siip group
2
Pablo González
Machine Learning Engineer, BigML

3. BigML, Inc #DutchMLSchool 3
Agenda
• Background
• Our solution
• Smart Video Review Framework

4. BigML, Inc #DutchMLSchool 4

5. BigML, Inc #DutchMLSchool 5
“Our football belongs to everyone”
• Attack plan to counter racism and discrimination
• Applying smart technology
• Aligns with a safer, more secure stadium
environment for players, fans, and employees
Background

6. BigML, Inc #DutchMLSchool 6
A nice and safe stadium experience for everyone
Our solution

7. BigML, Inc #DutchMLSchool 7
2 innovative solutions
• Access right to stadium is linked to a real
identity through a fan-app (pre-registration)
• Early signalling or detection of unwanted
behaviour or events using AI/ML
Our solution

8. BigML, Inc #DutchMLSchool 8
Market potential
• All sport stadiums
• Festivals, concerts, and large events
• Easy integration with existing technology,
systems, and processes in stadiums
• Easy connection to ticket-selling agencies
Our solution
Scalability

9. BigML, Inc #DutchMLSchool 9
Privacy and Ethics by Design
• Pre-registration: Data is securely stored only on individual’s own
device. Only necessary data is shared with consent to attend a
game
• AI: Use of aggregated metadata, no PII or personal
characteristics. Non-discriminatory model that explains what
prediction is based on.
• Segregation of pre-registration and AI: Human in the loop
• Guidance Ethics by University of Applied Science Windesheim
Our solution

10. BigML, Inc #DutchMLSchool 10
Smart Video Review Framework
Focus your attention on what’s important
!
x2 x3
!
!

11. BigML, Inc #DutchMLSchool 11
The framework:
fi
nd, group, label, review
Find interesting events
Flags
Chants
Group events
Experts review clusters
and assign their own labels
1 2
Find anomalous events
Cluster similar events
Smoke bombs / Flares
Label event groups
3
4
Assign labels to
known events
Models learn new
labels from
expert’s feedback
Smart Video Review Framework

12. BigML, Inc #DutchMLSchool 12
Avigilon Video Security System
The hardware
Smart Video Review Framework

13. BigML, Inc #DutchMLSchool 13
The software
Smart Video Review Framework

14. BigML, Inc #DutchMLSchool 14
The problem: We are trying to
fi
nd black swans
The Black Swan: The Impact of the Highly Improbable
Nassim Nicholas Taleb
Smart Video Review Framework

15. BigML, Inc #DutchMLSchool 15
The problem: We are trying to
fi
nd black swans
A methodology that combines supervised and
unsupervised learning to capture variance in data
Our approach
• Our objective is not to replace the reviewer,
but to make her work easier
• Only the reviewer is able to confidently
identify real black swans
• We will capture the variance in the data
trying to maximize the probability of
finding a black swan
+3 MORE +10 MORE +6 MORE
+8 MORE
+5 MORE
Reviewer’s feedback
+2 MORE
Feature extraction
Clustering
Anomaly Detection
Supervised Models
Anomalous group. It should be
reviewed first
Smart Video Review Framework

16. BigML, Inc #DutchMLSchool 16
Flare/Smoke Bombs Analysis
Binary Classification: Smoke/Flare vs not Detection: Flare Detection
Smart Video Review Framework

17. BigML, Inc #DutchMLSchool 17
Flags/Banners Analysis
Detection: Waving-flag and Banner Clustering and Pattern Matching
Smart Video Review Framework

18. BigML, Inc #DutchMLSchool 18
Chants Analysis
raw microphones data
1 3 seconds sound blocks
2
2D spectrograms
3
Filter out high frequencies
and obtain chromagram
4
Smart Video Review Framework

19. BigML, Inc #DutchMLSchool 19
Chants Analysis
raw microphones data
1 3 seconds sound blocks
2
2D spectrograms
3
Filter out high frequencies
and obtain chromagram
4
Chromagram Feature
Extraction
5
Chants Clustering and
Fingerprinting Matching
7
Chant / Noise
Classification
6
Chant 1
Chant 2
Chant 3
Smart Video Review Framework

20. BigML, Inc #DutchMLSchool 20
Monkey Chants Analysis
raw microphones data
1 3 seconds sound blocks
2
2D spectrograms
3
Filter out high frequencies
and obtain MEL Spectrogram
4
monkey chant audio blocks
non monkey chant audio blocks
Monkey Chant:
Periodic sound
around 400 Hz
BigML Image
Classification model
Smart Video Review Framework