This document discusses using machine learning for automatic summarization of sport videos. It presents three potential models for classifying success or failure states in video frames: (1) classifying cropped patches of people near the goal post, (2) extracting pose estimation keypoints and classifying those, and (3) combining models 1 and 2 by classifying patches and keypoints together. It then describes in more detail how model 1 works by sampling video frames, detecting people, cropping patches, and classifying the patches. Model 2 and 3 variations are also proposed. The document concludes that GANs have potential for many use cases if developed responsibly.

1. Machine Learning for automatic
summarisation of sport videos
Luca Marchesotti, Sparkd.ai
@lucamarchesotti

2. Introduction
@lucamarchesotti

3. @lucamarchesotti

4. ATHLETIC PERFORMANCE
GAMING
FANS, MEDIA & CONTENT
WEARABLE TECH
@lucamarchesotti
END-2-END

6. For Fans..

7. For Clubs..

11. ..the fastest
Game on grass

12. ball speed
100mph

13. ball speed
100mph

14. 82 % players suffer an injury.
• John C Murphy1,2, Conor Gissane3, Catherine Blake2
Injury in elite county-level hurling: a
prospective study

22. Saman Sarraf and Mehdi Noori, “Multimodal deep learning approach for event detection in sports, using Amazon SageMaker”, AWS blog 2021

24. Vats, Kanav, et al. "Event detection in coarsely annotated sports videos via parallel multi-receptive field 1d convolutions." Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition Workshops. 2020.

25. Cioppa, Anthony, et al. "A context-aware loss function for action spotting in soccer videos." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

26. Cioppa, Anthony, et al. "A context-aware loss function for action spotting in soccer videos." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

27. Cioppa, Anthony, et al. "A context-aware loss function for action spotting in soccer videos." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

28. Cioppa, Anthony, et al. "A context-aware loss function for action spotting in soccer videos." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

29. Challenges
Only 1 camera, no visual cues on filming style
No consistent visual patterns in players behaviour
Ball simply invisible..
1
2
3

31. Score detection

34. Tracking
Need
states
frame
success failure
goal point wide none
Classes taxonomy
Let’s start simple

35. Tracking
Need
states
frame
success failure
goal point wide none
Classes taxonomy
We are going to
classify at this level

36. Tracking
Need
states
Examples - Success class

37. Tracking
Need
states
Examples - Failure class

38. Tracking
Need
states
Classification ideas
● Model 1:
○ classify each people image
● Model 2:
○ extract pose estimation key points and use them as features
● Model 3:
○ merge information from model 1 and 2
3 possible models

39. Tracking
Need
states
Classification ideas
● Model 1:
○ classify each people image
● Model 2:
○ extract pose estimation key points and use them as features
● Model 3:
○ merge information from model 1 and 2
3 possible models

40. Tracking
Need
states
Classification ideas
● Model 1:
○ classify each people image
● Model 2:
○ extract pose estimation key points and use them as features
● Model 3:
○ merge information from model 1 and 2
3 possible models

41. Tracking
Need
states
Model 1 - How it works
Video of the match
...
...
Previous frame Next frame
Frame sampling

42. Tracking
Need
states
Model 1 - How it works
...
...
Previous frame Next frame
H-detection model
...
...
Previous frame Next frame

43. Tracking
Need
states
Model 1 - How it works
People detection

44. Tracking
Need
states
Model 1 - How it works
People detection

45. Tracking
Need
states
Crop and extract patches near
to the H-post

46. Tracking
Need
states
Model 1 - How it works
Feed each patch into Pour Success/
Failure CNN model and get the
corresponding prediction
26% 98% 18%

47. Tracking
Need
states
Model 2 - Variations
Extract key points for each patch

48. Tracking
Need
states
Model 2 - Variations
Feed a Success/Failure classifier with key
points and get the corresponding
prediction
26% 98% 18%

49. Tracking
Need
states
Model 3 - Combine Model 1 and Model 2
Feed a Success/Failure classifier with
both the patch and the predicted key
points and get the corresponding
prediction
26% 98% 18%

50. Tracking
Need
states

51. Tracking
Need
states

52. @lucamarchesotti

53. @lucamarchesotti

54. @lucamarchesotti

55. @lucamarchesotti

56. Conclusions
@lucamarchesotti

57. 1. GANs are not a “bad” technology
2. Incredible potential in many different use cases
3. The fight is not over!!
@lucamarchesotti