The document presents a hierarchical relational network approach for group activity recognition and retrieval. The key aspects are: (1) a relational layer that captures person representations based on a relationships graph, (2) stacking multiple relational layers to process different relationship graphs hierarchically, and (3) applications to supervised group activity recognition and unsupervised scene/person retrieval.

1. Hierarchical Relational Networks
for Group Activity Recognition and Retrieval
ECCV 2018
Mostafa S. Ibrahim, Greg Mori
Simon Fraser University
Prepared by
Mostafa Saad Ibrahim
PhD Student @ Simon Fraser University

2. Human Activities
Actions
Interactions
Gestures
Group Activity

3. Group Activity Recognition
Standing
Standing
Blocking
Blocking
Moving
Moving
Spiking
Jumping
Waiting
Moving
Moving
Moving
Left Spike Activity

4. Ibrahim et al. 2016, CVPR
Learn People
Representations
Aggregate People
Representations
Learn Group
Representations

5. Relationships?
Left team:
- Fake jump by a player
- Spike by another player
Right team:
- 3 players jump to block

6. Our work
◼ Relational layer to capture person relational
representation
◼ Layer input: K vectors / persons relationships
◼ Output: K relational vector representations
◼ Applications
◼ Supervised: Group Activity Recognition
◼ Unsupervised: Scene and Person Retrieval

7. Relationships Representation
….
Relationships given
as a graph
CNN CNN...
Features extracted
from fine tuned CNN

8. Inferring person's relational represt.

9. Relational Layer
Input
- K feature vectors of people
- Relationships Graph
Output
- K relational feature vectors
Processing
- Each layer learns a shared MLP
- It learns to infer edge representation
- Re-represent each person based on neighbours
p1
p2
p4
….
P`1
= F(P1
⊕P2
) + F(P1
⊕P4
)

10. Hierarchical Relational Network
What if we have different graphs
representing potential relationships?
Assume closed shape is a clique of
relationships (pairwise sub-graph)
Graph of
4 cliques
Graph of
2 cliques (2 teams)
Graph of
1 cliques (full pairwise)

11. Hierarchical Relational Network

12. Experiments

13. ◼ 4830 annotated frames from 55 YouTube videos.
◼ 9 person level labels, and 8 group activity labels.
Volleyball Dataset

14. Group Recognition on Volleyball

15. Can we utilize this layer in
unsupervised domains?

16. Scene Retrieval based on actions
F1 matches F2 IIF:
IOU(actions) >= 0.5

17. Scene Retrieval: Naive Approach
AutoEncoder Layer 1
CNN
CNN’
Euclideanloss
AutoEncoder Layer 2
AutoEncoder Layer 1’
-
What if we could jointly compress each person while
considering his relationships?

18. Scene Retrieval: Relational AutoEncoder

19. Scene Retrieval - Autoencoder - Variants

20. Scene Retrieval - Autoencoder - Baselines

21. Scene Retrieval - Denoising Autoencoder

22. Person Retrieval

23. Scene Retrieval Visualization: Q1@K
Query 1
Hit@5: Retrieved 1
=============
IoU = (1+2+6+2)/(1+2+6+3+1)
= 0.85
- 1 spike
- 2 block
- 6 move
- 3 stand
- 1 spike
- 2 block
- 6 move
- 2 stand
- 1 waiting

24. Scene Retrieval Visualization: Q1@K
Query 1
Hit@5: Retrieved 2
=============
IoU = (1+2+6+2)/(1+2+6+3+1)
= 0.85
- 1 spike
- 2 block
- 6 move
- 3 stand
- 1 spike
- 2 block
- 6 move
- 2 stand
- 1 waiting

25. Scene Retrieval Visualization: Q1@K
Query 1
Hit@5: Retrieved 3
=============
IoU = 6/(1+2+12+3)
= 0.33
- 1 spike
- 2 block
- 6 move
- 3 stand
- 12 move

26. Scene Retrieval Visualization: Q1@K
Query 1
Hit@5: Retrieved 4
=============
IoU = 3/(1+2+6+3+2+6)
= 0.15
- 1 spike
- 2 block
- 6 move
- 3 stand
- 1 sit
- 3 move
- 2 dig
- 6 waiting

27. Scene Retrieval Visualization: Q1@K
Query 1
Hit@5: Retrieved 5
=============
IoU = (1+2+4+3)/(1+3+6+3+1)
= 0.71
- 1 spike
- 2 block
- 6 move
- 3 stand
- 1 spike
- 3 block
- 4 move
- 3 stand
- 1 dig

28. Summary
◼ Relational layer
◼ Flexible: Each layer processes a graph, not just
all-pairwise style
◼ Hierarchical: Stack layers of whatever graphs
◼ Generic: Given K vectors / graph
◼ Plug and Play layer (image objects, set of images/docs)
◼ Applications
◼ Supervised: Group Activity Recognition
◼ Unsupervised: Scene and Person Retrieval
◼ Autoencoder / Denoising Autoencoder

30. Inferring Edge Representation
CNN
M features of Pi
...
CNN
M features of Pj
...
2-layers MLP F(Pi
⊕ Pj
)
...FC2: S features
...FC1: S features