1. Using video to make your assistant see
Roland Memisevic
AI Frontiers 2018

3. 3
TwentyBN’s real-time vision system

5. Crowd-Acting
1
We create labels describing
complex scenes and actions
We distribute the labels
using our patented platform
to workers all over the world
Workers record, submit and
verify videos, resulting in
high quality, densely labelled
video data for training nets

6. Data variability
Camera angles and scene layouts
Multi-person actions and
localization
Interactivity
Complex object interactions

7. 15 people, 3 street signs

8. Contrastive examples

9. Network
training
Data
platform
The data loop
Real-time
engineering
User
Interface

10. Dog
Cat

11. Something-something (V2): recognize complex activities end-to-end

12. (Mahdisoltani et al 2018)

13. 13
GradCam visualizations

14. 14
20bn-kitchenware

15. 20bn-kitchenware
from scratch
ImageNet baselines
Smth-Smth: classification on 40
action groups
Holding something
Smth-Smth: classification on 178
class actions
Holding [something] next to [something]
Captioning on full captions
Holding a metallic cup next to a big blue
box
Captioning on “single-object” captions
Holding cup next to box

16. https://github.com/TwentyBN/smth-smth-v2-baseline-with-models

17. 20bn.com

18. Twenty Billion Neurons fact sheet:
● Founded in 2015 by three Machine Learning researchers
● The company mission is to make cameras see just like
the human eye
● The company builds computer vision systems that run
at the edge and that are driven by a single RGB camera
● The company counts 20 full-time staff across Berlin and
Toronto

19. 19
Contrastive groups
After training on all 174 classes,
evaluate classifier within
contrastive group:
argmax p(class|video, group)

20. 20
Contrastive groups
After training on all 174 classes,
evaluate classifier within
contrastive group:
argmax p(class|video, group)

21. Common sense score