For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2023/09/learning-for-360-vision-a-presentation-from-google/ Yu-Chuan Su, Research Scientist at Google, presents the “Learning for 360° Vision,” tutorial at the May 2023 Embedded Vision Summit. As a core building block of virtual reality (VR) and augmented reality (AR) technology, and with the rapid growth of VR and AR, 360° cameras are becoming more available and more popular. People now create, share and watch 360° content in their everyday lives, and the amount of 360° content is increasing rapidly. While 360° cameras offer tremendous new possibilities in various domains, they also introduce new technical challenges. For example, the distortion of 360° content in the planar projection degrades the performance of visual recognition algorithms. In this talk, Su explains how his company enables visual recognition on 360° imagery using existing models. Google’s solution improves 360° video processing and builds a foundation for further exploration of this new media.

1. Learning for 360°
Vision
Yu-Chuan Su
Research Scientist
Google

2. Expected market growth
[Precedence Research]
Applications
Growing Popularity of 360° Media
Virtual / Augmented Reality
Drone
Photography Surveillance
© 2023 Google 2

3. 360° Image vs. Perspective Image
© 2023 Google 3

4. 360° Image vs. Perspective Image
𝜃
𝜙
© 2023 Google 4

5. • Applications depend on visual recognition
[Zhang et al., ECCV 2014], [Hu et al., CVPR 2017], [Lai et al., TOG 2017], [Chou et al., AAAI
2018], [Yu et al., AAAI 2018], [Lee et al., CVPR 2018]
• CNNs for spherical data
[Boomsma and Frellsen, NeurIPS 2017], [Cohen et al., ICLR 2018], [Cheng et al., CVPR
2018], [Esteves et al., ECCV 2018], [Coors et al., ECCV 2018], [Zhang et al., ECCV 2018],
[Khasanova and Frossard, ICML 2019]
o New architectures designed specifically for spherical data
o Not data efficient – require annotations in spherical formats
Visual Recognition on 360° Imagery
© 2023 Google 5

6. Strategy I (Equirectangular)
[Hu CVPR 2017], [Lai TOG 2017]
• Fast – single projection
• Inaccurate – distortion in projection
Applying CNNs to 360° Imagery
© 2023 Google 6

7. Strategy II (Perspective)
[Zhang ECCV 2014], [Chou AAAI 2018],
[Yu AAAI 2018]
• Accurate – exact convolution output
• Slow – repeated projection
Applying CNNs to 360° Imagery
© 2023 Google 7

8. • Fast – single equirectangular projection
• Accurate – simulate exact convolution output
• Data efficient – does not require additional annotations
Spherical Convolution
© 2023 Google 8

9. • Goal – account for distortion
• Problem – distortion is polar angle (θ) / row dependent
• Solution – untie kernel weights along the rows*
Spherical Convolution Model Architecture
θ = 90° θ = 54° θ = 18°
Source
* We use one set of weights every 5 rows in practice
© 2023 Google 9

10. • Goal – accelerate training
• Idea – require 𝑁𝑒 to reproduce all intermediate features
Layer-wise Training
© 2023 Google 10

11. Apply SphConv to Faster R-CNN
• Pano2Vid (Train)
o 360° video dataset
o Sample 1,056 frames for training
• Pascal VOC 2007 (Test)
o Perspective object detection dataset
o Project bounding boxes to spheres at five polar angle (𝜽)
Experiment: Datasets
© 2023 Google 11

12. SphConv Evaluation
12
50% more accurate
than Strategy I
Orders of magnitude
faster than Strategy II
* [Hu et al., CVPR 2017], [Lai et al., TOG 2017]
# [Zhang et al., ECCV 2014], [Chou et al., AAAI 2018], [Yu et al., AAAI 2018]
© 2023 Google
RMSE
Accuracy

13. • Model size increases linearly w.r.t. input resolution (H)
• SphConv kernels are highly correlated → weights are redundant
Problem of Spherical Convolution
CNN SphConv
Asymptotic c2k2 c2k2H
VGG* 56 MB 29 GB
* Over 640x320 input images
© 2023 Google 13

14. Idea – generates SphConv kernels from source kernels
Kernel Transformer Network
© 2023 Google 14

15. Kernel Transformer Network Architecture
15
© 2023 Google
CNN SphConv KTN
Asymptotic c2k2 c2k2H c2k2+c2
VGG 56 MB 29 GB 70 MB

16. Transferability of Kernel Transformer Network
We need only one KTN for different tasks!
© 2023 Google 16

17. • Appy KTN to Faster R-CNN
[Ren et al., NIPS 2015]
o Extract equirectangular feature map
o Project features to tangent planes
o Spherical non-maximum suppression
• Does not need new annotation
Application: 360° Object Detection
© 2023 Google 17

18. Experiment: Object Detection Accuracy
1[Cheng et al., CVPR 2018],
2[Cohen et al., ICLR 2018],
3[Esteves et al., ECCV 2018],
4[Coors et al., ECCV 2018]
• KTN is much more compact than SphConv
• KTN outperforms recent CNNs on spherical data
© 2023 Google 18

19. © 2023 Google 19

20. Experiment: KTN Transferability
© 2023 Google 19

21. Experiment: KTN Transferability
KTN performs almost identically
regardless of the training source!
© 2023 Google 21

22. Conclusion
1[Cheng et al., CVPR 2018],
2[Cohen et al., ICLR 2018],
3[Esteves et al., ECCV 2018],
4[Coors et al., ECCV 2018]
• Accounts for distortion in 360° images
• Does not require annotated 360° images
• Is transferable across different source CNNs
Open door to off-the-shelf CNN recognition on 360° images
© 2023 Google 22

23. Resources
• Su and Grauman, Learning Spherical
Convolution for Fast Features from 360°
Imagery, NeurIPS 2017
• Su and Grauman, Kernel Transformer
Networks for Compact Spherical
Convolution, CVPR 2018
• Su and Grauman, Learning Spherical
Convolution for 360 Recognition, PAMI
2021
• Zhang et al., PanoContext: A Whole-
Room 3D Context Model for Panoramic
Scene Understanding, ECCV 2014
• Hu et al., Deep 360 Pilot: Learning a
Deep Agent for Piloting through 360°
Sports Video, CVPR 2017
• Lai et al., Semantic-driven Generation of
Hyperlapse from 360° Video, TOG 2017
• Boomsma and Frellsen, Spherical
convolutions and their application in
molecular modelling, NeurIPS 2017
23
© 2023 Google

24. Resources
• Cheng et al., Cube Padding for Weakly-
Supervised Saliency Prediction in 360°
Videos, CVPR 2018
• Lee et al., A Memory Network Approach
for Story-Based Temporal Summarization
of 360° Videos, CVPR 2018
• Chou et al., Self-view Grounding Given a
Narrated 360° Video, AAAI 2018
• Yu et al., A Deep Ranking Model for
Spatio-Temporal Highlight Detection
from a 360 Video, AAAI 2018
• Cohen et al., Spherical CNNs, ICLR 2018
• Esteves et al., Learning SO(3)
Equivariant Representations with
Spherical CNNs, ECCV 2018
• Coors et al., SphereNet: Learning
Spherical Representations for Detection
and Classification in Omnidirectional
Images, ECCV 2018
• Zhang et al., Saliency Detection in 360°
Videos, ECCV 2018
• Khasanova and Frossard, Geometry
Aware Convolutional Filters for
Omnidirectional Images Representation,
ICML 2019
24
© 2023 Google