Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Curriculum Learning for Recurrent Video Object Segmentation

295 views

Published on

https://imatge-upc.github.io/rvos-mots/

Video object segmentation can be understood as a sequence-to-sequence task that can benefit from the curriculum learning strategies for better and faster training of deep neural networks. This work explores different schedule sampling and frame skipping variations to significantly improve the performance of a recurrent architecture. Our results on the car class of the KITTI-MOTS challenge indicate that, surprisingly, an inverse schedule sampling is a better option than a classic forward one. Also, that a progressive skipping of frames during training is beneficial, but only when training with the ground truth masks instead of the predicted ones.

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Curriculum Learning for Recurrent Video Object Segmentation

  1. 1. CURRICULUM LEARNING FOR RECURRENT VIDEO OBJECT SEGMENTATION Co-directors: Xavier Giró Nieto and Carles Ventura Royo Author: Maria Gonzàlez Calabuig
  2. 2. Introduction Dataset The model Experiment sets Techniques Qualitative results YouTube-VOS Conclusions CONTENTS
  3. 3. INTRODUCTION
  4. 4. INTRODUCTION Curriculum Learning for Recurrent VOS - 4 of 144 Curriculum Learning: Methodology inspired by the learning process of humans. The training data is presented in a meaningful way, from simple to complex concepts. Yoshua Bengio et al. “Curriculum Learning”, ICML. 2019.
  5. 5. INTRODUCTION Curriculum Learning for Recurrent VOS - 5 of 144 Curriculum Learning: Methodology inspired by the learning process of humans. The training data is presented in a meaningful way, from simple to complex concepts. 4 curriculums Yoshua Bengio et al. “Curriculum Learning”, ICML. 2019.
  6. 6. INTRODUCTION Curriculum Learning for Recurrent VOS - 6 of 144 Curriculum Learning: Methodology inspired by the learning process of humans. The training data is presented in a meaningful way, from simple to complex concepts. 4 curriculums THE DATASET Yoshua Bengio et al. “Curriculum Learning”, ICML. 2019.
  7. 7. INTRODUCTION Curriculum Learning for Recurrent VOS - 7 of 144 Curriculum Learning: Methodology inspired by the learning process of humans. The training data is presented in a meaningful way, from simple to complex concepts. 4 curriculums THE DATASET THE MODEL Yoshua Bengio et al. “Curriculum Learning”, ICML. 2019.
  8. 8. INTRODUCTION Curriculum Learning for Recurrent VOS - 8 of 144 THE TASK Semi-supervised or “one-shot” Video Object Segmentation
  9. 9. INTRODUCTION Curriculum Learning for Recurrent VOS - 9 of 144 THE TASK Semi-supervised or “one-shot” Video Object Segmentation
  10. 10. INTRODUCTION Curriculum Learning for Recurrent VOS - 10 of 144 THE TASK Semi-supervised or “one-shot” Video Object Segmentation
  11. 11. INTRODUCTION Curriculum Learning for Recurrent VOS - 11 of 144 THE TASK Estimated by the modelGiven to the model Semi-supervised or “one-shot” Video Object Segmentation
  12. 12. DATASET
  13. 13. KITTI-MOTS DATASET Curriculum Learning for Recurrent VOS - 13 of 144 Andreas Geiger, Philip Lenz, and Raquel Urtasun. “Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite”, CVPR 2012.
  14. 14. KITTI-MOTS DATASET Curriculum Learning for Recurrent VOS - 14 of 144 Andreas Geiger, Philip Lenz, and Raquel Urtasun. “Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite”, CVPR 2012. Its video sequences present challenges:
  15. 15. KITTI-MOTS DATASET Curriculum Learning for Recurrent VOS - 15 of 144 Its video sequences present challenges: Andreas Geiger, Philip Lenz, and Raquel Urtasun. “Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite”, CVPR 2012.
  16. 16. KITTI-MOTS DATASET Curriculum Learning for Recurrent VOS - 16 of 144 Its video sequences present challenges: Andreas Geiger, Philip Lenz, and Raquel Urtasun. “Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite”, CVPR 2012.
  17. 17. KITTI-MOTS DATASET Curriculum Learning for Recurrent VOS - 17 of 144 Its video sequences present challenges: Andreas Geiger, Philip Lenz, and Raquel Urtasun. “Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite”, CVPR 2012.
  18. 18. THE MODEL
  19. 19. THE MODEL End-to-End Recurrent Network for video object segmentation: RVOS Curriculum Learning for Recurrent VOS - 19 of 144 Carles Ventura, Miriam Bellver, Andreu Girbau, Amaia Salvador, Ferran Marques and Xavier Giro-i-Nieto. “RVOS: End-to-End Recurrent Network for Video Object Segmentation”, CVPR 2019.
  20. 20. THE MODEL End-to-End Recurrent Network for video object segmentation: RVOS Curriculum Learning for Recurrent VOS - 20 of 144 Athar, A., Mahadevan, S., Oˇsep, A., Leal-Taix´e, L., Leibe, B.: Stem-seg: Spatio-temporal embeddings for instance segmentation in videos., ECCV (2020)
  21. 21. EXPERIMENT SETS
  22. 22. SETS OF EXPERIMENTS All techniques tested on two sets of experiments: Resolution Batch Size Length clip 287x950 2 3 Resolution Batch Size Length clip 256x448 4 5 Curriculum Learning for Recurrent VOS - 22 of 144
  23. 23. METRICS The results have been evaluated on the official metrics of the MOTS Challenge. - sMOTSA has been defined as the reference metric: Curriculum Learning for Recurrent VOS - 23 of 144 Paul Voigtlaender et al. “MOTS: Multi-Object Tracking and Segmentation”, CVPR 2019.
  24. 24. METRICS The results have been evaluated on the official metrics of the MOTS Challenge. - sMOTSA has been defined as the reference metric: Curriculum Learning for Recurrent VOS - 25 of 144 Paul Voigtlaender et al. “MOTS: Multi-Object Tracking and Segmentation”, CVPR 2019.
  25. 25. EVALUATION METHOD Proposal: Evaluation averaged per sequence Curriculum Learning for Recurrent VOS - 25 of 144 Sequences
  26. 26. SCHEDULE SAMPLING
  27. 27. SCHEDULE SAMPLING Curriculum Learning for Recurrent VOS - 27 of 144 VOS requires information about the previous step.
  28. 28. SCHEDULE SAMPLING Curriculum Learning for Recurrent VOS - 28 of 144 Train using the model’s outputs.
  29. 29. SCHEDULE SAMPLING Curriculum Learning for Recurrent VOS - 29 of 144 Train using the ground-truth annotations.
  30. 30. SCHEDULE SAMPLING Curriculum Learning for Recurrent VOS - 30 of 144 TEACHER FORCING
  31. 31. SCHEDULE SAMPLING Curriculum Learning for Recurrent VOS - 31 of 144 TEACHER FORCING Fast and efficient
  32. 32. SCHEDULE SAMPLING Curriculum Learning for Recurrent VOS - 32 of 144 TEACHER FORCING Fast and efficient Leads to exposure bias
  33. 33. SCHEDULE SAMPLING Curriculum Learning for Recurrent VOS - 33 of 144 time
  34. 34. SCHEDULE SAMPLING Curriculum Learning for Recurrent VOS - 34 of 144 time
  35. 35. Schedule Sampling SCHEDULE SAMPLING Curriculum Learning for Recurrent VOS - 35 of 144
  36. 36. Schedule Sampling Linear SCHEDULE SAMPLING Curriculum Learning for Recurrent VOS - 36 of 144
  37. 37. Schedule Sampling Linear Step SCHEDULE SAMPLING Curriculum Learning for Recurrent VOS - 37 of 144
  38. 38. Schedule Sampling Linear Forward Step Forward SCHEDULE SAMPLING Curriculum Learning for Recurrent VOS - 38 of 144
  39. 39. Schedule Sampling Linear Forward Step Forward SCHEDULE SAMPLING Curriculum Learning for Recurrent VOS - 39 of 144
  40. 40. Schedule Sampling Linear Forward Step Forward SCHEDULE SAMPLING Curriculum Learning for Recurrent VOS - 40 of 144
  41. 41. Schedule Sampling Linear Forward Inverse Step Forward Inverse SCHEDULE SAMPLING Curriculum Learning for Recurrent VOS - 41 of 144
  42. 42. Schedule Sampling Linear Forward Inverse Step Forward Inverse SCHEDULE SAMPLING Curriculum Learning for Recurrent VOS - 42 of 144
  43. 43. Schedule Sampling Linear Forward Inverse Step Forward Inverse SCHEDULE SAMPLING Curriculum Learning for Recurrent VOS - 43 of 144
  44. 44. Schedule Sampling Linear Forward Inverse Step Forward Inverse SCHEDULE SAMPLING Curriculum Learning for Recurrent VOS - 44 of 144
  45. 45. SCHEDULE SAMPLING Curriculum Learning for Recurrent VOS - 45 of 144 RESULTS ON THE FORWARD STRATEGIES
  46. 46. SCHEDULE SAMPLING Curriculum Learning for Recurrent VOS - 46 of 144 RESULTS ON THE INVERSE STRATEGIES
  47. 47. SCHEDULE SAMPLING Curriculum Learning for Recurrent VOS - 47 of 144 OVERVIEW
  48. 48. SCHEDULE SAMPLING Curriculum Learning for Recurrent VOS - 48 of 144 OVERVIEW
  49. 49. SCHEDULE SAMPLING Curriculum Learning for Recurrent VOS - 49 of 144 OVERVIEW
  50. 50. SCHEDULE SAMPLING Curriculum Learning for Recurrent VOS - 50 of 144 OVERVIEW
  51. 51. FRAME SKIPPING
  52. 52. FRAME SKIPPING Curriculum Learning for Recurrent VOS - 52 of 144 KITTI-MOTS has slow-motion video sequences. frame #1
  53. 53. FRAME SKIPPING Curriculum Learning for Recurrent VOS - 53 of 144 KITTI-MOTS has slow-motion video sequences. frame #1 frame #2
  54. 54. FRAME SKIPPING Curriculum Learning for Recurrent VOS - 54 of 144 KITTI-MOTS has slow-motion video sequences. frame #1 frame #2 frame #3
  55. 55. FRAME SKIPPING Curriculum Learning for Recurrent VOS - 55 of 144 KITTI-MOTS has slow-motion video sequences. frame #1 frame #2 frame #3 frame #4
  56. 56. FRAME SKIPPING Curriculum Learning for Recurrent VOS - 56 of 144 KITTI-MOTS has slow-motion video sequences. frame #1 frame #2 frame #3 frame #4 frame #5
  57. 57. FRAME SKIPPING Curriculum Learning for Recurrent VOS - 57 of 144 KITTI-MOTS has slow-motion video sequences. frame #1 frame #2 frame #3 frame #4 frame #5 frame #6
  58. 58. FRAME SKIPPING Curriculum Learning for Recurrent VOS - 58 of 144 Ideally: … ... . … … .. N fram es of the sequence
  59. 59. FRAME SKIPPING Curriculum Learning for Recurrent VOS - 59 of 144 Ideally: But we have limitations (e.g. memory constraints) … … .. N fram es of the sequence … ... .
  60. 60. FRAME SKIPPING Curriculum Learning for Recurrent VOS - 60 of 144
  61. 61. FRAME SKIPPING Curriculum Learning for Recurrent VOS - 61 of 144
  62. 62. FRAME SKIPPING Frame Skipping Curriculum Learning for Recurrent VOS - 62 of 144
  63. 63. FRAME SKIPPING Frame Skipping From 0 to 9 Curriculum Learning for Recurrent VOS - 63 of 144
  64. 64. FRAME SKIPPING Frame Skipping From 0 to 9 Curriculum Learning for Recurrent VOS - 64 of 144 time
  65. 65. FRAME SKIPPING Frame Skipping From 0 to 9 Curriculum Learning for Recurrent VOS - 65 of 144 time
  66. 66. FRAME SKIPPING Frame Skipping From 0 to 9 Curriculum Learning for Recurrent VOS - 66 of 144 time
  67. 67. FRAME SKIPPING Frame Skipping From 0 to 9 Curriculum Learning for Recurrent VOS - 67 of 144 time ...
  68. 68. FRAME SKIPPING Frame Skipping From 0 to 9 From 1 to 5 Curriculum Learning for Recurrent VOS - 68 of 144 time
  69. 69. FRAME SKIPPING Frame Skipping From 0 to 9 From 1 to 5 Curriculum Learning for Recurrent VOS - 69 of 144 time
  70. 70. FRAME SKIPPING Frame Skipping From 0 to 9 From 1 to 5 Curriculum Learning for Recurrent VOS - 70 of 144 time
  71. 71. FRAME SKIPPING Frame Skipping From 0 to 9 From 1 to 5 Curriculum Learning for Recurrent VOS - 71 of 144 time ...
  72. 72. FRAME SKIPPING Frame Skipping From 0 to 9 From 1 to 5 All training All training Curriculum Learning for Recurrent VOS - 72 of 144
  73. 73. FRAME SKIPPING Frame Skipping From 0 to 9 From 1 to 5 All training All training Curriculum Learning for Recurrent VOS - 73 of 144
  74. 74. FRAME SKIPPING Frame Skipping From 0 to 9 From 1 to 5 All training All training Curriculum Learning for Recurrent VOS - 74 of 144
  75. 75. FRAME SKIPPING Frame Skipping From 0 to 9 From 1 to 5 All training First half training All training First half training Curriculum Learning for Recurrent VOS - 75 of 144
  76. 76. FRAME SKIPPING Frame Skipping From 0 to 9 From 1 to 5 All training First half training All training First half training Curriculum Learning for Recurrent VOS - 76 of 144
  77. 77. FRAME SKIPPING Frame Skipping From 0 to 9 From 1 to 5 All training First half training All training First half training Curriculum Learning for Recurrent VOS - 77 of 144
  78. 78. FRAME SKIPPING Curriculum Learning for Recurrent VOS - 78 of 144 RESULTS ON THE FRAME SKIPPING APPLIED DURING ALL TRAINING
  79. 79. FRAME SKIPPING Curriculum Learning for Recurrent VOS - 79 of 144 RESULTS ON THE FRAME SKIPPING APPLIED ONLY WITH GROUND-TRUTH
  80. 80. FRAME SKIPPING Curriculum Learning for Recurrent VOS - 80 of 144 OVERVIEW
  81. 81. FRAME SKIPPING Curriculum Learning for Recurrent VOS - 81 of 144 OVERVIEW
  82. 82. TEMPORAL AND SPATIAL RECURRENCES
  83. 83. TEMPORAL AND SPATIAL RECURRENCES Curriculum Learning for Recurrent VOS - 83 of 144 KITTI-MOTS is a crowded dataset:
  84. 84. TEMPORAL AND SPATIAL RECURRENCES Curriculum Learning for Recurrent VOS - 84 of 144 time (frame sequence) space(objectsequence)
  85. 85. TEMPORAL AND SPATIAL RECURRENCES Curriculum Learning for Recurrent VOS - 85 of 144 time (frame sequence) TEMPORAL RECURRENCE
  86. 86. TEMPORAL AND SPATIAL RECURRENCES Curriculum Learning for Recurrent VOS - 86 of 144 space(objectsequence) SPATIAL RECURRENCE
  87. 87. TEMPORAL AND SPATIAL RECURRENCES Proposed curriculum: Curriculum Learning for Recurrent VOS - 87 of 144
  88. 88. TEMPORAL AND SPATIAL RECURRENCES Temporal and Spatial Recurrence Only temporal during the first half of training Curriculum Learning for Recurrent VOS - 88 of 144
  89. 89. TEMPORAL AND SPATIAL RECURRENCES Temporal and Spatial Recurrence Spatio-temporal during all training Only temporal during the first half of training Curriculum Learning for Recurrent VOS - 89 of 144
  90. 90. TEMPORAL AND SPATIAL RECURRENCES Temporal and Spatial Recurrence Spatio-temporal during all training Only temporal during all training Only temporal during the first half of training Curriculum Learning for Recurrent VOS - 90 of 144
  91. 91. TEMPORAL AND SPATIAL RECURRENCES Temporal and Spatial Recurrence Spatio-temporal during all training Only temporal during all training Only temporal during the first half of training Only temporal during the second half of training Curriculum Learning for Recurrent VOS - 91 of 144
  92. 92. TEMPORAL AND SPATIAL RECURRENCES Curriculum Learning for Recurrent VOS - 92 of 144
  93. 93. TEMPORAL AND SPATIAL RECURRENCES Curriculum Learning for Recurrent VOS - 93 of 144
  94. 94. TEMPORAL AND SPATIAL RECURRENCES Curriculum Learning for Recurrent VOS - 94 of 144 Ground-truth
  95. 95. TEMPORAL AND SPATIAL RECURRENCES Curriculum Learning for Recurrent VOS - 95 of 144 Only Spatio-Temporal Ground-truth
  96. 96. TEMPORAL AND SPATIAL RECURRENCES Curriculum Learning for Recurrent VOS - 96 of 144 Only Spatio-Temporal Only Temporal Ground-truth
  97. 97. TEMPORAL AND SPATIAL RECURRENCES Curriculum Learning for Recurrent VOS - 97 of 144 Only Spatio-Temporal Only Temporal Only Temporal first half Ground-truth
  98. 98. TEMPORAL AND SPATIAL RECURRENCES Curriculum Learning for Recurrent VOS - 98 of 144 Only Spatio-Temporal Only Temporal Only Temporal first half Only Temporal second half Ground-truth
  99. 99. TEMPORAL AND SPATIAL RECURRENCES Curriculum Learning for Recurrent VOS - 99 of 144
  100. 100. LOSS PENALIZATION BY OBJECT AREA
  101. 101. LOSS PENALIZATION BY OBJECT AREA Curriculum Learning for Recurrent VOS - 101 of 144 KITTI-MOTS contains instances with different resolution:
  102. 102. LOSS PENALIZATION BY OBJECT AREA Curriculum Learning for Recurrent VOS - 102 of 144 KITTI-MOTS contains instances with different resolution:
  103. 103. LOSS PENALIZATION BY OBJECT AREA Curriculum Learning for Recurrent VOS - 103 of 144 KITTI-MOTS contains instances with different resolution:
  104. 104. LOSS PENALIZATION BY OBJECT AREA Curriculum Learning for Recurrent VOS - 104 of 144 An hypothesis is made: DIFFICULT
  105. 105. LOSS PENALIZATION BY OBJECT AREA Curriculum Learning for Recurrent VOS - 105 of 144 An hypothesis is made: DIFFICULT EASY
  106. 106. LOSS PENALIZATION BY OBJECT AREA Curriculum Learning for Recurrent VOS - 106 of 144 A curriculum is created: time
  107. 107. LOSS PENALIZATION BY OBJECT AREA Curriculum Learning for Recurrent VOS - 107 of 144 A curriculum is created: time
  108. 108. LOSS PENALIZATION BY OBJECT AREA Curriculum Learning for Recurrent VOS - 108 of 144
  109. 109. LOSS PENALIZATION BY OBJECT AREA Curriculum Learning for Recurrent VOS - 109 of 144 Resolution Batch Size Length clip 287x950 2 3 Resolution Batch Size Length clip 256x448 4 5
  110. 110. LOSS PENALIZATION BY OBJECT AREA Curriculum Learning for Recurrent VOS - 110 of 144 Resolution Batch Size Length clip 287x950 2 3 Resolution Batch Size Length clip 256x448 4 5
  111. 111. LAST MINUTE RESULTS
  112. 112. LAST MINUTE RESULTS Curriculum Learning for Recurrent VOS - 112 of 144
  113. 113. QUALITATIVE RESULTS
  114. 114. YouTube-VOS
  115. 115. YouTube-VOS Curriculum Learning for Recurrent VOS - 119 of 144 Ning Xu et al. “YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark”, ECCV 2018
  116. 116. YouTube-VOS Curriculum Learning for Recurrent VOS - 120 of 144 - Training parameters: Resolution Batch Size Length clip 256x448 4 5
  117. 117. YouTube-VOS Curriculum Learning for Recurrent VOS - 121 of 144 - Training parameters: - Evaluated with the official metrics of the YouTube-VOS challenge. Resolution Batch Size Length clip 256x448 4 5
  118. 118. YouTube-VOS Curriculum Learning for Recurrent VOS - 122 of 144 - Training parameters: - Evaluated with the official metrics of the YouTube-VOS challenge. Resolution Batch Size Length clip 256x448 4 5
  119. 119. YouTube-VOS Curriculum Learning for Recurrent VOS - 123 of 144 - Training parameters: - Evaluated with the official metrics of the YouTube-VOS challenge. Resolution Batch Size Length clip 256x448 4 5
  120. 120. YouTube-VOS Curriculum Learning for Recurrent VOS - 124 of 144 Forward Linear Inverse Linear Forward Step Inverse Linear
  121. 121. YouTube-VOS Curriculum Learning for Recurrent VOS - 125 of 144 Results on KITTI-MOTS Results on YouTube-VOS
  122. 122. YouTube-VOS Curriculum Learning for Recurrent VOS - 126 of 144 adapted Frame skipping from 0 to 3From 0 to 9
  123. 123. YouTube-VOS Curriculum Learning for Recurrent VOS - 127 of 144 Results on YouTube-VOSResults on KITTI-MOTS
  124. 124. CONCLUSIONS
  125. 125. CONCLUSIONS Curriculum Learning for Recurrent VOS - 129 of 144 SCHEDULE SAMPLING FRAME SKIPPING LOSS PENALIZATION BY OBJECT AREATEMPORAL AND SPATIAL RECURRENCES
  126. 126. CONCLUSIONS Curriculum Learning for Recurrent VOS - 130 of 144 SCHEDULE SAMPLING FRAME SKIPPING LOSS PENALIZATION BY OBJECT AREATEMPORAL AND SPATIAL RECURRENCES
  127. 127. CONCLUSIONS Curriculum Learning for Recurrent VOS - 131 of 144 SCHEDULE SAMPLING FRAME SKIPPING LOSS PENALIZATION BY OBJECT AREATEMPORAL AND SPATIAL RECURRENCES
  128. 128. CONCLUSIONS Curriculum Learning for Recurrent VOS - 132 of 144 SCHEDULE SAMPLING FRAME SKIPPING LOSS PENALIZATION BY OBJECT AREATEMPORAL AND SPATIAL RECURRENCES
  129. 129. CONCLUSIONS Curriculum Learning for Recurrent VOS - 133 of 144 SCHEDULE SAMPLING FRAME SKIPPING LOSS PENALIZATION BY OBJECT AREATEMPORAL AND SPATIAL RECURRENCES
  130. 130. CONCLUSIONS Curriculum Learning for Recurrent VOS - 134 of 144 Importance of knowing the dataset. KITTI-MOTS YouTube-VOS
  131. 131. CONCLUSIONS Curriculum Learning for Recurrent VOS - 135 of 144 Importance of knowing the dataset. KITTI-MOTS YouTube-VOS
  132. 132. CONCLUSIONS Curriculum Learning for Recurrent VOS - 136 of 144 Importance of knowing the dataset. KITTI-MOTS YouTube-VOS
  133. 133. FUTURE WORK Curriculum Learning for Recurrent VOS - 137 of 144
  134. 134. FUTURE WORK Curriculum Learning for Recurrent VOS - 138 of 144 Schedule Sampling
  135. 135. FUTURE WORK Curriculum Learning for Recurrent VOS - 139 of 144 Schedule Sampling Frame Skipping
  136. 136. FUTURE WORK Curriculum Learning for Recurrent VOS - 140 of 144 Schedule Sampling Frame Skipping Loss penalization by object area
  137. 137. FUTURE WORK Curriculum Learning for Recurrent VOS - 141 of 144 Schedule Sampling Frame Skipping Loss penalization by object area Other curriculums
  138. 138. FUTURE WORK Curriculum Learning for Recurrent VOS - 142 of 144 Schedule Sampling Frame Skipping Loss penalization by object area Other curriculums Combination of the best curriculums
  139. 139. WORKSHOP SUBMISSIONS Curriculum Learning for Recurrent VOS - 143 of 144 Acceptance Notification: August 3, 2020 PAD2020
  140. 140. Curriculum Learning for Recurrent Video Object Segmentation Maria Gonzalez Calabuig Barcelona, 24th July 2020

×