Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

[Paper introduction] GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking

1,963 views

Published on

[Partly in Japanese]
I present it in 関東コンピュータビジョン勉強会(2015/07/25).

Main References:
http://crcv.ucf.edu/projects/GMMCP-Tracker/CVPR15_GMMCP_Presentation.pptx

If you find a problem, please let me know.
Thanks!

Published in: Technology
  • Be the first to comment

[Paper introduction] GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking

  1. 1. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking 2015 / 7 / 26 (Fri.) 関東コンピュータビジョン勉強会 発表者: @hokkun_cv GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking 1 Afshin Dehghan, Shayan Modiri Assari, Mubarak Shah University of Central Florida
  2. 2. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking About me • 東大院・学際情報学府・M2 • 相澤研究室所属 •  普段は食べものの研究をしています • 2014/5のCV勉強会(CNNについて)ぶりの発表 参加です 2 • Preferred Networksでインターン→アルバイト中 •  メンターが@tabe2314さん • 今日はその課題の中で出てきたタスクに関連する 論文を紹介します
  3. 3. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking 対象とする問題 • Multiple Object Tracking (MOT) •  YouTubeデモ (GMMCP) 3 ※筆者は物体追跡については専門ではないので細かいとこ ろに誤りがある可能性があります.遠慮無く指摘をお願い 致します.
  4. 4. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking ちなみに • 筆者らはMultiple Object Trackingにかかわる論文 をもうひとつCVPR2015で発表している(強い) •  Target Identity-aware Network Flow for Online Multiple Target Tracking •  筆頭著者も一緒(Ph.Dの学生,ちなみに去年も2本筆頭で発表.強い) 4
  5. 5. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking その他 • H. Possegger et al., In Defense of Color-based Model-free Tracking •  モデルフリートラッキング(非detection based) • T. Liu et al., Real-time part-based visual tracking via adaptive correlation filters •  パートベースのトラッキング • S. Tang et al., Subgraph Decomposition for Multi- Target Tracking 5
  6. 6. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking Data Association (Naïvest) 6 Frame n Frame n+1 Bipartite Matching Problem
  7. 7. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking Tracking 7 Detection Data Association http://crcv.ucf.edu/projects/GMMCP-Tracker/CVPR15_GMMCP_Presentation.pptx
  8. 8. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking Tracking 8 Detection Data Association http://crcv.ucf.edu/projects/GMMCP-Tracker/CVPR15_GMMCP_Presentation.pptx
  9. 9. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking Data Association (Naïvest) 9 Frame n Frame n+1 Bipartite Matching Problem
  10. 10. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking Data Association (Naïvest) 10 Frame n Frame n+1 Bipartite Matching Problem
  11. 11. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking Data Association (Network Flow) 11 Frame n Frame n+1 Frame n+2 Frame n+3 sources sinks minimum-cost maximum-flow problem • incorporating motion feature • multi-commodity network
  12. 12. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking 12 Frame  1   Frame  2   Frame  3  
  13. 13. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking However, • Data association with network flow is simplified formulation of this problem • Assuming no simplification is closer to the tracking scenario in real world. 13
  14. 14. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking Data Association (Not Simplify) 14 Frame n Frame n+1 Frame n+2 Frame n+3 重み  =  0.95   重み  =  0.10   うまいこと重みが最大 になるクリークを探す
  15. 15. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking Preliminary: clique (クリーク) • 任意の2点を結ぶ枝がある頂点集合のこと •  see wikipedia in detail • 今回は「各クラスタから1つのノードを選んでで きる部分グラフ」という理解でOK 15
  16. 16. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking Data Association (Not Simplify) 16 Frame n Frame n+1 Frame n+2 Frame n+3 Input: k-partite complete graph (完全k部グラフ) A person form a clique ↓ maximum clique problem
  17. 17. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking GMCP Tracker[1] • The same team s ECCV 2012 paper • They formulate MOT as generalized maximum clique problem. (cf. former page) 17[1] Amir Roshan Zamir et al., GMCP-Tracker: Global Multi-object Tracking Using Generalized Minimum Clique Graphs, ECCV, 2012.
  18. 18. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking However (2), • Due to complexity of the model, these approaches have been solved by approximate solutions. • GMCP Tracker also used a greedy local neighborhood search, which is prone to local minima. • GMCP Tracker doesn t follow a joint optimization for all the tracks simultaneously (one by one). 18
  19. 19. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking Contribution 1.  this approach doesn t involve any simplification neither in formulation nor in optimization (Binary Integer Problem). 2.  they propose a more efficient occlusion handling strategy, which can handle long-term occlusions (e.g. 150 frames) and can speed-up the whole algorithm. 19
  20. 20. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking Contribution 1.  this approach doesn t involve any simplification neither in formulation nor in optimization (Binary Integer Problem). 2.  they propose a more efficient occlusion handling strategy, which can handle long-term occlusions (e.g. 150 frames) and can speed-up the whole algorithm. 20
  21. 21. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking 21 Low-level Tracklets Segment 01 Segment 05 Segment 06 Segment 10 Mid-level Tracklets Final Trajectories GMMCP GMMCP Input Video Human Detection Detected Humans
  22. 22. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking 22 Low-level Tracklets Segment 01 Segment 05 Segment 06 Segment 10 Mid-level Tracklets Final Trajectories GMMCP GMMCP Input Video Human Detection Detected Humans
  23. 23. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking Step 0: Low-level Tracklet • In GMCP, the nodes at first step are each detections. 23 Frames  1-­‐10   • In GMMCP, the nodes are (low-level) tracklet •  How to find: bounding boxes that overlap more than 60% between two frames are regarded as being connected.
  24. 24. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking 24 Low-level Tracklets Segment 01 Segment 05 Segment 06 Segment 10 Mid-level Tracklets Final Trajectories GMMCP GMMCP Input Video Human Detection Detected Humans
  25. 25. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking Step 1: Mid-level Tracklet 25 • 各クラスタ(青円)からひとつのノード(赤線) を選び,クリークを作る Frames  1-­‐10   Frames  11-­‐20   Frames  21-­‐30   Frames  31-­‐40   Frames  41-­‐50   Frames  51-­‐60  
  26. 26. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking Step 1: Mid-level Tracklet 26 • エッジの重み = (見た目特徴) + (動き特徴) • これを基に最適化をすると・・ Frames  1-­‐10   Frames  11-­‐20   Frames  21-­‐30   Frames  31-­‐40   Frames  41-­‐50   Frames  51-­‐60  
  27. 27. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking Step 1: Mid-level Tracklet 27 • このような三人の軌跡が同時に検出できる • オクルージョンに対応するため,ダミーノードを 入れてある Frames  1-­‐10   Frames  11-­‐20   Frames  21-­‐30   Frames  31-­‐40   Frames  41-­‐50   Frames  51-­‐60  
  28. 28. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking 28 Low-level Tracklets Segment 01 Segment 05 Segment 06 Segment 10 Mid-level Tracklets Final Trajectories GMMCP GMMCP Input Video Human Detection Detected Humans
  29. 29. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking Step 2: Final Trajectories • The another but similar problem with step 1. • They solve GMMCP: •  Nodes are Mid-level Tracklet •  For appearance feature, they use median (or average) feature among detections in each frame •  For motion feature, they use middle point of mid-level tracklet as the location of each node 29
  30. 30. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking Appearance Affinity • Feature: Invariant Color Histogram [2] •  Deformation and viewpoint invariant • Affinity: Histogram Intersection 30[1] J. Domke et al., Deformation and Viewpoint Invariant Color Histogram, BMVC, 2006 min(H1[i], H2[i])
  31. 31. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking Motion Affinity 31[1] J. Domke et al., Deformation and Viewpoint Invariant Color Histogram, BMVC, 2006 今の位置   前の位置+速度度から 予想される位置  
  32. 32. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking Optimization • GMMCP is NP Hard, but they solve without any simplification. • They formulate GMMCP as Binary Integer Problem (BIP, 0-1整数計画問題) 32
  33. 33. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking 33http://www.dais.is.tohoku.ac.jp/ shioura/teaching/dais08/dais02.pdf
  34. 34. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking 34http://www.dais.is.tohoku.ac.jp/ shioura/teaching/dais08/dais02.pdf
  35. 35. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking Optimization • GMMCP is NP Hard, but they solve without any simplification. • They formulate GMMCP as Binary Integer Problem (BIP, 0-1整数計画問題) 35 • これは実は組合せ最適化と言われる問題 • cf. 0-1ナップザック問題,巡回セールスマン問 題
  36. 36. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking BIP in this case • C is weight matrix (?) • x is boolean column vector •  the elements of x is all of edges and nodes • Ax = b is equality constraints • Mx <= n is inequality constraints 36
  37. 37. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking 37 各クラスタごとに1に なってるのは定数K Notation : i th node in j th cluster : edge between and h: Number of clusterseij mn vm n vi j vi j あるノードから伸び るエッジはh-1(かゼ ロ) クリークを作ってい るかどうか 3種の制約
  38. 38. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking Contribution 1.  this approach doesn t involve any simplification neither in formulation nor in optimization (Binary Integer Problem). 2.  they propose a more efficient occlusion handling strategy, which can handle long-term occlusions (e.g. 150 frames) and can speed-up the whole algorithm. 38
  39. 39. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking Occlusion Handling • Detector can detect not all the persons in each frame •  Occlusion, Detection Error, … • They add Dummy Node to each cluster • Cost of dummy edge ( = edge connected to dummy node) is fixed value. 39
  40. 40. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking Occlusion Handling 40 = さっきまで出てきてた重み ( 見た目 + 動き )cj1 cj2 = 定数c_d cj3 , cj4 = 0
  41. 41. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking Occlusion Handling • How/How many do we add dummy nodes? •  Many dummy nodes increase computational complexity • cf. case of GMCP: •  They add dummy node by the motion-based way •  ある答えに対して等速度運動を仮定して,大きくハズレ てしまうようなクラスタにダミーノードを足す •  Many dummy nodes increase computational complexity (大事なので2度) 41
  42. 42. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking Occlusion Handling • Aggregated Dummy Nodes (ADN) •  no longer be boolean variable •  can take any integer value •  add only one ADN to each cluster •  Not connected to other nodes! • New Solution: Mixed-Binary-Integer Programming 42 Constraint 1 Constraint 2 Constraint 3 各クラスタごとに1に なってるのは定数K あるクラスタから伸 びるエッジは1か0 クリークを作ってい るかどうか
  43. 43. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking Occlusion Handling 43 cj1 cj2 はない ( cj3 , cj4= 0 cd 2 cj3 , cj4 = = さっきまで出てきてた重み ( 見た目 + 動き )
  44. 44. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking ここからひたすら結果 44
  45. 45. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking Metrics 45
  46. 46. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking 46 Dataset Method MOTA MOTP MT ML IDS TownCenter MPT 72.9 71.3 - - - GMCP 75.59 71.93 - - - Ours 77.37 66.38 86.09 4.35 68 TUDCrossing MWIS 85.9 73 - - 2 GMCP 91.63 75.6 - - 0 Ours 91.9 70 75 0 2 TUDStadmitte DLP 79.3 73.9 - - 4 GMCP 77.7 63.4 - - 0 Ours 82.4 73.9 80 0 0 Parking Lot1 H2T 88.4 81.9 78.57 0 21 GMCP 90.43 74.1 - - - Ours 92.9 73.6 92.86 0 4
  47. 47. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking 47 Dataset Method MOTA MOTP MT ML IDS ParkingLot2 KSP 45.4 57.8 46.15 0 531 DCT 60.1 56.1 76.92 0 234 CMOT 80.7 58 84.62 0 61 GMCP 75.6 58.1 61.54 0 76 IHTLS 78.8 57.9 84.62 0 50 Ours 87.6 58.1 92.31 0 7 Pizza KSP 51.8 65.7 39.13 0 249 DCT 53.5 65.8 69.57 0 185 IHTLS 57.6 66.8 43.48 4.35 105 CMOT 56.9 63.3 30.43 4.35 87 GMCP 57.6 68.6 26.9 4.35 52 Ours 59.5 64.1 30.43 0 55
  48. 48. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking 時間的評価 48
  49. 49. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking 時間的評価 49
  50. 50. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking 時間的評価 50
  51. 51.                                                                                              GMMCP  Tracker:  Globally  Op3mal  Generalized  Maximum  Mul3  Clique  Problem  for  Mul3ple   Object  Tracking   TUD-Stadmitte Mid-­‐level  Tracklets   Final  Trajectories  
  52. 52.                                                                                              GMMCP  Tracker:  Globally  Op3mal  Generalized  Maximum  Mul3  Clique  Problem  for  Mul3ple   Object  Tracking  
  53. 53.                                                                                              GMMCP  Tracker:  Globally  Op3mal  Generalized  Maximum  Mul3  Clique  Problem  for  Mul3ple   Object  Tracking  
  54. 54. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking まとめ (拝借) • Formulate MOT as GMMCP •  a new graph theoretic problem • Formulate GMMCP as a MBIP •  GMMCP is NP Hard but no approximate solutions • An efficient occlusion handling through AND • Performance close to real-time • Improving state-of-art on several sequences
  55. 55. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking project page • 本物のproject pageが情報量多くてこれがCVPR複 数本2年連続で通す人のページか,と思いました • http://crcv.ucf.edu/projects/GMMCP-Tracker/ 55
  56. 56. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking Implementation Detail • Detection: DPM • K: target-specific • (1st layer ) number of cluster: 5 (2-6で実験) • (2nd layer) number of cluster: 6 56

×