Wbc master thesisdefense

1. Tracking Colliding Cells Nhat ‘Rich’ Nguyen Committee Members: Min Shin, PhD, Richard Souvenir, PhD, and Toan Huynh, MD

3. Lookout for signs of diseaseMicroscopic scale: 7,000 to 25,000 in a drop of blood 2

4. Flowing Rolling Adhering Motion Behaviors 3 Microscopy Video of white blood cells

5. Biologists learn cell behaviors by tracking a few cells 4

6. Manual Automatic Tedious Little Effort Expensive Inexpensive Subjective Objective 5

7. Previous Automatic Methods Ray et al. [2002] Active Contour Cui et al. [2005] Monte Carlo Mukherjee et al. [2004] Level Set Analysis 6

8. Previous Automatic Methods Eden et al. [2005] SmoothnessConstraints Li et al. [2005] Lineage Construction Smith et al. [2008] Probabilistic Formalization 7

9. Qualitative Comparison 8

10. Challenges As many cells move at a wide range of speed Collisions! 9

11. Challenges In a collision, cell motion and appearance 1. could be different 2. change abruptly 10

12. Example Eden et al. [2005] broken tracks robust tracks Our Method 11

13. Approach To improve tracking accuracy of colliding cells by: having separate collision states to describe cells inside and outside of collisions testing multiple hypotheses of cell motion and appearance as transitions between abrupt motion patterns. 12

14. Cell Detection Classify each pixel in the image as a Cell or Background 13

15. Detection Problems 14 Variation in cell appearances within an image time Varied appearance of a cell over time

16. AdaBoost 15 Idea: combine many “rules of thumb” to a highly accurate prediction rule. Input: visual features from training samples. Schema: maintain a strategy to determine “rules of thumb” using weight distribution. Output: a single strong classifier which is a linear combination of the set of weak classifiers.

17. Training 100 Cell Samples 100 Background Samples Features Mean Intensity Standard Dev. of Intensity Radial Mean  Decision Rules on feature scores 16

18. Detection Procedure Scan each pixel p in the image Compute image feature vector V from a window centered around p Classify p as a Cell pixel if the feature score in V satisfies the learned decision rule; otherwise classify p as a Background pixel. Cluster groups of Cell pixels into cell observation. 17

19. Cell Tracking 1. Predict using multiple hypotheses 2. Correspond predictions and measurements 3. Update based on the current state 18

22. Motion and Appearance Model 21 Collision States: State Transition: Hypotheses: to predict the state in the next frame control input vector State Vector*: Motion and Appearance Models: (for ) state transition matrix control input matrix process noise vector ~N(0,Qs) Observation Vector:

23. 22

24. 23 Tracking Steps 3 1 2 4

25. 24 Tracking Steps 3 1 2 4

26. 25 Tracking Steps 3 1 2 4

27. 26 Tracking Steps 3 1 2 4

28. 27 Tracking Steps 3 1 2 4

29. 28 Tracking Steps 3 1 correspond to H01 no measurement 2 4 correspond to H00 no prediction

30. 29 Tracking Steps 3 1 colliding cells lost cell 2 4 5 non-colliding cell new cell

31. 30 Tracking Steps 3 keep moving stay colliding 1 4 split away 2 5 keep moving start moving

32. 31

33. Data 8 300 ~6k image sequences cells tracks cell positions 32

34. Colliding vs. Non-colliding 33 112 188 colliding cells non-colliding cells

35. Evaluating Methods 34 Smoothness Constraints (SC) Single Hypothesis (SH) Multiple Hypotheses (MH)

36. Detection Performance 35 Recall : TP / (TP + FN) 75% Precision: TP / (TP + FP) 77%

37. Tracking Performance 36 RMSE : Root mean squared errors of position (pixel) -0.03 -0.17 +0.36 -0.21 +0.33 -0.20 SH introduces additional error in positions. MH does not introduce any additional error. Estimating colliding cells’ positions is more difficult.

38. Tracking Performance 37 PTP: Percentage of Tracked Positions (%) +27 +9 +23 +3 +4 +24 +28 Large improvement in colliding positions. Improvement overall. Tracking colliding cells is more difficult.

39. Comparisons MH SC SH 38

40. Discussions 39 The effect of collision duration on tracking 6 112 Exclude SC from being considered for collision. Classify colliding positions into bins based on the number of frames of the collision. colliding cells bins of collision duration

41. Discussions 40 The effect of collision duration on RMSE

42. Discussions 41 The effect of collision duration on PTP

43. Discussions 42 The impact of detection on tracking 38 596 Data with good detection results before and after collision (+/- 2 frames) cell positions treated colliding cells

44. Discussions 43 The impact of detection on RMSE -0.17 -0.13 -1.05 -1.09 Different improvement between dataset. Different improvement between methods.

45. Discussions 44 The impact of detection on PTP +9 +7 +16 +18 Large improvement between dataset. Large different between methods. MH achieves high performance in tracking.

46. Results Region B Our Method 45

47. Future Work 1. Add more features to improve detection. 46 5 7 6 8

48. Future Work 2. Incorporate a probabilistic approach to transition between collision states. 47 72 73 75 76

49. Future Work 3. Expand to track cells with more complex motions and behaviors. 48 49 50 51 52

50. Conclusion 49 A method for tracking colliding cells. Incorporate Kalman filter and multiple hypotheses for each collision state. Improve 28%in tracked position coverage compared to a previous work . Achieve 88% in tracked position coverage in tracking colliding cells.

51. Questions/Comments 50

52. S. J. Schmugge, S. Keller, N. Nguyen, R. Souvenir, T. H. Huynh, M. Clemens, M. C. Shin. "Segmentation of Vessels Cluttered with Cells using a Physics based Model". 11th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), New York, September 6-10 2008. N. Nguyen, S. Keller, T. Huynh, M. Shin. “Tracking Colliding Cells”. IEEE Workshop on Applications of Computer Vision (WACV), Snowbird, UT December 07-09 2009. to be submitted to IEEE Transactions on Medical Imaging N. Nguyen, S. Keller, Eric Norris, T. Huynh, M. Shin. “Tracking Colliding Cells in Intravital Microscopy Images”. Related Publications 51

53. More Results 52 Eden et al. [2005] Our Method

54. 53 Tracking Steps Predict motion Predict collision Get measurements Get errors in position & area Match with minimal error

55. Collision States: Hypotheses: to predict the state in the next frame control input vector State Vector*: Motion and Appearance Models: (for ) state transition matrix control input matrix process noise vector ~N(0,Qs) Observation Vector: Measurement Model: measurement noise vector ~N(0,R) measurement transition matrix 54

56. State Vector of cell i : Predicted State Vector: Zero Matrix Zero Matrix Zero Vector Zero Vector Predicted Covariance: 55

57. z1 x’1a Calculate error between all possible pairs Sm & On Criteria: Position & Area. Assign pairs via Greedy Search HФ 2. Correspondence X1 x’1b HФ Cell Detector z2 x'2a HФ X2 HФ x’2b z3 x’3a HФ X3 HФ x’3b zn X1 x’ka HФ z1 x’1a Xk HФ x’kb X2 x’2b z3 Cell 3 is missing x’3a X3 New cell has entered z2 X4 Ck zn x’kb 56

58. Predicted State Vector: Hypothesized Measurement Vector: measurement transition matrix Error of hypothesis : observation from the detector weight vector Rule 1: Rule 2: error threshold of Unlikely (i, k) pair Stop corresponding condition: 57

60. 59 II. AdaBoost1. Adaptive Boosting method Robert Schapire algorithm (1996). Idea: Combine many “rules of thumb”  a highly accurate prediction rule. Maintain a strategy to determine “rules of thumb”  weight. Terms: Learner = Hypothesis = Classifier. Weak Learner: <50% error rate. Strong Learner: linear combination of weak learner.

61. 60 II. AdaBoost2. An Example of Boosting Plastic Apple Real Apple

62. 61 II. AdaBoost3. Features Space Weight Color

63. 62 II. AdaBoost4. 1st Iteration Wrong guess Wrong guess Weak Learner Optimal Guessing line

64. 63 II. AdaBoost5. 1st Reweight Wrong guess Gets larger Get smaller Right guess Gets smaller

65. 64 II. AdaBoost6. 2nd Iteration More weighted ones get classify correctly Wrong guess Wrong guess

66. 65 II. AdaBoost7. 2nd Reweight smaller Wrong guess Gets larger smaller Right guess Gets smaller

67. 66 II. AdaBoost8. 3rd Iteration

68. 67 II. AdaBoost9. 4th Iteration

69. 68 II. AdaBoost10. Combination 3rd 2nd 1st 4th

70. 69 II. AdaBoost11. Decision Plastic Apple Region Decision Boundary Real Apple Region

71. 70 Tracking Steps

72. 71 Tracking Steps

73. 72

74. 73

75. Tracking Performance 74 +.24 +.04 +.23 +.03 +.23 +.09 +.21 +.18

76. Discussions 75 The impact of detection on PTP +7 +16 +11 Large improvement between dataset. Large different between methods. MH achieves high performance in tracking.

77. Training 100 Cell Samples 100 Background Samples Features Mean Intensity Standard Dev. of Intensity Radial Mean  Decision Rules on feature scores 76

78. 77