0
XuehanXiong, Daniel Munoz, J. Andrew Bagnell, Martial Hebert<br />Carnegie Mellon University<br />3-D Scene Analysis via S...
Introduction<br />Range scanners as standard equipment<br />Scan segmentation = distribute the points in object classes<br...
Definition<br />3D point cloud classification <br />= assign one of the predefined class labels to each point of a cloud b...
Segmentation algorithms<br />Exploit different features<br />Automatic trade off<br />Enforce spatial contiguity<br />Adja...
Classification = Training + Validation<br />Data: labeled instances  3D scan manually labeled<br />Training set<br />Vali...
Some State-of-the-art Classifiers<br />Support vector machine<br />Random forests<br />Apache Mahout<br />Perceptron<br />...
Approach – Generative Model<br />Learn p(y) and p(x|y) – classification step<br />Use Bayes rule:<br />Classify as:<br />p...
Approach – Generative Model<br />Carnegie Mellon University, Artificial Intelligence, Fall 2010 <br />
State of the Art 3D Point Cloud Classifier<br />Markov Random Fields<br />Scan points modeled as random variables = nodes<...
Markov Random Fields<br />Conditional Probability Query: P(Y| X = xi) = ?<br />Generate joint distribution, exhaustively s...
Xiong et al. Approach<br />Explicit joint probability distribution model:<br />Does not model P(y|x) directly<br />Exact i...
Overview<br />2 level hierarchy<br />Top-level: region, mixed labels<br />Bottom-level: points<br />K-means++, k = 1% poin...
Overview<br />At point level, train 2 classifiers:<br />Inter-level context + point cloud descriptors<br />Neighboring poi...
Base Classifier (LogR)<br />Assumption: <br />log p(y|x) of each class is a linear function of x + a normalization constan...
Base Classifier<br />Max-likelihood estimation:<br />Use regularization to avoid over fitting<br />Concave problem, solved...
Contextual Features<br />Construct a sphere around the region centroid (O)<br />12 meter radius<br />Divide the sphere in ...
Multi-Round Stacking (MRS)<br />X = {xi} – training set<br />Y = {yi} – label distribution (ground truth)<br />w1 = T(X, Y...
MRS – Avoid Overfitting<br />Generate multiple temporary classifiers<br />Partition the training set into 5 disjoint sets<...
Examine the w parameters computed<br />A tree trunk region likely has:<br /> vegetation above, but not below<br />car and ...
Stacked 3D Parsing Algorithm (S3DP)<br />Labeled point cloud<br />Construct 2-level hierarchy<br />Top<br />Bottom<br />Ex...
Stacked 3D Parsing Algorithm (S3D)<br />Parse UP the hierarchy:<br />Apply N rounds of MRS on (Xb, Yb):<br /> N+1 classifi...
Stacked 3D Parsing Algorithm (S3D)<br />Parse DOWN the hierarchy:<br />Apply N rounds of MRS on (Xt, Yt):<br /> N+1 classi...
Experimental Setup - Features<br />Bottom level<br />Local neighborhood: 0.8m/2m radius<br />Compute covariance matrix and...
Experimental Setup - Features<br />Bottom & top levels<br />Bounding box enclosing the points<br />Over local neighborhood...
Evaluation Metrics<br />Questions answered<br />Correct answers<br />Objects correctly classified<br />Misclassified objec...
Experimental Results<br />VMR-Oakland-v2 Dataset<br />CMU Campus<br />3.1 M points<br />36 sets, each ~85,000 points<br />...
A. VMR-Oakland-v2 Dataset<br />M3N<br />Conditional Random Fields <br />MRF trained discriminatively<br />Pairwise model:<...
A. VMR-Oakland-v2 Dataset<br />M3N    <br />
A. VMR-Oakland-v2 Dataset<br />
Experimental Results<br />GML-PCV Dataset<br />2 aerial datasets, A and B<br />Each dataset split in training and test, ~1...
B. GML-PCV Dataset<br />
Experimental Results<br />RSE-RSS Dataset<br />10 scans, each ~ 65,000 points, Velodyne laser on the ground<br />Most diff...
Final Comments<br />S3DP performs a series of simple predictions<br />Effective encoding of neighboring contexts<br />Lear...
References<br />X. Xiong, D. Munoz, J. A. Bagnell, M. Hebert, 3-D Scene Analysis via Sequenced Predictions over Points and...
Upcoming SlideShare
Loading in...5
×

3D Scene Analysis via Sequenced Predictions over Points and Regions

1,561

Published on

I gave this talk in Machine Vision seminar at Jacobs University. I presented the state of the art in 3D point cloud classification and I described X. Xiong et al approach in a paper published in 2010.

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,561
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
15
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Inferring labels only on local features is very difficult: e.g: the viewpoint from which the objects are perceived can widely vary, the sensor irregularly samples points from objects, there is often local ambiguity in appearance.
  • Exploit different features:Trees may require different features from carsGround can be detected simply based on a “height” featureAs the number of features grows, it becomes important to learn how to trade them off automaticallyEnforce spatial contiguity:Adapt to the scanner used:- Particularly relevant, because real-world scans can violate standard assumptions made in synthetic data used to evaluate segmentation algorithms
  • - CPQ: the probability distribution over the values of y conditioned by X = xi
  • No real improvement beyond the 2 levelsThe regions in the hierarchy do not change during the procedureMain idea about neighboring regions predictions: contextual information should refine the prediction.
  • Used to classify regions
  • - Large radii are needed to cover the large spatial extent from an object’s centroid to its neighboring regions’ points
  • By sequentially training a series of classifiers, we can ideally learn how to fix the mistake from previous one. Additionally, we can use these previous predictions as contextual cues
  • Top left: ground truth, initial labelingTop right: one round of stacking: pole mistaken as building Bottom left: 2 rounds – better results Bottom right:4 rounds Regions that do not fit the context are iteratively corrected
  • F1 score is used instead of comparing accuracy because accuracy can hide performance of classes with few samples and there is a class imbalance in all datasets-&gt; recall = objects correctly classified/total number of objects-&gt; precision = correct/total classified
  • (f) top border is vegetation =&gt; missclasified =&gt; higher order potentials over region prefer for the region to have same label =&gt; S3DP is better at boundary regions(e) context learn fail =&gt; shrubs labeled as bulding, S3DP learned that vegetation is “above” and corrected this label assignment
  • Shape info lost (aerial set)Car &amp; low vegetation are the sameGround points not on the same elevation =&gt; little info given by elevLogR does NOT diff between low &amp; high vegetation, S3DP uses stacking and learns that low veg has a higher ground distrib
  • Transcript of "3D Scene Analysis via Sequenced Predictions over Points and Regions"

    1. 1. XuehanXiong, Daniel Munoz, J. Andrew Bagnell, Martial Hebert<br />Carnegie Mellon University<br />3-D Scene Analysis via Sequenced Predictions over Points and Regions<br />Presenter: Flavia Grosan<br />Jacobs University Bremen, 2011<br />me@flaviagrosan.com<br />http://flaviagrosan.com<br />
    2. 2. Introduction<br />Range scanners as standard equipment<br />Scan segmentation = distribute the points in object classes<br />Scene understanding<br />Robot localization<br />Difficulties in 3D:<br />No color information<br />Often noisy and sparse data<br />Handling of previously unseen object instances of configurations<br />
    3. 3. Definition<br />3D point cloud classification <br />= assign one of the predefined class labels to each point of a cloud based on:<br /> Local properties of the point<br />Global properties of the cloud<br />
    4. 4. Segmentation algorithms<br />Exploit different features<br />Automatic trade off<br />Enforce spatial contiguity<br />Adjacent points in the scan tend to have the same label<br />Adapt to the scanner used<br />Different scanners produce qualitatively different outputs<br />
    5. 5. Classification = Training + Validation<br />Data: labeled instances  3D scan manually labeled<br />Training set<br />Validation set<br />Test set<br />Training<br />Estimate parameters on training set<br />Tune parameters on validation set <br />Report results on test set<br />Anything short of this yields over-optimistic claims<br />Evaluation<br />Many different metrics<br />Ideally, the criteria used to train the classifier should<br />be closely related to those used to evaluate the classifier<br />Statistical issues<br />Want a classifier which does well on test data<br />Overfitting: fitting the training data very closely, but not<br />generalizing well<br />Error bars: want realistic (conservative) estimates of accuracy<br />Training<br />Data<br />Validation<br />Data<br />Test<br />Data<br />
    6. 6. Some State-of-the-art Classifiers<br />Support vector machine<br />Random forests<br />Apache Mahout<br />Perceptron<br />Nearest neighbor – kNN<br />Bayesian classifiers<br />Logistic regression<br />
    7. 7. Approach – Generative Model<br />Learn p(y) and p(x|y) – classification step<br />Use Bayes rule:<br />Classify as:<br />p(y)<br />p(x|y)<br />p(y|x)<br />
    8. 8. Approach – Generative Model<br />Carnegie Mellon University, Artificial Intelligence, Fall 2010 <br />
    9. 9. State of the Art 3D Point Cloud Classifier<br />Markov Random Fields<br />Scan points modeled as random variables = nodes<br />Each random variable corresponds to the label of each point<br />Proximity links between points = edges<br />Defines joint distribution<br />Pairwise Markov networks<br />Node and edges associated with potentials<br />Node potential = a points ‘individual’ preference for different labels<br />Edge potential = encode interactions between labels of related points<br />
    10. 10. Markov Random Fields<br />Conditional Probability Query: P(Y| X = xi) = ?<br />Generate joint distribution, exhaustively sum out the joint.<br />Bad News: NP-hard <br />
    11. 11. Xiong et al. Approach<br />Explicit joint probability distribution model:<br />Does not model P(y|x) directly<br />Exact inference is hard <br />Approximate inference leads to poor results<br />Instead, directly design and train an inference procedure via sequence of predictions from simple machine learning modules<br />Use discriminative model<br />Logistic regression<br />Max-likelihood estimation problem<br />
    12. 12. Overview<br />2 level hierarchy<br />Top-level: region, mixed labels<br />Bottom-level: points<br />K-means++, k = 1% points to establish <br />initial clusters<br />Predict label distribution per region<br />Update each region’s intra-level context <br /> using neighboring regions predictions<br />Pass the predicted label distribution to <br /> the region’s points inter-level context<br />
    13. 13. Overview<br />At point level, train 2 classifiers:<br />Inter-level context + point cloud descriptors<br />Neighboring points predictions<br />Move up in the hierarchy:<br />Average the predicted label <br />distribution of points over a region<br />Send the average as inter-level<br />context to the region<br />Validation set determines the number of up-down iterations<br />
    14. 14. Base Classifier (LogR)<br />Assumption: <br />log p(y|x) of each class is a linear function of x + a normalization constant<br />Ci– RV for the class of a region<br />xi – features <br />yi – ground truth distribution of K labels<br />w – parameters  ?<br />
    15. 15. Base Classifier<br />Max-likelihood estimation:<br />Use regularization to avoid over fitting<br />Concave problem, solved with stochastic gradient descent<br />Choose an initial guess for w<br />Take a small step in the direction opposite the gradient<br />This gives a new configuration<br />Iterate until gradient is 0<br />
    16. 16. Contextual Features<br />Construct a sphere around the region centroid (O)<br />12 meter radius<br />Divide the sphere in 3 slices on vertical: 4m radius<br />Average points’ label distribution within each slice <br />A feature vector of length K/slice<br />Average angles formed between z-axis and the [O, Ni] vector<br />Ni = neighboring point (not part of this region)<br />Models the spatial configuration of neighboring points<br />3(K+1) contextual features  add them to xi (a region’s features)<br />
    17. 17. Multi-Round Stacking (MRS)<br />X = {xi} – training set<br />Y = {yi} – label distribution (ground truth)<br />w1 = T(X, Y)- first trained classifier<br />Y’ = P(X, w1)<br />Use Y’ to compute new contextual features for X X’<br />w2 = T(X’, Y’) – train a second classifier<br />Repeat until no improvement seen<br />w1 is optimistically correct  w2 prone to overfitting<br />
    18. 18. MRS – Avoid Overfitting<br />Generate multiple temporary classifiers<br />Partition the training set into 5 disjoint sets<br />Train temporary classifier γ = T(X – Xi, Y – Yi)<br />Use γ only on Xi to generate Y’I<br />Discard γ afterwards<br />Perform one or more <br /> rounds of stacking <br />
    19. 19. Examine the w parameters computed<br />A tree trunk region likely has:<br /> vegetation above, but not below<br />car and ground below, but not on top<br />
    20. 20. Stacked 3D Parsing Algorithm (S3DP)<br />Labeled point cloud<br />Construct 2-level hierarchy<br />Top<br />Bottom<br />Extract point cloud features<br />Create ground truth label distribution<br />(Xt, Yt) - top<br />(Xb, Yb) - bottom<br />
    21. 21. Stacked 3D Parsing Algorithm (S3D)<br />Parse UP the hierarchy:<br />Apply N rounds of MRS on (Xb, Yb):<br /> N+1 classifiers <br />Yb label prediction from the last round<br />Extend each region’s feature vector with the average of its children’s probability distribution in Yb<br />Apply N rounds of MRS on (Xt, Yt):<br /> N+1 classifiers<br />Save ft and fb for inference<br />
    22. 22. Stacked 3D Parsing Algorithm (S3D)<br />Parse DOWN the hierarchy:<br />Apply N rounds of MRS on (Xt, Yt):<br /> N+1 classifiers <br />Yt label prediction from the last round<br />Extend each point’s feature vector with the average of its parents’ probability distribution in Yt<br />Apply N rounds of MRS on (Xb, Yb):<br /> N+1 classifiers<br />Save ft and fb for inference<br />
    23. 23. Experimental Setup - Features<br />Bottom level<br />Local neighborhood: 0.8m/2m radius<br />Compute covariance matrix and eigenvalues a1> a2> a3<br />Scattered points: a1≅ a2≅a3 (vegetation)<br />Linear structures: a1, a2>>a3 (wires)<br />Solid surface: a1>> a2,a3 (tree trunks)<br />Scalar projection of local tangent and normal directions on to z-axis<br />
    24. 24. Experimental Setup - Features<br />Bottom & top levels<br />Bounding box enclosing the points<br />Over local neighborhood at bottom level<br />Over region itself at top level<br />Relative elevations<br />Take a horizontal cells of 10m x 10m, centered in centroid<br />Compute min z- and max z- coordinates<br />Compute 2 differences in elevation between region’s centroid elevation and it’s cells 2 extrema<br />
    25. 25. Evaluation Metrics<br />Questions answered<br />Correct answers<br />Objects correctly classified<br />Misclassified objects<br />Unclassified objects<br />TP<br />FP<br />Recall= <br />= fraction of all objects correctly classified <br />For a class k:<br />Precision=<br />= fraction of all questions correctly answered<br />
    26. 26. Experimental Results<br />VMR-Oakland-v2 Dataset<br />CMU Campus<br />3.1 M points<br />36 sets, each ~85,000 points<br />6 training sets<br />6 validation sets<br />All remaining – test sets<br />Labels: wire, pole, ground, vegetation, tree-trunk, building, car<br />Comparison with associative Max-Margin Markov Network (M3N) algorithm<br />
    27. 27. A. VMR-Oakland-v2 Dataset<br />M3N<br />Conditional Random Fields <br />MRF trained discriminatively<br />Pairwise model:<br />Associative (Potts) model:<br />
    28. 28. A. VMR-Oakland-v2 Dataset<br />M3N <br />
    29. 29. A. VMR-Oakland-v2 Dataset<br />
    30. 30. Experimental Results<br />GML-PCV Dataset<br />2 aerial datasets, A and B<br />Each dataset split in training and test, ~1 M points each<br />Each training set split in learning and validation<br />Labels: ground, roof/building, tree, low vegetation/shrub, car<br />Comparison with Non-Associative Markov Network (NAMN)<br />Pairwise Markov network constructed over segments<br />Edge potentials non-zero for different labels<br />
    31. 31. B. GML-PCV Dataset<br />
    32. 32. Experimental Results<br />RSE-RSS Dataset<br />10 scans, each ~ 65,000 points, Velodyne laser on the ground<br />Most difficult set: noise, sparse measurements and ground truth<br />Labels:ground, street signs, tree, building, fence, person, car, background<br />Comparison with the approach from Lai and Fox:<br />Use information from World Wide Web (Google 3D Warehouse) to reduce the need for manually labeled training data <br />
    33. 33. Final Comments<br />S3DP performs a series of simple predictions<br />Effective encoding of neighboring contexts<br />Learning of meaningful spatial layouts<br />E.g.: tree-trunks are below vegetation<br />Usable in many environments scanned with different sensors<br />S3DP requires about 42 seconds<br />
    34. 34. References<br />X. Xiong, D. Munoz, J. A. Bagnell, M. Hebert, 3-D Scene Analysis via Sequenced Predictions over Points and Regions, ICRA 2011<br />D. Anguelov, B. Taskar, V. Chatalbashev, Discriminative Learning of Markov Random Fields for Segmentation of 3D Scan Data, Computer Vision and Pattern Recognition, 2005<br />G. Obozinski, Practical Machine Learning CS 294, Berkeley University, Multi-Class and Structured Classification, 2008<br />A. Kulesza, F. Pereira, Structured Learning with Approximate Inference, In Proceedings of NIPS'2007<br />K. Lai, D. Fox, 3D Laser Scan Classification Using Web Data and Domain Adaptation, In International Journal of Robotics Research, Special Issue on Robotics: Science & Systems 2009, July 2010   <br />D. Munoz, J.A. Bagnell, M. Hebert, Stacked Hierarchical Labeling, Paper and Presentation, European Conference on Computer Vision, 2010<br />D. Munoz, J. A. Bagnell, N. Vandapel, M. Hebert, Contextual Classification with Functional Max-Margin Markov Networks, Paper and Presentation, IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), June, 2009<br />R. Shapovalov, A. Velizhev, O. Barinova, Non-Associative Markov Networks for 3D Point Cloud Classification, PCV 2010<br />C. Sutton, An Introduction to Conditional Random Fields, Statistical Machine Learning Class, University of Edinburgh<br />M. Jordan, Machine Learning Class, University of California, Berkeley, Classification lecture<br />D. Munoz, D. Bagnell, N. Vandapel, M. Hebert, Contextual Classification with Functional Max-Margin Markov Networks, Paper Presentation, 2009<br />P.J. Flynn, A.K. Jain, Surface Classification: Hypothesis Testing and Parameter Estimation, CVPR, 1988<br />S.L. Julien, Combining SVM with graphical models for supervised classification: an introduction to Max-Margin Markov Networks, University of California, Berkeley, 2003<br />D. Koller, N. Friedman, L. Getoor,B. Taskar, Graphical Models in a Nutshell, In Introduction to Statistical Relational Learning, 2007 <br />
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×