Enhancing Worker Digital Experience: A Hands-on Workshop for Partners
ST.Monteiro-EmbeddedFeatureSelection.pdf
1. Embedded Feature Selection
of Hyperspectral Bands with
Boosted Decision Trees
Sildomar Monteiro and Richard Murphy
The University of Sydney
2. Rio Tinto Centre for Mine Automation
• Totally Autonomous Mine in 10 years:
– Brings together all elements of systems, perception,
machine learning, data fusion and more
– A grand challenge for Field Robotics
• Driven by safety, predictability and efficiency
Dr Sildomar Monteiro IGARSS 2011 2
2
3. Goal: Mine Picture Compilation
• Provide a complete and accurate model of the mine
– Mine planning and better prediction outcomes
• Maintain and update a multi-scale probabilistic
representation
– Geology
– Geometry
– Equipment
– And other properties of interest for the mining process
Dr Sildomar Monteiro IGARSS 2011 3
4. Today
Today
Geology Feedback to Batch
Floor mapping using
ripped trench sections
Cone logging
Dr Sildomar Monteiro IGARSS 2011 4
4
7. Hyperspectral sensing for mining
• Geology classification (material identification) still has
many challenges
• Environmental conditions
– Illumination, temperature, dust
• Timely data acquisition and processing is needed
– Algorithms and calibration
• High spectral similarity between (ore-bearing) rock
types
– Few, if any, distinctive spectral features
Dr Sildomar Monteiro IGARSS 2011 7
9. Hyper-spectral Sensors
Multispectral
Hyper-spectral
SWIR
VisNIR 970-2500 nm
400-970 nm
Band n
Band 6
Band 5
Band 4
Band 3
Dr Sildomar MonteiroBand 2 IGARSS 2011 9
Band 1
11. Hyperspectral Band Selection
• Feature Selection (vs Dimensionality Reduction)
– Remove correlated inputs
– Physical interpretation (band wavelengths)
• Faster data processing
• Possible faster data acquisition
• Can be tailored to application
• Indicate multispectral bands
Dr Sildomar Monteiro IGARSS 2011 11
12. Boosting
• Sound theoretical foundation
– Additive Logistic Regression [Friedman, 2000]
• Empirical studies show that boosting
– Yields small classification error rates
– Is very resilient to overfitting
• State-of-the-art results in many applications, e.g. face
recognition in computer vision
• The idea of Boosting is to train many “weak” learners
on various distributions (or set of weights) of the input
data and then combine the resulting classifiers into a
single “committee”
Dr Sildomar Monteiro IGARSS 2011 12
13. Decision Trees
• Advantages:
– Robustness and interpretability
• Disadvantages
– Low accuracy and high variance
• Binary decision trees (x )
f (x , , , a,b ) a (x ) b
b a b
• Boosted trees
– Accurate, robust and interpretable
M
G ( x) sign m f m ( x )
m1 m
2 3
1
Dr Sildomar Monteiro IGARSS 2011 13
14. Embedded Feature Selection
• Relative Importance of input variables
1
ˆ
F (x )
2
Ij Ex . varx x j
xj
• Approximation for decision trees (heuristic)
[Friedman, 1999]
J 1
ˆ
I j2 (T ) ˆ
it2 ( (t ) j)
t 1
• Least-squares improvement criterion
2 wl wr 2
i Rl , Rr yl yr
wl w r
Dr Sildomar Monteiro IGARSS 2011 14
15. Embedded Feature Selection (cont.)
• Boosted Decision Trees
M
ˆ 1 ˆ
I j2 I j2 Tm
M m 1
• The Multi-class case
K
ˆ 1 ˆ
Ij I jk
K k 1
Dr Sildomar Monteiro IGARSS 2011 15
16. Experiments
• Hyperspectral data acquired using a field
spectrometer (ASD)
– 429 bands (same as hyperspectral camera)
– Wavelengths from 350 nm to 2500 nm
• Samples of ore-bearing rocks
– Martite, goethite, kaolinite, etc (total 9 classes)
– Different illumination and physical conditions (direct sunlight,
shadow and viewing angles)
• Methodology of experiments
– Metrics: accuracy, precision, recall, F, Kappa, AUC
– 4-fold cross-validation
Dr Sildomar Monteiro IGARSS 2011 16
22. Summary
• Boosting increases the performance of decision trees
while keeping model interpretability
• We presented two approaches to perform feature
selection using boosted decision trees
• Calculating the relative importance of features was
more efficient than the counting of features
• The reduced set is able to predict the classes
accurately, and more efficiently than using all features
Dr Sildomar Monteiro IGARSS 2011 22
23. Conclusions
• The standard learning procedure of boosted decision
trees can perform feature selection automatically
• The feature selection is embedded in the internal
structure of the model, no need for extra parameters
or separate selection algorithms
• Instability of the models can be an issue
• Future work: how to determine the optimal number of
features (using statistical tests)
Dr Sildomar Monteiro IGARSS 2011 23
