For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2021/09/is-my-model-performing-well-it-depends-a-presentation-from-bmw-group/ Vladimir Haltakov, Self-Driving Car Engineer at BMW Group, presents the “Is My Model Performing Well? It Depends…” tutorial at the May 2021 Embedded Vision Summit. There are many statistical metrics used to measure the performance of machine learning models. While they work well when the model itself is the final product, classical measures are often not enough when the model is a part of a more complex system. In this presentation, Haltakov focuses on products where machine learning models do not generate the final output. The challenge in these cases is to define a metric that accurately captures the effect of different failures of the model on the final system performance. This becomes even more difficult when there are conflicting requirements within the system. He discusses strategies to deal with these problems and shares insights from real-world examples.

1. © 2021 BMW Group
Is My Model Performing Well?
It Depends…
Vladimir Haltakov
BMW Group

2. © 2021 BMW Group
How to Develop a Good Machine Learning Model?
• Experience with traffic light recognition in production
• Red Light Warning
• Urban Cruise Control
• Real world challenges
• Optimizing the customer function instead of the model
• Conflicting requirements in the project
2

3. © 2021 BMW Group
Classic Computer Vision Pipeline
3
Image
Computer
Vision Model
Output
Problem Loss Function Evaluation Metrics
Classification Cross Entropy Loss Recall, Precision, F1 Score
Regression Mean Squared Error Loss Mean Squared Error
Segmentation Pixel-wise Cross Entropy Loss Intersection-over-Union
Well known loss functions and evaluation measures

4. © 2021 BMW Group
Complex Computer Vision Pipeline
4
Image
Computer
Vision Model
Complex
Function
Logic
Output
• How does the model accuracy correlate with the function accuracy?
• How do false positives and false negatives influence the output?
• Which loss function to choose for training?
• When is my computer vision model good enough?
• Why did I improve the accuracy of the model, but the application got worse?

5. © 2021 BMW Group
Example Pipeline – Traffic Light Warning
5
Image
Computer
Vision Model
Complex
Function
Logic
Output
camera image detected traffic lights 3D estimation +
fusion with map
red light warning

6. © 2021 BMW Group
Example Scenario – Redundant Traffic Lights
6
False negatives and false positives don’t hurt the performance a lot

7. © 2021 BMW Group
Example Scenario – Single Traffic Light
7
False negatives and green false positives reduce the function availability

8. © 2021 BMW Group
Example Scenario – Different Signal Groups
8
Green false negatives lead to false warnings

9. © 2021 BMW Group
Example Scenario – No Traffic Light
9
Red false positives lead to false warnings

10. © 2021 BMW Group
The Problem
Measurable model problems are not directly correlated to problems on the output.
Sometimes improvement of the model’s accuracy lead to regression in the application
Strategies to deal with the problem
1. Formulate a joint optimization
2. Adapt the loss function to the problem
3. Optimize application specific error measures
10

11. © 2021 BMW Group
Joint Neural Network
1. Formulate a Joint Optimization
• Model the function logic as a neural network
• Optimize the computer vision and the function logic networks jointly
✓ Get the most performance out of the data
✘ May be infeasible for complex functions and too data intensive
11
Image
Computer
Vision Neural
Network
Function
Neural
Network
Output

12. © 2021 BMW Group
2. Adapt the Loss Function to the Problem
Add application specific terms to the loss function of the model
Examples for the traffic light warning function
• Penalize FN more for big traffic lights (closer to the car)
• Penalize FN less if there are other objects of the same type
• Penalize FP less if there are already objects of the same type present
✓ Constraints can be automatically optimized
✘ Not always possible
✘ Hyper-parameter tuning is difficult (see Resources for help)
12

13. © 2021 BMW Group
3. Optimize Application Specific Error Metrics
Work on solving only problems of the model that hurt the application
1. Evaluate the final output according to the application specific metrics
2. Review the error cases and categorize the model’s issues
3. Work on solving these categories without reducing overall performance
Example
13
10x False Warning
20x Missed Warning
6x FP red light
2x FN green light
5x Wrong distance estimation
13x FN red light
4x FP green light

14. © 2021 BMW Group
Conflicting Requirements
• Conflicting requirements depending on the scenario
• Conflicting requirements between functions
Example: Traffic Light Warning vs. Traffic Light Braking
14
Image
Computer
Vision Model
Complex
Function 1
Logic
Output
Function 1
Complex
Function 2
Logic
Output
Function 2

15. © 2021 BMW Group
Dealing with Conflicting Requirements
• Provide more information from the model
• Existence probabilities
• Classification scores
• Position covariances
• Tracker uncertainties
• The functions need to choose which data to use
• The functions are able to control the trade-offs
15
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
True
Positive
Rate
False Positive Rate
ROC Curve for Red Lights
Traffic Light Warning
Traffic Light Braking

16. © 2021 BMW Group
Model Calibration
• Interpret the output of the model as a
probability
• The probability usually doesn’t match the
real distribution
• Most models are not calibrated (e.g.
neural networks, SVM, decision trees)
• Various methods for calibration: Platt
Scaling, Isotonic Regression (see
Resources)
16
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Fraction
of
positives
Output probability
Calibration Plot
Perfect Calibration Uncalibrated Model

17. © 2021 BMW Group
Summary and Important Lessons
• In complex systems, the accuracy of the computer vision model does not directly
correlate to the performance of the final application
• Define problem specific evaluation metrics and loss functions
• Provide more information from the computer vision model to handle conflicting
requirements
• Calibrate the model output
17

18. © 2021 BMW Group
Resources
18
2021 Embedded Vision Summit
Watch my other talk
“Data Collection in the Wild.”
Model tuning and calibration
How we can make machine learning algorithms
tunable?
https://engraved.ghost.io/how-we-can-make-
machine-learning-algorithms-tunable/
Model Calibration
https://scikit-
learn.org/stable/modules/calibration.html