This poster was presented at the University of Florida AI Days 2023. It is based on this paper: https://arxiv.org/abs/2308.12252 Title: How Safe Am I Given What I See? Calibrated Prediction of Safety Chances for Image-Controlled Autonomy Abstract: End-to-end learning has emerged as a major paradigm for developing autonomous systems. Unfortunately, with its performance and convenience comes an even greater challenge of safety assurance. A key factor of this challenge is the absence of the notion of a low-dimensional and interpretable dynamical state, around which traditional assurance methods revolve. Focusing on the online safety prediction problem, this paper proposes a configurable family of learning pipelines based on generative world models, which do not require low-dimensional states. To implement these pipelines, we overcome the challenges of learning safety-informed latent representations and missing safety labels under prediction-induced distribution shift. These pipelines come with statistical calibration guarantees on their safety chance predictions based on conformal prediction. We perform an extensive evaluation of the proposed learning pipelines on two case studies of image-controlled systems: a racing car and a cartpole.

1. ■ Adding physical constraints to latent states
similar to Neural ODEs (Wen, Wang, and
Metaxas 2022)
■ Jointly learning forecasters and evaluators
to overcome distribution shifts
■ Using world-model transformer
architectures
■ Applying predictors to physical systems
Challenges:
■ Unknown states and dynamical models
■ High-dimensional (1000+) observations
■ Difficult to reliably quantify confidence
Contributions:
■ Modular family of online safety
predictors
■ Conformal confidence calibration
■ Evaluation on racing car and cart pole
■ Given a system that contains state
(x), observation (y), and a safety
property φ (x) :
■ Is the system safe now?
─ φ (xi
| yi
)
■ Will the system be safe after k steps?
─ φ ( xi+k
| yi
)
■ What’s the probability of safety after k
steps?
─ P(φ (xi+k
) | yi
)
Conformal confidence calibration:
Draw N observation-state pairs {b1
, b2
, ..., bN
} M times from calibration datasets to
get M datasets {B1
, B2
, ..., BM
}. By implementing Conformal Calibration Algorithm
on predictor g given miscoverage level 𝜶 and M datasets, we can get an upper
calibration error bound c for a newly drawn BM+1
that:
P( |qM+1
− pM+1
| ≤ c ) ≥ 1 − α
where q is the mean true safety and p is the mean safety chance prediction scores:
Preprint: Mao, Z.; Sobolewski, C.; and Ruchkin, I. "How Safe
Am I Given What I See? Calibrated Prediction of Safety
Chances for Image-Controlled Autonomy."
arXiv:2308.12252 (2023).
Conformal calibration: Vovk, V.; Gammerman, A.; and
Shafer, G. 2005. Algorithmic Learning in a Random World.
New York: Springer, 2005 edition. ISBN 978-0-387-00152-4.
World model: Ha, D.; and Schmidhuber, J. 2018. Recurrent
World Models Facilitate Policy Evolution. In Advances in
Neural Information Processing Systems, volume 31.
Zhenjiang Mao, Carson Sobolewski, Ivan Ruchkin
Trustworthy Engineered Autonomy (TEA) Lab
Department of Electrical and Computer Engineering, University of Florida
How Safe Am I Given What I See?
Calibrated Prediction of Safety Chances for Image-Controlled Autonomy
APPROACH
FUTURE WORK
RESULTS
CHALLENGES & CONTRIBUTIONS
PROBLEM
REFERENCES
■ Left: Monolithic vs composite predictor for the racing car
■ Right: Image vs latent pred. for the racing car; shaded uncertainty shows STD across controllers
and variance due to resampling; “mon.”,“comp.”,“c-sp”, and “ind” stand respectively for
monolithic, composite, controller-specific, and controller-independent.
Racing car, monolithic predictor, horizon k = 100.
Left: uncalibrated; Right: calibrated w/ isotonic regression & conformal bounds for α = 0.05.
■ Left: Composite predictors have a lower FPR, while monolithic ones have higher a F1 score
■ Right: Latent predictors outperform image predictors
Lower and bounded expected calibration error (ECE) after calibration