Presented at 13th ACM/IEEE Intl. Conf. on Cyber-Physical Systems, part of CPS-IoT Week, on May 4, 2022. Presentation video: https://youtu.be/nnhcUhih-vQ Abstract: Closed-loop verification of cyber-physical systems with neural network controllers offers strong safety guarantees under certain assumptions. It is, however, difficult to determine whether these guarantees apply at run time because verification assumptions may be violated. To predict safety violations in a verified system, we propose a three-step confidence composition (CoCo) framework for monitoring verification assumptions. First, we represent the sufficient condition for verified safety with a propositional logical formula over assumptions. Second, we build calibrated confidence monitors that evaluate the probability that each assumption holds. Third, we obtain the confidence in the verification guarantees by composing the assumption monitors using a composition function suitable for the logical formula. Our CoCo framework provides theoretical bounds on the calibration and conservatism of compositional monitors. Two case studies show that compositional monitors are calibrated better than their constituents and successfully predict safety violations.

1. Confidence Composition (CoCo) for
Monitors of Verification Assumptions
Ivan Ruchkin, Matthew Cleaveland, Radoslav Ivanov*,
Pengyuan Lu, Taylor Carpenter, Oleg Sokolsky, Insup Lee
Computer and Information Science Department
University of Pennsylvania
13th
ACM/IEEE Intl. Conf. on Cyber-Physical Systems
CPS-IoT Week
May the 4th, 2022
* Now @ RPI CS

2. 2
Req: no obstacle collisions
Req: no pipeline loss
Detection
confidence
Dynamics
confidence Detection
confidence
Distance d
Position y
Motivation: unmanned underwater vehicle (UUV)

3. — Build a comprehensive model
○ Bayesian/Markov networks [Pearl’88, Koller’09]
Our goal: combine run-time confidences into a predictive probability of safety
Existing work:
— Aggregate predictions of the same phenomenon
○ Forecast combination and ensemble learning[Ranjan’10, Sagi’18]
Motivation: unmanned underwater vehicle (UUV)
3
Req: no obstacle collisions
Req: no pipeline loss
Detection
confidence
Distance d
Position y
Dynamics
confidence Detection
confidence
→ but we have different phenomena
→ but requires dependencies between confidences

4. - Key concepts: verification assumptions, confidence monitors
- CoCo: a confidence composition framework
- Confidence calibration bounds
- Experimental results for UUV
- Key concepts: verification assumptions, confidence monitors
- CoCo: a confidence composition framework
- Confidence calibration bounds
- Experimental results for UUV
Agenda
4

5. Challenge: fragmented guarantees & monitors
5
Sensor
data
Control
NN
Perception
NN
Dynamics
Simulation
Confidence
monitor
Closed-loop
NN
verification
?
?
“Safe”
Confidence
monitor

6. Assumptions relate guarantees & monitors
6
Sensor
data
Assn 1
Assn 2
Monitor of
Assn 1
Monitor of
Assn 2
Sufficient
“Safe”
Dynamics
Simulation
Control
NN
Perception
NN
Closed-loop
NN
verification

7. — Discrete-output monitors: “Yes”, “No”, “Maybe” via sequential detection [Scharf’91, Poor’13]
— Limitations of discrete assumption monitoring:
○ Too coarse for highly uncertain assumptions → uninformative
○ Errors accumulate combinatorially → decreased performance [Ruchkin’20]
How to monitor assumptions?
7
— Key idea: Confidence monitoring instead of discrete monitoring [Ruchkin’21]
○ Confidence C in assumption A is an estimate of Pr(A)
○ Expected calibration error (ECE) for confidence C predicting satisfaction of A:
▹ ECE(C, A) = E[| Pr( A | C ) - C |]

8. Assumptions relate guarantees & monitors
8
Sensor
data
Assn 1
Assn 2
“Safe”
Dynamics
Simulation
Control
NN
Perception
NN
Closed-loop
NN
verification
Monitor of
Assn 1
Monitor of
Assn 2
Sufficient

9. - Key concepts: verification assumptions, confidence monitors
- CoCo: a confidence composition framework
- Confidence calibration bounds
- Experimental results for UUV
- Key concepts: verification assumptions, confidence monitors
- CoCo: a confidence composition framework
- Confidence calibration bounds
- Experimental results for UUV
Agenda
9

10. Confidence composition (CoCo) framework
10
Design-time phase Run-time phase
Safety requirement
“In next 30s, track pipe unless avoiding obstacles”
□30
( d ≥ 5 ∧ (d ≥ 30 → 10 ≤ y ≤ 50) )
Dynamics
NN controller
Environment
Perception
Model
Closed-loop
NN verification

11. Confidence in the
safety guarantees
Confidence composition (CoCo) framework
11
Safety requirement
“In next 30s, track pipe unless avoiding obstacles”
□30
( d ≥ 5 ∧ (d ≥ 30 → 10 ≤ y ≤ 50) )
Dynamics
NN controller
Environment
Perception
A1: “The obstacle is far enough
(d ≥ 30)”
A2: “Currently tracking the pipe
(10 ≤ y ≤ 50)”
A3: “Our observations are
consistent with the dynamics”
A4: “Perception noise is
within known bounds”
Model
Assumption confidence monitors
Obstacle monitor M1
Model invalidator M3
Pipe monitor M2
M4
c1
c2
c3
c4
C = f(M1, M2, M3, M4)
Confidence composition
“Safe”
Closed-loop
NN verification
Combined assumptions
(A1 → A2) ∧ A3 ∧ A4
Design-time phase Run-time phase
M2
M1
M3
M4

12. How to compose confidences?
12
C = f(M1, M2, M3, M4)
Composed confidence
M2
M1
M3
M4
A2
A1
A3
A4
A = (A1 → A2) ∧ A3 ∧ A4
Combined assumptions
Verification
Given: calibrated monitors
ECE(M1, A1) ≤ e 1
, …
Safety outcome
S ∈ { , } Goal: calibrate C to S
ECE(C, S) ≤ e
Theorem: the Goal is achieved if
C is calibrated to A: ECE(C, A) ≤ g(e)
Monitors have
- Unknown dependencies
- Idiosyncratic inaccuracies

13. ECE(M1*M2, A1∧A2) ≤ max[4e1
e2
, (Var[M1]*Var[M2])0.5
+ e1
+ e2
+ e1
e2
]
ECE(w1
*M1+w2
*M2, A1∧A2) ≤ max[e1
+ e2
+ e1
e2
, max[w1
, w2
] + e1
+ e2
− e1
e2
]
How to compose confidences?
13
C = f(M1, M2, M3, M4)
Composed confidence
M2
M1
M3
M4
A2
A1
A3
A4
A = (A1 → A2) ∧ A3 ∧ A4
Combined assumptions
Given: calibrated monitors
ECE(M1, A1) ≤ e 1
, …
Pr( )=?
Problem: conjunctive composition of ECEs
Given: ECE(M1, A1) ≤ e1
, ECE(M2, A2) ≤ e2
Find: f, ef
s.t. ECE(f(M1, M2), A1 ∧ A2) ≤ ef
Two data-free candidates for f:
Monitors have
- Unknown dependencies
- Idiosyncratic inaccuracies
Theorem: the Goal is achieved if
C is calibrated to A: ECE(C, A) ≤ g(e)
Two data-driven candidates: logistic regression, Bayesian estimation
ECE(C, S) ≤ e

14. - Key concepts: verification assumptions, confidence monitors
- CoCo: a confidence composition framework
- Confidence calibration bounds
- Experimental results for UUV
- Key concepts: verification assumptions, confidence monitors
- CoCo: a confidence composition framework
- Confidence calibration bounds
- Experimental results for UUV
Agenda
14

15. — Case study setup: 194 simulated UUV executions
○ Random independent violations of assumptions
— Composed confidence
○ Improves calibration to safety over individual monitors
Confidence composition is useful in practice
15
— Another case study: mountain car
○ Similar outcomes, see paper

16. I. Algorithmic confidence composition
— Synthesizing assumptions and monitors
— Composing confidence for arbitrary propositional formulas
— Minimizing ECE for a family of composition functions
II. More accurate and optimistic confidence
— Incorporating offline testing/simulation data and online predictions
III. Confidence-based recovery with probabilistic guarantees
This paper enables interesting future work
16

17. — CoCo framework for composing confidence monitors of verification assumptions
— First compositional bounds for calibration error
— Useful predictor of run-time safety, as seen in two case studies
Talk summary
17
A2
A1
A3
A4
(A1 → A2) ∧ A3 ∧ A4 f(M1, M2, M3, M4)
M2
M1
M3
M4
https://github.com/bisc/coco-case-studies

18. — [Pearl’88] Pearl. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann, 1988.
— [Scharf’91] Scharf. Statistical Signal Processing. Pearson, 1991.
— [Koller’09] Koller, Friedman, Bach. Probabilistic Graphical Models: Principles and Techniques. MIT Press, 2009.
— [Ranjan’10] Ranjan, Gneiting. Combining probability forecasts. In the Journal of the Royal Statistical Society, 2010.
— [Poor’13] Poor. An Introduction to Signal Detection and Estimation. Springer, 2013.
— [Sagi’18] Sagi, Rokach. Ensemble learning: A survey. In Wiley Interdisciplinary Reviews, 2018.
— [Ruchkin’20] Ruchkin, Sokolsky, Weimer, Hedaoo, Lee. Compositional Probabilistic Analysis of Temporal Properties Over
Stochastic Detectors. In EMSOFT, 2020.
— [Ruchkin’21] Ruchkin, Cleaveland, Sokolsky, Lee. Confidence Monitoring and Composition for Dynamic Assurance of
Learning-Enabled Autonomous Systems. In Formal Methods in Outer Space, 2021.
References
18