Download to read offline
![79
Detector model as probability space
Given: mutually exclusive H1 and H0
Stochastic detector D – a triple (Ω, 𝓕, Pr):
– Ω: set of six possible outcomes for a pair of variables:
Detection Outcome (DO) ∈ {T, F, U}
Ground Truth (GT) ∈ {T, F}
– 𝓕: sigma-algebra of events over Ω (i.e., the powerset of Ω)
dot = (T, *), dof = (F, *), dou = (U, *)
gtt = (*, T), gtf = (*, F)
– Pr: 𝓕 → [0, 1], a probability measure over 𝓕
D T F U
T
dot dof dou
F
D T F U
T gtt
F gtf
D T F U
T o1
o2
o3
F o4
o5
o6](https://image.slidesharecdn.com/emsoft20-simple-200901155701/85/Compositional-Probabilistic-Analysis-of-Temporal-Properties-over-Stochastic-Detectors-79-320.jpg)
![82
Composition of detectors (ours)
● Operand detectors D1, D2 … DN
– A trace of value pairs (DO, GT) for each
●
LTL-like formula φ
– 3-value semantics [[[φ]]]
– 2-value semantics [[φ]]
●
Composite detector is (DO, GT) such that
– DO = [[[φ(DO1, … DON )]]]
– GT = [[φ(GT1, … GTN )]]](https://image.slidesharecdn.com/emsoft20-simple-200901155701/85/Compositional-Probabilistic-Analysis-of-Temporal-Properties-over-Stochastic-Detectors-82-320.jpg)
Uploaded byIvan Ruchkin
Compositional Probabilistic Analysis of Temporal Properties over Stochastic Detectors
monitor spec, detector model ● Trace collection: run missions, log detectors ● NCC estimation: - Vary #traces - Vary independence assumptions - Compare to BBC on full data ● Metrics: accuracy, computation time Evaluation of NCC estimates 66 Results: accuracy vs. #traces Evaluation of NCC estimates 67 Results: accuracy vs. assumptions Evaluation of NCC estimates 68 Conclusions ● NCC provides accurate error rate estimates with modest trace data ● Accuracy depends on independence assumptions ● Computation time scales linearly with formula size ● Mixed approach is promising for runtime verification
More Related Content
Similar to Compositional Probabilistic Analysis of Temporal Properties over Stochastic Detectors
![[DSC Croatia 22] Towards Classification Trustworthiness in Safety-Critical Au...](https://cdn.slidesharecdn.com/ss_thumbnails/dejanaugrenovicdsc22v3-220608091438-d25c74e1-thumbnail.jpg?width=640&height=640&fit=bounds)
![Tracking[1]](https://cdn.slidesharecdn.com/ss_thumbnails/tracking1-150428062953-conversion-gate02-thumbnail.jpg?width=640&height=640&fit=bounds){kind=tracking}
Compositional Probabilistic Analysis of Temporal Properties over Stochastic Detectors
- 1. 1 Compositional Probabilistic Analysis ofTemporal Properties over Stochastic Detectors Ivan Ruchkin, Oleg Sokolsky, Jim Weimer, Tushar Hedaoo, Insup Lee PRECISE Center Department of Computer and Information Science University of Pennsylvania The International Conference on Embedded Software (EMSOFT) September 22, 2020
- 2. 2 System example: Unmanned UnderwaterVehicle (UUV)
- 3. 3 System example: Unmanned UnderwaterVehicle (UUV)
- 4. 4 Forward sonar: obstacles Side sonar: pipeline GPS/IMU: pose Dopplerradar: seafloor System example: Unmanned Underwater Vehicle (UUV)
- 5. 5 Forward sonar: obstacles Side sonar: pipeline GPS/IMU: pose Dopplerradar: seafloor “The UVV should eventually re-discover the pipeline after avoiding an obstacle.” System example: Unmanned Underwater Vehicle (UUV)
- 6. 6 Motivating property “The UVVshould eventually re-discover the pipeline after avoiding an obstacle.”
- 7. 7 Motivating property “The UVVshould eventually re-discover the pipeline after avoiding an obstacle.” Run-time monitor
- 8. 8 Motivating property “The UVVshould eventually re-discover the pipeline after avoiding an obstacle.” Run-time monitor Detectors
- 9. 9 Motivating property “The UVVshould eventually re-discover the pipeline after avoiding an obstacle.” Temporal relation Run-time monitor Detectors
- 10. 10 Motivating property “The UVVshould eventually re-discover the pipeline after avoiding an obstacle.” Temporal relation Run-time monitor – What are the error rates of this property’s monitor given uncertain, unreliable detectors? Detectors
- 11. 11 Motivating property – Whatare the error rates of this property’s monitor given uncertain, unreliable detectors?
- 12. 12 Motivating property Data-driven ● Develop amonitor ● Sample & label monitor outputs ● Empirically estimate its error rate – What are the error rates of this property’s monitor given uncertain, unreliable detectors?
- 13. 13 Motivating property Data-driven ● Develop amonitor ● Sample & label monitor outputs ● Empirically estimate its error rate Pro: accurate Con: expensive data, effortful development – What are the error rates of this property’s monitor given uncertain, unreliable detectors?
- 14. 14 Motivating property Data-driven ● Develop amonitor ● Sample & label monitor outputs ● Empirically estimate its error rate Pro: accurate Con: expensive data, effortful development Model-based ● Specify a monitor ● Model the system ● Theoretically estimate the error rate – What are the error rates of this property’s monitor given uncertain, unreliable detectors?
- 15. 15 Motivating property Data-driven ● Develop amonitor ● Sample & label monitor outputs ● Empirically estimate its error rate Pro: accurate Con: expensive data, effortful development Model-based ● Specify a monitor ● Model the system ● Theoretically estimate the error rate Pro: no data needed Con: inaccurate, poor scalability – What are the error rates of this property’s monitor given uncertain, unreliable detectors?
- 16. 16 Motivating property Data-driven ● Develop amonitor ● Sample & label monitor outputs ● Empirically estimate its error rate Pro: accurate Con: expensive data, effortful development Model-based ● Specify a monitor ● Model the system ● Theoretically estimate the error rate Pro: no data needed Con: inaccurate, poor scalability Mixed ● Specify a monitor ● Sample & label detector outputs ● Estimate the error rate using the spec and detector data – What are the error rates of this property’s monitor given uncertain, unreliable detectors?
- 17. 17 Motivating property Data-driven ● Develop amonitor ● Sample & label monitor outputs ● Empirically estimate its error rate Pro: accurate Con: expensive data, effortful development Model-based ● Specify a monitor ● Model the system ● Theoretically estimate the error rate Pro: no data needed Con: inaccurate, poor scalability Mixed ● Specify a monitor ● Sample & label detector outputs ● Estimate the error rate using the spec and detector data Pro: cheap data, accurate, scalable, low effort – What are the error rates of this property’s monitor given uncertain, unreliable detectors?
- 18. 18 Approach outline Computational assistant Detector model Monitor’serror rate Detector data Monitor spec
- 19. 19 Approach outline Computational assistant Detector model Monitor’serror rate Detector data Monitor spec
- 20.
- 21. 21 Side sonar input Frontsonar input Obstacle det Our detector model Pipeline det Detection output Detection output
- 22. 22 Side sonar input Frontsonar input Obstacle det Our detector model Pipeline det Detection output Detection output Property monitor Violation
- 23. 23 Side sonar input Frontsonar input Obstacle det Our detector model Pipeline det Detection output Detection output Error Error Property monitor Error Violation
- 24. 24 Side sonar input Frontsonar input Obstacle det Our detector model Pipeline det Detection output Detection output Error Error Property monitor Error Violation P( ) = ?
- 25. 25 Side sonar input Frontsonar input Obstacle det Our detector model Pipeline det Detection output Detection output Obst. ground truth Pipe ground truth Error Error Property monitor Error Violation P( ) = ?
- 26. 26 Obstacle det Our detectormodel Pipeline det Detection output Detection output Error Error Property monitor Obst. ground truth Pipe ground truth Error Violation P( ) = ?
- 27. 27 D2 Our detector model D1 Detection output Detection output Error Error M Groundtruth Ground truth Error Violation P( ) = ?
- 28. 28 D2 Our detector model D1 Detection output Detection output Error Error M r.v.GT2 r.v. GT1 Ground truth Ground truth Error Violation P( ) = ?
- 29. 29 D2 Our detector model D1 Detection output Detection output Error Error M r.v.GT2 r.v. GT1 Ground truth Ground truth r.v. GT2 Values: GT2 = true↔gtt(D2 ) GT2 = false↔gtf(D2 ) Error Violation P( ) = ?
- 30. 30 D2 Our detector model D1 Detection output Detection output Error Error M r.v.DO2 r.v. DO1r.v. GT1 Ground truth Ground truth r.v. GT2 Values: GT2 = true↔gtt(D2 ) GT2 = false↔gtf(D2 ) Error Violation P( ) = ?
- 31. 31 D2 Our detector model D1 Detection output Detection output Error Error M r.v.DO2 Values: DO2 = true↔dot(D2 ) DO2 = false↔dof(D2 ) DO2 = unknown↔dou(D2 ) r.v. DO1r.v. GT1 Ground truth Ground truth r.v. GT2 Values: GT2 = true↔gtt(D2 ) GT2 = false↔gtf(D2 ) Error Violation P( ) = ?
- 32. 32 D2 Our detector model D1 Detection output Detection output Error Error M= φ(D1 , D2 ) r.v. DO2 Values: DO2 = true↔dot(D2 ) DO2 = false↔dof(D2 ) DO2 = unknown↔dou(D2 ) r.v. GT1 r.v. DO1 Ground truth Ground truth r.v. GT2 Values: GT2 = true↔gtt(D2 ) GT2 = false↔gtf(D2 ) Error Violation P( ) = ? DOM , GTM
- 33. 33 D2 Our detector model D1 Detection output Detection output Error Error Error M= φ(D1 , D2 ) Violation P( ) = ?r.v. GT2 Values: GT2 = true↔gtt(D2 ) GT2 = false↔gtf(D2 ) r.v. DO2 Values: DO2 = true↔dot(D2 ) DO2 = false↔dof(D2 ) DO2 = unknown↔dou(D2 ) r.v. GT1 FPR: Pr( dot(...) | gtf(...) ) FNR: Pr(¬dot(...) | gtt(...) ) DOM , GTM Ground truth Ground truth r.v. DO1
- 34. 34 Three-valued LTL fordetectors ● Syntax of LTL3d Semantics is given by detector compositions “The UUV should re-discover the pipeline within d seconds after losing it” Monitor for this property
- 35. 35 Three-valued LTL fordetectors ● Syntax of LTL3d ● Semantics is given by detector compositions “The UUV should re-discover the pipeline within d seconds after losing it” Monitor for this property
- 36. 36 Three-valued LTL fordetectors ● Syntax of LTL3d ● Semantics is given by detector compositions ● “The UUV should re-discover the pipeline within d seconds after losing it” Monitor for this property
- 37. 37 Three-valued LTL fordetectors ● Syntax of LTL3d ● Semantics is given by detector compositions ● “The UUV should re-discover the pipeline within d seconds after losing it” ● Monitor for this property
- 38. 38 Approach outline Computational assistant Detector model Monitor’serror rate Detector data Monitor spec
- 39. 39 Approach outline Computational assistant Detector model Monitor’serror rate Detector data Monitor spec
- 40. 40 Approach outline Computational assistant Detector model Monitor’serror rate Detector data Monitor spec
- 41. 41 Three estimators forerror rates
- 42. 42 Three estimators forerror rates ● Black-box calculation (BBC) – Inputs: labeled monitor traces – Baseline data-driven estimator Exact compositional calculation (ECC) Inputs: detector probabilities, independence assumptions, reasoning rules Noisy compositional calculation (NCC) Inputs: labeled detector traces, independence assumptions, reasoning rules
- 43. 43 Three estimators forerror rates ● Black-box calculation (BBC) – Inputs: labeled monitor traces – Baseline data-driven estimator ● Exact compositional calculation (ECC) – Inputs: detector probabilities, independence assumptions, reasoning rules Noisy compositional calculation (NCC) Inputs: labeled detector traces, independence assumptions, reasoning rules
- 44. 44 Three estimators forerror rates ● Black-box calculation (BBC) – Inputs: labeled monitor traces – Baseline data-driven estimator ● Exact compositional calculation (ECC) – Inputs: detector probabilities, independence assumptions, reasoning rules ● Noisy compositional calculation (NCC) – Inputs: labeled detector traces, independence assumptions, reasoning rules
- 45. 45 Three estimators forerror rates ● Black-box calculation (BBC) – Inputs: labeled monitor traces – Baseline data-driven estimator ● Exact compositional calculation (ECC) – Inputs: detector probabilities, independence assumptions, reasoning rules ● Noisy compositional calculation (NCC) – Inputs: labeled detector traces, independence assumptions, reasoning rules Computational assistant: https://github.com/bisc/prob-comp-asst
- 46. 46 ECC/NCC steps Step 1:rewrite with event predicates Step 2: reduce to known/ estimable probabilities Step 3: check if an independence holds Step 4: numerically estimate probabilities Step 5: calculate rate estimate
- 47. 47 ECC/NCC steps ● Step 1:simplify/rewrite with event predicates Step 2: reduce to known/ estimable probabilities Step 3: check if an independence holds Step 4: numerically estimate probabilities Step 5: calculate rate estimate
- 48. 48 ECC/NCC steps ● Step 1:simplify/rewrite with event predicates Step 2: reduce to known/ estimable probabilities Step 3: check if an independence holds Step 4: numerically estimate probabilities Step 5: calculate rate estimate
- 49. 49 ECC/NCC steps ● Step 1:simplify/rewrite with event predicates Step 2: reduce to known/ estimable probabilities Step 3: check if an independence holds Step 4: numerically estimate probabilities Step 5: calculate rate estimate
- 50. 50 ECC/NCC steps ● Step 1:simplify/rewrite with event predicates ● Step 2: reduce to known/ estimable probabilities Step 3: check if an independence holds Step 4: numerically estimate probabilities Step 5: calculate rate estimate
- 51. 51 ECC/NCC steps ● Step 1:simplify/rewrite with event predicates ● Step 2: reduce to known/ estimable probabilities Step 3: check if an independence holds Step 4: numerically estimate probabilities Step 5: calculate rate estimate
- 52. 52 ECC/NCC steps ● Step 1:simplify/rewrite with event predicates ● Step 2: reduce to known/ estimable probabilities ● Step 3: check if an independence holds Step 4: numerically estimate probabilities Step 5: calculate rate estimate
- 53. 53 Independence assumptions ● Assumption 1:detections determined by GTs ● Assumption 2: detection determined by its GT
- 54. 54 ECC/NCC steps ● Step 1:simplify/rewrite with event predicates ● Step 2: reduce to known/ estimable probabilities ● Step 3: check if an independence holds Step 4: numerically estimate probabilities Step 5: calculate rate estimate
- 55. 55 ECC/NCC steps ● Step 1:simplify/rewrite with event predicates ● Step 2: reduce to known/ estimable probabilities ● Step 3: check if an independence holds ● Step 4: numerically estimate probabilities Step 5: calculate rate estimate
- 56. 56 ECC/NCC steps ● Step 1:simplify/rewrite with event predicates ● Step 2: reduce to known/ estimable probabilities ● Step 3: check if an independence holds ● Step 4: numerically estimate probabilities Step 5: calculate rate estimate
- 57. 57 ECC/NCC steps ● Step 1:simplify/rewrite with event predicates ● Step 2: reduce to known/ estimable probabilities ● Step 3: check if an independence holds ● Step 4: numerically estimate probabilities Step 5: calculate rate estimate
- 58. 58 ECC/NCC steps ● Step 1:simplify/rewrite with event predicates ● Step 2: reduce to known/ estimable probabilities ● Step 3: check if an independence holds ● Step 4: numerically estimate probabilities ● Step 5: calculate the rate estimate
- 59. 59 Approach outline Computational assistant Detector model Monitor’serror rate Detector data Monitor spec
- 60. 60 Approach outline Computational assistant Detector model Monitor’serror rate Detector data Monitor spec
- 61. 61 Approach outline Computational assistant Detector model Monitor’serror rate Detector data Monitor spec
- 62. 62 Evaluation of NCCestimates
- 63. 63 Evaluation of NCCestimates ● How accurate are they? ● How are they affected by the quantity of trace data? ● How are they affected by the independence assumptions? ● What are their computational costs?
- 64. 64 ● How accurate arethey? ● How are they affected by the quantity of trace data? ● How are they affected by the independence assumptions? – Sensitive to accurate assumptions ● What are their computational costs? – 0-10 seconds, depending on formula size Evaluation of NCC estimates
- 65. 65 Evaluation setup ● Setup: UUVGazebo sim, randomized missions ● Goal: estimate error rates of two monitors ● 73 missions (total 7.7 hrs of sim time), each: – Two pairs of traces: (DO, GT) for pipe det & monitor https://github.com/uuvsimulator/uuv_simulator
- 66. 66 Evaluation variables ● Independent: – Missionconfiguration – Monitor formula; deadline d for pipe loss (sec) – Detector/monitor traces, for NCC/BBC resp. Dependent (error rate estimates): ECC: the true value based on exact probabilities NCC: our approach on cheap (detector) data BBC: data-driven on expensive (monitor) data
- 67. 67 Evaluation variables ● Independent: – Missionconfiguration – Monitor formula; deadline d for pipe loss (sec) – Detector/monitor traces, for NCC/BBC resp. ● Dependent (error rate estimates): – ECC: the true value based on exact probabilities – NCC: our approach on cheap (detector) data – BBC: data-driven on expensive (monitor) data
- 68. 68 ECC estimate (true,exact value) NCC estimate (ours, detector traces) BBC estimate (baseline, monitor traces) Interpretation: given enough data, the estimates are close
- 69. 69 NCC estimate (ours,detector traces) BBC estimate (baseline, monitor traces) Interpretation: less data favors NCC; more data favors BBC
- 70. 70 Summary Logical composition ofdetectors with LTL3d Probabilistic estimation of error rates Rule-based computational assistant Accuracy on par with data-driven estimates While using cheaper data & scalable analysis Preferred when little or no data available Computational assistant Detector model Monitor’s error rate Detector data Monitor spec
- 71. 71 Summary ● Logical compositionof detectors with LTL3d Probabilistic estimation of error rates Rule-based computational assistant Accuracy on par with data-driven estimates While using cheaper data & scalable analysis Preferred when little or no data available Computational assistant Detector model Monitor’s error rate Detector data Monitor spec
- 72. 72 Summary ● Logical compositionof detectors with LTL3d ● Probabilistic estimation of error rates – Rule-based computational assistant Accuracy on par with data-driven estimates While using cheaper data & scalable analysis Preferred when little or no data available Computational assistant Detector model Monitor’s error rate Detector data Monitor spec
- 73. 73 Summary ● Logical compositionof detectors with LTL3d ● Probabilistic estimation of error rates – Rule-based computational assistant ● Accuracy on par with data-driven estimates – While using cheaper data & scalable analysis Computational assistant Detector model Monitor’s error rate Detector data Monitor spec
- 74. 74 Refences ● The computational assistantand UUV case study data: https://github.com/bisc/prob-comp-asst ● The original paper: https://dx.doi.org/10.1109/TCAD.2020.3012643 ● Supplementary materials: https://www.researchgate.net/publication/342993188_Suppl ementary_Materials_for_Compositional_Probabilistic_Analy sis_of_Temporal_Properties_over_Stochastic_Detectors
- 75.
- 76. 76 Our framework atglance Inputs: – Set of detectors with event/error probabilities – Logical property over detectors – Detector independence assumptions – Labeled traces from detectors Method: a modeling & analysis framework based on – Logical composition model – Algebraic probability calculations – Axiomatic independence inference – Probability estimation from data Output: estimate of an error rate for the property monitor
- 77. 77 Our framework atglance
- 78. 78 Detector model asvariables ● Given: mutually exclusive H1 and H0 ● Atomic detector D is a pair (DO, GT) of r.v.s: – Detection outcome DO ∈ {T, F, U} – Ground truth GT ∈ {T, F} ● A probability space with marginal events: dot(D) = (T, *) gtt(D) = (*, T) dof(D) = (F, *) gtf(D) = (*, F) dou(D) = (U, *)
- 79. 79 Detector model asprobability space Given: mutually exclusive H1 and H0 Stochastic detector D – a triple (Ω, 𝓕, Pr): – Ω: set of six possible outcomes for a pair of variables: Detection Outcome (DO) ∈ {T, F, U} Ground Truth (GT) ∈ {T, F} – 𝓕: sigma-algebra of events over Ω (i.e., the powerset of Ω) dot = (T, *), dof = (F, *), dou = (U, *) gtt = (*, T), gtf = (*, F) – Pr: 𝓕 → [0, 1], a probability measure over 𝓕 D T F U T dot dof dou F D T F U T gtt F gtf D T F U T o1 o2 o3 F o4 o5 o6
- 80. 80 Detector examples ● Cointoss with faulty perception: H1 – heads, H0 – tails ● Detecting obstacles: H1 – obstacle ahead, H0 – no obstacle ● Perfect coin toss: H1 – heads, H0 – tails OD dot dof dou gtt 0.2 0.09 0.09 gtf 0.01 0.6 0.01 PCT dot dof dou gtt 0.5 0 0 gtf 0 0.5 0 FCT dot dof dou gtt 0.3 0.05 0.15 gtf 0.05 0.3 0.15
- 81. 81 Composition of detectors(general) ● Operand detectors D1, D2 … DN – A value pair (DO, GT) for each ● Composition operator op – 3-value version op3 – 2-value version op2 ● Composite detector is (DO, GT) such that – DO = op3(DO1, … DON ) – GT = op2(GT1, … GTN )
- 82. 82 Composition of detectors(ours) ● Operand detectors D1, D2 … DN – A trace of value pairs (DO, GT) for each ● LTL-like formula φ – 3-value semantics [[[φ]]] – 2-value semantics [[φ]] ● Composite detector is (DO, GT) such that – DO = [[[φ(DO1, … DON )]]] – GT = [[φ(GT1, … GTN )]]
- 83.
- 84. 84 Strong negation ● Flips hypotheses/outcomes DCH1 CH0 NC H1 0.3 0.05 0.15 H0 0.05 0.3 0.15 ¬s D CH1 CH0 NC H1 0.05 0.3 0.15 H0 0.3 0.05 0.15
- 85. 85 Weak negation ● Flips hypotheses,true on unknown D CH1 CH0 NC H1 0.3 0.05 0.15 H0 0.05 0.3 0.15 ¬w D CH1 CH0 NC H1 0.45 0.05 0 H0 0.2 0.3 0
- 86. 86 Strong exclusive negation ● Flipshypotheses, false on unknown D CH1 CH0 NC H1 0.3 0.05 0.15 H0 0.05 0.3 0.15 ¬se D CH1 CH0 NC H1 0.3 0.2 0 H0 0.05 0.45 0
- 87. 87 Binary composition example Twoindependent faulty coin tosses FCT1 Λ FCT2 dot dof dou gtt 0.09 0.0475 0.1125 gtf 0.0325 0.53 0.1875 FCT1 dot dof dou gtt 0.3 0.05 0.15 gtf 0.05 0.3 0.15 FCT2 ch1 dof dou gtt 0.3 0.05 0.15 gtf 0.05 0.3 0.15 FCT1 V FCT2 dot dof dou gtt 0.53 0.0325 0.1875 gtf 0.0475 0.09 0.1125
- 88. 88 Negations, implications, modalities Strong negation:¬s D Weak negation: ¬w D Strong exclusive negation: ¬se D Strong implication: D1→s D2 := ¬s D1 V D2 Weak implication: D1 →w D2 := ¬w D1 V D2 Strong exclusive implication: D1 →se D2 := ¬se D1 V D2 Box modality: Diamond modality:
- 89.
- 90. 90 Error rates ofdetectors ● False positive rate: ● False negative rate:
- 91. 91 Error rates forexamples ● Coin toss with faulty perception: H1 – heads, H0 – tails ● Detecting obstacles: H1 – obstacle ahead, H0 – no obstacle ● Perfect coin toss: H1 – heads, H0 – tails FCT dot dof dou gtt 0.3 0.05 0.15 gtf 0.05 0.3 0.15 DO dot dof dou gtt 0.2 0.09 0.09 gtf 0.01 0.6 0.01 PCT dot dof dou gtt 0.5 0 0 gtf 0 0.5 0 fpr = 0.05 fnr = 0.2 fpr = 0.01 fnr = 0.18 fpr = 0 fnr = 0
- 92. 92 Error rates forcomposition example Two independent coin tosses with faulty perception CT1 Λ CT2 dot dof dou gtt 0.09 0.0475 0.1125 gtf 0.0325 0.53 0.1875 CT1 dot dof dou gtt 0.3 0.05 0.15 gtf 0.05 0.3 0.15 CT2 dot dof dou gtt 0.3 0.05 0.15 gtf 0.05 0.3 0.15 CT1 V CT2 dot dof dou gtt 0.53 0.0325 0.1875 gtf 0.0475 0.09 0.1125 fpr = 0.05, fnr = 0.2 fpr = 0.0325, fnr = 0.155 fpr = 0.0475, fnr = 0.22
- 93. 93 Reasoning rules ● 𝓡log– tautologies of LTL3d ● 𝓡ev – tautologies of event predicates ● 𝓡prob – algebraic probability rules ● 𝓡indep – conditional independence rules
- 94. 94 Related work: combining detectors ● Ensemblesand cross-validation – Good performance, weak guarantees ● Hypothesis tests – Either too simple or too complex ● Logics of probability (probability operator) – No need for explicit statements about probability ● Probabilistic logics (probability of formulas) – No need for uncertain statements
- 95. 95 Related work: predictingerror rates ● Model-free – E.g., estimation from data – Requires extensive data collection & labeling – Difficulty in estimating rare cases ● Model-based – E.g., model checking probabilistic automata – Requires comprehensive (~ inaccurate) modeling – Limited scalability