May 1, 2013Metric Driven Formal VerificationRaik BrinkmannOneSpin Solutions
May 1, 2013OneSpin Solutions - Who we are…OneSpin providesenduring solutions thatenable the mostthorough & easiest to usel...
May 1, 2013Formal Verification with OneSpin 360• Unique push button observation coverage analysis for verification progres...
May 1, 2013OutlineImportance of Progress MetricsCoverage MetricsControl vs. Observation CoveragePractical Observation Cove...
May 1, 2013Why do we need a metric?DUVTests / Scenarios CheckersBus FunctionalModel (BFM)Test #1Test #2Stimulus / Scenario...
May 1, 2013Standard Formal ABVProgress of Formal Verification?• Unclear how many assertions to write and where to put them...
May 1, 2013Progress Metric & Coverage TaxonomyCoverageStructural FunctionalProgress MetricBug RateCode Circuit• Line/State...
May 1, 2013Control / Simulation Coverage:• Focused on quality of stimuli• How to judge quality of checkers?Observation & C...
May 1, 2013Been there? Done what?• Has the statement been reached?• Idea:– If a statement has not been reachedduring verif...
May 1, 2013Control Coverage for FormalNo explicit stimulus generation for formalFormal considers all possible input stimul...
May 1, 2013(Formal) Observation CoverageVerification tries to answer the question:• “Does a design satisfy a specification...
May 1, 2013Practical Observation CoverageMutation coverage• Particular changes depending on error model, usually static• R...
May 1, 2013always @(posedge clk or posedge reset)if (reset)z <= 1’b0;elsebegincase (i)3b001: z <= a;3b010: z <= b;3b100: z...
May 1, 2013Merging Observation and ControlObservation Coverage Control Coverage Merged Resultobserved reached1) coveredunk...
May 1, 2013Interfacing with UCISUCIS• Coverage standard evolved from Mentor Graphics‘ UCDB• Only supports control coverage...
May 1, 2013Simple Quantify MDV Example• Mixed control and observationcoverage results indicate:– holes indicating missinga...
May 1, 2013Industry ExamplesDesign LOCs #Assertions #Locations RuntimeFIFO 321 30 21 100sFSM-DDR2-Read 839 6 93 106svCore-...
May 1, 2013SummaryCoverage is an important aspect of verificationNew metric introduced: “Quantify MDV”• Combines observati...
May 1, 2013Thank You!
May 1, 2013What about redundancy?Redundant code cannot be observed!• Redundant code will be marked uncovered if not treate...
May 1, 2013Functional vs. StructuralCoverage Type Functional StructuralControl How much of the specified behaviorhas been ...
May 1, 2013Verification Environment with Exhaustive Coverage AnalysisDUVTests / Scenarios CheckersSpecVerification Environ...
May 1, 2013What does block level coverage meanin the larger context?System-leveldesignBlock-leveldesign (RTL)Architecture-...
Upcoming SlideShare
Loading in …5
×

TRACK H: Formal metric driven verification/ Raik Brinkmann

1,304 views
1,190 views

Published on

Published in: Technology, Design
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,304
On SlideShare
0
From Embeds
0
Number of Embeds
763
Actions
Shares
0
Downloads
9
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

TRACK H: Formal metric driven verification/ Raik Brinkmann

  1. 1. May 1, 2013Metric Driven Formal VerificationRaik BrinkmannOneSpin Solutions
  2. 2. May 1, 2013OneSpin Solutions - Who we are…OneSpin providesenduring solutions thatenable the mostthorough & easiest to uselogic verification availableOneSpin Solutions was spunout of Infineon in 2005Technologies includeAssertion-basedverification (ABV) andequivalence checking (EC)Production-proventools used inthousands of designsThe company is a completelyindependent entity owned byAzini Capital and OneSpin’semployeesThe original technologies havebeen augmented & enhancedto provide the most thoroughformal capabilities availableEngineering HQ basedin Munich, GermanySales offices in US,France, Germany,Israel, Japan, andTurkey
  3. 3. May 1, 2013Formal Verification with OneSpin 360• Unique push button observation coverage analysis for verification progress• Faster assertion development in timing diagram style using transaction level assertions• 100% functional coverage using unique gap free verification• Unique structural assertion debugger and active value/driver tracing through RTL• Incremental compilation of assertions for quick turn around• Automatic clock and reset detection for easy design setupUnique productivity features• HDL-linting• Structural assertion synthesis• Coverage closure for simulation• SoC connectivity verification• Register verification• Formal score boarding• Protocol verification IP• X-propagation verificationLarge selection of push-button design verification solutions, included• Verification of logic, sequential and power optimizations• Supporting RTL and gate level for FPGA and ASIC• Integrated with major vendors and many synthesis flowsPush-button equivalence checking
  4. 4. May 1, 2013OutlineImportance of Progress MetricsCoverage MetricsControl vs. Observation CoveragePractical Observation CoverageCombining Control and Observation CoverageExamplesSummary
  5. 5. May 1, 2013Why do we need a metric?DUVTests / Scenarios CheckersBus FunctionalModel (BFM)Test #1Test #2Stimulus / Scenario GenerationConstraintsFormalEngineStimulus / Scenario ExclusionCheck #1Check #2Check #3Check #4SimulationFormalVerification Environment (simplified):Verification Process:• Plan: Verification planning• Do: Write tests / properties and verify and fix DUV, tests, properties• Check: Are we done, (still) making progress?• Act: Adapt planning or get ready for tape-outProgress Metric : Key Component of Verification FlowPlan DoCheckAct
  6. 6. May 1, 2013Standard Formal ABVProgress of Formal Verification?• Unclear how many assertions to write and where to put them• Unclear verification progress• Unclear how formal verification affects overall verification qualityIntegration with Simulation?• Weak integration into verification flow and sign-off• Additional effort to existing simulation effort• Formal used as “point tool” for verifying specific aspects (e.g. hard to testcorner cases) Low acceptance of formal ABVWe need a practical progress metric!
  7. 7. May 1, 2013Progress Metric & Coverage TaxonomyCoverageStructural FunctionalProgress MetricBug RateCode Circuit• Line/Statement/Block• Expression/Branch/Condition• FSM• Toggle• LatchScenariosImplementationartefactsAnalyticalBugsFixed100%timeok#AssertionComplete• Gap Free Analysis • Verification PlanAnecdotalHave I written enough tests / properties?How much has been verified?Where are the gaps in my verification?Are we done and ready for tape-out?if (…)…if (…)else …
  8. 8. May 1, 2013Control / Simulation Coverage:• Focused on quality of stimuli• How to judge quality of checkers?Observation & Control Coveragecoverage metricsstimulus generation checkers/assertionsplanDUVreportHow good are mytest vectors &constraints?How good are mycheckers andassertions?How much of my DUV is verified?When is my verification finished?Observation Coverage:• Focused on quality of checkers• Exposes unverified DUV parts
  9. 9. May 1, 2013Been there? Done what?• Has the statement been reached?• Idea:– If a statement has not been reachedduring verification, it can’t break acheck.– If a statement has been reached, wouldsome check fail?• Can measure quality of stimuli.case (state)…burst:if (cancel_i)done_o <= 1…activecase (state)…burst:if (cancel_i)done_o <= X…mutate• Has the statement been verified?• Idea:If a statement is modified, some checkshould fail.But would some check fail, if thestatement cannot be reached?• Can measure quality of checkers.Been there! Done that!Example: Statement CoverageCoverageControl ObservationBeen there! Done that!
  10. 10. May 1, 2013Control Coverage for FormalNo explicit stimulus generation for formalFormal considers all possible input stimuli by default (exhaustive verification)However: Control coverage still an issue because• Formal requires constraints excluding illegal environment behavior• Constraints may accidentally be too strong and exclude vital functionality• Constrained functionality can neither be controlled nor observed(Structural) control coveragequantifies degree to whichlocations of a design have beenactivated during verification.controlcontrollablereachedunreachedunknownuncontrollabledeadconstrained
  11. 11. May 1, 2013(Formal) Observation CoverageVerification tries to answer the question:• “Does a design satisfy a specification?”(Observation) coverage tries to answer the question:• “What causes a design to satisfy a specification?”Causality:• “When do we say A is a cause of B?”Common approach: Counterfactual causality:• “A is a cause of B if, had A not happened, then B would not have happened.”• A code line is a cause of a design satisfying an assertion. Source: H. Chockler, IBM(Structural) observationcoverage quantifies degree towhich locations of a designhave been responsible to makechecks pass.observationunknownobservableobservedunobservedunobservable
  12. 12. May 1, 2013Practical Observation CoverageMutation coverage• Particular changes depending on error model, usually static• Replacing for example “a <= b;” with “a <= 1´b1;”• Problems:• Can lead to vacuously holding assertions• Requires several modifications at each location, high run time“Quantify MDV” metric• Use abstraction by free variables, allow dynamic behavioral change• Replacing for example “a <= b;” with “a <= free_b;”• Advantages:• No unintended vacuity• One modification for each location leads to faster results• Multiple locations can be checked at the same time to prove “unobserved”
  13. 13. May 1, 2013always @(posedge clk or posedge reset)if (reset)z <= 1’b0;elsebegincase (i)3b001: z <= a;3b010: z <= b;3b100: z <= c;default: z <= <input>;endcaseendM5always @(posedge clk or posedge reset)if (reset)z <= 1’b0;elsebegincase (i)3b001: z <= a;3b010: z <= b;3b100: z <= <input>;default: z <= 1b1;endcaseendM4always @(posedge clk or posedge reset)if (reset)z <= 1’b0;elsebegincase (i)3b001: z <= a;3b010: z <= <input>;3b100: z <= c;default: z <= 1b1;endcaseendM3Formal Observation Coverage Examplemodule select1(onehot, a, b, c, z, clk, reset);input clk;input reset;input [2:0] i;input a;input b;input c;output reg z;always @(posedge clk or posedge reset)if (reset)z <= 1b0; // L1: not covered (reset case)elsebegincase (i)3b001: z <= a; // L2: covered by assertion3b010: z <= b; // L3: not covered3b100: z <= c; // L4: not covereddefault: z <= 1b1; // L5: not coveredendcaseend// if there is no reset, then a is stored in z if ‘i is 3b001A: assert property( @(posedge clk)disable iff (reset)i == 3b001 |=> z == $past(a));endmoduleQ: Which assignment locations Lx in designM are observed by proven assertion A?2. Re-Check property A for each M1..M5always @(posedge clk or posedge reset)if (reset)z <= <input>;elsebegincase (i)3b001: z <= a;3b010: z <= b;3b100: z <= c;default: z <= 1b1;endcaseendM1always @(posedge clk or posedge reset)if (reset)z <= 1’b0;elsebegincase (i)3b001: z <= <input>;3b010: z <= b;3b100: z <= c;default: z <= 1b1;endcaseendM2Assertion A holds on M1:L1 not observedAssertion A fails on M2: L2is observedMAL3 not observedL4 not observedL5 not observed1. Modify each location L1..L5of M: Producing M1..M5A: The locations Lx for which A fails afterreplacing the assignment with a free input.
  14. 14. May 1, 2013Merging Observation and ControlObservation Coverage Control Coverage Merged Resultobserved reached1) coveredunknown reached reachedunobserved reached unobservedunknown unreached unreachedunobserved unreached uncoveredunobservable2) constrained constrainedunobservable2) dead dead1. If a location is uncontrollable, it is also unobservable.2. If a location is observed, it is also reached and thus controllable.observationunknownobservableobservedunobservedunobservablecontrolcontrollablereachedunreachedunknownuncontrollabledeadconstrained2.1.Been there! Done that!
  15. 15. May 1, 2013Interfacing with UCISUCIS• Coverage standard evolved from Mentor Graphics‘ UCDB• Only supports control coverage, but not observation coverageHow to interface?• Use user defined structures to store Quantify MDV results• Use result merging to enhance standard UCIS coverageResult Merging from UCIS to Quantify MDV:• Covered bins -> Reached locationsMerging Quantify MDV to UCIS:• Dead/Constrained -> Coverage Excludes• Reached locations -> Covered bins• Covered locations -> Covered binsConclusions on UCIS interfacing• Merging results with UCIS is possible and useful now• Direct support for observation coverage by UCIS is desirable
  16. 16. May 1, 2013Simple Quantify MDV Example• Mixed control and observationcoverage results indicate:– holes indicating missingassertions– constrained code indicatingover-constrainingverificationholeverifiedcodeconstrainedcodedeadcodeManagementoverview
  17. 17. May 1, 2013Industry ExamplesDesign LOCs #Assertions #Locations RuntimeFIFO 321 30 21 100sFSM-DDR2-Read 839 6 93 106svCore-Processor 295 8 86 204sSQRT 383 2 35 257sIFX-Aurix-1*) 25563 85 2316 4dIFX-Aurix-2 27374 157 1993 5dIFX-Aurix-3 57253 253 5309 7d**)*) “Quantification of Formal Properties for Productive Automotive Microcontroller Verification”, HolgerBusch, DVCon, San Jose - February 26, 2013**) interrupted at 80% completion
  18. 18. May 1, 2013SummaryCoverage is an important aspect of verificationNew metric introduced: “Quantify MDV”• Combines observation and control coverage• Agnostic to assertion style, assertion language, design style and designlanguage• Scales to relevant big industry designs• Useful for• Quick feedback on formal verification quality, as well as• Criterion for (formal) verification closure• Can be combined with simulation based verification and metrics
  19. 19. May 1, 2013Thank You!
  20. 20. May 1, 2013What about redundancy?Redundant code cannot be observed!• Redundant code will be marked uncovered if not treated specially.What are sources of redundancy?• Redundant logic not driving any outputs• Safety logic meant to implement fail safe devicesHow to mitigate?• Identify redundant code and mark it• Remove redundant code before running coverage• Selectively activate/de-activate redundant parts
  21. 21. May 1, 2013Functional vs. StructuralCoverage Type Functional StructuralControl How much of the specified behaviorhas been activated?How much of the DUV implementation hasbeen activated?Observation How much of the specified behaviorhas been verified?How much of the DUV implementation hasbeen verified?Functional vs. Structural• Functional - relates to specification– Measures completeness of requirements verification– Can identify gaps in verification plan• Structural - relates to DUV– Measures completeness of design intent verification– Can identify unverified parts of DUVControl vs. Observation• Control - relates to activation– Measures quality of stimuli– Can identify unreachable code / over-constraining• Observation - relates to causality– Measures quality of checkers– Can identify verification gaps• Control coverage doesn’t imply any observation coverage.• Observation coverage doesn’t imply any functional coverage.• 100% observation coverage implies 100% control coverage.• 100% Functional coverage + all assertions proven implies 100% observation coverage (if DUV contains no deadcode, no constrained code, no redundant code)
  22. 22. May 1, 2013Verification Environment with Exhaustive Coverage AnalysisDUVTests / Scenarios CheckersSpecVerification EnvironmentVerificationPlanCoverageAnalysisControlCoverageObservationCoverageFunctionalCoverageHave I writtenenough stimuli?Which parts ofmy DUV havebeen exercised?Which parts of myDUV have beenchecked?Did I writeenough checks?Are all specifiedfunctionsimplemented?Are all specifiedfunctions verified?FunctionalStructural
  23. 23. May 1, 2013What does block level coverage meanin the larger context?System-leveldesignBlock-leveldesign (RTL)Architecture-leveldesignDesign startVerification closureSystem-levelsimulationBlock-levelsimulationSub-system-levelsimulationFormal ABV withCovage AnalysisUCDBAggregation of module / blocklevel coverage results with higherlevel contextHigh-Level/RTL Design & Verification Flow

×