Is it time to declare a
verification war?

Brian Bailey
Email: brian_bailey@acm.org
Tel: 503 632 7448
Cell: 503 753 6040
W...
In the beginning
                            • The year app. 500 BC
                            • Sun Tzu published “on
  ...
Why was it so important
• Sun Tzu was the first to recognize the importance of
  positioning in strategy and that position...
Who is the Enemy?
         • When thinking about verification,
           the battle is being fought against
           Mu...
Sun Tzu’s 13 Chapters
1.    Laying Plans explores the five key elements that define competitive position (mission, climate...
The Key Essence
• Know the enemy
  – Understand the design and the best ways to
    approach the verification challenge
• ...
Outline
• Know yourself
  – I am going to revisit some fundamentals of
    verification that many forget
  – Take a look a...
Verification definition
   "Confirmation by examination and provisions of
     objective evidence that specified requireme...
Verification definition
• 4 Key phrases
  – Confirmation by examination
     • If you don’t look, then there is no hope of...
Verification fundamentals

                             Propagation success

                              Bug
           ...
Fundamentals - Coverage
• The extent or degree to which something is
  observed, analyzed, and reported.
                 ...
Structural Metrics
• These metrics are automatically extracted
  from the implementation source
    Easy to implement
    ...
Structural Coverage

                Propagation success

                    Bug
                                        ...
Structural Coverage

            Propagation success

         Shows if you have reached
             Bug
                ...
Structural metrics
• Path coverage
   This is the set of all combinations of all branches
   It identifies exhaustive exec...
Path Coverage

                Propagation success

                    Bug
                                              ...
Path Coverage

                Propagation success

                 Bug
                                                 ...
Structural coverage – dirty word
• When did structural coverage become a dirty
  word?
  – Directed tests target specific ...
Functional Coverage
• Identifies indicators of functionality
  –   A data value
  –   A specific state
  –   A sequence of...
Functional coverage
• It is a third independent model
  – Does not define the correct behaviors, just the behaviors
    th...
Adding Propagation
• Path Coverage
  – Already talked about this
• OCCOM    (Observability-based Code COverage Metrics )
 ...
OCCOM results




• Notes:
  – Shows the problems with coupling code coverage
    and random techniques
  – Shows that peo...
Summary of coverage methods

             Cost to     Cost to     Imp or     Completion     Objective   Benefits   Analysi...
Arm yourself with the best weapons
• A look at some new technologies

  – Functional Qualification from Certess (Now part ...
Mutation Analysis
• Similar to manufacturing test
  – Looks for a change in values seen on an output
  – Stuck-at faults: ...
Mutation analysis
• Performs complete stimulate and propagate
  analysis
  – Addresses --
     • If you don’t look it hasn...
Functional Qualification
• Based on mutation analysis
• Several differences:
  – Functional Qualification includes the det...
Some other differences
• For SW, mutation analysis was used to “cover” the
  program
• Functional qualification is being u...
Certitude Metrics - ST References
    Global Metric
          Representing the overall quality of the Verification Environ...
Case study 1 : 3rd Party - IP qualification
    •    Case study 1:                                              Activation...
Raising the abstraction
• Just as combinatorial equivalence checking
  allowed us to move from gate to RTL
  verification
...
Uses for SEC
• Untimed input in C/C++/SystemC
  – Ensure hand coded RTL functionally matches
• Synthesis verification
  – ...
A simple example
Always @ (posedge CLK) begin
  Out = A + B + C
end


          A       B       C                         ...
Commercialization
• Large pool of researchers
  – Shares many technologies with property checking
• Internally generated t...
Using the right weapon
• We have a number of different products in
  the verification portfolio
  – Each have very differe...
Thinking slightly differently
• But it is not just the application of the tools
  – It is also the application of technolo...
Intelligent Testbenches
• Two primary types
  – Graph based
  – Simulation based
• Graph based ones basically create a for...
Typical Questions That Must Be Answered
      for Successful Design Reuse
                                        Will thi...
Behavioral Indexing – Automated Extraction

                                        Behavioral Indexing System       Index...
Behavioral Indexing – Interactive Extraction
                                        Behavioral Indexing System


       P...
Behavioral Indexing Enables Many Capabilities
       Executable specification cross-correlated to RTL to facilitate dynami...
And More…




        Live annotation of transactions and design behaviors on the waveforms
        Live annotation of tra...
Check RTL Correctness Across Revisions

                             Automatic Implication Analysis

    Rev i            ...
Is it “The Art of Verification”




          Copyright © 2008-2009 Brian Bailey Consulting   44
Verification is Not Art
• We denigrate ourselves by calling it
• While we have not yet formalized and
  perfected the phil...
Thank You

Questions?


Email: brian_bailey@acm.org
Tel: 503 632 7448
Cell: 503 753 6040
Web: brianbailey.us




         ...
Upcoming SlideShare
Loading in...5
×

Schulz sv q2_2009

324

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
324
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Schulz sv q2_2009

  1. 1. Is it time to declare a verification war? Brian Bailey Email: brian_bailey@acm.org Tel: 503 632 7448 Cell: 503 753 6040 Web: brianbailey.us Copyright © 2008-2009 Brian Bailey Consulting
  2. 2. In the beginning • The year app. 500 BC • Sun Tzu published “on the art of war” 孫子兵法 • Since then it has been used for – Military strategies – Political – Business – Anything requiring tactics Copyright © 2008-2009 Brian Bailey Consulting 2
  3. 3. Why was it so important • Sun Tzu was the first to recognize the importance of positioning in strategy and that position is affected both by: – objective conditions in the physical environment – subjective opinions of competitive actors in that environment • He taught that strategy was not a to do list – requires quick and appropriate responses to changing conditions • Begins to sound like verification – Can we learn from Sun Tzu? Copyright © 2008-2009 Brian Bailey Consulting 3
  4. 4. Who is the Enemy? • When thinking about verification, the battle is being fought against Murphy the Design – Murphy is powerful and indiscriminate – He will show up anywhere and attempt to cause the maximum damage – He is also indestructible – Kill him and he shows up somewhere else – Is it possible to know he is dead – no we know he will never die • Why then do we bother fighting? – Because we are the verification heroes! Copyright © 2008-2009 Brian Bailey Consulting 4
  5. 5. Sun Tzu’s 13 Chapters 1. Laying Plans explores the five key elements that define competitive position (mission, climate, ground, leadership, and methods) and how to evaluate your competitive strengths against your competition. 2. Waging War explains how to understand the economic nature of competition and how success requires making the winning play, which in turn, requires limiting the cost of competition and conflict. 3. Attack by Stratagem defines the source of strength as unity, not size, and the five ingredients that you need to succeed in any competitive situation. 4. Tactical Dispositions explains the importance of defending existing positions until you can advance them and how you must recognize opportunities, not try to create them. 5. Energy explains the use of creativity and timing in building your competitive momentum. 6. Weak Points & Strong explains how your opportunities come from the openings in the environment caused by the relative weakness of your competitors in a given area. 7. Maneuvering explains the dangers of direct conflict and how to win those confrontations when they are forced upon you. 8. Variation in Tactics focuses on the need for flexibility in your responses. It explains how to respond to shifting circumstances successfully. 9. The Army on the March describes the different situations in which you find yourselves as you move into new competitive arenas and how to respond to them. Much of it focuses on evaluating the intentions of others. 10. Terrain looks at the three general areas of resistance (distance, dangers, and barriers) and the six types of ground positions that arise from them. Each of these six field positions offer certain advantages and disadvantages. 11. The Nine Situations describe nine common situations (or stages) in a competitive campaign, from scattering to deadly, and the specific focus you need to successfully navigate each of them. 12. The Attack by Fire explains the use of weapons generally and the use of the environment as a weapon specifically. It examines the five targets for attack, the five types of environmental attack, and the appropriate responses to such attack. 13. The Use of Spies focuses on the importance of developing good information sources, specifically the five types of sources and how to manage them. Copyright © 2008-2009 Brian Bailey Consulting 5
  6. 6. The Key Essence • Know the enemy – Understand the design and the best ways to approach the verification challenge • Know yourself – Understand (and improve) the process – Understand its strengths and weaknesses • Prepare yourself – Make sure the tools you use are those most likely to lead to success – Be flexible in the way that you use them • Use feedback – Based on objective observations Copyright © 2008-2009 Brian Bailey Consulting 6
  7. 7. Outline • Know yourself – I am going to revisit some fundamentals of verification that many forget – Take a look at coverage metrics – Primarily for the people newer to verification • Also a good reminder to more seasoned amongst us • Arm yourself with the best weapons – Some recent tools that help • Make verification more objective • Raise the level of abstraction • Preserve and use knowledge Copyright © 2008-2009 Brian Bailey Consulting 7
  8. 8. Verification definition "Confirmation by examination and provisions of objective evidence that specified requirements have been fulfilled." IEEE Definition of verification.  Verification is all about answering the question: Have we implemented something correctly ? Copyright © 2008-2009 Brian Bailey Consulting 8
  9. 9. Verification definition • 4 Key phrases – Confirmation by examination • If you don’t look, then there is no hope of finding incorrect behavior – provisions of objective evidence • Requires a second independent model of functionality – specified requirements • This introduces the needs for coverage to ensure the right things have been verified – have been fulfilled • Requires an act of verification to be associated with a coverage measurement Copyright © 2008-2009 Brian Bailey Consulting 9
  10. 10. Verification fundamentals Propagation success Bug Checker Stimulus 2. Propagate 3. Detect 1. Activate Design Propagation failure Remember: If you don’t look it hasn’t been verified If it hasn’t been verified against something objective, then it isn’t trustworthy Only place this happens is in the checker Copyright © 2008-2009 Brian Bailey Consulting 10
  11. 11. Fundamentals - Coverage • The extent or degree to which something is observed, analyzed, and reported. thefreedictionary.com – High coverage does not imply high quality • With the metrics that are in use today – While coverage metrics are objective (the information they provide is impassionate) • The decision about which to apply is subjective • The analysis of results is often subjective • Most coverage metrics do not provide “Confirmation by examination and provisions of objective evidence” – This include code coverage, functional coverage and almost all other metrics in use Copyright © 2008-2009 Brian Bailey Consulting 11
  12. 12. Structural Metrics • These metrics are automatically extracted from the implementation source Easy to implement Cannot identify what is missing • They tell you that something was ‘reached’ – A line of code – A decision point – A state They do not tell you that it was reached for the right reason They do not tell you that the right thing happened after that Copyright © 2008-2009 Brian Bailey Consulting 12
  13. 13. Structural Coverage Propagation success Bug Checker Stimulus 1. Activate Design Propagation failure Copyright © 2008-2009 Brian Bailey Consulting 13
  14. 14. Structural Coverage Propagation success Shows if you have reached Bug Checker every point in the implementation Stimulus where a bug may be 1. Activate Does not show you that the Design bug would have been detected Propagation failure Coverage != Verification Copyright © 2008-2009 Brian Bailey Consulting 14
  15. 15. Structural metrics • Path coverage This is the set of all combinations of all branches It identifies exhaustive execution of a model This is the closest form of coverage that relates to formal methods Still cannot identify missing functionality Expensive computationally – Limited path sets have been defined but not in use Does not identify data errors that do not affect control • Would not have identified Intel FP bug since this was related to values in a ROM Copyright © 2008-2009 Brian Bailey Consulting 15
  16. 16. Path Coverage Propagation success Bug Checker Stimulus 2. Propagate 1. Activate Design Propagation failure Copyright © 2008-2009 Brian Bailey Consulting 16
  17. 17. Path Coverage Propagation success Bug Checker Stimulus 2. Propagate 1. Activate Shows if you have executed all Design possible paths in the implementation Propagation failure Does not show you that the bug would have been detected Copyright © 2008-2009 Brian Bailey Consulting 17
  18. 18. Structural coverage – dirty word • When did structural coverage become a dirty word? – Directed tests target specific functionality – structural coverage was an impartial way to identify holes • Along came constrained/pseudo-random generation – No longer targeting specific functionality – Needed a way to ensure important functionality was executed • Thus functional coverage was born – There is nothing wrong with structural coverage coupled to directed testing • Unless used irresponsibly • Becoming less useful with increased concurrency Copyright © 2008-2009 Brian Bailey Consulting 18
  19. 19. Functional Coverage • Identifies indicators of functionality – A data value – A specific state – A sequence of states – etc It does not identify that the functionality is correct – Still expects that the comparison of two models is happening – Suffers from some of the same problems as code coverage Copyright © 2008-2009 Brian Bailey Consulting 19
  20. 20. Functional coverage • It is a third independent model – Does not define the correct behaviors, just the behaviors that should be detectable – Different language • Good and bad No way to define completion • Implementation is subjective • Closest theoretical model is path coverage – Expensive in terms of execution time • Higher cost than structural coverage An extra model to define Slows down simulator Analysis of holes is easier Functional Coverage != Verification Copyright © 2008-2009 Brian Bailey Consulting 20
  21. 21. Adding Propagation • Path Coverage – Already talked about this • OCCOM (Observability-based Code COverage Metrics ) – Srinivas Devadas Abhijit Ghosh Kurt Keutzer – DAC 1998 – Computes the probability that an effect of the fault would be propagated – During normal simulation – variables are tagged • Positive and negative tags are used – In a post processing step, these tags are used to compute the probabilities of propagation Copyright © 2008-2009 Brian Bailey Consulting 21
  22. 22. OCCOM results • Notes: – Shows the problems with coupling code coverage and random techniques – Shows that people creating directed tests consider propagation • This is a flaw with random methods – Some of the new intelligent testbench tools will make this a lot worse Copyright © 2008-2009 Brian Bailey Consulting 22
  23. 23. Summary of coverage methods Cost to Cost to Imp or Completion Objective Benefits Analysis implement execute spec Code Low Low Imp Low Yes Low Difficult Path Low Low Imp High Yes High Moderate Toggle Low Low Imp Medium Yes Medium Moderate Functional Medium Medium Spec ?* No Medium Easy Assertion High High Spec ?* No High Difficult * No direct way to measure today Copyright © 2008-2009 Brian Bailey Consulting 23
  24. 24. Arm yourself with the best weapons • A look at some new technologies – Functional Qualification from Certess (Now part of Springsoft) – Raising abstraction with Calypto – Behavioral Indexing from Jasper Copyright © 2008-2009 Brian Bailey Consulting 24
  25. 25. Mutation Analysis • Similar to manufacturing test – Looks for a change in values seen on an output – Stuck-at faults: what fault model is the equivalent for designer errors? • Mutation fault model based on two hypotheses: – that programmers or engineers write code that is close to being correct – that a test that distinguishes the good version from all its mutants is also sensitive to more complex errors – Potentially huge number of faults • Concept introduced in 1971 – First tool implementation in 1980 Copyright © 2008-2009 Brian Bailey Consulting 25
  26. 26. Mutation analysis • Performs complete stimulate and propagate analysis – Addresses -- • If you don’t look it hasn’t been verified – But not – • If it hasn’t been verified against something objective, then it isn’t trustworthy • This is the same as manufacturing test – It is not good enough to know that something was different – Must be able to detect that it was in error Copyright © 2008-2009 Brian Bailey Consulting 26
  27. 27. Functional Qualification • Based on mutation analysis • Several differences: – Functional Qualification includes the detection phase. For a fault to be detected there must be a check made so that at least one testcase fails – Functional qualification does not depend on propagation to a primary output. Directly supports white box assertions – Uses very different fault injections schemes to provide relevant results faster – Applied to hardware instead of software Copyright © 2008-2009 Brian Bailey Consulting 27
  28. 28. Some other differences • For SW, mutation analysis was used to “cover” the program • Functional qualification is being used as a quality measure for the testbench – First time verification engineers have a tool that allows them to objectively measure themselves • Statistical in nature – Technology advancements to improve performance – Do not have to run all faults or all testcases – Stop as soon as a testbench flaw has been revealed – Tackle the difficult problems first, and let the easy ones take case of themselves Copyright © 2008-2009 Brian Bailey Consulting 28
  29. 29. Certitude Metrics - ST References Global Metric Representing the overall quality of the Verification Environment ST reference : 75%, but usually higher Activation Score Measures the ability of the test suite to exercise all the RTL of the IP Similar to code coverage ST reference : 95%, & 100% explained Missing % should deeply studied & fixed or explained Propagation Score Measures the ability of the test suite to propagate mutations to the outputs of the IP ST reference : 80%, but should probably be enhanced by adding more test scenarios to reach 90% Detection Score Measures the ability of the environment to catch errors ST reference : 90%, but usually higher DAC’2008 - Anaheim Elevating Confidence in Design IP 29
  30. 30. Case study 1 : 3rd Party - IP qualification • Case study 1: Activation Score (A/F) • Application: 3rd party IP 95% IP ST Ref ST Avg 3rd Party IP Activation Score (A/F) 95% 97% 97% • HDL Directed Environment 90% Propagation Score (P/A) 80% 90% 80% 85% Global Metric (D/F) 75% 80% 66% • ~300 tests, 30 minutes 80% Detection Score (D/P) 90% 93% 85% 75% • Code Coverage ~100% 70% 65% Detection Score (D/P) 60% Propagation Score (P/A) • Challenges ST Ref • Convince 3rd Party IP provider ST Avg • High revenue, high visibility chip; 3rd Party IP reduce respin risk Global Metric (D/F) • Results • Helped us to push IP provider to improve verification environment • and monitor progress • Low detection score highlighted manual waveform checks DAC’2008 - Anaheim Elevating Confidence in Design IP 30
  31. 31. Raising the abstraction • Just as combinatorial equivalence checking allowed us to move from gate to RTL verification – Sequential equivalence checking will allow a migration to higher levels of abstraction • Higher simulation speeds • Enable longer runs • Use real scenarios (live data) Copyright © 2008-2009 Brian Bailey Consulting 31
  32. 32. Uses for SEC • Untimed input in C/C++/SystemC – Ensure hand coded RTL functionally matches • Synthesis verification – Ensure constraints, setup and options produced a valid result • Power optimization – Clock gating – Power optimizations Copyright © 2008-2009 Brian Bailey Consulting 32
  33. 33. A simple example Always @ (posedge CLK) begin Out = A + B + C end A B C A B C + + + CLK CLK Out Out Copyright © 2008-2009 Brian Bailey Consulting 33
  34. 34. Commercialization • Large pool of researchers – Shares many technologies with property checking • Internally generated tools – IBM • Esterel Studio – 2007 – Only within Esterel environment • Calypto introduced SLEC at DAC 2005 – Enables retimed RTL to be verified – Untimed to timed functional equivalency – Integrated with Mentor, Forte, Cadence synthesis products Copyright © 2008-2009 Brian Bailey Consulting 34
  35. 35. Using the right weapon • We have a number of different products in the verification portfolio – Each have very different characteristics – Each have different areas where they can excel – They also have weaknesses • Need to carefully match the tool to the situation – Verification planning helps in this regard – Requires thought upfront Copyright © 2008-2009 Brian Bailey Consulting 35
  36. 36. Thinking slightly differently • But it is not just the application of the tools – It is also the application of technology to make tools – For years formal methods were used to make formal verification tools • Seems obvious, but limited usefulness • Capacity constraints, etc. – Started to target technology to fit the problem • Bounded model checking • Semi-formal verification • Application of abstractions – Look for completely different applications • Accelerating design reuse (ActiveDesign) Copyright © 2008-2009 Brian Bailey Consulting 36
  37. 37. Intelligent Testbenches • Two primary types – Graph based – Simulation based • Graph based ones basically create a formal model of the system and use it to create stimulus paths that in turn exercise the design • Simulation based ones “learn” as they simulate so that they can get to an “objective” faster • Neither type of company wants to be called “formal” – but they both employ formal technologies • Now Jasper is adding a new paradigm for intelligence: – ActiveDesign with Behavioral Indexing Copyright © 2008-2009 Brian Bailey Consulting 37
  38. 38. Typical Questions That Must Be Answered for Successful Design Reuse Will this IP block operate correctly in my target use environment? What part of the RTL How do I correctly and pertains to feature XYZ? efficiently program these registers? If I change these lines of RTL, ANSWERS what features might break? What is the minimum latency of this path? Can this block be configured for my new Can this event application? ever happen? SATA: What is the latency PCIe: How do I program the through the data-link layer? number of links and lanes? •The design is the authority of what it can and cannot do! •Jasper’s ActiveDesign™ System extracts definitive answers - 38 - ©2008 Jasper Design Automation
  39. 39. Behavioral Indexing – Automated Extraction Behavioral Indexing System Indexed Database (Under Construction) Partial RTL, Legacy RTL, … RTL Design Information such as Auto-exploration such as • Inputs • FSM states • Outputs • FSM transitions • Flops • Coverage-driven • Counters, relevant values reachability analysis • FSMs, states •… •… - 39 - ©2008 Jasper Design Automation
  40. 40. Behavioral Indexing – Interactive Extraction Behavioral Indexing System Partial RTL, Legacy RTL, … Indexed Database (Under Construction) RTL Spec Captures and Analyzes Labels Automatically-Generated Behaviors Timing Diagrams, etc. Observes Meaningful Behaviors - 40 - ©2008 Jasper Design Automation
  41. 41. Behavioral Indexing Enables Many Capabilities Executable specification cross-correlated to RTL to facilitate dynamic analysis Executable specification cross-correlated to RTL to facilitate dynamic analysis of proposed uses and/or changes to the design of proposed uses and/or changes to the design Automatically-extracted waveforms for design behavior queries Automatically-extracted waveforms for design behavior queries to provide definitive answers to reuse questions to provide definitive answers to reuse questions -- without a testbench or simulator -- without a testbench or simulator Waveform manipulation - 41 - ©2008 Jasper Design Automation
  42. 42. And More… Live annotation of transactions and design behaviors on the waveforms Live annotation of transactions and design behaviors on the waveforms Implication analysis and re-verification of design modifications Implication analysis and re-verification of design modifications Accelerated firmware development through automatic generation of Accelerated firmware development through automatic generation of programming sequences programming sequences - 42 - ©2008 Jasper Design Automation
  43. 43. Check RTL Correctness Across Revisions Automatic Implication Analysis Rev i Rev i+1 Identical trace length (unaffected) Different trace length (potentially affected) Trace No Trace AD (broken) ActiveDesign - 43 - ©2008 Jasper Design Automation
  44. 44. Is it “The Art of Verification” Copyright © 2008-2009 Brian Bailey Consulting 44
  45. 45. Verification is Not Art • We denigrate ourselves by calling it • While we have not yet formalized and perfected the philosophy of verification – We do create highly adaptive strategies – We do use highly sophisticated tools – We do not believe that defeat is inevitable The last word goes to Sun Tzu: To subdue the enemy without fighting is the supreme excellence Copyright © 2008-2009 Brian Bailey Consulting 45
  46. 46. Thank You Questions? Email: brian_bailey@acm.org Tel: 503 632 7448 Cell: 503 753 6040 Web: brianbailey.us Copyright © 2008-2009 Brian Bailey Consulting
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×