Lacey coverage dallas-june20_2006
Upcoming SlideShare
Loading in...5
×
 

Lacey coverage dallas-june20_2006

on

  • 441 views

 

Statistics

Views

Total Views
441
Views on SlideShare
441
Embed Views
0

Actions

Likes
0
Downloads
9
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Lacey coverage dallas-june20_2006 Lacey coverage dallas-june20_2006 Presentation Transcript

  • What’s with all this talk about coverage? David Lacey and Rob Porter Hewlett Packard Company June 20, 2006 © 2006 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice
  • “You have this awesome generation that pseudo-randomly creates all sorts of good scenarios. You also have created equally awesome scoreboard and temporal checker infrastructure that will catch all the bugs. Next, you run it like mad, with all sorts of seeds to hit as much of the verification space as possible.” Peet James Verification Plans 30 September 2 2006
  • Given all that, what really happened? • Where did all those transaction go? • Which lines of RTL were exercised? • Which sections of the specification were tested? • Which corner cases of my implementation were hit? • What was the distribution of transaction types issues? • Do I need to create new tests? • Can I stop running simulations? Coverage helps provides the Coverage is a pieceanswers! the final answer of the puzzle, not 30 September 3 2006
  • Coverage provides… • An understanding of which portions of the design have been exercised • Increased observability of simulation behavior • Feedback on which tests are effective • Feedback to direct future verification efforts 30 September 4 2006
  • Agenda • Coverage terms and tools • How to get started • Coverage planning • Coverage execution • Coverage analysis • Coverage results 30 September 5 2006
  • Coverage terms and tools 30 September 6 2006
  • Coverage terms • Coverage strategy − Approach defined to utilize coverage technology − Generate, gather, and analyze coverage data • Coverage model − Collection of coverage spaces − Definition of one or more coverage spaces of interest • Coverage space − Set of coverage points associated with a single aspect of the design and a single coverage technology • Coverage technology − Specific mechanism such as code, functional, assertion, transaction • Coverage point − A specific named aspect of the design behavior − FCP, line of code, state transition, transaction or sequence of transactions • Coverage data − Raw data collected from all coverage points and coverage spaces • Coverage results − Interpretation of coverage data in context of coverage model 30 September 7 2006
  • Coverage model vs. coverage tools • WHAT −Coverage model High Level Architecture Specification Bug rates Transaction Sim cycles Coverage Functional Model Code Assertion Design detail Low Level • HOW Coverage tools Coverage tools 30 September 8 2006
  • Code coverage • Line/block, branch, path, expression, state • Measures controllability aspect of our stimulus − i.e. What lines of code have we exercised • Does not connect us to the actual functionality of the chip − No insight into functional correctness • Takes a blind approach to coverage (low observability) − Activating an erroneous statement does not mean the error will propagate to an observable point during the course of a simulation • Generates a lot of data − Difficult to interpret what is significant and what is not 30 September 9 2006
  • Assertion coverage Assertions monitor and report undesirable behavior • Ensures that preconditions of an assertion check have been met // SVA: if asserting stop or flush, no new request assert property (@(posedge clk) disable iff (rst_n) ((Flush | SMQueStop) |-> SMQueNew)) else $error(“Illegal behavior”); precondition check 30 September 10 2006
  • Functional coverage • Similar in nature to assertions Assertions monitor and report undesirable behavior Functional coverage monitors and reports desirable behavior • Functional coverage − Specific design details − Corner cases of interest to engineers − Architectural features 30 September 11 2006
  • Transaction coverage • A transaction is the logging of any data structure − A packet on a bus − Does not have to be a system packet • Example transaction coverage points − All transaction types were seen on each interface − Transactions with specific data were seen • Source, destination, address, address ranges • Sequences of transactions − Have recording monitor watch for sequence − Implement advanced queries to look for sequence • Two parts to transaction coverage − Record the right data − Correct queries 30 September 12 2006
  • EDA tools • Code, FCPs and transactions are recorded into vendor specific databases − Tools are provided to look at coverage data − Report engines provide text reports • Debug tools for FCPs and assertions • Tools to encourage coverage-driven methodologies • Coverage is still a young technology − Tools still expanding set of capabilities − Development areas such as data aggregation, multiple view extraction 30 September 13 2006
  • How do I get started with this coverage stuff? 30 September 14 2006
  • Coverage roadmap – getting started Planning PSL / SVA / 1. Choose Code specification 4. Collect OVL form data FCP Txn 2. Identify Spec, design coverage 5. Analyze Coverage tools model data Consumption Code / 3. Implement Assertion / FCP coverage 6. React to / Txn Adjust stimulus model data Execution 30 September 15 2006
  • Coverage planning PSL / SVA / Planning 1. Choose Code OVL specification 4. Collect form data FCP Txn 2. Identify Spec, design coverage 5. Analyze Coverage tools model data Consumption Code / 3. Implement Assertion / FCP coverage 6. React to Adjust stimulus / Txn model data Execution 30 September 16 2006
  • Coverage planning Start looking at coverage up front! Coverage results only as good as coverage model • Identify content of the coverage model − Coverage types to be used • Identify required tools • Coverage infrastructure • Coverage execution • Maintenance and tool enhancements • Define coverage goals and metrics • Coverage reviews 30 September 17 2006
  • Who, what, when, where, why • Who creates coverage model? − Who analyses the data? Logic and DV engineers − Who owns coverage? • What to cover in the model? Concern areas • When to add coverage points? Add with RTL − When to analyze coverage data? Analyze continuously spec, design, assertions, • Where to look for ideas? test plan • Why mess with coverage? Because… 30 September 18 2006
  • For FCPs, ask yourself… −What should be covered? −Where is best place to put FCPs? −When to look for condition? −Why have coverage point? 30 September 19 2006
  • Watch out for… • Too much data − Need information, not data − Need supporting tools to get correct views of data • Ineffective use of coverage − FCPs that fire every clock cycle − Duplication of coverage with different tools • Reading too much into grading tests − Random tests produce different results with different seeds 30 September 20 2006
  • Cost of coverage • Plan for the costs of using coverage − Get solid infrastructure setup − Plan for slower simulations • Some level of cost is acceptable − Getting value back for investment • Be smart − Architect coverage plan up front to ensure success 30 September 21 2006
  • Coverage execution PSL / SVA / OVL Planning 1. Choose specificatio 4. Collect Code n form data FCP Txn 2. Identify Spec, design coverage 5. Analyze Coverage tools model data Consumption Code / 3. Assertion / Implement coverage 6. React to Adjust stimulus FCP / Txn model data Execution 30 September 22 2006
  • Describing coverage model • Code coverage − RTL code, pragmas • Assertion and functional coverage − Use assertion language or library (PSL, SVA, OVL) • Transaction − Use hooks into Transaction Level Modeling // SVA cover example // PSL cover example always @(posedge clk) begin default clock = (posedge clk); if (reset_n) sequence qFullCondition = myQfull: cover (q_full) {reset_n ? (q_full : 1’b0); $info (“queue was full”); cover qFullCondition; end 30 September 23 2006
  • Data collection • Collect data across volume simulation • Aggregate multiple databases • Location of coverage data repository • Manage volume of data 30 September 24 2006
  • Coverage analysis Planning PSL / SVA / 1. Choose Code OVL specification 4. Collect form data FCP Txn 2. Identify Spec, design coverage 5. Analyze Coverage tools model data Consumption Code / 3. Implement Assertion / coverage 6. React to Adjust stimulus FCP / Txn model data Execution 30 September 25 2006
  • The analysis • Easy to generate a ton of data − Want information, not data Need to organize the data Can’t look at it all at once Determine views needed 30 September 26 2006
  • Views of coverage data • Un-hit coverage • Functionality groups • Block, chip, system • Current milestone functionality • Instance or module specific • Across environments, time, model releases • Cross views 30 September 27 2006
  • Our use of coverage • Aggregate data for each verification environment • Views: Verification effectiveness − Verification environment • Views: TR readiness − Major sub-blocks and chip • Filtering infrastructure − Milestone specific functionality − Unreachable • Aggregate coverage data across windows of time • Metrics provided for each team and full chips 30 September 28 2006
  • Analysis is done… now what? • Understand all un-hit coverage • Fill coverage holes • Look for hard to hit coverage • Track coverage metrics Don’t play games with metrics just to get coverage goals met Really understand the results 30 September 29 2006
  • Coverage Planning results PSL / SVA / OVL 1. Choose specification form 4. Collect data Code FCP Txn 2. Identify Spec, design coverage 5. Analyze Coverage tools model data Consumption Code / 3. Implement Assertion / coverage 6. React to Adjust stimulus FCP / Txn model data Execution 30 September 30 2006
  • Success stories • They exist! • Check them out in Assertion-Based Design 30 September 31 2006
  • HP coverage data • SX1000 chipset − 6,500 FCPs • SX2000 chipset − 25,000 FCPs • Current efforts − 135,000 assertions and 650,000 FCPs − 56,000 transaction points • Coverage goals − 100% coverage with understood exceptions − Team defined goals per milestone 30 September 32 2006
  • Resources − J. Bergeron, Writing Testbenches: Functional Verification of HDL Models, Second Edition, Kluwer Academic Publishers, 2003. − H. Foster, A. Krolnick, D. Lacey, Assertion-Based Design, Second Edition, Kluwer Academic Publishers, 2004. − P. James, Verification Plans: The Five-Day Verification Strategy for Modern Hardware Verification Languages, Kluwer Academic Publishers, 2004. − A. Piziali, Functional Verification Coverage Measurement and Analysis, Kluwer Academic Publishers, 2004. − B. Cohen, Using PSL/Sugar with Verilog and VHDL, Guide to Property Specification Language for ABV, VhdlCohen Publishing, 2003. David Lacey, Hewlett Packard, david.lacey@hp.com Rob Porter, Hewlett Packard, robert.porter@hp.com 30 September 33 2006