This document discusses coverage in hardware verification. It defines key coverage terms like coverage model, coverage space, and coverage point. It describes different coverage techniques like code coverage, functional coverage, assertion coverage, and transaction coverage. The document provides guidance on planning a coverage strategy, including identifying what to cover, when to add coverage points, and avoiding common pitfalls. It outlines the steps to implement coverage, from planning to collecting data to analysis. The goal is to help engineers get started with and effectively use coverage to improve verification.

1. What’s with all
this talk about
coverage?
David Lacey and Rob Porter
Hewlett Packard Company
June 20, 2006
© 2006 Hewlett-Packard Development Company, L.P.
The information contained herein is subject to change without notice

2. “You have this awesome generation that
pseudo-randomly creates all sorts of
good scenarios. You also have created
equally awesome scoreboard and
temporal checker infrastructure that will
catch all the bugs. Next, you run it like
mad, with all sorts of seeds to hit as
much of the verification space as
possible.”
Peet James
Verification Plans
30 September
2 2006

3. Given all that, what really happened?
• Where did all those transaction go?
• Which lines of RTL were exercised?
• Which sections of the specification were tested?
• Which corner cases of my implementation were hit?
• What was the distribution of transaction types issues?
• Do I need to create new tests?
• Can I stop running simulations?
Coverage helps
provides the
Coverage is a pieceanswers! the final answer
of the puzzle, not
30 September
3 2006

4. Coverage provides…
• An understanding of which portions of the design
have been exercised
• Increased observability of simulation behavior
• Feedback on which tests are effective
• Feedback to direct future verification efforts
30 September
4 2006

5. Agenda
• Coverage terms and tools
• How to get started
• Coverage planning
• Coverage execution
• Coverage analysis
• Coverage results
30 September
5 2006

6. Coverage terms
and tools
30 September
6 2006

7. Coverage terms
• Coverage strategy
− Approach defined to utilize coverage technology
− Generate, gather, and analyze coverage data
• Coverage model
− Collection of coverage spaces
− Definition of one or more coverage spaces of interest
• Coverage space
− Set of coverage points associated with a single aspect of the design and a single
coverage technology
• Coverage technology
− Specific mechanism such as code, functional, assertion, transaction
• Coverage point
− A specific named aspect of the design behavior
− FCP, line of code, state transition, transaction or sequence of transactions
• Coverage data
− Raw data collected from all coverage points and coverage spaces
• Coverage results
− Interpretation of coverage data in context of coverage model
30 September
7 2006

8. Coverage model vs. coverage tools
• WHAT
−Coverage model High Level
Architecture
Specification
Bug rates Transaction
Sim cycles Coverage Functional
Model
Code Assertion
Design detail
Low Level
• HOW
Coverage tools Coverage tools
30 September
8 2006

9. Code coverage
• Line/block, branch, path, expression, state
• Measures controllability aspect of our stimulus
− i.e. What lines of code have we exercised
• Does not connect us to the actual functionality of the chip
− No insight into functional correctness
• Takes a blind approach to coverage (low observability)
− Activating an erroneous statement does not mean the error will
propagate to an observable point during the course of a simulation
• Generates a lot of data
− Difficult to interpret what is significant and what is not
30 September
9 2006

10. Assertion coverage
Assertions monitor and report undesirable
behavior
• Ensures that preconditions of an assertion check
have been met
// SVA: if asserting stop or flush, no new request
assert property (@(posedge clk) disable iff (rst_n)
((Flush | SMQueStop) |->
SMQueNew))
else $error(“Illegal behavior”);
precondition
check
30 September
10 2006

11. Functional coverage
• Similar in nature to assertions
Assertions monitor and report undesirable behavior
Functional coverage monitors and reports desirable
behavior
• Functional coverage
− Specific design details
− Corner cases of interest to engineers
− Architectural features
30 September
11 2006

12. Transaction coverage
• A transaction is the logging of any data structure
− A packet on a bus
− Does not have to be a system packet
• Example transaction coverage points
− All transaction types were seen on each interface
− Transactions with specific data were seen
• Source, destination, address, address ranges
• Sequences of transactions
− Have recording monitor watch for sequence
− Implement advanced queries to look for sequence
• Two parts to transaction coverage
− Record the right data
− Correct queries
30 September
12 2006

13. EDA tools
• Code, FCPs and transactions are recorded into
vendor specific databases
− Tools are provided to look at coverage data
− Report engines provide text reports
• Debug tools for FCPs and assertions
• Tools to encourage coverage-driven
methodologies
• Coverage is still a young technology
− Tools still expanding set of capabilities
− Development areas such as data aggregation, multiple
view extraction
30 September
13 2006

14. How do I get
started with this
coverage stuff?
30 September
14 2006

15. Coverage roadmap – getting started
Planning
PSL / SVA / 1. Choose Code
specification 4. Collect
OVL
form data FCP
Txn
2. Identify
Spec, design coverage 5. Analyze Coverage tools
model data
Consumption
Code / 3. Implement
Assertion / FCP coverage 6. React to
/ Txn Adjust stimulus
model data
Execution
30 September
15 2006

16. Coverage
planning PSL / SVA /
Planning
1. Choose Code
OVL specification 4. Collect
form data FCP
Txn
2. Identify
Spec, design coverage 5. Analyze Coverage tools
model data
Consumption
Code / 3. Implement
Assertion / FCP coverage 6. React to Adjust stimulus
/ Txn model data
Execution
30 September
16 2006

17. Coverage planning
Start looking at coverage up front!
Coverage results only as good as coverage
model
• Identify content of the coverage model
− Coverage types to be used
• Identify required tools
• Coverage infrastructure
• Coverage execution
• Maintenance and tool enhancements
• Define coverage goals and metrics
• Coverage reviews
30 September
17 2006

18. Who, what, when, where, why
• Who creates coverage model?
− Who analyses the data? Logic and DV engineers
− Who owns coverage?
• What to cover in the model? Concern areas
• When to add coverage points? Add with RTL
− When to analyze coverage data? Analyze continuously
spec, design, assertions,
• Where to look for ideas? test plan
• Why mess with coverage? Because…
30 September
18 2006

19. For FCPs, ask yourself…
−What should be covered?
−Where is best place to put FCPs?
−When to look for condition?
−Why have coverage point?
30 September
19 2006

20. Watch out for…
• Too much data
− Need information, not data
− Need supporting tools to get correct
views of data
• Ineffective use of coverage
− FCPs that fire every clock cycle
− Duplication of coverage with different
tools
• Reading too much into grading tests
− Random tests produce different results
with different seeds
30 September
20 2006

21. Cost of coverage
• Plan for the costs of using coverage
− Get solid infrastructure setup
− Plan for slower simulations
• Some level of cost is acceptable
− Getting value back for investment
• Be smart
− Architect coverage plan up front to ensure success
30 September
21 2006

22. Coverage
execution PSL / SVA
/ OVL
Planning
1. Choose
specificatio 4. Collect
Code
n form data FCP
Txn
2. Identify
Spec, design coverage 5. Analyze Coverage tools
model data
Consumption
Code / 3.
Assertion / Implement
coverage 6. React to Adjust stimulus
FCP / Txn
model data
Execution
30 September
22 2006

23. Describing coverage model
• Code coverage
− RTL code, pragmas
• Assertion and functional coverage
− Use assertion language or library (PSL, SVA, OVL)
• Transaction
− Use hooks into Transaction Level Modeling
// SVA cover example
// PSL cover example
always @(posedge clk) begin
default clock = (posedge clk);
if (reset_n)
sequence qFullCondition =
myQfull: cover (q_full)
{reset_n ? (q_full : 1’b0);
$info (“queue was full”);
cover qFullCondition;
end
30 September
23 2006

24. Data collection
• Collect data across volume simulation
• Aggregate multiple databases
• Location of coverage data repository
• Manage volume of data
30 September
24 2006

25. Coverage
analysis
Planning
PSL / SVA / 1. Choose Code
OVL specification 4. Collect
form data FCP
Txn
2. Identify
Spec, design coverage 5. Analyze Coverage tools
model data
Consumption
Code / 3. Implement
Assertion / coverage 6. React to Adjust stimulus
FCP / Txn model data
Execution
30 September
25 2006

26. The analysis
• Easy to generate a ton
of data
− Want information, not
data
Need to organize the
data
Can’t look at it all at
once
Determine views
needed
30 September
26 2006

27. Views of coverage data
• Un-hit coverage
• Functionality groups
• Block, chip, system
• Current milestone functionality
• Instance or module specific
• Across environments, time,
model releases
• Cross views
30 September
27 2006

28. Our use of coverage
• Aggregate data for each verification environment
• Views: Verification effectiveness
− Verification environment
• Views: TR readiness
− Major sub-blocks and chip
• Filtering infrastructure
− Milestone specific functionality
− Unreachable
• Aggregate coverage data across windows of time
• Metrics provided for each team and full chips
30 September
28 2006

29. Analysis is done… now what?
• Understand all un-hit coverage
• Fill coverage holes
• Look for hard to hit coverage
• Track coverage metrics
Don’t play games with
metrics just to get
coverage goals met
Really understand the
results
30 September
29 2006

30. Coverage Planning
results PSL / SVA /
OVL
1. Choose
specification
form
4. Collect
data
Code
FCP
Txn
2. Identify
Spec, design coverage 5. Analyze Coverage tools
model data
Consumption
Code / 3. Implement
Assertion / coverage 6. React to Adjust stimulus
FCP / Txn model data
Execution
30 September
30 2006

31. Success stories
• They exist!
• Check them out in Assertion-Based Design
30 September
31 2006

32. HP coverage data
• SX1000 chipset
− 6,500 FCPs
• SX2000 chipset
− 25,000 FCPs
• Current efforts
− 135,000 assertions and 650,000 FCPs
− 56,000 transaction points
• Coverage goals
− 100% coverage with understood exceptions
− Team defined goals per milestone
30 September
32 2006

33. Resources
− J. Bergeron, Writing Testbenches: Functional Verification of HDL
Models, Second Edition, Kluwer Academic Publishers, 2003.
− H. Foster, A. Krolnick, D. Lacey, Assertion-Based Design, Second
Edition, Kluwer Academic Publishers, 2004.
− P. James, Verification Plans: The Five-Day Verification Strategy
for Modern Hardware Verification Languages, Kluwer Academic
Publishers, 2004.
− A. Piziali, Functional Verification Coverage Measurement and
Analysis, Kluwer Academic Publishers, 2004.
− B. Cohen, Using PSL/Sugar with Verilog and VHDL, Guide to
Property Specification Language for ABV, VhdlCohen Publishing,
2003.
David Lacey, Hewlett Packard, david.lacey@hp.com
Rob Porter, Hewlett Packard, robert.porter@hp.com
30 September
33 2006