The document discusses verification metrics for CPU design projects. There are two key types of metrics: 1) verification test plan metrics like tests completed and assertions written to track progress, and 2) health of the design metrics like bug rates and simulation passing rates. While metrics provide useful information, they have limitations and challenges like historical data sometimes being misleading. Examples of metrics include a bug rate graph showing a "knee in the curve" and a functional coverage closure chart.

1. Verification Metrics
Dave Williamson
CPU Verification and Modeling Manager
Austin Design Center
June 2006
1

2. Verification Metrics: Why do we care?
 Predicting functional closure of a design is hard
 Design verification is typically the critical path
 CPU design projects rarely complete on schedule
 Cost of failure to predict design closure is significant
2

3. Two key types of metrics
 Verification test plan based metrics
 Amount of direct tests completed
 Amount of random testing completed
 Number of assertions written
 Amount of functional coverage written and hit
 Verification reviews completed
 Health of the design metrics
 Simulation passing rates
 Bug rate
 Code stability
 Design reviews completed
3

4. Challenges and limitations
 Limitations of test plan based metrics
 Will give a best case answer for completion date
 The plan will grow as testing continues
 Limitations of health of the design based metrics
 Can give false impressions if used independent from test plan metrics
 Requires good historical data on similar project for proper interpretation
 General concerns to be aware of for all metrics
 What you measure will affect what you do
 Gathering metrics is not free
 Historical data can be misleading
 Don’t be a slave to the metrics:
 they are a great tool, but not the complete answer
4

5. Bug rate example
Bug History
1200 20
Knee in curve
18
1000
16
Bug Rate Rolling Average
14
800
Total Bug Count
12
600 10
8
400
6
4
200
2
0 0
13
17
21
25
29
33
37
41
49
65
69
85
97
101
105
109
1
5
9
45
53
57
61
73
77
81
89
93
113
Week number
Total Bug Count Weekly Bug Count (4wk rolling average)
5

6. Bug rate by unit example
Bug breakdown per design unit
300
250
200
150
100
50
0
1
5
9
3
1
5
9
13
17
21
29
33
37
45
49
53
57
61
65
69
73
77
89
93
97
25
41
81
85
10
10
10
11
6

7. Functional Coverage closure example
New coverage
points added
7