Verification Challenges and Methodologies

Shivananda
(Shivoo)
R
Koteshwar

Director,
MediaTek

shivoo.koteshwar@gmail.com
/
Facebook:
shivoo.koteshwar

BLOG:
http://shivookoteshwar.wordpress.com

SLIDESHARE:
www.slideshare.net/shivoo.koteshwar

BMS College and Mentor Graphics,
Bangalore
Jul 2015

1.  Basics

2.  Verification
Challenges

3.  Verification
Strategies

4.  Verification
Methodologies

5.  The
market

6.  The
Opportunities

7.  Skills
needed

8.  Q&A

•  Design
synthesis:

!  Given
an
I/O
function,
develop
a
procedure
to

manufacture
a
device
using
known
materials
and

processes

•  Veriﬁcation:

!  Predictive
analysis
to
ensure
that
the
synthesized
design,

when
manufactured,
will
perform
the
given
I/O
function

•  Test:

!  A
manufacturing
step
that
ensures
that
the
physical

device,
manufactured
from
the
synthesized
design,
has
no

manufacturing
defect.

3

!  Goal:
Validate
a
model
of
the
design

!  Testbench
wraps
around
the
design
under
test
(DUT)

!  Inputs
provide
(deterministic
or
random)
stimulus

!  Reference
signals:
clock(s),
reset,
etc.

!  Data:
bits,
bit
words

!  Protocols:
PCI,
SPI,
AMBA,
USB,
etc.

!  Outputs
capture
responses
and
make
checks

!  Data:
bits,
bit
words

!  Protocols:
PCI,
SPI,
AMBA,
USB,
etc.

http://www.xilinx.com/support/ sw_manuals/xilinx8/download/qst.zip
Design entry (VHDL,
Verilog, State diagrams,
pre-designed cores)
Net lists
Placement, routing

Veriﬁcation
is
the
process
of
verifying
the
transformation
steps
in

the
design
ﬂow
are
executed
correctly.

Algorithm
Architecture/
Spec RTL Gate GDSII ASIC
End
productIdea
Product
Acceptance
Test
Transformations
C-Model
Spec
Acceptance
Review
Simulation/
Code Review
Formal
Functional/ Timing
Verification ATE
Sign-Off
Review

!  Ensure
full
conformance
with
speciﬁcation:

!  Must
avoid
false
positives
(untested

functionalities)

???Pass
Fail
Good Bad(bug)
RTL code
√

Tape out!
Debug
testbench
Debug RTL
code
Testbench
Simulation
result
False
positive
results in
shipping a
bad design
How do we achieve this goal?

Design Errors Simulation –Practical Problem

!  Coverage

!  Code
Coverage

▪  Statement
or
Block
Coverage

▪  Path
Coverage

▪  Expression
Coverage

!  Functional
Coverage

!  Veriﬁcation
languages
can
raise
the
level
of
abstraction

!  Best
way
to
increase
productivity
is
to
raise
the
level
of

abstraction
used
to
perform
a
task

!  VHDL
and
Verilog
are
simulation
languages,
not

veriﬁcation
languages

!  VHDL
simulation
tools
can
automatically
calculate
a
metric
called
code

coverage
(assuming
you
have
licenses
for
this
feature).

!  Code
coverage
tracks
what
lines
of
code
or
expressions
in
the
code
have

been
exercised.

!  Code
coverage
cannot
detect
conditions
that
are
not
in
the
code

!  Code
coverage
on
a
partially
implemented
design
can
reach
100%.

It

cannot
detect
missing
features
and
many
boundary
conditions
(in

particular
those
that
span
more
than
one
block)

!  Code
coverage
is
an
optimistic
metric.

In
combinational
logic
code
in
an

HDL,
a
process
may
be
executed
many
times
during
a
given
clock
cycle

due
to
delta
cycle
changes
on
input
signals.

This
can
result
in
several

diﬀerent
branches
of
code
being
executed.

However,
only
the
last
branch

of
code
executed
before
the
clock
edge
truly
has
been
covered.

!  Hence,
code
coverage
cannot
be
used
exclusively
to
indicate
we
are
done

testing.

12

!  Functional
coverage
is
code
that
observes
execution
of
a
test
plan.

As
such,
it
is
code
you

write
to
track
whether
important
values,
sets
of
values,
or
sequences
of
values
that

correspond
to
design
or
interface
requirements,
features,
or
boundary
conditions
have
been

exercised

!  Specifically,
100%
functional
coverage
indicates
that
all
items
in
the
test
plan
have
been

tested.

Combine
this
with
100%
code
coverage
and
it
indicates
that
testing
is
done

!  Functional
coverage
that
examines
the
values
within
a
single
object
is
called
either
point

coverage
or
item
coverage

!  One
relationship
we
might
look
at
is
different
transfer
sizes
across
a
packet
based
bus.

For
example,

the
test
plan
may
require
that
transfer
sizes
with
the
following
size
or
range
of
sizes
be
observed:
1,
2,

3,
4
to
127,
128
to
252,
253,
254,
or
255.

!  Functional
coverage
that
examines
the
relationships
between
different
objects
is
called
cross

coverage.

An
example
of
this
would
be
examining
whether
an
ALU
has
done
all
of
its

supported
operations
with
every
different
input
pair
of
registers

!  VHDL’s
Open
Source
VHDL
Verification
Methodology
(OSVVM)
provides
a
package,

CoveragePkg,
with
a
protected
type
that
facilitates
capturing
the
data
structure
and
writing

functional
coverage

!  Functional
Coverage
provides
additional
supporting
data
that
the
design
is
tested.
It’s
a

supplement
to
Primitive
testing
directed,
algorithmic,
file
based,
or
constrained
random

test
methods

13

!  It
is
quite
possible
to
achieve
100%
code
coverage
but
only
50%
functional

coverage

!  Here
the
design
is
half
complete

!  Equally,
it
is
possible
to
have
50%
code
coverage
but
100%
functional

coverage

!  Indicates
that
the
functional
coverage
model
is
missing
some
key
features
of
the
design

!  Indicates
the
design
contains
untested
code
that
is
not
part
of
the
test
plan

!  This
can
come
from
an
incomplete
test
plan,
extra
undocumented
features
in
the
design,
or

case
statement
others
branches
that
do
not
get
exercised
in
normal
hardware
operation

!  Untested
features
need
to
either
be
tested
or
removed

!  As
a
result,
even
with
100%
functional
coverage
it
is
still
a
good
idea
to
use
code
coverage
as
a

fail
safe
for
the
test
plan

!  Code
Coverage
is
quantitative
coverage
and
functional
coverage
is
qualitative

coverage

!  The
two
coverage
approaches
are
complementary,
and
high
quality

veriﬁcation
will
beneﬁt
from
both.

14
Test Done = Test Plan Executed and All Code Executed
REF: https://www.doulos.com/knowhow/sysverilog/uvm/easier_uvm_guidelines/coverage-driven

!  IP
/
Module
Level
Veriﬁcation

Study DUT and related
Specification
Gather requirements for
features to be verified and set priorities
Review Requirements with IP Architect/Designer
(Requirements should cover all parameters for module)
Design Test Infrastructure on paper / document
(includes re-usable verification components)
Review TB Architecture with Verification Team
Design Test Infrastructure
Code Testcases as per Test-Bench Plan. Also code
Functional Coverpoints / Assertions
Complete Verification such that Functional Coverage
100% and Code Coverage numbers are logged.
Review Code Coverage numbers with Designer
to eliminate dead code possibilities.
Sign off Module Level Verification by checking in
the files having relevant data such as logs.

!  SoC
Level
Veriﬁcation

Study SoC and related
Specification
Gather requirements for
critical data paths set priorities
Review Requirements with IP Architect/Designer
(Requirements should cover all parameters for module)
Design Test Infrastructure on paper / document
Identify testcases that can be re-used
Review TB Architecture with Verification Team
Design Test Infrastructure
Code Testcases as per Test-Bench Plan. Also code
Functional Coverpoints / Assertions
Complete Verification such that Functional Coverage
100% and Code Coverage numbers are logged.
Review Code Coverage numbers with Designer
to eliminate dead code possibilities.
Sign off SoC Verification by checking in
the files having relevant data such as logs.

TESTBENCH
ENVIRONMENT /
ARCHITECTURE

!  C
type
data
types
like
int,
typedef,
struct,
union,
enum

!  Dynamic
data
types
:
struct,
classes,
dynamic
queues,

dynamic
arrays

!  New
operators
and
built
in
methods

!  Enhanced
ﬂow
control
like,
foreach,
return,
break,

continue

!  Inter-‐process
synchronization
–
Semaphores,

Mailboxes,
Event
Extension

!  Assertions
and
Coverage

!  Clocking
Domains

!  Direct
Programming
Interface
(DPI)
-‐
VPI

!  Hardware
speciﬁc
procedures

REF: http://www.eetimes.com/document.asp?doc_id=1277143

!  Provides
a
reusable
,standard
infrastructure
in
form
of

base
classes
which
are
pre-‐defined.
These
can
be

extended
and
enhanced
as
per
user
needs

!  Defines
rules
to
create
behavioral
models
also
known

as
Verification
Components
(OVC/UVC)

!  Defines
standards
for
higher
level
of
modelling
input

stimulus
using
Transaction
Level
Modelling
(TLM)

!  Defines
rules
to
have
a
layered
structure
of
test-‐
benches

!  In
summary
Methodology
=
standardization
of
the

way
of
creating
complex
test-‐benches
with

constrained
random
test-‐vectors.

!  OVM

!  Open
Verification
Methodology

!  Derived
mainly
from
the
URM
(Universal
Reuse
Methodology)
which
was,
to
a
large
part,
based
on
the
eRM

(e
Reuse
Methodology)
for
the
e
Verification
Language
developed
by
Verisity
Design
in
2001

!  The
OVM
also
brings
in
concepts
from
the
Advanced
Verification
Methodology
(AVM)

!  System
Verilog

!  RVM

!  Reference
Verification
Methodology

!  Complete
set
of
metrics
and
methods
for
performing
Functional
verification
of
complex
designs

!  The
SystemVerilog
implementation
of
the
RVM
is
known
as
the
VMM
(Verification
Methodology
Manual)

!  OVL

!  Open
Verification
Language

!  OVL
library
of
assertion
checkers
is
intended
to
be
used
by
design,
integration,
and
verification
engineers
to

check
for
good/bad
behavior
in
simulation,
emulation,
and
formal
verification.

!  Accellera
-‐
http://www.accellera.org/downloads/standards/ovl/

!  UVM

!  Standard
Universal
Verification
Methodology

!  Accellera
-‐
http://www.accellera.org/downloads/standards/uvm

!  System
Verilog

!  OS-‐VVM

!  VHDL

!  Accellera

OVC: OVM Verification Component
UVC: Universal Verification Component

!  Simple
reusable
OVCs

interfaced
to
a
DUT.

!  Due
to
standard
nature

third
party
OVCs
can
be

quickly
interfaced
and

used.

!  A
virtual
sequencer

controls
the
sequencer

which
controls
the
OVC

sequences
to
generate

scenarios.

!  Run
the
most
important
tests
ﬁrst
when
you
get
a

new
build

!  Do
not
start
over
on
your
test
pass
every
time
you

receive
a
new
build

!  Regression
tests
that
have
been
run
already
many

times
are
unlikely
to
reveal
new
bugs.
If
your
testcase

fully
automated,
by
all
means,
run
all
of
them
for
each

build.

!  Prioritize
tests
into
“Must-‐Pass”
types
with
a
more

focused
list
of
tests
which
can
reduce
the
time
of
the

regression.
Major
builds
will
warrant
running
all

testcases.

!  Automate
whenever
it
makes
sense
to
do
so.

!  Automation
is
a
means
of
reducing
manual

eﬀort
in
running
repetitive
tasks
such
as

regressions.

!  Automation
can
be
done
also
in
creating
test-‐
benches
so
that
a
standard
infrastructure
is

maintained
across
the
team.

!  This
can
be
done
using
PERL
scripts.

!  Why
use
PERL
?

-‐  Free
and
works
with
most
UNIX
and
Linux
versions

-‐  Ease
to
work
with
,
smaller
learning
curve.

-‐  Advance
PERL
with
OOPs
available
makes
scripting

easier

!  Scripting

!  PERL,
Python,
C++

!  Languages
and
Methodologies

!  Verilog,
VHDL,
System
Verilog,
UVM,
OVM,
VMM

!  Problem
Solving
Skills

!  Diligent
and
Methodical

!  Documentation
Skills

!  Reading
Skills!

Visit my slideshare to
view all these
presentations

Shivananda
(Shivoo)
R
Koteshwar

Director,
Mediatek

shivoo.koteshwar@gmail.com/
Facebook:
shivoo.koteshwar

BLOG:
http://shivookoteshwar.wordpress.com

SLIDESHARE:
www.slideshare.net/shivoo.koteshwar

Verification Challenges and Methodologies

More Related Content

What's hot

Viewers also liked

Similar to Verification Challenges and Methodologies

More from Dr. Shivananda Koteshwar

Recently uploaded

In this document

Verification Challenges and Methodologies