SlideShare a Scribd company logo
1 of 41
Download to read offline
Systems and Technology Group
March 5, 2007 © 2007 IBM Corporation
Experience with Formal Methods,
Especially Sequential Equivalence Checking
Mark Firstenberg
firsten@us.ibm.com
Systems and Technology Group
© 2007 IBM Corporation2 March 5, 2007
From Bob Fredieu’s Talk
Statements Comments
‘Current Trends: Boredom’
‘What has not helped:
Formal Verification –
Can’t get people to do it’
What?!
Explore Something New!
‘Don’t Waste Money’ Pursue Efficiency/
Improve Quality
Wrong Expectations?
Benefits Unknown?
Purpose of this talk:
Provide a user’s perspective of Formal Methods
Systems and Technology Group
© 2007 IBM Corporation3 March 5, 2007
Why Consider Formal Methods?
Formal methods are preferable to pattern based
simulation since formal proofs:
– Provide complete coverage while pattern based
simulation does not,
– Require a fraction of the computer resources pattern
based simulation does, and
– Require fewer human resources than pattern based
simulation does.
At least that is the theory . . .
Systems and Technology Group
© 2007 IBM Corporation4 March 5, 2007
Verification Process: New Design
Layout
Schematic
RTL
Spec.
Logic Designer
Circuit Designer
Layout Designer
Pattern-Based SimulationFormal Methods
RTL-Level Simulation
Gate-Level Simulation
Model Checking
Equivalence Checking
LVS
(Semi-Formal
Methods)
Design Activity
Systems and Technology Group
© 2007 IBM Corporation5 March 5, 2007
Verification Process: New Design (notes)
Overall task is to verify that the design specification and the
final product (i.e. layout) are functionally equivalent
Since the level of abstraction between specification and final
product is too great, break the design task into activities
which produce functionally equivalent representations:
– Logic designer translates the specification to RTL by hand
– Circuit designer (or synthesizer) translates RTL to schematics
– Layout designer (or layout tool) translates schematics to layout
Then verify that adjacent design representations are
functionally equivalent
Systems and Technology Group
© 2007 IBM Corporation6 March 5, 2007
Verification Process: New Design (notes)
Pattern-Based Simulation Methods
– RTL-Level Simulation
• Required if the specification is not machine readable
(typically the case)
– Gate-Level Simulation
• Can be used for RTL to schematic comparison, but formal
methods are preferable
• Typically used for POR sequence testing
Systems and Technology Group
© 2007 IBM Corporation7 March 5, 2007
Verification Process: New Design (notes)
Formal Methods
– Model Checking
• Used to prove specific assertions about the design
• Inherently incomplete (in that all interesting assertions cannot
be specified) so is used as a backup to RTL-Level Simulation
– Equivalence Checking for RTL to schematic comparison
– LVS for schematic to layout comparison
– Semi-Formal Methods
• Accelerates triggering RTL based assertions
• Is not a pure formal method since it uses RTL-Level
Simulation to traverse portions of the state space
Systems and Technology Group
© 2007 IBM Corporation8 March 5, 2007
Verification Process: Derivative Design
Layout
Schematic
RTL
Spec.
Layout
Schematic
RTL
Spec.
Golden Design Derivative Design
RTL-to-RTL
Equivalence
Checking
RTL-Level
Simulation
RTL-Level
Simulation
Equivalence
Checking
LVS
Systems and Technology Group
© 2007 IBM Corporation9 March 5, 2007
Verification Process: Derivative Design (notes)
Previously verified ‘golden design’ used as a starting point
Use primary verification methods for adjacent derivative
design representations:
– RTL-Level Simulation for specification to RTL comparison
– Equivalence Checking for RTL to schematic comparison
– LVS for schematic to layout comparison
Bridging the gap between ‘golden’ and ‘derivative’ designs:
– Either migrate ‘golden model’ RTL-Level Simulation
environment and tests to derivative design RTL-Level
Simulation environment (effort may be significant depending on
type of changes in the derivative design)
– Or perform RTL (golden) to RTL (derivative) Equivalence
Checking taking derivative design changes into account
Systems and Technology Group
© 2007 IBM Corporation10 March 5, 2007
Equivalence Checking Base Definitions
Equivalence Checking: The process of comparing two design
representations to ensure that their digital function is the same.
Note that equivalence checking says nothing about functional
correctness!
Logic Cone: The collection of non-state device logic which feeds
a single state device within a design or a single primary output of
a design. Logic cone inputs are state device outputs and/or
design primary inputs.
Boolean Function: The function of a single logic cone.
Sequential Behavior: The digital behavior of a sequence of logic
cones and any intervening state devices (i.e. multi-cycle
behavior) as observed at the output of the sequential path.
Register Re-timing: The movement of logic across state devices
which preserves sequential behavior.
Systems and Technology Group
© 2007 IBM Corporation11 March 5, 2007
Formal Equivalence Checking Types
Boolean Equivalence Checking: The process of
comparing corresponding logic cones from two
different design representations to ensure that their
digital function is the same.
Sequential Equivalence Checking: The process of
comparing corresponding sequential path outputs
from two different design representations to ensure
that their digital function is the same.
Systems and Technology Group
© 2007 IBM Corporation12 March 5, 2007
Boolean vs Sequential Equivalence Checking Example
A(31:0)
B(31:0)
Sum(31:0)
Impl. 1
ScanIn ScanOut
R
e
g
A
R
e
g
B
R
e
g
S
A
D
D
B(31:0)
Sum(31:0)
Impl. 2
ScanIn ScanOut
R
e
g
S
A
D
D
A(31:0)
R
e
g
X
Two Cycle Adder Two Cycle Adder Re-timed
Systems and Technology Group
© 2007 IBM Corporation13 March 5, 2007
Boolean vs Sequential Equivalence Checking Example
(notes)
Boolean equivalence will detect mismatches at all
comparison points
– Impl. 1 RegA and RegB versus Impl. 2 RegX, because
• The logic cones feeding the registers have changed.
• If the size of RegA plus the size of RegB does not equal the size of
RegX, there will be extra state devices in either Impl. 1 or Impl. 2
without corresponding state devices in the other design.
– Impl. 1 and Impl. 2 RegS’s (and thus their Sum’s) will mismatch
because of logic cone differences.
– Impl. 1 and Impl. 2 ScanOut’s will mismatch because of the
RegS differences.
Systems and Technology Group
© 2007 IBM Corporation14 March 5, 2007
Boolean vs Sequential Equivalence Checking Example
(notes)
Sequential equivalence would find the two designs
equivalent except for the ScanOut
– The two Sum’s are functionally equivalent and
– The two ScanOut’s are different since the logic cone
differences feeding the RegS’s are visible during scan chain
shifting.
– Since only primary outputs are compared, a mismatch in state
devices between the two designs does not necessarily matter.
– Initial values in the ‘Reg’s could cause mismatches during the
first two cycles, so initial state device values matter!
If scan operations were disabled, the two designs would be
sequentially equivalent
Note that this includes sequential path latency
Systems and Technology Group
© 2007 IBM Corporation15 March 5, 2007
Boolean vs Sequential Equivalence Checking Comparison
Boolean equivalence checking is ‘stricter’ and more microscopic
– Register re-timing is reported as a mismatch.
– State device correspondence is required.
– Design constraints cannot propagate through state devices.
– State device initialization does not need to be considered.
Sequential equivalence checking takes a more macroscopic view
of the design
– Register re-timing is not reported as a mismatch.
– State device correspondence is not required.
– Design constraints applied at primary inputs propagate.
– State device initialization does need to be considered.
Systems and Technology Group
© 2007 IBM Corporation16 March 5, 2007
Semi-Formal Methods
Combination of pattern-based simulation and formal methods
– Simulate to get deep into design’s state space
– Then use formal methods to reach design assertions
– Accelerate determination if assertion can be triggered
Since simulation is involved, method is not purely formal
(diagram from 9/12/05 SixthSense presentation, Jason Baumgartner, et. al.)
Systems and Technology Group
© 2007 IBM Corporation17 March 5, 2007
Building a Sequential Equivalence Checker Using
Formal/Semi-Formal Methods
Semi-Formal Methods use formal methods to
determine if design assertions can be triggered
If the assertion is a checker comparing
corresponding outputs of two design
representations, such a tool could be used to
formally prove design equivalence (once the
dependence on pattern-based simulation is
eliminated)
Since the formal methods are designed to
propagate sequentially from an initial state, the
result would be a Sequential Equivalence Checker
Systems and Technology Group
© 2007 IBM Corporation18 March 5, 2007
Sequential Equivalence Checking Test Bench
‘old’ design
‘new’ design
Primary
Input
Stimulus
Generator
Primary
Output
Compare
User supplied mapping
and constraint information
Systems and Technology Group
© 2007 IBM Corporation19 March 5, 2007
The user supplies:
– ‘old’ design
– ‘new’ design
– Mapping and constraint information
Sequential equivalence checking tool
automatically generates the test bench
– Correlates output pins, connecting them to checkers
– Correlates input pins, driven from a stimulus generator
– Both output checkers and stimulus generator can be
modified via user supplied signal mappings and input
constraints
Sequential Equivalence Checking Test Bench (notes)
Systems and Technology Group
© 2007 IBM Corporation20 March 5, 2007
Input/Output Pin Correlation
By default, input/output pins from both designs are paired
based on pin names
For any uncorrelated input pins:
– Test bench allowed to stimulate them with different values
– Could lead to ‘false mismatches’ (which are annoying!)
– Fix/waive such warnings (via mapping file) before pursuing
mismatches
For any uncorrelated output pins:
– Test bench eliminates the pins
– Could lead to ‘false matches’ (which are not acceptable!)
– Fix/waive such warnings (via mapping file) before declaring
equivalence
Systems and Technology Group
© 2007 IBM Corporation21 March 5, 2007
Black Boxing Design Sections
PIs POsBBPIs BBPOs
Parent Unit X
Sub-Unit Y
Black-boxed
Sub-Unit Y
Systems and Technology Group
© 2007 IBM Corporation22 March 5, 2007
Black Boxing Design Sections (notes)
After ‘Sub-Unit Y’ has been ‘black boxed’, ‘Parent Unit X’s:
– Primary inputs will be the combination of ‘X’s Primary Inputs
(PIs) and ‘Y’s Black Box Primary Outputs (BBPOs)
– Primary outputs will be the combination of ‘X’s Primary Outputs
(POs) and ‘Y’s Black Box Primary Inputs (BBPIs)
‘Black Boxing’ helps alleviate tool capacity issues
– Reduces amount of logic being checked
‘Black Boxing’ increases complexity in other ways
– Increases number of items to prove (primary outputs)
– Increases number of items to control (primary inputs)
– Requires separate verification of the ‘black boxed’ unit
– Creates another equivalence checking boundary which needs
to be maintained
Systems and Technology Group
© 2007 IBM Corporation23 March 5, 2007
Sequential Equivalence Checking Debug Phases
mismatches?
‘unsolved’s?
Run
Comparison
no
Fix Problem
yes
Create Custom
‘.config’ file, etc.
Done
no
yes
Verify
Constraints
yes
failures?
no
Systems and Technology Group
© 2007 IBM Corporation24 March 5, 2007
Sequential Equivalence Checking Debug Phases (notes)
Comparison can produce three results for each ‘property’
– ‘Mismatch’: designer action required
– ‘Match’: equivalence proven
– ‘Unsolved’: incomplete proof (neither ‘mismatch’ or ‘match’)
Debug Phases
1. Designer resolves mismatches (via design or constraints
changes)
2. Application Engineer resolves ‘unsolved’s (via ‘.config’ files, if
not other methods)
3. Designer resolves incorrect constraints
Debug activities can be intermixed . . .
Systems and Technology Group
© 2007 IBM Corporation25 March 5, 2007
Debugging Sequential Equivalence Checking Mismatches
Trace (.aet file) is created for each mismatch:
– OUTPUT_MISMATCH shows when failure occurred
– Corresponding NEW and OLD model signals available
– Only as many cycles as needed are displayed . . .
– . . . making it easy to trace with little or no design knowledge!
Systems and Technology Group
© 2007 IBM Corporation26 March 5, 2007
XBOX 360 Processor Follow-on: Description
Goals
– Cost reduction of a multi-processor SOC
– Maintain ‘mission’ function and performance
Design Methods
– Process technology transfer
– 100+ changes to ‘pervasive’ (IBM term for ‘POR/DFT/debug’) function
– HDL: VHDL
Verification Methods
– Pervasive function: pattern based simulation
– Mission function: combination of sequential equivalence
checking and pattern based simulation (from previous design)
– Performance: sequential equivalence
Systems and Technology Group
© 2007 IBM Corporation27 March 5, 2007
XBOX 360 Processor Follow-on: ‘Latest is Greatest’
Comparisons
Golden
Model
Model
A
Model
B
Model
C
time
Systems and Technology Group
© 2007 IBM Corporation28 March 5, 2007
XBOX 360 Processor Follow-on: ‘Latest is Greatest’
Comparison (notes)
Always compare back to the ‘golden model’
– Problems (and solutions) accumulate
• A difficult problem can’t be avoided (unless designed away)
• Solutions must be carried forward (unless designed away)
– Always a desire to get to latest model . . .
– . . . more like traditional verification
Configuration management is less of an issue
(mainly needed for ‘golden model’)
Systems and Technology Group
© 2007 IBM Corporation29 March 5, 2007
XBOX 360 Processor Follow-on: Results
Sequential equivalence was used to compare the entire design
– ~25 design sections and ~25 arrays
– Only the physical layer of the I/O unit (PHY macro) was not compared
(since the old and new designs did not split the analog and digital
functions in the same way, so comparison was not possible)
Design issues uncovered
– Removal of piping latches detected (i.e. latency difference)
– Undocumented change discovered
Other discoveries
– Numerous signal constraint/documentation errors
– Differences between ‘pervasive’ and ‘mission’ simulation
environment’s use of POR state . . .
Systems and Technology Group
© 2007 IBM Corporation30 March 5, 2007
XBOX 360 Processor Follow-on: Lessons Learned
Only ‘black box’:
– At agreed upon equivalence checking boundaries
– When forced to because of tool capacity issues
Discovering uncorrelated inputs via debugging is painful!
The driver of a signal knows more about how to constrain that
signal than the consumer does, but the consumer does the
debugging (and thus proposes the constraint)
Must verify all constraints (i.e. equivalence checking assumptions),
using either:
– Formal Methods, or
– Simulation assertions
Don’t wait till the last minute to start using POR based state device
initialization
We needed POR results (for state device initialization) but POR is a
pervasive function (which was subject to change)!
Systems and Technology Group
© 2007 IBM Corporation31 March 5, 2007
PowerPC 464FP FPU: Description
Goals
– Cycle time improvement of an FPU
– Use the same process technology
– Maintain ‘mission’ function
Design Methods
– Significant register re-timing
– Micro-architectural changes (while maintaining operation latency)
– HDL: Verilog
Verification Methods
– Original design verified with many directed tests, with unknown
coverage . . .
– Sequential equivalence (to ‘hold the line’)
– Pattern based simulation (concentrating on random testing)
Systems and Technology Group
© 2007 IBM Corporation32 March 5, 2007
PowerPC 464FP FPU: ‘Step-Wise’ Comparisons
Golden
Model
Model
A
Model
B
Model
C
time
Systems and Technology Group
© 2007 IBM Corporation33 March 5, 2007
PowerPC 464FP FPU: ‘Step-Wise’ Comparisons (notes)
Based on ‘A==B, B==C, therefore A==C’ mentality
– Problems (and solutions) can be isolated
• Usually trying to isolate design changes which will be difficult to
verify (ex: register re-timings)
• Assumes difficult design changes are staged independently!
– Comparisons do not have to be resolved in order and debug
can progress in parallel
– Model patches may be required . . .
Strong configuration management required
– Constantly going back to old models (not just the ‘golden
model’)!
– Had better be versioning libraries and tools . . .
Systems and Technology Group
© 2007 IBM Corporation34 March 5, 2007
PowerPC 464FP FPU: Lessons Learned
Since there is less pressure to keep up, verification can quickly fall
behind
Accidentally disabling all clocks is a great way to get everything to
match ;^)
– Little warning is given for this mistake (other than the comparison job
runs very quickly), although a new option is being added to fix this
– Leads back to the importance of verifying constraints!
Although state device correlation is not required, it does help . . .
. . . constantly renaming state devices and/or using the same state
device names when changing function can be very disruptive!
Success with sequential equivalence checking is related more
closely to the nature of the design (ex: arithmetic versus control)
and the changes being made (ex: register re-timing versus DFT
changes) than to design size (ex: multi-processor SOC versus FPU)
Systems and Technology Group
© 2007 IBM Corporation35 March 5, 2007
Per Process Diagram
portals_
<hdl>
portals_
<hdl>
portals_
<hdl>
portals_
<hdl>
portals_
<hdl>
chip.proto
unitX.proto
unit1.proto . . .
sub-unitY.proto
portals_
<hdl>
sub-unit1.proto
. . .
chip.<hdl>
unitX.<hdl>
sub-unitY.<hdl>
portals_
<hdl>
. . .
testbench.<hdl>
sub-unitN.proto
unit1.proto
sub-unitN.protosub-unit1.proto
testbench.proto
mesabld runTest.pl
testbench.m2msmod
test.aet
test.source
run_aet_
to_reset
merge_
reset_files
insert.reset
POR.reset
equiv_check
-no_verify
POR.reset.rawchip.sxs_latch_list
chip.black_box.lst
cycle #
Database
Build
Simulate
Per Comparison
Reset File
Generation
Systems and Technology Group
© 2007 IBM Corporation36 March 5, 2007
Process Flow: Per Design Targets
portals_
<hdl>
portals_
<hdl>
portals_
<hdl>
portals_
<hdl>
portals_
<hdl>
chip.proto
unitX.proto
unit1.proto . . .
sub-unitY.proto
portals_
<hdl>
sub-unit1.proto
. . .
chip.<hdl>
unitX.<hdl>
sub-unitY.<hdl>
portals_
<hdl>
. . .
testbench.<hdl>
sub-unitN.proto
unit1.proto
sub-unitN.protosub-unit1.proto
testbench.proto
mesabld runTest.pl
testbench.m2msmod
test.aet
test.source
run_aet_
to_reset
merge_
reset_files
insert.reset
POR.reset
equiv_check
-no_verify
POR.reset.rawchip.sxs_latch_list
chip.black_box.lst
cycle #
Database
Build
Simulate
Per Comparison
Reset File
Generation
Systems and Technology Group
© 2007 IBM Corporation37 March 5, 2007
Process Flow: Per Comparison Targets
equiv_check
summarize_
equiv_check
_results
equiv_check
summarize_
equiv_check
_results
equiv_check
summarize_
equiv_check
_results
old_vs_new_xyz_equiv_check.rpt
old_xyz_self_equiv.rpt
new_xyz_self_equiv.rptnew_xyz.proto
old_xyz.proto
old_xyz.proto
new_xyz.proto
xyz.mapping
old_xyz.POR.reset
new_xyz.POR.reset
Main Equivalence Checking
mapping2
bugspray
xyz_bugspray.vhdl
Optional Self
Equivalence Checking
Systems and Technology Group
© 2007 IBM Corporation38 March 5, 2007
Process Flow (notes)
Significant process steps to support main point
tool (equiv_check) execution
Generated ‘makefile’ based flow
– Converts a design ‘hierarchical definition’ (.hd) file into a
‘makefile’ containing targets which perform analysis tasks
on that design
• Reproducible and reliable (execute only required targets)
• Proper sequencing and parallel execution of targets
• Build avoidance (don’t re-execute up to date targets)
• Scalable and easily modified (via ‘.hd’ file changes)
– User can choose to know as much or as little about target
specifics as he/she wants
Systems and Technology Group
© 2007 IBM Corporation39 March 5, 2007
Process Flow (notes)
Tool catalog
– Standard IBM internal tools
• Support mixing VHDL and Verilog source files during database builds
(portals_<hdl>)
• Cycle-Based Simulation build (mesabld)
• Waveform to reset file conversion (run_aet_to_reset)
– Project specific tools
• Running simulation test (runTest.pl)
– SixthSense flow tools
• Makefile (flow control) generator (hd2make – not depicted)
• ‘.reset’ file merging/checking (merge_reset_files)
• ‘equiv_check’ results checking (summarize_equiv_check_results)
• Constraint to assertion converter (mapping2bugspray)
– SixthSense
• ‘No verify’ test bench compile (equiv_check –no_verify)
• Test bench compile and verify (equiv_check)
Systems and Technology Group
© 2007 IBM Corporation40 March 5, 2007
Summary
Pursue Formal Methods . . .
. . . as long as you don’t expect them to work perfectly right
out of the box or without some flow development!
Formal Methods require the thoroughness that verification
engineers naturally possess
Analyzing a design from a different perspective is an
effective means of uncovering new problems
Embrace Assertion Based Verification
– Encourage designers to document their RTL (what a concept!)
– Enables formal methods
No excuse to be bored ;^)
Systems and Technology Group
© 2007 IBM Corporation41 March 5, 2007
Tool References
SixthSense
– (Semi-)Formal Tool and Sequential Equivalence Checker
– IBM internal tool (www.research.ibm.com/sixthsense)
Calypto SLEC
– Sequential Equivalence Checker
– Commercial EDA tool (www.calypto.com)
0-In
– Semi-Formal Tool
– Commercial EDA tool (www.mentor.com)
Magellan
– Semi-Formal Tool
– Commercial EDA tool (www.synopsys.com)
Incisive Formal Verifier
– Assertion Based Verification
– Commercial EDA tool (www.cadence.com)

More Related Content

What's hot

Finding Bugs Faster with Assertion Based Verification (ABV)
Finding Bugs Faster with Assertion Based Verification (ABV)Finding Bugs Faster with Assertion Based Verification (ABV)
Finding Bugs Faster with Assertion Based Verification (ABV)DVClub
 
Session 8 assertion_based_verification_and_interfaces
Session 8 assertion_based_verification_and_interfacesSession 8 assertion_based_verification_and_interfaces
Session 8 assertion_based_verification_and_interfacesNirav Desai
 
Formal Method for Avionics Software Verification
 Formal Method for Avionics Software Verification Formal Method for Avionics Software Verification
Formal Method for Avionics Software VerificationAdaCore
 
Verification Engineer - Opportunities and Career Path
Verification Engineer - Opportunities and Career PathVerification Engineer - Opportunities and Career Path
Verification Engineer - Opportunities and Career PathRamdas Mozhikunnath
 
SOC Verification using SystemVerilog
SOC Verification using SystemVerilog SOC Verification using SystemVerilog
SOC Verification using SystemVerilog Ramdas Mozhikunnath
 
System verilog important
System verilog importantSystem verilog important
System verilog importantelumalai7
 
Upgrading to System Verilog for FPGA Designs, Srinivasan Venkataramanan, CVC
Upgrading to System Verilog for FPGA Designs, Srinivasan Venkataramanan, CVCUpgrading to System Verilog for FPGA Designs, Srinivasan Venkataramanan, CVC
Upgrading to System Verilog for FPGA Designs, Srinivasan Venkataramanan, CVCFPGA Central
 
Model-Based Testing: Theory and Practice. Keynote @ MoTiP (ISSRE) 2012.
Model-Based Testing: Theory and Practice. Keynote @ MoTiP (ISSRE) 2012.Model-Based Testing: Theory and Practice. Keynote @ MoTiP (ISSRE) 2012.
Model-Based Testing: Theory and Practice. Keynote @ MoTiP (ISSRE) 2012.Wolfgang Grieskamp
 
Interface-Implementation Contract Checking
Interface-Implementation Contract CheckingInterface-Implementation Contract Checking
Interface-Implementation Contract CheckingDharmalingam Ganesan
 
Hands-on Experience Model based testing with spec explorer
Hands-on Experience Model based testing with spec explorer Hands-on Experience Model based testing with spec explorer
Hands-on Experience Model based testing with spec explorer Rachid Kherrazi
 
Session 7 code_functional_coverage
Session 7 code_functional_coverageSession 7 code_functional_coverage
Session 7 code_functional_coverageNirav Desai
 
Control Flow Testing
Control Flow TestingControl Flow Testing
Control Flow TestingHirra Sultan
 
Design For Testability
Design For TestabilityDesign For Testability
Design For TestabilityWill Iverson
 
Code Management Workshop
Code Management WorkshopCode Management Workshop
Code Management WorkshopSameh El-Ashry
 
Assessing Model-Based Testing: An Empirical Study Conducted in Industry
Assessing Model-Based Testing: An Empirical Study Conducted in IndustryAssessing Model-Based Testing: An Empirical Study Conducted in Industry
Assessing Model-Based Testing: An Empirical Study Conducted in IndustryDharmalingam Ganesan
 

What's hot (20)

Finding Bugs Faster with Assertion Based Verification (ABV)
Finding Bugs Faster with Assertion Based Verification (ABV)Finding Bugs Faster with Assertion Based Verification (ABV)
Finding Bugs Faster with Assertion Based Verification (ABV)
 
Session 8 assertion_based_verification_and_interfaces
Session 8 assertion_based_verification_and_interfacesSession 8 assertion_based_verification_and_interfaces
Session 8 assertion_based_verification_and_interfaces
 
Rv11
Rv11Rv11
Rv11
 
Formal Method for Avionics Software Verification
 Formal Method for Avionics Software Verification Formal Method for Avionics Software Verification
Formal Method for Avionics Software Verification
 
ASIC design verification
ASIC design verificationASIC design verification
ASIC design verification
 
Verification Engineer - Opportunities and Career Path
Verification Engineer - Opportunities and Career PathVerification Engineer - Opportunities and Career Path
Verification Engineer - Opportunities and Career Path
 
SOC Verification using SystemVerilog
SOC Verification using SystemVerilog SOC Verification using SystemVerilog
SOC Verification using SystemVerilog
 
Dsp
DspDsp
Dsp
 
System verilog important
System verilog importantSystem verilog important
System verilog important
 
Upgrading to System Verilog for FPGA Designs, Srinivasan Venkataramanan, CVC
Upgrading to System Verilog for FPGA Designs, Srinivasan Venkataramanan, CVCUpgrading to System Verilog for FPGA Designs, Srinivasan Venkataramanan, CVC
Upgrading to System Verilog for FPGA Designs, Srinivasan Venkataramanan, CVC
 
Model-Based Testing: Theory and Practice. Keynote @ MoTiP (ISSRE) 2012.
Model-Based Testing: Theory and Practice. Keynote @ MoTiP (ISSRE) 2012.Model-Based Testing: Theory and Practice. Keynote @ MoTiP (ISSRE) 2012.
Model-Based Testing: Theory and Practice. Keynote @ MoTiP (ISSRE) 2012.
 
Design Verification
Design VerificationDesign Verification
Design Verification
 
Design for Testability
Design for TestabilityDesign for Testability
Design for Testability
 
Interface-Implementation Contract Checking
Interface-Implementation Contract CheckingInterface-Implementation Contract Checking
Interface-Implementation Contract Checking
 
Hands-on Experience Model based testing with spec explorer
Hands-on Experience Model based testing with spec explorer Hands-on Experience Model based testing with spec explorer
Hands-on Experience Model based testing with spec explorer
 
Session 7 code_functional_coverage
Session 7 code_functional_coverageSession 7 code_functional_coverage
Session 7 code_functional_coverage
 
Control Flow Testing
Control Flow TestingControl Flow Testing
Control Flow Testing
 
Design For Testability
Design For TestabilityDesign For Testability
Design For Testability
 
Code Management Workshop
Code Management WorkshopCode Management Workshop
Code Management Workshop
 
Assessing Model-Based Testing: An Empirical Study Conducted in Industry
Assessing Model-Based Testing: An Empirical Study Conducted in IndustryAssessing Model-Based Testing: An Empirical Study Conducted in Industry
Assessing Model-Based Testing: An Empirical Study Conducted in Industry
 

Similar to Experience with Formal Methods, Especially Sequential Equivalence Checking

Brian muirhead v1-27-12
Brian muirhead v1-27-12Brian muirhead v1-27-12
Brian muirhead v1-27-12NASAPMC
 
Testware Hierarchy for Test Automation
Testware Hierarchy for Test AutomationTestware Hierarchy for Test Automation
Testware Hierarchy for Test AutomationGregory Solovey
 
DOTNET 2013 IEEE MOBILECOMPUTING PROJECT Model based analysis of wireless sys...
DOTNET 2013 IEEE MOBILECOMPUTING PROJECT Model based analysis of wireless sys...DOTNET 2013 IEEE MOBILECOMPUTING PROJECT Model based analysis of wireless sys...
DOTNET 2013 IEEE MOBILECOMPUTING PROJECT Model based analysis of wireless sys...IEEEGLOBALSOFTTECHNOLOGIES
 
cupdf.com_chapter-11-system-level-verification-issues-the-importance-of-verif...
cupdf.com_chapter-11-system-level-verification-issues-the-importance-of-verif...cupdf.com_chapter-11-system-level-verification-issues-the-importance-of-verif...
cupdf.com_chapter-11-system-level-verification-issues-the-importance-of-verif...SamHoney6
 
12 Rational Solo Pruebas 2009
12 Rational Solo Pruebas 200912 Rational Solo Pruebas 2009
12 Rational Solo Pruebas 2009Pepe
 
Enabling Automated Software Testing with Artificial Intelligence
Enabling Automated Software Testing with Artificial IntelligenceEnabling Automated Software Testing with Artificial Intelligence
Enabling Automated Software Testing with Artificial IntelligenceLionel Briand
 
2cee Master Cocomo20071
2cee Master Cocomo200712cee Master Cocomo20071
2cee Master Cocomo20071CS, NcState
 
Verilog Ams Used In Top Down Methodology For Wireless Integrated Circuits
Verilog Ams Used In Top Down Methodology For Wireless Integrated CircuitsVerilog Ams Used In Top Down Methodology For Wireless Integrated Circuits
Verilog Ams Used In Top Down Methodology For Wireless Integrated CircuitsRégis SANTONJA
 
Scalable Software Testing and Verification of Non-Functional Properties throu...
Scalable Software Testing and Verification of Non-Functional Properties throu...Scalable Software Testing and Verification of Non-Functional Properties throu...
Scalable Software Testing and Verification of Non-Functional Properties throu...Lionel Briand
 
Rejunevating software reengineering processes
Rejunevating software reengineering processesRejunevating software reengineering processes
Rejunevating software reengineering processesmanishthaper
 
testing throughout-the-software-life-cycle-section-2
testing throughout-the-software-life-cycle-section-2testing throughout-the-software-life-cycle-section-2
testing throughout-the-software-life-cycle-section-2Dr. Ahmed Al Zaidy
 
SSBSE 2020 keynote
SSBSE 2020 keynoteSSBSE 2020 keynote
SSBSE 2020 keynoteShiva Nejati
 
Basic software-testing-concepts
Basic software-testing-conceptsBasic software-testing-concepts
Basic software-testing-conceptsmedsherb
 
Performance Evaluation using Blackboard Technique in Software Architecture
Performance Evaluation using Blackboard Technique in Software ArchitecturePerformance Evaluation using Blackboard Technique in Software Architecture
Performance Evaluation using Blackboard Technique in Software ArchitectureEditor IJCATR
 
Controller Software Verification Using AVM Meta and HybridSAL
Controller Software Verification Using AVM Meta and HybridSALController Software Verification Using AVM Meta and HybridSAL
Controller Software Verification Using AVM Meta and HybridSALJoseph Porter
 

Similar to Experience with Formal Methods, Especially Sequential Equivalence Checking (20)

Firstenberg q207
Firstenberg q207Firstenberg q207
Firstenberg q207
 
Presentation Of Mbt Tools
Presentation Of Mbt ToolsPresentation Of Mbt Tools
Presentation Of Mbt Tools
 
Brian muirhead v1-27-12
Brian muirhead v1-27-12Brian muirhead v1-27-12
Brian muirhead v1-27-12
 
Testware Hierarchy for Test Automation
Testware Hierarchy for Test AutomationTestware Hierarchy for Test Automation
Testware Hierarchy for Test Automation
 
DOTNET 2013 IEEE MOBILECOMPUTING PROJECT Model based analysis of wireless sys...
DOTNET 2013 IEEE MOBILECOMPUTING PROJECT Model based analysis of wireless sys...DOTNET 2013 IEEE MOBILECOMPUTING PROJECT Model based analysis of wireless sys...
DOTNET 2013 IEEE MOBILECOMPUTING PROJECT Model based analysis of wireless sys...
 
cupdf.com_chapter-11-system-level-verification-issues-the-importance-of-verif...
cupdf.com_chapter-11-system-level-verification-issues-the-importance-of-verif...cupdf.com_chapter-11-system-level-verification-issues-the-importance-of-verif...
cupdf.com_chapter-11-system-level-verification-issues-the-importance-of-verif...
 
12 Rational Solo Pruebas 2009
12 Rational Solo Pruebas 200912 Rational Solo Pruebas 2009
12 Rational Solo Pruebas 2009
 
Enabling Automated Software Testing with Artificial Intelligence
Enabling Automated Software Testing with Artificial IntelligenceEnabling Automated Software Testing with Artificial Intelligence
Enabling Automated Software Testing with Artificial Intelligence
 
2cee Master Cocomo20071
2cee Master Cocomo200712cee Master Cocomo20071
2cee Master Cocomo20071
 
Verilog Ams Used In Top Down Methodology For Wireless Integrated Circuits
Verilog Ams Used In Top Down Methodology For Wireless Integrated CircuitsVerilog Ams Used In Top Down Methodology For Wireless Integrated Circuits
Verilog Ams Used In Top Down Methodology For Wireless Integrated Circuits
 
Testing
TestingTesting
Testing
 
Scalable Software Testing and Verification of Non-Functional Properties throu...
Scalable Software Testing and Verification of Non-Functional Properties throu...Scalable Software Testing and Verification of Non-Functional Properties throu...
Scalable Software Testing and Verification of Non-Functional Properties throu...
 
Rejunevating software reengineering processes
Rejunevating software reengineering processesRejunevating software reengineering processes
Rejunevating software reengineering processes
 
testing throughout-the-software-life-cycle-section-2
testing throughout-the-software-life-cycle-section-2testing throughout-the-software-life-cycle-section-2
testing throughout-the-software-life-cycle-section-2
 
SSBSE 2020 keynote
SSBSE 2020 keynoteSSBSE 2020 keynote
SSBSE 2020 keynote
 
Introduction & Manual Testing
Introduction & Manual TestingIntroduction & Manual Testing
Introduction & Manual Testing
 
Manualtestingppt
ManualtestingpptManualtestingppt
Manualtestingppt
 
Basic software-testing-concepts
Basic software-testing-conceptsBasic software-testing-concepts
Basic software-testing-concepts
 
Performance Evaluation using Blackboard Technique in Software Architecture
Performance Evaluation using Blackboard Technique in Software ArchitecturePerformance Evaluation using Blackboard Technique in Software Architecture
Performance Evaluation using Blackboard Technique in Software Architecture
 
Controller Software Verification Using AVM Meta and HybridSAL
Controller Software Verification Using AVM Meta and HybridSALController Software Verification Using AVM Meta and HybridSAL
Controller Software Verification Using AVM Meta and HybridSAL
 

More from DVClub

IP Reuse Impact on Design Verification Management Across the Enterprise
IP Reuse Impact on Design Verification Management Across the EnterpriseIP Reuse Impact on Design Verification Management Across the Enterprise
IP Reuse Impact on Design Verification Management Across the EnterpriseDVClub
 
Cisco Base Environment Overview
Cisco Base Environment OverviewCisco Base Environment Overview
Cisco Base Environment OverviewDVClub
 
Intel Xeon Pre-Silicon Validation: Introduction and Challenges
Intel Xeon Pre-Silicon Validation: Introduction and ChallengesIntel Xeon Pre-Silicon Validation: Introduction and Challenges
Intel Xeon Pre-Silicon Validation: Introduction and ChallengesDVClub
 
Verification of Graphics ASICs (Part II)
Verification of Graphics ASICs (Part II)Verification of Graphics ASICs (Part II)
Verification of Graphics ASICs (Part II)DVClub
 
Verification of Graphics ASICs (Part I)
Verification of Graphics ASICs (Part I)Verification of Graphics ASICs (Part I)
Verification of Graphics ASICs (Part I)DVClub
 
Stop Writing Assertions! Efficient Verification Methodology
Stop Writing Assertions! Efficient Verification MethodologyStop Writing Assertions! Efficient Verification Methodology
Stop Writing Assertions! Efficient Verification MethodologyDVClub
 
Validating Next Generation CPUs
Validating Next Generation CPUsValidating Next Generation CPUs
Validating Next Generation CPUsDVClub
 
Verification Automation Using IPXACT
Verification Automation Using IPXACTVerification Automation Using IPXACT
Verification Automation Using IPXACTDVClub
 
Validation and Design in a Small Team Environment
Validation and Design in a Small Team EnvironmentValidation and Design in a Small Team Environment
Validation and Design in a Small Team EnvironmentDVClub
 
Trends in Mixed Signal Validation
Trends in Mixed Signal ValidationTrends in Mixed Signal Validation
Trends in Mixed Signal ValidationDVClub
 
Verification In A Global Design Community
Verification In A Global Design CommunityVerification In A Global Design Community
Verification In A Global Design CommunityDVClub
 
Design Verification Using SystemC
Design Verification Using SystemCDesign Verification Using SystemC
Design Verification Using SystemCDVClub
 
Verification Strategy for PCI-Express
Verification Strategy for PCI-ExpressVerification Strategy for PCI-Express
Verification Strategy for PCI-ExpressDVClub
 
SystemVerilog Assertions (SVA) in the Design/Verification Process
SystemVerilog Assertions (SVA) in the Design/Verification ProcessSystemVerilog Assertions (SVA) in the Design/Verification Process
SystemVerilog Assertions (SVA) in the Design/Verification ProcessDVClub
 
Efficiency Through Methodology
Efficiency Through MethodologyEfficiency Through Methodology
Efficiency Through MethodologyDVClub
 
Pre-Si Verification for Post-Si Validation
Pre-Si Verification for Post-Si ValidationPre-Si Verification for Post-Si Validation
Pre-Si Verification for Post-Si ValidationDVClub
 
OpenSPARC T1 Processor
OpenSPARC T1 ProcessorOpenSPARC T1 Processor
OpenSPARC T1 ProcessorDVClub
 
Intel Atom Processor Pre-Silicon Verification Experience
Intel Atom Processor Pre-Silicon Verification ExperienceIntel Atom Processor Pre-Silicon Verification Experience
Intel Atom Processor Pre-Silicon Verification ExperienceDVClub
 
Using Assertions in AMS Verification
Using Assertions in AMS VerificationUsing Assertions in AMS Verification
Using Assertions in AMS VerificationDVClub
 
Low-Power Design and Verification
Low-Power Design and VerificationLow-Power Design and Verification
Low-Power Design and VerificationDVClub
 

More from DVClub (20)

IP Reuse Impact on Design Verification Management Across the Enterprise
IP Reuse Impact on Design Verification Management Across the EnterpriseIP Reuse Impact on Design Verification Management Across the Enterprise
IP Reuse Impact on Design Verification Management Across the Enterprise
 
Cisco Base Environment Overview
Cisco Base Environment OverviewCisco Base Environment Overview
Cisco Base Environment Overview
 
Intel Xeon Pre-Silicon Validation: Introduction and Challenges
Intel Xeon Pre-Silicon Validation: Introduction and ChallengesIntel Xeon Pre-Silicon Validation: Introduction and Challenges
Intel Xeon Pre-Silicon Validation: Introduction and Challenges
 
Verification of Graphics ASICs (Part II)
Verification of Graphics ASICs (Part II)Verification of Graphics ASICs (Part II)
Verification of Graphics ASICs (Part II)
 
Verification of Graphics ASICs (Part I)
Verification of Graphics ASICs (Part I)Verification of Graphics ASICs (Part I)
Verification of Graphics ASICs (Part I)
 
Stop Writing Assertions! Efficient Verification Methodology
Stop Writing Assertions! Efficient Verification MethodologyStop Writing Assertions! Efficient Verification Methodology
Stop Writing Assertions! Efficient Verification Methodology
 
Validating Next Generation CPUs
Validating Next Generation CPUsValidating Next Generation CPUs
Validating Next Generation CPUs
 
Verification Automation Using IPXACT
Verification Automation Using IPXACTVerification Automation Using IPXACT
Verification Automation Using IPXACT
 
Validation and Design in a Small Team Environment
Validation and Design in a Small Team EnvironmentValidation and Design in a Small Team Environment
Validation and Design in a Small Team Environment
 
Trends in Mixed Signal Validation
Trends in Mixed Signal ValidationTrends in Mixed Signal Validation
Trends in Mixed Signal Validation
 
Verification In A Global Design Community
Verification In A Global Design CommunityVerification In A Global Design Community
Verification In A Global Design Community
 
Design Verification Using SystemC
Design Verification Using SystemCDesign Verification Using SystemC
Design Verification Using SystemC
 
Verification Strategy for PCI-Express
Verification Strategy for PCI-ExpressVerification Strategy for PCI-Express
Verification Strategy for PCI-Express
 
SystemVerilog Assertions (SVA) in the Design/Verification Process
SystemVerilog Assertions (SVA) in the Design/Verification ProcessSystemVerilog Assertions (SVA) in the Design/Verification Process
SystemVerilog Assertions (SVA) in the Design/Verification Process
 
Efficiency Through Methodology
Efficiency Through MethodologyEfficiency Through Methodology
Efficiency Through Methodology
 
Pre-Si Verification for Post-Si Validation
Pre-Si Verification for Post-Si ValidationPre-Si Verification for Post-Si Validation
Pre-Si Verification for Post-Si Validation
 
OpenSPARC T1 Processor
OpenSPARC T1 ProcessorOpenSPARC T1 Processor
OpenSPARC T1 Processor
 
Intel Atom Processor Pre-Silicon Verification Experience
Intel Atom Processor Pre-Silicon Verification ExperienceIntel Atom Processor Pre-Silicon Verification Experience
Intel Atom Processor Pre-Silicon Verification Experience
 
Using Assertions in AMS Verification
Using Assertions in AMS VerificationUsing Assertions in AMS Verification
Using Assertions in AMS Verification
 
Low-Power Design and Verification
Low-Power Design and VerificationLow-Power Design and Verification
Low-Power Design and Verification
 

Recently uploaded

Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
IEEE Computer Society’s Strategic Activities and Products including SWEBOK Guide
IEEE Computer Society’s Strategic Activities and Products including SWEBOK GuideIEEE Computer Society’s Strategic Activities and Products including SWEBOK Guide
IEEE Computer Society’s Strategic Activities and Products including SWEBOK GuideHironori Washizaki
 
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...Daniel Zivkovic
 
99.99% of Your Traces Are (Probably) Trash (SRECon NA 2024).pdf
99.99% of Your Traces  Are (Probably) Trash (SRECon NA 2024).pdf99.99% of Your Traces  Are (Probably) Trash (SRECon NA 2024).pdf
99.99% of Your Traces Are (Probably) Trash (SRECon NA 2024).pdfPaige Cruz
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
100+ ChatGPT Prompts for SEO Optimization
100+ ChatGPT Prompts for SEO Optimization100+ ChatGPT Prompts for SEO Optimization
100+ ChatGPT Prompts for SEO Optimizationarrow10202532yuvraj
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
Governance in SharePoint Premium:What's in the box?
Governance in SharePoint Premium:What's in the box?Governance in SharePoint Premium:What's in the box?
Governance in SharePoint Premium:What's in the box?Juan Carlos Gonzalez
 

Recently uploaded (20)

Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
IEEE Computer Society’s Strategic Activities and Products including SWEBOK Guide
IEEE Computer Society’s Strategic Activities and Products including SWEBOK GuideIEEE Computer Society’s Strategic Activities and Products including SWEBOK Guide
IEEE Computer Society’s Strategic Activities and Products including SWEBOK Guide
 
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
 
99.99% of Your Traces Are (Probably) Trash (SRECon NA 2024).pdf
99.99% of Your Traces  Are (Probably) Trash (SRECon NA 2024).pdf99.99% of Your Traces  Are (Probably) Trash (SRECon NA 2024).pdf
99.99% of Your Traces Are (Probably) Trash (SRECon NA 2024).pdf
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
100+ ChatGPT Prompts for SEO Optimization
100+ ChatGPT Prompts for SEO Optimization100+ ChatGPT Prompts for SEO Optimization
100+ ChatGPT Prompts for SEO Optimization
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
Governance in SharePoint Premium:What's in the box?
Governance in SharePoint Premium:What's in the box?Governance in SharePoint Premium:What's in the box?
Governance in SharePoint Premium:What's in the box?
 

Experience with Formal Methods, Especially Sequential Equivalence Checking

  • 1. Systems and Technology Group March 5, 2007 © 2007 IBM Corporation Experience with Formal Methods, Especially Sequential Equivalence Checking Mark Firstenberg firsten@us.ibm.com
  • 2. Systems and Technology Group © 2007 IBM Corporation2 March 5, 2007 From Bob Fredieu’s Talk Statements Comments ‘Current Trends: Boredom’ ‘What has not helped: Formal Verification – Can’t get people to do it’ What?! Explore Something New! ‘Don’t Waste Money’ Pursue Efficiency/ Improve Quality Wrong Expectations? Benefits Unknown? Purpose of this talk: Provide a user’s perspective of Formal Methods
  • 3. Systems and Technology Group © 2007 IBM Corporation3 March 5, 2007 Why Consider Formal Methods? Formal methods are preferable to pattern based simulation since formal proofs: – Provide complete coverage while pattern based simulation does not, – Require a fraction of the computer resources pattern based simulation does, and – Require fewer human resources than pattern based simulation does. At least that is the theory . . .
  • 4. Systems and Technology Group © 2007 IBM Corporation4 March 5, 2007 Verification Process: New Design Layout Schematic RTL Spec. Logic Designer Circuit Designer Layout Designer Pattern-Based SimulationFormal Methods RTL-Level Simulation Gate-Level Simulation Model Checking Equivalence Checking LVS (Semi-Formal Methods) Design Activity
  • 5. Systems and Technology Group © 2007 IBM Corporation5 March 5, 2007 Verification Process: New Design (notes) Overall task is to verify that the design specification and the final product (i.e. layout) are functionally equivalent Since the level of abstraction between specification and final product is too great, break the design task into activities which produce functionally equivalent representations: – Logic designer translates the specification to RTL by hand – Circuit designer (or synthesizer) translates RTL to schematics – Layout designer (or layout tool) translates schematics to layout Then verify that adjacent design representations are functionally equivalent
  • 6. Systems and Technology Group © 2007 IBM Corporation6 March 5, 2007 Verification Process: New Design (notes) Pattern-Based Simulation Methods – RTL-Level Simulation • Required if the specification is not machine readable (typically the case) – Gate-Level Simulation • Can be used for RTL to schematic comparison, but formal methods are preferable • Typically used for POR sequence testing
  • 7. Systems and Technology Group © 2007 IBM Corporation7 March 5, 2007 Verification Process: New Design (notes) Formal Methods – Model Checking • Used to prove specific assertions about the design • Inherently incomplete (in that all interesting assertions cannot be specified) so is used as a backup to RTL-Level Simulation – Equivalence Checking for RTL to schematic comparison – LVS for schematic to layout comparison – Semi-Formal Methods • Accelerates triggering RTL based assertions • Is not a pure formal method since it uses RTL-Level Simulation to traverse portions of the state space
  • 8. Systems and Technology Group © 2007 IBM Corporation8 March 5, 2007 Verification Process: Derivative Design Layout Schematic RTL Spec. Layout Schematic RTL Spec. Golden Design Derivative Design RTL-to-RTL Equivalence Checking RTL-Level Simulation RTL-Level Simulation Equivalence Checking LVS
  • 9. Systems and Technology Group © 2007 IBM Corporation9 March 5, 2007 Verification Process: Derivative Design (notes) Previously verified ‘golden design’ used as a starting point Use primary verification methods for adjacent derivative design representations: – RTL-Level Simulation for specification to RTL comparison – Equivalence Checking for RTL to schematic comparison – LVS for schematic to layout comparison Bridging the gap between ‘golden’ and ‘derivative’ designs: – Either migrate ‘golden model’ RTL-Level Simulation environment and tests to derivative design RTL-Level Simulation environment (effort may be significant depending on type of changes in the derivative design) – Or perform RTL (golden) to RTL (derivative) Equivalence Checking taking derivative design changes into account
  • 10. Systems and Technology Group © 2007 IBM Corporation10 March 5, 2007 Equivalence Checking Base Definitions Equivalence Checking: The process of comparing two design representations to ensure that their digital function is the same. Note that equivalence checking says nothing about functional correctness! Logic Cone: The collection of non-state device logic which feeds a single state device within a design or a single primary output of a design. Logic cone inputs are state device outputs and/or design primary inputs. Boolean Function: The function of a single logic cone. Sequential Behavior: The digital behavior of a sequence of logic cones and any intervening state devices (i.e. multi-cycle behavior) as observed at the output of the sequential path. Register Re-timing: The movement of logic across state devices which preserves sequential behavior.
  • 11. Systems and Technology Group © 2007 IBM Corporation11 March 5, 2007 Formal Equivalence Checking Types Boolean Equivalence Checking: The process of comparing corresponding logic cones from two different design representations to ensure that their digital function is the same. Sequential Equivalence Checking: The process of comparing corresponding sequential path outputs from two different design representations to ensure that their digital function is the same.
  • 12. Systems and Technology Group © 2007 IBM Corporation12 March 5, 2007 Boolean vs Sequential Equivalence Checking Example A(31:0) B(31:0) Sum(31:0) Impl. 1 ScanIn ScanOut R e g A R e g B R e g S A D D B(31:0) Sum(31:0) Impl. 2 ScanIn ScanOut R e g S A D D A(31:0) R e g X Two Cycle Adder Two Cycle Adder Re-timed
  • 13. Systems and Technology Group © 2007 IBM Corporation13 March 5, 2007 Boolean vs Sequential Equivalence Checking Example (notes) Boolean equivalence will detect mismatches at all comparison points – Impl. 1 RegA and RegB versus Impl. 2 RegX, because • The logic cones feeding the registers have changed. • If the size of RegA plus the size of RegB does not equal the size of RegX, there will be extra state devices in either Impl. 1 or Impl. 2 without corresponding state devices in the other design. – Impl. 1 and Impl. 2 RegS’s (and thus their Sum’s) will mismatch because of logic cone differences. – Impl. 1 and Impl. 2 ScanOut’s will mismatch because of the RegS differences.
  • 14. Systems and Technology Group © 2007 IBM Corporation14 March 5, 2007 Boolean vs Sequential Equivalence Checking Example (notes) Sequential equivalence would find the two designs equivalent except for the ScanOut – The two Sum’s are functionally equivalent and – The two ScanOut’s are different since the logic cone differences feeding the RegS’s are visible during scan chain shifting. – Since only primary outputs are compared, a mismatch in state devices between the two designs does not necessarily matter. – Initial values in the ‘Reg’s could cause mismatches during the first two cycles, so initial state device values matter! If scan operations were disabled, the two designs would be sequentially equivalent Note that this includes sequential path latency
  • 15. Systems and Technology Group © 2007 IBM Corporation15 March 5, 2007 Boolean vs Sequential Equivalence Checking Comparison Boolean equivalence checking is ‘stricter’ and more microscopic – Register re-timing is reported as a mismatch. – State device correspondence is required. – Design constraints cannot propagate through state devices. – State device initialization does not need to be considered. Sequential equivalence checking takes a more macroscopic view of the design – Register re-timing is not reported as a mismatch. – State device correspondence is not required. – Design constraints applied at primary inputs propagate. – State device initialization does need to be considered.
  • 16. Systems and Technology Group © 2007 IBM Corporation16 March 5, 2007 Semi-Formal Methods Combination of pattern-based simulation and formal methods – Simulate to get deep into design’s state space – Then use formal methods to reach design assertions – Accelerate determination if assertion can be triggered Since simulation is involved, method is not purely formal (diagram from 9/12/05 SixthSense presentation, Jason Baumgartner, et. al.)
  • 17. Systems and Technology Group © 2007 IBM Corporation17 March 5, 2007 Building a Sequential Equivalence Checker Using Formal/Semi-Formal Methods Semi-Formal Methods use formal methods to determine if design assertions can be triggered If the assertion is a checker comparing corresponding outputs of two design representations, such a tool could be used to formally prove design equivalence (once the dependence on pattern-based simulation is eliminated) Since the formal methods are designed to propagate sequentially from an initial state, the result would be a Sequential Equivalence Checker
  • 18. Systems and Technology Group © 2007 IBM Corporation18 March 5, 2007 Sequential Equivalence Checking Test Bench ‘old’ design ‘new’ design Primary Input Stimulus Generator Primary Output Compare User supplied mapping and constraint information
  • 19. Systems and Technology Group © 2007 IBM Corporation19 March 5, 2007 The user supplies: – ‘old’ design – ‘new’ design – Mapping and constraint information Sequential equivalence checking tool automatically generates the test bench – Correlates output pins, connecting them to checkers – Correlates input pins, driven from a stimulus generator – Both output checkers and stimulus generator can be modified via user supplied signal mappings and input constraints Sequential Equivalence Checking Test Bench (notes)
  • 20. Systems and Technology Group © 2007 IBM Corporation20 March 5, 2007 Input/Output Pin Correlation By default, input/output pins from both designs are paired based on pin names For any uncorrelated input pins: – Test bench allowed to stimulate them with different values – Could lead to ‘false mismatches’ (which are annoying!) – Fix/waive such warnings (via mapping file) before pursuing mismatches For any uncorrelated output pins: – Test bench eliminates the pins – Could lead to ‘false matches’ (which are not acceptable!) – Fix/waive such warnings (via mapping file) before declaring equivalence
  • 21. Systems and Technology Group © 2007 IBM Corporation21 March 5, 2007 Black Boxing Design Sections PIs POsBBPIs BBPOs Parent Unit X Sub-Unit Y Black-boxed Sub-Unit Y
  • 22. Systems and Technology Group © 2007 IBM Corporation22 March 5, 2007 Black Boxing Design Sections (notes) After ‘Sub-Unit Y’ has been ‘black boxed’, ‘Parent Unit X’s: – Primary inputs will be the combination of ‘X’s Primary Inputs (PIs) and ‘Y’s Black Box Primary Outputs (BBPOs) – Primary outputs will be the combination of ‘X’s Primary Outputs (POs) and ‘Y’s Black Box Primary Inputs (BBPIs) ‘Black Boxing’ helps alleviate tool capacity issues – Reduces amount of logic being checked ‘Black Boxing’ increases complexity in other ways – Increases number of items to prove (primary outputs) – Increases number of items to control (primary inputs) – Requires separate verification of the ‘black boxed’ unit – Creates another equivalence checking boundary which needs to be maintained
  • 23. Systems and Technology Group © 2007 IBM Corporation23 March 5, 2007 Sequential Equivalence Checking Debug Phases mismatches? ‘unsolved’s? Run Comparison no Fix Problem yes Create Custom ‘.config’ file, etc. Done no yes Verify Constraints yes failures? no
  • 24. Systems and Technology Group © 2007 IBM Corporation24 March 5, 2007 Sequential Equivalence Checking Debug Phases (notes) Comparison can produce three results for each ‘property’ – ‘Mismatch’: designer action required – ‘Match’: equivalence proven – ‘Unsolved’: incomplete proof (neither ‘mismatch’ or ‘match’) Debug Phases 1. Designer resolves mismatches (via design or constraints changes) 2. Application Engineer resolves ‘unsolved’s (via ‘.config’ files, if not other methods) 3. Designer resolves incorrect constraints Debug activities can be intermixed . . .
  • 25. Systems and Technology Group © 2007 IBM Corporation25 March 5, 2007 Debugging Sequential Equivalence Checking Mismatches Trace (.aet file) is created for each mismatch: – OUTPUT_MISMATCH shows when failure occurred – Corresponding NEW and OLD model signals available – Only as many cycles as needed are displayed . . . – . . . making it easy to trace with little or no design knowledge!
  • 26. Systems and Technology Group © 2007 IBM Corporation26 March 5, 2007 XBOX 360 Processor Follow-on: Description Goals – Cost reduction of a multi-processor SOC – Maintain ‘mission’ function and performance Design Methods – Process technology transfer – 100+ changes to ‘pervasive’ (IBM term for ‘POR/DFT/debug’) function – HDL: VHDL Verification Methods – Pervasive function: pattern based simulation – Mission function: combination of sequential equivalence checking and pattern based simulation (from previous design) – Performance: sequential equivalence
  • 27. Systems and Technology Group © 2007 IBM Corporation27 March 5, 2007 XBOX 360 Processor Follow-on: ‘Latest is Greatest’ Comparisons Golden Model Model A Model B Model C time
  • 28. Systems and Technology Group © 2007 IBM Corporation28 March 5, 2007 XBOX 360 Processor Follow-on: ‘Latest is Greatest’ Comparison (notes) Always compare back to the ‘golden model’ – Problems (and solutions) accumulate • A difficult problem can’t be avoided (unless designed away) • Solutions must be carried forward (unless designed away) – Always a desire to get to latest model . . . – . . . more like traditional verification Configuration management is less of an issue (mainly needed for ‘golden model’)
  • 29. Systems and Technology Group © 2007 IBM Corporation29 March 5, 2007 XBOX 360 Processor Follow-on: Results Sequential equivalence was used to compare the entire design – ~25 design sections and ~25 arrays – Only the physical layer of the I/O unit (PHY macro) was not compared (since the old and new designs did not split the analog and digital functions in the same way, so comparison was not possible) Design issues uncovered – Removal of piping latches detected (i.e. latency difference) – Undocumented change discovered Other discoveries – Numerous signal constraint/documentation errors – Differences between ‘pervasive’ and ‘mission’ simulation environment’s use of POR state . . .
  • 30. Systems and Technology Group © 2007 IBM Corporation30 March 5, 2007 XBOX 360 Processor Follow-on: Lessons Learned Only ‘black box’: – At agreed upon equivalence checking boundaries – When forced to because of tool capacity issues Discovering uncorrelated inputs via debugging is painful! The driver of a signal knows more about how to constrain that signal than the consumer does, but the consumer does the debugging (and thus proposes the constraint) Must verify all constraints (i.e. equivalence checking assumptions), using either: – Formal Methods, or – Simulation assertions Don’t wait till the last minute to start using POR based state device initialization We needed POR results (for state device initialization) but POR is a pervasive function (which was subject to change)!
  • 31. Systems and Technology Group © 2007 IBM Corporation31 March 5, 2007 PowerPC 464FP FPU: Description Goals – Cycle time improvement of an FPU – Use the same process technology – Maintain ‘mission’ function Design Methods – Significant register re-timing – Micro-architectural changes (while maintaining operation latency) – HDL: Verilog Verification Methods – Original design verified with many directed tests, with unknown coverage . . . – Sequential equivalence (to ‘hold the line’) – Pattern based simulation (concentrating on random testing)
  • 32. Systems and Technology Group © 2007 IBM Corporation32 March 5, 2007 PowerPC 464FP FPU: ‘Step-Wise’ Comparisons Golden Model Model A Model B Model C time
  • 33. Systems and Technology Group © 2007 IBM Corporation33 March 5, 2007 PowerPC 464FP FPU: ‘Step-Wise’ Comparisons (notes) Based on ‘A==B, B==C, therefore A==C’ mentality – Problems (and solutions) can be isolated • Usually trying to isolate design changes which will be difficult to verify (ex: register re-timings) • Assumes difficult design changes are staged independently! – Comparisons do not have to be resolved in order and debug can progress in parallel – Model patches may be required . . . Strong configuration management required – Constantly going back to old models (not just the ‘golden model’)! – Had better be versioning libraries and tools . . .
  • 34. Systems and Technology Group © 2007 IBM Corporation34 March 5, 2007 PowerPC 464FP FPU: Lessons Learned Since there is less pressure to keep up, verification can quickly fall behind Accidentally disabling all clocks is a great way to get everything to match ;^) – Little warning is given for this mistake (other than the comparison job runs very quickly), although a new option is being added to fix this – Leads back to the importance of verifying constraints! Although state device correlation is not required, it does help . . . . . . constantly renaming state devices and/or using the same state device names when changing function can be very disruptive! Success with sequential equivalence checking is related more closely to the nature of the design (ex: arithmetic versus control) and the changes being made (ex: register re-timing versus DFT changes) than to design size (ex: multi-processor SOC versus FPU)
  • 35. Systems and Technology Group © 2007 IBM Corporation35 March 5, 2007 Per Process Diagram portals_ <hdl> portals_ <hdl> portals_ <hdl> portals_ <hdl> portals_ <hdl> chip.proto unitX.proto unit1.proto . . . sub-unitY.proto portals_ <hdl> sub-unit1.proto . . . chip.<hdl> unitX.<hdl> sub-unitY.<hdl> portals_ <hdl> . . . testbench.<hdl> sub-unitN.proto unit1.proto sub-unitN.protosub-unit1.proto testbench.proto mesabld runTest.pl testbench.m2msmod test.aet test.source run_aet_ to_reset merge_ reset_files insert.reset POR.reset equiv_check -no_verify POR.reset.rawchip.sxs_latch_list chip.black_box.lst cycle # Database Build Simulate Per Comparison Reset File Generation
  • 36. Systems and Technology Group © 2007 IBM Corporation36 March 5, 2007 Process Flow: Per Design Targets portals_ <hdl> portals_ <hdl> portals_ <hdl> portals_ <hdl> portals_ <hdl> chip.proto unitX.proto unit1.proto . . . sub-unitY.proto portals_ <hdl> sub-unit1.proto . . . chip.<hdl> unitX.<hdl> sub-unitY.<hdl> portals_ <hdl> . . . testbench.<hdl> sub-unitN.proto unit1.proto sub-unitN.protosub-unit1.proto testbench.proto mesabld runTest.pl testbench.m2msmod test.aet test.source run_aet_ to_reset merge_ reset_files insert.reset POR.reset equiv_check -no_verify POR.reset.rawchip.sxs_latch_list chip.black_box.lst cycle # Database Build Simulate Per Comparison Reset File Generation
  • 37. Systems and Technology Group © 2007 IBM Corporation37 March 5, 2007 Process Flow: Per Comparison Targets equiv_check summarize_ equiv_check _results equiv_check summarize_ equiv_check _results equiv_check summarize_ equiv_check _results old_vs_new_xyz_equiv_check.rpt old_xyz_self_equiv.rpt new_xyz_self_equiv.rptnew_xyz.proto old_xyz.proto old_xyz.proto new_xyz.proto xyz.mapping old_xyz.POR.reset new_xyz.POR.reset Main Equivalence Checking mapping2 bugspray xyz_bugspray.vhdl Optional Self Equivalence Checking
  • 38. Systems and Technology Group © 2007 IBM Corporation38 March 5, 2007 Process Flow (notes) Significant process steps to support main point tool (equiv_check) execution Generated ‘makefile’ based flow – Converts a design ‘hierarchical definition’ (.hd) file into a ‘makefile’ containing targets which perform analysis tasks on that design • Reproducible and reliable (execute only required targets) • Proper sequencing and parallel execution of targets • Build avoidance (don’t re-execute up to date targets) • Scalable and easily modified (via ‘.hd’ file changes) – User can choose to know as much or as little about target specifics as he/she wants
  • 39. Systems and Technology Group © 2007 IBM Corporation39 March 5, 2007 Process Flow (notes) Tool catalog – Standard IBM internal tools • Support mixing VHDL and Verilog source files during database builds (portals_<hdl>) • Cycle-Based Simulation build (mesabld) • Waveform to reset file conversion (run_aet_to_reset) – Project specific tools • Running simulation test (runTest.pl) – SixthSense flow tools • Makefile (flow control) generator (hd2make – not depicted) • ‘.reset’ file merging/checking (merge_reset_files) • ‘equiv_check’ results checking (summarize_equiv_check_results) • Constraint to assertion converter (mapping2bugspray) – SixthSense • ‘No verify’ test bench compile (equiv_check –no_verify) • Test bench compile and verify (equiv_check)
  • 40. Systems and Technology Group © 2007 IBM Corporation40 March 5, 2007 Summary Pursue Formal Methods . . . . . . as long as you don’t expect them to work perfectly right out of the box or without some flow development! Formal Methods require the thoroughness that verification engineers naturally possess Analyzing a design from a different perspective is an effective means of uncovering new problems Embrace Assertion Based Verification – Encourage designers to document their RTL (what a concept!) – Enables formal methods No excuse to be bored ;^)
  • 41. Systems and Technology Group © 2007 IBM Corporation41 March 5, 2007 Tool References SixthSense – (Semi-)Formal Tool and Sequential Equivalence Checker – IBM internal tool (www.research.ibm.com/sixthsense) Calypto SLEC – Sequential Equivalence Checker – Commercial EDA tool (www.calypto.com) 0-In – Semi-Formal Tool – Commercial EDA tool (www.mentor.com) Magellan – Semi-Formal Tool – Commercial EDA tool (www.synopsys.com) Incisive Formal Verifier – Assertion Based Verification – Commercial EDA tool (www.cadence.com)