DVClub Bristol 22-April-2009




The Verification Methodology
Landscape




Jonathan Bromley, Doulos
                                                                         1
                      Copyright © 2008 by Doulos. All rights reserved.
The Verification
Methodology Landscape



CONTENTS


The M-word

Languages, methodologies, tools and standards

The big players: OVM, VMM, eRM

Interoperability and convergence

Conclusions?
                                                           2
        Copyright © 2008 by Doulos. All rights reserved.
Verification Environment


             Coverage                Scoreboard /                      Transaction
             collector                 Checker                          recording



  Test
controller
                                            Monitor



                                    TLM
Stimulus
                   Driver                                              DUT
generator
                                                                                     Transactions
                                                                                                    3
                    Copyright © 2008 by Doulos. All rights reserved.
Constrained Random Verification

                                          Checker

                                       Did we see a bug?
Constrained random
     stimulus



   11001001                                                                        000010
   01001010                                                                        010011
   00001001                             Design                                     000010
   01110110                             Under                                      100100
   01100110                              Test                                      001000
   01001001                                                                        110010
   01001110                                                                        000011

                                                               Functional
           Constraints                                         Coverage

        Header   Payload   Checksum                                 Are we done?

                     Increase coverage
                                                                                            4
                             Copyright © 2008 by Doulos. All rights reserved.
The Verification Space
                                      Verification

                                                                                      Formal
                                                                                     Verification


Acceleration                                                         Property                   Equivalence
                     Simulation
+ Emulation                                                          Checking                    Checking


                                         Dynamic
                                          Formal
                                                                        Simulation              Simulation
 Simulation                Simulation                                     Formal                  Formal

 Transaction Level         Constrained
                                                                         Assertions             Coverage
     Modelling              Random


                            Intelligent
                            Testbench                                                                         5
                              Copyright © 2008 by Doulos. All rights reserved.
What Our Customers Want
• Ease of deployment
   •   Customizable environment
        •   but it must do something useful straight out of the box
   •   Simple, uniform interface to any verification IP block
   •   Gentle learning curve for the whole team


• Power
   •   Complex testcases co-ordinated across the whole environment
   •   Randomization
   •   Sophisticated coverage analysis


• Interoperability
   •   Every customer we meet has legacy verification IP

                                                                                6
                             Copyright © 2008 by Doulos. All rights reserved.
Methodology (or just a toolkit?)
• Tame the language monster
   •   e, SystemVerilog, C++ are big and complicated
   •   Many ways to solve a problem
   •   Wheel reinvention is a hazard


• Toolkit
   •   OOP encourages encapsulation of standard functionality
       in base classes
   •   Ideal for component hierarchy, reporting, block-to-block
       communication, DUT connectivity



• Methodology
   •   Published methodology encourages interoperability and re-use
   •   Promote best practice
                                                                             7
                          Copyright © 2008 by Doulos. All rights reserved.
Mature Language Standards
List unchanged for 4 years...                                        Crude Caricature

        •        IEEE 1076 VHDL
                                                                   FPGA, RTL, Europe, Mil-Aero
        •    IEEE 1850™ PSL


        •    IEEE 1364 Verilog                                     ASIC, RTL, USA/RoW

•   IEEE 1800™ SystemVerilog                                       Hardware verification



             •    IEEE 1647™ e                                     Hardware verification



        •    ISO/IEC 14882 C++                                     Modelling, verification

    •       IEEE 1666™ SystemC                                      Virtual hardware prototypes for S/W dev



                 •   Tcl/Tk, Perl                                  Scripting
                                                                                                              8
                                    Copyright © 2008 by Doulos. All rights reserved.                          8
New Standards Activity

•   Verilog and SystemVerilog unified
     •   LRM this year, currently in ballot feedback
     •   Major enhancements to assertions


•   Verification methodology
     •   OVM 2.0, OVM-SC
     •   VMM open-source
     •   eRM3 - e / SV interoperability (Cadence)
     •   OVM / VMM interoperability (Accellera)


•   SystemC TLM-2.0
                                                                      9
                   Copyright © 2008 by Doulos. All rights reserved.
The Big Methodology Players


        www.ovmworld.org



        www.vmmcentral.org



eRM     www.cadence.com




                                                              10
           Copyright © 2008 by Doulos. All rights reserved.
Static View of Testbench
                                                                                            Environment

              Configuration                                            checker



eRM                Sequ-                                                           Sequ-
                   encer                                                           encer
                          active                                                       passive
                          agent                                                         agent

                   Driver       Monitor                                            Driver   Monitor
                   BFM           BFM                                               BFM       BFM



          simple ports (e)                                   DUT
      virtual interfaces (SV)
                                                                                                          11
                                Copyright © 2008 by Doulos. All rights reserved.
Static View of Testbench (VMM)

                 ...
                 test_env env = new;
                 env.run();
                 ...

 Constraints;
directed tests
                    Generator               Self Check
    High level
  transactions                                                                                         Functional
                      Transactor                                                           Monitor     Coverage
   Atomic
transactions

 Checker               Driver           Properties                    Checker                Monitor




                                       DUT

                                                                                                                    12
                                        Copyright © 2008 by Doulos. All rights reserved.
OVM Key Features


•   Constrained random, coverage-driven verification
•   Separation of tests from verification environment
•   Configuration of verification environment
     •   through a table

•   Verification IP reuse (canonical structure and guidelines)


•   TLM communication
•   Automation (where missing from SystemVerilog language)
•   Hierarchical sequential stimulus (sequences)
•   Standardized messaging

                                                                              13
                           Copyright © 2008 by Doulos. All rights reserved.
eRM Key Features


•   Constrained random, coverage-driven verification
•   Separation of tests from verification environment
•   Configuration of verification environment
     •   through AOP extension and pre-run constraints

•   Verification IP reuse (rigorously standardized rules)


•   Communication via ports
•   Automation (using e language's macro features)
•   Hierarchical sequential stimulus (sequences)
•   Standardized messaging

                                                                             14
                          Copyright © 2008 by Doulos. All rights reserved.
VMM Key Features


•   Constrained random, coverage-driven verification
•   Configuration of verification environment
     •   through configuration objects passed to verification components

•   Verification IP reuse (conventions)


•   Communication via channels, callbacks, notifications
•   Automation (scripts and macros)
•   Hierarchical sequential stimulus (scenarios)
•   Standardized messaging


•   Strongly influenced by RVM (Synopsys Vera)
                                                                             15
                          Copyright © 2008 by Doulos. All rights reserved.
Structure of an OVM Component
class my_driver extends ovm_driver #(my_transaction);                        Base class
  // ovm_seq_item_pull_port #(...) seq_item_port;                            TLM port (inherited)

  my_dut_if_wrapper m_dut_if;                                                Connection to DUT

  function new(string name, ovm_component parent);
    super.new(name, parent);                                                 Constructor
  endfunction: new

  function void build;
    super.build();                                                           Build phase
  endfunction: build                                                         callback

  virtual task run;
    forever begin
      ...                                                                    Run phase
    end                                                                      callback
  endtask: run
endclass: my_driver

                                                                                                    16
                          Copyright © 2008 by Doulos. All rights reserved.
Phase Methods (OVM)

build                                                              Call factory

connect                                                            Make TLM connections

end_of_elaboration                                                 After connections hardened

start_of_simulation                                                Get ready to run

    run                                                            Task (executed concurrently)

          extract

          check                                                    Post-processing

          report


                  Similar phase arrangements in VMM, eRM
                                                                                                  17
                     Copyright © 2008 by Doulos. All rights reserved.
Reconfigurable Environment (OVM)
 By type or instance path
                                Test                                               Customize environments
                               Test
set_inst_override_*
                                                                      set_config_*



               Reusable verification environment                                         Customize types

        Scoreboard                       Virtual                       set_type_override_*
                                       sequencer


                                                                                            Config          Sequencer

          Monitor            Existing                Existing
                            verification            verification
                            component               component                                Monitor          Driver




                                           DUT
                                                                                                                        18
                                     Copyright © 2008 by Doulos. All rights reserved.
Layered Sequential Stimulus

Tests enumerate possible
top-level sequences



Virtual or layered
                                                                      seq1                      seq2    seq2
sequences




Constrained random
sequence of transactions                                                    tx1                  tx2   tx3


Randomized transactions are not enough



Drive transactions into DUT                              tx1                                Driver           DUT




                                                                                                                   19
                                         Copyright © 2008 by Doulos. All rights reserved.
Virtual Sequences
            Component hierarchy                                            Stimulus hierarchy
                                                                        (co-ordinated interfaces)
                 ovm_env

              ovm_sequencer                                                 ovm_sequence


 ovm_agent                           ovm_agent

ovm_sequencer                    ovm_sequencer                              ovm_sequence


                                                                         ovm_sequence_item

 ovm_driver                           ovm_driver


   dut_if                                 dut_if


                   DUT
                                                                                                    20
                     Copyright © 2008 by Doulos. All rights reserved.
Scenario Generator (VMM)

Verification environment

       Scenario generator

                            scenario_set                              burst
                                                                                      items
                            [0] atomic
        select_scenario
                            [1] burst
             select
                            [2] RMW



                               copies of items


                  generator's output channel
                                                                             Downstream transactor
                                                                                                     21
                                 Copyright © 2008 by Doulos. All rights reserved.
Now and Next

•   VMM: rapidly growing collection of "applications"
     •   register abstraction layer, hierarchy, ...



•   OVM/VMM interoperability toolkits/standards


•   OVM/eRM mixed-language tools


•   OVM-SC


•   Increasing availability of verification IP

                                                                          22
                       Copyright © 2008 by Doulos. All rights reserved.
Conclusion

•   Interesting times
     •   standards don't always keep up with user needs



•   Challenges for users choosing a new approach:
     •   tools?
     •   methodology?
     •   decisions are not yet completely decoupled



•   Training is important:
     •   VMM, OVM, eRM are not difficult ...
     •   ... but jump-starting your efforts pays dividends


                                                                         23
                      Copyright © 2008 by Doulos. All rights reserved.
24
Copyright © 2008 by Doulos. All rights reserved.

Jonathan bromley doulos

  • 1.
    DVClub Bristol 22-April-2009 TheVerification Methodology Landscape Jonathan Bromley, Doulos 1 Copyright © 2008 by Doulos. All rights reserved.
  • 2.
    The Verification Methodology Landscape CONTENTS TheM-word Languages, methodologies, tools and standards The big players: OVM, VMM, eRM Interoperability and convergence Conclusions? 2 Copyright © 2008 by Doulos. All rights reserved.
  • 3.
    Verification Environment Coverage Scoreboard / Transaction collector Checker recording Test controller Monitor TLM Stimulus Driver DUT generator Transactions 3 Copyright © 2008 by Doulos. All rights reserved.
  • 4.
    Constrained Random Verification Checker Did we see a bug? Constrained random stimulus 11001001 000010 01001010 010011 00001001 Design 000010 01110110 Under 100100 01100110 Test 001000 01001001 110010 01001110 000011 Functional Constraints Coverage Header Payload Checksum Are we done? Increase coverage 4 Copyright © 2008 by Doulos. All rights reserved.
  • 5.
    The Verification Space Verification Formal Verification Acceleration Property Equivalence Simulation + Emulation Checking Checking Dynamic Formal Simulation Simulation Simulation Simulation Formal Formal Transaction Level Constrained Assertions Coverage Modelling Random Intelligent Testbench 5 Copyright © 2008 by Doulos. All rights reserved.
  • 6.
    What Our CustomersWant • Ease of deployment • Customizable environment • but it must do something useful straight out of the box • Simple, uniform interface to any verification IP block • Gentle learning curve for the whole team • Power • Complex testcases co-ordinated across the whole environment • Randomization • Sophisticated coverage analysis • Interoperability • Every customer we meet has legacy verification IP 6 Copyright © 2008 by Doulos. All rights reserved.
  • 7.
    Methodology (or justa toolkit?) • Tame the language monster • e, SystemVerilog, C++ are big and complicated • Many ways to solve a problem • Wheel reinvention is a hazard • Toolkit • OOP encourages encapsulation of standard functionality in base classes • Ideal for component hierarchy, reporting, block-to-block communication, DUT connectivity • Methodology • Published methodology encourages interoperability and re-use • Promote best practice 7 Copyright © 2008 by Doulos. All rights reserved.
  • 8.
    Mature Language Standards Listunchanged for 4 years... Crude Caricature • IEEE 1076 VHDL FPGA, RTL, Europe, Mil-Aero • IEEE 1850™ PSL • IEEE 1364 Verilog ASIC, RTL, USA/RoW • IEEE 1800™ SystemVerilog Hardware verification • IEEE 1647™ e Hardware verification • ISO/IEC 14882 C++ Modelling, verification • IEEE 1666™ SystemC Virtual hardware prototypes for S/W dev • Tcl/Tk, Perl Scripting 8 Copyright © 2008 by Doulos. All rights reserved. 8
  • 9.
    New Standards Activity • Verilog and SystemVerilog unified • LRM this year, currently in ballot feedback • Major enhancements to assertions • Verification methodology • OVM 2.0, OVM-SC • VMM open-source • eRM3 - e / SV interoperability (Cadence) • OVM / VMM interoperability (Accellera) • SystemC TLM-2.0 9 Copyright © 2008 by Doulos. All rights reserved.
  • 10.
    The Big MethodologyPlayers www.ovmworld.org www.vmmcentral.org eRM www.cadence.com 10 Copyright © 2008 by Doulos. All rights reserved.
  • 11.
    Static View ofTestbench Environment Configuration checker eRM Sequ- Sequ- encer encer active passive agent agent Driver Monitor Driver Monitor BFM BFM BFM BFM simple ports (e) DUT virtual interfaces (SV) 11 Copyright © 2008 by Doulos. All rights reserved.
  • 12.
    Static View ofTestbench (VMM) ... test_env env = new; env.run(); ... Constraints; directed tests Generator Self Check High level transactions Functional Transactor Monitor Coverage Atomic transactions Checker Driver Properties Checker Monitor DUT 12 Copyright © 2008 by Doulos. All rights reserved.
  • 13.
    OVM Key Features • Constrained random, coverage-driven verification • Separation of tests from verification environment • Configuration of verification environment • through a table • Verification IP reuse (canonical structure and guidelines) • TLM communication • Automation (where missing from SystemVerilog language) • Hierarchical sequential stimulus (sequences) • Standardized messaging 13 Copyright © 2008 by Doulos. All rights reserved.
  • 14.
    eRM Key Features • Constrained random, coverage-driven verification • Separation of tests from verification environment • Configuration of verification environment • through AOP extension and pre-run constraints • Verification IP reuse (rigorously standardized rules) • Communication via ports • Automation (using e language's macro features) • Hierarchical sequential stimulus (sequences) • Standardized messaging 14 Copyright © 2008 by Doulos. All rights reserved.
  • 15.
    VMM Key Features • Constrained random, coverage-driven verification • Configuration of verification environment • through configuration objects passed to verification components • Verification IP reuse (conventions) • Communication via channels, callbacks, notifications • Automation (scripts and macros) • Hierarchical sequential stimulus (scenarios) • Standardized messaging • Strongly influenced by RVM (Synopsys Vera) 15 Copyright © 2008 by Doulos. All rights reserved.
  • 16.
    Structure of anOVM Component class my_driver extends ovm_driver #(my_transaction); Base class // ovm_seq_item_pull_port #(...) seq_item_port; TLM port (inherited) my_dut_if_wrapper m_dut_if; Connection to DUT function new(string name, ovm_component parent); super.new(name, parent); Constructor endfunction: new function void build; super.build(); Build phase endfunction: build callback virtual task run; forever begin ... Run phase end callback endtask: run endclass: my_driver 16 Copyright © 2008 by Doulos. All rights reserved.
  • 17.
    Phase Methods (OVM) build Call factory connect Make TLM connections end_of_elaboration After connections hardened start_of_simulation Get ready to run run Task (executed concurrently) extract check Post-processing report Similar phase arrangements in VMM, eRM 17 Copyright © 2008 by Doulos. All rights reserved.
  • 18.
    Reconfigurable Environment (OVM) By type or instance path Test Customize environments Test set_inst_override_* set_config_* Reusable verification environment Customize types Scoreboard Virtual set_type_override_* sequencer Config Sequencer Monitor Existing Existing verification verification component component Monitor Driver DUT 18 Copyright © 2008 by Doulos. All rights reserved.
  • 19.
    Layered Sequential Stimulus Testsenumerate possible top-level sequences Virtual or layered seq1 seq2 seq2 sequences Constrained random sequence of transactions tx1 tx2 tx3 Randomized transactions are not enough Drive transactions into DUT tx1 Driver DUT 19 Copyright © 2008 by Doulos. All rights reserved.
  • 20.
    Virtual Sequences Component hierarchy Stimulus hierarchy (co-ordinated interfaces) ovm_env ovm_sequencer ovm_sequence ovm_agent ovm_agent ovm_sequencer ovm_sequencer ovm_sequence ovm_sequence_item ovm_driver ovm_driver dut_if dut_if DUT 20 Copyright © 2008 by Doulos. All rights reserved.
  • 21.
    Scenario Generator (VMM) Verificationenvironment Scenario generator scenario_set burst items [0] atomic select_scenario [1] burst select [2] RMW copies of items generator's output channel Downstream transactor 21 Copyright © 2008 by Doulos. All rights reserved.
  • 22.
    Now and Next • VMM: rapidly growing collection of "applications" • register abstraction layer, hierarchy, ... • OVM/VMM interoperability toolkits/standards • OVM/eRM mixed-language tools • OVM-SC • Increasing availability of verification IP 22 Copyright © 2008 by Doulos. All rights reserved.
  • 23.
    Conclusion • Interesting times • standards don't always keep up with user needs • Challenges for users choosing a new approach: • tools? • methodology? • decisions are not yet completely decoupled • Training is important: • VMM, OVM, eRM are not difficult ... • ... but jump-starting your efforts pays dividends 23 Copyright © 2008 by Doulos. All rights reserved.
  • 24.
    24 Copyright © 2008by Doulos. All rights reserved.