0
Heuristic Stimuli Generation
     For Coverage Closure
Exploiting Simulation Feedback
          Giovanni Squillero
       ...
GOAL
• To propose a methodology for coverage-
  directed stimuli generation based on
  simulation feedback
• Such stimuli ...
Acknowledgements
•   Danilo Ravotto
•   Ernesto Sanchez
•   Matteo Sonza Reorda
•   Alberto Tonda

• + many others


DVClu...
Outline
• Proposed methodology
• Case studies
• Conclusions




DVClub 2010-01-18   giovanni.squillero@polito.it   4
Design Choices
• Being able to tackle real problems
• Develop a versatile and broadly applicable
  methodology
      – Com...
Feedback-Based Approach
• Simulation-based approach
• Exploits feedback from simulation
• Incremental improvement/refineme...
Proposed Methodology
                                                                Stimuli




   Stimuli               ...
Stimuli
                                                              Stimuli




   Stimuli                              ...
Stimuli
•   Sequences of bits
•   Sequences of keys
•   Full fledged assembly language programs
•   VHDL test case
•   Ext...
Stimuli Generator
                                                              Stimuli




   Stimuli                    ...
Stimuli Generator
• Exploit an Evolutionary Algorithm to generate
  stimuli to maximize a given function




DVClub 2010-0...
Evolutionary Algorithms
• Meta-heuristic optimization algorithm based
  on the concept of population and exploiting
  some...
Evolutionary Algorithms
• Succession of random and deterministic steps
      – A systematic way of throwing dices
      – ...
Evolutionary Algorithms
• Population
      – Multiple solutions considered in each step
      – More resistant than pure h...
Evolutionary Algorithms
• Why using an evolutionary algorithm?
      – Adaptative
      – Able to find unexpected solution...
Evolutionary Algorithms
• Problem: Fitness function
      – GOAL: Optimize the wheel
      – FITNESS: Minimize the number ...
Evolutionary Algorithms
• Problem: Black magic




DVClub 2010-01-18          giovanni.squillero@polito.it   17
µGP (MicroGP)
• CAD Group general-purpose evolver
      – 3 versions (only 2 released under GPL)
      – Project started i...
µGP (MicroGP)

           Evolutionary Optimization: the µGP
           toolkit
           E. Sanchez, M. Schillaci, G. Sq...
µGP (MicroGP)

               http://ugp3.sourceforge.net/

      MicroGP++ (aka. ugp3, µGP3)
         • Information
     ...
System & Feedback
                                                              Stimuli




   Stimuli                    ...
System
• Strongly problem dependant
• Model via simulation/emulation
      – HDL (netlist to high-level)
      – HW accele...
Feedback (examples)
• From simulation
      – Code coverage metrics (e.g., instruction coverage)
      – HW specific metri...
Outline
• Proposed methodology
• Case studies
• Conclusions




DVClub 2010-01-18   giovanni.squillero@polito.it   24
Design Verification
• Devise a set of programs maximizing different
  coverage metrics




DVClub 2010-01-18        giovan...
Feedback
• Code coverage metrics
      – Statement coverage (SC)
      – Branch coverage (BC)
      – Condition coverage C...
System
• DLX/pII
      – Cleaned and simplified MIPS intended primarily
        for teaching purposes
      – 32-bit load/...
Experimental Results




DVClub 2010-01-18         giovanni.squillero@polito.it   28
Experimental Results
                                                            2,978
                                   ...
Experimental Results
                      20h         33h
                                                       27h     ...
Post-silicon Verification
• Generate functional test programs for post-
  silicon verification
• Activity performed in col...
System & Feedback
• Real Pentium 4 (no simulation)
• Performance counters as metric
      – Introduced in 1993 in IA32 arc...
Experimental Results
• Two sets of experiments:
      – Mispredicted branches over the total branches
      – Clock cycles...
Mobile Phone
• 12m-activity performed in collaboration with
  Motorola Research Center in Turin
  (“Automation of Current ...
System
• Real prototypical mobile phone
      – P2K platform
      – GSM/3G networks
      – Complex applications
• Radio ...
Stimuli
• Sequence of keys
• Power supply
• Network status




DVClub 2010-01-18   giovanni.squillero@polito.it   36
Feedback
• Physical measures
      – Power consumption (+ autocorrelation)
      – Network status
• FSM transition coverag...
Experimental Results
• Major bugs discovered in the prototype
      – Test infrastructure
      – LCD display
      – Fake...
Outline
• Proposed methodology
• Case studies
• Conclusions




DVClub 2010-01-18   giovanni.squillero@polito.it   39
Conclusions
•   Simulation-based & feedback-based
•   Evolutionary algorithm
•   Broadly applicable
•   Human resources
  ...
Conclusions
• May exploit other methodologies
      – Useful starting point
      – Effective completion
• May be coupled ...
2010 bristol q1_heuristic-stimuli-generation
Upcoming SlideShare
Loading in...5
×

2010 bristol q1_heuristic-stimuli-generation

239

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
239
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "2010 bristol q1_heuristic-stimuli-generation"

  1. 1. Heuristic Stimuli Generation For Coverage Closure Exploiting Simulation Feedback Giovanni Squillero Politecnico di Torino - Italy CAD Group ( ) giovanni.squillero@polito.it
  2. 2. GOAL • To propose a methodology for coverage- directed stimuli generation based on simulation feedback • Such stimuli could be added as new content to improve existing validation suites DVClub 2010-01-18 giovanni.squillero@polito.it 2
  3. 3. Acknowledgements • Danilo Ravotto • Ernesto Sanchez • Matteo Sonza Reorda • Alberto Tonda • + many others DVClub 2010-01-18 giovanni.squillero@polito.it 3
  4. 4. Outline • Proposed methodology • Case studies • Conclusions DVClub 2010-01-18 giovanni.squillero@polito.it 4
  5. 5. Design Choices • Being able to tackle real problems • Develop a versatile and broadly applicable methodology – Compatible with different environment – Compatible with any coverage metric • Minimize effort to change goal/target – Exploit common aspects DVClub 2010-01-18 giovanni.squillero@polito.it 5
  6. 6. Feedback-Based Approach • Simulation-based approach • Exploits feedback from simulation • Incremental improvement/refinement of the solution (trial-and-error) • Trade-off between computational resources and confidence • May exploit heuristics or problem-specific knowledge DVClub 2010-01-18 giovanni.squillero@polito.it 6
  7. 7. Proposed Methodology Stimuli Stimuli System Generator Feedback DVClub 2010-01-18 giovanni.squillero@polito.it 7
  8. 8. Stimuli Stimuli Stimuli System Generator Feedback DVClub 2010-01-18 giovanni.squillero@polito.it 8
  9. 9. Stimuli • Sequences of bits • Sequences of keys • Full fledged assembly language programs • VHDL test case • External world • … DVClub 2010-01-18 giovanni.squillero@polito.it 9
  10. 10. Stimuli Generator Stimuli Stimuli System Generator Feedback DVClub 2010-01-18 giovanni.squillero@polito.it 10
  11. 11. Stimuli Generator • Exploit an Evolutionary Algorithm to generate stimuli to maximize a given function DVClub 2010-01-18 giovanni.squillero@polito.it 11
  12. 12. Evolutionary Algorithms • Meta-heuristic optimization algorithm based on the concept of population and exploiting some principles of natural evolution DVClub 2010-01-18 giovanni.squillero@polito.it 12
  13. 13. Evolutionary Algorithms • Succession of random and deterministic steps – A systematic way of throwing dices – Better than pure random “The great effect produced by the accumulation in one direction, during successive generations, of differences absolutely inappreciable by an uneducated eye” DVClub 2010-01-18 giovanni.squillero@polito.it 13
  14. 14. Evolutionary Algorithms • Population – Multiple solutions considered in each step – More resistant than pure hill-climbing – Different solutions may interbreed DVClub 2010-01-18 giovanni.squillero@polito.it 14
  15. 15. Evolutionary Algorithms • Why using an evolutionary algorithm? – Adaptative – Able to find unexpected solutions – Better than random DVClub 2010-01-18 giovanni.squillero@polito.it 15
  16. 16. Evolutionary Algorithms • Problem: Fitness function – GOAL: Optimize the wheel – FITNESS: Minimize the number of “bumps” DVClub 2010-01-18 giovanni.squillero@polito.it 16
  17. 17. Evolutionary Algorithms • Problem: Black magic DVClub 2010-01-18 giovanni.squillero@polito.it 17
  18. 18. µGP (MicroGP) • CAD Group general-purpose evolver – 3 versions (only 2 released under GPL) – Project started in 2002 – 11 developers + contractors, students, … • Current version – ≈ 300 file, > 40,000 lines in C++ DVClub 2010-01-18 giovanni.squillero@polito.it 18
  19. 19. µGP (MicroGP) Evolutionary Optimization: the µGP toolkit E. Sanchez, M. Schillaci, G. Squillero Springer, 2010 ISBN: 978-0-387-09425-0 DVClub 2010-01-18 giovanni.squillero@polito.it 19
  20. 20. µGP (MicroGP) http://ugp3.sourceforge.net/ MicroGP++ (aka. ugp3, µGP3) • Information • Download • Credits DVClub 2010-01-18 giovanni.squillero@polito.it 20
  21. 21. System & Feedback Stimuli Stimuli System Generator Feedback DVClub 2010-01-18 giovanni.squillero@polito.it 21
  22. 22. System • Strongly problem dependant • Model via simulation/emulation – HDL (netlist to high-level) – HW accelerated (e.g., exploiting FPGA) – Architectural simulator – ISA simulator • Real device DVClub 2010-01-18 giovanni.squillero@polito.it 22
  23. 23. Feedback (examples) • From simulation – Code coverage metrics (e.g., instruction coverage) – HW specific metrics (e.g., toggle coverage) – High-level information (e.g., FSM coverage) • From the real system – Performance counters – Physical measures (e.g., temperature, time, power consumption) DVClub 2010-01-18 giovanni.squillero@polito.it 23
  24. 24. Outline • Proposed methodology • Case studies • Conclusions DVClub 2010-01-18 giovanni.squillero@polito.it 24
  25. 25. Design Verification • Devise a set of programs maximizing different coverage metrics DVClub 2010-01-18 giovanni.squillero@polito.it 25
  26. 26. Feedback • Code coverage metrics – Statement coverage (SC) – Branch coverage (BC) – Condition coverage CC) – Expression coverage (EC) • HW specific metric – Toggle coverage (TC) DVClub 2010-01-18 giovanni.squillero@polito.it 26
  27. 27. System • DLX/pII – Cleaned and simplified MIPS intended primarily for teaching purposes – 32-bit load/store architecture, 5-stage pipeline – VHDL RTL description • 4,558 statements • 3,695 branches • 193 conditional statements (1,764 expressions) • 8,283 logic bits DVClub 2010-01-18 giovanni.squillero@polito.it 27
  28. 28. Experimental Results DVClub 2010-01-18 giovanni.squillero@polito.it 28
  29. 29. Experimental Results 2,978 instructions 230K instructions DVClub 2010-01-18 giovanni.squillero@polito.it 29
  30. 30. Experimental Results 20h 33h 27h 95h 37h ≈1 week DVClub 2010-01-18 giovanni.squillero@polito.it 30
  31. 31. Post-silicon Verification • Generate functional test programs for post- silicon verification • Activity performed in collaboration with the ETM Group, Intel (Phoenix) • Intel Pentium 4 – 42-55 millions of transistors – 2GHz clock – NetBurst architecture DVClub 2010-01-18 giovanni.squillero@polito.it 31
  32. 32. System & Feedback • Real Pentium 4 (no simulation) • Performance counters as metric – Introduced in 1993 in IA32 architecture – 48 event detectors + 18 programmable counters – Instruction-tagging (for discriminating non- speculative performance events) DVClub 2010-01-18 giovanni.squillero@polito.it 32
  33. 33. Experimental Results • Two sets of experiments: – Mispredicted branches over the total branches – Clock cycles in which the trace cache is delivering µops to the execution unit instead of decoding or building traces (hard metric!) • Intrinsic features of the µarchitectural design • Time (both): 12h/program Haifa 2008 G. Squillero 33
  34. 34. Mobile Phone • 12m-activity performed in collaboration with Motorola Research Center in Turin (“Automation of Current Drain Measures”) DVClub 2010-01-18 giovanni.squillero@polito.it 34
  35. 35. System • Real prototypical mobile phone – P2K platform – GSM/3G networks – Complex applications • Radio Communication Analyzer, anechoic chamber • Controllable power supply • Custom hardware DVClub 2010-01-18 giovanni.squillero@polito.it 35
  36. 36. Stimuli • Sequence of keys • Power supply • Network status DVClub 2010-01-18 giovanni.squillero@polito.it 36
  37. 37. Feedback • Physical measures – Power consumption (+ autocorrelation) – Network status • FSM transition coverage DVClub 2010-01-18 giovanni.squillero@polito.it 37
  38. 38. Experimental Results • Major bugs discovered in the prototype – Test infrastructure – LCD display – Fake deep-sleep state DVClub 2010-01-18 giovanni.squillero@polito.it 38
  39. 39. Outline • Proposed methodology • Case studies • Conclusions DVClub 2010-01-18 giovanni.squillero@polito.it 39
  40. 40. Conclusions • Simulation-based & feedback-based • Evolutionary algorithm • Broadly applicable • Human resources – Limited (set-up ) • Computational resources – Easily parallelizable (generation level) – Trade-off quality vs. effort DVClub 2010-01-18 giovanni.squillero@polito.it 40
  41. 41. Conclusions • May exploit other methodologies – Useful starting point – Effective completion • May be coupled with other methodologies – Rule-based instruction randomizers – Simulation-based approaches DVClub 2010-01-18 giovanni.squillero@polito.it 41
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×