Heuristic Stimuli GenerationHeuristic Stimuli GenerationFor Coverage ClosureFor Coverage ClosureExploiting Simulation Feed...
GOALGOAL• To propose a methodology for coverage-directed stimuli generation based onsimulation feedback• Such stimuli coul...
AcknowledgementsAcknowledgements• Danilo Ravotto• Ernesto Sanchez• Matteo Sonza Reorda• Alberto Tonda• + many othersDVClub...
OutlineOutline• Proposed methodology• Case studies• ConclusionsDVClub 2010-01-18 giovanni.squillero@polito.it 4
Design ChoicesDesign Choices• Being able to tackle real problems• Develop a versatile and broadly applicablemethodology– C...
Feedback-Based ApproachFeedback-Based Approach• Simulation-based approach• Exploits feedback from simulation• Incremental ...
Proposed MethodologyProposed MethodologyDVClub 2010-01-18 giovanni.squillero@polito.it 7StimuliGeneratorSystemStimuliFeedb...
StimuliStimuliDVClub 2010-01-18 giovanni.squillero@polito.it 8StimuliGeneratorSystemStimuliFeedback
StimuliStimuli• Sequences of bits• Sequences of keys• Full fledged assembly language programs• VHDL test case• External wo...
Stimuli GeneratorStimuli GeneratorDVClub 2010-01-18 giovanni.squillero@polito.it 10StimuliGeneratorSystemStimuliFeedback
Stimuli GeneratorStimuli Generator• Exploit an Evolutionary Algorithm to generatestimuli to maximize a given functionDVClu...
Evolutionary AlgorithmsEvolutionary Algorithms• Meta-heuristic optimization algorithm basedon the concept of population an...
Evolutionary AlgorithmsEvolutionary Algorithms• Succession of random and deterministic steps– A systematic way of throwing...
Evolutionary AlgorithmsEvolutionary Algorithms• Population– Multiple solutions considered in each step– More resistant tha...
Evolutionary AlgorithmsEvolutionary Algorithms• Why using an evolutionary algorithm?– Adaptative– Able to find unexpected ...
Evolutionary AlgorithmsEvolutionary Algorithms• Problem: Fitness function– GOAL: Optimize the wheel– FITNESS: Minimize the...
Evolutionary AlgorithmsEvolutionary Algorithms• Problem: Black magicDVClub 2010-01-18 giovanni.squillero@polito.it 17
µGP (MicroGP)µGP (MicroGP)• CAD Group general-purpose evolver– 3 versions (only 2 released under GPL)– Project started in ...
µGP (MicroGP)µGP (MicroGP)Evolutionary Optimization: the µGPtoolkitE. Sanchez, M. Schillaci, G. SquilleroSpringer, 2010ISB...
µGP (MicroGP)µGP (MicroGP)DVClub 2010-01-18 giovanni.squillero@polito.it 20http://ugp3.sourceforge.net/MicroGP++ (aka. ugp...
System & FeedbackSystem & FeedbackDVClub 2010-01-18 giovanni.squillero@polito.it 21StimuliGeneratorSystemStimuliFeedback
SystemSystem• Strongly problem dependant• Model via simulation/emulation– HDL (netlist to high-level)– HW accelerated (e.g...
Feedback (examples)Feedback (examples)• From simulation– Code coverage metrics (e.g., instruction coverage)– HW specific m...
OutlineOutline• Proposed methodology• Case studies• ConclusionsDVClub 2010-01-18 giovanni.squillero@polito.it 24
Design VerificationDesign Verification• Devise a set of programs maximizing differentcoverage metricsDVClub 2010-01-18 gio...
FeedbackFeedback• Code coverage metrics– Statement coverage (SC)– Branch coverage (BC)– Condition coverage CC)– Expression...
SystemSystem• DLX/pII– Cleaned and simplified MIPS intended primarilyfor teaching purposes– 32-bit load/store architecture...
Experimental ResultsExperimental ResultsDVClub 2010-01-18 giovanni.squillero@polito.it 28
Experimental ResultsExperimental ResultsDVClub 2010-01-18 giovanni.squillero@polito.it 29230Kinstructions2,978instructions
Experimental ResultsExperimental ResultsDVClub 2010-01-18 giovanni.squillero@polito.it 3020h 33h27h37h95h≈1 week
Post-silicon VerificationPost-silicon Verification• Generate functional test programs for post-silicon verification• Activ...
System & FeedbackSystem & Feedback• Real Pentium 4 (no simulation)• Performance counters as metric– Introduced in 1993 in ...
Experimental ResultsExperimental Results• Two sets of experiments:– Mispredicted branches over the total branches– Clock c...
Mobile PhoneMobile Phone• 12m-activity performed in collaboration withMotorola Research Center in Turin(“Automation of Cur...
SystemSystem• Real prototypical mobile phone– P2K platform– GSM/3G networks– Complex applications• Radio Communication Ana...
StimuliStimuli• Sequence of keys• Power supply• Network statusDVClub 2010-01-18 giovanni.squillero@polito.it 36
FeedbackFeedback• Physical measures– Power consumption (+ autocorrelation)– Network status• FSM transition coverageDVClub ...
Experimental ResultsExperimental Results• Major bugs discovered in the prototype– Test infrastructure– LCD display– Fake d...
OutlineOutline• Proposed methodology• Case studies• ConclusionsDVClub 2010-01-18 giovanni.squillero@polito.it 39
ConclusionsConclusions• Simulation-based & feedback-based• Evolutionary algorithm• Broadly applicable• Human resources– Li...
ConclusionsConclusions• May exploit other methodologies– Useful starting point– Effective completion• May be coupled with ...
Heuristic Stimuli Generation for Coverage Closure Exploiting Simulation Feedback
Upcoming SlideShare
Loading in …5
×

Heuristic Stimuli Generation for Coverage Closure Exploiting Simulation Feedback

292 views

Published on

Published in: Technology, Design
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
292
On SlideShare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Heuristic Stimuli Generation for Coverage Closure Exploiting Simulation Feedback

  1. 1. Heuristic Stimuli GenerationHeuristic Stimuli GenerationFor Coverage ClosureFor Coverage ClosureExploiting Simulation FeedbackExploiting Simulation FeedbackGiovanni SquilleroPolitecnico di Torino - ItalyCAD Group ( )giovanni.squillero@polito.it
  2. 2. GOALGOAL• To propose a methodology for coverage-directed stimuli generation based onsimulation feedback• Such stimuli could be added as new content toimprove existing validation suitesDVClub 2010-01-18 giovanni.squillero@polito.it 2
  3. 3. AcknowledgementsAcknowledgements• Danilo Ravotto• Ernesto Sanchez• Matteo Sonza Reorda• Alberto Tonda• + many othersDVClub 2010-01-18 giovanni.squillero@polito.it 3
  4. 4. OutlineOutline• Proposed methodology• Case studies• ConclusionsDVClub 2010-01-18 giovanni.squillero@polito.it 4
  5. 5. Design ChoicesDesign Choices• Being able to tackle real problems• Develop a versatile and broadly applicablemethodology– Compatible with different environment– Compatible with any coverage metric• Minimize effort to change goal/target– Exploit common aspectsDVClub 2010-01-18 giovanni.squillero@polito.it 5
  6. 6. Feedback-Based ApproachFeedback-Based Approach• Simulation-based approach• Exploits feedback from simulation• Incremental improvement/refinement of thesolution (trial-and-error)• Trade-off between computational resourcesand confidence• May exploit heuristics or problem-specificknowledgeDVClub 2010-01-18 giovanni.squillero@polito.it 6
  7. 7. Proposed MethodologyProposed MethodologyDVClub 2010-01-18 giovanni.squillero@polito.it 7StimuliGeneratorSystemStimuliFeedback
  8. 8. StimuliStimuliDVClub 2010-01-18 giovanni.squillero@polito.it 8StimuliGeneratorSystemStimuliFeedback
  9. 9. StimuliStimuli• Sequences of bits• Sequences of keys• Full fledged assembly language programs• VHDL test case• External world• …DVClub 2010-01-18 giovanni.squillero@polito.it 9
  10. 10. Stimuli GeneratorStimuli GeneratorDVClub 2010-01-18 giovanni.squillero@polito.it 10StimuliGeneratorSystemStimuliFeedback
  11. 11. Stimuli GeneratorStimuli Generator• Exploit an Evolutionary Algorithm to generatestimuli to maximize a given functionDVClub 2010-01-18 giovanni.squillero@polito.it 11
  12. 12. Evolutionary AlgorithmsEvolutionary Algorithms• Meta-heuristic optimization algorithm basedon the concept of population and exploitingsome principles of natural evolutionDVClub 2010-01-18 giovanni.squillero@polito.it 12
  13. 13. Evolutionary AlgorithmsEvolutionary Algorithms• Succession of random and deterministic steps– A systematic way of throwing dices– Better than pure random“The great effect produced by theaccumulation in one direction,during successive generations,of differences absolutely inappreciableby an uneducated eye”DVClub 2010-01-18 giovanni.squillero@polito.it 13
  14. 14. Evolutionary AlgorithmsEvolutionary Algorithms• Population– Multiple solutions considered in each step– More resistant than pure hill-climbing– Different solutions may interbreedDVClub 2010-01-18 giovanni.squillero@polito.it 14
  15. 15. Evolutionary AlgorithmsEvolutionary Algorithms• Why using an evolutionary algorithm?– Adaptative– Able to find unexpected solutions– Better than randomBetter than randomDVClub 2010-01-18 giovanni.squillero@polito.it 15
  16. 16. Evolutionary AlgorithmsEvolutionary Algorithms• Problem: Fitness function– GOAL: Optimize the wheel– FITNESS: Minimize the number of “bumps”DVClub 2010-01-18 giovanni.squillero@polito.it 16
  17. 17. Evolutionary AlgorithmsEvolutionary Algorithms• Problem: Black magicDVClub 2010-01-18 giovanni.squillero@polito.it 17
  18. 18. µGP (MicroGP)µGP (MicroGP)• CAD Group general-purpose evolver– 3 versions (only 2 released under GPL)– Project started in 2002– 11 developers + contractors, students, …• Current version– ≈ 300 file, > 40,000 lines in C++DVClub 2010-01-18 giovanni.squillero@polito.it 18
  19. 19. µGP (MicroGP)µGP (MicroGP)Evolutionary Optimization: the µGPtoolkitE. Sanchez, M. Schillaci, G. SquilleroSpringer, 2010ISBN: 978-0-387-09425-0DVClub 2010-01-18 giovanni.squillero@polito.it 19
  20. 20. µGP (MicroGP)µGP (MicroGP)DVClub 2010-01-18 giovanni.squillero@polito.it 20http://ugp3.sourceforge.net/MicroGP++ (aka. ugp3, µGP3)• Information• Download• Credits
  21. 21. System & FeedbackSystem & FeedbackDVClub 2010-01-18 giovanni.squillero@polito.it 21StimuliGeneratorSystemStimuliFeedback
  22. 22. SystemSystem• Strongly problem dependant• Model via simulation/emulation– HDL (netlist to high-level)– HW accelerated (e.g., exploiting FPGA)– Architectural simulator– ISA simulator• Real deviceDVClub 2010-01-18 giovanni.squillero@polito.it 22
  23. 23. Feedback (examples)Feedback (examples)• From simulation– Code coverage metrics (e.g., instruction coverage)– HW specific metrics (e.g., toggle coverage)– High-level information (e.g., FSM coverage)• From the real system– Performance counters– Physical measures (e.g., temperature, time, powerconsumption)DVClub 2010-01-18 giovanni.squillero@polito.it 23
  24. 24. OutlineOutline• Proposed methodology• Case studies• ConclusionsDVClub 2010-01-18 giovanni.squillero@polito.it 24
  25. 25. Design VerificationDesign Verification• Devise a set of programs maximizing differentcoverage metricsDVClub 2010-01-18 giovanni.squillero@polito.it 25
  26. 26. FeedbackFeedback• Code coverage metrics– Statement coverage (SC)– Branch coverage (BC)– Condition coverage CC)– Expression coverage (EC)• HW specific metric– Toggle coverage (TC)DVClub 2010-01-18 giovanni.squillero@polito.it 26
  27. 27. SystemSystem• DLX/pII– Cleaned and simplified MIPS intended primarilyfor teaching purposes– 32-bit load/store architecture, 5-stage pipeline– VHDL RTL description• 4,558 statements• 3,695 branches• 193 conditional statements (1,764 expressions)• 8,283 logic bitsDVClub 2010-01-18 giovanni.squillero@polito.it 27
  28. 28. Experimental ResultsExperimental ResultsDVClub 2010-01-18 giovanni.squillero@polito.it 28
  29. 29. Experimental ResultsExperimental ResultsDVClub 2010-01-18 giovanni.squillero@polito.it 29230Kinstructions2,978instructions
  30. 30. Experimental ResultsExperimental ResultsDVClub 2010-01-18 giovanni.squillero@polito.it 3020h 33h27h37h95h≈1 week
  31. 31. Post-silicon VerificationPost-silicon Verification• Generate functional test programs for post-silicon verification• Activity performed in collaborationwith the ETM Group, Intel (Phoenix)• Intel Pentium 4– 42-55 millions of transistors– 2GHz clock– NetBurst architectureDVClub 2010-01-18 giovanni.squillero@polito.it 31
  32. 32. System & FeedbackSystem & Feedback• Real Pentium 4 (no simulation)• Performance counters as metric– Introduced in 1993 in IA32 architecture– 48 event detectors + 18 programmable counters– Instruction-tagging (for discriminating non-speculative performance events)DVClub 2010-01-18 giovanni.squillero@polito.it 32
  33. 33. Experimental ResultsExperimental Results• Two sets of experiments:– Mispredicted branches over the total branches– Clock cycles in which the trace cache is deliveringµops to the execution unit instead of decoding orbuilding traces (hard metric!)• Intrinsic features of the µarchitectural design• Time (both): 12h/programHaifa 2008 G. Squillero 33
  34. 34. Mobile PhoneMobile Phone• 12m-activity performed in collaboration withMotorola Research Center in Turin(“Automation of Current Drain Measures”)DVClub 2010-01-18 giovanni.squillero@polito.it 34
  35. 35. SystemSystem• Real prototypical mobile phone– P2K platform– GSM/3G networks– Complex applications• Radio Communication Analyzer, anechoicchamber• Controllable power supply• Custom hardwareDVClub 2010-01-18 giovanni.squillero@polito.it 35
  36. 36. StimuliStimuli• Sequence of keys• Power supply• Network statusDVClub 2010-01-18 giovanni.squillero@polito.it 36
  37. 37. FeedbackFeedback• Physical measures– Power consumption (+ autocorrelation)– Network status• FSM transition coverageDVClub 2010-01-18 giovanni.squillero@polito.it 37
  38. 38. Experimental ResultsExperimental Results• Major bugs discovered in the prototype– Test infrastructure– LCD display– Fake deep-sleep stateDVClub 2010-01-18 giovanni.squillero@polito.it 38
  39. 39. OutlineOutline• Proposed methodology• Case studies• ConclusionsDVClub 2010-01-18 giovanni.squillero@polito.it 39
  40. 40. ConclusionsConclusions• Simulation-based & feedback-based• Evolutionary algorithm• Broadly applicable• Human resources– Limited (set-up )• Computational resources– Easily parallelizable (generation level)– Trade-off quality vs. effortDVClub 2010-01-18 giovanni.squillero@polito.it 40
  41. 41. ConclusionsConclusions• May exploit other methodologies– Useful starting point– Effective completion• May be coupled with other methodologies– Rule-based instruction randomizers– Simulation-based approachesDVClub 2010-01-18 giovanni.squillero@polito.it 41

×