Automation in ASIC verification generates large amounts of test results that can overload engineers. The document proposes addressing this by using visual analytics through a game engine to help engineers focus verification efforts. It describes a prototype tool built in Unity that visualizes test results across files and commits as an interactive "test execution cityscape". Future work includes evaluating the tool with additional engineers and students.

1. Research Institutes of Sweden
ENABLING VISUAL
ANALYTICS WITH
- Exploring Regression Test Results
in ASIC Verification
Markus Borg, Andreas Brytting, Daniel Hansson
November 17, 2017
Swedish Institute of Computer Science
Software and Systems Engineering Laboratory

2. 2
Automation in ASIC verification generates large
amounts of test results
Visual analytics through a game engine helps
focusing design and verification effort

3. Development engineer, ABB, Malmö, Sweden 2007-2010
 Editor and compiler development
 Process automation
PhD student, Lund University, Sweden 2010-2015
 Machine learning for software engineering
 Bug report management and traceability
Senior researcher, RISE SICS AB, Lund, Sweden 2015-
 Software engineering for machine learning
 V&V for self-driving cars
3
Markus Borg?

4.  MSc thesis student Andreas Brytting
 ASIC verification tool vendor
 Research project
4
Credits

5. 5
WP3 Test effectiveness
Task 2 Test result overload

6.  ASIC design vs. software engineering
 Challenge and solution proposal
 Tool demo
 Evaluation plans
6
Outline

7. 7
ASIC development
ASIC
Verification
ASIC
Design
output from design phase
register-transfer level
(coding)
functional correctness
probing,
temperatures
system level,
customer
applications

8.  Abstraction
 hardware description languages (VHDL,
Verilog, SystemVerilog, … )
 Version control systems
 Test automation
 Coverage testing
 Steps toward continous integration
8
Software engineering influences ASIC development
(NCSU 2017)

9.  What to verify?
 Functional verification
 Timing verification
 Performance verification
 How to verify?
 Simulation-based verification
 Emulation-based verification (FPGA)
 Formal verification
9
ASIC verification

10.  Directed testing
 Manually craft test cases
 High maintenance costs
 Good for finite condition spaces
 Random testing
 Automatically generate test cases
 Simple to build generator
 No maintenance
 Needs infinite compute cycles
10
Test case creation constrained random testing
&
coverage metrics

11. 11
Peculiarities of ASIC verification
Bugs in code escape to silicon => ”re-spin”
70% effort and critical path
Exhaustive testing possible?
Huge ”test farms”
Tool vendors get paid
www.erikjohanssonphoto.com

12.  CPU designs are increasingly complex
 New hardware architectures
 multicore
 multithreading
 …
 Requirements on small form factors
 Trade-off between high performance and
low power consumption
 Business demands on short time-to-
market
12
Contemporary challenges in ASIC verification
Purna Mohanty,
Aims Technology

13.  Test automation and regression testing
 As often as possible:
 Check out the latest source code
 Run test cases
 Report results
 Blame all committers since last pass
13
State-of-practice solution

14. Large amounts of test results
Test result matrices used to hide information
14
The backside…
Wikimedia Commons, Brukar:Bep
Creative Commons CC-BY 2.5

15.  One of the largest chip
manufacturers worldwide
 Development of a CPU
 Source code
 61,200 files (~3.8 GB)
 Verilog, SystemVerilog, assembler,
C, and shell scripts
 95 committers
 Regression testing
 500 test cases
 2 h execution time
 3-8 executions per day
15
Case under study – An ASIC project
Wikimedia Commons,
User: Zollo, CC-BY 2.0

16. We target three use cases:
1. General exploration of the design-under-test (DUT)
2. Localization of error-prone parts of the design
3. Detection of potential coverage holes
16
Visual analytics
(Illinois Applied Research Institute)

17.  Why game engine?
 Interaction out-of-the-box
 Why Unity?
 Fairly simple
 Scales well
 Very popular
 Unity???
 Cross-platform game engine and IDE
 Drag-and-drop 2D and 3D scenes
 Scripting in C#
17

18. 18

19.  Each building is a file, height represents #commits
 Organized in blocks based on folder structure
 Color indicates fraction of failed test executions
19
Prototype visualization – test execution cityscape

20.  Two senior verification engineers in India
 Generally positive
 First-person shooter controls not necessarily intuitive
20
Prototype evaluation
Flickr: billsoPHOTO
CC BY-SA 2.0

21.  Filtering of commit sets
 Search lights
 Tooltips
 Search for folder name
 Export to log files
21
Prototype evolved into a tool

22. 22
Tool demo

23.  Explore city layouts adhering to
physical mapping, i.e., “floor plan”
 Evaluate in focus groups with
additional verification engineers
 Controlled experiment with students
 Task: distribute verification effort
 Treatment: Cityscape or matrix
 Measurement: Time and coverage
23
Future work
Flickr: jamesjoel
CC BY-ND 2.0

24. 24
markus.borg@ri.se
@mrksbrg
mrksbrg.com
Automation in ASIC verification generates large
amounts of test results
Visual analytics through a game engine helps
focusing design and verification effort