Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Enabling Visual Analytics with Unity - Exploring Regression Test Results in ASIC Verification

159 views

Published on

Invited talk at TOCSYC project meeting, November 17, 2017. Västerås, Sweden.

Published in: Science
  • Be the first to comment

  • Be the first to like this

Enabling Visual Analytics with Unity - Exploring Regression Test Results in ASIC Verification

  1. 1. Research Institutes of Sweden ENABLING VISUAL ANALYTICS WITH - Exploring Regression Test Results in ASIC Verification Markus Borg, Andreas Brytting, Daniel Hansson November 17, 2017 Swedish Institute of Computer Science Software and Systems Engineering Laboratory
  2. 2. 2 Automation in ASIC verification generates large amounts of test results Visual analytics through a game engine helps focusing design and verification effort
  3. 3. Development engineer, ABB, Malmö, Sweden 2007-2010  Editor and compiler development  Process automation PhD student, Lund University, Sweden 2010-2015  Machine learning for software engineering  Bug report management and traceability Senior researcher, RISE SICS AB, Lund, Sweden 2015-  Software engineering for machine learning  V&V for self-driving cars 3 Markus Borg?
  4. 4.  MSc thesis student Andreas Brytting  ASIC verification tool vendor  Research project 4 Credits
  5. 5. 5 WP3 Test effectiveness Task 2 Test result overload
  6. 6.  ASIC design vs. software engineering  Challenge and solution proposal  Tool demo  Evaluation plans 6 Outline
  7. 7. 7 ASIC development ASIC Verification ASIC Design output from design phase register-transfer level (coding) functional correctness probing, temperatures system level, customer applications
  8. 8.  Abstraction  hardware description languages (VHDL, Verilog, SystemVerilog, … )  Version control systems  Test automation  Coverage testing  Steps toward continous integration 8 Software engineering influences ASIC development (NCSU 2017)
  9. 9.  What to verify?  Functional verification  Timing verification  Performance verification  How to verify?  Simulation-based verification  Emulation-based verification (FPGA)  Formal verification 9 ASIC verification
  10. 10.  Directed testing  Manually craft test cases  High maintenance costs  Good for finite condition spaces  Random testing  Automatically generate test cases  Simple to build generator  No maintenance  Needs infinite compute cycles 10 Test case creation constrained random testing & coverage metrics
  11. 11. 11 Peculiarities of ASIC verification Bugs in code escape to silicon => ”re-spin” 70% effort and critical path Exhaustive testing possible? Huge ”test farms” Tool vendors get paid www.erikjohanssonphoto.com
  12. 12.  CPU designs are increasingly complex  New hardware architectures  multicore  multithreading  …  Requirements on small form factors  Trade-off between high performance and low power consumption  Business demands on short time-to- market 12 Contemporary challenges in ASIC verification Purna Mohanty, Aims Technology
  13. 13.  Test automation and regression testing  As often as possible:  Check out the latest source code  Run test cases  Report results  Blame all committers since last pass 13 State-of-practice solution
  14. 14. Large amounts of test results Test result matrices used to hide information 14 The backside… Wikimedia Commons, Brukar:Bep Creative Commons CC-BY 2.5
  15. 15.  One of the largest chip manufacturers worldwide  Development of a CPU  Source code  61,200 files (~3.8 GB)  Verilog, SystemVerilog, assembler, C, and shell scripts  95 committers  Regression testing  500 test cases  2 h execution time  3-8 executions per day 15 Case under study – An ASIC project Wikimedia Commons, User: Zollo, CC-BY 2.0
  16. 16. We target three use cases: 1. General exploration of the design-under-test (DUT) 2. Localization of error-prone parts of the design 3. Detection of potential coverage holes 16 Visual analytics (Illinois Applied Research Institute)
  17. 17.  Why game engine?  Interaction out-of-the-box  Why Unity?  Fairly simple  Scales well  Very popular  Unity???  Cross-platform game engine and IDE  Drag-and-drop 2D and 3D scenes  Scripting in C# 17
  18. 18. 18
  19. 19.  Each building is a file, height represents #commits  Organized in blocks based on folder structure  Color indicates fraction of failed test executions 19 Prototype visualization – test execution cityscape
  20. 20.  Two senior verification engineers in India  Generally positive  First-person shooter controls not necessarily intuitive 20 Prototype evaluation Flickr: billsoPHOTO CC BY-SA 2.0
  21. 21.  Filtering of commit sets  Search lights  Tooltips  Search for folder name  Export to log files 21 Prototype evolved into a tool
  22. 22. 22 Tool demo
  23. 23.  Explore city layouts adhering to physical mapping, i.e., “floor plan”  Evaluate in focus groups with additional verification engineers  Controlled experiment with students  Task: distribute verification effort  Treatment: Cityscape or matrix  Measurement: Time and coverage 23 Future work Flickr: jamesjoel CC BY-ND 2.0
  24. 24. 24 markus.borg@ri.se @mrksbrg mrksbrg.com Automation in ASIC verification generates large amounts of test results Visual analytics through a game engine helps focusing design and verification effort

×