Diab Compiler Quality Overview


Published on

Diab Compiler is the best in the Market, find out why...

  • Be the first to comment

  • Be the first to like this

Diab Compiler Quality Overview

  1. 1. Wind River Diab Compiler –Testing and Quality
  2. 2. Introduction  The compiler is a key tool for any safety critical applications  Compiler code generation issues can be very costly for customers with fielded products  Diab Compiler team strives to provide maximum visibility of defects found in the compiler – Access to information is key to assisting customers investigate and uncover product impacts before they happen in the field – This openness risks negatively impacting perception of customers  Wind River has built up a significant test harness to validate each new compiler release – Always room for further improvements in testing and new investments – Recently Wind River has been receiving more reported defects from our Automotive customers – Wind River is investigating why this is happening even when we have been increasing investment in compiler testing2 | © 2011 Wind River. All Rights Reserved.
  3. 3. Challenges of Compiler Testing  Compiler testing is complex  Virtually infinite number of option combinations and code that can be run through the compiler  Testing must be done across 10 major processor architectures and almost 100 sub-architectures  Wide range of customer markets and applications  Determining a representative set of tests is challenging3 | © 2011 Wind River. All Rights Reserved.
  4. 4. Diab Compiler Testing  Wind River is running millions of compiler tests – Standards test suites: e.g Plum Hall, Modena, Paranoia  Added ACE SuperTest C/C++ compiler test and validation suite – NEW investment in 2011 – World’s largest compiler test suite with over 2 Million tests – Benchmarks: EEMBC, Nullstone, PowerStone, customer supplied benchmarks  Added D-Bench in 2011 – Regression tests (6500+ test cases)  Verification tests for fixed defects are continuously be added to the regression test suite  Standard compiler tests may not always be representative of Automotive customers use cases4 | © 2011 Wind River. All Rights Reserved.
  5. 5. Getting Closer to the Customer  Customer specific tests are always better than off-the shelf application tests – Actual customer use cases of the compiler  Testing the compiler using customer supplied tests/benchmarks leads to improved test coverage of the compiler for customers – Code from key Automotive customers is already used in Diab testing  Wind River would be happy to incorporate more tests from customers into the test harness – However, customers aren’t always willing to share their source code and WR can’t always verify correct execution of customer applications  Wind River can test the compiler using customer option sets used in various customer projects – There is certainly an opportunity to review the current test process and ensure we are aligned with customer option sets5 | © 2011 Wind River. All Rights Reserved.
  6. 6. In Quality Improvement Plan Development  Investigate root cause of recent customer reported defects – Review defect reports from Automotive customers from last two years – Look for patterns that indicate where updates are needed to Diab test harness – Review tested option sets to ensure they continue to be in-line with customer use cases  Acquire code coverage data from key customers – Compare current compiler test suite coverage to help guide new test investments  Look for additional compiler test suites to add value to current test harness – Assuming low impact based on generic nature of available test suites and current coverage from existing suites being used for Diab  Acquire more “Automotive” source code from customers and partners – Need help from customers and partners in the Automotive industry – Agree with stakeholders on representative code for compiler testing – Currently discussing with customers and topic for upcoming German “road trip”6 | © 2011 Wind River. All Rights Reserved.
  7. 7. Sharing Information with Customers  Wind River’s primary customer support and communication tool is the Wind River On-line Support web portal (OLS)  OLS provides information on Diab compiler defects for customers  The defect tool has some limitations such as advanced sorting – Some of these limitations are due to OLS – Some of these limitations are due to the limits of WR’s underlying ClearQuest defect tracking database  Improvements are being investigated – Better export to Excel is currently being tested – How to better characterize and sort by arch and versions is being studied7 | © 2011 Wind River. All Rights Reserved.