Tool Qualification v12.02

1,614 views

Published on

How to be sure that a software development and test tool works according to a dedicated specification.

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,614
On SlideShare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
58
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Tool Qualification v12.02

  1. 1. Tool Qualification Minimize the riskof systematic faults in the developed product due to malfunctions of the software tool (introduce or fail to detect errors) Debug and Test Tools
  2. 2. ISO 26262Part 8 Supported Processes, Chapter 11 Safety-Lifecycle Process ― Big portions of this procoss is covered by software tools today This works only if ― The risk of systematic fault in the developed product due to malfunctions of the software tool (introduce or fail to detect errors) is minimized ― The software tool works accurate and correct within such a process Detailed and up front analysis / risk assessment of the software tools used within a safety project Debug and Test Tools
  3. 3. Software Tools Software Development Tools: May introduce an error in the final product Software Verification Tools: May fail to detect an error in the final product Other Software Tools ― Depending on the size of a company we are talking of a big number of tools (including in-house tools, Excel, …) Debug and Test Tools
  4. 4. Part 1 – OverviewThe Blue Box and ISO26262 Use Cases Debug and Test Tools
  5. 5. Reference V ISO 26262-6 ISO 26262-5, 10 Hardware Debug and Test Tools
  6. 6. iSYSTEM Tools within a Safety Project I Software Development (ISO 26262-6, 9) ― Not so critical ― Assembler (interrupt handler, time-critical algorithms, …) Software Test (ISO 26262-6, 9) ― Unit Testing ― Integration Testing Hardware-Software Integration and Testing (ISO 26262-5, 10/-4,8) System Integration Testing (ISO 26262-4,8) ISO 26262-6 mentions: „…The test environment … shall correspond as closely as possible to the target environment …“ Debug and Test Tools
  7. 7. iSYSTEM Tools within a Safety Project II - Unit Testingisystem.connectisystem.test Debug and Test Tools
  8. 8. iSYSTEM Tools within a Safety Project II - Unit Testingisystem.connectisystem.test +Trace &Object CodeLevel Coverage Debug and Test Tools
  9. 9. iSYSTEM Tools within a Safety Project II - Unit Testingisystem.connectisystem.test Debug and Test Tools
  10. 10. iSYSTEM Tools within a Safety Project II - Hardware-Software Integration Testingisystem.connectisystem.test+ Trace & Profilerisystem.connectisystem.test+ Trace &I/O Module Debug and Test Tools
  11. 11. Part 2 – How to get confidence in the use of software tools Detailed and up front analysis / risk assessment of the software tools used within a safety project Debug and Test Tools
  12. 12. Debug and Test Tools
  13. 13. e? Confidence in the use of e nc id software tools ISO 26262-8-11 on f c and sk How high is the risk that the malfunction of a tool leads to an error in the ri ss final product or errors are not detected (Tool Impact, TI)? se as ou How much do you trust your own methods and procedures (your y processes) in order to uncover suchhmalfunction of a tool (Tool Error Detection (TD)? hic w on high se average low ba Trust in processminimal risk ti on high risk a o rm f in e th  Beside the tool itself also the process plays a major role in getting is confidence t a h TCL2, TCL3: Tool Qualification applies W Debug and Test Tools
  14. 14. How iSYSTEM helps you getting/creating confidence? Process ― SPICE, CMMI, … iSYSTEM: ISO9001:2008 ― Output: Process documentation, external audits Transparency es ur ― Release and test process (http://www.isystem.com/downloads/winidea/release-policy) eas new features), item (bugs, changes, new ― Output: Test documentation, release notes (major features, …) tracking lists l M o na Documentation a ti is be used in a safety project or to be qualified) ― Description of features/functions (to an rg ― Explanation how these features/functions should (not) be used O ― Manuals: User Manual, Hardware Reference Manual, Getting Started Guides, PCB Desing Guidelines, ... ― System Requirements in general or specific to a micro controller ― Description of well-known workarounds dedicated to a specific micro controller ― … Pre-Qualification Environment: fitIDEA ― Regression test suite, reference hardware, reference application, reference test cases Debug and Test Tools
  15. 15. Result of Step 1 Determine a „Tool Confidence Level“ could be l? minimal risk, high trust = t oo e no tool qualificationr needed w a ft so a y lif or you see a high risk and an average trust in your processes andqua a tool qualification would to apply w Ho high average low Trust in processminimal risk high risk Debug and Test Tools
  16. 16. ISO 26262-8-11 Tool Qualification Methods TCL2 A B C D A-D: Measure of theMethods safety relevance of a 1a Increased confidence from use ++ ++ ++ + malfunction (Automotive Safety 1b Evaluation of the tool dev process ++ ++ ++ + Integrity Level (ASIL)) 1c Validation of the software tool + + + ++ 1d Dev. in accordance with a safety standard + + + ++ TCL3 A B C D 1a Increased confidence from use ++ ++ + + 1b Evaluation of the tool dev process ++ ++ + + 1c Validation of the software tool + + ++ ++ 1d Dev. in accordance with a safety standard + + ++ ++ + recommended ++ highly recommended  Qualifying a software tool means checking/discussing all 4 methods Debug and Test Tools
  17. 17. What are we talking about? The company iSYSTEM Development and test tools  Standard debugging and IDE functions (memory read, write, step, memory dump, download, flash programming, etc.)  Advanced debugging with trace and profiling (especially time measurements)  Software test with code coverage and unit test Debug and Test Tools
  18. 18. TCL2 A B C D1a Increased confidence from use ++ ++ ++ +1b Evaluation of the tool dev process ++ ++ ++ +1c Validation of the software tool + + + ++1d Dev. in accordance with a safety standard + + + ++TCL3 A B C D1a Increased confidence from use ++ ++ + +1b Evaluation of the tool dev process ++ ++ + +1c Validation of the software tool + + ++ ++1d Dev. in accordance with a safety standard + + ++ ++ Debug and Test Tools
  19. 19. Increased Confidence From Use? How is a software tool used today? ― Dedicated version and configuration ll ? ta ― For one or more specific micro controllers a ― In conjunction with other software tools (e.g., compilers) hod et New project: m s ― The tool setup may definitely differ fromi the previous project th ― Very new controllers will be used er id ns co Dynamic release policy of the software tool manufacturer e w ― Official and regression tested release d ul ho ― Hotfixes (customer wishes) S ― Fairly good tested software versions Debug and Test Tools
  20. 20. YES! – Because … In the market since 1986 Thousends of users, word-wide• Similar projects successfully done in the past Great cooperation between you and iSYSTEM ― Transparency ISO 262 62 ― Openness ― Pragmatism Debug and Test Tools
  21. 21. How iSYSTEM helps you getting/creating confidence? Process ― SPICE, CMMI, … iSYSTEM: ISO9001:2008 ! EM ― Output: Process documentation, external audits ST Transparency i SY = ― Release and test process (http://www.isystem.com/downloads/winidea/release-policy) ― Output: Test documentation, release notes se (major new features), item (bugs, changes, new features, …) tracking lists u in ce Documentation en fid ― Description of features/functions (to be used in a safety project or to be qualified) on ― Explanation how these features/functions should (not) be used c d ― Manuals: User Manual, Hardware Reference Manual, Getting Started Guides, PCB Desing se ea in general or specific to a micro controller Guidelines, ... cr ― System Requirements n ― DescriptionIof well-known workarounds dedicated to a specific micro controller ― … Pre-Qualification Environment: fitIDEA ― Regression test suite, reference hardware, reference application, reference test cases Debug and Test Tools
  22. 22. TCL2 A B C D1a Increased confidence from use ++ ++ ++ +1b Evaluation of the tool dev process ++ ++ ++ +1c Validation of the software tool + + + ++1d Dev. in accordance with a safety standard + + + ++TCL3 A B C D1a Increased confidence from use ++ ++ + +1b Evaluation of the tool dev process ++ ++ + +1c Validation of the software tool + + ++ ++1d Dev. in accordance with a safety standard + + ++ ++ Debug and Test Tools
  23. 23. Evaluation of the tool dev process Most of the software tool manufacturers do have defined and documented their processes (and lives those, hopefully), ISO9001 or similar We guess that non of them have a real process model in place (such as CMMI, SPICE, …) Audit/Assessment of these processes is possible (quality manual, process descriptions, external audit, …) Zertificates?!  It may be a big challenge to evaluate a dev process, again and again  Too many software tools are already in use, also Excel and so forth Debug and Test Tools
  24. 24. How iSYSTEM helps you getting/creating confidence?! s se Process es ― SPICE, CMMI, … iSYSTEM: ISO9001:2008 oc pr st ― Output: Process documentation, external audits te Transparency and ev ― Release and test process (http://www.isystem.com/downloads/winidea/release-policy) ― Output: Test documentation, release notes d s (major new features), item (bugs, changes, new features, …) tracking lists M TE YS Documentation iS e at ― Description of features/functions (to be used in a safety project or to be qualified) u ― Explanation how these lfeatures/functions should (not) be used a ev ― Manuals: User Manual, Hardware Reference Manual, Getting Started Guides, PCB Desing Guidelines, ... to e bl ― System Requirements in general or specific to a micro controller si os ― Description of well-known workarounds dedicated to a specific micro controller ― … p t is I Pre-Qualification Environment: fitIDEA ― Regression test suite, reference hardware, reference application, reference test cases Debug and Test Tools
  25. 25. TCL2 A B C D1a Increased confidence from use ++ ++ ++ +1b Evaluation of the tool dev process ++ ++ ++ +1c Validation of the software tool + + + ++1d Dev. in accordance with a safety standard + + + ++TCL3 A B C D1a Increased confidence from use ++ ++ + +1b Evaluation of the tool dev process ++ ++ + +1c Validation of the software tool + + ++ ++1d Dev. in accordance with a safety standard + + ++ ++ Debug and Test Tools
  26. 26. Validation of the software tool It is also the responsibility of the tool manufacturer to provide means and methods to enable a customer to perform a validation Ideal: Availability of the manufacturer’s test tool suite for a customer. Documentation, Test suite with reference hardware, reference applications and appropriate test cases. isystem.connect for Python wrapper isystem.connect test testIDEA isystem.connect iSYSTEM Development Environment & Debugger winIDEA Reference Target Reference Application Debug and Test Tools
  27. 27. How iSYSTEM helps you getting/creating confidence? e! Process ― SPICE, CMMI, … iSYSTEM: ISO9001:2008 os ― Output: Process documentation, external audits rp pu ct Transparency exa ― Release and test process (http://www.isystem.com/downloads/winidea/release-policy) s hi ― Output: Test documentation, release notes t(major new features), item (bugs, changes, new features, …) tracking lists f or A DE itI (to be used in a safety project or to be qualified) Documentation f ― Description of features/functions s de ― Explanation how these features/functions should (not) be used vi ― Manuals: User Manual, Hardware Reference Manual, Getting Started Guides, PCB Desing o Guidelines, ... pr EM ― System Requirements in general or specific to a micro controller ― Description T well-known workarounds dedicated to a specific micro controller Sof ― … i SY Pre-Qualification Environment: fitIDEA ― Regression test suite, reference hardware, reference application, reference test cases Debug and Test Tools
  28. 28. fitIDEA - Tool Pre-Qualification Environment Test GUI Test Cases Consulting & Test Reports Integration Safety Manual & Services Documentation isystem.connect for Python wrapper isystem.connect test testIDEA isystem.connect iSYSTEM Development Environment & Debugger winIDEA Reference Target Reference Application Debug and Test Tools
  29. 29. Tool Pre-Qualification Environment• If a specific function of an iSYSTEM tool has to get verified in the customer’s use case context• Consists of reference hardware plus test cases to verify several functions of an on-chip debug and trace tool such as: ― Standard debugging and IDE functions, e.g. memory read, write, step, memory dump, download, flash programming, etc. ― Advanced debugging with trace and profiling (especially time measurements) ― Software test with code coverage and unit test ―…• Is an extract of iSYSTEM’s internally used regression test tool suite• May be ported/adapted to a customer’s target system Debug and Test Tools
  30. 30. TCL2 A B C D1a Increased confidence from use ++ ++ ++ +1b Evaluation of the tool dev process ++ ++ ++ +1c Validation of the software tool + + + ++1d Dev. in accordance with a safety standard + + + ++TCL3 A B C D1a Increased confidence from use ++ ++ + +1b Evaluation of the tool dev process ++ ++ + +1c Validation of the software tool + + ++ ++1d Dev. in accordance with a safety standard + + ++ ++ Debug and Test Tools
  31. 31. Development in accordance with a saftey standard  View and adapt recommendations of such standards  Transfer know-how to software developers (how others have to do it) Debug and Test Tools
  32. 32. How iSYSTEM helps you getting/creating confidence? Process ― SPICE, CMMI, … iSYSTEM: ISO9001:2008 ― Output: Process documentation, external audits Transparency n! ar le ― Release and test process (http://www.isystem.com/downloads/winidea/release-policy) to ― Output: Test documentation, release notes (major new features), item (bugs, changes, new features, …) tracking lists g n i i ll w Documentation is EM ― Description of features/functions (to be used in a safety project or to be qualified) T ― Explanation how these features/functions should (not) be used SHardware Reference Manual, Getting Started Guides, PCB ― Manuals: User Manual,Y iS Desing Guidelines, ... ― System Requirements in general or specific to a micro controller ― Description of well-known workarounds dedicated to a specific micro controller ― … Pre-Qualification Environment: fitIDEA ― Regression test suite, reference hardware, reference application, reference test cases Debug and Test Tools
  33. 33. 62 ISO 262 Summary Detailed and up front analysis / risk assessment of the software tools used within a safety project Beside the tool itself also the process plays a major role in getting confidence It is also the responsibility of the tool manufacturer to provide means and methods to enable a customer to perform a validation Tool manufacturers will improve their development and test processes Transparency first, qualification second Debug and Test Tools

×