Your SlideShare is downloading. ×
Tool Qualification v12.02
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Tool Qualification v12.02

673
views

Published on

How to be sure that a software development and test tool works according to a dedicated specification.

How to be sure that a software development and test tool works according to a dedicated specification.

Published in: Technology, Business

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
673
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
34
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Tool Qualification Minimize the riskof systematic faults in the developed product due to malfunctions of the software tool (introduce or fail to detect errors) Debug and Test Tools
  • 2. ISO 26262Part 8 Supported Processes, Chapter 11 Safety-Lifecycle Process ― Big portions of this procoss is covered by software tools today This works only if ― The risk of systematic fault in the developed product due to malfunctions of the software tool (introduce or fail to detect errors) is minimized ― The software tool works accurate and correct within such a process Detailed and up front analysis / risk assessment of the software tools used within a safety project Debug and Test Tools
  • 3. Software Tools Software Development Tools: May introduce an error in the final product Software Verification Tools: May fail to detect an error in the final product Other Software Tools ― Depending on the size of a company we are talking of a big number of tools (including in-house tools, Excel, …) Debug and Test Tools
  • 4. Part 1 – OverviewThe Blue Box and ISO26262 Use Cases Debug and Test Tools
  • 5. Reference V ISO 26262-6 ISO 26262-5, 10 Hardware Debug and Test Tools
  • 6. iSYSTEM Tools within a Safety Project I Software Development (ISO 26262-6, 9) ― Not so critical ― Assembler (interrupt handler, time-critical algorithms, …) Software Test (ISO 26262-6, 9) ― Unit Testing ― Integration Testing Hardware-Software Integration and Testing (ISO 26262-5, 10/-4,8) System Integration Testing (ISO 26262-4,8) ISO 26262-6 mentions: „…The test environment … shall correspond as closely as possible to the target environment …“ Debug and Test Tools
  • 7. iSYSTEM Tools within a Safety Project II - Unit Testingisystem.connectisystem.test Debug and Test Tools
  • 8. iSYSTEM Tools within a Safety Project II - Unit Testingisystem.connectisystem.test +Trace &Object CodeLevel Coverage Debug and Test Tools
  • 9. iSYSTEM Tools within a Safety Project II - Unit Testingisystem.connectisystem.test Debug and Test Tools
  • 10. iSYSTEM Tools within a Safety Project II - Hardware-Software Integration Testingisystem.connectisystem.test+ Trace & Profilerisystem.connectisystem.test+ Trace &I/O Module Debug and Test Tools
  • 11. Part 2 – How to get confidence in the use of software tools Detailed and up front analysis / risk assessment of the software tools used within a safety project Debug and Test Tools
  • 12. Debug and Test Tools
  • 13. e? Confidence in the use of e nc id software tools ISO 26262-8-11 on f c and sk How high is the risk that the malfunction of a tool leads to an error in the ri ss final product or errors are not detected (Tool Impact, TI)? se as ou How much do you trust your own methods and procedures (your y processes) in order to uncover suchhmalfunction of a tool (Tool Error Detection (TD)? hic w on high se average low ba Trust in processminimal risk ti on high risk a o rm f in e th  Beside the tool itself also the process plays a major role in getting is confidence t a h TCL2, TCL3: Tool Qualification applies W Debug and Test Tools
  • 14. How iSYSTEM helps you getting/creating confidence? Process ― SPICE, CMMI, … iSYSTEM: ISO9001:2008 ― Output: Process documentation, external audits Transparency es ur ― Release and test process (http://www.isystem.com/downloads/winidea/release-policy) eas new features), item (bugs, changes, new ― Output: Test documentation, release notes (major features, …) tracking lists l M o na Documentation a ti is be used in a safety project or to be qualified) ― Description of features/functions (to an rg ― Explanation how these features/functions should (not) be used O ― Manuals: User Manual, Hardware Reference Manual, Getting Started Guides, PCB Desing Guidelines, ... ― System Requirements in general or specific to a micro controller ― Description of well-known workarounds dedicated to a specific micro controller ― … Pre-Qualification Environment: fitIDEA ― Regression test suite, reference hardware, reference application, reference test cases Debug and Test Tools
  • 15. Result of Step 1 Determine a „Tool Confidence Level“ could be l? minimal risk, high trust = t oo e no tool qualificationr needed w a ft so a y lif or you see a high risk and an average trust in your processes andqua a tool qualification would to apply w Ho high average low Trust in processminimal risk high risk Debug and Test Tools
  • 16. ISO 26262-8-11 Tool Qualification Methods TCL2 A B C D A-D: Measure of theMethods safety relevance of a 1a Increased confidence from use ++ ++ ++ + malfunction (Automotive Safety 1b Evaluation of the tool dev process ++ ++ ++ + Integrity Level (ASIL)) 1c Validation of the software tool + + + ++ 1d Dev. in accordance with a safety standard + + + ++ TCL3 A B C D 1a Increased confidence from use ++ ++ + + 1b Evaluation of the tool dev process ++ ++ + + 1c Validation of the software tool + + ++ ++ 1d Dev. in accordance with a safety standard + + ++ ++ + recommended ++ highly recommended  Qualifying a software tool means checking/discussing all 4 methods Debug and Test Tools
  • 17. What are we talking about? The company iSYSTEM Development and test tools  Standard debugging and IDE functions (memory read, write, step, memory dump, download, flash programming, etc.)  Advanced debugging with trace and profiling (especially time measurements)  Software test with code coverage and unit test Debug and Test Tools
  • 18. TCL2 A B C D1a Increased confidence from use ++ ++ ++ +1b Evaluation of the tool dev process ++ ++ ++ +1c Validation of the software tool + + + ++1d Dev. in accordance with a safety standard + + + ++TCL3 A B C D1a Increased confidence from use ++ ++ + +1b Evaluation of the tool dev process ++ ++ + +1c Validation of the software tool + + ++ ++1d Dev. in accordance with a safety standard + + ++ ++ Debug and Test Tools
  • 19. Increased Confidence From Use? How is a software tool used today? ― Dedicated version and configuration ll ? ta ― For one or more specific micro controllers a ― In conjunction with other software tools (e.g., compilers) hod et New project: m s ― The tool setup may definitely differ fromi the previous project th ― Very new controllers will be used er id ns co Dynamic release policy of the software tool manufacturer e w ― Official and regression tested release d ul ho ― Hotfixes (customer wishes) S ― Fairly good tested software versions Debug and Test Tools
  • 20. YES! – Because … In the market since 1986 Thousends of users, word-wide• Similar projects successfully done in the past Great cooperation between you and iSYSTEM ― Transparency ISO 262 62 ― Openness ― Pragmatism Debug and Test Tools
  • 21. How iSYSTEM helps you getting/creating confidence? Process ― SPICE, CMMI, … iSYSTEM: ISO9001:2008 ! EM ― Output: Process documentation, external audits ST Transparency i SY = ― Release and test process (http://www.isystem.com/downloads/winidea/release-policy) ― Output: Test documentation, release notes se (major new features), item (bugs, changes, new features, …) tracking lists u in ce Documentation en fid ― Description of features/functions (to be used in a safety project or to be qualified) on ― Explanation how these features/functions should (not) be used c d ― Manuals: User Manual, Hardware Reference Manual, Getting Started Guides, PCB Desing se ea in general or specific to a micro controller Guidelines, ... cr ― System Requirements n ― DescriptionIof well-known workarounds dedicated to a specific micro controller ― … Pre-Qualification Environment: fitIDEA ― Regression test suite, reference hardware, reference application, reference test cases Debug and Test Tools
  • 22. TCL2 A B C D1a Increased confidence from use ++ ++ ++ +1b Evaluation of the tool dev process ++ ++ ++ +1c Validation of the software tool + + + ++1d Dev. in accordance with a safety standard + + + ++TCL3 A B C D1a Increased confidence from use ++ ++ + +1b Evaluation of the tool dev process ++ ++ + +1c Validation of the software tool + + ++ ++1d Dev. in accordance with a safety standard + + ++ ++ Debug and Test Tools
  • 23. Evaluation of the tool dev process Most of the software tool manufacturers do have defined and documented their processes (and lives those, hopefully), ISO9001 or similar We guess that non of them have a real process model in place (such as CMMI, SPICE, …) Audit/Assessment of these processes is possible (quality manual, process descriptions, external audit, …) Zertificates?!  It may be a big challenge to evaluate a dev process, again and again  Too many software tools are already in use, also Excel and so forth Debug and Test Tools
  • 24. How iSYSTEM helps you getting/creating confidence?! s se Process es ― SPICE, CMMI, … iSYSTEM: ISO9001:2008 oc pr st ― Output: Process documentation, external audits te Transparency and ev ― Release and test process (http://www.isystem.com/downloads/winidea/release-policy) ― Output: Test documentation, release notes d s (major new features), item (bugs, changes, new features, …) tracking lists M TE YS Documentation iS e at ― Description of features/functions (to be used in a safety project or to be qualified) u ― Explanation how these lfeatures/functions should (not) be used a ev ― Manuals: User Manual, Hardware Reference Manual, Getting Started Guides, PCB Desing Guidelines, ... to e bl ― System Requirements in general or specific to a micro controller si os ― Description of well-known workarounds dedicated to a specific micro controller ― … p t is I Pre-Qualification Environment: fitIDEA ― Regression test suite, reference hardware, reference application, reference test cases Debug and Test Tools
  • 25. TCL2 A B C D1a Increased confidence from use ++ ++ ++ +1b Evaluation of the tool dev process ++ ++ ++ +1c Validation of the software tool + + + ++1d Dev. in accordance with a safety standard + + + ++TCL3 A B C D1a Increased confidence from use ++ ++ + +1b Evaluation of the tool dev process ++ ++ + +1c Validation of the software tool + + ++ ++1d Dev. in accordance with a safety standard + + ++ ++ Debug and Test Tools
  • 26. Validation of the software tool It is also the responsibility of the tool manufacturer to provide means and methods to enable a customer to perform a validation Ideal: Availability of the manufacturer’s test tool suite for a customer. Documentation, Test suite with reference hardware, reference applications and appropriate test cases. isystem.connect for Python wrapper isystem.connect test testIDEA isystem.connect iSYSTEM Development Environment & Debugger winIDEA Reference Target Reference Application Debug and Test Tools
  • 27. How iSYSTEM helps you getting/creating confidence? e! Process ― SPICE, CMMI, … iSYSTEM: ISO9001:2008 os ― Output: Process documentation, external audits rp pu ct Transparency exa ― Release and test process (http://www.isystem.com/downloads/winidea/release-policy) s hi ― Output: Test documentation, release notes t(major new features), item (bugs, changes, new features, …) tracking lists f or A DE itI (to be used in a safety project or to be qualified) Documentation f ― Description of features/functions s de ― Explanation how these features/functions should (not) be used vi ― Manuals: User Manual, Hardware Reference Manual, Getting Started Guides, PCB Desing o Guidelines, ... pr EM ― System Requirements in general or specific to a micro controller ― Description T well-known workarounds dedicated to a specific micro controller Sof ― … i SY Pre-Qualification Environment: fitIDEA ― Regression test suite, reference hardware, reference application, reference test cases Debug and Test Tools
  • 28. fitIDEA - Tool Pre-Qualification Environment Test GUI Test Cases Consulting & Test Reports Integration Safety Manual & Services Documentation isystem.connect for Python wrapper isystem.connect test testIDEA isystem.connect iSYSTEM Development Environment & Debugger winIDEA Reference Target Reference Application Debug and Test Tools
  • 29. Tool Pre-Qualification Environment• If a specific function of an iSYSTEM tool has to get verified in the customer’s use case context• Consists of reference hardware plus test cases to verify several functions of an on-chip debug and trace tool such as: ― Standard debugging and IDE functions, e.g. memory read, write, step, memory dump, download, flash programming, etc. ― Advanced debugging with trace and profiling (especially time measurements) ― Software test with code coverage and unit test ―…• Is an extract of iSYSTEM’s internally used regression test tool suite• May be ported/adapted to a customer’s target system Debug and Test Tools
  • 30. TCL2 A B C D1a Increased confidence from use ++ ++ ++ +1b Evaluation of the tool dev process ++ ++ ++ +1c Validation of the software tool + + + ++1d Dev. in accordance with a safety standard + + + ++TCL3 A B C D1a Increased confidence from use ++ ++ + +1b Evaluation of the tool dev process ++ ++ + +1c Validation of the software tool + + ++ ++1d Dev. in accordance with a safety standard + + ++ ++ Debug and Test Tools
  • 31. Development in accordance with a saftey standard  View and adapt recommendations of such standards  Transfer know-how to software developers (how others have to do it) Debug and Test Tools
  • 32. How iSYSTEM helps you getting/creating confidence? Process ― SPICE, CMMI, … iSYSTEM: ISO9001:2008 ― Output: Process documentation, external audits Transparency n! ar le ― Release and test process (http://www.isystem.com/downloads/winidea/release-policy) to ― Output: Test documentation, release notes (major new features), item (bugs, changes, new features, …) tracking lists g n i i ll w Documentation is EM ― Description of features/functions (to be used in a safety project or to be qualified) T ― Explanation how these features/functions should (not) be used SHardware Reference Manual, Getting Started Guides, PCB ― Manuals: User Manual,Y iS Desing Guidelines, ... ― System Requirements in general or specific to a micro controller ― Description of well-known workarounds dedicated to a specific micro controller ― … Pre-Qualification Environment: fitIDEA ― Regression test suite, reference hardware, reference application, reference test cases Debug and Test Tools
  • 33. 62 ISO 262 Summary Detailed and up front analysis / risk assessment of the software tools used within a safety project Beside the tool itself also the process plays a major role in getting confidence It is also the responsibility of the tool manufacturer to provide means and methods to enable a customer to perform a validation Tool manufacturers will improve their development and test processes Transparency first, qualification second Debug and Test Tools