Software Testing and Analysis Tools for Productivity and Quality

1,720
-1

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,720
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
42
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Software Testing and Analysis Tools for Productivity and Quality

  1. 1. Software Testing and Analysis Tools for Productivity and Quality
  2. 2. Summary ● Verifysoft and its partners ● Cost of software errors ● Testing tools in the software development process ● Code complexity / Static Analysis. ● Code Coverage ● Functional Test 2
  3. 3. Verifysoft Technology GmbH ● Verifysoft Technology GmbH ● Created in June 2003 ● Offenburg (Germany) ● 200+ customers in Europe (2009) ● Distributor and developper of Software testing tools ● distribution, consulting and support 3
  4. 4. Activities ● Tools ● Seminars 4
  5. 5. our partners ● Testwell, Tampere (Finland) ● Conformiq, Saratoga (USA) ● Coverity, San Francisco (USA) 5
  6. 6. our customers Main sectors of application: Development of critical software: – Aerospace – Automotive – Medical – Nuclear 6
  7. 7. References and many more... 7
  8. 8. Cost of software errors ● The losses due to errors of programming increase every year Estimation : ??? 8
  9. 9. Cost of software errors ● Software bugs cost in Europe 100-150.000.000.000 Euros/year → Tendancy increasing ! The complexity of Software doubles every 18 months. To face it: the "know-how" of developers!!! (Les Hatton, Kingston University London) 9
  10. 10. Cost of software errors ● Software bugs cost up to 14.000.000 Euros per company and per year 50% of the companies found in the first year of utilisation up to 10 critical errors. IDC-Survey in 2008 „Improving Software Quality to Drive Business Agility“, within American companies with 250 -10.000 employees www.idc.com 10
  11. 11. Cost of software errors 40-60% of the global budget is usually used for test and correction The purpose of testing is to obtain a product „without errors“ 11
  12. 12. Software test Testing is necessary for obtaining certifications IEC 61508 EN 50128 (Railway) IEC 62304 (Medical) EN 62138 (Nuclear) DO-178B (Aerospace) 12
  13. 13. Software test Test: is essential But ... expensive … repetitive activity → Test-Automatisation → use of testing tools ! 13
  14. 14. Software development process Requirements Software Analysis Validation System Conformiq System Testing Qtronic Design Functional Test Architecture Integrating Design Testing Detailed Testwell CTA++ Unit Testing Unittest Specifications Testwell CTC++ Code Coverage Black box tests Coding Testwell CMT++ White box tests CMTJava Complexity Measures 14
  15. 15. Software development process Requirements Software Analysis Validation System Conformiq System Testing Qtronic Design Functional Test Architecture Integrating Design Testing Detailed Testwell CTA++ Unit Testing Unittest Specifications Testwell CTC++ Code Coverage Black box tests Coding Testwell CMT++ White box tests CMTJava Complexity Measures 15
  16. 16. Testwell CMT++ / CMTJava Testwell CMT++ Testwell CMTJava Code Complexity Measures Tools for C/C++ and Java 16 Testwell CTC++, CMT++ and CMTJava are products of Testwell Oy, Tampere (Finland)
  17. 17. Testwell CMT++ / CMTJava 17 Testwell CTC++, CMT++ and CMTJava are products of Testwell Oy, Tampere (Finland)
  18. 18. Testwell CMT++ / CMTJava ● Why analysing the code complexity? ● Code complexity correlates with the defect rate and robustness of the application ● Complex code is difficult to test → more errors in the final application ● Complex code is difficult to maintain ● Unnecessary complex code is often the reason for bad code quality and erroneous programs. 18 Testwell CTC++, CMT++ and CMTJava are products of Testwell Oy, Tampere (Finland)
  19. 19. Testwell CMT++ / CMTJava Testwell CMT++ and Testwell CMTJava analyses your applications for the following metrics: ● Lines-of-code (LOC) metrics ● Halstead's metrics ● McCabe cyclomatic number v(G) ● Maintainability Index 19 Testwell CTC++, CMT++ and CMTJava are products of Testwell Oy, Tampere (Finland)
  20. 20. Testwell CMT++ / CMTJava Lines-of-code (LOC) metrics: ● LOCphy number of physical lines ● LOCpro number of lines with program code ● LOCbl number of blanc lines ● LOCcom number of lines with comments 20 Testwell CTC++, CMT++ and CMTJava are products of Testwell Oy, Tampere (Finland)
  21. 21. Testwell CMT++ / CMTJava Lines-of-code (LOC) metrics: ● Recommanded standard for fonctions: between 4 and 40 program lines A function definition contains at least a prototype, one line of code, and a pair of braces, which makes 4 lines. A function longer than 40 program lines probably implements many functions. 21 Testwell CTC++, CMT++ and CMTJava are products of Testwell Oy, Tampere (Finland)
  22. 22. Testwell CMT++ / CMTJava Lines-of-code (LOC) metrics: ● Recommanded standard for files: between 4 and 400 program lines The smallest entity that may reasonably occupy a whole source file is a function, and the minimum length of a function is 4 lines. Files longer than 400 program lines (10..40 functions) are usually too long to be understood as a whole. Comments : 30 to 75% 22 Testwell CTC++, CMT++ and CMTJava are products of Testwell Oy, Tampere (Finland)
  23. 23. Testwell CMT++ / CMTJava McCabe Cyclomatic Number v(G) describes the complexity of the control flow of a program. The greater the cyclomatic number is the more execution paths there are through the function, and the harder it is to understand. 23 Testwell CTC++, CMT++ and CMTJava are products of Testwell Oy, Tampere (Finland)
  24. 24. Testwell CMT++ / CMTJava McCabe Cyclomatic Number v(G) ● recommanded standard for fonctions: <15 If a function has a cyclomatic number of 15, there are at least 15 (but probably more) execution paths through it. More than 15 paths are hard to identify and test. ● recommanded standard for files: <100 24 Testwell CTC++, CMT++ and CMTJava are products of Testwell Oy, Tampere (Finland)
  25. 25. Testwell CMT++ / CMTJava Halstead's metrics ● B estimated number of bugs ● D difficulty level, error proneness ● E effort to implement ● L program level (abstraction level of the program) ● N program length ● N1 number of operators ● N2 numbre of operands ● n vocabulary size (unique operators + unique operands) ● n1 number of unique operators ● n2 number of unique operands ● T implementation time / time to understand ● V volume: size of the implementation of an algorithm 25 Testwell CTC++, CMT++ and CMTJava are products of Testwell Oy, Tampere (Finland)
  26. 26. Testwell CMT++ / CMTJava Maintainability Index (MI) indicates when it becomes cheaper and less risky to rewrite the code instead to change it. ● 85 and more good maintainability ● 65-85 moderate maintainability ● < 65 difficult to maintain → with really bad pieces of code (big, uncommented, unstructured) the MI value can be even negative 26 Testwell CTC++, CMT++ and CMTJava are products of Testwell Oy, Tampere (Finland)
  27. 27. Testwell CMT++ / CMTJava 27 Testwell CTC++, CMT++ and CMTJava are products of Testwell Oy, Tampere (Finland)
  28. 28. Testwell CMT++ / CMTJava Development View Management View Controlling View 28 Pdf-reports
  29. 29. Software development process Requirements Software Analysis Validation System Conformiq System Testing Qtronic Design Functional Test Architecture Integrating Design Testing Detailed Testwell CTA++ Unit Testing Unittest Specifications Testwell CTC++ Code Coverage Black box tests Coding Testwell CMT++ White box tests CMTJava Complexity Measures 29
  30. 30. Testwell CTC++ Test Coverage Analyser Testwell CTC++ Test Coverage Analyser for C/C++ CTC add-on for Java and C# 30 Testwell CTC++, CMT++ and CMTJava are products of Testwell Oy, Tampere (Finland)
  31. 31. Testwell CTC++ Test Coverage Analyser Code coverage shows the parts of the code ● executed / not executed ● tested / not tested 31 Testwell CTC++, CMT++ and CMTJava are products of Testwell Oy, Tampere (Finland)
  32. 32. Testwell CTC++ Test Coverage Analyser Why measuring the code coverage? ● helps to write better (more adapted) tests/test cases ● helps to avoid that you spend time on writing redundant test cases ● you know when you can stop testing ● you can proof to your customers that your code is tested according to their requirements ● you can be sure that your outsourcing/development partner delivers quality according to your requirements ● ensures high quality with high code coverage ● helps to find "dead code" ● required to obtain certifications. 32 Testwell CTC++, CMT++ and CMTJava are products of Testwell Oy, Tampere (Finland)
  33. 33. Testwell CTC++ Test Coverage Analyser Testwell CTC++ for all coverage levels: ● Function Coverage ● Decision Coverage ● Condition/Branch Coverage ● Modified Condition/Decision Coverage (MC/DC) ● Multicondition Coverage (MCC) 33 Testwell CTC++, CMT++ and CMTJava are products of Testwell Oy, Tampere (Finland)
  34. 34. Testwell CTC++ Test Coverage Analyser Testwell CTC++ can be used to obtain certifications in automotive, avionics and medical industries i.e. DO-178B - all levels: A, B, C - of the Federal Aviation Administration, FAA or for EN 61508 34 Testwell CTC++, CMT++ and CMTJava are products of Testwell Oy, Tampere (Finland)
  35. 35. Testwell CTC++ Test Coverage Analyser Testwell CTC++ is very easy to use ● no need of code modifications ● support of existing makefiles ● GUI integration in several IDEs – Microsoft Visual Studio – WindRiver Tornado – Borland C++ 5.02 – Metrowerks CodeWarrior – Eclipse – ... 35 Testwell CTC++, CMT++ and CMTJava are products of Testwell Oy, Tampere (Finland)
  36. 36. Testwell CTC++ Test Coverage Analyser Reports in Text, XML or HTML ● shows the untested code parts ● shows how many time each code part has been executed ● different coverage reports – Summary-Levels – Untested Code – Execution Profile Listing 36 Testwell CTC++, CMT++ and CMTJava are products of Testwell Oy, Tampere (Finland)
  37. 37. Testwell CTC++ Test Coverage Analyser Testwell CTC++ ideal for embedded targets ● very low instrumentation overhead ● performs code coverage in all targets → “host target add-on” is provided in source code → can be easily adapted to new targets ● works even with smallest targets and microcontrollers ● works with all compilers / cross-compilers 37 Testwell CTC++, CMT++ and CMTJava are products of Testwell Oy, Tampere (Finland)
  38. 38. Testwell CTC++ Test Coverage Analyser „CTC++ Add-on for Java and C#“ extension of Testwell CTC++ for Java and C# → You only need one code coverage tool for C, C++, Java, C#, ... 38 Testwell CTC++, CMT++ and CMTJava are products of Testwell Oy, Tampere (Finland)
  39. 39. Software development process Requirements Software Analysis Validation System Conformiq System Testing Qtronic Design Functional Test Architecture Integrating Design Testing Detailed Testwell CTA++ Unit Testing Unittest Specifications Testwell CTC++ Code Coverage Black box tests Coding Testwell CMT++ White box tests CMTJava Complexity Measures 39
  40. 40. Conformiq Qtronic Model-Based Automated Test Design for functional tests (black box tests) for software and systems 40 Conformiq Qtronic is a product and a trademark of Conformiq Inc., Saratoga (USA)
  41. 41. Conformiq Qtronic Manual tests takes time... and leads to risks  incorrect tests  missed tests  redundant tests  maintenance of tests 41 Conformiq Qtronic is a product and a trademark of Conformiq Inc., Saratoga (USA)
  42. 42. Conformiq Qtronic Our solution: Automated Test Design → model driven testing, model based testing, specification based testing, specification driven testing, ... 42 Conformiq Qtronic is a product and a trademark of Conformiq Inc., Saratoga (USA)
  43. 43. Manual vs. Automatic Automatic test generation and execution based on your design models X instead of writting the tests manually 43 Conformiq Qtronic is a product and a trademark of Conformiq Inc., Saratoga (USA)
  44. 44. Model Driven Testing The tool automates the design of functional tests (black box tests) for software and systems 44 Conformiq Qtronic is a product and a trademark of Conformiq Inc., Saratoga (USA)
  45. 45. Manual vs. Automatic Manual test script Test Plan 1 Script execution Automatic test generation System Model Q Script execution 45 Conformiq Qtronic is a product and a trademark of Conformiq Inc., Saratoga (USA)
  46. 46. Model Driven Testing Benefits: ● enforces the quality of requirements documentation ● creates a common platform for designers and testers ● self documented 46 Conformiq Qtronic is a product and a trademark of Conformiq Inc., Saratoga (USA)
  47. 47. Modelling ● Textually in Java (with elements in C#) → „Qtronic Modelling Language“ (QML) ● Grafically in UML State Charts (optional) ● The model can be created with: – Text editor („Java“) – Qtronic Modeller (UML State Charts) – or Third Party Modeling (UML) tools 47 Conformiq Qtronic is a product and a trademark of Conformiq Inc., Saratoga (USA)
  48. 48. Test generation Process: Creation of the model Import of the model Selection of the test coverage requirements and formats for the test scripts. Automatic test generation with Conformiq Qtronic Test execution in your environment. 48 Conformiq Qtronic is a product and a trademark of Conformiq Inc., Saratoga (USA)
  49. 49. Test generation Formats for the test scripts generation: Python TCL TTCN-3 C, C++ Visual Basic Java Junit Perl Excel HTML Word Shell Scripts 49 Conformiq Qtronic is a product and a trademark of Conformiq Inc., Saratoga (USA)
  50. 50. Test scripts generation 50 Conformiq Qtronic is a product and a trademark of Conformiq Inc., Saratoga (USA)
  51. 51. Test scripts generation Test scripts Generation progress List of test scripts 51 Conformiq Qtronic is a product and a trademark of Conformiq Inc., Saratoga (USA)
  52. 52. Test scripts generation Test scripts Generation progress Requirement Tracebility of the test scripts List of test scripts 52 Conformiq Qtronic is a product and a trademark of Conformiq Inc., Saratoga (USA)
  53. 53. Test scripts generation Test scripts Generation progress Message Sequence Chart Requirement-Tracebility for a test script Of the test scripts Test Step View List of with detailled information test scripts about contents 53 Conformiq Qtronic is a product and a trademark of Conformiq Inc., Saratoga (USA)
  54. 54. Test scripts generation Test scripts Generation progress Requirement Tracebility Message Sequence Chart of the test scripts Execution Trace View: for a test script shows the execution's Test Step View List of trace in the model with detailled information test scripts about contents 54 Conformiq Qtronic is a product and a trademark of Conformiq Inc., Saratoga (USA)
  55. 55. Conformiq Qtronic Model Test scripts generation 55 Conformiq Qtronic is a product and a trademark of Conformiq Inc., Saratoga (USA)
  56. 56. Test scripts execution Test scripts execution Model Model in your existing evironment (manually or automatically) Tes t Harnes s Tes t E xecution Test Scripts SystemUnder Test Platform Reports 56 Conformiq Qtronic is a product and a trademark of Conformiq Inc., Saratoga (USA)
  57. 57. Review → Cost of software errors → Testing tools in the software development process → Code Complexity / Static Analysis → Code Coverage → Automatic Test Generator 57
  58. 58. more Information Tel: +49 781 63 92 027 quality@verifysoft.com www.verifysoft.com Thank you! 58
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×