OCCF: A Framework for Developing Test Coverage Measurement Tools Supporting Multiple Programming Languages

830 views

Published on

http://www.icst.lu/
I talked about this slide at ICST 2013 Testing Tools Track.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
830
On SlideShare
0
From Embeds
0
Number of Embeds
24
Actions
Shares
0
Downloads
11
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

OCCF: A Framework for Developing Test Coverage Measurement Tools Supporting Multiple Programming Languages

  1. 1. ICST Testing Tools Track 2013 2013/03/20 Kazunori Sakamoto, Kiyofumi Shimojo,Ryohei Takasawa, Hironori Washizaki, Yoshiaki Fukazawa Waseda University Dept. Computer Science and Engineering
  2. 2. Motivating Example • Coverage based tools have differencesint min(int a, int b) { • Many tools say return a < b ? a : b statement coverage is} Java 100% (1/1)@assert void testMin() { • EMMA says assertEquals(1, min(1, 2)); line coverage} is 86% (6/7) JUnit • Tools use different measurement units 2013/3/20 ICST Testing Tools Track 2013 2
  3. 3. Comparison of 25 coverage toolsTool Mecha Hierar Free State Deci Cond Condit Lang nism chical ment sion ition ion/D uages view ecisionCobertura Binary ✔ ✔ ✔ ✔ JavaEMMA Binary ✔ ✔ ✔ ✔ Java Java,JCover Code ✔ ✔ ✔ GroovyClover Code ✔ ✔ ✔ JavaAgitar Binary ✔ ✔ ✔ JavaOpenCover Processor ✔ ✔ ✔ ✔ C#NCover Source ✔ ✔ ✔ C#dotCover Binary ✔ ✔ C#gcov Compiler ✔ ✔ ✔ C++COVTOOL Code ✔ ✔ C++ 2013/3/20 ICST Testing Tools Track 2013 3
  4. 4. Comparison of 25 coverage toolsTool Mecha Hierar Free Sta Dec Con Condit Lang nism chical tem isio diti ion/D uages view ent n on ecisionBullseye Code ✔ ✔ ✔ ✔ ✔ C++Intel Code Compiler ✔ ✔ JavaCoverage ToolSquish Coco Code ✔ ✔ ✔ ✔ ✔ JavaTCAT Code ✔ ✔ C++, JavaParasoft C++, C# Code ✔ ✔ ✔ ✔ ✔Test Java C++, C#PurifyPlus Binary ✔ ✔ ✔ ✔ ✔ JavaSemantic C++, C# Code ✔ ✔ ✔Designs Java, etc.Coverage C++, C# Code ✔ ✔ ✔Validator Java, etc. 2013/3/20 ICST Testing Tools Track 2013 4
  5. 5. Comparison of 25 coverage toolsTool Mecha Hierar Free Stat Dec Con Condit Lang nism chical em isio diti ion/D uages view ent n on ecisionScriptCover Code ✔ ✔ JavaScriptCoverage.py Processor ✔ ✔ ✔ Pythonrcov Processor ✔ ✔ ✔ RubySimpleCov Processor ✔ ✔ ✔ RubyDevel::Cover Processor ✔ ✔ ✔ ✔ ✔ Perlxdebug Code ✔ ✔ ✔ PHPLuaCov Processor ✔ ✔ Lua • Tools support different criteria and langs • Differences between tools cause problems (*)*) Rüdiger Lincke, Jonas Lundberg and Welf Löwe: “Comparing Software Metrics Tools,” ISSTA 08Proceedings of the 2008 international symposium on Software testing and analysis, pp. 131-142, 2008. 2013/3/20 ICST Testing Tools Track 2013 5
  6. 6. Problems and tool solutions• Differences between tools – Hard to measure coverage of software using multiple langs such as Web apps – OCCF aids to provide consistent criteria• High development costs – Hard to develop coverage tools supporting new programming languages – OCCF aids to develop new coverage tools2013/3/20 ICST Testing Tools Track 2013 6
  7. 7. • Inserts automaticallyMeasurement approach • • Outputs coverage info Has no side effect • Measures completely Statement and int func(int a) { branch coverage if (a == 0) { int func(int a) { printf("a == 0"); if (branch(2, a == 0)) { } stmt(0); else if (false) { printf("a == 0"); printf("a != 0"); } Inserts into AST } else if (branch(3,false)){ } Function stmt(1);int func(int a) printf("a != 0"); } } Statement Statement Statement void stmt(int id) { Statement RECORD_STATEMENT(id); printf("a == 0") printf("a != 0") stmt (0) stmt (1) } Abstraction syntax tree(AST) 2013/3/20 ICST Testing Tools Track 2013 7
  8. 8. Architecture Test Case Coverage FaultMinimization Tool Report Tool Localization Tool Tools OCCF Trans C Java Python formers AST Transformer OCCF Core Cold spot AST Generator C Java Python Parsers (Code2Xml) 2013/3/20 ICST Testing Tools Track 2013 8
  9. 9. Process and design library Instrumentation Minimization code Test Case Instrumentation ~~~~ ~~~~ library Tool ~~~~ ~~~~ ~~~~ ~~~~ ~~~~ coverage Testing Framework data OCCF CoverageProduction Modified 0101 Report 0001 code Production Tool 1110 code 1010 Localization ~~~~ ~~~~ ~~~~ ~~~~ ~~~~ Pass ~~~~ Fail Fault ~~~~ Tool Pass FailTest code Modified Instrumentation Test code Test code Results 2013/3/20 ICST Testing Tools Track 2013 9
  10. 10. Process and design in detail XML object in .NET Instrumentation Framework AST Generator code AST Finder AST Inserter Code Generator~~~~ AST AST AST ~~~~~~~~ ~~~~~~~~ ~~~~ ~~~~source source code code C Java Python 2013/3/20 ICST Testing Tools Track 2013 10
  11. 11. Implementation• OCCF is developed in C# – Works on .NET Framework and Mono• AST Generator utilizes existing parser libraries• ANTLR for C, C#, Java, JavaScript, Lua – http://www.antlr.org/ – Reuse existing grammar files on the repository• Standard library (parser module) for Python 2, 3 – http://docs.python.org• ruby_parser for Ruby 1.8, 1.9 – https://github.com/seattlerb/ruby_parser• srcML for C++ – http://www.sdml.info/projects/srcml/2013/3/20 ICST Testing Tools Track 2013 11
  12. 12. Demo• Insert instrumentation code in eight languages: C, C++, C#, Java, JavaScript, Python, Ruby, Lua• Let me show four materials in Java – Original code – Original AST in XML – Modified AST in XML – Modified code2013/3/20 ICST Testing Tools Track 2013 12
  13. 13. Hot spot in OCCF• Finding nodes to be inserted – Providing enhancements for LINQ to XML e.g. root.Descendants("statement") .Select(e => e.FirstElement())• Instrumentation library – Recording statements and branches • Text file, Binary file, Shared memory, TCP/IP, … – Basically, implemented in each language• Function call for instrumentation library – Function calls are inserted into each langs2013/3/20 ICST Testing Tools Track 2013 13
  14. 14. Finding statements in C#var decls = root.Descendants("declaration_statement");var stmts = root.Descendants("embedded_statement") .Where(e => e.FirstElement().Name() != "block");return stmts.Concat(decls); Finderstatement : declaration_statement | labeled_statement | embedded_statement ;embedded_statement : block | selection_statement // if, switch |… ; Grammar 2013/3/20 ICST Testing Tools Track 2013 14
  15. 15. Finding statements in JavaScriptreturn root.Descendants("statement") Finder .Select(e => e.FirstElement()) .Where(e => e.Name() != "statementBlock") .Where(e => e.Name() != "labeledStatement") .Where(e => e.Name() != "emptyStatement");statement : statementBlock | variableStatement | emptyStatement | expressionStatement | ifStatement |…; Grammar 2013/3/20 ICST Testing Tools Track 2013 15
  16. 16. Finding branches in Creturn root.Descendants("selection_statement") .Where(e => e.FirstElement().Value == "if") .Select(e => e.NthElement(2)); Finder We canselection_statement consider while : if ( expression ) statement as a branch | switch ( expression ) statement ;iteration_statement : while ( expression ) statement |…; Grammar 2013/3/20 ICST Testing Tools Track 2013 16
  17. 17. Finding branches in Pythonvar ifs = root.Descendants("if_stmt") .SelectMany(e => e.Elements("test"));var whiles = root.Descendants("while_stmt") .SelectMany(e => e.Elements("test"));return ifs.Concat(whiles); Finderif_stmt : ’if’ test ’:’ suite (’elif’ test ’:’ suite)* (’else’ ’:’ suite)? ;while_stmt : "while" test ":" suite ("else" ":" suite)? Grammar ; 2013/3/20 ICST Testing Tools Track 2013 17
  18. 18. Commonality between langs• Finding statements – Most of grammars have similar statement nodes 1. Finding statement nodes by a name 2. Excluding unused statement nodes • Should consider what is a statement• Finding branches – Most of grammars have branch statements 1. Finding branch statement nodes by a name 2. Extracting condition nodes (or find directory) • Should consider what is a branch (if, while, … ?) 2013/3/20 ICST Testing Tools Track 2013 18
  19. 19. Customization • Changing AST Finders e.g. Extracting only if statementsvar ifs = root.Descendants("if_stmt") .SelectMany(e => e.Elements("test"));var whiles = root.Descendants("while_stmt") .SelectMany(e => e.Elements("test"));return ifs.Concat(whiles); Finderreturn root.Descendants("if_stmt") .SelectMany(e => e.Elements("test")); Finder 2013/3/20 ICST Testing Tools Track 2013 19
  20. 20. Sample applications for FizzBuzzString answer(int n) { @Test void fizz3() { if (n % 15 == 0) { assertEquals("fizz", // should return fizzbuzz FizzBuzz.answer(3)); return "fizzzbuzzz"; } } else if (n % 3 == 0) { @Test void fizz6() { return "fizz"; assertEquals("fizz", } else if (n % 5 == 0) { FizzBuzz.answer(6)); return "buzz"; } } else { @Test void fizzBuzz15() { return "" + n; assertEquals("fizzbuzz", } FizzBuzz.answer(15));} } Java JUnit 2013/3/20 ICST Testing Tools Track 2013 20
  21. 21. Apps: Coverage reporter String answer(int n) {1: executed if (n % 15 == 0) { Statement // should return fizzbuzz coverage2: executed return "fizzzbuzzz"; 57%3: executed } else if (n % 3 == 0) { (4 / 7)4: executed return "fizz";5: } else if (n % 5 == 0) {6: return "buzz"; } else {7: return "" + n; } } Java 2013/3/20 ICST Testing Tools Track 2013 21
  22. 22. Apps: Fault localization risk String answer(int n) {1: 0.5 if (n % 15 == 0) { // should return fizzbuzz Calculate2: 1.0 return "fizzzbuzzz"; risk metric3: 0 } else if (n % 3 == 0) {4: 0 return "fizz";5: NaN } else if (n % 5 == 0) {6: NaN return "buzz"; We can change } else { definition of7: NaN return "" + n; risk metric by } writing script } Java 2013/3/20 ICST Testing Tools Track 2013 22
  23. 23. Apps: Test minimization@Test void fizz3() { assertEquals("fizz", FizzBuzz.answer(3)); fizz3 and fizz6 are}@Test void fizz6() { duplicated test cases assertEquals("fizz", because they execute FizzBuzz.answer(6)); same path}@Test void fizzBuzz15() { But different test assertEquals("fizzbuzz", oracles have different FizzBuzz.answer(15));} detection capabilities .. JUnit2013/3/20 ICST Testing Tools Track 2013 23
  24. 24. Limitation• Data flow coverage – OCCF cannot support to implement data flow analyzer due to language independence.• Supportable languages – OCCF cannot support languages where we cannot insert instrumentation code.• Measurement environment – OCCF works on only .NET Framework and Mono 2013/3/20 ICST Testing Tools Track 2013 24
  25. 25. Conclusion• Problems – Differences between coverage tools – High development costs for various langs• OCCF Solution – Providing common code – Extracting commonality between langs • Most of ASTs have similar structures• Please contact us if you want to develop coverage tools with OCCF – We would like to support you 2013/3/20 ICST Testing Tools Track 2013 25

×