Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Improving software quality and devop automation with STAMP, OW2con'16, Paris.

251 views

Published on

DevOps has emerged, as a major cultural movement to handle the need for increased agility in software development. While this movement is loosely bound to development methods, a number of practices have emerged to operationalize this extreme agility: loosely coupled software architectures meant to support incremental updates and build (services or micro services with clear APIs); a very high degree of automation at multiple stages of the development lifecycle. Quality assurance is a major challenge in this context. There is very little time for manual testing and the responsibility of bug detection is placed upon the automated test suites.
STAMP (Software Testing AMPlification), is a new European R&D project, which aims to bolster automated test suites through the automatic transformation of test assets. The key technical challenge that STAMP aims at overcoming is to reduce the cost due to regression bugs that propagate to production, through advanced research in automatic test generation. The key novelty of our research agenda is to leverage existing assets (such as test cases or execution logs) in order to increase test effectiveness. This innovative research is at the crossroads of program analysis and transformation, software testing, automatic deployment and search-based software engineering.

Published in: Technology
  • Be the first to comment

Improving software quality and devop automation with STAMP, OW2con'16, Paris.

  1. 1. STAMP Software testing amplification to increase quality in DevOps 1
  2. 2. 2 DevOps
  3. 3. 3 DevOps Test assets are available BUT They are manually developed and their quality is not sufficient
  4. 4. 4 DevOps The main objective of STAMP is to automatically transform existing test assets in order to detect regression bugs before production and drive down the cost of software testing.
  5. 5. 5 STAMP in DevOps
  6. 6. Continuous test amplification 6
  7. 7. Continuous test amplification 7 behav. diff.
  8. 8. Continuous test amplification 8 behav. diff.
  9. 9. Continuous test amplification 9 behav. diff. perf. diff.
  10. 10. Continuous test amplification 10 behav. diff. perf. diff.
  11. 11. Continuous test amplification 11 behav. diff. perf. diff.
  12. 12. Unit test cases amplification 12 Program P Input space Output space state
  13. 13. Unit test cases amplification 13 Program P Input space Output space state
  14. 14. DSpot: amplification of JUnit test suites 14 original test suite and program under test update in the program under test P DSpot TS P' TS++
  15. 15. DSpot •First experiments with amplification •Automatic transformation of Junit test cases 15 P TS P' TS+ search-based input space amplification observation space amplification run and look for differences TS++
  16. 16. Search-based input space amplification • Mutate existing test cases • transform literal values • replace literal values by others found in the scope • add new invocations to methods not well covered • create new objects • Search-based • ‘stack’ transformations iteratively • select the ones that have a good fitness (e.g. cover new branches) 16
  17. 17. Observation space amplification • Remove all assertions from original test cases • keep invocations that can be inside the assertion • Add new observation points • run all test cases on original program • record all values that can be observed (getters, is_*, etc.) • Filter new observation points • several runs of all test cases with observation points on the original program • discard all points that vary (e.g., OS-dependent values, IO, etc.) 17
  18. 18. Example 18 public void testSearch() { final ArrayStack<E> stack = makeObject(); stack.push(((E)("First Item"))); stack.push(((E)("Second Item"))); assertEquals("Top item is 'Second Item'", 1, stack.search("Second Item")); assertEquals("Next Item is 'First Item'", 2, stack.search("First Item")); assertEquals("Cannot find 'Missing Item'", -1, stack.search("Missing Item"));} public void testSearch_cf5() { final ArrayStack<E> stack = makeObject(); stack.push(((E)("First Item"))); stack.push(((E)("Second Item"))); E o_5 = stack.peek(); assertFalse(((java.lang.String)o_5).isEmpty()); E o_6 = stack.pop(); assertFalse(((java.lang.String)o_6).isEmpty()); Object o_7_0 = stack.search("Second Item"); Object o_9_0 = stack.search("First Item"); assertEquals("Cannot find 'Missing Item'", -1, stack.search("Missing Item")); assertEquals(((java.lang.String)o_5).length(), 11); assertEquals(((java.lang.String)o_6).length(), 11);} new calls and asserts extracted from asserts new asserts
  19. 19. Methodology 19 Unit test ampli Config. test ampli Online test ampli
  20. 20. Methodology 20 Micro service architecture Unit test ampli Config. test ampli Online test ampli
  21. 21. Methodology 21 Micro service architecture Unit test ampli Config. test ampli Online test ampli Information management Software quality Cloud computing E-Health Smart cities
  22. 22. STAMP •3-year project •9 EU partners •Full OSS •Starts 01/12/16 22

×