Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Automated acceptance test

1,071 views

Published on

How to create a stable automated test suite in DevOps.

Published in: Software
  • D0WNL0AD FULL ▶ ▶ ▶ ▶ http://1url.pw/xqXhJ ◀ ◀ ◀ ◀
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Automated acceptance test

  1. 1. Automated Acceptance Test ~ How to build it stable & maintainable
  2. 2. Bryan QA Programmer Automation Engineer DevOps Evangelist & Facilitator Big Data Ad Exchange (RTB) Low Latency High Concurrency (We are hiring)
  3. 3. Automated Acceptance Test In Continuous Delivery • “Are we good to go live?” - Test jobs are your “eyes and ears” - Optimize for them! • Test code equals production code Common problems: • Costly to maintain test buckets • Spent much time running and debugging false alarms
  4. 4. Acceptance Test “ An Executable Specification of System Behaviour ”
  5. 5. Culture ~ Responsibilities?
  6. 6. < 100 ms What’s AD Exchange
  7. 7. “ Don’t have a separate Testing/QA team! Quality is down to everyone - Developer owns Acceptance Tests!! ” ― David Farley, Pipeline Conf. 2015
  8. 8. “ Not enough to have acceptance test and acceptance test got to be created and maintained by developer will make code more testable and maintainable. ” ― Jez Humble, Pipeline Conf. 2016
  9. 9. don’t hire too many dedicated testers developer relies on them, lazier and write more bugs hire people who can development and test functions
  10. 10. code review occupies a central position no separate group of tester or QA people relies on automated testing
  11. 11. “Testing is essentially the responsibility of the person who develops a given feature” (tens of thousands of regression tests) developer responsible for writing unit tests, regression tests and performance tests
  12. 12. regulated FX exchange London & Tokyo launched 2010 ~ $2 trillion traded in 2015 one of UK’s fastest growing technology companies
  13. 13. high quality - very low rate of production issues (order of magnitude below industry average) “lowest bug count we’ve ever found”
 - independent analysis by a well-known tool vendor
  14. 14. 2 million lines of code (50:50 test/production) half the codebase < 18 months old
  15. 15. Process / Discipline ~ Definition of done
  16. 16. Definition of Done • Automation task and sprint end demo • Code review • Pipeline dashboard - pipeline always in deliverable status - statistic analysis report
  17. 17. • Build and unit tests • Full acceptance against the change sets • Only delivered if above passed Definition of Done
  18. 18. • Code review of automated test scripts • Loop test suite 20 times before code merge Loop of Your Tests stage concurrency: 1, name: ‘Test Looping before merge'
 docker.image('qa/chrome-slave:2.53.0').inside('-v /dev/shm:/dev/shm --privileged=true') {
 wrap([$class: 'Xvfb', additionalOptions: '-fbdir /tmp']) {
 git branch: 'develop', url: 'http://abc.com/qa/accept-test.git'
 withEnv(["baseUrl=${baseUrl}","mySQLUrl=${mySQLUrl}"]) {
 for(int i = 0; i < count; i++) { println(“Loop of: ${i}”) sh 'mvn -B test -fae —DsuiteXmlFile=testng.xml’ }
 }
 }
 }
  19. 19. Jenkins Scriptable Build & Multibranch Pipeline
  20. 20. Immutable Environment • Dockerize environments for build & test - Always fresh - Quick - Immutable - Scalable & parallel • Dockerize testing infrastructure
  21. 21. Strategies ~ How to make good one
  22. 22. Test Automation Pyramid src: http://www.ontestautomation.com/tag/mike-cohn/
  23. 23. Test Isolation • Reliability/Repeatability - Provide consistent, repeatable results. • Isolation - Tests should not depend, or be affected by, the results of other tests: testContext - users, accounts, advertiser names etc: alias name - external services, 3rd-party integrations: stub, simulator account: ‘Johnny’ ==> ‘Johnny_4534031’ book: ‘DevOps 101’ ==> ‘DevOps 101_1234567’
  24. 24. Parallelisation • Commit stage < 15 mins • Acceptance Test stage < 45 mins • Fail FASTer! • Radical parallelisation - throw-away environments (e.g. containers)
  25. 25. Separation of Concern • Use Domain Specific Language (DSL) - Focus on “What” not “How” • Ease of Development - Hide details of how the tests talk to the SUT • Ease of Maintenance - When tests break, we Identify the problem and fix it quickly.
  26. 26. ‘What’ Not ‘How’
  27. 27. ‘What’ Not ‘How’
  28. 28. ‘What’ Not ‘How’
  29. 29. Focus on ‘What’ Not ‘How’
  30. 30. API Example @Given("^I create a advertiser with those fields fill : name '(.+)', state '(.+)', currency '(.+)'$")
 public void createAdvertiser(final String name, final String state, final String currency) {
 advertiserAPI.createDefaultAdvertiser(name, state, currency);
 } package com.vpon.dsp.driver.web.rest.service.adv;
 public class AdvertiserServiceImpl {
 static { AdvertiserService advService = ServiceGenerator.createService(AdvertiserService.class); }
 public static String createAdvertiser(Advertiser obj) throws IOException{
 Call<Advertiser> advExec = advService.createAdvertiser(obj);
 Response<Advertiser> resp = advExec.execute(); // id should not be null if created successfully
 if(resp.isSuccessful() && null != resp.body().id) {
 ... }}} package com.vpon.dsp.dsl.web;
 public String createDefaultAdvertiser(String name) {
 Advertiser obj = defaultAdvertiser(name);
 return AdvertiserServiceImpl.createAdvertiser(obj);
 } DSL layer: Driver layer: Test case layer:
  31. 31. Implementation ~ Right tools
  32. 32. API Test “ API is more than just a status code, Verify every part of it.”
  33. 33. API Test
  34. 34. Concerns? • Support DevOps - high frequent delivery & BDD • Maintainability - UI, API are volatile, avoid boilerplate code • Tools: SoapUI, REST-assured, POSTMAN, Unirest
  35. 35. Test Case Example
  36. 36. Goal @end2end
 Feature: LineItem end-to-end system targeting test 
 Background: An Advertiser, LineItem with different System targeting conditions created
 Given There is a 'advTargeting' advertiser and following 'System' targeting lineItems:
 | name | category | targeting |
 | LI_OS_iOS | OS Family | Apple iOS |
 | LI_LANG_Eng | Language | English |
 
 Scenario Outline: Bid accordingly with lineItem targeting setting
 Given I enable '<lineItemName>' lineItem
 When I send a bid request with '<criteria>' targeting only
 And I send a 'default' bid request with no targeting criteria
 Then I should receive '1' successful bid response
 
 Examples:
 | lineItemName | criteria |
 | "LI_OS_iOS" | "iOS" |
 | "LI_LANG_Eng" | "English" |
  37. 37. Retrofit A type-safe HTTP client for Android and Java
  38. 38. GUI (Flaky) Test Stability is super important for automation in CD • 1% failure rate with 100 test scripts ( 0.99¹⁰⁰) = 63% chance of failure • You can’t get away with flaky tests in CI/CD • Spend much time debugging false alarms
  39. 39. Selenium + jQuery Selector • Implement JQueryBy class - Tells Selenium how to find element with jQuery locator • If no jQuery in application page - Inject one for testing • jQuery - Powerful API for navigating and selecting content - Manipulate UI for easy assertion - CSS based, a whole lot better than XPath - Has “qualified” feature (ex: has, contains) for filtering • No xPath locator - Slow & tight to DOM structure - Easily broken with html changes - Bad readability and hard to maintain
  40. 40. Selenium PageObject • Separate test logic from web implementation - No html stuff (locator, Selenium code, …) in test case • Increase maintainability - No need to update all hundreds of TCs when UI changes - Object classes are the only place to modify • Increase stability - More tuning on the framework, it gets more stable src: Martin
  41. 41. Handling Wait in UI Test User Perform Action if_Async CheckPageReady() Check Ajax Request Complete Wait Dynamic Element Presence Make Assertion Check DB / Queue Status Match Thread.sleep()
  42. 42. Handling Wait in UI Test • Each page loading is following the same ‘wait’ checking procedure • Each page can define it’s own ‘getPageLoadCondition()’ public T initPage(Class<T> clazz) { T page = PageFactory.initElements(getDriver(), clazz); checkPageReady(); //<== check for DOM, JS lib ExpectedCondition pageLoadCondition = ((AbstractPage) page).getPageLoadCondition(); 
 // for extended page does not want to check loading condition ... if (null != pageLoadCondition) { waitForCondition(pageLoadCondition); //<== check for page’s load condition
 }
 return page;
 }
  43. 43. Handling Wait in UI Test private static void checkPageReady() {
 /* for generic loading, but not all browsers compatible... final String chk1 = “(document.readyState==='complete'|| document.readyState===‘interactive')"; */
 final String chk1 = "(typeof jQuery != 'undefined') && ($(window).load(function() {return true;}))";
 //for jQuery loaded and no ajax requests pending
 final String chk2 = "($.active === 0)";
 final String condition = "return " + chk1 + " && " + chk2 + ";";
 ExpectedCondition<Boolean> response = new ExpectedCondition<Boolean>() {
 public Boolean apply(WebDriver d) {
 return (Boolean)((JavascriptExecutor) d).executeScript(condition);
 }
 };
 new WebDriverWait(getDriver(), LOAD_TIMEOUT).until(response);
 }
  44. 44. //for those with data grid loading
 public static ExpectedCondition getDataLoadingCondition() {
 LogUtils.printDebugMessage(logger, "Before returning _loading wait condition...");
 return new ExpectedCondition<Boolean>() {
 public Boolean apply(WebDriver d) {
 boolean result = (Boolean)((JavascriptExecutor) d).executeScript(
 "return ($("div[id$='_loading'][style*='display: none;']").length===2" +
 " && $.active===0 “ + “ && $("div[id$='_loading'][style$='display: block;']").length===0)"
 );
 return result;
 }
 };
 } Wait Ajax Data Loading (1) $.active === 0 => No ajax reqs pending (2) $(”div[id$=‘_loading’]…..”).length===0) => loading icon disappear
  45. 45. Wait Dynamic Web Element public String getToolTipText() {
 SeleniumDriver.onPage(ByJQuery.jQuerySelector("div#content"));
 . . . . . 
 //mouseover
 new Actions(getDriver()).moveToElement(e).build().perform();
 String toolTipLocator = "div.tooltip[style$='display: block;']";
 LogUtils.printMessage(logger, "trying to get tooltips of a warning ...");
 return SeleniumDriver.waitUntilElementPresence( ByJQuery.jQuerySelector(toolTipLocator)).getText();
 }
  46. 46. Capacity Testing • Performance testing for components • Long run and stress • Use production traffic - boost confidence level • Tools: Gor, Gatling
  47. 47. Reference [How google Test Software] [Continuous Delivery: Reliable Software Releases] [From research paper “Moving Fast with Software Verification - 2015”] Others: https://dzone.com/articles/dev-centric-culture-breaking-down-the-walls http://blog.xebialabs.com/2015/06/22/guidelines-for-a-successful-test-strategy/ http://www.androidwarriors.com/2015/12/retrofit-20-android-example-web.html https://gortool.com/ http://gatling.io/#/ [About Vpon] [About Me]
  48. 48. Q & A

×