Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Fullstack End-to-End Test
Automation with Node.js
One year later
SF Selenium Meetup @ Saucelabs
Chris Clayman
Mek Srunyu S...
2
Agenda
● Background
○ What we presented last year
● Async / Await - an alternative to Webdriver’s built-in control flow....
3
Background
Back in 2015, we started looking at node.js
for end-to-end functional test framework.
● Kept Node.js adoption...
4
Why we chose node.js
QA, Automation engineers
Frontend engineers
Backend engineers
Java
Javascript
Java
Python
Javascrip...
5
What we presented last year
Input / update
data
Get data
Input / update
data
Get data
UI Framework : node.js
● selenium-...
6
Selenium-webdriver usage
7
Mocha usage
8
Heavily dependent on selenium-webdriver control flows
http://seleniumhq.github.io/selenium/docs/api/javascript/module/se...
9
Achieving Sync-like Code
Code written using Webdriver Promise Manager
JavaScript selenium tests using Promise Manager
dr...
10
MochaJS with Webdriver Wrapper
Provided Mocha test wrapper with Promise Manager
Selenium-webdriver’s wrapper for mocha ...
11
test.it('Verify data from both frontend and backend', function() {
var webClient = new WebClient();
var projectFromBack...
Alternatives to Webdriver’s Built-in
Control Flow
Async / Await
JS and Webdriver: the Good, the Bad...
Good Stuff
● Functional programming
● More Collaboration with
front-end development...
14
...and the Ugly
const promise = require('selenium-webdriver').promise
PageObject.prototype.getMessages = function() {
c...
15
...and the (kind of) Ugly
test.it('Archives job', function () {
const flow = promise.controlFlow();
let job;
flow.execu...
16
JobsPage.prototype.getJobList = function () {
this.waitForDisplayed(By.css('.job-table'));
const jobNames = [];
const d...
17
As if we had trainer
wheels on...
Doing complex things
with selenium-webdriver
promise manager ended
up taking more tim...
18
What We Wanted
19
Async/Await
Introducing Async/Await
● ES2016 language specification grabbed from C#
● Async functions awaits and return...
20
function notAsync () {
foo().then((bar) => {
console.log(bar)
});
}
Async/Await
async function isAsync() {
const bar = ...
21
PageObject.prototype.getMessages = function() {
const els = this.driver.findElements(By.css('.classname');
const defer ...
22
test.it('Archives job', function () {
const flow = promise.controlFlow();
let job;
flow.execute(function* () {
// Creat...
23
How to use Async / Await
How did we get Async / Await?
● Babel compiles latest JS syntax to ES5
compatible code
● Babel...
24
'use strict';
var BasePage = require('./BasePage');
var By = require('selenium-webdriver').By;
//Constructor for the To...
25
What we presented last year
Input / update
data
Get data
Input / update
data
Get data
UI Framework : node.js
● selenium...
26
Current
Database
Backend : Go, Python
Browser
Read & Write
- Click, Drag
- Enter text
- Get text
Frontend : Javascript
...
27
Async / Await Cons
● Error handling / stack tracing
● Forces devs to actually understand promises
● Async or bust - dif...
28
● Async / await was almost designed for browser
automation ‘desync’ing. The glove fits
● Refactoring out of Promise Man...
Custom reporter with screenshots from
Saucelabs
Extending MochaJS
30
Why Roll Your Own Reporter?
● Start a conversation with developers:
○ What is the most important data you need to see i...
31
Keeping Reporting Simple
32
Parallelization Try mocha-parallel-tests. But be careful!
33
● Take advantage of scalability to deliver what
other devs want
● Keep an eye on the OS-community for new
solutions
The...
Type-safe JavaScript
Facebook FlowType
35
Flowtype and Eslint
Don’t like the willy-nilly-ness of JavaScript? Lint! Type check!
● Static type analysis is availabl...
And best practices
Robust visual diffs
37
Robust automated visual tests
Powered by
38
Size of test
The test pyramid*
Cost
Time
Coverage
Martin Fowler - Test Pyramid, Google Testing Blog
2014 Google Test Au...
39
Google suggests a 70/20/10 ratio for test
amount allocation. (Of a total 100 tests)
○ 70% unit tests,
○ 20% integration...
40
Visual diff test pyramid
Visual
tests
UI Selenium
tests
REST API
tests
QA
test pyramid
Big
E2E tests
Medium
Integration...
41
Applitools commandline image diff
Powered by
Using Applitools eyes.images SDK
var Eyes = require('eyes.images').Eyes;
a...
42
Visual diff test pyramid
Browser
Screenshots
Backend
image diffs
Powered by
Test trend analysis and etc.
Whats next ?
44
Important attributes of CI/CD systems
● Trustworthy results
● Tests that do not add value
are removed
● Tests have the ...
45
Test trend analytics
Work in Progress!
46
Testability as a product requirement
Collaboration between Frontend and Automation teams
Surface data attributes in the...
47
Github links
The Airware Github repos https://github.com/airware
Our Async / Await Example
https://github.com/airware/w...
Thank you
Questions ?
Upcoming SlideShare
Loading in …5
×

Fullstack End-to-end test automation with Node.js, one year later

3,533 views

Published on

Airware's cloud automation team returns with a year’s worth of lessons learned, and will share the challenges involved with building a full-stack test automation framework with Node.js while using the latest and greatest in JavaScript tools.

Topics
Async / Await - an alternative to Webdriver’s built-in control flow. Limitations with control flow. Use Babel to write the latest ES6 JavaScript syntax. Custom reporter with screenshots from Sauce Labs. Parallel tests and accurate reporting.
Type-safe JavaScript with Facebook’s Flow-type library.
Robust visual diffs

Published in: Technology

Fullstack End-to-end test automation with Node.js, one year later

  1. 1. Fullstack End-to-End Test Automation with Node.js One year later SF Selenium Meetup @ Saucelabs Chris Clayman Mek Srunyu Stittri
  2. 2. 2 Agenda ● Background ○ What we presented last year ● Async / Await - an alternative to Webdriver’s built-in control flow. ○ Limitations with control flow ○ Use Babel to write the latest ES6 JavaScript syntax. ES6 Pageobjects ● Extending MochaJS ○ Custom reporter with screenshots from Sauce Labs ○ Parallel tests and accurate reporting ● Type-safe JavaScript with Facebook’s Flow-type library. ● Robust visual diffs ● What’s next
  3. 3. 3 Background Back in 2015, we started looking at node.js for end-to-end functional test framework. ● Kept Node.js adoption in mind ○ More and more company moving to node.js ○ Share code with fullstack developers ● Team presented at San Francisco Selenium Meetup in Nov 2015 ○ Event - http://www.meetup.com/seleniumsanfrancisco/events/226089563/ ○ Recording - https://www.youtube.com/watch?v=CqeCUyoIEo8 ○ Slides - http://www.slideshare.net/MekSrunyuStittri/nodejs-and-selenium-webdriver-a-journey-from-the-java-side
  4. 4. 4 Why we chose node.js QA, Automation engineers Frontend engineers Backend engineers Java Javascript Java Python Javascript Java Ruby Javascript Node.js Company A Company B Company C Node.js Javascript Node.js Go, Python Airware
  5. 5. 5 What we presented last year Input / update data Get data Input / update data Get data UI Framework : node.js ● selenium-webdriver ● mocha + wrapper ● Applitools ● co-wrap for webclient ● chai (asserts) Rest API Framework : node.js ● co-requests ● mocha ● co-mocha ● chai (asserts) ● json, jayschema WebClients Pageobjects Webclient adaptor Database Backend : Go, Python Browser Frontend : Javascript Microservice #1 Microservice #2 ... #n Rest APIs
  6. 6. 6 Selenium-webdriver usage
  7. 7. 7 Mocha usage
  8. 8. 8 Heavily dependent on selenium-webdriver control flows http://seleniumhq.github.io/selenium/docs/api/javascript/module/selenium-webdriver/lib/promise.html What is the promise manager / control flow ● Implicitly synchronizes asynchronous actions ● Coordinate the scheduling and execution of all commands. Maintains a queue of scheduled tasks and executing them. What we presented last year driver.get('http://www.google.com/ncr'); driver.findElement({name: 'q'}).sendKeys('webdriver'); driver.findElement({name: 'btnGn'}).click(); driver.get('http://www.google.com/ncr') .then(function() { return driver.findElement({name: 'q'}); }) .then(function(q) { return q.sendKeys('webdriver'); }) .then(function() { return driver.findElement({name: 'btnG'}); }) .then(function(btnG) { return btnG.click(); }); The core Webdriver API is built on top of the control flow, allowing users to write the below. Instead of thatThis
  9. 9. 9 Achieving Sync-like Code Code written using Webdriver Promise Manager JavaScript selenium tests using Promise Manager driver.get("http://www.google.com"); driver.findElement(webdriver.By.name('q')).sendKeys('webdriver'); driver.findElement(webdriver.By.name('btnG')).click(); driver.getTitle().then(function(title) { console.log(title); }); Equivalent Java code driver.get("http://www.google.com"); driver.findElement(By.name("q")).sendKeys("webdriver"); driver.findElement(By.name("btnG")).click(); assertEquals("webdriver - Google Search", driver.getTitle()); Hey, we look similar now!
  10. 10. 10 MochaJS with Webdriver Wrapper Provided Mocha test wrapper with Promise Manager Selenium-webdriver’s wrapper for mocha methods that automatically handles all calls into the promise manager which makes the code very sync like. var test = require('selenium-webdriver/testing'); var webdriver = require('selenium-webdriver'); var By = require('selenium-webdriver').By; var Until = require('selenium-webdriver').until; test.it('Login and make sure the job menu is there', function() { driver.get(url, 5000); driver.findElement(By.css('input#email')).sendKeys('useremail@email.com'); driver.findElement(By.css('input#password')).sendKeys(password); driver.findElement(By.css('button[type="submit"]')).click(); driver.wait(Until.elementLocated(By.css('li.active > a.jobs'))); var job = driver.findElement(By.css('li.active a.jobs')); job.getText().then(function (text) { assert.equal(text, 'Jobs', 'Job link title is correct'); }); }); Mocha wrapper makes the code very “synchronous” like.
  11. 11. 11 test.it('Verify data from both frontend and backend', function() { var webClient = new WebClient(); var projectFromBackend; // API Portion of the test var flow = webdriver.promise.controlFlow(); flow.execute(function *(){ yield webClient.login(Constants.USER001_EMAIL, Constants.USER_PASSWORD); var projects = yield webClient.getProjects(); projectFromBackend = projectutil.getProjectByName(projects, Constants.QE_PROJECT); }); // UI Portion of the test var login = new LoginPage(driver); login.enterUserInfo(Constants.USER001_EMAIL, Constants.USER_PASSWORD); var topNav = new TopNav(driver); topNav.getProjects().then(function (projects){ Logger.debug('Projects from backend:', projectsFromBackend); Logger.debug('Projects from frontend:', projects); assert.equal(projectsFromBackend.size, projects.size); }); Heavily dependent on the promise manager / control flow Here we handle execution order including a generator call What We Presented Last Year Context switching to REST API calls
  12. 12. Alternatives to Webdriver’s Built-in Control Flow Async / Await
  13. 13. JS and Webdriver: the Good, the Bad... Good Stuff ● Functional programming ● More Collaboration with front-end development teams ● JavaScript Developers writing Selenium tests ● Fast paced open source community ● Able to build things really quick ● JavaScript is fun! Bad Stuff ● Webdriver’s Control Flow & Promise Manager ○ Not agnostic ○ Parent variable declarations ○ Iteration can be hacky ● Context switching between Webdriver and non-Webdriver asynchronous function calls
  14. 14. 14 ...and the Ugly const promise = require('selenium-webdriver').promise PageObject.prototype.getMessages = function() { const els = this.driver.findElements(By.css('.classname'); const defer = promise.defer(); const flow = promise.controlFlow(); flow.execute(function* () { const textArray = yield els.map((el) => { return el.getText(); }); defer.fulfill(textArray); }); return defer.promise; } We want our tests to be: ● Readable / Flat structure ✔ ● Agnostic / Context-free ❌ ● De-asynchronous ✔ ● In line with ECMA standards ❌ Context Switch
  15. 15. 15 ...and the (kind of) Ugly test.it('Archives job', function () { const flow = promise.controlFlow(); let job; flow.execute(function* () { // Create a job through API job = yield webClient.createJob(); driverutil.goToJob(driver, job.id); }); const jobInfo = new ViewJob(driver); jobInfo.clickArchive(); jobInfo.isModalDisplayed().then((displayed) => { assert.isTrue(displayed, 'Modal should be displayed'); }); flow.execute(function* () { yield webClient.deleteJob(job.id); }); }); We want our tests to be: ● Readable / Flat structure ● Agnostic / Context-free ❌ ● De-asynchronous ✔ ● In line with ECMA standards ❌ Context Switch Context Switch
  16. 16. 16 JobsPage.prototype.getJobList = function () { this.waitForDisplayed(By.css('.job-table')); const jobNames = []; const defer = promise.defer(); const flow = promise.controlFlow(); const jobList = this.driver.findElement(By.css('.job-table')); // get entries flow.execute(() => { jobList.findElements(By.css('.show-pointer')).then((jobs) => { // Get text on all the elements jobs.forEach((job) => { let jobName; flow.execute(function () { job.findElement(By.css('.job-table-row-name')).then((element) => { element.getText().then((text) => { jobName = text; }); // look up more table cells... }); }).then(() => { jobNames.push({ jobName: jobName }); }); }); }); }).then(() => { // fulfill results defer.fulfill(jobNames); }); return defer.promise; }; ...and the Really Ugly Not planning for complexity + promise chaining = We want our tests to be: ● Readable / Flat structure ❌ ● Agnostic / Context-free ❌ ● ‘De-asynchronous’ ✔ ● In line with ECMA standards ❌
  17. 17. 17 As if we had trainer wheels on... Doing complex things with selenium-webdriver promise manager ended up taking more time and being more cumbersome. What It Felt Like
  18. 18. 18 What We Wanted
  19. 19. 19 Async/Await Introducing Async/Await ● ES2016 language specification grabbed from C# ● Async functions awaits and returns a promise ● No callbacks, no control flow libraries, no promise chaining, nothing but simple syntax. ● ‘De-asynchronous’ done easy
  20. 20. 20 function notAsync () { foo().then((bar) => { console.log(bar) }); } Async/Await async function isAsync() { const bar = await foo(); console.log(bar); } Given function foo() that returns a promise... ES5 Javascript ES2016 Latest Javascript
  21. 21. 21 PageObject.prototype.getMessages = function() { const els = this.driver.findElements(By.css('.classname'); const defer = promise.defer(); const flow = promise.controlFlow(); flow.execute(function* () { const textArray = yield* els.map((el) => { return el.getText(); }); defer.fulfill(textArray); }); return defer.promise; } async getMessages() { const els = await this.driver.findElements(By.css('.classname'); return Promise.all(els.map((el) => { return el.getText(); })); } We want our tests to be ● Readable / Flat structure ✔ ● Portable / Context-free ✔ ● De-asynchronous ✔ ● In line with ECMA standards ✔ Promise Manager vs. Async/Await
  22. 22. 22 test.it('Archives job', function () { const flow = promise.controlFlow(); let job; flow.execute(function* () { // Create a job through API job = yield webClient.createJob(); driverutil.goToJob(driver, job.id); }); const jobInfo = new ViewJob(driver); jobInfo.clickArchive(); jobInfo.isModalDisplayed().then((displayed) => { assert.isTrue(displayed, 'Modal should be displayed'); }); flow.execute(function* () { yield webClient.deleteJob(job.id); }); }); Promise Manager vs. Async/Await it('Archives job', async function () { const job = await webClient.createJob(); await driverutil.goToJob(driver, job.id); const jobInfo = new ViewJob(driver); await jobInfo.clickArchive(); const displayed = await jobInfo.isModalDisplayed(); assert.isTrue(displayed, 'Modal should be displayed'); await webClient.deleteJob(job.id); }); We want our tests to be ● Readable / Flat structure ✔ ● Portable / Context-free ✔ ● De-asynchronous ✔ ● In line with ECMA standards ✔
  23. 23. 23 How to use Async / Await How did we get Async / Await? ● Babel compiles latest JS syntax to ES5 compatible code ● Babel-register can be a pre-runtime compiler in Mocha. ● See the repo!
  24. 24. 24 'use strict'; var BasePage = require('./BasePage'); var By = require('selenium-webdriver').By; //Constructor for the Top Navigation Bar function TopNav(webdriver) { BasePage.call(this, webdriver); this.isLoaded(); } //BasePage and Constructor wiring TopNav.prototype = Object.create(BasePage.prototype); TopNav.prototype.constructor = TopNav; TopNav.prototype.isLoaded = function () { this.waitForDisplayed(By.css('.options')); return this; }; TopNav.prototype.openProjectDropdown = function () { this.waitForDisplayed(By.css('.options')); this.waitForEnabled(By.css('.options ul:nth-of-type(1)')); this.click(By.css('.options ul:nth-of-type(1)')); return this; }; 'use strict'; import BasePage from './BasePage'; import { By } from 'selenium-webdriver'; import ui from './../util/ui-util'; export default class TopNav extends BasePage { //Constructor for the Top Navigation Bar constructor(webdriver: WebDriverClass) { super(webdriver); } async isLoaded(): Promise<this> { await ui.waitForDisplayed(this.driver, By.css('.options')); return this; } async openProjectDropdown(): Promise<this> { await ui.waitForDisplayed(this.driver, By.css('.options')); await ui.waitForEnabled(this.driver, By.css('.options ul:nth-of-type(1)')); await ui.click(this.driver, By.css('.options ul:nth-of-type(1)')); return this; } } ES6 Pageobjects with flow annotation
  25. 25. 25 What we presented last year Input / update data Get data Input / update data Get data UI Framework : node.js ● selenium-webdriver ● mocha + wrapper ● Applitools ● co-wrap for webclient ● chai (asserts) Rest API Framework : node.js ● co-requests ● mocha ● co-mocha ● chai (asserts) ● json, jayschema WebClients Pageobjects Webclient adaptor Database Backend : Go, Python Browser Frontend : Javascript Microservice #1 Microservice #2 ... #n Rest APIs
  26. 26. 26 Current Database Backend : Go, Python Browser Read & Write - Click, Drag - Enter text - Get text Frontend : Javascript Microservice #1 Microservice #2 ... #n Rest APIs Read & Write API Calls Get, Post, Put, Delete UI tests ● selenium-webdriver ● requests Rest API tests ● requests ● Json, jayschema WebClients - Job Client - Project Client - File Client - etc.. UI Pageobjects - Project List - Job List - Job info - Maps - Annotations Node.js common tooling ● Mocha ● Babel (ES6) ● Bluebird ● Asserts (Chai) ● Flow type (Facebook) Visual tests ● Applitools (visual diffs)
  27. 27. 27 Async / Await Cons ● Error handling / stack tracing ● Forces devs to actually understand promises ● Async or bust - difficult to incrementally refactor ● Promise Wrapper optimizations gone ● Chewier syntax than control flow - `await` everywhere
  28. 28. 28 ● Async / await was almost designed for browser automation ‘desync’ing. The glove fits ● Refactoring out of Promise Manager is cumbersome ● Simplifying test syntax -> less dependencies and opinion -> happy and efficient devs The Bottom Line...
  29. 29. Custom reporter with screenshots from Saucelabs Extending MochaJS
  30. 30. 30 Why Roll Your Own Reporter? ● Start a conversation with developers: ○ What is the most important data you need to see in order to be efficient and successful? ● 5 things important to Airware cloud team: ○ Failures First ○ Flakiness vs. Failure ○ Assertion messaging ○ Screenshots ○ Video
  31. 31. 31 Keeping Reporting Simple
  32. 32. 32 Parallelization Try mocha-parallel-tests. But be careful!
  33. 33. 33 ● Take advantage of scalability to deliver what other devs want ● Keep an eye on the OS-community for new solutions The Bottom Line...
  34. 34. Type-safe JavaScript Facebook FlowType
  35. 35. 35 Flowtype and Eslint Don’t like the willy-nilly-ness of JavaScript? Lint! Type check! ● Static type analysis is available with Flow and TypeScript ● Both have awesome IDE / editor plugins ● We picked Flow in order to start annotating our code piece by piece. ● We added Webdriver, Mocha, and Chai type definitions ● ESlint has some great plugins related to test structuring. We’ve also written our own ● The bottom line: the best time to capture test errors is as you write them
  36. 36. And best practices Robust visual diffs
  37. 37. 37 Robust automated visual tests Powered by
  38. 38. 38 Size of test The test pyramid* Cost Time Coverage Martin Fowler - Test Pyramid, Google Testing Blog 2014 Google Test Automation Conference Big E2E tests Medium Integration tests Small Unit tests As you move up the pyramid, your tests gets bigger. At the same time the number of tests (pyramid width) gets smaller. ● Cost: Test execution, setup time and maintenance is less as you come down ● Feedback: Detection cycle (time) is less as you come down ● Stability: Smaller tests are less flaky ● Coverage: E2E workflow tests have better coverage but with tradeoffs; longer time, flakiness # Number of tests
  39. 39. 39 Google suggests a 70/20/10 ratio for test amount allocation. (Of a total 100 tests) ○ 70% unit tests, ○ 20% integration tests ○ 10% end-to-end tests The exact mix will be different for each team, but we should try to retain that pyramid shape. The ideal ratio Google Testing Blog 2014 Google Test Automation Conference 10 20 70
  40. 40. 40 Visual diff test pyramid Visual tests UI Selenium tests REST API tests QA test pyramid Big E2E tests Medium Integration tests Small Unit tests Engineering test pyramid Browser Screenshots Backend image diffs Visual diff test pyramid Apply the test pyramid concept to automated visual test suite ● Browser screenshot tests are big E2E tests ● Add smaller visual testing with commandline image diffs. No browsers involved.
  41. 41. 41 Applitools commandline image diff Powered by Using Applitools eyes.images SDK var Eyes = require('eyes.images').Eyes; await image = getImage("store.applitools.com","/download/contact_us.png/" + version); // Visual validation point #1 await eyes.checkImage(img, 'Contact-us page'); ● Downloads or streams the images as PNG format ● Uploads images to validate against Applitools service More info - https://eyes.applitools.com/app/tutorial.html?accountId=m989aQAuq8e107L5sKPP9tWCCPU10JcYV8FtXpBk1pRrlE110 Thanks Liran Barokas !!
  42. 42. 42 Visual diff test pyramid Browser Screenshots Backend image diffs Powered by
  43. 43. Test trend analysis and etc. Whats next ?
  44. 44. 44 Important attributes of CI/CD systems ● Trustworthy results ● Tests that do not add value are removed ● Tests have the privilege (not the right) to run in CI ● Metadata from build and test results ● Trend analysis ● Is the feature ready to ship ● Cutting edge technology ○ Visual diff ■ Applitools ○ Kubernetes ○ Containers ○ Unikernels Denali Lumma (Uber), testing in 2020
  45. 45. 45 Test trend analytics Work in Progress!
  46. 46. 46 Testability as a product requirement Collaboration between Frontend and Automation teams Surface data attributes in the UI DOM ● uuids - test can cross validate by making API calls in UI tests ● Image group, image names, photo clustering attributes ● etc.. Testability as a product requirement! :)
  47. 47. 47 Github links The Airware Github repos https://github.com/airware Our Async / Await Example https://github.com/airware/webdriver-mocha-async-await-example Flowtype interfaces for Webdriver, Chai, Mocha https://github.com/airware/forseti-flow-interfaces
  48. 48. Thank you
  49. 49. Questions ?

×