Quest to the best test automation for low code development platform kherrazi rachid

1. The quest to the best
test automation tool for
low code development
platforms
Rachid Kherrazi
28-1-2018 version 0.1 (Proposal for the QA Test 2018)

2. Low code development
platforms
Samen aan de slag
Technology for creating applications through
configuration of functions, rather than coding
those functions
Shift from coding to modeling
Promedico has selected OutSystems as the
agile development platform
– Agility
– Time to market
– Short life cycle apps
Platform: design, implementation, integration,
deployment
Rachid Kherrazi

3. Benefits and challenges
Samen aan de slag
High speed of development, many deploys:
1. Regression testing is key
2. Without test automation, testing / QA would be a
bottleneck for the whole development
It is about modeling and code generation:
1. Full automation regression test with low maintenance is key
2. Should be robust against dynamic generated ID’s after each
deploy
Rachid Kherrazi
Example:
id="Common_CW_wt37_block_wtMainContent_SilkUIFramework_wt14_block_wtColumn1_APRO_Patterns_wt82_block_wtSections_SilkUIFramework_wt35_block_wtContent
_APRO_Patterns_wt47_block_wtTab1"

4. The quest to the best tool for low code development
platform
Criteria: Suitability & Maintainability
Robust against dynamic ID’s generated by OutSystems
Low maintenance effort
Easy to use, low learning curve, no programming skills.
And of course other aspects (cost, open source,
community)
What was our goal?
Samen aan de slag Rachid Kherrazi

5. The road to the best tool
We used well-know tool selection and implementation process (ISTQB)
Rachid KherraziSamen aan de slag

6. What were the choices?
Samen aan de slag
Tools using tekst
and images (OCR)
Tools using code and
object libraries
“What you see = what you test”
“What you code = what you test”
V.S.
Some of above tools have advanced capabilities like key-driven, data-driven, Given-When-Then
This allow you to organize test to improve readability and maintenance of scripts.
Rachid Kherrazi
2 types of tools

7. Main issue with these tools
Samen aan de slag
Sensitive for SUT configuration and screen resolutions
Closed/limited support and community
High license cost
V.S.
Some of above tools have advanced capabilities like key-driven, data-driven, Given-When-Then
This allow you to organize test to improve readability and maintenance of scripts.
Programming skills is required, (Xpath, CSS, DOM)
Maintenance high
High learning curve
Rachid Kherrazi
“What you see = what you test”
“What you code = what you test”

8. Some attempts to improve
maintainability and readability
Samen aan de slag
Separation of concern: feature file (Business Language), step
definition (programming)
Use of Page and Data Objects design patterns (in programming
part)
Robust locators (starts-with, contains, end-with…)
Image / OCR based test automation: they use advanced
capabilities like smart recording, modular reuse of components
and scenario’s
Rachid Kherrazi

9. Our choice (Architecture)
Samen aan de slag
Specflow Step definition UI
Fitnesse Fixture UI
Fitnesse
Hsac
BrowserTest
Selenium
web driver
UI
ICTestAutomation CitraTest OCR/images UI
Logical test cases
Should be done manually
Good design, maintenance and skills are
required
Selenium
web driver
Selenium
web driver
Key driven
Data driven
Collaboration
Communication
Documentation
Wiki /table format
Given when then
Automatic generated
Programming skills and maintenance not required
UISelenium
web driver
UISelenium
web driver
Open source
License cost
Selenide
Our best choice
Rachid Kherrazi
Note: Tools above are only as examples for explanation purpose

10. Our choice:
Fitnesse - Hsac BrowserTest
Samen aan de slag
Easy way to set up tests: Wiki (improve
communication, collaboration and
documentation)
Write tests in natural / non-technical language
Hide (Java) implementation (programming like
step definition and fixture is not needed)
Nice reporting and monitoring overviews,…..
Advanced capabilities (modular reuse of
scenario’s, symbolic links, support development
of Domain Specific language (DSL).
Java implementatie:
….
public boolean click(final String
place) {
return clickImp(place);
}
….
Rachid Kherrazi

11. Samen aan de slag
Fitnesse
HSAC
Selenium
Custom
Wiki webserver
Fixtures (Java)
Hsac BrowserTest
Hsac XmlHttpTest
Hsac JsonHttpTest
Framework (Java)
Selenium/FitnesseHsac BrowserTest Stack
Fitnesse - Hsac BrowserTest
The framework allows us also to test web services and going beyond the UI testing
Rachid Kherrazi

12. Samen aan de slag
Architecture stack
Fitnesse-HsacBrowserTest stack
Java
Selenium
Hsac-BrowserTest
Gecko Chrome IE
Fitnesse
Galen
OS
fixture
fixture
8
Executeprogram Script.wiki
Note: it is just an illustration of the stack, not all components are mentioned
Fixtures e.g.
- string fixture
- Json fixture
- Xml fixtures
- Xls fixtures
- Promedico fixture

13. Fixtures
Samen aan de slag

14. Samen aan de slag
Galen framework
1) Define layout in galen spec file
2) And Hsac BrowserTest can check against this spec in different browser formats
Example of galen spec file
Just an extra example of new features

15. Samen aan de slag
Integration with other tools
Dashboards
Check in
Check in
Check in
Version control
Tester
Tester
Tester
Check in
Automatic Smoke test (after each deployment)
Automatic night runs (regression test)
A high-quality automated environment that supports multi-user
Rachid Kherrazi

16. Recap & Results
Samen aan de slag
Integration testing
(web services, mocks )
E2E
UI testing
The Outsystems development environment is provided with a self-
healing and validation engine (TrueChange™) that assures robust
and error-free changes across all applications and their modules.
Unit testing & reviews, critical units
Start situation:
12 developers
4 testers
0% automated testing
Current situation:
More than 60% of all test cases are fully automated.
Every night regression test (integrated in Jenkins)
This includes also security and performance test scripts
The framework allows us to build a set of effective regression test in a efficient way
Thanks to the smart advanced capabilities (key, data driven approach, symbolic links,…)
Rachid Kherrazi

17. Lessons learned-
Maintainability
Samen aan de slag
INVEST IN WEL-KNOWN MATURE OPEN SOURCE TOOLS
FOCUS ON AUTOMATING THE RIGHT THINGS
DON’T AUTOMATE LOW-VALUE FEATURES
CUT YOUR CONFIGURATION MATRIX
AVOID COMPLEX TEST SCENARIOS
KEEP TESTS GRANULAR AND INDEPENDENT
GO BEYOND THE UI FOR YOUR FUNCTIONAL TESTS
Rachid Kherrazi

18. Live demo
Samen aan de slag

19. Samen aan de slag

20. Hsac vs Selenide
Samen aan de slag
Feature Fitnesse -Hsac BrowserTest Selenide -IntelliJ
Installation Only 1 Jar file IntelliJ + Jar file
Programming skills Not required (high level logical
language)
Java,
Step definition /fixture Automatic generation Fixture
Robust against Dynamic ID’s Yes ?
Design and maintenance Not applicable design and maintenance of abstraction
layer is required
Collaboration Wiki
Living document
Specification by example
-
Test driven Supported ?
Keyword driven Scenario, modular -
Data driven Parametrization -
Tag’s Test case organization and selection
With regulare expressions
?
Screenshot Yes yes
Reporting Yes, history overview ?
Extendable yes Yes
Target group Testers Developers (evolution of developer)
Rachid Kherrazi

21. Hsac vs Selenide
Samen aan de slag
Feature Fitnesse -Hsac BrowserTest Selenide -IntelliJ
Mocks yes -
Smart waits, solution for Ajax Yes Yes
Error messages yes Yes
Collections Yes, list fixture Yes
Upload files ? Yes
Download file yes Yes
Custom matchers yes Yes
Find element by text yes Yes
Search for parent and children yes Yes
Parallel tests yes Yes
Random data, date, BSN generation Yes ?
Json support Yes ?
Pdf and text Yes
Symbolic links Yes
Support extentions Yes
Windows Yes
Rachid Kherrazi

22. It is backward compatibility: the old style pages can still be edited
Improved version control
New up to date web drivers (e.g. Gecko driver)
Some new features:
Execute program: we can test not-browser related functionalities like UZI pas
Galen framework: automated layout testing for responsive websites
And lot of other features …
Samen aan de slag
What's in it for us?

23. What is Gecko Driver?
Samen aan de slag

24. Why Gecko Driver?
Samen aan de slag
Geckodriver is the next generation of the
Selenium FirefoxDriver
Firefox browser above version 48 can’t be used without
geckodriver. If you try to call the firefox browser from any
language binding, you’d get an error.
From Selenium version 3.0, will have Geckodriver as the default
running implementation for Firefox tests.
Simply we should update. We have no other choices…….
…..if we want to benefit from functionalities of the new versions of Selenium and FireFox

25. Samen aan de slag
Java based forms
Windows e.g.
Batch files
exe
print forms
….
With old version

26. Samen aan de slag
With new
version we
can test also
not browser
related
applications
With new version

27. In following example test script will access and
control:
Browser (Apro log in page)
Batch file (to kill ZorgID process)
Java form (to start ZorgID and log in)
Exe (upload excel file to Apro)
Post message in notepad
Samen aan de slag
Live demo

28. Samen aan de slag
Cookies fixture
clear session storage clears HTML5's sessionStorage (for the domain of the current open page in the browser)
click <clickPlace> until value of <checkPlace> is <expectedValue>
Clicks 'clickPlace' until value of 'checkPlace' is 'expectedValue'. When 'checkPlace' cannot be found, or does not have value 'expectedValue' this method will return 'false'. (Test will be stopped if 'clickPlace' cannot be
found.)
click <clickPlace> until value of <checkPlace> is not
<currentValue>
Clicks 'clickPlace' until value of 'checkPlace' is no longer 'currentValue'. When 'checkPlace' cannot be found, or still has value 'currentValue' this method will return 'false'. (Test will be stopped if 'clickPlace' cannot be
found.)
control click <place>
performs mouse click while holding control key at the designated place (e.g. link/button/checkbox/radio button). If 'send command for control on Mac' is enabled 'Cmd' (i.e. command) key will be pressed instead of 'Ctrl'
(i.e. control).
copy Copies the current selection to the browser's system clipboard.
cut Copies the current selection to the browser's system clipboard and removes that selection.
drag and drop <place> to <target> drags the designated place to the target (does not work for HTML5 drag 'n' drop, because of Selenium issue)
enter date <value> as <place> replaces the current value of an input of type 'date' (i.e. using the HTML5 datepicker) with the supplied value. The value must be in the format yyyy-mm-dd.
execute javascript <JavaScript> until <checkPlace> is
<expectedValue>
Executes 'JavaScript' until value of 'checkPlace' is 'expectedValue', or until the maximum number of retries has been reached.
execute javascript <JavaScript> until <checkPlace> is
<expectedValue>
Executes 'JavaScript' until value of 'checkPlace' is no longer 'expectedValue', or until the maximum number of retries has been reached.
get selection text Gets the text currently selected.
is enabled <place> whether specified place is enabled (i.e. can be clicked) (used in combination with 'ensure' and 'reject')
is visible <place> whether specified place is visible in browser's window (used in combination with 'ensure' and 'reject')
is visible on page <place> whether specified place is visible on the page in the browser's window, scrolling may be required to actually show it (used in combination with 'ensure' and 'reject')
location returns the current web address (i.e. URL of page)
normalized value of <input> retrieves the normalized current value of the input (for use with 'check' verb or variable assignment). Normalized means all sequences of whitespace are collapsed to a single space.
number for <list item text> retrieves the number of the item (in a numbered list) with the supplied text
number of times <text> is visible how often specified text is visible in browser's window
number of times <text> is visible on page how often specified text is visible on the page in the browser's window, scrolling may be required to actually show it
page title retrieves the current page's title (for use with 'check' verb)
paste Pastes the current browser's system clipboard content to the currently active element.
refresh until value of <place> is <expectedValue> Refreshes current page until value of 'place' is 'expectedValue'. When 'place' cannot be found, or does not have value 'expectedValue' this method will return 'false'.
refresh until value of <place> is not <currentValue> Refreshes current page until value of 'place' is no longer 'currentValue'. When 'place' cannot be found, or still has value 'currentValue' this method will return 'false'.
repeat at most <count> times Sets the maximum number of requests sent by 'repeat until ...' commands. The commands will return false if this number is reached without the condition being met.
repeat at most times Returns the maximum number of requests sent by 'repeat until ...' commands.
repeat count Returns the number of requests sent by the last 'repeat until ...' command.
repeat interval Returns the number of milliseconds between requests sent by 'repeat until ...' commands.
right click <place> performs right mouse button click (a.k.a. context click) at the designated place (e.g. link/button/checkbox/radio button)
row exists where <selectColumn> is <selectValue> determines whether there is a row where the column with header 'selectColumn' is 'selectValue'
select all Selects the entire content of the currently active element.
send command for control on Mac Returns whether BrowserTest should press 'Cmd' (i.e. command) key instead of 'Ctrl' (i.e. control) when brower is running on Mac.
set browser height <newHeight> Sets the height of the browser's window to 'newHeight' (in pixels).
set send command for control on Mac to <true/false> Controls whether BrowserTest should press 'Cmd' (i.e. command) key instead of 'Ctrl' (i.e. control) when brower is running on Mac. By default this is false.
shift click <place> performs mouse click while holding shift key at the designated place (e.g. link/button/checkbox/radio button)
switch to default content Activates main/top-level (i)frame (i.e. makes it the current frame).
target of link <link> Retrieves the destination of the link (i.e. the address or URL).
time spent repeating Returns the number of milliseconds the last 'repeat until ...' command took.
tooltip for <place> Retrieves the value of the tooltip that is shown when the mouse is placed on the place (the 'title' attribute)
value of <columnName> in row number <rowIndex> retrieves the value from the column with header 'columnName' in row number 'rowIndex'
value of <columnName> in row where <selectColumn> is
<selectValue>
finds the first row where the column with header 'selectColumn' is 'selectValue' and retrieves the value from the column with header 'columnName'
value of <input> retrieves the current value of the input (for use with 'check' verb or variable assignment)
value of attribute <attribute> on <place> retrieves the value of the given attribute on the element identified by place
value of attribute <attribute> on <place> in <container> retrieves the value of the given attribute on the element identified by place inside the given container
value of column number <columnIndex> in row number
<rowIndex>
retrieves the value from column number 'columnIndex' in row number 'rowIndex'
values of <place> retrieves all values of place, which is expected to be a 'select' or list