6 Traits of a Successful Test Automation Architecture

6 TRAITS OF
A SUCCESSFUL TEST AUTOMATION
ARCHITECTURE
ERDEM YILDIRIM

https://www.linkedin.com/in/yildirim-erdem
About Me
2000 – Software Developer
2006 – Discovered Test Automation &
Decided to be a Test Engineer
2010 – Test Lead at STM (Defence
Technologies and Engineering)
2013 – Test Lead at Innova (12 PRJs, 20 TEs)

 When/What to Automate?
 Design Principles/Patterns
 Locator Strategy
 Methodology
 Framework/Language Selection
 OOP Principles
AGENDA

NEVER ENDING DEBATE
A TALE OF TWO VALUES
AUTOMATED TEST
MANUAL TEST

AUTOMATION
Expected Action
Machines are great for
running straigtforward
tests
Gives Confidence
EXPLORATORY
Unexpected Scenario
Humans are great for tests
requiring intelligence and
experience
Finds new defects
COMPLEMANTARY

WHAT IS YOUR AUTOMATION
ARCHITECTURE SCREAMING

PREVENT/SOLVE AUTOMATION ISSUES
HIGH MAINTENANCE COST
LACK OF TEST AUTOMATION EXPERIENCE
STALLED AUTOMATION
UNREALISTIC EXPECTATIONS

BEING MULTIPLIER
increase the productivity & efficiency
of the automation project and team.

st
trait
Which Test Levels To
Be Automated

Automation Levels
Scenario Depth
UI
Tests
Integration
Tests
Unit Tests
Test Data Preparation
CI–ContinuousIntegration
Horizontal E2E (UI & Integration Layer) Login / Add to Chart / Checkout / Pay
Vertical E2E
Implementing
tests on all level
of the pyramid
Deciding
Exploratory
Test

UI
Tests
Integration
Tests
Unit Tests
Test
Type
Low
Medium
High
Business Logic
Coverage
Code
Coverage
Low
Medium
High
Execution
Time/Costs
Short/
Low
Medium
Long / High

SUSTAINABLE
automation solutions

TESTER
CODER
DESIGNER
is a
also a

Design Principles
Modular
Reusable
Flexible
Extensible
Loosely Coupled
Readable
Manintainable (Easily)
Principle of Good Enough
Seperation of Concerns
High Cohesion
KISS
YAGNI
DRY

KISS
Keep It Simple Stupid
YAGNI
You Aren’t Going to Need It
DRY
Don’t Repeat Yourself

Layers in Your Automation Code
Tests
Framework
Selenium/GEB/etc.
Browser (Web App)

How to Design a Framework
User Login
- LoginLink.Click
- UserNameField.EnterText."erdem"
- PswdField.EnterText."yildirim"
- RememberPswd.Click
- LoginButton.Click
- Confirm Login
Send Email
- LoginLink.Click
- UserNameField.EnterText."erdem"
- PswdField.EnterText."erdem"
- LoginButton.Click
- Confirm Login
- Click compose
- type msg
- type subj
- send
- confirm send
- logout
Linear Framework
userLogin ()
sendEmail ()
Function userLogin(userName, userPassword){
- LoginLink.Click
- UserNameField.EnterText.userName
- PswdField.EnterText.userPassword
- LoginButton.Click
- Confirm Login
}
Function sendEmail(userName, userPassword){
- LoginLink.Click
- UserNameField.EnterText."karthik"
- PswdField.EnterText."karthik"
- LoginButton.Click
- Confirm Login
- Click compose
- type msg
- type subj
- send
- confirm send
- logout
}
Functional Decomposition Framework
+Data Driven

What is a Design Pattern?
Reusable solution to a commonly occurring problem within a
given context.

Spaghetti Pattern
No Pattern (a.k.a) Antipattern

Advantages of Spaghetti Pattern
• Quick start
• Smaller code base
• Smoke tests

Disadvantages of Spaghetti Pattern
• Anti-pattern
• Tight Coupling
• No Random Order
• No parallel test runs
• Covers up failures
• Difficult to extend or reuse; INEFFICIENT

Hermetic Test Pattern
Polar opposite of the Spaghetti pattern.
It states that each test should be completely independent and
self-sufficient.
Advantages Disadvantages
• Clean Start
• Resilience
• Modular
• Random Run Order
• Parallel Testing
• Upfront Design
• Runtime Increase
• Resource Usage Increase

Factory Design Pattern
interface Dog
{
public void speak ();
}
class Poodle implements Dog
{
public void speak()
{
System.out.println("The poodle says "arf"");
}
}
class Rottweiler implements Dog
{
public void speak()
{
System.out.println("The Rottweiler says (in a very deep voice) "WOOF!"");
}
}
class SiberianHusky implements Dog
{
public void speak()
{
System.out.println("The husky says "Dude, what's up?"");
}
class DogFactory
{
public static Dog getDog(String criteria)
{
if ( criteria.equals("small") )
return new Poodle();
else if ( criteria.equals("big") )
return new Rottweiler();
else if ( criteria.equals("working") )
return new SiberianHusky();
return null;
}
}
1
3
2
Create a Dog Interface Define Concrete Dog Classes
Define Java Factory Class

Factory Design Pattern
public class JavaFactoryPatternExample
{
public static void main(String[] args)
{
// create a small dog
Dog dog = DogFactory.getDog("small");
dog.speak();
// create a big dog
dog = DogFactory.getDog("big");
dog.speak();
// create a working dog
dog = DogFactory.getDog("working");
dog.speak();
}
}
4
Factory Pattern Test Program
The poodle says "arf"
The Rottweiler says (in a very deep voice) "WOOF!"
The husky says "Dude, what's up?"
Brings
Decoupling
Encapsulation

Page Object Model (POM)
Helps to hide the page implementation; the test is no longer allowed to directly
interact with a given page, but instead uses a framework of classes and methods to
accomplish the same goal. This pattern abstracts the business logic/implementation
details.
Advantages Disadvantages
• Busines Point of View
• Testing Behavior
• DRY
• Modular & Reusable
• Clear Intentions
• Complexity (Wrong Usage)

Page - Spec
public class LoginSpec extends PageSpec {
def "Login tests with user information"() {
given: "Home Page"
def loginPage = to LoginPage
when: "user information is entered"
loginPage.loginOl(userName, password)
then: "login to the system, open main page"
at resultPage
where:
userName | password | resultPage | aciklama
"buro1@innova.com.tr" | "123456" | MemberHomePage | "positive-test"
"wrong@mail.com" | "123" | LoginPage | "negative-test"
"" | "" | LoginPage | "exceptional-test"
}
}
class LoginPage extends BasePage {
static url = "giris"
static at = { waitFor { title == "Innova Legalite User Login" } }
def getUser() { $("input", name: "username") }
def getPass() { $("input", name: "password") }
def getEnter() { $("button", 0) }
def loginOl(String username, String password) {
user.value(username)
pass.value(password)
enter.click()
}
}

Layers in Your Framework Design
The Framework should also be divided up into layers of functionality.
Tests
Framework
Selenium/GEB/etc.
Browser (Web App)
Workflows
Pages
Navigation Modules
Selenium/GEB/etc.

PageFactory Pattern
The PageFactory Class in Selenium is an extension to the Page Object
design pattern. It is used to initialize the elements of the Page Object
or instantiate the Page Objects itself.
It is used to initialize elements of a Page class without having to use
‘FindElement’ or ‘FindElements’. Annotations can be used to supply
descriptive names of target objects to improve code readability.
In fact, we can use the page object pattern without using the Page
Factory class.

PageFactory Pattern
There are different opinions in sector.
• Even the creator of PageFactory doesn’t encourage to use PageFactory.
• Jim Holmes: "It's simpler and cleaner for people to stand up their own
page objects. PageFactory is a needless complication"
• PageFactory is useful; It handles WebElements nicely and helps test
codes/scenarios be easier to read.
• The only difference I can see is that it makes tracing through code a
little harder for people unfamiliar to Selenium and saves a few
characters per instruction.

Setup And Teardown Method
Setup
Exercise
Verify
Teardown
SUT
Install

Prod Like Testing Stage
Simulate a test stage as if every integrated system/comp is up
before they are really up
And leave the stage at starting point
Setup
Exercise
Verify
Teardown
SUT
Install
Create
Return
ValuesIndirect
Inputs
Record
Mock
Object

Simlicity is the ultimate sophistication
Leanardo da Vinci

LOCATORS
Your locator strategy will determine your tests’ destiny as flaky or
robust.
Good: If you know the right strategy you can apply it.
Bad: It is also dependent to AUT code/developers not only to you.
Critical: Communication & Collaboration, mentoring dev teams.

LOCATORS
unique
descriptive
unlikely to change

LOCATORSCSS selectors
ID
Xpath
Custom Locators
Class
Name
Link Text
Partial Link Text
Tag Name
Dynamic Locators

LOCATORS
in order
ID
Custom Locators
Class
Name
CSS selectors
Xpath
Link Text
Partial Link Text
Tag Name
Dynamic Locators

E2E Test Automation
High business logic coverage,
Testing from the user perspective,
Prevents production incidents,
Gives the confidence that everything is OK
Low code coverage,
Slow execution,

Continuous Testing
To develop software within shorter delivery cycles; Agile, DevOps
and Continuous Delivery approaches are on the forefront.
Continuous Delivery supports Continuous Testing which is a
testing strategy that consists of a large number of automated unit
and acceptance tests but a small number of automated end-to-
end tests.
Continuous Testing claims that E2E tests are inefficient and they
should be limited.
It is a means of using Vertical E2E Test Automation effectively but
avoiding Horizontal E2E Test Automation

TDD – Test Driven Development

TDD – Test Driven Development
RULES
Write only enough of a unit test to fail.
Write only enough production code to make the failing unit test pass.
WHY TDD
Simplest way to achieve both good quality code and good test coverage
.

BDD – Behaviour Driven Development
Developing unambiguous features in collaboration with BA, QA, Dev
Writing features’ definition and also the behaviours driving the design
process.
Preventing defects rather than finding defects at early stages.

BDD – Behaviour Driven Development
Communication / Collaboration
Automation
Living Documentation
Drives whole project team
Preventative Tests
Healthy test basis / requirements
Clear understanding of what a feature means
Replaces scripted test with automation scripts

BDD Styles
Cucumber
Readable/executable scenarios by every role in project
Spock
Fastest coding and most concise, readable code

Spock BDD
public class LoginSpec extends PageSpec {
def "Login tests with user information"() {
given: "Home Page"
def loginPage = to LoginPage
when: "user information is entered"
loginPage.loginOl(userName, password)
then: "login to the system, open main page"
at resultPage
where:
userName | password | resultPage | aciklama
"buro1@innova.com.tr" | "123456" | MemberHomePage | "positive-test"
"wrong@mail.com" | "123" | LoginPage | "negative-test"
"" | "" | LoginPage | "exceptional-test"
}
}
All (BDD, automation code) in one file;
very concise/readable code,
very fast and fun to code

Cucumber BDD
Two seperate files; one for Scenarios, one for code
readable for non-developers

xDD: TDD vs BDD
BDD focuses on the behavioural aspect of the system rather
than the implementation aspect of the system that TDD
focuses on.
TDD captures low-level requirements (usually at the
class/method level). BDD captures high-level requirements.
Therefore, BDD is still TDD - but at a higher level

Develop “Software that matters”
along with “Software that works”
Enter the User Stories (feature + scenarios) in a BDD tool,
Write E2E automation code to map the User Stories to tests,
Write production code using TDD to make the tests pass.

Dev Team
Version
Control
Build & Unit
Tests
UI Test
Automation
(Smoke)
Deploy (Prod)
UI Test
Automation
(Smoke)
UI Test
Automation
(Full Set)
Monitoring
Dev CI Pre-Production Production
Check in
Trigger
Trigger
Approval
Approval
Approval
Approval
Feedback
Feedback
Feedback
Feedback
Feedback
Automated Test Runs
After Nightly Build [1]
• Unit Test
Automation
• Web Service Test
Automation
• UI Test Automation
(Full Set)
• Cross Browser &
Platform Tests
Automated Test Runs
After Weekly Build [2]
• Automated Non
Functional Tests
(Performance &
Load Tests)
• Static Code Quality
Analysis
Feedback
Feedback
Feedback

th
trait
TOOL
FRAMEWORK
LANGUAGE
SELECTION

NO SUCH TOOL
The Illusion of Using one Test Automation Tool for All Projects

BEST TOOL
Depends on the project dynamics

CONSIDER
Technologies used in SUT development code,
3rd party components
LOOK FOR
Efficiency
(produce more with less effort)

Just a Few Examples
on Personel Experience
GEB, Spock, Groovy
the power of Selenium WebDriver,
the elegance of jQuery content selection,
the robustness of Page Object modelling,
the expressiveness of the Groovy language,
the velocity of Spock style BDD
What bring us?

CODECEPTJS
Backend agnostic; the power of Selenium WebDriver,
webdriverio, protractor, nightmarejs, appium, puppiteer
Facilitated one language for all above frameworks,
Scenario driven highly readable,
Built in POM support,
Support for both Spock style BDD and Cucumber BDD
Fast to develop and learn.

CUCUMBER
Readable BDD for whole stakeholders,
Ideal for automating backend, middleware applications.
Increase the communication / collaboration in team.
Resolve the issue of developing the wrong/misunderstood
requirement.

4 PILLARS OF OOP
Encapsulation
Inheritance
Polymorphism
Abstraction

ENCAPSULATION
Hiding data and binding that data to public methods that other classes can utilize to
access to data to protect the data integrity and consistency. Accessing data from the
getter/setter methods.
class Account
{
private double balance;
public setBalance(double balance)
{
if(balance > 0) // Validating input data in order to maintain data integrity
this.balance = balance;
}
}
public static void main()
{
Account theGeekyAsian = new Account(); //Creating an object of class Account
theGeekyAsian.setBalance(1000); //Balance set successful
}

INHERITANCE
Extend data and behaviours. Inheritance is a way to reuse code. The class which is
inherited from, is called the base class, and the class which inherits the code from the
base class is called a derived class. A derived class can use all the functions which are
defined in the base class, making the code reusable.
class Shape
{
private int height;
private int width;
public void setWidth(int width)
{
this.width = width;
}
public void setHeight(int height)
{
this.height = height;
}
}
class Rectangle extends Shape
{
public int getArea()
{
return height * width; //Rectangle's getArea method inherited the height and width from Shape
}
}

POLYMORPHISM
The ability to take multiple forms. The name polymorphism defines itself. Poly means
many, and morph means forms. It is a process of a method of a class taking
different/many forms.
Static Polymorphism – Known as Method Overloading
Dynamic Polymorphism – Known as Method Overriding
class Numbers
{
public int sum(int numberA, int numberB)
{
return numberA + numberB;
}
public double sum(double numberA, double numberB) //Here, method overloading takes place,
{ //by providing another feature to the same method 'sum'
return numberA + numberB;
}
}
class Test
{
public static void main(String[] args)
{
Numbers objNumber = new Numbers();
System.out.println("Sum of two integer numbers is: " + objNumber.sum(2,3);
System.out.println("Sum of two double numbers is: " + objNumber.sum(2.5,3.2);
}
}

ABSTRACTION
In abstraction, we provide only essential/necessary features (methods) of an entity to
the outside world (the user) while hiding other irrelevant details, in order to reduce
operational complexity at the user-end.
class Car
{
public void changeGear(){}
public void steerCar(){}
public void accelerateCar() //Accelerator is accessible by the user to accelerate the car
{
/* Something happens here */
what_happens_when_he_accelerates_the_car(someValue);
}
private void exhaustSystem(){}
private void what_happens_when_he_accelerates_the_car(String someValue)
{
/* Something happens here */
}
private void silencer(){}
}

Team Building
Roles – Intersection of Interests and Works
Developer in Test
Test Automation Engineer
Test Analyst
Career Planning
Give Venue to Team for Sharing Knowledge
Engaging – Motivating Environment
Communication - Collaboration

Enjoy the brainstorming
sessions.

BE A MULTIPLIER
increase the productivity & efficiency
of the automation project and team.
don’t forget to

ReferencesTest Automation Patterns: Issues and Solutions, Dorothy Graham, Seretta Gamba
V. Garousi and M. V. Mäntylä, "When and what to automate in software testing? A multi-vocal literature review," Information and Software Technology, vol. 76, pp. 92-117, 2016.
Testistanbul 2016 - Keynote: "Why Automated Verification Matters" by Kristian Karl
Effective Software Testing: 50 Ways to Improve Your Software Testing, Addison-Wesley Longman Publishing Co., Inc. Boston, MA, USA ©2002
End-to-End Integration Testing Design, Computer Software and Applications Conference, 2001. COMPSAC 2001. 25th Annual International
Selenium Design Patterns and Best Practices, Dima Kovalenko
Behavior Driven Development Webapps mit Groovy Spock und Geb
https://blog.testlodge.com/tdd-vs-bdd/
https://www.itexico.com/blog/software-development-kiss-yagni-dry-3-principles-to-simplify-your-life Tight
Khanh Le Common Design Principles and Design Patterns in Automation Testing
https://medium.freecodecamp.org/test-driven-development-what-it-is-and-what-it-is-not-41fa6bca02a2
https://blog.jbrains.ca/permalink/the-worlds-shortest-article-on-behavior-driven-development-revisited
https://blog.jbrains.ca/permalink/how-test-driven-development-works-and-more
Selenium Design Patterns, Liraz Shay
http://thegeekyasian.com/4-pillars-of-oop/
https://www.it.uu.se/edu/course/homepage/ood/ht12/overview/patterns/Lecture6.pdf
https://alvinalexander.com/java/java-factory-pattern-example
https://mentormate.com/blog/bbd-using-web-accessibility-rules-specflow-net-cucumber-examples/
http://www.diogonunes.com/blog/four-testing-quadrants/
https://www.youtube.com/watch?v=Jpjfm7s572Y
Applying the Pillars of Object-Oriented Programming to Test Automation - with Angie Jones
https://www.youtube.com/watch?v=Jpjfm7s572Y
Creating A Test Automation Framework Architecture With Selenium (Step-By-Step), John Sonmez, Bulldog Mindset.
Common design principles and design patterns in automation testing, KMSTechnolog

Image References
https://tenor.com/view/peepshow-interesting-story-gif-4852643
Andreas von der Heydt 12 Minutes To Create A Mind Changing Presentation
Damon Nofar 5 Ways To Surprise Your Audience (and keep their attention)
Slidemodel.com,
pexels.com
(Re)Building an engineering culture: DevOps at Target, Heather Mickman, Ross Clanton

Thank You
Erdem YILDIRIM
https://www.linkedin.com/in/yildirim-erdem

6 Traits of a Successful Test Automation Architecture

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to 6 Traits of a Successful Test Automation Architecture

Similar to 6 Traits of a Successful Test Automation Architecture (20)

Recently uploaded

Recently uploaded (20)

6 Traits of a Successful Test Automation Architecture