This document discusses 6 traits of a successful test automation architecture: 1. Deciding which test levels to automate by considering factors like efficiency, expected vs unexpected scenarios, and intelligence vs repetitiveness. 2. Design principles for test automation including modularity, reusability, and separation of concerns. 3. Locator strategy which determines whether tests are flaky or robust, prioritizing unique, descriptive locators that are unlikely to change. 4. Methodology such as behavior driven development, test driven development, and continuous testing approaches. 5. Framework and language selection considering project dynamics and technologies. Examples mentioned are Geb, Spock, Groovy, and CodeceptJS. 6.

1. 6 TRAITS OF
A SUCCESSFUL TEST AUTOMATION
ARCHITECTURE
ERDEM YILDIRIM

2. https://www.linkedin.com/in/yildirim-erdem
About Me
2000 – Software Developer
2006 – Discovered Test Automation &
Decided to be a Test Engineer
2010 – Test Lead at STM (Defence
Technologies and Engineering)
2013 – Test Lead at Innova (12 PRJs, 20 TEs)

4.  When/What to Automate?
 Design Principles/Patterns
 Locator Strategy
 Methodology
 Framework/Language Selection
 OOP Principles
AGENDA

6. What is The Point?
EFFICIENCY

7. NEVER ENDING DEBATE
A TALE OF TWO VALUES
AUTOMATED TEST
MANUAL TEST

8. EXPECTED

9. UNEXPECTED

10. REPETITIVE

11. INTELLIGENCE

12. AUTOMATION
Expected Action
Machines are great for
running straigtforward
tests
Gives Confidence
EXPLORATORY
Unexpected Scenario
Humans are great for tests
requiring intelligence and
experience
Finds new defects
COMPLEMANTARY

13. ARCHITECTURE

14. WHAT IS YOUR AUTOMATION
ARCHITECTURE SCREAMING

15. TEST AUTOMATION ARCHITECT

16. PREVENT/SOLVE AUTOMATION ISSUES
HIGH MAINTENANCE COST
LACK OF TEST AUTOMATION EXPERIENCE
STALLED AUTOMATION
UNREALISTIC EXPECTATIONS

17. BEING MULTIPLIER
increase the productivity & efficiency
of the automation project and team.

18. st
trait
Which Test Levels To
Be Automated

19. Automation Levels
Scenario Depth
UI
Tests
Integration
Tests
Unit Tests
Test Data Preparation
CI–ContinuousIntegration
Horizontal E2E (UI & Integration Layer) Login / Add to Chart / Checkout / Pay
Vertical E2E
Implementing
tests on all level
of the pyramid
Deciding
Exploratory
Test

21. UI
Tests
Integration
Tests
Unit Tests
Test
Type
Low
Medium
High
Business Logic
Coverage
Code
Coverage
Low
Medium
High
Execution
Time/Costs
Short/
Low
Medium
Long / High

23. SUSTAINABLE
automation solutions

24. nd
trait
DESIGN

25. TESTER
CODER
DESIGNER
is a
also a

26. Design Principles
Modular
Reusable
Flexible
Extensible
Loosely Coupled
Readable
Manintainable (Easily)
Principle of Good Enough
Seperation of Concerns
High Cohesion
KISS
YAGNI
DRY

27. KISS
Keep It Simple Stupid
YAGNI
You Aren’t Going to Need It
DRY
Don’t Repeat Yourself

28. #Code Example @before DRY

29. #Code Example @after DRY

30. Layers in Your Automation Code
Tests
Framework
Selenium/GEB/etc.
Browser (Web App)

31. How to Design a Framework
User Login
- LoginLink.Click
- UserNameField.EnterText."erdem"
- PswdField.EnterText."yildirim"
- RememberPswd.Click
- LoginButton.Click
- Confirm Login
Send Email
- LoginLink.Click
- UserNameField.EnterText."erdem"
- PswdField.EnterText."erdem"
- RememberPswd.Click
- LoginButton.Click
- Confirm Login
- Click compose
- type msg
- type subj
- send
- confirm send
- logout
Linear Framework
userLogin ()
sendEmail ()
Function userLogin(userName, userPassword){
- LoginLink.Click
- UserNameField.EnterText.userName
- PswdField.EnterText.userPassword
- RememberPswd.Click
- LoginButton.Click
- Confirm Login
}
Function sendEmail(userName, userPassword){
- LoginLink.Click
- UserNameField.EnterText."karthik"
- PswdField.EnterText."karthik"
- RememberPswd.Click
- LoginButton.Click
- Confirm Login
- Click compose
- type msg
- type subj
- send
- confirm send
- logout
}
Functional Decomposition Framework
+Data Driven

32. What is a Design Pattern?
Reusable solution to a commonly occurring problem within a
given context.

33. Spaghetti Pattern
No Pattern (a.k.a) Antipattern

34. Advantages of Spaghetti Pattern
• Quick start
• Smaller code base
• Smoke tests

35. Disadvantages of Spaghetti Pattern
• Anti-pattern
• Tight Coupling
• No Random Order
• No parallel test runs
• Covers up failures
• Difficult to extend or reuse; INEFFICIENT

36. Hermetic Test Pattern
Polar opposite of the Spaghetti pattern.
It states that each test should be completely independent and
self-sufficient.
Advantages Disadvantages
• Clean Start
• Resilience
• Modular
• Random Run Order
• Parallel Testing
• Upfront Design
• Runtime Increase
• Resource Usage Increase

37. Factory Design Pattern
interface Dog
{
public void speak ();
}
class Poodle implements Dog
{
public void speak()
{
System.out.println("The poodle says "arf"");
}
}
class Rottweiler implements Dog
{
public void speak()
{
System.out.println("The Rottweiler says (in a very deep voice) "WOOF!"");
}
}
class SiberianHusky implements Dog
{
public void speak()
{
System.out.println("The husky says "Dude, what's up?"");
}
class DogFactory
{
public static Dog getDog(String criteria)
{
if ( criteria.equals("small") )
return new Poodle();
else if ( criteria.equals("big") )
return new Rottweiler();
else if ( criteria.equals("working") )
return new SiberianHusky();
return null;
}
}
1
3
2
Create a Dog Interface Define Concrete Dog Classes
Define Java Factory Class

38. Factory Design Pattern
public class JavaFactoryPatternExample
{
public static void main(String[] args)
{
// create a small dog
Dog dog = DogFactory.getDog("small");
dog.speak();
// create a big dog
dog = DogFactory.getDog("big");
dog.speak();
// create a working dog
dog = DogFactory.getDog("working");
dog.speak();
}
}
4
Factory Pattern Test Program
The poodle says "arf"
The Rottweiler says (in a very deep voice) "WOOF!"
The husky says "Dude, what's up?"
Brings
Decoupling
Encapsulation

39. Page Object Model (POM)
Helps to hide the page implementation; the test is no longer allowed to directly
interact with a given page, but instead uses a framework of classes and methods to
accomplish the same goal. This pattern abstracts the business logic/implementation
details.
Advantages Disadvantages
• Busines Point of View
• Testing Behavior
• DRY
• Modular & Reusable
• Clear Intentions
• Complexity (Wrong Usage)

40. Page - Spec
public class LoginSpec extends PageSpec {
def "Login tests with user information"() {
given: "Home Page"
def loginPage = to LoginPage
when: "user information is entered"
loginPage.loginOl(userName, password)
then: "login to the system, open main page"
at resultPage
where:
userName | password | resultPage | aciklama
"buro1@innova.com.tr" | "123456" | MemberHomePage | "positive-test"
"wrong@mail.com" | "123" | LoginPage | "negative-test"
"" | "" | LoginPage | "exceptional-test"
}
}
class LoginPage extends BasePage {
static url = "giris"
static at = { waitFor { title == "Innova Legalite User Login" } }
def getUser() { $("input", name: "username") }
def getPass() { $("input", name: "password") }
def getEnter() { $("button", 0) }
def loginOl(String username, String password) {
user.value(username)
pass.value(password)
enter.click()
}
}

41. Layers in Your Framework Design
The Framework should also be divided up into layers of functionality.
Tests
Framework
Selenium/GEB/etc.
Browser (Web App)
Workflows
Pages
Navigation Modules
Selenium/GEB/etc.

42. PageFactory Pattern
The PageFactory Class in Selenium is an extension to the Page Object
design pattern. It is used to initialize the elements of the Page Object
or instantiate the Page Objects itself.
It is used to initialize elements of a Page class without having to use
‘FindElement’ or ‘FindElements’. Annotations can be used to supply
descriptive names of target objects to improve code readability.
In fact, we can use the page object pattern without using the Page
Factory class.

43. PageFactory Pattern
There are different opinions in sector.
• Even the creator of PageFactory doesn’t encourage to use PageFactory.
• Jim Holmes: "It's simpler and cleaner for people to stand up their own
page objects. PageFactory is a needless complication"
• PageFactory is useful; It handles WebElements nicely and helps test
codes/scenarios be easier to read.
• The only difference I can see is that it makes tracing through code a
little harder for people unfamiliar to Selenium and saves a few
characters per instruction.

44. “BEST” PATTERN TO USE ?

45. Setup And Teardown Method
Setup
Exercise
Verify
Teardown
SUT
Install

46. Prod Like Testing Stage
Simulate a test stage as if every integrated system/comp is up
before they are really up
And leave the stage at starting point
Setup
Exercise
Verify
Teardown
SUT
Install
Create
Return
ValuesIndirect
Inputs
Record
Mock
Object

47. Simlicity is the ultimate sophistication
Leanardo da Vinci

48. rd
trait
LOCATOR
STRATEGY

49. LOCATORS
Your locator strategy will determine your tests’ destiny as flaky or
robust.
Good: If you know the right strategy you can apply it.
Bad: It is also dependent to AUT code/developers not only to you.
Critical: Communication & Collaboration, mentoring dev teams.

50. LOCATORS
unique
descriptive
unlikely to change

51. LOCATORSCSS selectors
ID
Xpath
Custom Locators
Class
Name
Link Text
Partial Link Text
Tag Name
Dynamic Locators

52. LOCATORS
in order
ID
Custom Locators
Class
Name
CSS selectors
Xpath
Link Text
Partial Link Text
Tag Name
Dynamic Locators

53. th
trait
Methodology

54. E2E Test Automation
High business logic coverage,
Testing from the user perspective,
Prevents production incidents,
Gives the confidence that everything is OK
Low code coverage,
Slow execution,

55. Continuous Testing
To develop software within shorter delivery cycles; Agile, DevOps
and Continuous Delivery approaches are on the forefront.
Continuous Delivery supports Continuous Testing which is a
testing strategy that consists of a large number of automated unit
and acceptance tests but a small number of automated end-to-
end tests.
Continuous Testing claims that E2E tests are inefficient and they
should be limited.
It is a means of using Vertical E2E Test Automation effectively but
avoiding Horizontal E2E Test Automation

56. TDD – Test Driven Development

57. TDD – Test Driven Development
RULES
Write only enough of a unit test to fail.
Write only enough production code to make the failing unit test pass.
WHY TDD
Simplest way to achieve both good quality code and good test coverage
.

58. BDD – Behaviour Driven Development
Developing unambiguous features in collaboration with BA, QA, Dev
Writing features’ definition and also the behaviours driving the design
process.
Preventing defects rather than finding defects at early stages.

59. BDD – Behaviour Driven Development
Communication / Collaboration
Automation
Living Documentation
Drives whole project team
Preventative Tests
Healthy test basis / requirements
Clear understanding of what a feature means
Replaces scripted test with automation scripts

60. BDD Styles
Cucumber
Readable/executable scenarios by every role in project
Spock
Fastest coding and most concise, readable code

61. Spock BDD
public class LoginSpec extends PageSpec {
def "Login tests with user information"() {
given: "Home Page"
def loginPage = to LoginPage
when: "user information is entered"
loginPage.loginOl(userName, password)
then: "login to the system, open main page"
at resultPage
where:
userName | password | resultPage | aciklama
"buro1@innova.com.tr" | "123456" | MemberHomePage | "positive-test"
"wrong@mail.com" | "123" | LoginPage | "negative-test"
"" | "" | LoginPage | "exceptional-test"
}
}
All (BDD, automation code) in one file;
very concise/readable code,
very fast and fun to code

62. Cucumber BDD
Two seperate files; one for Scenarios, one for code
readable for non-developers

63. xDD: TDD vs BDD
BDD focuses on the behavioural aspect of the system rather
than the implementation aspect of the system that TDD
focuses on.
TDD captures low-level requirements (usually at the
class/method level). BDD captures high-level requirements.
Therefore, BDD is still TDD - but at a higher level

64. Develop “Software that matters”
along with “Software that works”
Enter the User Stories (feature + scenarios) in a BDD tool,
Write E2E automation code to map the User Stories to tests,
Write production code using TDD to make the tests pass.

65. Dev Team
Version
Control
Build & Unit
Tests
UI Test
Automation
(Smoke)
Deploy (Prod)
UI Test
Automation
(Smoke)
UI Test
Automation
(Full Set)
Monitoring
Dev CI Pre-Production Production
Check in
Trigger
Trigger
Approval
Approval
Approval
Approval
Feedback
Feedback
Feedback
Feedback
Feedback
Automated Test Runs
After Nightly Build [1]
• Unit Test
Automation
• Web Service Test
Automation
• UI Test Automation
(Full Set)
• Cross Browser &
Platform Tests
Automated Test Runs
After Weekly Build [2]
• Automated Non
Functional Tests
(Performance &
Load Tests)
• Static Code Quality
Analysis
Feedback
Feedback
Feedback

66. th
trait
TOOL
FRAMEWORK
LANGUAGE
SELECTION

67. NO SUCH TOOL
The Illusion of Using one Test Automation Tool for All Projects

68. BEST TOOL
Depends on the project dynamics

69. CONSIDER
Technologies used in SUT development code,
3rd party components
LOOK FOR
Efficiency
(produce more with less effort)

70. Just a Few Examples
on Personel Experience
GEB, Spock, Groovy
the power of Selenium WebDriver,
the elegance of jQuery content selection,
the robustness of Page Object modelling,
the expressiveness of the Groovy language,
the velocity of Spock style BDD
What bring us?

71. CODECEPTJS
Backend agnostic; the power of Selenium WebDriver,
webdriverio, protractor, nightmarejs, appium, puppiteer
Facilitated one language for all above frameworks,
Scenario driven highly readable,
Built in POM support,
Support for both Spock style BDD and Cucumber BDD
Fast to develop and learn.

72. CUCUMBER
Readable BDD for whole stakeholders,
Ideal for automating backend, middleware applications.
Increase the communication / collaboration in team.
Resolve the issue of developing the wrong/misunderstood
requirement.

73. th
trait
OOP
Principles

74. 4 PILLARS OF OOP
Encapsulation
Inheritance
Polymorphism
Abstraction

75. ENCAPSULATION
Hiding data and binding that data to public methods that other classes can utilize to
access to data to protect the data integrity and consistency. Accessing data from the
getter/setter methods.
class Account
{
private double balance;
public setBalance(double balance)
{
if(balance > 0) // Validating input data in order to maintain data integrity
this.balance = balance;
}
}
public static void main()
{
Account theGeekyAsian = new Account(); //Creating an object of class Account
theGeekyAsian.setBalance(1000); //Balance set successful
}

76. INHERITANCE
Extend data and behaviours. Inheritance is a way to reuse code. The class which is
inherited from, is called the base class, and the class which inherits the code from the
base class is called a derived class. A derived class can use all the functions which are
defined in the base class, making the code reusable.
class Shape
{
private int height;
private int width;
public void setWidth(int width)
{
this.width = width;
}
public void setHeight(int height)
{
this.height = height;
}
}
class Rectangle extends Shape
{
public int getArea()
{
return height * width; //Rectangle's getArea method inherited the height and width from Shape
}
}

77. POLYMORPHISM
The ability to take multiple forms. The name polymorphism defines itself. Poly means
many, and morph means forms. It is a process of a method of a class taking
different/many forms.
Static Polymorphism – Known as Method Overloading
Dynamic Polymorphism – Known as Method Overriding
class Numbers
{
public int sum(int numberA, int numberB)
{
return numberA + numberB;
}
public double sum(double numberA, double numberB) //Here, method overloading takes place,
{ //by providing another feature to the same method 'sum'
return numberA + numberB;
}
}
class Test
{
public static void main(String[] args)
{
Numbers objNumber = new Numbers();
System.out.println("Sum of two integer numbers is: " + objNumber.sum(2,3);
System.out.println("Sum of two double numbers is: " + objNumber.sum(2.5,3.2);
}
}

78. ABSTRACTION
In abstraction, we provide only essential/necessary features (methods) of an entity to
the outside world (the user) while hiding other irrelevant details, in order to reduce
operational complexity at the user-end.
class Car
{
public void changeGear(){}
public void steerCar(){}
public void accelerateCar() //Accelerator is accessible by the user to accelerate the car
{
/* Something happens here */
what_happens_when_he_accelerates_the_car(someValue);
}
private void exhaustSystem(){}
private void what_happens_when_he_accelerates_the_car(String someValue)
{
/* Something happens here */
}
private void silencer(){}
}

79. PEOPLE

80. Team Building
Roles – Intersection of Interests and Works
Developer in Test
Test Automation Engineer
Test Analyst
Career Planning
Give Venue to Team for Sharing Knowledge
Engaging – Motivating Environment
Communication - Collaboration

81. Give Your Team a Venue

82. Enjoy the brainstorming
sessions.

83. Be open to
new ideas / techs.

84. BE A MULTIPLIER
increase the productivity & efficiency
of the automation project and team.
don’t forget to

85. ReferencesTest Automation Patterns: Issues and Solutions, Dorothy Graham, Seretta Gamba
V. Garousi and M. V. Mäntylä, "When and what to automate in software testing? A multi-vocal literature review," Information and Software Technology, vol. 76, pp. 92-117, 2016.
Testistanbul 2016 - Keynote: "Why Automated Verification Matters" by Kristian Karl
Effective Software Testing: 50 Ways to Improve Your Software Testing, Addison-Wesley Longman Publishing Co., Inc. Boston, MA, USA ©2002
End-to-End Integration Testing Design, Computer Software and Applications Conference, 2001. COMPSAC 2001. 25th Annual International
Selenium Design Patterns and Best Practices, Dima Kovalenko
Behavior Driven Development Webapps mit Groovy Spock und Geb
https://blog.testlodge.com/tdd-vs-bdd/
https://www.itexico.com/blog/software-development-kiss-yagni-dry-3-principles-to-simplify-your-life Tight
Khanh Le Common Design Principles and Design Patterns in Automation Testing
https://medium.freecodecamp.org/test-driven-development-what-it-is-and-what-it-is-not-41fa6bca02a2
https://blog.jbrains.ca/permalink/the-worlds-shortest-article-on-behavior-driven-development-revisited
https://blog.jbrains.ca/permalink/how-test-driven-development-works-and-more
Selenium Design Patterns, Liraz Shay
http://thegeekyasian.com/4-pillars-of-oop/
https://www.it.uu.se/edu/course/homepage/ood/ht12/overview/patterns/Lecture6.pdf
https://alvinalexander.com/java/java-factory-pattern-example
https://mentormate.com/blog/bbd-using-web-accessibility-rules-specflow-net-cucumber-examples/
http://www.diogonunes.com/blog/four-testing-quadrants/
https://www.youtube.com/watch?v=Jpjfm7s572Y
Applying the Pillars of Object-Oriented Programming to Test Automation - with Angie Jones
https://www.youtube.com/watch?v=Jpjfm7s572Y
Creating A Test Automation Framework Architecture With Selenium (Step-By-Step), John Sonmez, Bulldog Mindset.
Common design principles and design patterns in automation testing, KMSTechnolog

86. Image References
https://tenor.com/view/peepshow-interesting-story-gif-4852643
Andreas von der Heydt 12 Minutes To Create A Mind Changing Presentation
Damon Nofar 5 Ways To Surprise Your Audience (and keep their attention)
Slidemodel.com,
pexels.com
(Re)Building an engineering culture: DevOps at Target, Heather Mickman, Ross Clanton

87. Thank You
Erdem YILDIRIM
https://www.linkedin.com/in/yildirim-erdem