The document provides a detailed guide on building a test automation framework, emphasizing requirements like minimal cost of ownership, integration with existing tools, and transparency. It discusses the distinction between frameworks and libraries, outlines essential components such as test libraries, test cases, and assertion strategies, and introduces various technologies involved like JUnit 5 and Cucumber. Additionally, it highlights the importance of logging, reporting, and configuration, concluding with a call to action for test automation enthusiasts.

1. How to Build Your Own Test
Automation Framework?
Dmitry Buzdin
TAPOST 18
October 11th/12th

2. Working as a consultant
• DevOps (Jenkins, Docker, Cloud etc)
• Test Automation (TDD, BDD, ATDD, ?DD)
• Software Architecture (aka Microservices)
• Java/Web Development
@buzdin
About Trainer

3. Agenda
• What is expected from a test framework?
• What makes a test framework?
• How to make one?
• What should be considered?

4. Test Framework Expectations

5. Lets Create a Test Automation
Framework!

6. Requirements
• Minimal cost of ownership
• No vendor lock-in
• Large user base
• Integration with existing tools
• Longevity
• Transparency

7. Assembling open source test
framework based on Java

8. What Makes a Test
Framework?

9. Framework vs Library
• Frameworks have key distinguishing features that separate them
from normal libraries:
• inversion of control: In a framework, unlike in libraries or in
standard user applications, the overall program's flow of control
is not dictated by the caller, but by the framework.
• extensibility: A user can extend the framework - usually by
selective overriding; or programmers can add specialized user
code to provide specific functionality.
• non-modifiable framework code: The framework code, in
general, is not supposed to be modified, while accepting user-
implemented extensions. In other words, users can extend the
framework, but should not modify its code.
https://en.wikipedia.org/wiki/Software_framework

10. Where would you find
the building blocks?

11. Download a Cucumber

12. VS
How to know if the component is good or bad?

13. We need some
architecture guidelines

14. ISTQB Generic Test Automation Architecture

15. Not sure if this is related to the
real world automated testing?

16. Test Definition Layer
Test Procedures
Test Cases
Test Conditions
Test Data
Test Library

17. Test Library
• Collection of reusable elements to compose test cases
• Directly related to business logic of SUT
• The most valuable asset, which should outlive
framework changes

18. Test Step
@Step
public void login(String username, String password) {
// performs login
}
@Step
public void navigateToPage(String page) {
// navigates browser to specified page
}
Test Library == Test Steps

19. Test Step Benefits
• Using minimal features of the actual framework
• Independent from each other
• Parametrised with inputs
Test Framework should support technology agnostic test steps

20. Test Conditions
• Test conditions are methods to check expectations
• Important selection factors:
• Possibility to write/find custom assertions
• Clarity of failure messages
• Additional assertion strategies
Test Conditions == Assertions

21. Fluent Assertions
http://joel-costigliola.github.io/assertj/
Readable assertions with auto-completion support

22. Power Assertions
available in Groovy

23. Soft Assertions
@org.junit.Rule
public final JUnitSoftAssertions softly;
@Step
public void ensureAllFieldsAreDisabledForUser() {
…
softly.assertThat(user.getText()).isEqualTo("user");
softly.assertThat(nameInput.isEnabled()).isFalse();
softly.assertThat(surnameInput.isEnabled()).isFalse();
softly.assertThat(submitButton.isEnabled()).isFalse();
softly.assertThat(logoutButton.isEnabled()).isTrue();
…
}
All assertion failures are recorded until end of the test

24. Assertion Groups
assertAll("person",
() -> assertEquals("John", person.getFirstName()),
() -> assertEquals("Doe", person.getLastName())
);
All assertions in a group are evaluated at the same time.
You will see all errors in output together.
http://junit.org/junit5/

25. Test Data
• Test Data is used to define test inputs for
abstract test cases
• Test Data can come from different sources and
formats

26. Data Tables
Spock Framework

27. JUnit 5 Data-Driven
@ParameterizedTest
@CsvFileSource(resources = "/data/users.csv")
public void testWithParameters(
String user,
String password,
Integer id) {
logger.info("{}:{}:{}", user, password, id);
}

28. Test Cases
“A set of input values, execution preconditions, expected
results and execution postconditions, developed for a
particular objective or test condition, such as to exercise
a particular program path or to verify compliance with a
specific requirement.”
http://glossary.istqb.org/search/test%20case

29. Test Case in Manual Testing
No Test Step Expectation Data
1 User logs in
navigated to
home page
username=“joe”
2 User selects item
navigated to
item page
item=“123”
3 User clicks “Add to Cart”
Shopping cart
updated
4 User clicks “Go To Cart”
navigated to
shopping cart
5 User clicks “Checkout”
Checkout screen
opens

30. Automated Testing
Where are all the test steps?

31. Proper Test Case
@Test
public void checkoutTest() {
login(“joe”, “password”);
assertOnPage(“/home”);
viewItem(123);
assertOnPage(“/item.123”);
addToCart();
assertCartItems(1);
goToCart();
…
}
Test Case contains:
• test steps
• assertions
• test data

32. Abstract vs Concrete
• Abstract (high-level) test cases
• Concrete (low-level) test cases = Data-Driven
• Relation between them should be kept
Test Framework should distinguish between those
@TestTemplate
public void abstractTest(Data data) {
}

33. Test Procedurs
Defines how to select and run a set of test cases?
Test Procedures == Test Suites

34. Testing Assets
Test Plans / Test Suites
Test Cases
Test Steps
Test Libraries
Test Data
Assertions

35. There is one more component!

36. Test Language
•You could write test cases in:
•Gherkin
•Java/Groovy
•Excel
•HTML
•Markdown
All of that could coexist
in a single framework!

37. Gherkin Language
Cross-platform specification
Cucumber JVM works both with JUnit and TestNG
Test
Step
Test
Conditions
Test
Step
Test
Case

38. Plain Language
@Test
public void ensureAllFieldsAreDisabledForUser() {
goToPage(“/login“);
login(“janedoe”, “password”);
assertPage(“/home”);
logout();
}
Same test in Java

39. Multi-Language Framework
Gherkin Test Cases
Test Steps
Test Adapters
JUnit Test Cases
Cucumber Step Defs
Reusing the same test steps, data and assertions

40. Test Execution Layer
Test Execution
Test Logging
Test Reporting

41. Test Execution
• API for Test Case creation
• Extension model
• Engine to run the tests
• External API to IDE, Build tools and GUI

42. JUnit 5 Architecture
Test Runner
Test Engine
Test
Language

43. Test Engine Responsibilities
• Provides API/Language for Test Definition
• Discovers test cases
• Knows how to run tests
• Provide engine-specific extension points

44. Test Runner Responsibilities
• Integrates with IDE, Build and others tools
• Integrates with Test Engines
• Runs tests according to test plan
• Starts multiple engines if needed
• Provides extension points

45. Test Logging
• Recording test execution event for further analysis
• Test logs are attached to test reports
• Vital to decrease investigation time of failures

46. System.out.println(
“HTTP Result for ” +
req.getUrl() +
“ is ” +
resp.getStatus()
);
This isn’t a good test logging strategy!

47. Logger Frameworks
• Possibility to change level of detail (runtime)
• Log message tagging
• Configurable log message format
private static final Logger logger =
LoggerFactory.getLogger(“NAME”);
logger.info(“Hello, logger!”);

48. Log Management Tools

49. Test Reporting
• Has to make sense to your Product Owner
• Technical drill-down should be available
• Aid in decision making process

50. Typical Test Report
This is a very poor test report

51. Open Questions
• What feature failed?
• Was this test failing recently?
• Is this the only test with such exception?
• Are there any screenshots?
• Which test inputs were used?
• Which environment?

52. Allure Framework 2.0

53. Test Case Run Details

54. Test Metadata
@Epic("JIRA-123")
@Story("JIRA-456")
@Feature(“Login")
@Description("hello")
@Owner("dmitry.buzdin")
@Severity(SeverityLevel.NORMAL)
@Test
public void allureTest() {…}

55. More Features
• Test steps are auto discovered
• Test data is captured
• All attachments are placed to according places
• Historical analysis: flakiness, screenshot diffs

56. Test Configuration
• Loading environment configurations
• Configuration profiles
• Different configuration sources (json, yaml, xml,
external system)
• Configuring SUT before test run

57. OWNER Library
http://owner.aeonbits.org/

58. Dependency Injection
• Necessary because of all building blocks
• Keeping intermediate results
• Keeping virtual user sessions
* same name for inversion of control

59. Dependency Scopes
• Test run scope
• Test suite scope
• Test case scope
• User session scope

60. Test Adaptation Layer
GUI
API
Services
Protocols
Databases
Simulators
Emulators

61. Test Adaptation Layer
• Means to interact with SUT
• GUI: Web, Mobile, Desktop, Image Recognition
• Protocols: HTTP, AMQP, SSH
• Database connectors

62. Test Adapters
• Adapters should be configurable
• Adapters should be independent from other test
framework components
• Adapters should be easy to use and replace

63. Rest Assured
http://rest-assured.io/
when().
get("/lotto/{id}", 5).
then().
statusCode(200).
body("lotto.lottoId", equalTo(5),
"lotto.winners.winnerId", containsOnly(23, 54));
Adapter DSL + Inversion of Control + Assertions
Easy to use, but hard to integrate with other tools

64. Sometimes,
simple HttpClient
is a better choice…

65. Building a Test Framework

66. Pieces of the Puzzle
Configuration
Test Reporting Test Engine Assertions
Logging Dependency
Injection
Test Language Test Adapters
Test Runner
Test Data

67. Sample Java Stack
Component Technology
Test Engine JUnit 5
Test Reporting Allure 2
Assertions AssertJ
Configurations OWNER
Test Language Cucumber-JVM
Logging Logback
Test Adapters Rest-assured, Selenide etc.
Dependency Injection Guice
Data-driven JUnit 5 + Extensions

68. How it all fits together?
• Test Automator value is in knowing all pieces of
puzzle and integrating those
• Skills and experience required!
• Plugins, plugins, plugins…

69. Tool Evaluation Checklist
• Documentation quality
• Number of extensions points
• Number of plugins
• StackOverflow responsiveness
• Project statistics
• Open standards

70. Git Hub Stats

71. Open Standards / APIs
• Gherkin language
• WebDriver https://www.w3.org/TR/webdriver/
• JUnit 5 Platform
• Allure Reporting Result Format

72. Summary

73. You have to pick and choose
components in order to build your
framework

74. Test Automation time is now!

75. October 26th Allure Meetup in Riga