Bdd test automation analysis

1. BDD Test Automation Overview
Amit Verma © 1

2. Agenda
• Benefits of BDD Test Automation
• BDD & TDD Comparison
• Cucumber-JVM and J-Behave Comparison
• Open Source Test Automation Framework
• Test Automation Suites & Data Management
• N-1 Sprint vs In Sprint Test Automation Process
• Service Virtualization Solutions
• Q & A
Amit Verma © 2

3. Benefits of BDD Test Automation for Front End & Back End
3
Faster Script Development
Inbuilt TC & TS Parametrization
Inbuilt Test Data Handling
Feature Driven Automation
Test Cases In Business Language
Eliminate Need Of Test Management Tool
Tightly Coupled Test Case & Test Code
Parameters passing flexibility (All Data Types)
No Custom, Redundant Code
Easy Script Maintenance
Better Test Suite Management
Easy handling of I/O (URI, Res Status, Req/Res Header & body)
Detailed Execution Report Of Tested Feature
Better Collaboration among Dev, QA & SA
Amit Verma ©

4. BDD vs TDD Comparison
4
Parameters TDD BDD
Test Runner X-Unit BDD
Parallel Execution Of Tests Methods, Classes Features
Parallel Execution Of Builds Yes Yes
Test Script Programming Language High Level High Level
Scalability Yes Yes
Tools & Technologies Open-Source Open-Source
Tests Dependency Support Good Good
Code Examples Yes Yes
Forums Support High High
Official Site Yes Yes
General Usage Guide Yes Yes
Technologies Used To Build Application Doesn’t Matter Doesn’t Matter
Testing Approach Test Driven Behaviour/Feature Driven
Test Focus Unit Tests Functional, Integration, E2E
Participants QA QA, Developers, SME, Customer
Stakeholders Suitability Technical Technical & Business
Suitability For AUT UI UI & API
Our Customers Experience Front End Front End & Back End
Test Case Format English Business Language (Gherkin)
Code Reusability Medium Very High
Ease Of Automation Maintenance Medium High
Ease Of Adoption High Very High
Test Tagging / Filtering Level Test Case Test Case / Scenario, Feature, Module
Test Case Management External-3rd Party Inbuilt
Test Suite Management Good Excellent
Amit Verma ©

5. BDD vs TDD Comparison
5
Parameters TDD BDD
Test Case & Test Scripts Mapping Indirect Direct
Test Case Parametrization Support Limited* Very Good & Inbuilt
Hooks Support Average Very Good
Tests Execution Priority Support Good Very Good
Test Script Components Binding One To One One To One & One To Many
Documentation Average Good
Automation Code Quality High Very High
Usage Stacks (StackShare) 49 529
Followers (StackShare) 68 423
Reddit Points (StackShare) 0 14
StackOverflow Q&A (StackShare) 7.4K 9K
GitHub Stars (StackShare) 1.5K 2.9K
GitHub Forks (StackShare) 822 586
Commit Frequency (StackShare) Frequent Daily
Community Activity Medium Very High
Interest Over Time Declining Increasing
Code Duplicity High Very Low
Test Case Aligned With Business Behavior No Yes
Custom Code Very High Low
Test Coverage Medium Very High
Script Development Time Very High Low
Test Automation Cost Very High Medium
Test Automation Maintenance Cost High Low
Test Reporting Poor (Pass/Fail) Excellent (Detailed)
Flakiness Handling Support No Runtime
Test Suite Parametrization Support No Yes
Amit Verma ©

6. 6
Parameters TDD BDD
Scenario: POST API with 1 request URL, 1 Request Header,
10 Request Body & 15 Response body parameters
Test Coverage: I/O combinations like valid/invalid data & datatype,
boundary conditions, R vs O parameters, reserved keywords etc
Test Cases: Approx. 60-80
12-15 Days
Average of 4-5 Test Cases
Implementation & Testing
Per Day
5-6 Days
1-2 Days For Writing FF
2-3 Days For Step Definition
1 Day For Testing
Test Case & Test Code Binding Loosely Coupled & Hazy Tightly Coupled & Lucid
Inbuilt Features Limited Huge
Getting Started Guide No Yes
Test Parameter Variants Low High
Test Parameters Injection Poor Very Good
Tabular Parameters Support No Yes
Test Data Management Poor* Very Good & Inbuilt
Pretty Console Output No Yes
Multi Format Reports Support No Yes
Test Case Syntax Highlighting No Yes
RegEx Support No Yes
Test Case Conformity To Standards Medium Very Good
Framework Status Need to Build Ready For Use
Framework Features For Various Automation Need Need to Build Already Build
Test case dependency management Yes-Need to Build Inbuilt
Single Framework For UI & API automation needs No
Yes
Amit Verma ©
BDD vs TDD Comparison

7. Cucumber-JVM vs J-Behave Features Availability
7
Feature J-Behave Cucumber JVM
Official site ✔ ✔
Getting started guide ✔ ✔
General Usage guide ✔ ✔
API documentation ✔ ✔
Code examples ✔ ✔
Specialized forums ✔ ✔
Parameter variants ✔ ✔
Parameters injection ✔ ✔
Tabular parameters ✔ ✔
Examples ✔ ✔
Multi-line input ✔ ✔
Formatted input ✔ ✔
Auto Complete ✔ ✔
Feature/Story scope ✔ ✔
Feature J-Behave Cucumber JVM
Scenario scope ✔ ✔
Step scope ✔ X
Composite Steps ✔ X
Background & Hooks X ✔
Before/After Story ✔ ✔
Before/After Stories ✔ X
Before/After Scenario ✔ ✔
Before/After Run X ✔
Given Stories (Story Parametrization) ✔ X
Code Binding ✔ ✔
One to Many Mapping ✔ ✔
One to One Mapping ✔ X
Gherkin Given / When / Then ✔ ✔
Gherkin And, But X ✔
Feature J-Behave Cucumber JVM
Console output ✔ ✔
Pretty console output ✔ ✔
Structured file (e.g. XML) ✔ ✔
Well-formatted readable file ✔ ✔
Usage report ✔ ✔
Extra report types ✔ ✔
Feature ✔ ✔
Scenario ✔ ✔
Scenario Outline X ✔
Examples ✔ ✔
Background X ✔
Tag ✔ ✔
Tables ✔ ✔
External Data Sources (Jira) ✔ X
Amit Verma ©

8. 8
Feature J-Behave Cucumber JVM
Official Documentation 4 2
Getting Started Guides 4 3
General Usage Guides 4 2
API Documentation 4 4
Code Examples 4 3
Forums 4 4
Up-to-date Blogs, Tutorials 2 3
Active Community 3 4
User stories/features 4 4
Scenario 4 4
Background 4 4
Feature J-Behave Cucumber JVM
Scenario Outline 4 4
Examples 4 4
Given/When/Then/And/But 4 4
Tags 4 4
Tables 4 4
Flexibility in passing parameters 4 4
Syntax highlight, auto-complete 4 4
Composite Steps 4 1
Backgrounds and Hooks 3 4
Formatting Flexibility 3 4
Binding to code (Annotations) 4 4
Feature J-Behave Cucumber JVM
Lambda Exp support 3 4
External Data Support 4 3
Standard console output 4 4
Pretty console output 4 4
JSON output 4 4
XML output (Structured) 4 4
HTML output 4 4
Usage report 4 4
Visual (graphs, charts) 3 4
Report attachments 3 4
Code quality 4 3
Cucumber-JVM vs J-Behave Features Weightage
0 No
1 Poor
2 Good
3 Very Good
4 Excellent
Score
J-Behave Cucumber JVM
Available Features
37 / 42 36 / 42
Features Score
124 120
Stack Overflow Tag Analysis J-Behave Cucumber
All questions 454 664
Zero answers 90 102
Questions with no upvoted answers 213 291
GitHub Status J-Behave Cucumber
Stars 195 1506
Watchers 35 164
Forks 225 1289
Commit Regular Frequent
Engine
Feature
Documentation
Parameters
Passing
Flexibility
Auto
Complete
Scoping
Composite
Steps
Backgrounds Hooks
Code
Binding
Formatting
Flexibility
Built-In
reports
Conformity
To
Gherkin
Standards
Input Data
Sources
Total
External Inclusions
J-Behave 4 4 4 4 4 0 4 4 3 4 4 4 2 45
Cucumber-JVM 3 3 4 2 1 4 3 4 4 4 4 0 0 36
http://mkolisnyk.blogspot.com/2013/03/jbehave-vs-cucumber-jvm-comparison.html
https://medium.com/agile-vision/jbehave-vs-cucumber-jvm-comparison-and-experience-sharing-439dfdf5922d
Amit Verma ©

9. 9
S
Y
S
T
E
M
U
N
D
E
R
T
E
S
T
REST Client
Config
Feature File
Data Handler
Reporting
Integrations*
Web UI Client
I
/
O
H
A
N
D
L
E
R
Runner
Test Code
Appium
Server
Android Client
iOS Client
M
E
D
I
A
H
A
N
D
L
E
R
A
U
T
H
H
A
N
D
L
E
R
Assertion
Handler
SOAP Client
DB Client
ESB Client
GCS Client
Redis Client
Oracle Client
Mongo Client
MySQL Client
Spanner Client
ActiveMQ Client
Tibco Client
Kafka Client
API Client
Storage
Client
Version
Control
CI / CD
Build
Web UI
API / Services
Messaging
Database
Storage
Component
Open Source
Customizable
Integration
E2E & NFR
Open Source Test Automation Framework Architecture
Amit Verma ©

10. Test Automation Suites & Data
10
Test Suite Type Environment
Fit / Sanity Dev
Feature QA
Regression & Scope Stage / Release
Smoke Pre-Prod / Prod
Test Case Type Scope
Functional / Component Level API Business & Auth
Integration UI, SOAP & Rest CRUD
System / E2E ESB, DB, 3rd Party & Down Streams
NFR Circuit Breaker, Timeout, Lockout
Batch Job Data Load / Feed
Test Data Management
Framework Managed - Dynamic Create / Retrieve / Update on Fly
Test Case Managed - Static Static Setup* (Periodic Updates if
Corrupted / Expired)
Amit Verma ©

11. 11
Application
Code
Unit &
Integration
Tests
Step
Definitions
Create DEV
Branch-A
Create AUTO
Branch-A
Commit
AUTO
Branch-A
Commit DEV
Branch-A
Dry Run
Test Feature
&
Report Bugs
Defect Fix
Merge DEV & AUTO
Branch-A
(Common Repo*)
Create & Deploy
Artifact-EAR
Static Code Analysis
Checkout AUTO
Branch-A
Create & Run
Artifact-Jar
Regression
Defect
Broken Fix
Checkout DEV
Branch-A
Destroy Dynamic
Environment
(Common Repo*)
Spin Dynamic
Environment
(Common Repo*) Merge DEV & AUTO
Branch-A
(Common Repo*)
Sync – Master
AUTO Branch-A
Sync – Master
DEV Branch-A
Lead Approval
DEV Branch-A
Lead Approval
AUTO Branch-A
Checkout AUTO
Branch-A
Checkout DEV
Branch-A
Destroy Dynamic
Environment
(Common Repo*)
Spin Dynamic
Environment
(Common Repo*)
Push To Master
Promote Artifacts To
Nexus For UAT
Create & Deploy
Artifact-EAR
Create & Run
Artifact-Jar
Feature Development
& Testing
CI Build Testing
(Auto Trigger)
CI Artifact Testing
(Manual Trigger)
In Sprint Test Automation Process
PM BA/SA/SME
Dev QE
Dev-Ops
Stories
Writing
Sprints
Planning
Stories
Walkthrough
& Sizing
Planning
Stories
Closure
Features
Showcase
Retrospective
Branch Cut &
Release
Deployment
Regression &
Release
Sign-Off
Prod Deployment
& Smoke Test
Sprint Closure Release
Write FF
Review FF
Sign-Off FF
Story In Play
Amit Verma ©

12. 12
Write FF
Review FF
Sign-Off FF
Application
Code
Unit & Integration
Tests
Stories
Writing
Sprints
Planning
Stories
Walkthrough
& Sizing
Nth -1 Sprint
Defect Fix
Stories
Closure
Features
Showcase
Retrospective
Branch Cut &
Release Deployment
Regression & Release
Sign-Off
Prod Deployment
& Smoke Test
Nth Sprint Feature
Development
Sprint Closure Release
Nth Sprint Story In
Play
Planning
Step
Definitions
Dry Run
Test Feature &
Report Bugs
Nth-1 Sprint Feature
Testing
PM BA/SA/SME Dev QE Dev-Ops
N-1 Sprint Test Automation Process
Amit Verma ©

13. 13
Why?
• Emulate behaviour of unavailable components in heterogeneous
system for software development and testing purpose when they
are unavailable due to…
• Not Developed / Under Development
• Requires Access Fee
• Test Data Setup Constraint
• Error Out Cases Validation
• Unreliable & Nondeterministic Behaviour
What?
• Component’s Requires Virtualization
• Restful & SOAP Services
• Messaging Systems (ESB/JMS)
• File Transfers
• Any other Software / Application
Mocking /
Stubbing
•Done within dev code
•Not suitable for system testing as it does not replicate real behaviour of application
•Not suitable for NFR’s such as security, Hystrix Circuit Breaker / Circuit Timeout
•Not suitable to carry out performance / stress / load test
Service Virtualization
Amit Verma ©

14. Solution Support
Traffic Parrot SOAP & Rest Services,
JMS IBM MQ,
File Transfers
Mountebank SOAP & Rest Services,
SMTP, TCP
CA Service Virtualization SOAP & Rest Services,
Mainframe
Mock Lab
(Built on Wire Mock)
SOAP & Rest Services
UP9 SOAP & Rest Services
Custom
Implementation
SOAP & Rest Services,
JMS ActiveMQ /
TibcoMQ / Kafka*,
File Transfers*
Parameters Custom Implementation Tools
Virtualization Suitability Test Automation & Manual Testing Test Automation & Manual Testing
Services Virtualization Good Good
Messaging Virtualization Good* Limited
TCP, SMTP, FTP
Virtualization
Good* Limited
Feature Specific Test Data Dynamic Static
Dynamic System Responses High Medium
Network Latency Support Good Good
Scalability High Medium
Cost of Deployment NA (Managed within JRE) Paid
Effort of Implementation High (Requires Coding) Low (UI Interface, No coding)
Transactions Capacity Sufficient for Functional Test Automation
but not suitable for Performance Testing
Sufficient for Performance Testing but not suitable
for Functional Test Automation
(Static Data / Responses & deployment challenges*)
14
Virtualization Solutions Comparison
Amit Verma ©

15. Thank You
Q & A
Amit Verma © 15