What are the advantages of test automation? What are different automation considerations? what are various tools used in practice? What are various non-functional test automation dimensions?

1. Functional &
Non-functional Test
Automation
Dr Ganesh Neelakanta Iyer
QA Architect
GANESHNIYER
gaiyer@progress.com
http://ganeshniyer.com

2. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.2
About Me
GANESHNIYER

3. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.3
Quick Poll

4. Do you have some test
automation in place for your
testing?

5. Do you have required functional
test automation in place?

6. Do you have test automation for
functional and non-functional
(performance, load, scalability,
security, …) pieces of testing?

7. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.7
Agenda
 Goal of successful automation
 Types of test automation frameworks
 Functional Test automation
 Non-Functional test automation – Major types

8. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.8
Goals of Successful Automation

9. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.9
The Goal: Successful Automation
 Suggestions:
• Automate the ‘Right’ Test Cases
• Test Early / Test Often
• Define Quality Test Data
• Design Automated Tests that can resist change
• Selecting the right Automation Tool
Graphic from qatestlab.com

10. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.10
Automate the ‘Right’ Test Cases
• Repetitively Run Tests (aka Smoke
or Acceptance tests)
• Data intensive tests
• Frequently Used Functionality
• High Risk, Business-critical
functionality
• Tests that are difficult to run
manually
• Tests that are run on differing
hardware/software platforms and
configurations
• Tests that take a lot of effort or
time
Graphic from qatestlab.com

11. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.11
Test Early / Test Often
 Start as early in the process as possible
– Could start with Automated Unit testing
– Gradually build your Automation Suite
 The more you test, the more bugs you find.
 The earlier bugs are found the cheaper they are to fix!

12. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.12
Define Quality Test Data
 With good data…
– Writing and Maintaining Automated tests is easier.
– With Data driven tests you can run more iterations and find more bugs!
– Can extend existing Automated tests as new features are developed – without
having to edit the actual automated test itself.
– Stay proactive in updating/defining data.

13. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.13
Automation Approach
 Keep Automated tests small and focused
– Easy to debug
– Easy to manage
– Can be reused and shared
 These can then be grouped into a larger Automated test suite organized by
functional area.
 With the right framework, users can create large/complex Automated tests that are
also easy to maintain!
Graphic from checkpointech.com

14. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.14
Design Automated Tests Resistant to Change
 If possible, avoid using screen coordinates.
 Componentize your test scripts
 Ensure that your development team uses unique names for each control and
object!

15. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.15
How long do Automated Tests Last?
 At some point, the application will change and the script will break.
• If you are using GUI Capture/Playback a major revision to the UI may break your tests.
 If the script breaks before the automation effort was repaid, it is better off as a manual
test.

16. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.16
Selecting the ‘‘Right” Automation Tool
Can be used by business users
No programming skills required
Simple
Automate 60-70% of core
business processes in a few days
Rapid return on investment
Fast
When underlying applications change,
automation continues to work
Simple maintenance of your
business process portfolio
Resilient
True end-to-end business process validation
covering all major technologies and platforms
Complete

17. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.17
Return On Investment (ROI)

18. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.18
Classic ROI Calculation
ROI = BENEFIT/COST
Automation Cost = Price Of HW + Price of SW + Development Cost +
Maintenance Cost + Execution Cost
Manual Testing Cost = Development Cost + Maintenance Cost +
Execution Cost
ROI = (Manual Testing Cost - Automation Cost)/Automation Cost
Looks right, isn’t it?

19. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.19
Problems with Classic ROI Calculation
 You can’t compare Automated Testing and Manual Testing. There
are not the same
 You can’t compare cost of multiple execution of automated tests vs.
manual tests. You would never dream of executing that many test
cases manually.

20. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.20
Automation Real ROI
 ROI value IS NOT the value of Automation vs. Cost of executing
these tests manually
 Automation ROI value IS the benefit of this type of testing, and it
can be:
 Reducing Time to Market
 Increased Test Efficiency (Productivity)
 Increased Test Effectiveness

21. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.21
Types of test automation
frameworks

22. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.22
Types of Test Automation Frameworks
Modular Data-Driven
Keyword–Driven Model-Based

23. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.23
Modular framework
 The Modular framework is the natural progression from Record-and-
Playback
 The modular framework seeks to minimize this repetition of code by
grouping similar actions into “modules” (e.g.: login)
 Test Data is in a script

24. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.24
Data-Driven/Keyword Driven
 These frameworks are similar in that the data is separated from the test
script
 The script is just a "driver" or delivery mechanism for the data.
 The difference:
• In keyword-driven testing, the navigation data and test data are contained in the data
source
• In data-driven testing, only test data is contained in the data source.

25. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.25
Model-Based testing
 Model-based testing is software testing in which test cases are derived
in whole or in part from a model that describes some (usually functional)
aspects of the system.
 Model-based testing for complex software systems is still an evolving
field.

26. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.26
Functional test automation

27. Typically what we speak about
is all about functional
automation

28. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.28
Functional Testing
Web
 Telerik Test
Studio
 Sahi
 Selenium
 Sikuli
Web Mobile
Desktop Backend
Mobile
 Mobile Test
Studio
 Telerik
DeviceCloud
 Appium
Desktop
 QTP

29. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.29
Testing Mobile Apps in Cloud
Telerik DeviceCloud
>300
29
27
Devices
Manufacturers
OS versions

30. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.30
Compatibility Testing; Web, Device, OS, DB,….
Automation:
 Telerik Test
Studio
 Telerik
DeviceCloud
 Sahi
 Selenium
Compatibility
Testing
Accessibility
testing
Globalization and
localization testing
Compatibility
under different
situations

31. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.31
Non-Functional test automation –
Major types

32. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.32
Major Non-functional test dimensions in a Nut-shell
Non-
functional
Security
API
Connectivity
PerformanceResilience
Integration

33. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.33
Security
Security
Traversal
vulnerability
User access/ roles
Identity federation
management
Multi-tenant
penetration
User
Access /
Roles
Web UI
Security
Multi-
Tenant
DB
Vulnerabilities
Automation:
 Veracode
 Websecurify
 ZAP tool
 Scripts

34. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.34
Security at various stages
Planning
Architecture
and Design
Implementation
Level
Whitebox
testing
Blackbox
testing
Reviews
Code Doc Log Regional Security Compliance Ruleset System Config Network Sniffing File Integrity Checking
PO/
PM
QA
Dev Architect
/ Lead
Dev QA QA
Dev QA QADev Architect
/ Lead
Dev Architect
/ Lead
QA

35. Different components of each aspect- what needs to be tested/assured
Security
protection
methods
Authentication
Access Control
Cryptography
Firewall, Anti-virus
etc
Secured
Infrastructure
Web Security
URL Manipulation
SQL Injection,
Other injection
attacks
Session fixation,
XSS
API Security
Testing
Cloud
Security
Multi-tenant
penetration (for
ISVs)
Data protection in
Cloud
Cloud Identity
Management
Cloud Infrastructure
security
Database
Security
Reliability and
Integrity
DB Disclosure
Security in data
mining
Data security
Vulnerabilities
and attacks
Network, Memory-
related, Metadata
injection
Traversal
vulnerability
DoS/DDoS attacks
Encryption in
networks/browsers
Programmed
Injection
Privacy, Legal
issues and
Ethics
Web privacy
Law and
Compliance
Email security
Ethical issues
Emerging
Technologies
and Security
Aspects
Mobile App
Mobile Web
IoT sensors
Information Security
Security Issues within the code

36. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.36
How we plan?– Four types of activities
High-level test approach and test execution frequency
QA Reviews
• Doc Review
• Code Review
Static Analysis (Whitebox Testing)
•Analyze source for security vulnerabilities typically at the
programming and/or testing software life cycle (SLC)
phases
Service Level Dynamic Testing
• Test the run time services exposed by applications
for various security testing
Business Level Dynamic Testing
•Application Runtime analysis, E2E business use
cases
To happen regularly on
a scheduled manner
For short releases 
Once in the release
For long releases, 2-3
times during the
release cycle
Sanity Suite to run on
Continuous Integration
Automate the cases
and run once in a
sprint
Automate and run once
in a sprint
Use tools for scanning
1-3 times based on
release cycle

38. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.38
API Invocation
API
Invocation
Connectivity and
invocation
API load
API security
Multi-tenancy
Automation:
 SoapUI
 Node.js
 PySys
 In-house
Frameworks

39. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.39
High Availability (HA) and Performance Testing
Automation:
 Telerik Test
Studio
 NeoLoad
 JMeter
HA &
Performance
Time to deploy
Multi-tenancy
Latency
Connectivity and
reliability with
3rd parties
Reliability and
availability
HA/
Performance
Testing
Availability
24 x 7 –
Endurance
Reliability
Latency
Stress

40. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.40
24 x 7 Testing
VMs / Build Integration Machines
ANT ScriptTests
Reports using
Kendo UI controls
Decides
parallelism
HTTP POST reqBatch
Script
Decides
frequency
Logs
SUT
Uses any test
environments
and leverage
existing tests
Batch Script
Decides the
frequency
ANT Script
We configure
#Threads,
#Tests to run in
parallel
Reports
Shown using
Keno UI
controls /
widgets

41. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.41
Resilience and Failover
Ability of a server, network, storage
system, or an entire data center, to
continue operating even when there has
been an equipment failure, power
outage or other disruption
Software
error
Hardware
error
Human
error
Natural
Disaster
Picture Credit: http://imperfectspirituality.com/wp-content/uploads/2012/08/Moving-
mountains-with-resilience.jpg

42. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.42
Resilience and Failover
Resilience
Making one
instance down
Making multiple
instances down
Disaster Recovery
Application
Interface
(UI / API)
IaaS
Logs
ELB
HA Instance 2Stubs for
3rd Parties
HA Instance 1

43. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.43
Integration
Integration
P2P Integration
System
Integration
End-to-End
Integration
SSO
Customer
Scenarios
Picture Credit: http://goo.gl/wgNKi3

44. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.44
A Typical Integration Scenario in Progress
OpenEdge DB
OData
OData REST
Service
Telerik
Mobile
QA Dashboard App
DMZ REST
Wrapper
REST Service

45. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.45
Typical End-to-End Scenario
Registration
Land on
product
dashboard
Click on
subscription
details
Choose a
plan
Checkout
process
Generate
invoice
Product
usage
Log-out
LDAP Community
Export Compliance,
Marketing
Marketing,
Billing,
Support
Billing,
Invoicing
LDAP
Billing
Metering
Billing

46. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.46
Integration Test Challenges and Approaches
Multiple systems behave differently
No automated deletion of test data in some systems
Unpredictable delays in updating various systems
Different types of environments for testing

47. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.47
Integration Test Automation With Cloud
Telerik Test Studio as the UI framework, C# as the Runtime framework
Output
Repository
Test
Manager
Test
Logger
Run Time Test
Framework
API Calls
UI
Framework
Stubs for
3rd Parties
Test Repository
 Test suite
 Libraries (APIs)
Product
1
Product
2
Product
3

48. There are much more…

49. Internationalization Localization
Hot-Upgrade Multi-tenancy

50. Backward
Compatibility
Limits
Migration Installation

51. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.51
Major Test Dimensions in a Nut Shell
Security Performance
Resilience &
Failover
Compatibility
API
Connectivity
Integration
Traversal
vulnerability
Time to deploy
Making one
instance down
Accessibility
Connectivity and
invocation
P2P Integration
User access/ roles Multi-tenancy
Making multiple
instances down
Globalization and
localization
API load
System
Integration
Identity federation
management
Latency Disaster Recovery
Compatibility
under different
situations
API security
End-to-End
Integration
Multi-tenant
penetration
Connectivity and
reliability with
3rd parties
Multi-tenancy SSO
Reliability and
availability
Customer
Scenarios

52. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.52
Automation Frameworks
Telerik Test Studio, Telerik DeviceCloud
Sahi / Selenium scripts
Telerik Test Studio for Load / Performance Testing
Neoload, JMeter
SOAP UI / Node.js / PySys
Telerik Test Studio (with C#)
TestNG + Java + Sahi combined approach
Integration
Functional
(Web/Mobile UI)
Performance
API Invocation
Veracode, Websecurify, ZAP ToolSecurity

53. gaiyer@progress.com
http://ganeshniyer.com
GANESHNIYER