Cloud testing: challenges and opportunities, TaaS, Integration Testing

Test challenges and methodologies
with Cloud
Dr Ganesh Iyer, Progress Software

Ref: Ganesh Neelakanta Iyer, Jayakhanna Pasimutu and Ramesh Loganathan "PCTF: An Integrated, Extensible Cloud Test Framework for
Testing Cloud Platforms and Applications" The 13th International Conference on Quality Software 2013, (QSIC,'13),

Introduction
 Cloud Computing and Web 2.0 technologies
• Web-driven applications

• Driven by browser and do not need any installation
• On-demand resource availability
• Faster time to market
• Reduced capital and operational expenses

2

© 2013 Progress Software Corporation. All rights reserved.

Introduction

Paradigm
shift

Seamless
upgrades

Sharing of
resources:
Multitenancy

Traditionally,
applications are
designed first,
then developed,
tested and
distributed

Traditionally,
software upgrade
needs to get new
version of the
software, down
the system and
upgrade

Traditionally,
software
development &
deployment is
mostly in
dedicated
resources

3

But with Cloud,
upgrade should
happen live with
minimal or no
down time E.g.
Gmail


With Cloud, these
resources are
often being shared
among multiple
customers

Auto-elasticity:
Resources ondemand

Performance

Disaster recovery

SaaS Applications;
Browser based
configuration and
usage

New test
dimensions

Changes in
existing
dimensions

Compatibility

Availability and
Business
Continuity

Security: Multitenant penetration,
Identity federation
management

Common Cloud Testing Dimensions

Elasticity
Testing

Security
Testing

Performance
Testing

Resource
acquisition/
Release
Time

Traversal
vulnerability

Compatibility
Testing

Time to
deploy
Accessibility
testing

User access/
Roles

Multi-tenancy
Time to
Genesis

Provisionin
g on-the-go

Load
Testing for
ELB

4

Identity
federation
management
Communicati
on latency
over SSL
Multi-tenant
penetration

Connectivity
and reliability
with 3rd
parties
Reliability
and
Availability
Latency


Globalization
and
Localization
testing

API
Integration
Testing

Live
Testing

Connectivity
and
invocation
testing

Disaster
recovery

API load
testing

Live
upgrade
testing

API security
Compatibility
under
different
situations

Self-healing
ability

Multitenancy

Availability
and
business
continuity

Multitenancy
Testing
Multi-tenant
penetration

Rigid failure
containment
Availability
and
Business
continuity
Risk of
correlated
behaviors
Service
transition
activity
analysis

Elasticity Testing


Based on subscription plans, check the maximum vertical/horizontal limit



Auto provisioning/freeing on-the-go



Testing for Load Balancing



Performance





5

Test for the impact while auto scaling
Response time/Release time for provisioning of resources
Load Testing of Different subscriptions


Elasticity and Scalability

• Limitations on max objects/applications at a time
• Number of applications that can be developed per
platform instance

Load
requirements

Cloud App Development
Platform

Load
requirements

Elasticity Load Testing for different usage scenarios

• 100’s of administrators accessing the management

Time

Time



6

Load
requirements

Load
requirements

console

Time

Unpredictable/Predictable burst: Some tenants have a specific pattern in
their usage and test for sudden expected/unexpected variations in the
usage

Time

Security Testing Implications
Outsourced + Insourced
External (Data Storage)






Sensitive data management in the Cloud
S3 Storage or any other cloud vendor storage
DBaaS (OE Database)

Third party services to be tested only for connectivity
with over services





Functionality testing for our services
Single sign on account for all the services
Application Security – using both the ProPaaS and
third party services

Multi-tenant penetration testing

Proprietary (API’s)






7



De-perimeterised

Traversal vulnerability
Communication latency over SSL
API Level and Application Security
User access/Roles testing





Access from different Clients to the ProPaaS
platform testing
From VPN, Firewall settings, Antivirus software
Authentication/Authorization

Identity federation management testing


https://collaboration.opengroup.org/jericho/cloud_cube_model_v1.0.pdf

Security

User access/roles

Web UI

Vulnerabilities and Attacks

• Authentication and authorization
• Identity federation management Single Sign-On
• Access from different clients to the
platform
• VPN, firewall settings, anti-virus
• User privileges

• SQL Injection
• URL Manipulation
• Cross-site scripting
• Password cracking
• Hidden-field manipulation

• Multi-tenancy penetration testing
• Traversal vulnerability
• DDoS attacks

Multi-tenant Database
• Data management at DB (Encryption
security)
• Data retention and destruction for DB:
Erase and sanitize when space is
reallocated

8

Other Security Concerns
• Fault-injection-based testing for web-driven services (Including verification for all
input fields, network interface, environment variables etc)
• Fuzzy testing for web-driven services (Injecting random data into application to
determine whether it can run normally under the jumbled input)
• Data privacy: Custom SLA capabilities


Performance Testing
 Time to deploy
 Density (multi-tenancy)
 Reliability Testing
 Availability Testing


To facilitate Follow the Sun advantage model

 Connectivity and reliability with 3rd party components from our Cloud
• Reliability of the data sent between two systems

9


Latency under different conditions

Network

• The round-trip time between the browser and the server
• The number of round-trip times it takes to completely load a web
page
• The protocol’s flow & congestion control properties, and
• Competing traffic, unreliable network

Processing
Client side
10

• The time it takes to the server to prepare the content that will be sent
to the user.
• Resource sharing introduces contention risks, increased recovery
latencies

• The time the web browser needs to prepare the received content to
be presented
• Latency when accessed from different unreliable sources (e.g.
handheld devices, PDAs etc..)


Live Testing and Failover Testing
 Robustness of infrastructure against failures
 Live Upgrade Testing - Managing customer applications for maintenance/upgrade
 Recovery time in case of product/platform crash
 Self healing ability in case of product/platform
 Availability and Business continuity in multi-tenant environment

11


Multi-Tenancy Testing
 Verifiable resource accounting for Billing
•

In case, multiple tenant’s data need to be captured simultaneously by the billing agent

 Multi-tenancy penetration testing
•

Tenant level access and their boundaries based on authentication and authorization

 Rigid failure containment between tenants
•

Failure because of one tenant’s action shouldn’t stall other tenant’s activities or bring down the complete system down

 Testing for the risk of correlated behaviors
•

Example: multiple application instances execute the same recovery action or periodic maintenance actions
simultaneously

 Service transition activity analysis
•

12

No service impact on other tenant instances when each and every tenant-specific configuration parameter is changed.


Compatibility Testing

Languages

Browsers

Platforms

Devices

13


PCTF: Progress Cloud Test Framework
PCTF

Injection strings

Test Logger
Input

Crawler

Error

parameters

Injector

patterns

Error Pattern
Manager

Security
Testing

Test manager
Injection strings & Results
Pages/hyperlinks

Client shell
Output
Repository

• Parameter

Test Repository:

• Test suite

Plug-ins

configuration

• Libraries: SQL string library, Error

•

Test suite
selection

Result
Analyzer

• Result/Log
collection

API Integration
Testing

SLA

patterns library etc

System Under Test (Cloud
Platform/Application)

Test DB
(MySQL)

Synthetic Load
Generator

• SLA mapping table
• SLA metrics database

SLA information
Stubs for 3rd parties

SLA Monitor

Testing
14

Ganesh Neelakanta Iyer, Jayakhanna Pasimutu and Ramesh Loganathan "PCTF: An Integrated, Extensible Cloud Test Framework for
Testing Cloud Platforms and Applications" The 13th International Conference on Quality Software 2013, (QSIC,'13),

Integration Testing

15


What is Integration Testing?
 Often, many Cloud-based systems will be integrated to each other for delivering a
cloud-based offering.
 For example, for a SaaS application, the SSO (single-sign-on) mechanism may be
handled by a 3rd party system and payment mechanisms might be handled by another
3rd party system such as PayPal.
 Hence, we need to get sandbox environments for such payment gateways which are
identitical to its production environment in order to have the systems behave same in
both test as well as live environments.

16


Typical issues
 Multiple systems in an integrated Cloud-based product behave differently and
verification process for different systems differ each other.
 Some of the systems do not allow automated deletion of data created for testing. So it
imposes a unique requirement to have unique users created every time we perform
such test automation

 Unpredictable delays in updating various systems.
 Different types of environments for testing. For example, presence of web UI testing
and runtime testing in one test scenario poses its own unique challenges

17


Integration test automation challenges
 Unique requirements:
• UI components
• Runtime components

 Need to either develop a test framework that can efficiently test both run time and UI
components or use a combination of two frameworks for runtime/UI testing
 Needs to have mechanisms to initiate the test written in one framework from the other
one and to generate a combined test results

 Using the exposed APIs provided by other third party systems integrated as part of the
product

18


Integration test automation with Cloud

19


TaaS

20


Overview of Testing as a Service TaaS
 Wikipedia - Testing as a Service (TaaS, typically pronounced 'tass') is a model of
software testing whereby a provider undertakes the activity of software testing
applications/solutions for customers as a service on demand. …involves the ondemand test execution of well-defined suites of test material, generally on an
outsourced basis.
 Shared Services delivery model
 Pay per by drink and not by Glass

 Standardised, Repeatable services

21


Courtesy: http://tinyurl.com/taasmphasis

TaaS: Conceptual Model
Customer 1

Customer …n

Customer 2

Fixed price
per product
Commercial
Models
SLAs

Customer Service Management

Pre-defined

Automation Offering

Service
Products

Move towards
standardised services

Service Catalogue for business

Web

Performance Offering

Perf .Test
Web/ERP

Functional Test
Offering

Customer
Interface
Activities and
deliverables
Predefined

Manual Testing

Demand Management
Service Catalogue for Operations

Owned by service
provider.
Continuous
Improvement

Service
n…

Service 3

Test
Assets

Test
Assets

Service 2
Test
Assets

Global Delivery
Model
Testing
Framewor
k
Test
Assets

TAAS
Engine

Internal Service Management
Owned by
service
provider

22

HAAS/
Cloud

In house
Tools /
External Tools

Test
Analyst

Test
Process

High sharing
of resources

Courtesy: http://tinyurl.com/taasmphasis


Advantages of testing in the Cloud

Traditional Testing

Testing in Cloud

Low asset utilization
Scalability: Long time to increase
capacity

Less time (instantaneous) increase
and reduction in capacity

Long time to build datacenters

Purchased as a service from cloud
providers

Difficult to manage

Better management and increased
productivity

Duplicate test systems

Aggregated system

Creates unnecessary wastes
23

Improved asset utilization

Cleaner, greener testing, saving in
CO2 emissions


Conclusions
 Various Cloud test dimensions
• Elasticity, Multi-tenancy, Security, Live Upgrade, Performance

 Integration Testing: A unique requirement with Cloud
 Automation challenges and possible approaches
 TaaS

24


gaiyer@progress.com
http://ganeshniyer.com

Cloud testing: challenges and opportunities, TaaS, Integration Testing

More Related Content

What's hot

Similar to Cloud testing: challenges and opportunities, TaaS, Integration Testing

More from Dr Ganesh Iyer

Recently uploaded

Cloud testing: challenges and opportunities, TaaS, Integration Testing