SlideShare a Scribd company logo
1 of 25
Test challenges and methodologies
with Cloud
Dr Ganesh Iyer, Progress Software

Ref: Ganesh Neelakanta Iyer, Jayakhanna Pasimutu and Ramesh Loganathan "PCTF: An Integrated, Extensible Cloud Test Framework for
Testing Cloud Platforms and Applications" The 13th International Conference on Quality Software 2013, (QSIC,'13),
Introduction
 Cloud Computing and Web 2.0 technologies
• Web-driven applications

• Driven by browser and do not need any installation
• On-demand resource availability
• Faster time to market
• Reduced capital and operational expenses

2

© 2013 Progress Software Corporation. All rights reserved.
Introduction

Paradigm
shift

Seamless
upgrades

Sharing of
resources:
Multitenancy

Traditionally,
applications are
designed first,
then developed,
tested and
distributed

Traditionally,
software upgrade
needs to get new
version of the
software, down
the system and
upgrade

Traditionally,
software
development &
deployment is
mostly in
dedicated
resources

3

But with Cloud,
upgrade should
happen live with
minimal or no
down time E.g.
Gmail

© 2013 Progress Software Corporation. All rights reserved.

With Cloud, these
resources are
often being shared
among multiple
customers

Auto-elasticity:
Resources ondemand

Performance

Disaster recovery

SaaS Applications;
Browser based
configuration and
usage

New test
dimensions

Changes in
existing
dimensions

Compatibility

Availability and
Business
Continuity

Security: Multitenant penetration,
Identity federation
management
Common Cloud Testing Dimensions

Elasticity
Testing

Security
Testing

Performance
Testing

Resource
acquisition/
Release
Time

Traversal
vulnerability

Compatibility
Testing

Time to
deploy
Accessibility
testing

User access/
Roles

Multi-tenancy
Time to
Genesis

Provisionin
g on-the-go

Load
Testing for
ELB

4

Identity
federation
management
Communicati
on latency
over SSL
Multi-tenant
penetration

Connectivity
and reliability
with 3rd
parties
Reliability
and
Availability
Latency

© 2013 Progress Software Corporation. All rights reserved.

Globalization
and
Localization
testing

API
Integration
Testing

Live
Testing

Connectivity
and
invocation
testing

Disaster
recovery

API load
testing

Live
upgrade
testing

API security
Compatibility
under
different
situations

Self-healing
ability

Multitenancy

Availability
and
business
continuity

Multitenancy
Testing
Multi-tenant
penetration

Rigid failure
containment
Availability
and
Business
continuity
Risk of
correlated
behaviors
Service
transition
activity
analysis
Elasticity Testing


Based on subscription plans, check the maximum vertical/horizontal limit



Auto provisioning/freeing on-the-go



Testing for Load Balancing



Performance





5

Test for the impact while auto scaling
Response time/Release time for provisioning of resources
Load Testing of Different subscriptions

© 2013 Progress Software Corporation. All rights reserved.
Elasticity and Scalability

• Limitations on max objects/applications at a time
• Number of applications that can be developed per
platform instance

Load
requirements

Cloud App Development
Platform

Load
requirements

Elasticity Load Testing for different usage scenarios

• 100’s of administrators accessing the management

Time

Time



6

Load
requirements

Load
requirements

console

Time

Unpredictable/Predictable burst: Some tenants have a specific pattern in
their usage and test for sudden expected/unexpected variations in the
usage
© 2013 Progress Software Corporation. All rights reserved.

Time
Security Testing Implications
Outsourced + Insourced
External (Data Storage)






Sensitive data management in the Cloud
S3 Storage or any other cloud vendor storage
DBaaS (OE Database)

Third party services to be tested only for connectivity
with over services





Functionality testing for our services
Single sign on account for all the services
Application Security – using both the ProPaaS and
third party services

Multi-tenant penetration testing

Proprietary (API’s)






7



De-perimeterised

Traversal vulnerability
Communication latency over SSL
API Level and Application Security
User access/Roles testing





Access from different Clients to the ProPaaS
platform testing
From VPN, Firewall settings, Antivirus software
Authentication/Authorization

Identity federation management testing

© 2013 Progress Software Corporation. All rights reserved.

https://collaboration.opengroup.org/jericho/cloud_cube_model_v1.0.pdf
Security

User access/roles

Web UI

Vulnerabilities and Attacks

• Authentication and authorization
• Identity federation management Single Sign-On
• Access from different clients to the
platform
• VPN, firewall settings, anti-virus
• User privileges

• SQL Injection
• URL Manipulation
• Cross-site scripting
• Password cracking
• Hidden-field manipulation

• Multi-tenancy penetration testing
• Traversal vulnerability
• DDoS attacks

Multi-tenant Database
• Data management at DB (Encryption
security)
• Data retention and destruction for DB:
Erase and sanitize when space is
reallocated

8

Other Security Concerns
• Fault-injection-based testing for web-driven services (Including verification for all
input fields, network interface, environment variables etc)
• Fuzzy testing for web-driven services (Injecting random data into application to
determine whether it can run normally under the jumbled input)
• Data privacy: Custom SLA capabilities

© 2013 Progress Software Corporation. All rights reserved.
Performance Testing
 Time to deploy
 Density (multi-tenancy)
 Reliability Testing
 Availability Testing


To facilitate Follow the Sun advantage model

 Connectivity and reliability with 3rd party components from our Cloud
• Reliability of the data sent between two systems

9

© 2013 Progress Software Corporation. All rights reserved.
Latency under different conditions

Network

• The round-trip time between the browser and the server
• The number of round-trip times it takes to completely load a web
page
• The protocol’s flow & congestion control properties, and
• Competing traffic, unreliable network

Processing
Client side
10

• The time it takes to the server to prepare the content that will be sent
to the user.
• Resource sharing introduces contention risks, increased recovery
latencies

• The time the web browser needs to prepare the received content to
be presented
• Latency when accessed from different unreliable sources (e.g.
handheld devices, PDAs etc..)

© 2013 Progress Software Corporation. All rights reserved.
Live Testing and Failover Testing
 Robustness of infrastructure against failures
 Live Upgrade Testing - Managing customer applications for maintenance/upgrade
 Recovery time in case of product/platform crash
 Self healing ability in case of product/platform
 Availability and Business continuity in multi-tenant environment

11

© 2013 Progress Software Corporation. All rights reserved.
Multi-Tenancy Testing
 Verifiable resource accounting for Billing
•

In case, multiple tenant’s data need to be captured simultaneously by the billing agent

 Multi-tenancy penetration testing
•

Tenant level access and their boundaries based on authentication and authorization

 Rigid failure containment between tenants
•

Failure because of one tenant’s action shouldn’t stall other tenant’s activities or bring down the complete system down

 Testing for the risk of correlated behaviors
•

Example: multiple application instances execute the same recovery action or periodic maintenance actions
simultaneously

 Service transition activity analysis
•

12

No service impact on other tenant instances when each and every tenant-specific configuration parameter is changed.

© 2013 Progress Software Corporation. All rights reserved.
Compatibility Testing

Languages

Browsers

Platforms

Devices

13

© 2013 Progress Software Corporation. All rights reserved.
PCTF: Progress Cloud Test Framework
PCTF

Injection strings

Test Logger
Input

Crawler

Error

parameters

Injector

patterns

Error Pattern
Manager

Security
Testing

Test manager
Injection strings & Results
Pages/hyperlinks

Client shell
Output
Repository

• Parameter

Test Repository:

• Test suite

Plug-ins

configuration

• Libraries: SQL string library, Error

•

Test suite
selection

Result
Analyzer

• Result/Log
collection

API Integration
Testing

SLA

patterns library etc

System Under Test (Cloud
Platform/Application)

Test DB
(MySQL)

Synthetic Load
Generator

• SLA mapping table
• SLA metrics database

SLA information
Stubs for 3rd parties

SLA Monitor

Testing
14

Ganesh Neelakanta Iyer, Jayakhanna Pasimutu and Ramesh Loganathan "PCTF: An Integrated, Extensible Cloud Test Framework for
Testing Cloud Platforms and Applications" The 13th International Conference on Quality Software 2013, (QSIC,'13),
© 2013 Progress Software Corporation. All rights reserved.
Integration Testing

15

© 2013 Progress Software Corporation. All rights reserved.
What is Integration Testing?
 Often, many Cloud-based systems will be integrated to each other for delivering a
cloud-based offering.
 For example, for a SaaS application, the SSO (single-sign-on) mechanism may be
handled by a 3rd party system and payment mechanisms might be handled by another
3rd party system such as PayPal.
 Hence, we need to get sandbox environments for such payment gateways which are
identitical to its production environment in order to have the systems behave same in
both test as well as live environments.

16

© 2013 Progress Software Corporation. All rights reserved.
Typical issues
 Multiple systems in an integrated Cloud-based product behave differently and
verification process for different systems differ each other.
 Some of the systems do not allow automated deletion of data created for testing. So it
imposes a unique requirement to have unique users created every time we perform
such test automation

 Unpredictable delays in updating various systems.
 Different types of environments for testing. For example, presence of web UI testing
and runtime testing in one test scenario poses its own unique challenges

17

© 2013 Progress Software Corporation. All rights reserved.
Integration test automation challenges
 Unique requirements:
• UI components
• Runtime components

 Need to either develop a test framework that can efficiently test both run time and UI
components or use a combination of two frameworks for runtime/UI testing
 Needs to have mechanisms to initiate the test written in one framework from the other
one and to generate a combined test results

 Using the exposed APIs provided by other third party systems integrated as part of the
product

18

© 2013 Progress Software Corporation. All rights reserved.
Integration test automation with Cloud

19

© 2013 Progress Software Corporation. All rights reserved.
TaaS

20

© 2013 Progress Software Corporation. All rights reserved.
Overview of Testing as a Service TaaS
 Wikipedia - Testing as a Service (TaaS, typically pronounced 'tass') is a model of
software testing whereby a provider undertakes the activity of software testing
applications/solutions for customers as a service on demand. …involves the ondemand test execution of well-defined suites of test material, generally on an
outsourced basis.
 Shared Services delivery model
 Pay per by drink and not by Glass

 Standardised, Repeatable services

21

© 2013 Progress Software Corporation. All rights reserved.

Courtesy: http://tinyurl.com/taasmphasis
TaaS: Conceptual Model
Customer 1

Customer …n

Customer 2

Fixed price
per product
Commercial
Models
SLAs

Customer Service Management

Pre-defined

Automation Offering

Service
Products

Move towards
standardised services

Service Catalogue for business

Web

Performance Offering

Perf .Test
Web/ERP

Functional Test
Offering

Customer
Interface
Activities and
deliverables
Predefined

Manual Testing

Demand Management
Service Catalogue for Operations

Owned by service
provider.
Continuous
Improvement

Service
n…

Service 3

Test
Assets

Test
Assets

Service 2
Test
Assets

Global Delivery
Model
Testing
Framewor
k
Test
Assets

TAAS
Engine

Internal Service Management
Owned by
service
provider

22

HAAS/
Cloud

In house
Tools /
External Tools

Test
Analyst

Test
Process

High sharing
of resources

Courtesy: http://tinyurl.com/taasmphasis

© 2013 Progress Software Corporation. All rights reserved.
Advantages of testing in the Cloud

Traditional Testing

Testing in Cloud

Low asset utilization
Scalability: Long time to increase
capacity

Less time (instantaneous) increase
and reduction in capacity

Long time to build datacenters

Purchased as a service from cloud
providers

Difficult to manage

Better management and increased
productivity

Duplicate test systems

Aggregated system

Creates unnecessary wastes
23

Improved asset utilization

Cleaner, greener testing, saving in
CO2 emissions

© 2013 Progress Software Corporation. All rights reserved.
Conclusions
 Various Cloud test dimensions
• Elasticity, Multi-tenancy, Security, Live Upgrade, Performance

 Integration Testing: A unique requirement with Cloud
 Automation challenges and possible approaches
 TaaS

24

© 2013 Progress Software Corporation. All rights reserved.
gaiyer@progress.com
http://ganeshniyer.com

More Related Content

What's hot

Taking Testing to the Cloud
Taking Testing to the CloudTaking Testing to the Cloud
Taking Testing to the CloudCognizant
 
Software Testing in Cloud Platform A Survey_final
Software Testing in Cloud Platform A Survey_finalSoftware Testing in Cloud Platform A Survey_final
Software Testing in Cloud Platform A Survey_finalwww.pixelsolutionbd.com
 
A Year of “Testing” the Cloud for Development and Test
A Year of “Testing” the Cloud for Development and TestA Year of “Testing” the Cloud for Development and Test
A Year of “Testing” the Cloud for Development and TestTechWell
 
Automate across Platform, OS, Technologies with TaaS
Automate across Platform, OS, Technologies with TaaSAutomate across Platform, OS, Technologies with TaaS
Automate across Platform, OS, Technologies with TaaSAnand Bagmar
 
Cloud Testing - A New Age Approach to Testing
Cloud Testing - A New Age Approach to TestingCloud Testing - A New Age Approach to Testing
Cloud Testing - A New Age Approach to TestingSoftware Testing Solution
 
Applying a Comprehensive, Automated Assurance Framework to Validate Cloud Rea...
Applying a Comprehensive, Automated Assurance Framework to Validate Cloud Rea...Applying a Comprehensive, Automated Assurance Framework to Validate Cloud Rea...
Applying a Comprehensive, Automated Assurance Framework to Validate Cloud Rea...Cognizant
 
Testing the Migration of Monolithic Applications to Microservices on the Cloud
Testing the Migration of Monolithic Applications to Microservices on the CloudTesting the Migration of Monolithic Applications to Microservices on the Cloud
Testing the Migration of Monolithic Applications to Microservices on the CloudNagarro
 
Test automation in project management
Test automation in project managementTest automation in project management
Test automation in project managementambreprasad77
 
Complex End-to-End Testing
Complex End-to-End TestingComplex End-to-End Testing
Complex End-to-End TestingErika Barron
 
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the CloudVirtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloudguest2e9c5f40
 
Agile Testing in the Cloud
Agile Testing in the CloudAgile Testing in the Cloud
Agile Testing in the CloudCygnet Infotech
 
Performance Testing Cloud-Based Systems
Performance Testing Cloud-Based SystemsPerformance Testing Cloud-Based Systems
Performance Testing Cloud-Based SystemsTechWell
 
Non-functional Issues in Cloud Based Systems by Kees Blokland and Martin Pol
Non-functional Issues in Cloud Based Systems by Kees Blokland and Martin PolNon-functional Issues in Cloud Based Systems by Kees Blokland and Martin Pol
Non-functional Issues in Cloud Based Systems by Kees Blokland and Martin PolKees Blokland
 
From Relational Database Management to Big Data: Solutions for Data Migration...
From Relational Database Management to Big Data: Solutions for Data Migration...From Relational Database Management to Big Data: Solutions for Data Migration...
From Relational Database Management to Big Data: Solutions for Data Migration...Cognizant
 

What's hot (20)

Cloud testing
Cloud testingCloud testing
Cloud testing
 
Taking Testing to the Cloud
Taking Testing to the CloudTaking Testing to the Cloud
Taking Testing to the Cloud
 
Cloud based testing
Cloud based testingCloud based testing
Cloud based testing
 
Software Testing in Cloud Platform A Survey_final
Software Testing in Cloud Platform A Survey_finalSoftware Testing in Cloud Platform A Survey_final
Software Testing in Cloud Platform A Survey_final
 
Cloud testing v1
Cloud testing v1Cloud testing v1
Cloud testing v1
 
A Year of “Testing” the Cloud for Development and Test
A Year of “Testing” the Cloud for Development and TestA Year of “Testing” the Cloud for Development and Test
A Year of “Testing” the Cloud for Development and Test
 
Automate across Platform, OS, Technologies with TaaS
Automate across Platform, OS, Technologies with TaaSAutomate across Platform, OS, Technologies with TaaS
Automate across Platform, OS, Technologies with TaaS
 
Cloud Testing
Cloud TestingCloud Testing
Cloud Testing
 
Testing a SaaS Platform
Testing a SaaS PlatformTesting a SaaS Platform
Testing a SaaS Platform
 
Cloud Testing - A New Age Approach to Testing
Cloud Testing - A New Age Approach to TestingCloud Testing - A New Age Approach to Testing
Cloud Testing - A New Age Approach to Testing
 
Applying a Comprehensive, Automated Assurance Framework to Validate Cloud Rea...
Applying a Comprehensive, Automated Assurance Framework to Validate Cloud Rea...Applying a Comprehensive, Automated Assurance Framework to Validate Cloud Rea...
Applying a Comprehensive, Automated Assurance Framework to Validate Cloud Rea...
 
Testing the Migration of Monolithic Applications to Microservices on the Cloud
Testing the Migration of Monolithic Applications to Microservices on the CloudTesting the Migration of Monolithic Applications to Microservices on the Cloud
Testing the Migration of Monolithic Applications to Microservices on the Cloud
 
Test automation in project management
Test automation in project managementTest automation in project management
Test automation in project management
 
Complex End-to-End Testing
Complex End-to-End TestingComplex End-to-End Testing
Complex End-to-End Testing
 
Cloud testing
Cloud testingCloud testing
Cloud testing
 
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the CloudVirtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
 
Agile Testing in the Cloud
Agile Testing in the CloudAgile Testing in the Cloud
Agile Testing in the Cloud
 
Performance Testing Cloud-Based Systems
Performance Testing Cloud-Based SystemsPerformance Testing Cloud-Based Systems
Performance Testing Cloud-Based Systems
 
Non-functional Issues in Cloud Based Systems by Kees Blokland and Martin Pol
Non-functional Issues in Cloud Based Systems by Kees Blokland and Martin PolNon-functional Issues in Cloud Based Systems by Kees Blokland and Martin Pol
Non-functional Issues in Cloud Based Systems by Kees Blokland and Martin Pol
 
From Relational Database Management to Big Data: Solutions for Data Migration...
From Relational Database Management to Big Data: Solutions for Data Migration...From Relational Database Management to Big Data: Solutions for Data Migration...
From Relational Database Management to Big Data: Solutions for Data Migration...
 

Similar to Cloud testing: challenges and opportunities, TaaS, Integration Testing

Service Virtualization: Delivering Complex Test Environments on Demand
Service Virtualization: Delivering Complex Test Environments on DemandService Virtualization: Delivering Complex Test Environments on Demand
Service Virtualization: Delivering Complex Test Environments on DemandErika Barron
 
On-Demand Webinar: Software Virtualization Lifecycle
On-Demand Webinar: Software Virtualization LifecycleOn-Demand Webinar: Software Virtualization Lifecycle
On-Demand Webinar: Software Virtualization LifecycleSkytap Cloud
 
Modernizing Testing as Apps Re-Architect
Modernizing Testing as Apps Re-ArchitectModernizing Testing as Apps Re-Architect
Modernizing Testing as Apps Re-ArchitectDevOps.com
 
Cloud Performance Testing with LoadRunner
Cloud Performance Testing with LoadRunnerCloud Performance Testing with LoadRunner
Cloud Performance Testing with LoadRunnerRichard Bishop
 
15-factor-apps.pdf
15-factor-apps.pdf15-factor-apps.pdf
15-factor-apps.pdfNilesh Gule
 
Service Virtualization: What Testers Need to Know
Service Virtualization: What Testers Need to KnowService Virtualization: What Testers Need to Know
Service Virtualization: What Testers Need to KnowTechWell
 
CSC AWS re:Invent Enterprise DevOps session
CSC AWS re:Invent Enterprise DevOps sessionCSC AWS re:Invent Enterprise DevOps session
CSC AWS re:Invent Enterprise DevOps sessionTom Laszewski
 
Gomez Blazing Fast Cloud Best Practices
Gomez Blazing Fast Cloud Best Practices Gomez Blazing Fast Cloud Best Practices
Gomez Blazing Fast Cloud Best Practices Compuware APM
 
Breaking the Monolith
Breaking the MonolithBreaking the Monolith
Breaking the MonolithVMware Tanzu
 
Creating Complete Test Environments in the Cloud: Skytap & Parasoft Webinar
Creating Complete Test Environments in the Cloud: Skytap & Parasoft WebinarCreating Complete Test Environments in the Cloud: Skytap & Parasoft Webinar
Creating Complete Test Environments in the Cloud: Skytap & Parasoft WebinarSkytap Cloud
 
How Real Device Cloud Testing Ensures Exceptional Efficiency and Scalability ...
How Real Device Cloud Testing Ensures Exceptional Efficiency and Scalability ...How Real Device Cloud Testing Ensures Exceptional Efficiency and Scalability ...
How Real Device Cloud Testing Ensures Exceptional Efficiency and Scalability ...kalichargn70th171
 
How Real Device Cloud Testing Ensures Exceptional Efficiency and Scalability ...
How Real Device Cloud Testing Ensures Exceptional Efficiency and Scalability ...How Real Device Cloud Testing Ensures Exceptional Efficiency and Scalability ...
How Real Device Cloud Testing Ensures Exceptional Efficiency and Scalability ...kalichargn70th171
 
Best Practices In Load And Stress Testing Cmg Seminar[1]
Best Practices In Load And Stress Testing Cmg Seminar[1]Best Practices In Load And Stress Testing Cmg Seminar[1]
Best Practices In Load And Stress Testing Cmg Seminar[1]Munirathnam Naidu
 
1. introduction to_cloud_services_architecture
1. introduction to_cloud_services_architecture1. introduction to_cloud_services_architecture
1. introduction to_cloud_services_architectureCloud Genius
 
A Year of Testing in the Cloud: Lessons Learned
A Year of Testing in the Cloud: Lessons LearnedA Year of Testing in the Cloud: Lessons Learned
A Year of Testing in the Cloud: Lessons LearnedTechWell
 
Cloud Testing : An Overview
Cloud Testing : An OverviewCloud Testing : An Overview
Cloud Testing : An OverviewQA InfoTech
 
Cloud Computing & Business Intelligence
Cloud Computing & Business IntelligenceCloud Computing & Business Intelligence
Cloud Computing & Business IntelligenceSudip Chatterjee
 
Lecture32-Web-based-testing-II.pptx
Lecture32-Web-based-testing-II.pptxLecture32-Web-based-testing-II.pptx
Lecture32-Web-based-testing-II.pptxBalkrishanpatidar
 
APIsecure 2023 - Approaching Multicloud API Security USing Metacloud, David L...
APIsecure 2023 - Approaching Multicloud API Security USing Metacloud, David L...APIsecure 2023 - Approaching Multicloud API Security USing Metacloud, David L...
APIsecure 2023 - Approaching Multicloud API Security USing Metacloud, David L...apidays
 
Brocade vADC Portfolio Overview 2016
Brocade vADC Portfolio Overview 2016Brocade vADC Portfolio Overview 2016
Brocade vADC Portfolio Overview 2016Scott Sims
 

Similar to Cloud testing: challenges and opportunities, TaaS, Integration Testing (20)

Service Virtualization: Delivering Complex Test Environments on Demand
Service Virtualization: Delivering Complex Test Environments on DemandService Virtualization: Delivering Complex Test Environments on Demand
Service Virtualization: Delivering Complex Test Environments on Demand
 
On-Demand Webinar: Software Virtualization Lifecycle
On-Demand Webinar: Software Virtualization LifecycleOn-Demand Webinar: Software Virtualization Lifecycle
On-Demand Webinar: Software Virtualization Lifecycle
 
Modernizing Testing as Apps Re-Architect
Modernizing Testing as Apps Re-ArchitectModernizing Testing as Apps Re-Architect
Modernizing Testing as Apps Re-Architect
 
Cloud Performance Testing with LoadRunner
Cloud Performance Testing with LoadRunnerCloud Performance Testing with LoadRunner
Cloud Performance Testing with LoadRunner
 
15-factor-apps.pdf
15-factor-apps.pdf15-factor-apps.pdf
15-factor-apps.pdf
 
Service Virtualization: What Testers Need to Know
Service Virtualization: What Testers Need to KnowService Virtualization: What Testers Need to Know
Service Virtualization: What Testers Need to Know
 
CSC AWS re:Invent Enterprise DevOps session
CSC AWS re:Invent Enterprise DevOps sessionCSC AWS re:Invent Enterprise DevOps session
CSC AWS re:Invent Enterprise DevOps session
 
Gomez Blazing Fast Cloud Best Practices
Gomez Blazing Fast Cloud Best Practices Gomez Blazing Fast Cloud Best Practices
Gomez Blazing Fast Cloud Best Practices
 
Breaking the Monolith
Breaking the MonolithBreaking the Monolith
Breaking the Monolith
 
Creating Complete Test Environments in the Cloud: Skytap & Parasoft Webinar
Creating Complete Test Environments in the Cloud: Skytap & Parasoft WebinarCreating Complete Test Environments in the Cloud: Skytap & Parasoft Webinar
Creating Complete Test Environments in the Cloud: Skytap & Parasoft Webinar
 
How Real Device Cloud Testing Ensures Exceptional Efficiency and Scalability ...
How Real Device Cloud Testing Ensures Exceptional Efficiency and Scalability ...How Real Device Cloud Testing Ensures Exceptional Efficiency and Scalability ...
How Real Device Cloud Testing Ensures Exceptional Efficiency and Scalability ...
 
How Real Device Cloud Testing Ensures Exceptional Efficiency and Scalability ...
How Real Device Cloud Testing Ensures Exceptional Efficiency and Scalability ...How Real Device Cloud Testing Ensures Exceptional Efficiency and Scalability ...
How Real Device Cloud Testing Ensures Exceptional Efficiency and Scalability ...
 
Best Practices In Load And Stress Testing Cmg Seminar[1]
Best Practices In Load And Stress Testing Cmg Seminar[1]Best Practices In Load And Stress Testing Cmg Seminar[1]
Best Practices In Load And Stress Testing Cmg Seminar[1]
 
1. introduction to_cloud_services_architecture
1. introduction to_cloud_services_architecture1. introduction to_cloud_services_architecture
1. introduction to_cloud_services_architecture
 
A Year of Testing in the Cloud: Lessons Learned
A Year of Testing in the Cloud: Lessons LearnedA Year of Testing in the Cloud: Lessons Learned
A Year of Testing in the Cloud: Lessons Learned
 
Cloud Testing : An Overview
Cloud Testing : An OverviewCloud Testing : An Overview
Cloud Testing : An Overview
 
Cloud Computing & Business Intelligence
Cloud Computing & Business IntelligenceCloud Computing & Business Intelligence
Cloud Computing & Business Intelligence
 
Lecture32-Web-based-testing-II.pptx
Lecture32-Web-based-testing-II.pptxLecture32-Web-based-testing-II.pptx
Lecture32-Web-based-testing-II.pptx
 
APIsecure 2023 - Approaching Multicloud API Security USing Metacloud, David L...
APIsecure 2023 - Approaching Multicloud API Security USing Metacloud, David L...APIsecure 2023 - Approaching Multicloud API Security USing Metacloud, David L...
APIsecure 2023 - Approaching Multicloud API Security USing Metacloud, David L...
 
Brocade vADC Portfolio Overview 2016
Brocade vADC Portfolio Overview 2016Brocade vADC Portfolio Overview 2016
Brocade vADC Portfolio Overview 2016
 

More from Dr Ganesh Iyer

SRE Demystified - 16 - NALSD - Non-Abstract Large System Design
SRE Demystified - 16 - NALSD - Non-Abstract Large System DesignSRE Demystified - 16 - NALSD - Non-Abstract Large System Design
SRE Demystified - 16 - NALSD - Non-Abstract Large System DesignDr Ganesh Iyer
 
SRE Demystified - 14 - SRE Practices overview
SRE Demystified - 14 - SRE Practices overviewSRE Demystified - 14 - SRE Practices overview
SRE Demystified - 14 - SRE Practices overviewDr Ganesh Iyer
 
SRE Demystified - 13 - Docs that matter -2
SRE Demystified - 13 - Docs that matter -2SRE Demystified - 13 - Docs that matter -2
SRE Demystified - 13 - Docs that matter -2Dr Ganesh Iyer
 
SRE Demystified - 12 - Docs that matter -1
SRE Demystified - 12 - Docs that matter -1 SRE Demystified - 12 - Docs that matter -1
SRE Demystified - 12 - Docs that matter -1 Dr Ganesh Iyer
 
SRE Demystified - 01 - SLO SLI and SLA
SRE Demystified - 01 - SLO SLI and SLASRE Demystified - 01 - SLO SLI and SLA
SRE Demystified - 01 - SLO SLI and SLADr Ganesh Iyer
 
SRE Demystified - 11 - Release management-2
SRE Demystified - 11 - Release management-2SRE Demystified - 11 - Release management-2
SRE Demystified - 11 - Release management-2Dr Ganesh Iyer
 
SRE Demystified - 10 - Release management-1
SRE Demystified - 10 - Release management-1SRE Demystified - 10 - Release management-1
SRE Demystified - 10 - Release management-1Dr Ganesh Iyer
 
SRE Demystified - 09 - Simplicity
SRE Demystified - 09 - SimplicitySRE Demystified - 09 - Simplicity
SRE Demystified - 09 - SimplicityDr Ganesh Iyer
 
SRE Demystified - 07 - Practical Alerting
SRE Demystified - 07 - Practical AlertingSRE Demystified - 07 - Practical Alerting
SRE Demystified - 07 - Practical AlertingDr Ganesh Iyer
 
SRE Demystified - 06 - Distributed Monitoring
SRE Demystified - 06 - Distributed MonitoringSRE Demystified - 06 - Distributed Monitoring
SRE Demystified - 06 - Distributed MonitoringDr Ganesh Iyer
 
SRE Demystified - 05 - Toil Elimination
SRE Demystified - 05 - Toil EliminationSRE Demystified - 05 - Toil Elimination
SRE Demystified - 05 - Toil EliminationDr Ganesh Iyer
 
SRE Demystified - 04 - Engagement Model
SRE Demystified - 04 - Engagement ModelSRE Demystified - 04 - Engagement Model
SRE Demystified - 04 - Engagement ModelDr Ganesh Iyer
 
SRE Demystified - 03 - Choosing SLIs and SLOs
SRE Demystified - 03 - Choosing SLIs and SLOsSRE Demystified - 03 - Choosing SLIs and SLOs
SRE Demystified - 03 - Choosing SLIs and SLOsDr Ganesh Iyer
 
Machine Learning for Statisticians - Introduction
Machine Learning for Statisticians - IntroductionMachine Learning for Statisticians - Introduction
Machine Learning for Statisticians - IntroductionDr Ganesh Iyer
 
Making Decisions - A Game Theoretic approach
Making Decisions - A Game Theoretic approachMaking Decisions - A Game Theoretic approach
Making Decisions - A Game Theoretic approachDr Ganesh Iyer
 
Game Theory and Engineering Applications
Game Theory and Engineering ApplicationsGame Theory and Engineering Applications
Game Theory and Engineering ApplicationsDr Ganesh Iyer
 
Machine Learning and its Applications
Machine Learning and its ApplicationsMachine Learning and its Applications
Machine Learning and its ApplicationsDr Ganesh Iyer
 
How to become a successful entrepreneur
How to become a successful entrepreneurHow to become a successful entrepreneur
How to become a successful entrepreneurDr Ganesh Iyer
 
Dockers and kubernetes
Dockers and kubernetesDockers and kubernetes
Dockers and kubernetesDr Ganesh Iyer
 

More from Dr Ganesh Iyer (20)

SRE Demystified - 16 - NALSD - Non-Abstract Large System Design
SRE Demystified - 16 - NALSD - Non-Abstract Large System DesignSRE Demystified - 16 - NALSD - Non-Abstract Large System Design
SRE Demystified - 16 - NALSD - Non-Abstract Large System Design
 
SRE Demystified - 14 - SRE Practices overview
SRE Demystified - 14 - SRE Practices overviewSRE Demystified - 14 - SRE Practices overview
SRE Demystified - 14 - SRE Practices overview
 
SRE Demystified - 13 - Docs that matter -2
SRE Demystified - 13 - Docs that matter -2SRE Demystified - 13 - Docs that matter -2
SRE Demystified - 13 - Docs that matter -2
 
SRE Demystified - 12 - Docs that matter -1
SRE Demystified - 12 - Docs that matter -1 SRE Demystified - 12 - Docs that matter -1
SRE Demystified - 12 - Docs that matter -1
 
SRE Demystified - 01 - SLO SLI and SLA
SRE Demystified - 01 - SLO SLI and SLASRE Demystified - 01 - SLO SLI and SLA
SRE Demystified - 01 - SLO SLI and SLA
 
SRE Demystified - 11 - Release management-2
SRE Demystified - 11 - Release management-2SRE Demystified - 11 - Release management-2
SRE Demystified - 11 - Release management-2
 
SRE Demystified - 10 - Release management-1
SRE Demystified - 10 - Release management-1SRE Demystified - 10 - Release management-1
SRE Demystified - 10 - Release management-1
 
SRE Demystified - 09 - Simplicity
SRE Demystified - 09 - SimplicitySRE Demystified - 09 - Simplicity
SRE Demystified - 09 - Simplicity
 
SRE Demystified - 07 - Practical Alerting
SRE Demystified - 07 - Practical AlertingSRE Demystified - 07 - Practical Alerting
SRE Demystified - 07 - Practical Alerting
 
SRE Demystified - 06 - Distributed Monitoring
SRE Demystified - 06 - Distributed MonitoringSRE Demystified - 06 - Distributed Monitoring
SRE Demystified - 06 - Distributed Monitoring
 
SRE Demystified - 05 - Toil Elimination
SRE Demystified - 05 - Toil EliminationSRE Demystified - 05 - Toil Elimination
SRE Demystified - 05 - Toil Elimination
 
SRE Demystified - 04 - Engagement Model
SRE Demystified - 04 - Engagement ModelSRE Demystified - 04 - Engagement Model
SRE Demystified - 04 - Engagement Model
 
SRE Demystified - 03 - Choosing SLIs and SLOs
SRE Demystified - 03 - Choosing SLIs and SLOsSRE Demystified - 03 - Choosing SLIs and SLOs
SRE Demystified - 03 - Choosing SLIs and SLOs
 
Machine Learning for Statisticians - Introduction
Machine Learning for Statisticians - IntroductionMachine Learning for Statisticians - Introduction
Machine Learning for Statisticians - Introduction
 
Making Decisions - A Game Theoretic approach
Making Decisions - A Game Theoretic approachMaking Decisions - A Game Theoretic approach
Making Decisions - A Game Theoretic approach
 
Cloud and Industry4.0
Cloud and Industry4.0Cloud and Industry4.0
Cloud and Industry4.0
 
Game Theory and Engineering Applications
Game Theory and Engineering ApplicationsGame Theory and Engineering Applications
Game Theory and Engineering Applications
 
Machine Learning and its Applications
Machine Learning and its ApplicationsMachine Learning and its Applications
Machine Learning and its Applications
 
How to become a successful entrepreneur
How to become a successful entrepreneurHow to become a successful entrepreneur
How to become a successful entrepreneur
 
Dockers and kubernetes
Dockers and kubernetesDockers and kubernetes
Dockers and kubernetes
 

Recently uploaded

NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
Introduction to Quantum Computing
Introduction to Quantum ComputingIntroduction to Quantum Computing
Introduction to Quantum ComputingGDSC PJATK
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataSafe Software
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
Things you didn't know you can use in your Salesforce
Things you didn't know you can use in your SalesforceThings you didn't know you can use in your Salesforce
Things you didn't know you can use in your SalesforceMartin Humpolec
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.francesco barbera
 
Spring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfSpring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfAnna Loughnan Colquhoun
 

Recently uploaded (20)

NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
Introduction to Quantum Computing
Introduction to Quantum ComputingIntroduction to Quantum Computing
Introduction to Quantum Computing
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
Things you didn't know you can use in your Salesforce
Things you didn't know you can use in your SalesforceThings you didn't know you can use in your Salesforce
Things you didn't know you can use in your Salesforce
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.
 
Spring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfSpring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdf
 

Cloud testing: challenges and opportunities, TaaS, Integration Testing

  • 1. Test challenges and methodologies with Cloud Dr Ganesh Iyer, Progress Software Ref: Ganesh Neelakanta Iyer, Jayakhanna Pasimutu and Ramesh Loganathan "PCTF: An Integrated, Extensible Cloud Test Framework for Testing Cloud Platforms and Applications" The 13th International Conference on Quality Software 2013, (QSIC,'13),
  • 2. Introduction  Cloud Computing and Web 2.0 technologies • Web-driven applications • Driven by browser and do not need any installation • On-demand resource availability • Faster time to market • Reduced capital and operational expenses 2 © 2013 Progress Software Corporation. All rights reserved.
  • 3. Introduction Paradigm shift Seamless upgrades Sharing of resources: Multitenancy Traditionally, applications are designed first, then developed, tested and distributed Traditionally, software upgrade needs to get new version of the software, down the system and upgrade Traditionally, software development & deployment is mostly in dedicated resources 3 But with Cloud, upgrade should happen live with minimal or no down time E.g. Gmail © 2013 Progress Software Corporation. All rights reserved. With Cloud, these resources are often being shared among multiple customers Auto-elasticity: Resources ondemand Performance Disaster recovery SaaS Applications; Browser based configuration and usage New test dimensions Changes in existing dimensions Compatibility Availability and Business Continuity Security: Multitenant penetration, Identity federation management
  • 4. Common Cloud Testing Dimensions Elasticity Testing Security Testing Performance Testing Resource acquisition/ Release Time Traversal vulnerability Compatibility Testing Time to deploy Accessibility testing User access/ Roles Multi-tenancy Time to Genesis Provisionin g on-the-go Load Testing for ELB 4 Identity federation management Communicati on latency over SSL Multi-tenant penetration Connectivity and reliability with 3rd parties Reliability and Availability Latency © 2013 Progress Software Corporation. All rights reserved. Globalization and Localization testing API Integration Testing Live Testing Connectivity and invocation testing Disaster recovery API load testing Live upgrade testing API security Compatibility under different situations Self-healing ability Multitenancy Availability and business continuity Multitenancy Testing Multi-tenant penetration Rigid failure containment Availability and Business continuity Risk of correlated behaviors Service transition activity analysis
  • 5. Elasticity Testing  Based on subscription plans, check the maximum vertical/horizontal limit  Auto provisioning/freeing on-the-go  Testing for Load Balancing  Performance    5 Test for the impact while auto scaling Response time/Release time for provisioning of resources Load Testing of Different subscriptions © 2013 Progress Software Corporation. All rights reserved.
  • 6. Elasticity and Scalability • Limitations on max objects/applications at a time • Number of applications that can be developed per platform instance Load requirements Cloud App Development Platform Load requirements Elasticity Load Testing for different usage scenarios • 100’s of administrators accessing the management Time Time  6 Load requirements Load requirements console Time Unpredictable/Predictable burst: Some tenants have a specific pattern in their usage and test for sudden expected/unexpected variations in the usage © 2013 Progress Software Corporation. All rights reserved. Time
  • 7. Security Testing Implications Outsourced + Insourced External (Data Storage)     Sensitive data management in the Cloud S3 Storage or any other cloud vendor storage DBaaS (OE Database) Third party services to be tested only for connectivity with over services    Functionality testing for our services Single sign on account for all the services Application Security – using both the ProPaaS and third party services Multi-tenant penetration testing Proprietary (API’s)      7  De-perimeterised Traversal vulnerability Communication latency over SSL API Level and Application Security User access/Roles testing    Access from different Clients to the ProPaaS platform testing From VPN, Firewall settings, Antivirus software Authentication/Authorization Identity federation management testing © 2013 Progress Software Corporation. All rights reserved. https://collaboration.opengroup.org/jericho/cloud_cube_model_v1.0.pdf
  • 8. Security User access/roles Web UI Vulnerabilities and Attacks • Authentication and authorization • Identity federation management Single Sign-On • Access from different clients to the platform • VPN, firewall settings, anti-virus • User privileges • SQL Injection • URL Manipulation • Cross-site scripting • Password cracking • Hidden-field manipulation • Multi-tenancy penetration testing • Traversal vulnerability • DDoS attacks Multi-tenant Database • Data management at DB (Encryption security) • Data retention and destruction for DB: Erase and sanitize when space is reallocated 8 Other Security Concerns • Fault-injection-based testing for web-driven services (Including verification for all input fields, network interface, environment variables etc) • Fuzzy testing for web-driven services (Injecting random data into application to determine whether it can run normally under the jumbled input) • Data privacy: Custom SLA capabilities © 2013 Progress Software Corporation. All rights reserved.
  • 9. Performance Testing  Time to deploy  Density (multi-tenancy)  Reliability Testing  Availability Testing  To facilitate Follow the Sun advantage model  Connectivity and reliability with 3rd party components from our Cloud • Reliability of the data sent between two systems 9 © 2013 Progress Software Corporation. All rights reserved.
  • 10. Latency under different conditions Network • The round-trip time between the browser and the server • The number of round-trip times it takes to completely load a web page • The protocol’s flow & congestion control properties, and • Competing traffic, unreliable network Processing Client side 10 • The time it takes to the server to prepare the content that will be sent to the user. • Resource sharing introduces contention risks, increased recovery latencies • The time the web browser needs to prepare the received content to be presented • Latency when accessed from different unreliable sources (e.g. handheld devices, PDAs etc..) © 2013 Progress Software Corporation. All rights reserved.
  • 11. Live Testing and Failover Testing  Robustness of infrastructure against failures  Live Upgrade Testing - Managing customer applications for maintenance/upgrade  Recovery time in case of product/platform crash  Self healing ability in case of product/platform  Availability and Business continuity in multi-tenant environment 11 © 2013 Progress Software Corporation. All rights reserved.
  • 12. Multi-Tenancy Testing  Verifiable resource accounting for Billing • In case, multiple tenant’s data need to be captured simultaneously by the billing agent  Multi-tenancy penetration testing • Tenant level access and their boundaries based on authentication and authorization  Rigid failure containment between tenants • Failure because of one tenant’s action shouldn’t stall other tenant’s activities or bring down the complete system down  Testing for the risk of correlated behaviors • Example: multiple application instances execute the same recovery action or periodic maintenance actions simultaneously  Service transition activity analysis • 12 No service impact on other tenant instances when each and every tenant-specific configuration parameter is changed. © 2013 Progress Software Corporation. All rights reserved.
  • 13. Compatibility Testing Languages Browsers Platforms Devices 13 © 2013 Progress Software Corporation. All rights reserved.
  • 14. PCTF: Progress Cloud Test Framework PCTF Injection strings Test Logger Input Crawler Error parameters Injector patterns Error Pattern Manager Security Testing Test manager Injection strings & Results Pages/hyperlinks Client shell Output Repository • Parameter Test Repository: • Test suite Plug-ins configuration • Libraries: SQL string library, Error • Test suite selection Result Analyzer • Result/Log collection API Integration Testing SLA patterns library etc System Under Test (Cloud Platform/Application) Test DB (MySQL) Synthetic Load Generator • SLA mapping table • SLA metrics database SLA information Stubs for 3rd parties SLA Monitor Testing 14 Ganesh Neelakanta Iyer, Jayakhanna Pasimutu and Ramesh Loganathan "PCTF: An Integrated, Extensible Cloud Test Framework for Testing Cloud Platforms and Applications" The 13th International Conference on Quality Software 2013, (QSIC,'13), © 2013 Progress Software Corporation. All rights reserved.
  • 15. Integration Testing 15 © 2013 Progress Software Corporation. All rights reserved.
  • 16. What is Integration Testing?  Often, many Cloud-based systems will be integrated to each other for delivering a cloud-based offering.  For example, for a SaaS application, the SSO (single-sign-on) mechanism may be handled by a 3rd party system and payment mechanisms might be handled by another 3rd party system such as PayPal.  Hence, we need to get sandbox environments for such payment gateways which are identitical to its production environment in order to have the systems behave same in both test as well as live environments. 16 © 2013 Progress Software Corporation. All rights reserved.
  • 17. Typical issues  Multiple systems in an integrated Cloud-based product behave differently and verification process for different systems differ each other.  Some of the systems do not allow automated deletion of data created for testing. So it imposes a unique requirement to have unique users created every time we perform such test automation  Unpredictable delays in updating various systems.  Different types of environments for testing. For example, presence of web UI testing and runtime testing in one test scenario poses its own unique challenges 17 © 2013 Progress Software Corporation. All rights reserved.
  • 18. Integration test automation challenges  Unique requirements: • UI components • Runtime components  Need to either develop a test framework that can efficiently test both run time and UI components or use a combination of two frameworks for runtime/UI testing  Needs to have mechanisms to initiate the test written in one framework from the other one and to generate a combined test results  Using the exposed APIs provided by other third party systems integrated as part of the product 18 © 2013 Progress Software Corporation. All rights reserved.
  • 19. Integration test automation with Cloud 19 © 2013 Progress Software Corporation. All rights reserved.
  • 20. TaaS 20 © 2013 Progress Software Corporation. All rights reserved.
  • 21. Overview of Testing as a Service TaaS  Wikipedia - Testing as a Service (TaaS, typically pronounced 'tass') is a model of software testing whereby a provider undertakes the activity of software testing applications/solutions for customers as a service on demand. …involves the ondemand test execution of well-defined suites of test material, generally on an outsourced basis.  Shared Services delivery model  Pay per by drink and not by Glass  Standardised, Repeatable services 21 © 2013 Progress Software Corporation. All rights reserved. Courtesy: http://tinyurl.com/taasmphasis
  • 22. TaaS: Conceptual Model Customer 1 Customer …n Customer 2 Fixed price per product Commercial Models SLAs Customer Service Management Pre-defined Automation Offering Service Products Move towards standardised services Service Catalogue for business Web Performance Offering Perf .Test Web/ERP Functional Test Offering Customer Interface Activities and deliverables Predefined Manual Testing Demand Management Service Catalogue for Operations Owned by service provider. Continuous Improvement Service n… Service 3 Test Assets Test Assets Service 2 Test Assets Global Delivery Model Testing Framewor k Test Assets TAAS Engine Internal Service Management Owned by service provider 22 HAAS/ Cloud In house Tools / External Tools Test Analyst Test Process High sharing of resources Courtesy: http://tinyurl.com/taasmphasis © 2013 Progress Software Corporation. All rights reserved.
  • 23. Advantages of testing in the Cloud Traditional Testing Testing in Cloud Low asset utilization Scalability: Long time to increase capacity Less time (instantaneous) increase and reduction in capacity Long time to build datacenters Purchased as a service from cloud providers Difficult to manage Better management and increased productivity Duplicate test systems Aggregated system Creates unnecessary wastes 23 Improved asset utilization Cleaner, greener testing, saving in CO2 emissions © 2013 Progress Software Corporation. All rights reserved.
  • 24. Conclusions  Various Cloud test dimensions • Elasticity, Multi-tenancy, Security, Live Upgrade, Performance  Integration Testing: A unique requirement with Cloud  Automation challenges and possible approaches  TaaS 24 © 2013 Progress Software Corporation. All rights reserved.