SaaS Testing Overview
Ram Garg
CEO
Software Performance Assurance (SPA)
Software Performance Assurance Confidential

Outline
 SaaS Introduction
 What is it?
 Why SaaS?
 SaaS Challenges
 Key Challenges
 SaaS Testing
 How to approach SaaS testing?
 How SaaS Testing is different?
 SaaS Test Tools
Software Performance Assurance Confidential

What is SaaS?
SaaS Introduction
Software Performance Assurance Confidential

SaaS: What is it?
Software-as-a-Service (SaaS) is the application
layer of the cloud computing model.
Application Software as a Service (SaaS)
Platform Platform as a Service (PaaS)
Infrastructure Infrastructure as a Service (IaaS)
Software Performance Assurance Confidential

SaaS Definition
“Software as a service (SaaS), sometimes
referred to as "software on demand," is
software that is 100% deployed over the
internet … . With SaaS, a provider licenses
an application to customers as a service
on demand, through a subscription or a
"pay-as-you-go" model.”
– Wikipedia definition
Software Performance Assurance Confidential

Architecture: Three Key Differentiators
Multi-tenant
Efficient
Scalable Configurable
Key
Architecture
Differentiators
Software Performance Assurance Confidential

Why SaaS?
SaaS Introduction
Software Performance Assurance Confidential

The Software Business Model Changes …
non -SaaS SaaS
Software Ownership Customer Provider
Infrastructure
Customer Provider
management Responsibility
Installation Client - Server Central
Upgrades Larger, less frequent Shorter, frequent
Platform Multi-version Single-version
Software Performance Assurance Confidential

Changing the Business Model cont …
non -SaaS SaaS
Configurability
Low High, Critical
Requirement
Multi-tenancy No Yes
Backward compatibility No Yes
Maintenance Customer Provider
Software Performance Assurance Confidential

Changing the Business Model cont …
non -SaaS SaaS
Cost, Allocation High, CapEx Low, OpEx
Close the deal, Provide value,
Sales focus / Success
License Revenue Lack of Churn
Profits Initial sale + maintenance Ongoing Subscription
Software Performance Assurance Confidential

Ecosystem
SaaS (Software)  Starting point to every service within
cloud computing ecosystem
PaaS (Platform)  Highest on the maturity level and
growth curve amongst XaaS
IaaS  Highest revenue generating service
(Infrastructure)
within cloud model for next several
years

SaaS Catalysts
Intrigue of access to unlimited Computing and storage
resources, made available and paid for only as needed (Pay as
you go)
Decreasing cost of bandwidth
Customer frustration with traditional software buying cycle
(license, maintenance contract, upgrades)
Perceived increase in control over the vendor relationship
Early SaaS success stories – Salesforce.com, WebEx, etc.
Demonstrate broader value proposition
You focus on your application and not on infrastructure
Software Performance Assurance Confidential

SaaS Early Adoption
SaaS Introduction
Software Performance Assurance Confidential

SaaS Adoption: Verticals
AMI-Partners„ report -
"Software-as-Service Hot Adoption rate of
Spots” : Education, SaaS in SMB is
Healthcare, and Financial" higher than large
enterprises
All research points out that
In-Stat research on SaaS has been adopted
vertical market adoption
shows the healthcare heavily in :
market is currently most  HR applications (ERP,
satisfied with hosted Salesforce automation,
applications, with eRecruiting)
education identified as an  Online Collaboration (Web
important potential target conferencing)
market also.  CRM
 Healthcare
Software Performance Assurance Confidential

SaaS Adoption Factors
(source: Forrester)
Availability & Reliability (71%) Fast Deployment (45%)
Reduced dependencies on
Flexible consumption (35%)
internal IT (24%)
Simplified pricing model (32%)
Software Performance Assurance Confidential

The SaaS Market is and will
Continue to Grow…
Software Performance Assurance Confidential

Today and tomorrow …
Software Performance Assurance Confidential

Today and tomorrow …
Software Performance Assurance Confidential

Today and tomorrow …
Software Performance Assurance Confidential

Challenges
SaaS Challenges
Software Performance Assurance Confidential

Primary Concerns with the SaaS Model
Security concerns
Concerns over compliance with standards
Concerns over integration with legacy applications
Concerns over reliability and robustness of service
Not comfortable with putting mission-critical application on shared
infrastructure and have access over internet
SaaS can not be configured to meet my corporation‟s all specific
needs
Software Performance Assurance Confidential

Numerous Challenges
Security Performance Availability SLA
Compliance
Data Integrity Privacy with Regulatory
Standards
Lack of
Integration Multi-tenancy
confidence
Software Performance Assurance Confidential

SaaS Testing
Software Performance Assurance Confidential

Why SaaS Testing?
SaaS Testing
Software Performance Assurance Confidential

The average enterprise IT
Why SaaS Testing?
department devotes up to 50
percent of its entire technology
infrastructure to development A delay in website page
and test, with up to 90 percent loading by even one second
of the available test can impact conversions by
infrastructure remaining idle. 7% and customer satisfaction
IBM Research by 16%.
Aberdeen Group
Cloud
Testing
Those using cloud facilities to
run applications, 49.8% said
they were doing so
The average online experimentally or for
business had 8 days of prototyping; 28.6% for non-
downtime a year due to critical business applications
performance issues and 21.6% for business
leading to lost revenues critical applications
Aberdeen Group Evans Data Survey
Software Performance Assurance Confidential

Trend leads to trend …
Overall growth in Cloud Computing (investment,
revenue, ROI, market cap, etc.) is leading to a
marked increase in adoption of SaaS Testing

What is SaaS Testing?
SaaS Testing
Software Performance Assurance Confidential

Classifying SaaS Testing
“Testing The Cloud”
SaaS”
Testing applications which are in
cloud or migrating to cloud.
Testing Test the performance, reliability &
the SaaS security of applications
Cloud /
SaaS “Testing SaaS Using Cloud”
“Testing Using Cloud”
Testing applications using the
Testing cloud-based hardware, software,
Testing infrastructure and computing
resources.
SaaS
Using Tests performed like performance,
Cloud load & stress, security and
compatibility
Software Performance Assurance Confidential

How SaaS Testing is Different
Easier Challenges
• No client-server • Frequent live upgrades
installation
• Too many pages
• No multiplatform support
• Multiple privileges
• No multiple version
upgrades • Immature SaaS-specific
• No backward compatibility testing tools
• Access partitions critical
Software Performance Assurance Confidential

Approach
SaaS Testing
Software Performance Assurance Confidential

SaaS Testing Preparation
Prepare
Assess Understand Prepare
Test Plan Prepare
Testing Usage Test
and Test Test Data
Needs Patterns Environment
Suite
Software Performance Assurance Confidential

SaaS Testing Areas
Infrastructure
Testing
Network Application
testing testing
Secured
SaaS
Application
Ensure SaaS testing does not damage
the shared infrastructure
Software Performance Assurance Confidential

SaaS Functional Testing
• Perform Manual and Exploratory test (similar to traditional
applications)
• Check Browser compatibility
• Perform regression
• SaaS regression must cover significant more functionality as single
defect will impact all the customers
• Functional and regression test automation.
• This is important to comprehensive testing cost-effectively as above
point indicates that, in case of SaaS, regression should cover more
functionality.
• Test SaaS on cloud-based staging environment
• Testing on in-house environment and then directly deploying on cloud-
based production environment may result in chaos
• Test in a multi-user environment where different users are
performing different activities.
Software Performance Assurance Confidential

SaaS Security Testing
• Test application security
• Cross Site scripting, SQL Injection, HTTP header injection
• Test network security
• Secured communication, Backup and storage policies, Controlled
access to sites and servers
• Test for possible attacks
• Test roles and privileges in multi-tenant environment
• Many roles and privilege levels
• Test all the upgrades to detect broken security,
Ensure security patches are not missed
• Test data security / integrity among multiple tenants
Software Performance Assurance Confidential

SaaS Security Testing cont …
• Identify scenarios where application might be
vulnerable
• Test PCI compliance
• Test cookie setting, SSL configuration, information
leak, etc.
• Log errors, warnings and suspicious requests
• Tests tampering with the licensing mechanism;
gaining unauthorized access to application data,
editing registry entities or configuration files
Software Performance Assurance Confidential

SaaS Performance Testing
• Test performance extensively first time and for each upgrade
• SaaS is revenue generating and business impacting applications.
Single performance issue will impact all your customers
• Test performance extensively as SaaS need to handle large number
of users and we don‟t have luxury of re-booting or going down every
once in a while
• Test load characteristics, ramp up rate, protocols, load
origination,
• Establish ability to test performance of your SaaS isolated
fashion so that network related issues can be identified and
passed to network infrastructure and internet providers
• Test key metrics and SLAs
• Test performance for mobile users
Software Performance Assurance Confidential

In-Service Upgrade Testing
• Ability to upgrade (full release, patches) system with zero-
to-minimum down time is essential
• Global users make this complicated as somewhere in the
world someone might be accessing your application
• Thoroughly test in staging environment before perform
upgrade ass this is one of the most complex features
Remote Accessibility and Usage
• Global access with acceptable response time
• 24 hours availability
Software Performance Assurance Confidential

Disaster Recovery
• Shutdown to bring up application quickly: server configuration;
install and deploy software ; restore data
• Roll back to last known stable data
Localization
• Make sure it supports international characters as SaaS
application can be accessed from anywhere in the world
• Optimize by supporting target market language
Software Performance Assurance Confidential

Test Tools
SaaS Testing
Software Performance Assurance Confidential

Traditional and Cloud-based Test Tools
Cloud-based Test
Traditional Test Tools
Tools
QTP, WinRunner
Rational Robot
Rational Functional Tester
Functional
SilkTest
Gomez
PushToTest,
Selenium, STAFF, Fitness, Watir
SOASTA
CloudTesting
LoadRunner, Rational Robot, uTest
Rational Performance Tester Many others
Silk Performer
Performance
Jmeter, Grinder, WebLoad,
OpenSTA
Software Performance Assurance Confidential

Traditional Vs. Cloud-based Tools
Testing SaaS Using Testing SaaS Using
Traditional Tools Cloud-based Tools
Many providers who provide A few providers who provide
SaaS testing using traditional SaaS testing using their own
testing tools cloud-based test tools
Many open source matured Limited open source test tools
tools options
Many options for professional Captive professional service by
services by independent test tools provider which is not
service providers even core for their business
Professional service options Captive profession service is
allow cheaper cost sold at very high rate
Software Performance Assurance Confidential

Conclusion
Software Performance Assurance Confidential

Conclusion
Customers buy each month – keep them happy
SaaS is only a delivering model, customers will not compromise on
functionality, stability, security, and performance. They will not switch to
SaaS it if not better than traditional systems.
Save time on support, maintenance, and upgrade, etc.
You need to pay more attention to stuff like accessibility, security,
deployments. The way you earn reputation
Don‟t buy cloud-based test solution just because you are testing SaaS. In
most cases, traditional test solutions work just fine and save your 30% –
50% cost.
Software Performance Assurance Confidential