More Related Content


Recently uploaded(20)


SaaS Testing Overview - Foundation

  1. SaaS Testing Overview Ram Garg CEO Software Performance Assurance (SPA) Software Performance Assurance Confidential
  2. Outline  SaaS Introduction  What is it?  Why SaaS?  SaaS Challenges  Key Challenges  SaaS Testing  How to approach SaaS testing?  How SaaS Testing is different?  SaaS Test Tools Software Performance Assurance Confidential
  3. What is SaaS? SaaS Introduction Software Performance Assurance Confidential
  4. SaaS: What is it? Software-as-a-Service (SaaS) is the application layer of the cloud computing model. Application Software as a Service (SaaS) Platform Platform as a Service (PaaS) Infrastructure Infrastructure as a Service (IaaS) Software Performance Assurance Confidential
  5. SaaS Definition “Software as a service (SaaS), sometimes referred to as "software on demand," is software that is 100% deployed over the internet … . With SaaS, a provider licenses an application to customers as a service on demand, through a subscription or a "pay-as-you-go" model.” – Wikipedia definition Software Performance Assurance Confidential
  6. Architecture: Three Key Differentiators Multi-tenant Efficient Scalable Configurable Key Architecture Differentiators Software Performance Assurance Confidential
  7. Why SaaS? SaaS Introduction Software Performance Assurance Confidential
  8. The Software Business Model Changes … non -SaaS SaaS Software Ownership Customer Provider Infrastructure Customer Provider management Responsibility Installation Client - Server Central Upgrades Larger, less frequent Shorter, frequent Platform Multi-version Single-version Software Performance Assurance Confidential
  9. Changing the Business Model cont … non -SaaS SaaS Configurability Low High, Critical Requirement Multi-tenancy No Yes Backward compatibility No Yes Maintenance Customer Provider Software Performance Assurance Confidential
  10. Changing the Business Model cont … non -SaaS SaaS Cost, Allocation High, CapEx Low, OpEx Close the deal, Provide value, Sales focus / Success License Revenue Lack of Churn Profits Initial sale + maintenance Ongoing Subscription Software Performance Assurance Confidential
  11. Ecosystem SaaS (Software)  Starting point to every service within cloud computing ecosystem PaaS (Platform)  Highest on the maturity level and growth curve amongst XaaS IaaS  Highest revenue generating service (Infrastructure) within cloud model for next several years
  12. SaaS Catalysts Intrigue of access to unlimited Computing and storage resources, made available and paid for only as needed (Pay as you go) Decreasing cost of bandwidth Customer frustration with traditional software buying cycle (license, maintenance contract, upgrades) Perceived increase in control over the vendor relationship Early SaaS success stories –, WebEx, etc. Demonstrate broader value proposition You focus on your application and not on infrastructure Software Performance Assurance Confidential
  13. SaaS Early Adoption SaaS Introduction Software Performance Assurance Confidential
  14. SaaS Adoption: Verticals AMI-Partners„ report - "Software-as-Service Hot Adoption rate of Spots” : Education, SaaS in SMB is Healthcare, and Financial" higher than large enterprises All research points out that In-Stat research on SaaS has been adopted vertical market adoption shows the healthcare heavily in : market is currently most  HR applications (ERP, satisfied with hosted Salesforce automation, applications, with eRecruiting) education identified as an  Online Collaboration (Web important potential target conferencing) market also.  CRM  Healthcare Software Performance Assurance Confidential
  15. SaaS Adoption Factors (source: Forrester) Availability & Reliability (71%) Fast Deployment (45%) Reduced dependencies on Flexible consumption (35%) internal IT (24%) Simplified pricing model (32%) Software Performance Assurance Confidential
  16. The SaaS Market is and will Continue to Grow… Software Performance Assurance Confidential
  17. Today and tomorrow … Software Performance Assurance Confidential
  18. Today and tomorrow … Software Performance Assurance Confidential
  19. Today and tomorrow … Software Performance Assurance Confidential
  20. Challenges SaaS Challenges Software Performance Assurance Confidential
  21. Primary Concerns with the SaaS Model Security concerns Concerns over compliance with standards Concerns over integration with legacy applications Concerns over reliability and robustness of service Not comfortable with putting mission-critical application on shared infrastructure and have access over internet SaaS can not be configured to meet my corporation‟s all specific needs Software Performance Assurance Confidential
  22. Numerous Challenges Security Performance Availability SLA Compliance Data Integrity Privacy with Regulatory Standards Lack of Integration Multi-tenancy confidence Software Performance Assurance Confidential
  23. SaaS Testing Software Performance Assurance Confidential
  24. Why SaaS Testing? SaaS Testing Software Performance Assurance Confidential
  25. The average enterprise IT Why SaaS Testing? department devotes up to 50 percent of its entire technology infrastructure to development A delay in website page and test, with up to 90 percent loading by even one second of the available test can impact conversions by infrastructure remaining idle. 7% and customer satisfaction IBM Research by 16%. Aberdeen Group Cloud Testing Those using cloud facilities to run applications, 49.8% said they were doing so The average online experimentally or for business had 8 days of prototyping; 28.6% for non- downtime a year due to critical business applications performance issues and 21.6% for business leading to lost revenues critical applications Aberdeen Group Evans Data Survey Software Performance Assurance Confidential
  26. Trend leads to trend … Overall growth in Cloud Computing (investment, revenue, ROI, market cap, etc.) is leading to a marked increase in adoption of SaaS Testing
  27. What is SaaS Testing? SaaS Testing Software Performance Assurance Confidential
  28. Classifying SaaS Testing “Testing The Cloud” SaaS” Testing applications which are in cloud or migrating to cloud. Testing Test the performance, reliability & the SaaS security of applications Cloud / SaaS “Testing SaaS Using Cloud” “Testing Using Cloud” Testing applications using the Testing cloud-based hardware, software, Testing infrastructure and computing resources. SaaS Using Tests performed like performance, Cloud load & stress, security and compatibility Software Performance Assurance Confidential
  29. How SaaS Testing is Different Easier Challenges • No client-server • Frequent live upgrades installation • Too many pages • No multiplatform support • Multiple privileges • No multiple version upgrades • Immature SaaS-specific • No backward compatibility testing tools • Access partitions critical Software Performance Assurance Confidential
  30. Approach SaaS Testing Software Performance Assurance Confidential
  31. SaaS Testing Preparation Prepare Assess Understand Prepare Test Plan Prepare Testing Usage Test and Test Test Data Needs Patterns Environment Suite Software Performance Assurance Confidential
  32. SaaS Testing Areas Infrastructure Testing Network Application testing testing Secured SaaS Application Ensure SaaS testing does not damage the shared infrastructure Software Performance Assurance Confidential
  33. SaaS Functional Testing • Perform Manual and Exploratory test (similar to traditional applications) • Check Browser compatibility • Perform regression • SaaS regression must cover significant more functionality as single defect will impact all the customers • Functional and regression test automation. • This is important to comprehensive testing cost-effectively as above point indicates that, in case of SaaS, regression should cover more functionality. • Test SaaS on cloud-based staging environment • Testing on in-house environment and then directly deploying on cloud- based production environment may result in chaos • Test in a multi-user environment where different users are performing different activities. Software Performance Assurance Confidential
  34. SaaS Security Testing • Test application security • Cross Site scripting, SQL Injection, HTTP header injection • Test network security • Secured communication, Backup and storage policies, Controlled access to sites and servers • Test for possible attacks • Test roles and privileges in multi-tenant environment • Many roles and privilege levels • Test all the upgrades to detect broken security, Ensure security patches are not missed • Test data security / integrity among multiple tenants Software Performance Assurance Confidential
  35. SaaS Security Testing cont … • Identify scenarios where application might be vulnerable • Test PCI compliance • Test cookie setting, SSL configuration, information leak, etc. • Log errors, warnings and suspicious requests • Tests tampering with the licensing mechanism; gaining unauthorized access to application data, editing registry entities or configuration files Software Performance Assurance Confidential
  36. SaaS Performance Testing • Test performance extensively first time and for each upgrade • SaaS is revenue generating and business impacting applications. Single performance issue will impact all your customers • Test performance extensively as SaaS need to handle large number of users and we don‟t have luxury of re-booting or going down every once in a while • Test load characteristics, ramp up rate, protocols, load origination, • Establish ability to test performance of your SaaS isolated fashion so that network related issues can be identified and passed to network infrastructure and internet providers • Test key metrics and SLAs • Test performance for mobile users Software Performance Assurance Confidential
  37. In-Service Upgrade Testing • Ability to upgrade (full release, patches) system with zero- to-minimum down time is essential • Global users make this complicated as somewhere in the world someone might be accessing your application • Thoroughly test in staging environment before perform upgrade ass this is one of the most complex features Remote Accessibility and Usage • Global access with acceptable response time • 24 hours availability Software Performance Assurance Confidential
  38. Disaster Recovery • Shutdown to bring up application quickly: server configuration; install and deploy software ; restore data • Roll back to last known stable data Localization • Make sure it supports international characters as SaaS application can be accessed from anywhere in the world • Optimize by supporting target market language Software Performance Assurance Confidential
  39. Test Tools SaaS Testing Software Performance Assurance Confidential
  40. Traditional and Cloud-based Test Tools Cloud-based Test Traditional Test Tools Tools QTP, WinRunner Rational Robot Rational Functional Tester Functional SilkTest Gomez PushToTest, Selenium, STAFF, Fitness, Watir SOASTA CloudTesting LoadRunner, Rational Robot, uTest Rational Performance Tester Many others Silk Performer Performance Jmeter, Grinder, WebLoad, OpenSTA Software Performance Assurance Confidential
  41. Traditional Vs. Cloud-based Tools Testing SaaS Using Testing SaaS Using Traditional Tools Cloud-based Tools Many providers who provide A few providers who provide SaaS testing using traditional SaaS testing using their own testing tools cloud-based test tools Many open source matured Limited open source test tools tools options Many options for professional Captive professional service by services by independent test tools provider which is not service providers even core for their business Professional service options Captive profession service is allow cheaper cost sold at very high rate Software Performance Assurance Confidential
  42. Conclusion Software Performance Assurance Confidential
  43. Conclusion Customers buy each month – keep them happy SaaS is only a delivering model, customers will not compromise on functionality, stability, security, and performance. They will not switch to SaaS it if not better than traditional systems. Save time on support, maintenance, and upgrade, etc. You need to pay more attention to stuff like accessibility, security, deployments. The way you earn reputation Don‟t buy cloud-based test solution just because you are testing SaaS. In most cases, traditional test solutions work just fine and save your 30% – 50% cost. Software Performance Assurance Confidential