TAAS is a model of software testing whereby TRASYS undertakes the activity of testing applications & solutions for our customers as a service with following traits:
On-demand services
service delivery function & service governance
Well-defined & repeatable services (service catalogue, RACI, entry & exit criteria, )
Outsouring using a shared service centre
Pay per use (based on test effort estimation techniques)
Well defined test process & methodology
Professional / certified test resources
Supported by (cloud based and virtualised) test tools & test environments
3. Testing As A Service: Definition
TAAS is a model of software testing whereby TRASYS undertakes
the activity of testing applications & solutions for our customers as a
service with following traits:
On-demand services
service delivery function & service governance
Well-defined & repeatable services (service catalogue, RACI, entry & exit
criteria, )
Outsouring using a shared service centre
Pay per use (based on test effort estimation techniques)
Well defined test process & methodology
Professional / certified test resources
Supported by (cloud based and virtualised) test tools & test environments
4. Testing as a Service: Key components
Managed Services
With KPIs and SLAs
Centralised
Resource allocation
&
global X-shore
capability
Defined list of services
and test activities
Remote, global
and secure
Acces to AUT and
test environments
Test tools
as a service
(SAAS Cloud)
Certified and
professional testers
Unit price for
each service
(test case, use case ,
function point, LOC, etc)
Test automation
and simulators
enhance
productivity
On demand
and virtualised
test environments
Global Network of Test
Centres Of Excellence
Testing as a
Service
5. Testing As A Service: Key Benefits
Testing As a Service
Benefits
Reduce TCO - cost of quality
(between 25% and 35%)
Optimize governance &
streamline communication
Access to ‘niche’ expertise Predict service cost
(fixed price)
Shift from CAPEX to OPEX
(pay for what you use)
Schedule acceleration & flexibility
(Rapid provisioning – shared HR and IT
resources)
Predict service outcome
(result commitment – SLA’s – defined services)
Reduce risk Levels & improve
product & service quality
7. Testing as a Service: Our assets
ISO 9001:2008 certified Test Centre of Excellence
8 Western European locations and Ukraine
ISTQB, Prince2 and Agile certified testers
80% certified
Resource pool > 100
Majority of testers are senior or expert level
( between 5 and 10 years experience)
Test tools & test environments offered as a service
Secure remote access
Cloud based
Virtualized and on demand
See annex 1 for more details
8. Testing as a Service: Service Delivery Model
Resource
allocation
Mgr.
Release
Test Centre Of Excellence
SAAS test mgt.
& defect mgt.
tools
Virtualised On Demand
test environments
Service catalogue
Test Mangement
& goveranance
Nearshore team
TRASYS Service Delivery Manager
Testing best practices &
metholodogy
Client Service Manager
Fixed Price
Service
Contract
SLAs and
metrics
A
Projects AApplications
Simulators &
test data mgt.
tools
On Demand Request
Service Delivery
Offshore team
Releases
On site team
9. Testing as a Service: Our vision
• Business & technology risks define test
strategy
• Traceability matrix
• Requirements coverage analysis
Risk & Requirements Based
• Test management & defect management
• Test automation, test data analysis & mgt.
• Performance testing & profiling
• Security , penetration & vulnerability
• Code review and inspection
Tool supported
• Focus on (important) business processes
• Black box testers need to master the business
• TCO & ROI drive test strategy & master plan
Business driven
• Based on best practices:
ISTQB , ISO 9126, ISO 29119
• Test design techniques for test cases & test data
• Test effort estimation techniques
• (Master) test plan & formal test strategy
Structured
Test strategy, tools & checklists (= framework) for:
• Mobile applications
• SAP, CRM & ERP
• Cloud SAAS
• Webservices & SOA
• Mainframe <> J2EE <> RIA & HTML5 <> …
Technology specific
• Test activities should be embedded in the SDLC
• Testing starts at inception of a project / release
• Testing should be adapted to the lifecycle:
Agile, iterative, V-model
Early & Adapted to lifecycle
11. Testing as a Service: Service Catalogue
Data driven I.M
test automation
Code review &
requirements review
Cloud testing
Keyword driven test
automation
Portability testing
Mobile application
testing
Agile test driven Security & reliability
Big data - DWH and
BI testing
Model based
test automation
Performance testing
& profiling
CRM and ERP
Test automation Non functional testing
Functional & business
oriented
Test solutions
Integration and SOA
testing
E2E business
process testing
Usability & Crowd
testing
Functional system
testing
13. Testing as a Service: Test solutions
Data driven I.M
test automation
Code review &
requirements review
Keyword driven test
automation
Portability testing
Cloud testing
Agile test driven Security & reliability
Mobile application
testing
Model based
test automation
Performance testing
& profiling
Test automation Non functional testing
CRM and ERP
Functional & business
oriented
Test solutions
Integration and SOA
testing
E2E business
process testing
Usability & Crowd
testing
Functional system
testing
Big data - DWH and
BI testing
14. Testing as a Service: CRM & ERP
Testing of a ‘commercial off-the-shelf’ database software application(s) supporting
customer relationship management and enterprise resource planning business
processes.
Some leading providers of CRM / ERP
Definition
• Succesfull selection and Integration of the ‘package’ leading to ‘acceptance’
• (Test) Data accuracy, data quality and security during storage and retrieval
• Understand the (impacted) business process(es)
• Understand the test object ( platform) , the changes and their impact
Key
challenges
• SAP solution manager, SAP TAO, SAP ECATT, SAP TDMS
• Predefined test (automation) libraries & accelerators for SAP, Peoplesoft & Oracle (order to
cash, procude to pay, etc)
• HP QTP
TRASYS
tools
• (automated) Data migration & platform migration testing
• Customisation, parameterisation and usability testing
• Assistance during evaluation / testing of prototypes - packages
• (automated) version upgrade (regression) testing
• Installability, configuration & deployment testing
• E2E business process testing & system integration testing
• (automated & data driven) functional acceptance testing and testing of reports
• Application Lifecycle Test Management services ( ALM)
TRASYS
Services
15. Testing as a Service: Cloud Testing
Testing conducted on the applications and services provided via cloud computing:
‘test object’ i.e application under test and / or testing services themselves are
provided via cloud computing using PAAS / SAAS and APIs
Defintion
• Performance, scalability, capacity and availability
• Access, privacy & security
• Migration, interoperability & reversibility (avoid “vendor lock in”)
Key
challenges
• HP Loadrunner
• SOASTA Cloud test
• SoapUI
TRASYS
tools
• Load balance and stress testing for a range of simulated usage conditions
• Role based & tool based security testing
• Cloud migration testing (data migration, repository migration)
• E2E system integration testing ( Hybrid cloud, interfaces between Cloud SAAS
and back end systems and databases, API testing, SOA testing)
TRASYS
Services
16. Testing as a Service: Mobile Application Testing
The functional and non functional testing of mobile-based native (or hybrid)
applications.
Some leading mobile app. platform providers: SAP, IBM, Kony, Antenna, Jquery mobile,
Adobe, Appcelrator
Definition
• Understand the native application platform (hardware, OS) and network types
• Testing needs to ascertain that the app can be successfully downloaded to the device,
executed on the device and interact with the supporting backend & infrastructure and
updates can be made (push out)
• Understand the non functional requirements, risks & ‘vulnerabilities’ (security, performance,
usability, portability…) related to mobile application development and (end user) usage
Key
challenges
• Predefined mobile application checklists and ‘standard’ test cases (MAT starters kit)
• Our mobile application test lab contains multiple fysical devices & emulators in order to
support different test objectives. This is also supported by remote cloud based devices .
• We can test multiple networks from our different locations (i.e GPRS tunneling effect)
• Automated mobile test execution using emulators and test automation tools (see annex 1)
TRASYS
tools &
assets
• Risk based multidevice and multi OS testing, including backward compatibility
• App lifecycle mgt testing ( from store readyness to push out of upgrade app versions)
• Functional testing , security testing, performance testing, E2E business process testing,
back end integration testing and usability testing for mobile applications
TRASYS
Services
17. Testing as a Service: Big Data, DWH and BI
Testing of the data warehouse i.e the main organisations ’repository
of transformed and cleaned historical data’ and business intelligence and
decision support applications
Some leading providers:
Defintion
• Availability of high quality (test) data in order to cover requirements (i.e reports or queries
and ETLs)
• High volatility of business rules & requirements in DWH/BI testing
• Sensitivity, complexity & confidentiality of (test) data
Key
challenges
• Tricentis Tosca Testsuite (for DWH testing)
• Analytical simulation engine (accelerator)
• Data driven testing tools and model based testing tools
(See next slide)
TRASYS
tools
• DWH, DI and Big data testing:
• Source data extraction testing
• ETL process validation (including statistical & dimensional analysis and data quality
validation)
• E2E integration testing of ETL batch jobs
• Report validation, Cube validation, dashboard validation
• Non functional testing (performance , security)
• (Test) data mining, combined with ‘synthetic’ test data creation
• Simulation of business rules while processing historical and/or near-to real time transactions
in order to test systems
TRASYS
Services
19. Supporting test services
Testing as a Service: Supporting test services
Data driven I.M
test automation
Code review &
requirements review
Keyword driven test
automation
Portability testing
Cloud testing
Agile test driven Security & reliability
Mobile application
testing
Model based
test automation
Performance testing
& profiling
Test automation Non functional testing
CRM and ERP
Functional & business
oriented
Test solutions
Integration and SOA
testing
E2E business
process testing
Usability & Crowd
testing
Functional system
testing
Big data - DWH and
BI testing
20. Testing as a Service:
Model Based Testing (MBT) & Data Driven Testing (DDT)
Testing based on a model of the component or system under test, e.g., reliability growth models, usage
models such as operational profiles or behavioral models such as decision table or state transition
diagram.Test cases are derived from a model such as finite state machine , UML, state chart, Markov
chain that describes (mostly functional) aspects of the application under test
-> Model based testing is supported by data driven test automation (DDT)
-> Model based testing is a way of simulation early in the SDLC
Defintion
• Cost reduction
• Increased test coverage
• early validation of design model
Key
advantages
• Test Optimal (MBT tool)
• MaTeLo (MBT tool)
• HP QTP (DDT automation tool)
• Selenium ( test automation tool)
TRASYS
tools
• Test data & test case generation using
design techniques (boundary values, pair wise,
branch, decision, transition, state, requirements,..)
• Model based & data driven test automation
• Test data management
TRASYS
Services
21. Testing as a Service: Security Testing
Defintion
Key
challenges
TRASYS
tools
• Vulnerability scanning & assessment
• Penetration testing / attack based testing (= systematic probing of systems with the goal
of gaining access to the information they contain)
-> Open Web Application Security Project (OWASP) top 10
most critical web application flaws
-> Open Source Security Testing Methodology (OSSTM)
TRASYS
Services
• Acunetix
• Webscarab
• IBM Appscan
• See appendix 1 for full list
Determine that an information system protects data and prevents unauthorized
access, whether accidental or deliberate, to programs and data.
The six basic security concepts that need to be covered by security testing are:
• confidentiality
• Integrity
• identification & authentication
• Availability
• Authorization
• non-repudiation
22. Testing as a Service: Usability & Crowd Testing
Defintion
Key
challenges
TRASYS
tools
• Requirements based usability testing, including
prototype & mock up validations during inception
of SDLC using for example ISO 9241
• Real-world usage validation in order to
continuously improve the systems
* Use of production data analytics & usability tools
* Crowd Testing = crowd sourcing + cloud test platform
TRASYS
Services
• Chalkmark, Clickheat
• Google analytics
• Usertesting.com, Userzoom
• Browser cam (cloud based)
• UserReplay
• Avoid that end users / customers leave (abandon) when a system is difficult to use, a
homepage fails to clearly deliver information i.e E-commerce: if users cannot find the
product, they cannot buy it either ( impact on revenue)
• The functionality of our application is state of the art, but users lose time trying to figure out
how to operate it ( impact on productivity)
Testing the ease with which the user interfaces can be used and whether the application or the
product built is user-friendly or not.
Usability testing also reveals whether users feel comfortable with your application or Web site
according to different parameters - the flow, navigation and layout, speed and content -
especially in comparison to prior or similar applications.
23. Testing as a Service: Performance Testing
Defintion
Key
challenges
TRASYS
tools
• Load testing
• Stress testing
• Scalability testing & capacity planning support
• Code review with performance focus – performance profiling
• Reliability testing
TRASYS
Services
• Loadrunner
• Jmeter
• Jprofiler
• Several code review & inspection tools ( see Annex 1)
• Avoid that end users / customers leave (abandon) when a system is ‘too slow’
to use, a homepage fails to deliver information with an acceptable response
time -> impact on revenue
• The functionality of our application is state of the art, but (internal or external)
users lose time to operate and use the functionality -> impact on productivity
• System is not available due to overload -> impact on revenue
• System is oversized or undersized due to lack of performance testing ->
impact on Capex / Opex
The process of testing to determine the performance of a software product.
25. Testing as a Service: Implementation approach
Critical Success Factors
Defined as major business
transformation & major
change IN YOUR
ORGANISATION
Managed as a project with
a plan, budget , resources,
scope, dependencies,
risks,…
Business case
supported by share -
and stakeholders
Service contract with
RACI and defined service
inputs and outputs
Collaboration model and
governance model
26. Testing as a Service: Implementation approach
Prepare phase
•Feasibility study – Proof of Concept – Business Case
•Test maturity assessment
•Information and communication (generate awareness)
Service Design
phase
•Target operating model & test process – governance model – collaboration model - RACI
•Service contract – document of understanding (scope of services & projects / assets – pricing – planning)
•Transition , change and deployment plan (including training and internal communication)
Implementation
phase
•Knowledge transfer – information transfer – access to test object , test environments & tools
•Proof of concepts
•Mass deployment
Operations
phase
•Run operations as a service
•Monitor and report on operations (KPIs, activity reporting)
•Manage demands and service delivery
Improvement
phase
•Operational excellence through continuous test process improvement and service improvement
27. Annex 1 : Test tools
Defect
management
Test
Management
Test
automation
Performance
testing &
profiling
Code
review
&
inspection
Usability
and
prototype
testing
Security
testing
Agile
testing
Mobile
application &
cloud testing
SAP
testing
HP Quality
Centre - ALM
HP Quality
Centre - ALM
HP QTP HP Loadrunner
FindBugs
Chalkmark Acunetix
WVS
Jira
Greenhopper
Fonemonkey SAP TAO
Jira Spiratest Selenium Jmeter
PMD
Clickheat
BackTrack
SmartBear
ALM
Perfectomobile
(cloud based)
SAP
Solution
Manager
Bugzilla Flexmonkey Jprofiler
CheckStyle
Google
analytics Paros Proxy
MS Visual
Studio and
TFS
IBM Worklight
(mobile) and IBM
Smartcloud
SAP
eCATT
Mantis SOASTA
JavaNCSS
Usertesting.
com Tamper
Data
Mylyn for
eclipse
SOASTA
(cloud based)
• CloudTest
• TouchTest
HP QTP
SAP plug
in
SOAP UI
(SOA testing)
Cobertura
Userzoom
WebScarab
HP Agile
Manager
(cloud
based)
Open Cloud
architecture
( Cloud Foundry)
Test Optimal
(model
based
testing)
SonarQube
Google
Website
Optimizer
CSRF
Tester
W3C mobile ok
checker
MaTeLo
(model
based
testing)
Browser
cam (cloud
based)
IBM
rational
AppScan
Deviceanywhere
(cloud based)
UserReplay Xamarin
(cloud based)
