QA in Cloud world & DevOps

By Dudi Vaanunu
QA IN CLOUD WORLD
& DEVOPS
ALL RIGHTS RESERVED © (Copy rights) to Smart Group LTD Dudi Vaanunu

o In this Meetup we are going to see how to
perform “cloud testing”.
To understand this we need to know the
cloud computing concept first.
In this Meetup I will explains what is cloud
computing, what is cloud testing, and the
major challenges with testing in cloud.
o DevOps – On time deployment on Quality
what is DevOps ? Integration with QA
CI ? CD ?
Agenda

o Director of QA at ControlUp technologies
o Director of QA Training divisions – Hitch colleges
o Senior consultant and lecturer
o Co-Founder at Smart Group LTD
(Smart Test , Smart Dev , Smart Professionals ,
Smart Sites)
o More then 13 years on hands-on experience
Dudi Vaanunu
About me

How many of you are in the cloud?
QUESTION:
4

o Several years back the industry witnessed
a new buzzword and technology called
“virtualization”.
With the advent of virtualization, the
ideology of sharing computing resources
across multiple operating systems to
reduce capital costs, increase scalability
and enable easy administration of the IT
infrastructure became the backbone of
several enterprises.
Cloud computing introduction

o Scalability :
The capability of a system, network, or
process to handle a growing amount of work,
or its potential to be enlarged to
accommodate that growth.

o Vertical Scaling (Scale-up): Generally refers to
adding more processors and RAM, buying a more
expensive and robust server.
o Less power consumption than running multiple
servers
o Cooling costs are less than scaling horizontally
o Generally less challenging to implement
o Less licensing costs
o (sometimes) uses less network hardware than
scaling horizontally (this is a whole different topic
that we can discuss later)

o Horizontal Scaling (Scale-out):
Generally refers to adding more servers
with less processors and RAM.
This is usually cheaper overall and can
literally scale infinitely (although we know
that there are usually limits imposed by
software or other attributes of an
environment’s infrastructure)
o Much cheaper than scaling vertically
o Easier to run fault-tolerance
o Easy to upgrade

o In the past few years, IT has witnessed an
evolution of virtualization in the form of cloud
computing.
Any novice to cloud must picture cloud
computing as a model that views everything
“As a Service”.
It is obvious to define cloud computing as a
total solution that delivers IT as a Service.
Like virtualization, its motto is resource
sharing whereby allocation and availability
are on demand via Internet.

Cloud Data models
(SaaS)
Software as a
Service
Email
CRM
iCloud
ERP
Consume
(PaaS)
Platform as
a Service
Application Development
Decision Support
Web
Streaming
Build On It
(IaaS)
Infrastructure
as a Service
Caching
Legacy
Networking Technology
Security System Management
Migrate To It
• End
Users
Saas
• Application
Developers
Paas
• Network
Architects
Iaas
ValueVisibilitytoendusers

o Infrastructure as a Service (IaaS):
This is the most fundamental layer which
forms the building block of the cloud.
It mainly comprises of physical resources
such as Storage, Network devices,
compute servers, etc.
All these computing resources are
available on demand, where a user pays
as per his usage.
IaaS
11

Platform as a Service :
The central layer in the cloud Is the platform. Here
there is no control over the underlying
infrastructure but you can access the deployed
applications.
Hence this provides the entire runtime environment
on demand which could be a development or test
environment.
Hence in this model, most commonly you would
have a VM that contains the complete environment
like OS, required middleware, etc. available when
you need it.
PaaS
12

o Software as a Service:
The topmost layer here is the application layer,
which is generally visible to any user.
Here, applications / products are available to the
user on demand via the internet.
Hence instead of having to acquire licenses for a
particular user, this proves to be the most cost
effective way of making sure that the license is
always in use.
Examples of this are Gmail, Google docs, etc.
SaaS
13

14
Applications
Data
Runtime
Middleware
O/S
Virtualization
Servers
Storage
Networking
YOUMANAGE
Traditional IT
Applications
Data
Runtime
Middleware
Virtualization
Servers
Storage
Networking
YOUMANAGEDELIVEREDASA
SERVICE
O/S
IaaS
Applications
Data
Runtime
Middleware
O/S
Virtualization
Servers
Storage
Networking
YOU
MANAGEDELIVEREDASA
SERVICE
PaaS
Applications
Data
Runtime
Middleware
O/S
Virtualization
Servers
Storage
Networking
DELIVEREDASASERVICE
SaaS

o Testing as a Service (TaaS) is an outsourcing model
in which testing activities associated with some of an
organization’s business activities are performed by a
service provider rather than employees.
o Enables your organization to concentrate on its core
development activities while we handle the validation
work independently.
Regardless of the scope and frequency of your
testing needs, it is often time saving and cost
effective to engage an independent team of software
testing professionals to do the job for you.
Testing as a Service (TaaS) ?
15

o KaaS is not a software package, a methodology, or a
framework– it is a model.
A collection of lessons learned, best practices, proven
workflows, and case studies that can help your
organization leverage knowledge from anywhere,
anything, and (more importantly) anyone in a
distributed computing model.
Knowledge as a Service (KaaS)
16

Cloud Environments
17
Private Environment Public Environment
Hybrid Environment

o There are 3 types of cloud testing environments:
Cloud testing environments:
• A test environment which is cloud based, where
the quality of applications deployed in the cloud
need to be validated.
• Private or public environment where the quality of
applications deployed in them need to be
validated.
• A hybrid environment, where the quality of
applications deployed in them need to be
validated.

o Are where services are available to
everyone, where the resources are
allocated and provisioned dynamically as
per request.
Public clouds

o Are generally managed within firewall rules
of a particular organization and are
available only to the users within the
company
Private clouds

o Are a mixture of private and public clouds.
Organizations can decide what services
they want to expose to everyone and what
services they want to expose to only the
users within the organization.
Hybrid clouds

o Cloud computing has today become one of
those “big bangs” in the industry.
Most organizations are now leaning to
adopting the cloud because of its flexibility,
scalability and reduced costs.
o Cloud testing is a form of software testing in
which web applications use cloud computing
environments
(a "cloud") to simulate real-world user traffic.
What is cloud testing ?

WHY?
Paradigm
shift
Traditionally
applications are
designed first,
then developed,
tested and
distributed.
SaaS
Applications;
Browser based
configuration and
usage.
Seamless
upgrades
Traditionally.
Software
upgrade needs to
get new version
of the software,
down the system
and upgrade.
But with Cloud,
upgrade should
happen live with
minimal or no
down time E.g.
Gmail
Sharing of
resources:
Multitenancy
Traditionally,
software
development &
deployment is
mostly in
dedicated
resources.
With Cloud,
these resources
are often being
shared among
multiple
customers.
New test
dimensions
Auto-elasticity:
Resources on-
demand.
Disaster recovery
(Cloud DR)
Availability and
Business
Continuity
Changes in
existing
dimensions
Performance
Compatibility
Security:
Multitenant
penetration,
identity federation
management.

o Cloud testing refers to testing of resources
such as hardware, software, etc.
That are available on demand.
Even the testing here can be viewed “as a
service”.
For cloud offerings it’s essential to make
sure that the service (product) not only
meets its functional requirements but also
non-functional requirements.
With the range of applications in the cloud
it is now becoming essential to develop
cloud testing strategy.
Cloud testing- Need for it
0
10
20
30
40
50
60
70
Web Site Test And
Development
E-mail and PC
Applications
HR and Payroll Finance and
Accounting
Applications in the cloud

Dynamic availability of testing environment:
The normal testing approach in any organization
is to invest in the hardware/ software
infrastructure needed for the testing.
Almost most of you will agree that the
environment supplied to testing teams very
rarely matches a customer environment on
account of rapidly changing requirements, as a
result of which it is very challenging for
companies to keep up.
Cloud is the answer to this problem whereby,
QA can easily replicate a customer environment
and find defects early in the cycle.
Benefits of Cloud Testing:

o Low cost: Another angle to the previous
point is that when companies invest in the
infrastructure, the usual case it many of
their servers are not utilized all the time.
As a result, they may have to incur
additional expenses on license renewal.
Transitioning to cloud helps in this
scenario as well, as QA can commission
appliances as and when they want, saving
huge costs for an organization.
Benefits of Cloud Testing:

o With the usage of cloud, it’s a rather easy
task for organizations to emulate an end-
user centric environment by customizing it
as per use saving cost and time.
o Test teams can easily perform load and
performance testing scenarios in various
permutations and combinations like –
different OSes, browsers, configurations,
etc.
Easily customizable:
01

o One of the most attractive features of the
cloud whereby the computing resources
can be increased or decreased wherever it
is necessary.
Extremely useful in situations where the
business requirements keep altering
frequently.
Scalability:
02

o Testing with respect to a cloud will employ
both traditional and new age methods to
perform testing.
Testing and cloud are equally large
domains in their own right. Let’s learn how
to view the two together in the below
sections.
Cloud testing deep dive
03

o Cloud testing can be broadly divided into four different categories based on
what they aim to do :
Cloud testing forms:
SaaS testing in cloud: Functional
and nonfunctional testing is
performed based on requirements.
12
3 4
Testing of the whole cloud: The
cloud is viewed as a whole entity
based on its features and testing is
carried out based on that.
Testing within a cloud : This is the
testing carried out inside of the
cloud by checking each of its
internal features
Testing across clouds: Based on
specifications, here the testing is
carried out on the different types of
cloud like public, private and hybrid
clouds.

o Testing in a cloud has to not only ensure
that the functional requirements are met,
but a strong emphasis needs to be laid on
nonfunctional testing as well. Let’s take a
look at the different kinds of testing that
are performed.
Types of testing performed in
cloud:

o Goes without saying, that functional testing has to be
performed to make sure that offering provides the services
that the user is paying for.
Functional tests ensure that the business requirements are
being met.
Some of the functional tests are described below:
o System Verification Testing: This makes sure that the various
modules function correctly with one another, thus making sure
that their behavior is as expected.
o Acceptance Testing: Here the cloud based solution is handed
over to the users to make sure it meets their expectations.
o Interoperability Testing: Any application must have the
flexibility to work without any issues not only in different
platforms, but also must work seamlessly when moving from
cloud infrastructure to another.
Functional testing:
01

o Non-functional tests mainly focus on the web
application based tests ensuring that they meet the
desired requirements.
Here are few forms of non-functional tests discussed
below :
o Availability Testing: The cloud supervisor/ vendor
has to make sure that the cloud is available round the
clock.
As there could be many mission critical activities going
on, the administrator has to make sure that there is no
adverse impact to the consumers
o Multi Tenancy Testing: Here, multiple users use a
cloud app.
Testing must be performed to ensure that there is
sufficient security and access control of data when
multiple users are using a single instance.
Non-Functional testing:
02

o Performance Testing: Verification of the
response time needs to be done to ensure
that everything is intact even when there is a
large amount of requests to be satisfied.
The network latency is also one of the critical
factors to evaluate performance.
o Also workload balancing needs to be done
when there is a reduction in load, by
decommissioning resources. Thus load and
stress testing are done in the cloud offering
to make sure applications are performing
optimally with increase/decrease in load and
stress.
02

o Security testing: Since with the cloud
everything is available anytime, it’s
essential to make sure that all user
sensitive information has no unauthorized
access and the privacy of users remains
intact. When maintaining the applications
in cloud, user data integrity must also be
verified.
02

o Disaster recovery Testing: As already
stated in availability testing, the cloud has
to be available at all times and if there are
any kind of failures like network outages,
breakdown due to extreme load, system
failures, etc, measure how fast the failure
is indicated and any data loss during this
period.
o Scalability Testing: Test to make sure that
offering has the capability to provide scale
up or scale down facilities as per the need.
02

Cloud Testing Challenges
37

o Challenge #1 : With everything available on
demand to any user, security is a primary issue
for the businesses as currently there is still a lot of
discussion and research going on in the industry
to set up security standards.
User privacy protection, security standards on
cloud, security of applications running within the
cloud, security testing techniques are some of
the primary issues that need to be addressed in
the cloud infrastructure.
01

o Challenge #2: Another big challenge is
the performance of an application in a
cloud
It will be shared across many users and
hence could lead to delays.
Also in case of some maintenance or
outage related activities, the bandwidth
may seem insufficient.
02

o Challenge #3: Sometimes for testing
purposes, we require certain
configurations: with respect to servers,
storage or networking which may not be
supported by the cloud provider.
This sometimes makes it difficult to
emulate customer environments.
03

o Challenge #4: Another commonly faced challenge is
with respect to integration testing whereby the
testers test the network, database, servers, etc.
In such situations the tester will not have control on
the underlying environment.
Secondly, the challenge is doubled when there has to
be an interaction between these components
because the tester will have to anticipate risks like
crashes, network breakdown or servers going kaput.
04

o Challenge#5: Data Security and Privacy
Cloud applications being a multi-tenant in
nature, risk of data theft is always remain.
Cloud users should be given an assurance
about safety of their data by suppliers.
o Challenge#6: Data Migration
Data migration from one Cloud provider to
another is a huge challenge as both
providers may have different database
schemas and it requires a lot of effort to
understand the data fields, relationships and
how are they mapped across SaaS
application
5&6

Could testing examples
43

o Failure due to one user action on cloud
should not affect other users performance
o Manual or automatic scaling should not
cause any disruption
o On all types of devices the performance of
the application should remain same
o Overbooking at supplier end should not
hamper the application performance
Performance testing

o Only authorized customer should get access
to data
o Data must be encrypted well
o Data must be deleted completely if it is not in
use by client
o Data should be accessible with insufficient
encryption
o Administration on suppliers end should not
access the customers data
o Check for various security settings like
firewall, VPN, Anti-virus etc.
Security Testing

o Valid input should give the expected
results
o Service should integrate properly with
other applications
o System should display customer account
type when successfully login to the cloud
o When customer chose to switch to other
service the running service should close
automatically
o System should display customer account
type when successfully login to the cloud
o When customer chose to switch to other
service the running service should close
automatically
Functional testing

o Validate the compatibility requirements of
the application under test system
o Check browser compatibility on cloud
environment
o Identify the defect that might arise while
connecting to cloud
o Any incomplete data on cloud should not
be transferred
o Verify that application works across
different platform of cloud
o Test application on in-house environment
and then deploy it on cloud environment
Interoperability & Compatibility
Testing
Testing

o Test protocol responsible for cloud
connectivity
o Check for data integrity while transferring
data
o Check for proper network connectivity
o Check if packets are being dropped by
firewall on either side
o Network interruptions
Network Testing
Testing

o Check for services when multiple users
access the cloud services
o Identify the defect responsible for
hardware or environment failure
o Check whether system fails under
increasing specific load
o Check how system changes over time
under a certain load
Load and Stress Testing
Testing

On time delivery on Quality
50

o In simple words the DevOps means ‘It is a combination of software
development and operations’.
o It came to emphasize the communication, collaboration, and cohesion
between the traditionally separated development and IT operations
teams.
o DevOps helps an organization deploy software more frequently, while
maintaining service stability and gaining the speed necessary for more
innovation.
o DevOps is the practice of operations and development engineers
participating together in the entire service lifecycle, from design through
the development process to production support.
o DevOps is also characterized by operations staff making use many of
the same techniques as developers for their systems work.
DevOps Overview

DevOps Overview
52

DevOps Overview
Operations
Development
Software
Engineering
QA
Quality
Assurance
DevOps

o Also known as CI
o Continuous integration is the practice of constantly merging development work with
a Master/Trunk/Mainline branch so that you can test changes and test that those
changes work with other changes.
The idea here is to test your code as often as possible so you can catch issues
early on.
o In the continuous integration process, most of the work is done by an automated
tests technique which requires a unit test framework.
o It is best practice to have a build server designed specifically for performing these
tests so your development team can continue merging requests even while tests
are being performed.
o Perform all tests automatically for every piece of code
Continuous Integration

o Continuous delivery is the continual
delivery of code to an environment once
the developer feels the code is ready to
ship - this could be QA, Staging or
production.
Continuous Delivery

o Also known as CD
o The production branch is always stable and ready to
be deployed by an automated process.
o The automated process is key because it should
be able to be performed by anyone in a matter
of minutes (preferably by the press of a button).
o After a deploy, logs must be inspected to
determine if your key metrics are affected,
positively or negatively.
o Once a piece of code has been through CI
o releases it automatically
o Very useful for bugs and minor changes
Continuous Deployment

CONTINUOUS DEPLOYMENT
Releases
per day 30+ 100+ 20+ 2+
Features
Released Thursday Tuesday Daily Monthly

o In the ideal workflow, the entire process
could be automated from start to finish:
o Step 1: Developer checks in code to
development branch.
o Step 2: Continuous integration server
picks up the change, merges it with
Master/Trunk/Mainline, performs unit
tests and votes on the merge to staging
environment based on test results.
How They Work Together

o Step 3. If Step 2 is successful,
developer deploys it to the
staging environment and QA tests
the environment.
o Step 4. If Step 3 passed, you
vote to move to production and
the continuous integration server
picks this up again and
determines if it’s ok to merge into
production.
o Step 5. If Step 4 is successful, it
will deploy to production
environment.
How They Work Together

o A single team composed of cross-functional members and all working in collaboration, DevOps
organizations can deliver with maximum speed, functionality, and innovation.
o Benefits of DevOps:
DevOps Overview
o Technical Benefits:
Continuous software delivery
Less complexity to manage
Faster resolution of problems
o Cultural Benefits:
Happier, more productive teams
Higher employee engagement
Greater professional development opportunities
o Business Benefits:
Faster delivery of features
More stable operating environments
Improved communication and collaboration
More time to innovate (rather than fix/maintain)

DevOps Architecture
61

o There are 9 types of DevOps tools which has known before choosing for the project…
o Collaboration Tools:
o This type of tool is crucial to helping teams work together more easily, regardless of time zones or
locations.
o A rapid action oriented communication designed to share knowledge and save time.
(See: Slack, Campfire).
o Planning Tools:
o This type of tool is designed to provide transparency to stakeholder and participants.
o Working together, teams can plan towards common goals, and better understanding of
dependencies. Bottlenecks and conflicting priorities are more visible. (See: Clarizenand Asana).
o Source Control Tools:
o Tools of this sort make up the building blocks for the entire process ranging across all key assets.
Whether code, configuration, documentation, database, compiled resources and your web site
html – you can only gain by managing them in your one true source of truth.(See: Git, Subversion).
Types Of DevOps Tools

o Issue Tracking Tools:
o These tools increase responsiveness and visibility.
o All teams should use the same issue tracking tool, unifying internal issue
tracking as well as customer generated ones. (See: Jira and ZenDesk ).
o Configuration Management Tools:
o Without this type of tool, it would be impossible to enforce desired state norms
or achieve any sort of consistency at scale.
o Infrastructure should be treated exactly as code that can be provisioned and
configured in a repeatable way. (See: Puppet, Chef, Salt).
o Database DevOps Tools:
o The database, obviously, needs to be an honored member of the managed
resources family. Managing source code, tasks, configuration, and deployments
is incomplete if the database is left out of the equation.(See: DBmaestro)

o Continuous Integration Tools:
o Continuous integration tools provide an immediate feedback loop by regularly merging code. Teams
merge developed code many times a day, getting feedback from automated test tools.
(See: Jenkins, Bamboo, TeamCity).
o Automated Testing Tools:
o Tools of this sort are tasked with verifying code quality before passing the build. The quicker the
feedback loop works – the higher the quality gets, and the quicker you reach the desired "definition
of done".(See: Telerik, QTP, TestComplete)
o Deployment Tools:
o In an effective DevOps environment, application deployments are frequent, predictable, and
reliable.
o Deployment tools are essential to checking those boxes. Continuous delivery means that
applications can be released to production at any time you want in order to improve time to market,
while keeping risk as low as possible. (See: IBM uDeploy, CA Release Automation, XebiaLabs)

How many of you use continuous integration tools?
QUESTION:
65

Jenkins
o The "leading open source automation server," Jenkins was
forked from Hudson and offers many of the same capabilities.
o It boasts easy installation and configuration, hundreds of
plugins, extensibility and a distributed architecture that allows
it to speed the process of testing.
o It has a very active user community with lots of scheduled
events that offer opportunities to learn more about the
software.
o There is also plenty of documentation on the website,
including a blog that is updated regularly.
o Jenkins Pipeline is a suite of plugins which supports
implementing and integrating continuous delivery pipelines
into Jenkins.
o Pipeline provides an extensible set of tools for modeling
simple-to-complex delivery pipelines "as code".
DevOps Tools

DevOps Tools

Thank you for participating
Dudi Vaanunu
Q&A
68

QA in Cloud world & DevOps

More Related Content

What's hot

Similar to QA in Cloud world & DevOps

Recently uploaded

QA in Cloud world & DevOps