Nonfunctional Testing: Examine the Other Side of the Coin

T14

Test
Techniques

10/6/16
13:30

Nonfunctional
Testing:
Examine
the

Other
Side
of
the
Coin

Presented
by:

Balaji
Arunachalam

Intuit,
Inc.

Brought
to
you
by:

350
Corporate
Way,
Suite
400,
Orange
Park,
FL
32073

888-‐-‐-‐268-‐-‐-‐8770
·∙·∙
904-‐-‐-‐278-‐-‐-‐0524
-‐
info@techwell.com
-‐
http://www.starwest.techwell.com/

Balaji
Arunachalam

Balaji
Arunachalam
initially
worked
at
Intuit
as
a
software
developer
and
then

moved
into
QE
out
of
a
passion
for
software
quality.
Balaji
is
currently
leading
the

QE
group
for
Intuit's
Core
Services,
a
group
comprised
of
strong
engineers
who

share
his
passion
for
software
quality.
The
QE
group
started
its
journey
with
test

automation,
continued
as
everything
automation,
and
matured
into
a
unique

automation-‐as-‐a-‐service
practice.
The
QE
group
is
performing
on-‐going
testing

while
innovating
tools
and
frameworks
that
make
the
job
easier
and
more

interesting.
Over
time,
the
team
has
evolved
into
hybrid
engineers
who
are
capable

of
doing
both
functional
and
nonfunctional
testing.

Nonfunctional Testing:
Examine the Other Side of the Coin
Balaji Arunachalam,
Director of Quality Engineering, Intuit
Note: The content is best read through slide-show

Intuit Proprietary & Confidential
Balaji Arunachalam,
Director, Quality Engineering, Intuit

What is Non-Functional Testing?
3

Why is it important?
4

Intuit Journey
5
Single Data Center
(No HA DR)
Dual Data
Center (HA DR)
Isolation and
Constant Refresh
AWS Cloud

What is HA/DR?
6
App Server
VM-1
App Server
VM-2
App Server
VM-3
DB
Load
Balancer
Real Time Sync
X
SITE-1
App Server
VM-1
App Server
VM-2
App Server
VM-3
DB
Load
Balancer
Backup
SITE-2
Load
Balancer
XX
X
BackupBackup
Recovery
(Passive)(Passive)

RTO/RPO
• RTO (Recovery Time Objective)- The
amount of time it takes to recover
the system in the event of system
outage (Downtime).
• RPO (Recovery Point Objective) –
The amount of data that could be
potentially lost during a data
recovery. This drives the backup
interval.
7
Time line
System failed System recovered
(RTO)
Last Data
Backup
(RPO)
Time line
System Failed System recovered

What is the RTO and RPO for this case?
8
App Server
VM-1
App Server
VM-2
App Server
VM-3
DB
Load
Balancer
Real Time Sync
X
SITE-1
App Server
VM-1
App Server
VM-2
App Server
VM-3
DB
Load
Balancer
Backup
SITE-2
Load
Balancer
XX
X
BackupBackup
Recovery
(Passive)(Passive)
2:00 PM
2:01 PM
Sync Delay: 3 minutes
5:00 PM
5:30 PM
Last Backup: 4 PM
RTO/RPO?
RTO: 1 Min
RPO: up to 3 mins
RTO: 30 Mins
RPO: 1 hour
X

Capacity Testing (Stress Testing)
9
2000 LB 2000 LB
2000 LB
500LB

Stability Testing (Reliability Testing)
10
2000 LB
2000 LB
2000 LB
2000 LB
2000 LB
After driving 12 hours
After driving 24 hours

Scalability Testing
11
2000 LB 2000 LB
2000 LB
2000 LB
2000 LB
2000 LB
2000 LB
Vertical
2000 LB
2000 LB
Horizontal
2000 LB
2000 LB
2000 LB
2000 LB

Capacity (Stress) Testing
12
0
20
40
60
80
100
120
0
500
1000
1500
2000
2500
3000
3500
0 200 400 600 800 1000 1200 1400 1600 1800 2000
ResponseTime(ms)
Throughput(TPS)
Concurrent User Count
Capacity Test - Single Site
Throughput (TPS) 99th Percentile RT (ms)
SLA - Under 100 ms
Capacity Target - 2000 TPS
• Run Capacity Test in each site.
• Plan Capacity Buffer for failure conditions
(what if I lose one full site in a Dual Active
Site situation).
• Account for dependency capacities and also
contingencies.

Stability Testing
13
• Run test simultaneously across multiple active sites.
• Typically run for 24 to 72 hours.
• Memory leaks, network instability, scheduled
maintenance impacts can be found.

Benchmarking
14
• API/Pages are benchmarked for response times
under constant load.
• Automatically runs every night.
• Can find new release regression, faulty nodes etc.,

Scalability Testing
15
• Conduct both Vertical and Horizontal Scalability
testing to find the best approach.
• Vertical scaling might be cost effective but may not
help with redundancy.
0
5
10
15
20
25
30
35
0
200
400
600
800
1000
1200
1400
0 2 4 6 8
ResponseTime(ms)
Throughput(TPS)
# of App Server VMs
Horizontal Scaling - e.g. Linear
0
5
10
15
20
25
30
35
0
100
200
300
400
500
600
700
Small Medium Large XL
Throughput(TPS) App Server VM Size
Vertical Scaling – e.g. Non-Linear

Gate Rush (Spike) Testing
16
• Understand and simulate your customer
traffic.
• Gate rush traffic is sometimes
unpredictable. The throughput can
skyrocket beyond the system capacity.
• How does the system handle without fall
over? (Queuing, throttling??)
0
200
400
600
800
1000
1200
1400
1600
6:00AM
6:25AM
6:50AM
7:15AM
7:40AM
8:05AM
8:30AM
8:55AM
9:20AM
9:45AM
10:10AM
10:35AM
11:00AM
11:25AM
11:50AM
12:15PM
12:40PM
1:05PM
1:30PM
1:55PM
2:20PM
2:45PM
3:10PM
3:35PM
4:00PM
4:25PM
4:50PM
5:15PM
5:40PM
6:05PM
6:30PM
6:55PM
7:20PM
7:45PM
8:10PM
8:35PM
9:00PM
9:25PM
9:50PM
THROUGHPUT
TIME
Gaterush Testing

Potential Bottlenecks
1. High CPU on the test driver
2. Low network throughput capacity
3. Web layer : Max Clients too low
4. Web layer: File descriptor limit too low
5. Web layer: Connector or HTTP Client pool too small
6. Web layer: High CPU
7. Web layer: Memory leak causing GC crisis
8. Web layer: Heap too small
9. Application layer: High CPU
10. Application layer: Connector thread limit reached
11. Application layer: JVM thread limit reached
12. Application layer: File descriptor limit too low
13. Application layer: Memory leak
14. Application layer: Heap too small
15. Application layer: Database connection pool too small
16. Application layer: Database connection pool max Active too small
17. Application layer: Database stored procedure slow
18. Misc: Licensing limits
19. Misc: Dependency bottleneck

RAS Testing – Steps Involve
FMEA
RAS Test Plan
RAS Test Scheduling
RAS Test Execution
RAS Test Report
11

RAS Testing – FMEA (Failure Mode and Effect Analysis)
Identify System
Failure Modes
Failure Mode Scoring
E.g., Application Server Failure, Disk Shortage, NFS Mount Failure
Brainstorming with Project Team, Network Team, DB Team and Ops Team as needed
Score each failure mode based on “Severity”, “Detection” and “Occurrence” out of 10.
Calculate total score for each failure mode by multiplying all the 3 scores from step 1.
Prioritize Failure Modes
Prioritize the failure modes based on the total score and identify the
Failure modes which needs to be tested.
12

RAS Testing – Single Site vs. Multi Site
Stable
Load
App
Server
Site-2
LTM
GTM
DNS
App
Server
App
Server
Gateway Gateway Gateway
Stable
Load
App
Server
Site-1
LTM
GTM
Golden Gate
DNS
GTM Request
Actual Service Request
Data Center State
Exchange
App
Server
App
Server
Gateway Gateway Gateway
Spectrum
Wily
New Relic
Splunk
Monitoring
ALERTS ALERTS
Database Failover:
Database cluster load balancing
Database node failover
Application layer Failover:
Application layer load balancing
Application Node failover
Gateway layer Failover:
Gateway layer load balancing
Gateway Node failover
Other RAS Testing:
Disk fillup, NFS mount down,
Network down etc
Tools:
1) Load generated using JMeter
2) Monitoring tools: Spectrum, Wily,
Site Scope, Splunk.
11
Multi Site Testing:
1) Data Center Failover caused by DB cluster failure
2) Data Center Failover caused by App layer failure
3) Data Center Failover caused by Web layer
failure.
4) Test for Switchback scenario of the above
failures.
5) Test for unnecessary data center failover.

• Questions:
– What Testing is this?
• Stability, Capacity or
Scalability
– What is the capacity of the
system?
– What could be the
bottleneck? And how can
we solve?
Quiz - 1

• Questions:
– What Testing is this? (Capacity, Stability or Scalability)
– What is the conclusion of this testing (good or bad)?
Quiz - 2
-0.1
6E-16
0.1
0.2
0.3
0.4
0.5
0.6
0
500
1000
1500
2000
2500
3000
3500
4000
0 200 400 600 800 1000 1200 1400 1600 1800
ResponseTime(sec)
Txn/sec
Number of Virtual Users
IDM 4.0.0.17 vs. 4.0.0.12
IDM 4.0.0.17 Hits/sec
IDM 4.0.0.12 Hits/sec
IDM 4.0.0.17 Response Time
IDM 4.0.0.12 Response Time

• Questions:
– If the SLA for response time is 120 ms, is the above test
successful?
– If not successful, why?
Quiz - 3

• Questions:
– What type of testing is this?
• Capacity, Scalability or
RAS/Failover
– What do you understand from
Fig.1?
Quiz - 4

• Questions:
– What type of Testing is this?
• Capacity, Stability or Scalability
– What is wrong with this trend in Old Gen Memory?
Quiz -
5

Mistakes and Learning in Performance
Testing
• Automate Everything
• Pay attention to System Profile.
– Load test should match not only overall TPS but also API/use case
profile.
• Pay attention to payload.
– Load test should match production payload size and content.
• Continuously update the load testing based on production
profile.
• Understand your dependencies and their capacities.
• Keep functional validation as part of load testing.
• Measurement of E2E response times from client call location.
• Pay attention to outliers in response times.
26

Where we started…
27
Dedicated but Limited
Non Functional Engineers
No Automation for
Non-Functional Testing
Single Point of Failure
Repeated and Boring Task

Where we are now…
28
Hybrid Engineers
Fully Automated
Non-Functional Testing
Multi Point of Support
Happy Engineers

Maturity Model of a Functional QE to
Hybrid Engineer
29
Functional QE Tasks
Performance/FMEA Testing
(Existing Projects)
FMEA Testing
(New Projects)
Performance Testing
(New Projects)
SWE in Quality I
SWE in Quality II
Senior SWE in Quality
(Hybrid)
Staff SWE in Quality or Above
(Hybrid)
Performance QE

Performance Test Automation Demo
30

Questions
31

Nonfunctional Testing: Examine the Other Side of the Coin

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Nonfunctional Testing: Examine the Other Side of the Coin

Similar to Nonfunctional Testing: Examine the Other Side of the Coin (20)

More from TechWell

More from TechWell (20)

Recently uploaded

Recently uploaded (20)

Nonfunctional Testing: Examine the Other Side of the Coin