Non functional requirements testing in agile, presented at Agile Testing Alliance Global Gathering 2014 in Bangalore

1. Non Functional Testing
in Agile
Agile Testing Alliance Global
Gathering
Nov.
7
S R V Subrahmaniam
PMP, ACP, CSP
Principal Consultant Agile
Altimetrik

2. Slide 2
Let us play the Tank Game!
Let us build a Tank Iteratively
We, in the next 30 minutes, look at building a
new version of Main Battle Tank Arjan
New set of features
Assume a tank is already in place, and
we will look enhancing the same
Typical Main Battle Tank
CONTEXT
A main battle tank (MBT), also known as a
battle tank or universal tank, is a tank that fills the
heavy direct fire role of many modern armies.
Image Source: http://upload.wikimedia.org/wikipedia/commons/5/5b/US_Navy_050619-N-6811L-
216_An_Australian_Army_Leopard_AS1_main_battle_tank_moves_up_the_beach_at_Sabina_Point_after_being_transported_ashore_by_a_U.S._Navy_Landing
_Craft,_Air_Cushion_(LCAC).jpg

3. Slide 3
Features for the game
today
GETTING
STARTED
Arjan needs to
travel in shallow
water (upto 5 m) –
it can travel in
mud and sand.
Upgrade the main
firing gun from
90mm to 130mm
caliber
Improve the
weapon delivery
control system –
including gun data
computer, director
and radar
Introduce ceramic vehicle armour
This might impact the performance of the tank in durability, moving in
multiple terrains, withstanding the impact of firing (from the tank & on the
tank). This (and a combination of these) may be simulated in specialized test
Environments and may require prolonged monitoring.

4. Slide 4
Our Plan today
AGENDA
BACKLOG CREATION
10 Minutes
ITERATION I
10 Minutes
RETROSPECTIVE
10 Minutes
DISCUSSIONS
30 Minutes

5. Slide 5
Let us start with the Tank Game!
Elicit Requirements
Create Backlog
Features > User Stories
EXERCISE
Backlog Creation
10 Minutes

6. Slide 6
Let us continue with the Tank Game!
Iteration 1 Planning
Execute Iteration
Iteration 1
EXERCISE
10 Minutes

7. Slide 7
Let continue with the Tank Game!
Iteration 1 Retrospective
EXERCISE
Iteration 1 Retro
10 Minutes

8. Slide 8
Let us look back
DISCUSSIONS
We make things good. We make things big. We make things fast.
REQUIREMENTS
Did you get your
User Stories right?
GROOMING
Typically they
„emerge“ with lots
of discussions
ELICITATION
Was it straight forward?
BACKLOG
Did you include
NFRs in your
backlog?

9. Slide 9
Let us look back on
what happened within
the Iteration
DISCUSSIONS
Plan
Build
Design
Test
Frame
Define Cycle
How were the
NFRs incorporated
in Iteration I?
How were these planned
to be verified?
Acceptance
Was the
Acceptance
smooth?

10. Slide 10
Nature of NFR testing
and defects pose a
challenge
CONTEXT
Complex set-up
Tools, test
organization and
execution is complex
Need for expert
services
Long Duration Tests
Tests take time with extended
monitoring
Specialized test
environment
Hard to reproduce
defects
Defect occurrance does
not follow a pattern
Design flaws
Defects may expose
architecture or design
flaws; Hard to fix

11. Slide 11
DEFINITION
NFRs can be
incorporated as individual
user stories.
Minor ones can be added
as „Acceptance Criteria“.
VERIFICATION
Test early – do what is possible.
Consider having parallel NFR test
iterations to the development
iterations where the previous
iterations software is subjected to
NFR tests.
Testing NFRs -
Recommendations
CONTEXT
FULL
VIEW
Final summary
testing at release
level.

12. Slide 12
Case in Point: Emergency call
Center product
EXAMPLE
An emergency call center application involving integration of multiple audio / VoIP
technologies. Has to comply with various safety and security standards.
Sprint 1
•NFR
environment
preparation
Sprint 2
•NFR Testing
with Sprint 1
output
Sprint 3
•NFR Testing
with Sprint 2
output
Defect Defect Defect Product
Backlog
Release
level
Test ing
Case Source: http://www.slideshare.net/salem_iyer/scrum-gathering-indiaregional2013loadandperformancetestsinagilescrumframeworkv4

13. Slide 13
Crispin’s Agile Testing
Quadrants
AGILE TESTING
Image Source: http://img3.douban.com/view/page_note/large/public/p14837373-1.jpg

14. Slide 14
Crispin’s Agile Testing
Quadrants – applied to
our discussion
AGILE TESTING
Within Iteration Outside of Iteration

15. Slide 15
Look out for contexts where -
CONTEXTUAL
FITMENT
Complex
Set-ups
Domain centric
Hardware Intensive
Safety Criticality
/ High Standards
compliance
Heterogenous
environment
Image Source: http://www.portlandwebworks.com/sites/default/files/stacey.png
Stacey‘s graph

16. QUESTIONS &
ANSWERS
Go ahead. Don‘t hesitate.

17. Slide 17
Let us stay in touch!
CONTACT
LinkedIn
linkedin.com/in/subrahmaniamsrv
Twitter
twitter.com/srv_subbu
E-Mail
Srv.subbu@yahoo.in

18. THANK
YOU
For your attention!