This document discusses the importance of non-functional testing (NFT) in software development. It defines NFT as testing aspects of a product that are not direct functional requirements, such as performance, reliability, security, and usability. The document recommends implementing NFT throughout the entire development lifecycle from development to post-production in order to reduce risks, costs, and improve quality. It provides examples of different types of NFT and emphasizes the resources required to properly perform serious NFT.

1. “Measure what is measurable and what is
not measurable – Make measurable”
-Galileo Galilei

2. ◦ Anything in the product behavior or surrounding
environments, which is not a functional aspect
◦ The functional execution of the product in order
to achieve “other” product requirements
◦ Focus on Performance, Dependability,
Operational aspects, production readiness issues
◦ Some methodologies and standards exist in the
market place (ISO 9126)

3. ◦ Functionality
 Interoperability , Security
◦ Reliability
 Availability , Recoverability
◦ Maintainability
◦ Usability
◦ Efficiency
 Performance aspects
◦ Portability
 Installability
Dependability

4. ◦ Complete and enhance Package testing as part of
product delivery readiness
◦ Improve quality of product in areas of performance,
stability, resilience, operability etc.
◦ Reduce production risks and costs associated with nonfunctional aspects of the product
◦ Optimize the way product is installed, setup,
configured, executes, managed and monitored
◦ Collect and produce measurements and metrics for
internal R&D use (Sizing, Capacity, targets etc.)
◦ Improve and enhance in-depth knowledge of the
product behavior, technologies in use
◦ Support Ad-Hoc requests for testing

5. 
Performance
◦ Volume test
◦ Load testing
◦ Stress (Load + Volume)
•
Availability & Resilience
•
H/A, BC ,DRP Testing
•
Operability/Inter-Operability
◦ Function under load
◦ Balancing/Parallelism/Scaling
•
Stability testing
•
Configuration
Installation & Upgrade
•
Architecture
•
Maintainability
 AKA Negative testing


Usability

Security
◦ Intrusion , Hardening
•
•
Cleanup / Purge
Conversion / Data Migration

6. Test environments
•
Long term use
•
•
Reuse environment from one release to another
Reflection of production
•
Customer data
•
Transactional data + History
•
All Server types
•
•
•
Application, Web, Communication
Auxiliary, Admin, Monitoring, Load servers
Large population of Customers, transactions and
historical data

7. Test environments – Continued
•
Server & Storage aspects
•
Multi-CPU, Multi-Core servers or Blades
•
Large amounts of Memory
•
Enterprise grade storage arrays
•
Large amount of storage
•
Multiple O/S environments
•
Virtualization
•
Clustering
•
•
O/S, Databases
Communications and Security devices

8. Test environments – Tools
•
In-house tools
•
•
Test automation and envelopes
•
•
Scripts
Application tools & Simulators
3rd party tools
•
Debuggers
•
Performance & Load testing tools
•
Profiling / Instrumentation tools
•
Memory management tools
•
Monitoring tools
•
Tuning tools
•
Simulators
•
Many others….

9. Administration
•
Direct Staffing
•
•
Environment & SCM
•
Application / SME’s
•
Testers
•
•
PM & Team Lead
Infrastructure
Additional Staff (Ad-hoc & Expert support)
•
Middleware
•
Performance
•
R&D Development
•
Application experts
•
Database Experts
•
System Unix/Linux/Microsoft
•
Storage guys
•
Etc…

12. Dev
QA
UAT
Prod
Post
Prod
No
Testing

13. Dev
QA
UAT
Prod
NFT
Post
Prod

14. Dev
QA
UAT
NFT
Prod
Post
Prod

15. Dev
QA
NFT
UAT
Prod
Post
Prod

16. Dev
NFT
QA
UAT
Prod
Post
Prod

17. Dev
QA
UAT
Prod
Post
Prod
NFT
NFT
NFT
NFT
NFT

18. Quick Recap
•
NFT is Critical for producing quality software
•
NFT has many categories and aspects
•
Choose the ones that are critical for your company
•
Finding and fixing problems late – is costly
•
NFT should be an integral part of the S/W Lifecycle
•
Make your company “NFT Aware”
•
Serious NFT is expensive to do
•
•
People, Equipment, Time & Effort, Overall costs
Method #6 seems to work best

19. Thank you !