We typically distinguish between functional and non-functional testing, which might mislead to under prioritize some important aspects of the quality of the application we are testing. In many cases when the system is not secure, performant or accessible, its functionality is affected or it’s not functional at all. In this talk, I will show techniques and tools that we use that will help you improve your ability to observe the system while you are performing functional testing in order to provide feedback about the so-called “non-functional” properties. I will also discuss how to properly prioritize the different characteristics of the system in order to focus your efforts on what is more important to your business at each moment.

1. abstracta.us
Extra-functional Testing:
Improve How you
Observe the System
During Functional Testing
Federico Toledo
Co-founder and Chief Quality Officer
federico@abstracta.us

2. abstracta.us
abstracta.us/blog
federico-toledo.com
About me

3. Mabl
/ USA
#QSConf2023
+20 November 9th & 10th
2 SPEAKERS TRACKS | ONLINE + IN PERSON
International thought
leaders
Free for attendees!
Call for Sponsors is open
Call for Speakers opens soon

4. abstracta.us
80% of the testing focuses on the
system’s functionality
Non-functional testing is often left for later:
● For when the system is more stable
● For when we are more mature
● Right before the release
● After the first version, for the optimization stage
Motivation

5. abstracta.us
● Demystify non-functional testing
● Share basic concepts for performance, security and accessibility
● Share some tools to help you kick off
● Discuss if the approach would make sense in your context
Goals

6. abstracta.us
01.
Introduction

7. Functional vs Non-functional
Quality Factors
Based on

8. abstracta.us
Is it really “non” functional?
Quality Factors
Non-Functional?
Para-Functional
Quasi-Functional
Extra-Functional

9. abstracta.us
Interrelation
Interrelation / Interconnection
Quality Factors
We can’t test quality
factors in isolation
We need to have a holistic
approach to quality

10. abstracta.us
What’s your quality strategy?
Build bridges between teams
Don’t lose focus from your main goal
● Be mindful about what’s important for the
business, at each moment
Unveil Risks Sooner

11. abstracta.us
02.
Extra-Functional Aspects:
Performance

12. abstracta.us
What is it?
Amount of useful work achieved by a system
● Related to
○ Velocity
○ Response times
○ Throughput
● Resources usage
○ CPU
○ Memory
○ Disk
○ Network
○ Etc.
Performance

13. abstracta.us
Goal:
● Detect performance degradations, risks or improvement
opportunities in every test cycle
How?
● Using tools to gather metrics and compare with expected
(or historical) behavior
● Using tools to perform automatic analysis
How to Incorporate Performance
into Functional Testing?

14. abstracta.us
Good to Know:
● Understand your system’s architecture, components and
their role and most relevant KPIs
● Have access to the tools your team is already using
○ Understand reports and be able to drill down
How to Incorporate Performance
into Functional Testing?

15. abstracta.us
● Test under load
● Volume of the testing database
● Analize how performance limitations can affect functionality
○ Simulate or generate resource limitations
Good to Know:
● Client side versus server side performance
Where Performance
meets Functionality

16. abstracta.us
Some Tools to
Try for Observing
Performance

17. abstracta.us
Performance Tools:
Lighthouse for Web
DEMO

18. abstracta.us
Performance Tools:
ChatGPT plugin for Website
Performance

19. abstracta.us
Performance Tools:
Apptim for Mobile
DEMO

20. abstracta.us
Performance Tools:
Application Performance
Management Tools (APMs)
DEMO

21. abstracta.us
How to Incorporate
Performance into Test
Automation?
Collect response times
● Analyze trends, assertions to identify degradations
Chrome DevTools:
● API to access performance metrics
● Assert deviations with previous versions or
comparing to an SLO
● Example using Chrome DevTools from Playwright
https://github.com/unlikelyzero/js-perf-toolkit

22. abstracta.us
How to Incorporate
Performance into Test
Automation?
From Functional Test Automation to
Performance Simulation
JMeter DSL:
● Convert Selenium scripts to JMeter
● Demo (repo)
Mabl:
● Reusing API testing to run performance tests

23. abstracta.us
02.
Extra-Functional Aspects:
Security

24. abstracta.us
What is it?
Capacity to protect system data and resources
from access, damage or non authorized uses.
Related to:
● Confidentiality
● Information integrity and availability
Security

25. abstracta.us
How to Incorporate Security
into Functional Testing?
Goal:
● Detect security vulnerabilities or risks
● Detect non compliance with standards
How?
● Using tools to scan for vulnerabilities
● Functional testing on access control features

26. abstracta.us
Where Security meets
Functionality
● Functional tests on the login, and access control (users, roles)
● Error handling (which information is exposed, i.e. stack traces)
● Input validation (SQL injection, prompt injection)
Good to Know:
● Most common attacks and vulnerabilities (SQL injection,
phishing, prompt injection, etc.)
● Applicable standards (PCI, OWASP, MASVS)
● Identify sensitive information (passwords, financial or medical
information, credit card numbers, etc)

27. abstracta.us
Some Tools to
Try for Observing
Security

28. abstracta.us
DEMO
Security Tools:
OWASP - ZAP

29. abstracta.us
Security Tools (mobile):
Mob SF - dynamic analysis

30. abstracta.us
How to Incorporate Security
into Test Automation?
OWASP ZAP:
● Run audits during automated test runs to detect new issues or threats
● Setting OWASP ZAP as a proxy in the webdriver

31. abstracta.us
How to Incorporate Security
into Test Automation?

32. abstracta.us
02.
Extra-Functional Aspects:
Accessibility

33. abstracta.us
What is it?
Capacity of a system of being accessed by all the
people in an effective way, without barriers,
regardless of their abilities or disabilities and
their context.
Accessibility

34. abstracta.us
How to Incorporate Accessibility
into Functional Testing?
Goal:
● Detect accessibility issues
● Detect non compliance with standards
How?
● Using tools to perform accessibility audits
● Manually verifying accessibility criteria

35. abstracta.us
How to Incorporate Accessibility
into Functional Testing?
Good to Know:
● Applicable standards (WCAG, ADA, etc.)
● Understand how support tools work and alternative uses of the
system in accessibility mode (only keyboard, screen readers,
magnifiers, etc)
● Understand how a system must be adjusted in order to effectively use
a support tool (ARIA labels, alternative text, tabulation order, etc)

36. abstracta.us
● Power users can be more efficient
● Usability improves accessibility and vice versa
○ Check usability, use Nielsen Heuristics
Good to Know:
● Applicable standards (WCAG, ADA, etc.)
● Support tools and accessibility mode (only keyboard, screen
readers, magnifiers, etc)
● What’s needed to make them work well (ARIA labels,
alternative text, tabulation order, etc.)
Where Accessibility meets
Functionality

37. abstracta.us
Some Tools to
Try for Observing
Accessibility

38. abstracta.us
Accessibility Tools:
Lighthouse for Web

39. abstracta.us
Accessibility Tools:
axe DevTools
DEMO

40. abstracta.us
Accessibility Tools:
WAVE (Web Accessibility
Evaluation Tool)

41. abstracta.us
● Keyboard only
● Screen reader
“Accessibility is not about compliance with a
standard or adding ARIA labels, it’s about
removing barriers.”
Accessibility Tools:

42. abstracta.us
How to Incorporate Accessibility
into Test Automation?
● axe DevTools API:
API to include accessibility audits to your scripts
● Mabl:
Low-code test automation tool, includes accessibility checks in runtime

43. abstracta.us
03.
Final Thoughts

44. abstracta.us
Tester Copilot,
how would that
be?

45. abstracta.us
● Extra-functional testing is not rocket science
○ Learn the basic concepts
○ Get familiar with the tools (Lighthouse, Apptim, ZAP, axe, etc.)
○ Share learnings with your team
● Integrate it to your functional testing and test automation
○ Start simple
○ Filter what’s important
● Understand business priorities
● Report considering priorities
● Connect, build bridges
Extra-Functional Testing,
Final Thoughts:

46. abstracta.us
Food for Thought
Guides
● Continuous Testing Guide
Blogposts
● From Functional Tester to Performance Tester
● What is Observability and How is it Relevant to Software Testing
● An End-to-End Guide of Load Testing
● How to Make Testing Progress Visible
● How to Take the Security of your Mobile Apps to the Next Level of OWASP
Quality Sense Podcasts
● Leandro Melendez - Performance Testing Explained Simple
● Sofia Palamarchuk - Mobile App Performance

47. abstracta.us
Let’s discuss if the
approach makes sense in
your context
Federico Toledo
federico@abstracta.us
¡Gracias!
Extra-functional Testing:
Improve How you Observe the System During Functional Testing