This document discusses extra-functional testing and how testers can improve their ability to observe non-functional aspects of a system during functional testing. It covers key concepts like observability, and how to incorporate testing of performance, security, and accessibility. Specific tools are recommended for each quality factor, and ways to automate extra-functional tests are described. The presentation aims to help testers expand their skills and collaborate more with other testing teams.

1. abstracta.us
Extra-functional Testing:
Improve How you
Observe the System
During Functional
Testing
Federico Toledo
federico@abstracta.us

2. abstracta.us
abstracta.us
02. Non-Functional Aspects
03. Tools
04. Conclusions
Agenda
01. Concepts and Motivation

3. abstracta.us
abstracta.us/blog
federico-toledo.com
About me

4. abstracta.us
01.
Concepts and Motivation

5. abstracta.us
Functional vs Non-functional
Quality Factors

6. abstracta.us
Observability?
● The capability of a system to allow the determination of
its internal state from its inputs and outputs.
Useful to be prepared to solve unexpected problems in:
● Performance
● Security
● Bugs in production
Improve How you Observe the System During
Functional Testing

7. abstracta.us
Observability
Pillars:
Metrics
● KPIs: response times, CPU, memory,
error rates, etc.
Logs
● Text records with their timestamp.
Traces
● Breakdown of a request throughout all
the layers of the application.

8. abstracta.us
Can a tester observe
security, performance or
accessibility during their
functional testing?

9. abstracta.us
Building Bridges
The aim is not to dissolve specialized teams,
instead, to build bridges between teams
● Functional - Security
● Functional - Infrastructure
● Functional - Development
Neither is to lose focus
● Be mindful about what’s important for the
business, at each moment

10. abstracta.us
02.
Non-Functional Aspects

11. abstracta.us
What is it?
Amount of useful work achieved by a system
● Related to
○ Velocity
○ Throughput
● Resources usage
○ CPU, memory, disk, network, etc.
Performance

12. abstracta.us
Goal:
● Detect performance degradations in every test cycle
● Analize how performance limitations can affect functionality
● Test under load, volume testing, simulate or generate resource limitations
General Knowledge
● Client side versus server side performance
● Understand architectures, components and their role and most relevant KPIs
● Understand and breakdown reports or analysis
Tools to Try
● Google Lighthouse (inside the browser)
● Apptim (mobile testing)
● APMs (Datadog, Dynatrace, NewRelic, Grafana)
How to Incorporate Performance in the
Extra-Functional Testing?

13. abstracta.us
What is it?
Capacity to protect system data and resources
from access, damage or non authorized uses.
Related to:
● Confidentiality
● Information integrity and availability
Security

14. abstracta.us
How to Incorporate Security in the
Extra-Functional Testing?
Goal:
● Detect security vulnerabilities
● Detect non compliance with some standards (PCI, OWASP top 10, MASVS-L1, etc)
General Knowledge
● Applicable standards (PCI, OWASP, MASVS)
● Identify sensible information (passwords, bank information, credit card numbers, user data, etc)
● Knowledge of most common attacks and vulnerabilities (SQL injection, man in the middle, phishing, etc)
Tools to Try
● Web/Backend: OWASP ZAP
● Mobile: Mob SF

15. abstracta.us
What is it?
Capacity of a system of being accessed by all the
people in an effective way, without barriers,
regardless of their abilities or disabilities and
their context.
Accessibility

16. abstracta.us
How to Incorporate Accessibility in the
Extra-Functional Testing?
Goal:
● Detect accessibility issues
● Detect non compliance with a standard (WCAG, ADA, etc.)
General Knowledge
● Applicable standards (WCAG, ADA, etc.)
● Understand how support tools work and alternative uses of the system in accessibility mode (only
keyboard, screen readers, magnifiers, etc)
● Understand how a system must be adjusted in order to effectively use a support tool (ARIA labels,
alternative text, tabulation order, etc)
Tools to Try
● Google Lighthouse (in the browser)
● WAVE
● axe DevTools
● Colour contrast analyzer

17. abstracta.us
03.
Tools

18. abstracta.us
Lighthouse

19. abstracta.us
Apptim

20. abstracta.us
Owasp - ZAP

21. abstracta.us
Extra-Functional Test Automation
Performance
● Apptim, DevTools:
○ API to access performance metrics
○ Assert deviations with previous versions or comparing to an SLO
● JMeter DSL:
○ Convert Selenium scripts to JMeter
Security
● OWASP ZAP:
○ Run audits during automated test runs to detect new issues or threats
Accessibility
● Mabl:
○ Low-code test automation tool, includes accessibility checks in runtime
● Axe:
○ API to include accessibility audits to your scripts

22. abstracta.us
Tester Copilot,
how would that
be?

23. abstracta.us
04.
Conclusions

24. abstracta.us
The Path to Become
an Extra-Functional
Tester
● Learn the basic concepts
● Learn the tools
● Integrate it to your functional testing
● Understand business priorities
● Report considering priorities
● Connect, build bridges
Cursos online de
testing de software
Aprende desde el lugar que te
encuentres, con los mejores instructores
y a tu propio ritmo

25. Food for thought
Guides
● Continuous Testing Guide
Blogposts
● De tester funcional a tester de performance
● ¿Qué es la observabilidad y cuál es su importancia para el testing?
● An End-to-End Guide of Load Testing
● How to Make Testing Progress Visible
Quality Sense Podcasts
● Leandro Melendez - Performance Testing Explained Simple
● Sofia Palamarchuk - Mobile App Performance

26. abstracta.us
Extra-functional Testing:
Improve How you Observe
the System During
Functional Testing
Federico Toledo
federico@abstracta.us
¡Gracias!