SlideShare a Scribd company logo
1 of 28
QA is not Quality
@antoligy
@antoligy
@antoligy
@antoligy
Who am I?
“Web performance advocate and aspiring polyglot”
Twitter — @antoligy
Web — alexwilson.tech
@antoligy
Take-away
● What Software Quality actually is
● How we can measure it
● Practical ways to improve the quality of what we’re building
@antoligy
So what actually is quality?
The difficulty in defining quality is to translate future needs of the user into measurable characteristics, so that a
product can be designed and turned out to give satisfaction at a price that the user will pay.
— William E Deming
Let’s ask the experts!
@antoligy
The word quality has multiple meanings. Two of these meanings dominate the use of the word: 1. Quality consists of
those product features which meet the need of customers and thereby provide product satisfaction. 2. Quality
consists of freedom from deficiencies. Nevertheless, in a handbook such as this it is convenient to standardize on a
short definition of the word quality as "fitness for use".
— Joseph M Juran
Quality is a customer determination, not an engineer's determination, not a marketing determination, nor a general
management determination. It is based on the customer's actual experience with the product or service, measured
against his or her requirements — stated or unstated, conscious or merely sensed, technically operational or entirely
subjective — and always representing a moving target in a competitive market.
— Armand V Feigenbaum
● Being “fit for purpose”, and serving a user need.
● Free from deficiencies.
● Being able to easily meet the needs of the future.
In simpler terms...
@antoligy
Quality Model: ISO-25000 (SQuaRE)
@antoligy
What is it?
A system provides functions that meet stated and implied needs when used under specified
conditions
How do you measure it?
● Functional completeness: CSAT, Traceability matrix (requirements vs test coverage)
Functional Suitability
@antoligy
How do you improve it?
● Hire a Business Analyst and QA to define complete functional requirements
● Implement user-testing
Example
The time we reverse engineered a product and its functional requirements while changing its
technology stack: It would have been far easier if we had mapped out what we knew.
Functional Suitability
@antoligy
What is it?
The performance relative to the amount of resources used under stated conditions
How do you measure it?
● “Core Metrics” — # Latency, # Request-Rate, # Errors
● User-centric metrics — # Time to interactive, # Time to first meaningful interaction
● Capacity — Load Testing
● Resource Utilisation — Tracing
Performance Efficiency
@antoligy
Performance Efficiency
@antoligy
How do you improve it?
● Choose the right technologies for the problems you are trying to solve
● Instrument “core metrics” across your stack.
● Load-test under realistic scenarios.
● Link performance improvements to business metrics (e.g. conversion)
Example
The time a website’s performance problems for years turned out to be an unnoticed,
unnecessary synchronous API callout blocking every request.
Usability
@antoligy
What is it?
A system can be used by specified users to achieve specific goals with effectiveness,
efficiency and satisfaction in a specified context of use
How do you measure it?
● Accessibility — WCAG 2.1 score
● User-friendliness — # User error rates, # L1 support-tickets, CSAT
● Operability — # Production incidents
Usability
How do you improve it?
● Automate accessibility testing
● Regularly conduct user interviews and perform user research
● Test on real user devices
● Invest in onboarding every type of user a system or product might have
Example
The time I built multi-language support into an events website to improve the accessibility for
its non-English-speaking users, but, didn’t train or make it easy for administrators to add new
locales.
@antoligy
Reliability
What is it?
A component functions consistently under specified conditions for a specified period of time
How do you measure it?
● Stability — Uptime, # of reported outages
● Availability — MTBF (mean-time-between-failures: mean-time-to-failure + mean-time-
to-repair)
@antoligy
Reliability
@antoligy
How do you improve it?
● Chaos-suite: Simulate disasters and failures before they hit you
● Knowledge-sharing exercises in the technology stack
● Apply SLAs and circuit-breakers to dependencies
● Instrument applications and services
● Invest in Observability tooling
Example
The time Puppet accidentally deleted a production database and server, but because we
used Puppet and orchestrated every aspect of AWS, we were able to recover within two
hours.
Security
What is it?
A system protects information and data so that persons or other products or systems have
the degree of data access appropriate to their types and levels of authorization
How do you measure it?
● Risk — OWASP Risk Rating (Likelihood * Impact)
● Auditability — # of user actions not recorded/attributed to a user
● Integrity — # of areas where PII may be found
● Compliance with standards such as PCI-DSS
@antoligy
How do you improve it?
● Scan dependencies for known vulnerabilities (e.g. Snyk)
● Vulnerability scanning of entire system (e.g. automated like Qualys)
● Bug-Bounties
● Role-Based-Access-Control, and Attribute-Based-Access-Control
Example
The time a Drupal website was hacked and would randomly serve cialis adverts to search-
bots only.
Security
@antoligy
What is it?
A system can be efficiently and effectively modified to improve it, correct it or adapt it to
changes in environment, and in requirements
How do you measure it?
● Testability — % unit code coverage, % scenario coverage, % feature coverage
● Understandability — # of lines of code, average file length, adherence to conventions
● Modifiability — Cyclomatic complexity, Cohesion
Maintainability
@antoligy
How do you improve it?
● Invest heavily in developer experience
● Lean on unit-testing to improve developer confidence
● Invest in code-review automation (e.g. Codacy)
● Reduce # caching layers and improve their utilisation
● Document your systems inside-out
● Invest in Observability tooling
Example
The time when revisiting our architectural documentation and increasing the verbosity of our
logging helped us diagnose an intermittent problem as being Redis replicating data
asynchronously.
Maintainability
@antoligy
Portability
What is it?
A system, product or component can be transferred from
one hardware, software or other operational or usage
environment to another.
How do you improve it?
Containerize
Portability & Compatibility
I’ll be honest: These are not things I think I have ever successfully solved. They are included for “completeness”.
Compatibility
What is it?
A component can exchange information with other
components, and/or perform its required functions, while
sharing the same hardware or software environment.
How do you improve it?
Namespacing, Containerize
@antoligy
Practical ways of improving quality
There are also some general principles we can implement.
● Defining “Done”
● Experiment and iterate
● Test in production
Practical Suggestions: Define “Done”
@antoligy
● Being dogmatic about every principle of quality will get us nowhere fast, so
prioritise based on what is important for our users.
● Periodically revisit this definition as a team, asking:
○ Do our stakeholders and users understand how this works?
○ Do we think this has helped improve our quality?
○ Do our users feel the same way?
● Work with “quality champions” early-on to define how new bodies of work will
be tested.
https://www.scrum.org/resources/blog/walking-through-definition-done
Practical Suggestions: Experiment and iterate!
@antoligy
● Get feedback from stakeholders and users early and often.
○ As quality improves, it will be easier to manage further changes upwards.
○ Team-members and developers in other areas of the business are stakeholders
too.
● Use a model of quality to measure and experiment with process and tooling,
and don’t be afraid to make mistakes.
● Link chosen metrics to tangible, easy-to-understand goals, and evaluate
success based on trends.
Practical Suggestions: Test in production
@antoligy
● Production is the only environment where we can prove things work.
● As well as testing safely out of production, make it safe to test in production.
○ Feature-flagging to decouple roll-out of features, from roll-out of code, so manual
tests are possible in prod before showing users a broken experience.
○ Canary releases to test with small percentage of user-base first.
○ Synthetic monitoring of critical features and journeys.
○ Especially on the web: Enhance progressively, and degrade gracefully. (i.e. if
JavaScript errors, continue serving critical functionalities)
● We’re already doing this every time we push to production!
https://martinfowler.com/articles/qa-in-production.html
Thanks for listening!
Any questions?
@antoligy

More Related Content

What's hot

Agile testing presentation
Agile testing presentationAgile testing presentation
Agile testing presentationKomal Garg
 
Tackling software testing challenges in the agile era
Tackling software testing challenges in the agile eraTackling software testing challenges in the agile era
Tackling software testing challenges in the agile eraQASymphony
 
Introduction to BDD
Introduction to BDDIntroduction to BDD
Introduction to BDDKnoldus Inc.
 
Testing in Agile Projects
Testing in Agile ProjectsTesting in Agile Projects
Testing in Agile Projectssriks7
 
Acceptance criteria
Acceptance criteriaAcceptance criteria
Acceptance criteriaSoftheme
 
Software Quality Assurance (QA) Testing Interview Questions & Answers
Software Quality Assurance (QA) Testing Interview Questions & AnswersSoftware Quality Assurance (QA) Testing Interview Questions & Answers
Software Quality Assurance (QA) Testing Interview Questions & AnswersJanBask Training
 
An introduction to Behavior-Driven Development (BDD)
An introduction to Behavior-Driven Development (BDD)An introduction to Behavior-Driven Development (BDD)
An introduction to Behavior-Driven Development (BDD)Suman Guha
 
QA Interview Questions With Answers
QA Interview Questions With AnswersQA Interview Questions With Answers
QA Interview Questions With AnswersH2Kinfosys
 
Agile tour ncr test360_degree - agile testing on steroids
Agile tour ncr test360_degree - agile testing on steroidsAgile tour ncr test360_degree - agile testing on steroids
Agile tour ncr test360_degree - agile testing on steroidsVipul Gupta
 
Agile Testing Transformation is as Easy as 1, 2, 3 by Michael Buening
Agile Testing Transformation is as Easy as 1, 2, 3 by Michael BueningAgile Testing Transformation is as Easy as 1, 2, 3 by Michael Buening
Agile Testing Transformation is as Easy as 1, 2, 3 by Michael BueningQA or the Highway
 
Test Automation Strategies and Frameworks: What Should Your Team Do?
Test Automation Strategies and Frameworks: What Should Your Team Do?Test Automation Strategies and Frameworks: What Should Your Team Do?
Test Automation Strategies and Frameworks: What Should Your Team Do?TechWell
 
Behavior Driven Development Pros and Cons
Behavior Driven Development Pros and ConsBehavior Driven Development Pros and Cons
Behavior Driven Development Pros and ConsIosif Itkin
 
Introduction to Agile Testing
Introduction to Agile TestingIntroduction to Agile Testing
Introduction to Agile TestingvodQA
 
Agile Test Automation: Truth, Oxymoron or Lie?
Agile Test Automation: Truth, Oxymoron or Lie?Agile Test Automation: Truth, Oxymoron or Lie?
Agile Test Automation: Truth, Oxymoron or Lie?Original Software
 
IIT Academy: 204 User stories and acceptance criteria
IIT Academy: 204 User stories and acceptance criteriaIIT Academy: 204 User stories and acceptance criteria
IIT Academy: 204 User stories and acceptance criteriaSteven HK Ma | 馬國豪
 
Is Test Planning a lost art in Agile? by Michelle Williams
Is Test Planning a lost art in Agile? by Michelle WilliamsIs Test Planning a lost art in Agile? by Michelle Williams
Is Test Planning a lost art in Agile? by Michelle WilliamsQA or the Highway
 

What's hot (20)

Agile testing presentation
Agile testing presentationAgile testing presentation
Agile testing presentation
 
Tackling software testing challenges in the agile era
Tackling software testing challenges in the agile eraTackling software testing challenges in the agile era
Tackling software testing challenges in the agile era
 
Introduction to BDD
Introduction to BDDIntroduction to BDD
Introduction to BDD
 
Testing in Agile Projects
Testing in Agile ProjectsTesting in Agile Projects
Testing in Agile Projects
 
Acceptance criteria
Acceptance criteriaAcceptance criteria
Acceptance criteria
 
Software Quality Assurance (QA) Testing Interview Questions & Answers
Software Quality Assurance (QA) Testing Interview Questions & AnswersSoftware Quality Assurance (QA) Testing Interview Questions & Answers
Software Quality Assurance (QA) Testing Interview Questions & Answers
 
Beginners QA Testing
Beginners QA TestingBeginners QA Testing
Beginners QA Testing
 
An introduction to Behavior-Driven Development (BDD)
An introduction to Behavior-Driven Development (BDD)An introduction to Behavior-Driven Development (BDD)
An introduction to Behavior-Driven Development (BDD)
 
QA Interview Questions With Answers
QA Interview Questions With AnswersQA Interview Questions With Answers
QA Interview Questions With Answers
 
Agile tour ncr test360_degree - agile testing on steroids
Agile tour ncr test360_degree - agile testing on steroidsAgile tour ncr test360_degree - agile testing on steroids
Agile tour ncr test360_degree - agile testing on steroids
 
Agile Testing Transformation is as Easy as 1, 2, 3 by Michael Buening
Agile Testing Transformation is as Easy as 1, 2, 3 by Michael BueningAgile Testing Transformation is as Easy as 1, 2, 3 by Michael Buening
Agile Testing Transformation is as Easy as 1, 2, 3 by Michael Buening
 
Top 10 Qualities of a QA Tester
Top 10 Qualities of a QA TesterTop 10 Qualities of a QA Tester
Top 10 Qualities of a QA Tester
 
Introduction to Agile Testing
Introduction to Agile TestingIntroduction to Agile Testing
Introduction to Agile Testing
 
Test Automation Strategies and Frameworks: What Should Your Team Do?
Test Automation Strategies and Frameworks: What Should Your Team Do?Test Automation Strategies and Frameworks: What Should Your Team Do?
Test Automation Strategies and Frameworks: What Should Your Team Do?
 
Behavior Driven Development Pros and Cons
Behavior Driven Development Pros and ConsBehavior Driven Development Pros and Cons
Behavior Driven Development Pros and Cons
 
New model
New modelNew model
New model
 
Introduction to Agile Testing
Introduction to Agile TestingIntroduction to Agile Testing
Introduction to Agile Testing
 
Agile Test Automation: Truth, Oxymoron or Lie?
Agile Test Automation: Truth, Oxymoron or Lie?Agile Test Automation: Truth, Oxymoron or Lie?
Agile Test Automation: Truth, Oxymoron or Lie?
 
IIT Academy: 204 User stories and acceptance criteria
IIT Academy: 204 User stories and acceptance criteriaIIT Academy: 204 User stories and acceptance criteria
IIT Academy: 204 User stories and acceptance criteria
 
Is Test Planning a lost art in Agile? by Michelle Williams
Is Test Planning a lost art in Agile? by Michelle WilliamsIs Test Planning a lost art in Agile? by Michelle Williams
Is Test Planning a lost art in Agile? by Michelle Williams
 

Similar to QA is not quality

Requirement management presentation to a software team
Requirement management presentation to a software teamRequirement management presentation to a software team
Requirement management presentation to a software teamrchakra
 
Independent verification & validation presented by Maneat v02
Independent verification & validation presented by Maneat v02Independent verification & validation presented by Maneat v02
Independent verification & validation presented by Maneat v02Dr. Pierpaolo Mangeruga
 
Software Test Engineer with 3.6 years of experience
Software Test Engineer with 3.6 years of experienceSoftware Test Engineer with 3.6 years of experience
Software Test Engineer with 3.6 years of experiencevijaya lakshmi
 
Effective User Story Writing
Effective User Story WritingEffective User Story Writing
Effective User Story WritingAhmed Misbah
 
3 Keys to Performance Testing at the Speed of Agile
3 Keys to Performance Testing at the Speed of Agile3 Keys to Performance Testing at the Speed of Agile
3 Keys to Performance Testing at the Speed of AgileNeotys
 
Owasp summit slides day 2
Owasp summit slides day 2Owasp summit slides day 2
Owasp summit slides day 2Dinis Cruz
 
Failure is an Option: Scaling Resilient Feature Delivery
Failure is an Option: Scaling Resilient Feature DeliveryFailure is an Option: Scaling Resilient Feature Delivery
Failure is an Option: Scaling Resilient Feature DeliveryOptimizely
 
What is Software Quality and how to measure it?
What is Software Quality and how to measure it?What is Software Quality and how to measure it?
What is Software Quality and how to measure it?Denys Zaiats
 
Quality at the speed of digital
Quality   at the speed of digitalQuality   at the speed of digital
Quality at the speed of digitalrajni singh
 
Software quality
Software qualitySoftware quality
Software qualityjagadeesan
 
#ATAGTR2019 Presentation "Top 10 quality engineering best practices to achiev...
#ATAGTR2019 Presentation "Top 10 quality engineering best practices to achiev...#ATAGTR2019 Presentation "Top 10 quality engineering best practices to achiev...
#ATAGTR2019 Presentation "Top 10 quality engineering best practices to achiev...Agile Testing Alliance
 
How to build confidence in your release cycle
How to build confidence in your release cycleHow to build confidence in your release cycle
How to build confidence in your release cycleDiUS
 
Software Quality for Developers
Software Quality for DevelopersSoftware Quality for Developers
Software Quality for DevelopersMaira Bay de Souza
 

Similar to QA is not quality (20)

Agile case studies
Agile case studiesAgile case studies
Agile case studies
 
Requirement management presentation to a software team
Requirement management presentation to a software teamRequirement management presentation to a software team
Requirement management presentation to a software team
 
Independent verification & validation presented by Maneat v02
Independent verification & validation presented by Maneat v02Independent verification & validation presented by Maneat v02
Independent verification & validation presented by Maneat v02
 
Software Test Engineer with 3.6 years of experience
Software Test Engineer with 3.6 years of experienceSoftware Test Engineer with 3.6 years of experience
Software Test Engineer with 3.6 years of experience
 
Effective User Story Writing
Effective User Story WritingEffective User Story Writing
Effective User Story Writing
 
BAAgileQA
BAAgileQABAAgileQA
BAAgileQA
 
3 Keys to Performance Testing at the Speed of Agile
3 Keys to Performance Testing at the Speed of Agile3 Keys to Performance Testing at the Speed of Agile
3 Keys to Performance Testing at the Speed of Agile
 
Resume
ResumeResume
Resume
 
QA in Agile World
QA in Agile WorldQA in Agile World
QA in Agile World
 
Owasp summit slides day 2
Owasp summit slides day 2Owasp summit slides day 2
Owasp summit slides day 2
 
Failure is an Option: Scaling Resilient Feature Delivery
Failure is an Option: Scaling Resilient Feature DeliveryFailure is an Option: Scaling Resilient Feature Delivery
Failure is an Option: Scaling Resilient Feature Delivery
 
What is Software Quality and how to measure it?
What is Software Quality and how to measure it?What is Software Quality and how to measure it?
What is Software Quality and how to measure it?
 
Quality at the speed of digital
Quality   at the speed of digitalQuality   at the speed of digital
Quality at the speed of digital
 
Software quality
Software qualitySoftware quality
Software quality
 
Software testing
Software testingSoftware testing
Software testing
 
#ATAGTR2019 Presentation "Top 10 quality engineering best practices to achiev...
#ATAGTR2019 Presentation "Top 10 quality engineering best practices to achiev...#ATAGTR2019 Presentation "Top 10 quality engineering best practices to achiev...
#ATAGTR2019 Presentation "Top 10 quality engineering best practices to achiev...
 
How to build confidence in your release cycle
How to build confidence in your release cycleHow to build confidence in your release cycle
How to build confidence in your release cycle
 
Agile testing
Agile testingAgile testing
Agile testing
 
Software Quality for Developers
Software Quality for DevelopersSoftware Quality for Developers
Software Quality for Developers
 
Agile testing
Agile testingAgile testing
Agile testing
 

Recently uploaded

UI5ers live - Custom Controls wrapping 3rd-party libs.pptx
UI5ers live - Custom Controls wrapping 3rd-party libs.pptxUI5ers live - Custom Controls wrapping 3rd-party libs.pptx
UI5ers live - Custom Controls wrapping 3rd-party libs.pptxAndreas Kunz
 
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...OnePlan Solutions
 
Understanding Flamingo - DeepMind's VLM Architecture
Understanding Flamingo - DeepMind's VLM ArchitectureUnderstanding Flamingo - DeepMind's VLM Architecture
Understanding Flamingo - DeepMind's VLM Architecturerahul_net
 
SAM Training Session - How to use EXCEL ?
SAM Training Session - How to use EXCEL ?SAM Training Session - How to use EXCEL ?
SAM Training Session - How to use EXCEL ?Alexandre Beguel
 
What’s New in VictoriaMetrics: Q1 2024 Updates
What’s New in VictoriaMetrics: Q1 2024 UpdatesWhat’s New in VictoriaMetrics: Q1 2024 Updates
What’s New in VictoriaMetrics: Q1 2024 UpdatesVictoriaMetrics
 
SoftTeco - Software Development Company Profile
SoftTeco - Software Development Company ProfileSoftTeco - Software Development Company Profile
SoftTeco - Software Development Company Profileakrivarotava
 
2024 DevNexus Patterns for Resiliency: Shuffle shards
2024 DevNexus Patterns for Resiliency: Shuffle shards2024 DevNexus Patterns for Resiliency: Shuffle shards
2024 DevNexus Patterns for Resiliency: Shuffle shardsChristopher Curtin
 
SpotFlow: Tracking Method Calls and States at Runtime
SpotFlow: Tracking Method Calls and States at RuntimeSpotFlow: Tracking Method Calls and States at Runtime
SpotFlow: Tracking Method Calls and States at Runtimeandrehoraa
 
Not a Kubernetes fan? The state of PaaS in 2024
Not a Kubernetes fan? The state of PaaS in 2024Not a Kubernetes fan? The state of PaaS in 2024
Not a Kubernetes fan? The state of PaaS in 2024Anthony Dahanne
 
Comparing Linux OS Image Update Models - EOSS 2024.pdf
Comparing Linux OS Image Update Models - EOSS 2024.pdfComparing Linux OS Image Update Models - EOSS 2024.pdf
Comparing Linux OS Image Update Models - EOSS 2024.pdfDrew Moseley
 
Effectively Troubleshoot 9 Types of OutOfMemoryError
Effectively Troubleshoot 9 Types of OutOfMemoryErrorEffectively Troubleshoot 9 Types of OutOfMemoryError
Effectively Troubleshoot 9 Types of OutOfMemoryErrorTier1 app
 
SensoDat: Simulation-based Sensor Dataset of Self-driving Cars
SensoDat: Simulation-based Sensor Dataset of Self-driving CarsSensoDat: Simulation-based Sensor Dataset of Self-driving Cars
SensoDat: Simulation-based Sensor Dataset of Self-driving CarsChristian Birchler
 
Ronisha Informatics Private Limited Catalogue
Ronisha Informatics Private Limited CatalogueRonisha Informatics Private Limited Catalogue
Ronisha Informatics Private Limited Catalogueitservices996
 
VK Business Profile - provides IT solutions and Web Development
VK Business Profile - provides IT solutions and Web DevelopmentVK Business Profile - provides IT solutions and Web Development
VK Business Profile - provides IT solutions and Web Developmentvyaparkranti
 
Simplifying Microservices & Apps - The art of effortless development - Meetup...
Simplifying Microservices & Apps - The art of effortless development - Meetup...Simplifying Microservices & Apps - The art of effortless development - Meetup...
Simplifying Microservices & Apps - The art of effortless development - Meetup...Rob Geurden
 
Large Language Models for Test Case Evolution and Repair
Large Language Models for Test Case Evolution and RepairLarge Language Models for Test Case Evolution and Repair
Large Language Models for Test Case Evolution and RepairLionel Briand
 
How to submit a standout Adobe Champion Application
How to submit a standout Adobe Champion ApplicationHow to submit a standout Adobe Champion Application
How to submit a standout Adobe Champion ApplicationBradBedford3
 
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...Angel Borroy López
 
Post Quantum Cryptography – The Impact on Identity
Post Quantum Cryptography – The Impact on IdentityPost Quantum Cryptography – The Impact on Identity
Post Quantum Cryptography – The Impact on Identityteam-WIBU
 
Odoo 14 - eLearning Module In Odoo 14 Enterprise
Odoo 14 - eLearning Module In Odoo 14 EnterpriseOdoo 14 - eLearning Module In Odoo 14 Enterprise
Odoo 14 - eLearning Module In Odoo 14 Enterprisepreethippts
 

Recently uploaded (20)

UI5ers live - Custom Controls wrapping 3rd-party libs.pptx
UI5ers live - Custom Controls wrapping 3rd-party libs.pptxUI5ers live - Custom Controls wrapping 3rd-party libs.pptx
UI5ers live - Custom Controls wrapping 3rd-party libs.pptx
 
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...
 
Understanding Flamingo - DeepMind's VLM Architecture
Understanding Flamingo - DeepMind's VLM ArchitectureUnderstanding Flamingo - DeepMind's VLM Architecture
Understanding Flamingo - DeepMind's VLM Architecture
 
SAM Training Session - How to use EXCEL ?
SAM Training Session - How to use EXCEL ?SAM Training Session - How to use EXCEL ?
SAM Training Session - How to use EXCEL ?
 
What’s New in VictoriaMetrics: Q1 2024 Updates
What’s New in VictoriaMetrics: Q1 2024 UpdatesWhat’s New in VictoriaMetrics: Q1 2024 Updates
What’s New in VictoriaMetrics: Q1 2024 Updates
 
SoftTeco - Software Development Company Profile
SoftTeco - Software Development Company ProfileSoftTeco - Software Development Company Profile
SoftTeco - Software Development Company Profile
 
2024 DevNexus Patterns for Resiliency: Shuffle shards
2024 DevNexus Patterns for Resiliency: Shuffle shards2024 DevNexus Patterns for Resiliency: Shuffle shards
2024 DevNexus Patterns for Resiliency: Shuffle shards
 
SpotFlow: Tracking Method Calls and States at Runtime
SpotFlow: Tracking Method Calls and States at RuntimeSpotFlow: Tracking Method Calls and States at Runtime
SpotFlow: Tracking Method Calls and States at Runtime
 
Not a Kubernetes fan? The state of PaaS in 2024
Not a Kubernetes fan? The state of PaaS in 2024Not a Kubernetes fan? The state of PaaS in 2024
Not a Kubernetes fan? The state of PaaS in 2024
 
Comparing Linux OS Image Update Models - EOSS 2024.pdf
Comparing Linux OS Image Update Models - EOSS 2024.pdfComparing Linux OS Image Update Models - EOSS 2024.pdf
Comparing Linux OS Image Update Models - EOSS 2024.pdf
 
Effectively Troubleshoot 9 Types of OutOfMemoryError
Effectively Troubleshoot 9 Types of OutOfMemoryErrorEffectively Troubleshoot 9 Types of OutOfMemoryError
Effectively Troubleshoot 9 Types of OutOfMemoryError
 
SensoDat: Simulation-based Sensor Dataset of Self-driving Cars
SensoDat: Simulation-based Sensor Dataset of Self-driving CarsSensoDat: Simulation-based Sensor Dataset of Self-driving Cars
SensoDat: Simulation-based Sensor Dataset of Self-driving Cars
 
Ronisha Informatics Private Limited Catalogue
Ronisha Informatics Private Limited CatalogueRonisha Informatics Private Limited Catalogue
Ronisha Informatics Private Limited Catalogue
 
VK Business Profile - provides IT solutions and Web Development
VK Business Profile - provides IT solutions and Web DevelopmentVK Business Profile - provides IT solutions and Web Development
VK Business Profile - provides IT solutions and Web Development
 
Simplifying Microservices & Apps - The art of effortless development - Meetup...
Simplifying Microservices & Apps - The art of effortless development - Meetup...Simplifying Microservices & Apps - The art of effortless development - Meetup...
Simplifying Microservices & Apps - The art of effortless development - Meetup...
 
Large Language Models for Test Case Evolution and Repair
Large Language Models for Test Case Evolution and RepairLarge Language Models for Test Case Evolution and Repair
Large Language Models for Test Case Evolution and Repair
 
How to submit a standout Adobe Champion Application
How to submit a standout Adobe Champion ApplicationHow to submit a standout Adobe Champion Application
How to submit a standout Adobe Champion Application
 
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
 
Post Quantum Cryptography – The Impact on Identity
Post Quantum Cryptography – The Impact on IdentityPost Quantum Cryptography – The Impact on Identity
Post Quantum Cryptography – The Impact on Identity
 
Odoo 14 - eLearning Module In Odoo 14 Enterprise
Odoo 14 - eLearning Module In Odoo 14 EnterpriseOdoo 14 - eLearning Module In Odoo 14 Enterprise
Odoo 14 - eLearning Module In Odoo 14 Enterprise
 

QA is not quality

  • 1. QA is not Quality @antoligy
  • 5. Who am I? “Web performance advocate and aspiring polyglot” Twitter — @antoligy Web — alexwilson.tech @antoligy
  • 6. Take-away ● What Software Quality actually is ● How we can measure it ● Practical ways to improve the quality of what we’re building @antoligy
  • 7. So what actually is quality?
  • 8. The difficulty in defining quality is to translate future needs of the user into measurable characteristics, so that a product can be designed and turned out to give satisfaction at a price that the user will pay. — William E Deming Let’s ask the experts! @antoligy The word quality has multiple meanings. Two of these meanings dominate the use of the word: 1. Quality consists of those product features which meet the need of customers and thereby provide product satisfaction. 2. Quality consists of freedom from deficiencies. Nevertheless, in a handbook such as this it is convenient to standardize on a short definition of the word quality as "fitness for use". — Joseph M Juran Quality is a customer determination, not an engineer's determination, not a marketing determination, nor a general management determination. It is based on the customer's actual experience with the product or service, measured against his or her requirements — stated or unstated, conscious or merely sensed, technically operational or entirely subjective — and always representing a moving target in a competitive market. — Armand V Feigenbaum
  • 9. ● Being “fit for purpose”, and serving a user need. ● Free from deficiencies. ● Being able to easily meet the needs of the future. In simpler terms... @antoligy
  • 10. Quality Model: ISO-25000 (SQuaRE) @antoligy
  • 11. What is it? A system provides functions that meet stated and implied needs when used under specified conditions How do you measure it? ● Functional completeness: CSAT, Traceability matrix (requirements vs test coverage) Functional Suitability @antoligy
  • 12. How do you improve it? ● Hire a Business Analyst and QA to define complete functional requirements ● Implement user-testing Example The time we reverse engineered a product and its functional requirements while changing its technology stack: It would have been far easier if we had mapped out what we knew. Functional Suitability @antoligy
  • 13. What is it? The performance relative to the amount of resources used under stated conditions How do you measure it? ● “Core Metrics” — # Latency, # Request-Rate, # Errors ● User-centric metrics — # Time to interactive, # Time to first meaningful interaction ● Capacity — Load Testing ● Resource Utilisation — Tracing Performance Efficiency @antoligy
  • 14. Performance Efficiency @antoligy How do you improve it? ● Choose the right technologies for the problems you are trying to solve ● Instrument “core metrics” across your stack. ● Load-test under realistic scenarios. ● Link performance improvements to business metrics (e.g. conversion) Example The time a website’s performance problems for years turned out to be an unnoticed, unnecessary synchronous API callout blocking every request.
  • 15. Usability @antoligy What is it? A system can be used by specified users to achieve specific goals with effectiveness, efficiency and satisfaction in a specified context of use How do you measure it? ● Accessibility — WCAG 2.1 score ● User-friendliness — # User error rates, # L1 support-tickets, CSAT ● Operability — # Production incidents
  • 16. Usability How do you improve it? ● Automate accessibility testing ● Regularly conduct user interviews and perform user research ● Test on real user devices ● Invest in onboarding every type of user a system or product might have Example The time I built multi-language support into an events website to improve the accessibility for its non-English-speaking users, but, didn’t train or make it easy for administrators to add new locales. @antoligy
  • 17. Reliability What is it? A component functions consistently under specified conditions for a specified period of time How do you measure it? ● Stability — Uptime, # of reported outages ● Availability — MTBF (mean-time-between-failures: mean-time-to-failure + mean-time- to-repair) @antoligy
  • 18. Reliability @antoligy How do you improve it? ● Chaos-suite: Simulate disasters and failures before they hit you ● Knowledge-sharing exercises in the technology stack ● Apply SLAs and circuit-breakers to dependencies ● Instrument applications and services ● Invest in Observability tooling Example The time Puppet accidentally deleted a production database and server, but because we used Puppet and orchestrated every aspect of AWS, we were able to recover within two hours.
  • 19. Security What is it? A system protects information and data so that persons or other products or systems have the degree of data access appropriate to their types and levels of authorization How do you measure it? ● Risk — OWASP Risk Rating (Likelihood * Impact) ● Auditability — # of user actions not recorded/attributed to a user ● Integrity — # of areas where PII may be found ● Compliance with standards such as PCI-DSS @antoligy
  • 20. How do you improve it? ● Scan dependencies for known vulnerabilities (e.g. Snyk) ● Vulnerability scanning of entire system (e.g. automated like Qualys) ● Bug-Bounties ● Role-Based-Access-Control, and Attribute-Based-Access-Control Example The time a Drupal website was hacked and would randomly serve cialis adverts to search- bots only. Security @antoligy
  • 21. What is it? A system can be efficiently and effectively modified to improve it, correct it or adapt it to changes in environment, and in requirements How do you measure it? ● Testability — % unit code coverage, % scenario coverage, % feature coverage ● Understandability — # of lines of code, average file length, adherence to conventions ● Modifiability — Cyclomatic complexity, Cohesion Maintainability @antoligy
  • 22. How do you improve it? ● Invest heavily in developer experience ● Lean on unit-testing to improve developer confidence ● Invest in code-review automation (e.g. Codacy) ● Reduce # caching layers and improve their utilisation ● Document your systems inside-out ● Invest in Observability tooling Example The time when revisiting our architectural documentation and increasing the verbosity of our logging helped us diagnose an intermittent problem as being Redis replicating data asynchronously. Maintainability @antoligy
  • 23. Portability What is it? A system, product or component can be transferred from one hardware, software or other operational or usage environment to another. How do you improve it? Containerize Portability & Compatibility I’ll be honest: These are not things I think I have ever successfully solved. They are included for “completeness”. Compatibility What is it? A component can exchange information with other components, and/or perform its required functions, while sharing the same hardware or software environment. How do you improve it? Namespacing, Containerize @antoligy
  • 24. Practical ways of improving quality There are also some general principles we can implement. ● Defining “Done” ● Experiment and iterate ● Test in production
  • 25. Practical Suggestions: Define “Done” @antoligy ● Being dogmatic about every principle of quality will get us nowhere fast, so prioritise based on what is important for our users. ● Periodically revisit this definition as a team, asking: ○ Do our stakeholders and users understand how this works? ○ Do we think this has helped improve our quality? ○ Do our users feel the same way? ● Work with “quality champions” early-on to define how new bodies of work will be tested. https://www.scrum.org/resources/blog/walking-through-definition-done
  • 26. Practical Suggestions: Experiment and iterate! @antoligy ● Get feedback from stakeholders and users early and often. ○ As quality improves, it will be easier to manage further changes upwards. ○ Team-members and developers in other areas of the business are stakeholders too. ● Use a model of quality to measure and experiment with process and tooling, and don’t be afraid to make mistakes. ● Link chosen metrics to tangible, easy-to-understand goals, and evaluate success based on trends.
  • 27. Practical Suggestions: Test in production @antoligy ● Production is the only environment where we can prove things work. ● As well as testing safely out of production, make it safe to test in production. ○ Feature-flagging to decouple roll-out of features, from roll-out of code, so manual tests are possible in prod before showing users a broken experience. ○ Canary releases to test with small percentage of user-base first. ○ Synthetic monitoring of critical features and journeys. ○ Especially on the web: Enhance progressively, and degrade gracefully. (i.e. if JavaScript errors, continue serving critical functionalities) ● We’re already doing this every time we push to production! https://martinfowler.com/articles/qa-in-production.html
  • 28. Thanks for listening! Any questions? @antoligy

Editor's Notes

  1. Hey!
  2. Hey — Anyone familiar with this? One of my favourite games, Super Metroid, came out 25 years ago today. For those of you who aren’t familiar, it’s a side-scrolling adventure game where you explore an alien world to stop space terrorists known as space pirates from developing and using bio-weaponry. If that doesn’t sound awesome, then it’s probably because I’ve undersold it: If you’re able to, definitely give it a try. Anyway: The game was built as a partnership between Nintendo and another company called Intelligent Systems, by a team of seventeen people called Team Deer Force. Anyone who has worked cross-organisation will know that there’s an awful lot that tends to go wrong and this was no exception: There were many logistical problems along the way, but team cohesion was never one of them and over two years they managed to build something which still holds its own today. Why this game still stands up so well is so very relevant to this topic, because it’s the same reason that many other titles from Nintendo have generally done so well over time. Nintendo championed extensive software testing as part of an extensive quality control process.
  3. It was actually this kind of process which helped them claw back the video-games market after an over saturation of low-quality software for the Atari 2600 platform nearly killed the entire industry in 1983. Specifically, remember ET? That game that Atari published, which sold so terribly that Atari then found themselves paying for their stock to be buried in landfill? Yeah I’m not going to show you what that game played like, but it isn’t pretty. And it’s a good example of what happens when we skip any semblance of quality process. At the other extreme, the quality process which led to titles like Super Metroid — I’ve found it rather hard trying to actually find out what their quality-control process looked like (it turns out that this is a closely guarded secret, like KFC’s blend of herbs and spices) — although it is possible to make guesses by looking at the general state of software testing and quality control in Japan at the time which was a blend of Japanese “kaizen” — the process of continuous improvement — and something called Total Quality Management which I’m not going to talk about today as it’s very heavy. It would have been vigorous to the point where everybody in this room would be complaining about it holding us back!
  4. I love this example because on the one hand there are these exceptional games which captured audiences, and on the other there are these truly awful experiences which actually scared scared away their users from ever using a games console ever again. Kind of like what happens when a user tries using a buggy cart to buy a book, and ends up abandoning that particular store never to return.
  5. So with that out of the way, who am I and why should you listen to me talk about this weird thing called quality? My name is Alex, and I’m a self-diagnosed full-stack software person. Primarily I have either built, or have led teams building, consumer-facing software — typically on the safe, easy-to-support land of the web. Where nothing ever randomly goes wrong. I’m just about getting a grip on this funny little notion of quality and wanted to share some thoughts.
  6. After this … Hopefully you’ll have taken away why quality problems can’t be pigeon-holed, and why they can only be owned by an entire team. Hopefully you’ll also have taken away a few distinct ways of looking at and measuring quality throughout your entire delivery pipeline. And, now that you’re hopefully able to measure quality throughout your pipeline, hopefully I’ll also have been able to give you a few ideas about how to improve it.
  7. Starting with basics, though: What is Quality?
  8. There have been a few! I’ve picked out a few quotes here from William E Deming, Armand V Feigenbaum and Joseph M Juran, who are three of the “founding fathers” of quality. Or at least are three of the people whom are attributed as such by the Six Sigma distribution of quality techniques developed by Motorola in the 1980s. William E Deming says: The difficulty in defining quality is to translate future needs of the user into measurable characteristics, so that a product can be designed and turned out to give satisfaction at a price that the user will pay. This is not easy, and as soon as one feels fairly successful in the endeavor, he finds that the needs of the consumer have changed, competitors have moved in, etc. Armand V Feigenbaum, who is credited with TQM, says: Quality is a customer determination, not an engineer's determination, not a marketing determination, nor a general management determination. It is based on the customer's actual experience with the product or service, measured against his or her requirements — stated or unstated, conscious or merely sensed, technically operational or entirely subjective — and always representing a moving target in a competitive market. Joseph M Juran says: The word quality has multiple meanings. Two of these meanings dominate the use of the word: 1. Quality consists of those product features which meet the need of customers and thereby provide product satisfaction. 2. Quality consists of freedom from deficiencies. Nevertheless, in a handbook such as this it is convenient to standardize on a short definition of the word quality as "fitness for use".
  9. I’ve often been told off for being too reductive: Unfortunately that’s a lot of this talk. The shorter version of what these people have said is that a product can be said to be of high quality when it is fit for purpose, serving a user need, when it is free from deficiencies and when it is easily able to meet the needs of the future. But this is still a little bit abstract. I think we can do a bit better with our definition of quality.
  10. Turns out that there is a well-crafted standard addressing this: ISO 25000, or SQuaRE: Software-product Quality Requirements and Evaluation. - ISO 25000’s model of quality defines a set of high-level characteristics and then sub-characteristics where each is a measurable thing. - Functional Suitability - Performance efficiency - Usability - Reliability - Security - Maintainability - Portability - Compatibility As you can see, there’s a lot here. There’s two extremes that people tend to drift towards: Either having a pool of QA who act as a barrier for all work, who are given the sole responsibility of making sure that a system actually works well, usually using a mixture of automated and manual tesiting. Or, not having a pool of QA at all and sort-of winging it because automated testing owned by the engineering team is considered to be enough. I think both methodologies have merits which have a legitimate place!
  11. The tester needs to gather information about the threat agent involved, the attack that will be used, the vulnerability involved, and the impact of a successful exploit on the business
  12. Portability and compatibility are two concerns in quality that I don’t feel like I’ve successfully addressed. There’s a lot to them, but the overall payoff unless you are extremely risk averse doesn’t seem too high. In the past I have definitely overprioritised these areas, and it’s resulted in wasting a lot of time, although I’m definitely interested in hearing other use-cases people are seeing of this. “Portability” definitely makes it easier to test and reproduce problems and other error states. Largely I’ve taken the same solution to both, and that is to focus on enforcing Heroku’s twelve-factor app principles. Containerising using a technology like Rkt or Docker will generally achieve this.
  13. Now unfortunately this isn’t one that has worked so well for people making video-games 20 years ago. Companies still follow feature-flagging and data-driven decisions in prod though.
  14. I haven’t really been talking