SlideShare a Scribd company logo
Testing System Qualities

     Rebecca Wirfs-Brock
             Joseph Yoder



      Copyright 2012 Rebecca Wirfs-Brock, Joseph Yoder,
        Wirfs-Brock Associates and The Refactory, Inc.
Introducing Rebecca
President, Wirfs-Brock Associates
Agile enthusiast (involved with experience
    reports since 1st agile conference, board
    president Agile Open Northwest)
First engineering job in Quality Assurance
Pattern enthusiast, author, and Hillside Board
    Treasurer
Old design geek (author of two object design
    books, inventor of Responsibility-Driven
    Design, advocate of CRC cards, hot spot
    cards, & other low-tech design tools, IEEE
    Software design columnist)
Consults and trains top companies on agile
    architecture, responsiblity-driven design,
    enterprise app design, agile use cases,
    design storytelling, pragmatic testing
Runs marathons!!!
Introducing Joseph
Founder and Architect, The Refactory, Inc.
Pattern enthusiast, author and Hillside Board
   President
Author of the Big Ball of Mud Pattern
Adaptive systems expert (programs adaptive
   software, consults on adaptive
   architectures, author of adaptive
   architecture patterns, metatdata maven,
   website: adaptiveobjectmodel.com)
Agile enthusiast and practitioner
Business owner (leads a world class
   development company)
Consults and trains top companies on design,
   refactoring, pragmatic testing
Amateur photographer, motorcycle enthusiast,
   enjoys dancing samba!!!
Some Agile Myths
• System Qualities can be
  added at the last moment.
• We can easily adapt to changing
  requirements (new requirements).
• You can change the system fast!!!
• Don’t worry about the architecture.
Most testers spend the majority of their time writing functional tests




…BUT THERE’S A LOT MORE TO TEST
THAT YOUR SOFTWARE WORKS AS
ADVERTISED
Pragmatic Testing Issues

  • What kinds of testing
    should you focus on?
  • Who writes them?
  • Who runs them?
  • When are they run?
  • How are they run?
Testing System Qualities
Qualities we could consider…
    •   Usability
    •   Security
    •   Performance
    •   Scalability
    •   Internationalization
    •   Availability
    •   Flexibility
    •   Accessibility
    •   Location
    •   Regulation
Discussion


What are the
 important
 qualities in
your system?
Functional Tests vs. System Quality Tests
  Functional:
    – How do I …?
    – Tests user stories work as advertised
       • “As a reviewer I want to add a note to a chart”
       • Compute the charge for an invoice
    – Tests boundary conditions
       • Can I add more than one note at the same place?
       • Are excess charges computed correctly?
System Quality Tests
• How does the system handle…?
  – system load …? number of add note
    transactions/minute under normal load
  – system support for…? simultaneously
    updating charts
  – usability…? ease of locating and selecting notes

• Tests that emphasize architecture capabilities
  and tangible system characteristics
Structure for A System
Quality Acceptance Test
Specify Measurable Results
• Meter: An appropriate • Scale: The values
         way to measure          you expect
                           – Natural Scale:
                              • Response time in
                                milliseconds
                           – Constructed:
                              • 1-10 ranking
                           – Proxy:
                              • Projecting throughput
                                using sample data
Example Performance Scenario

Source of   Stimulus       Artifact    Response       Response
Stimulus                                              Measure
 Users      Initiate      System       Transactions
            order                      processed      Average
                                                      latency of 2
                       Environment                    seconds
                          Normal
                          conditions


  “Users initiate 1,000 order transactions per
   minute under normal operations; transactions
   are processed with an average latency of 2
   seconds.”
Possible Performance Scenario Values
  Portion of                       Possible Values
  Scenario
  Source        External systems, users, components, databases
  Stimulus      Periodic events, sporadic or random events (or a
                combination)
  Artifact      The system’s services, data, or other resources
  Environment   The state the system can be in: normal, overloaded,
                partial operation, emergency mode…
  Response      Process the event or event sequence and possibly change
                the level of service
  Response      Times it takes to process the arriving events (latency or
  Measure       deadline by which event must be processed), the
                variation in this time, the number of events that can be
                processed within a particular time interval, or a
                characterization of events that cannot be processed
                (missed rate, data loss)
Example Security Scenario

 Source of    Stimulus   Artifact   Response      Response
 Stimulus                                         Measure
                         Data       System
              Tries to   within the maintains     Correct data
 Correctly
              transfer   system                   is restored
 identified                         audit trail
              money Environment                   within a day of
 individual
              between Normal                      reported event
              accounts operation

“A known, authorized user transfers money between
 accounts. The user is later identified as an embezzler
 by the institution they belong to and the system then
 restores funds to the original account.”
Possible Security Scenario Values
Portion of                            Possible Values
Scenario
Source        A human or another system. May be identified (correctly or not) or
              be unknown.
Stimulus      An attack or an attempt to break security by trying to display
              information, change information, access system services, or reduce
              system availability
Artifact      The system’s services or data.
Environment   The system might be online or offline, connected to or disconnected
              from the network, behind a firewall or open to the network.
Response      Authenticates user; hides identity of the user; blocks or allows access
              to data and/or services; records access/modification attempts by
              identity; stores data in encrypted format; recognizes unexplained high
              demand and informs a user or other system, or restricts availability.
Response      Time/effort/resources required to circumvent security measures with
Measure       probability of success; probability of detecting attack; probability of
              identifying individual responsible for attack or access/modification;
              time/effort/resources to restore data/services; extent to which
              data/services are damaged and/or legitimate access denied.
Example Modifiability Scenario

Source of     Stimulus    Artifact    Response       Response
Stimulus                                             Measure
 Developer   Add support UI, Source Modification made
             for new                with no schema     2 days to
                         code,
             service                changes            code and
                         Service
             code                                      test, 1 day
                         Code
                                                       to deploy
                         Table
                     Environment
                   Compile time, data definition

“A developer adds support for a new service code to the
system by adding the service code to the definitions table and
modifying the UI to make it available to users. The
modification is made with no data schema changes ”
Possible Modifiability Scenario Values
  Portion of                  Possible Values
  Scenario
  Source        End user, developer, system administrator
  Stimulus      Wishes to add/delete/modify/vary functionality.
                Wishes to change some system quality such as
                availability, responsiveness, increasing capacity.
  Artifact      What is to be changed: system user interface,
                platform, environment, or another system or API
                with which it interoperates
  Environment   When the change can be made: runtime, compile
                time, build time, when deployed,…
  Response      Locates places to be modified; makes
                modification without affecting other
                functionality; tests and deploys modification
  Response      End user, developer, system administrator
  Measure
Example Availability Scenario

Source of    Stimulus    Artifact    Response             Response
Stimulus                                                  Measure
 Unknown    Unexpected System        Record “raw”
            Report                   report in database    No lost
 Sensor
                                     and log event         data
                    Environment
                        Normal
                        conditions


  “An unknown sensor sends a report. The system stores
  the raw data in the unknown sensor database (to
  potentially be processed or purged later) and logs the
  event. ”
Example Usability Scenario

  Source of    Stimulus   Artifact   Response      Response
  Stimulus    Cancel      System                   Measure
              request                 System
                                      backs out    Cancel takes
   End user         Environment
                                      pending      less than one
                     At runtime       transaction second
                                      and releases
                                      resources

“A user can cancel a pending request within two seconds.”
Copyright 2012 Wirfs-Brock Associates and The Refactory
EXERCISES



WRITE A QUALITY SCENARIO FOR A DATA
COLLECTION AND ANALYSIS SYSTEM
Write A Quality Scenario
• Using the template handout,
  write a quality scenario. Be specific.
• Options:
   – Response to data received from an unknown sensor.
   – Adding new analyzer plug-ins.
   – Relocating a sensor and being able to correlate data
     from its previous location (if desired).
   – Detecting and troubleshooting equipment failures
   – Detecting unusual weather conditions and incidents

• What quality does your scenario address?
You can’t test warm and fuzzy…
“It should be easy to place an online order”



TURN VAGUE STATEMENTS INTO
CONCRETE MEASURABLE ACTIONS
Turning Warm Fuzzies into a
        Testable Usability Scenario

  Source of   Stimulus    Artifact     Response    Response
  Stimulus                System                   Measure
              Place                    Order
   Novice     order                    completed   Time to
   user                  Environment               complete order
                                                   entry task
                         Web interface
                         with online help

“80% of novice users should be able to place an order in
 under 3 minutes without assistance.”
                            or
“80% of novice users should be able to place an order in
 under 3 minutes only using online help.”
Agreeing on scenarios and target values
Some options…
• Toss out a reasonable number, then
     discuss to come to a consensus
• Average informed individuals’ estimates
• Use an existing system as baseline
• Values for similar scenarios
• Benchmark working code
• …
There is more than “pass” or “fail”
                  • Landing Zone: Lets you
                    define a range of
                    acceptable values
                     – Minimal: OK, we
                       can live with that
                     – Target: Realistic goal,
                       what we are aiming for
                     – Outstanding: This
                       would be great, if
                       everything goes well
Landing Zones
       • Roll up product or
         project success to
         several key scenarios
       • Easier to make sense of
         the bigger picture:
          – What happens when one
            quality scenario edges
            below minimum? How
            do others trend?
          – When will targets be
            achieved? At what cost?
Minimum     Target       Outstanding
            Throughput (txns      50,000      70,000       90,000
Performance per day)
            Average txn time      2 seconds   1 second     < 1 second

             Intersystem data     95%         97%          97%
             consistency (per
Data Quality cent critical data
             attributes
             consistent)
              Data accuracy       97%         99%          >99%


               Managing Landing Zones
       Too many scenarios and you lose track of what’s really important
       Define a core set, organize and group
       Roll up into aggregate scenarios
       Re-calibrate values as you implement more functionality…be agile
Quality Testing Cycle for TDD

        Identify and
                           Write Code
                                             Verify
           Write                            Quality
                           and Tests
          Quality                          Scenarios
         Scenarios


                        Clean up Code
                       (Refactor/revise
                           /rework)



Ship
                                            Check
            Ready to   all tests succeed
it!!!                                      all Tests
            Release?                       Succeed
Test Coverage Can Overlap…

               Smoke
               Tests   Quality
                       Scenarios
 Integration
 Tests
                                   Acceptance
                                   Tests
                                   (Functional and
                                   qualities)

               Unit
               Tests
Pragmatic Test Driven
           Development Is…
• Practical. Testing system qualities can fit into
  and enhance your current testing.
• Thoughtful. What qualities need to be tested?
  Who should write tests? When should you test
  for qualities?
• Realistic. You only have so much time and energy.
  Test essential qualities.
Summary
• Quality Scenarios are easy to write and read.
• Writing quality test scenarios drives out
  important cross-cutting concerns.
• If you don’t pay attention to qualities they
  can be hard to achieve at the last moment.
• Measuring system qualities can require
  specialized testing/measurement tools.
Resources
•   Agile Myths: agilemyths.com
•   Wirfs-Brock Associates: www.wirfs-brock.com
•   The Refactory: www.refactory.com
•   Joe’s website: joeyoder.com
•   Our Pragmatic TDD Course:
    – refactory.com/training/test-driven-development
    – wirfs-brock.com/pragmatictestdrivendevelopment.html
• Landing Zone blogs:
    – wirfs-brock.com/blog/2011/07/20/introducing-landing-zones/
    – wirfs-brock.com/blog/2011/08/16/who-defines-or-redefines-
                                                landing-zone-criteria/
    – wirfs-brock.com/blog/2011/08/05/landing-zone-targets-precision-
                                                specificity-and-wiggle-room/
• Introducing Pragmatic TDD:
    – wirfs-brock.com/blog/2011/09/23/what-is-pragmatic-testing-all-about/
    – http://adaptiveobjectmodel.com/2012/01/what-is-pragmatic-tdd/

More Related Content

Viewers also liked

AWS Summit Tokyo 2015 megane LT JAWS-UG TRAIN TRAIN 栄光に向かって走って行け
AWS Summit Tokyo 2015 megane LT JAWS-UG TRAIN TRAIN 栄光に向かって走って行けAWS Summit Tokyo 2015 megane LT JAWS-UG TRAIN TRAIN 栄光に向かって走って行け
AWS Summit Tokyo 2015 megane LT JAWS-UG TRAIN TRAIN 栄光に向かって走って行け
Hajime Ogushi
 
System quality attributes
System quality attributes System quality attributes
System quality attributes
Adil Mehmoood
 
Taming Big Balls of Mud with Diligence, Agile Practices, and Hard Work
Taming Big Balls of Mud with Diligence, Agile Practices, and Hard WorkTaming Big Balls of Mud with Diligence, Agile Practices, and Hard Work
Taming Big Balls of Mud with Diligence, Agile Practices, and Hard Work
Joseph Yoder
 
Cloud Computing Architecture Primer
Cloud Computing Architecture PrimerCloud Computing Architecture Primer
Cloud Computing Architecture Primer
Ilham Ahmed
 
Software Architecture: Test Case Writing
Software Architecture: Test Case WritingSoftware Architecture: Test Case Writing
Software Architecture: Test Case Writing
Sitdhibong Laokok
 
Cloud computing architecture and vulnerabilies
Cloud computing architecture and vulnerabiliesCloud computing architecture and vulnerabilies
Cloud computing architecture and vulnerabilies
Vinay Dwivedi
 
Introduction to cloud computing
Introduction to cloud computingIntroduction to cloud computing
Introduction to cloud computing
Vipin Batra
 
Mobile Cloud Computing Challenges and Security
Mobile Cloud Computing Challenges and SecurityMobile Cloud Computing Challenges and Security
Mobile Cloud Computing Challenges and Security
John Paul Prassanna
 
How to Implement Hybrid Cloud Solutions Successfully
How to Implement Hybrid Cloud Solutions SuccessfullyHow to Implement Hybrid Cloud Solutions Successfully
How to Implement Hybrid Cloud Solutions Successfully
SoftServe
 
Fog Computing with VORTEX
Fog Computing with VORTEXFog Computing with VORTEX
Fog Computing with VORTEX
Angelo Corsaro
 
Cloud computing security issues and challenges
Cloud computing security issues and challengesCloud computing security issues and challenges
Cloud computing security issues and challenges
Dheeraj Negi
 
AWSでアプリ開発するなら 知っておくべこと
AWSでアプリ開発するなら 知っておくべことAWSでアプリ開発するなら 知っておくべこと
AWSでアプリ開発するなら 知っておくべこと
Keisuke Nishitani
 

Viewers also liked (12)

AWS Summit Tokyo 2015 megane LT JAWS-UG TRAIN TRAIN 栄光に向かって走って行け
AWS Summit Tokyo 2015 megane LT JAWS-UG TRAIN TRAIN 栄光に向かって走って行けAWS Summit Tokyo 2015 megane LT JAWS-UG TRAIN TRAIN 栄光に向かって走って行け
AWS Summit Tokyo 2015 megane LT JAWS-UG TRAIN TRAIN 栄光に向かって走って行け
 
System quality attributes
System quality attributes System quality attributes
System quality attributes
 
Taming Big Balls of Mud with Diligence, Agile Practices, and Hard Work
Taming Big Balls of Mud with Diligence, Agile Practices, and Hard WorkTaming Big Balls of Mud with Diligence, Agile Practices, and Hard Work
Taming Big Balls of Mud with Diligence, Agile Practices, and Hard Work
 
Cloud Computing Architecture Primer
Cloud Computing Architecture PrimerCloud Computing Architecture Primer
Cloud Computing Architecture Primer
 
Software Architecture: Test Case Writing
Software Architecture: Test Case WritingSoftware Architecture: Test Case Writing
Software Architecture: Test Case Writing
 
Cloud computing architecture and vulnerabilies
Cloud computing architecture and vulnerabiliesCloud computing architecture and vulnerabilies
Cloud computing architecture and vulnerabilies
 
Introduction to cloud computing
Introduction to cloud computingIntroduction to cloud computing
Introduction to cloud computing
 
Mobile Cloud Computing Challenges and Security
Mobile Cloud Computing Challenges and SecurityMobile Cloud Computing Challenges and Security
Mobile Cloud Computing Challenges and Security
 
How to Implement Hybrid Cloud Solutions Successfully
How to Implement Hybrid Cloud Solutions SuccessfullyHow to Implement Hybrid Cloud Solutions Successfully
How to Implement Hybrid Cloud Solutions Successfully
 
Fog Computing with VORTEX
Fog Computing with VORTEXFog Computing with VORTEX
Fog Computing with VORTEX
 
Cloud computing security issues and challenges
Cloud computing security issues and challengesCloud computing security issues and challenges
Cloud computing security issues and challenges
 
AWSでアプリ開発するなら 知っておくべこと
AWSでアプリ開発するなら 知っておくべことAWSでアプリ開発するなら 知っておくべこと
AWSでアプリ開発するなら 知っておくべこと
 

Similar to Testing System Qualities Agile2012 by Rebecca Wirfs-Brock and Joseph Yoder

Software archiecture lecture05
Software archiecture   lecture05Software archiecture   lecture05
Software archiecture lecture05
Luktalja
 
Sda 3
Sda   3Sda   3
Design For Testability
Design For TestabilityDesign For Testability
Design For Testability
Giovanni Asproni
 
Quality Attributes Of a Software.pptx
Quality Attributes Of a Software.pptxQuality Attributes Of a Software.pptx
Quality Attributes Of a Software.pptx
jordan724752
 
RuSIEM overview (english version)
RuSIEM overview (english version)RuSIEM overview (english version)
RuSIEM overview (english version)
Olesya Shelestova
 
Ch20-Software Engineering 9
Ch20-Software Engineering 9Ch20-Software Engineering 9
Ch20-Software Engineering 9
Ian Sommerville
 
Automated Discovery of Performance Regressions in Enterprise Applications
Automated Discovery of Performance Regressions in Enterprise ApplicationsAutomated Discovery of Performance Regressions in Enterprise Applications
Automated Discovery of Performance Regressions in Enterprise Applications
SAIL_QU
 
Performance testing basics
Performance testing basicsPerformance testing basics
Performance testing basics
Charu Anand
 
Siddhi: A Second Look at Complex Event Processing Implementations
Siddhi: A Second Look at Complex Event Processing ImplementationsSiddhi: A Second Look at Complex Event Processing Implementations
Siddhi: A Second Look at Complex Event Processing Implementations
Srinath Perera
 
What is Platform Observability? An Overview
What is Platform Observability? An OverviewWhat is Platform Observability? An Overview
What is Platform Observability? An Overview
Kumar Kolaganti
 
High Availability HPC ~ Microservice Architectures for Supercomputing
High Availability HPC ~ Microservice Architectures for SupercomputingHigh Availability HPC ~ Microservice Architectures for Supercomputing
High Availability HPC ~ Microservice Architectures for Supercomputing
inside-BigData.com
 
ADDO Open Source Observability Tools
ADDO Open Source Observability Tools ADDO Open Source Observability Tools
ADDO Open Source Observability Tools
Mickey Boxell
 
What is onTune for management
What is onTune for managementWhat is onTune for management
What is onTune for management
TeemStone Pty Ltd
 
Performance testing and rpt
Performance testing and rptPerformance testing and rpt
Performance testing and rpt
SivaprasanthRentala1975
 
Proactive ops for container orchestration environments
Proactive ops for container orchestration environmentsProactive ops for container orchestration environments
Proactive ops for container orchestration environments
Docker, Inc.
 
Code Quality - Security
Code Quality - SecurityCode Quality - Security
Code Quality - Security
sedukull
 
Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Building an Event-oriented...
Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Building an Event-oriented...Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Building an Event-oriented...
Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Building an Event-oriented...
Data Con LA
 
Big Data and Machine Learning on AWS
Big Data and Machine Learning on AWSBig Data and Machine Learning on AWS
Big Data and Machine Learning on AWS
CloudHesive
 
Manual Testing Types Used in Software Testing
Manual Testing Types Used in Software TestingManual Testing Types Used in Software Testing
Manual Testing Types Used in Software Testing
seojayeshts
 
Regulated Reactive - Security Considerations for Building Reactive Systems in...
Regulated Reactive - Security Considerations for Building Reactive Systems in...Regulated Reactive - Security Considerations for Building Reactive Systems in...
Regulated Reactive - Security Considerations for Building Reactive Systems in...
Ryan Hodgin
 

Similar to Testing System Qualities Agile2012 by Rebecca Wirfs-Brock and Joseph Yoder (20)

Software archiecture lecture05
Software archiecture   lecture05Software archiecture   lecture05
Software archiecture lecture05
 
Sda 3
Sda   3Sda   3
Sda 3
 
Design For Testability
Design For TestabilityDesign For Testability
Design For Testability
 
Quality Attributes Of a Software.pptx
Quality Attributes Of a Software.pptxQuality Attributes Of a Software.pptx
Quality Attributes Of a Software.pptx
 
RuSIEM overview (english version)
RuSIEM overview (english version)RuSIEM overview (english version)
RuSIEM overview (english version)
 
Ch20-Software Engineering 9
Ch20-Software Engineering 9Ch20-Software Engineering 9
Ch20-Software Engineering 9
 
Automated Discovery of Performance Regressions in Enterprise Applications
Automated Discovery of Performance Regressions in Enterprise ApplicationsAutomated Discovery of Performance Regressions in Enterprise Applications
Automated Discovery of Performance Regressions in Enterprise Applications
 
Performance testing basics
Performance testing basicsPerformance testing basics
Performance testing basics
 
Siddhi: A Second Look at Complex Event Processing Implementations
Siddhi: A Second Look at Complex Event Processing ImplementationsSiddhi: A Second Look at Complex Event Processing Implementations
Siddhi: A Second Look at Complex Event Processing Implementations
 
What is Platform Observability? An Overview
What is Platform Observability? An OverviewWhat is Platform Observability? An Overview
What is Platform Observability? An Overview
 
High Availability HPC ~ Microservice Architectures for Supercomputing
High Availability HPC ~ Microservice Architectures for SupercomputingHigh Availability HPC ~ Microservice Architectures for Supercomputing
High Availability HPC ~ Microservice Architectures for Supercomputing
 
ADDO Open Source Observability Tools
ADDO Open Source Observability Tools ADDO Open Source Observability Tools
ADDO Open Source Observability Tools
 
What is onTune for management
What is onTune for managementWhat is onTune for management
What is onTune for management
 
Performance testing and rpt
Performance testing and rptPerformance testing and rpt
Performance testing and rpt
 
Proactive ops for container orchestration environments
Proactive ops for container orchestration environmentsProactive ops for container orchestration environments
Proactive ops for container orchestration environments
 
Code Quality - Security
Code Quality - SecurityCode Quality - Security
Code Quality - Security
 
Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Building an Event-oriented...
Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Building an Event-oriented...Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Building an Event-oriented...
Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Building an Event-oriented...
 
Big Data and Machine Learning on AWS
Big Data and Machine Learning on AWSBig Data and Machine Learning on AWS
Big Data and Machine Learning on AWS
 
Manual Testing Types Used in Software Testing
Manual Testing Types Used in Software TestingManual Testing Types Used in Software Testing
Manual Testing Types Used in Software Testing
 
Regulated Reactive - Security Considerations for Building Reactive Systems in...
Regulated Reactive - Security Considerations for Building Reactive Systems in...Regulated Reactive - Security Considerations for Building Reactive Systems in...
Regulated Reactive - Security Considerations for Building Reactive Systems in...
 

Recently uploaded

Three New Criminal Laws in India 1 July 2024
Three New Criminal Laws in India 1 July 2024Three New Criminal Laws in India 1 July 2024
Three New Criminal Laws in India 1 July 2024
aakash malhotra
 
Opencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of MünsterOpencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of Münster
Matthias Neugebauer
 
How to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptxHow to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptx
Adam Dunkels
 
“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...
“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...
“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...
Edge AI and Vision Alliance
 
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
digitalxplive
 
Types of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technologyTypes of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technology
ldtexsolbl
 
Using LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and MilvusUsing LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and Milvus
Zilliz
 
Best Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdfBest Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdf
Tatiana Al-Chueyr
 
WhatsApp Spy Online Trackers and Monitoring Apps
WhatsApp Spy Online Trackers and Monitoring AppsWhatsApp Spy Online Trackers and Monitoring Apps
WhatsApp Spy Online Trackers and Monitoring Apps
HackersList
 
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite SolutionIPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Networks
 
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
Priyanka Aash
 
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
maigasapphire
 
Google I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged SlidesGoogle I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged Slides
Google Developer Group - Harare
 
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and OllamaTirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
Zilliz
 
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
Torry Harris
 
Sonkoloniya documentation - ONEprojukti.pdf
Sonkoloniya documentation - ONEprojukti.pdfSonkoloniya documentation - ONEprojukti.pdf
Sonkoloniya documentation - ONEprojukti.pdf
SubhamMandal40
 
The Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF GuideThe Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF Guide
Shiv Technolabs
 
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-InTrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc
 
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
bhumivarma35300
 
find out more about the role of autonomous vehicles in facing global challenges
find out more about the role of autonomous vehicles in facing global challengesfind out more about the role of autonomous vehicles in facing global challenges
find out more about the role of autonomous vehicles in facing global challenges
huseindihon
 

Recently uploaded (20)

Three New Criminal Laws in India 1 July 2024
Three New Criminal Laws in India 1 July 2024Three New Criminal Laws in India 1 July 2024
Three New Criminal Laws in India 1 July 2024
 
Opencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of MünsterOpencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of Münster
 
How to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptxHow to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptx
 
“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...
“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...
“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...
 
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
 
Types of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technologyTypes of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technology
 
Using LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and MilvusUsing LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and Milvus
 
Best Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdfBest Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdf
 
WhatsApp Spy Online Trackers and Monitoring Apps
WhatsApp Spy Online Trackers and Monitoring AppsWhatsApp Spy Online Trackers and Monitoring Apps
WhatsApp Spy Online Trackers and Monitoring Apps
 
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite SolutionIPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite Solution
 
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
 
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
 
Google I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged SlidesGoogle I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged Slides
 
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and OllamaTirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
 
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
 
Sonkoloniya documentation - ONEprojukti.pdf
Sonkoloniya documentation - ONEprojukti.pdfSonkoloniya documentation - ONEprojukti.pdf
Sonkoloniya documentation - ONEprojukti.pdf
 
The Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF GuideThe Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF Guide
 
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-InTrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
 
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
 
find out more about the role of autonomous vehicles in facing global challenges
find out more about the role of autonomous vehicles in facing global challengesfind out more about the role of autonomous vehicles in facing global challenges
find out more about the role of autonomous vehicles in facing global challenges
 

Testing System Qualities Agile2012 by Rebecca Wirfs-Brock and Joseph Yoder

  • 1. Testing System Qualities Rebecca Wirfs-Brock Joseph Yoder Copyright 2012 Rebecca Wirfs-Brock, Joseph Yoder, Wirfs-Brock Associates and The Refactory, Inc.
  • 2. Introducing Rebecca President, Wirfs-Brock Associates Agile enthusiast (involved with experience reports since 1st agile conference, board president Agile Open Northwest) First engineering job in Quality Assurance Pattern enthusiast, author, and Hillside Board Treasurer Old design geek (author of two object design books, inventor of Responsibility-Driven Design, advocate of CRC cards, hot spot cards, & other low-tech design tools, IEEE Software design columnist) Consults and trains top companies on agile architecture, responsiblity-driven design, enterprise app design, agile use cases, design storytelling, pragmatic testing Runs marathons!!!
  • 3. Introducing Joseph Founder and Architect, The Refactory, Inc. Pattern enthusiast, author and Hillside Board President Author of the Big Ball of Mud Pattern Adaptive systems expert (programs adaptive software, consults on adaptive architectures, author of adaptive architecture patterns, metatdata maven, website: adaptiveobjectmodel.com) Agile enthusiast and practitioner Business owner (leads a world class development company) Consults and trains top companies on design, refactoring, pragmatic testing Amateur photographer, motorcycle enthusiast, enjoys dancing samba!!!
  • 4. Some Agile Myths • System Qualities can be added at the last moment. • We can easily adapt to changing requirements (new requirements). • You can change the system fast!!! • Don’t worry about the architecture.
  • 5. Most testers spend the majority of their time writing functional tests …BUT THERE’S A LOT MORE TO TEST THAT YOUR SOFTWARE WORKS AS ADVERTISED
  • 6. Pragmatic Testing Issues • What kinds of testing should you focus on? • Who writes them? • Who runs them? • When are they run? • How are they run?
  • 7. Testing System Qualities Qualities we could consider… • Usability • Security • Performance • Scalability • Internationalization • Availability • Flexibility • Accessibility • Location • Regulation
  • 8. Discussion What are the important qualities in your system?
  • 9. Functional Tests vs. System Quality Tests Functional: – How do I …? – Tests user stories work as advertised • “As a reviewer I want to add a note to a chart” • Compute the charge for an invoice – Tests boundary conditions • Can I add more than one note at the same place? • Are excess charges computed correctly?
  • 10. System Quality Tests • How does the system handle…? – system load …? number of add note transactions/minute under normal load – system support for…? simultaneously updating charts – usability…? ease of locating and selecting notes • Tests that emphasize architecture capabilities and tangible system characteristics
  • 11. Structure for A System Quality Acceptance Test
  • 12. Specify Measurable Results • Meter: An appropriate • Scale: The values way to measure you expect – Natural Scale: • Response time in milliseconds – Constructed: • 1-10 ranking – Proxy: • Projecting throughput using sample data
  • 13. Example Performance Scenario Source of Stimulus Artifact Response Response Stimulus Measure Users Initiate System Transactions order processed Average latency of 2 Environment seconds Normal conditions “Users initiate 1,000 order transactions per minute under normal operations; transactions are processed with an average latency of 2 seconds.”
  • 14. Possible Performance Scenario Values Portion of Possible Values Scenario Source External systems, users, components, databases Stimulus Periodic events, sporadic or random events (or a combination) Artifact The system’s services, data, or other resources Environment The state the system can be in: normal, overloaded, partial operation, emergency mode… Response Process the event or event sequence and possibly change the level of service Response Times it takes to process the arriving events (latency or Measure deadline by which event must be processed), the variation in this time, the number of events that can be processed within a particular time interval, or a characterization of events that cannot be processed (missed rate, data loss)
  • 15. Example Security Scenario Source of Stimulus Artifact Response Response Stimulus Measure Data System Tries to within the maintains Correct data Correctly transfer system is restored identified audit trail money Environment within a day of individual between Normal reported event accounts operation “A known, authorized user transfers money between accounts. The user is later identified as an embezzler by the institution they belong to and the system then restores funds to the original account.”
  • 16. Possible Security Scenario Values Portion of Possible Values Scenario Source A human or another system. May be identified (correctly or not) or be unknown. Stimulus An attack or an attempt to break security by trying to display information, change information, access system services, or reduce system availability Artifact The system’s services or data. Environment The system might be online or offline, connected to or disconnected from the network, behind a firewall or open to the network. Response Authenticates user; hides identity of the user; blocks or allows access to data and/or services; records access/modification attempts by identity; stores data in encrypted format; recognizes unexplained high demand and informs a user or other system, or restricts availability. Response Time/effort/resources required to circumvent security measures with Measure probability of success; probability of detecting attack; probability of identifying individual responsible for attack or access/modification; time/effort/resources to restore data/services; extent to which data/services are damaged and/or legitimate access denied.
  • 17. Example Modifiability Scenario Source of Stimulus Artifact Response Response Stimulus Measure Developer Add support UI, Source Modification made for new with no schema 2 days to code, service changes code and Service code test, 1 day Code to deploy Table Environment Compile time, data definition “A developer adds support for a new service code to the system by adding the service code to the definitions table and modifying the UI to make it available to users. The modification is made with no data schema changes ”
  • 18. Possible Modifiability Scenario Values Portion of Possible Values Scenario Source End user, developer, system administrator Stimulus Wishes to add/delete/modify/vary functionality. Wishes to change some system quality such as availability, responsiveness, increasing capacity. Artifact What is to be changed: system user interface, platform, environment, or another system or API with which it interoperates Environment When the change can be made: runtime, compile time, build time, when deployed,… Response Locates places to be modified; makes modification without affecting other functionality; tests and deploys modification Response End user, developer, system administrator Measure
  • 19. Example Availability Scenario Source of Stimulus Artifact Response Response Stimulus Measure Unknown Unexpected System Record “raw” Report report in database No lost Sensor and log event data Environment Normal conditions “An unknown sensor sends a report. The system stores the raw data in the unknown sensor database (to potentially be processed or purged later) and logs the event. ”
  • 20. Example Usability Scenario Source of Stimulus Artifact Response Response Stimulus Cancel System Measure request System backs out Cancel takes End user Environment pending less than one At runtime transaction second and releases resources “A user can cancel a pending request within two seconds.”
  • 21. Copyright 2012 Wirfs-Brock Associates and The Refactory
  • 22. EXERCISES WRITE A QUALITY SCENARIO FOR A DATA COLLECTION AND ANALYSIS SYSTEM
  • 23. Write A Quality Scenario • Using the template handout, write a quality scenario. Be specific. • Options: – Response to data received from an unknown sensor. – Adding new analyzer plug-ins. – Relocating a sensor and being able to correlate data from its previous location (if desired). – Detecting and troubleshooting equipment failures – Detecting unusual weather conditions and incidents • What quality does your scenario address?
  • 24. You can’t test warm and fuzzy… “It should be easy to place an online order” TURN VAGUE STATEMENTS INTO CONCRETE MEASURABLE ACTIONS
  • 25. Turning Warm Fuzzies into a Testable Usability Scenario Source of Stimulus Artifact Response Response Stimulus System Measure Place Order Novice order completed Time to user Environment complete order entry task Web interface with online help “80% of novice users should be able to place an order in under 3 minutes without assistance.” or “80% of novice users should be able to place an order in under 3 minutes only using online help.”
  • 26. Agreeing on scenarios and target values
  • 27. Some options… • Toss out a reasonable number, then discuss to come to a consensus • Average informed individuals’ estimates • Use an existing system as baseline • Values for similar scenarios • Benchmark working code • …
  • 28. There is more than “pass” or “fail” • Landing Zone: Lets you define a range of acceptable values – Minimal: OK, we can live with that – Target: Realistic goal, what we are aiming for – Outstanding: This would be great, if everything goes well
  • 29. Landing Zones • Roll up product or project success to several key scenarios • Easier to make sense of the bigger picture: – What happens when one quality scenario edges below minimum? How do others trend? – When will targets be achieved? At what cost?
  • 30. Minimum Target Outstanding Throughput (txns 50,000 70,000 90,000 Performance per day) Average txn time 2 seconds 1 second < 1 second Intersystem data 95% 97% 97% consistency (per Data Quality cent critical data attributes consistent) Data accuracy 97% 99% >99% Managing Landing Zones Too many scenarios and you lose track of what’s really important Define a core set, organize and group Roll up into aggregate scenarios Re-calibrate values as you implement more functionality…be agile
  • 31. Quality Testing Cycle for TDD Identify and Write Code Verify Write Quality and Tests Quality Scenarios Scenarios Clean up Code (Refactor/revise /rework) Ship Check Ready to all tests succeed it!!! all Tests Release? Succeed
  • 32. Test Coverage Can Overlap… Smoke Tests Quality Scenarios Integration Tests Acceptance Tests (Functional and qualities) Unit Tests
  • 33. Pragmatic Test Driven Development Is… • Practical. Testing system qualities can fit into and enhance your current testing. • Thoughtful. What qualities need to be tested? Who should write tests? When should you test for qualities? • Realistic. You only have so much time and energy. Test essential qualities.
  • 34. Summary • Quality Scenarios are easy to write and read. • Writing quality test scenarios drives out important cross-cutting concerns. • If you don’t pay attention to qualities they can be hard to achieve at the last moment. • Measuring system qualities can require specialized testing/measurement tools.
  • 35. Resources • Agile Myths: agilemyths.com • Wirfs-Brock Associates: www.wirfs-brock.com • The Refactory: www.refactory.com • Joe’s website: joeyoder.com • Our Pragmatic TDD Course: – refactory.com/training/test-driven-development – wirfs-brock.com/pragmatictestdrivendevelopment.html • Landing Zone blogs: – wirfs-brock.com/blog/2011/07/20/introducing-landing-zones/ – wirfs-brock.com/blog/2011/08/16/who-defines-or-redefines- landing-zone-criteria/ – wirfs-brock.com/blog/2011/08/05/landing-zone-targets-precision- specificity-and-wiggle-room/ • Introducing Pragmatic TDD: – wirfs-brock.com/blog/2011/09/23/what-is-pragmatic-testing-all-about/ – http://adaptiveobjectmodel.com/2012/01/what-is-pragmatic-tdd/