A New Model for Testing
@paul_gerrard
Paul Gerrard
paul@gerrardconsulting.com
gerrardconsulting.com
Programme Chair 2014
Helping clients transform their testing through
INNOVATION, COACHING and LEADERSHIP
Our CLIENTS
– Want to be agile rather ...
Typical tester job spec? (May 2014)
• Skills & Requirements
– Thorough C# knowledge including Linq
– Good .Net experience ...
Agenda
• Pressure to Change
• Three Innovations
• Where is this Leading?
• A New Model forTesting
• Consequences
• Shift-L...
What's happening?
• Stampede to mobile computing
• Big Data
• Internet of (Every)Thing, Pervasive Computing
• Continuous D...
Three Innovations That
Are Driving Change
Being agile
Shift Left
Analytics
#1
Agile doesn’t work
but being agile might
Agile is no longer innovative.
Challenging Agile and being agile
might be
Is c...
Agile doesn’t work?
• It works for some
• But many organisations have difficulties
• Mostly a cultural problem
– Requireme...
Pragmatism rules (again)
• Pragmatic companies use the things that work
and discard those that don’t
• Let's be honest: Mo...
#2
Shift Left
Developers in test?
Testers in development?
Testing (testers?) everywhere
TestingBDD/Test-Driven)
Trusted
Requirements +
Stories
Shift left (redistributed testing)
Functional
Requirements
Build (+...
From traditional delivery…
… to Continuous Delivery
Deployment into
Production is not the
Same as ‘Going Live’
Release to production and test
environments on-demand
Go Live w...
#3
Test Analytics
Analytics from production
Analytics from test
An opportunity for testing
Analysis Insight Decision
Development
and Testing
Production
Testing
IntroducingTest Analytics
“IntroducingTest Analytics”...
Data silos for test analytics
(an illustration)
Stakeholder Requirements Assurance (Production)
Application
Monitoring
Pro...
Where is this leading?
If I knew that, I wouldn't be here...
Three development patterns
Structured
Agile Continuous
Autonomous
There aren't three
patterns
There are many
From high process to DIY process
• In the past, high process was seen to be the
technical solution to the software problem...
The old ways won't
work in the future
We need a New Model ofTesting
(free from logistics)
A New Model for
Testing
Don't throw stones.Yet.
I'm working on a refined version (with others).
Want to help/review?
Forget Logistics
(for the time being)
Document or not?
Agile v waterfall?
Planned v exploratory?
This technology v that te...
ALLTesting is
Exploratory
We explore sources of knowledge ...
... to build test models ...
... that inform our testing.
AllTesting is Based on
Models
Examples of models
Models are innate, essential, human
Judgement, exploring and testing
Testing
(the system)
Our model(s) are adequate
Our model(s) are not adequate
Exploring
(s...
Exploration process
ExplorationDefinitions
specs/stories
Oracles
Sources
Design
Models
Test
Models
Enquiring
Challenging
S...
Testing process
Testing
System
Under Test
Refining
Informing Applying
Interpreting
Test
Models
Revise the
System
More expl...
New Model Testing
29 page paper: http://dev.sp.qa/download/newModel
Some Consequences
Do developers and
testers think the same
way?
Similar processes, not thinking the same
Easier to embed testers in developm...
Covers functional and
non-functional testing?
Ignoring logistics means we can have
the same foundational process for
ALL t...
Relation to
TDD and BDD?
TDD is not testing
BDD is modelling using stories
Testing v Checking?
I submit that a Check is "a test that
can be applied by a human or tool"
But only humans can interpret
Test automation from a
different perspective
Automation uses different test models
Is that why automation efforts fail so ...
DeveloperTesting
The model works (for me)
Capabilities
Enquiring, Modelling, Predicting, Challenging
Informing,Applying, Interpreting, Refining
Reporting and Logging
• Analysis, enquiry and elicitation
• Modelling
• Creation of custom models, using
heuristics, guesses, brainstorming,
ide...
Shift-Left is More
Important than Agile
Developers write code;
testers test because
developers can’t
No they don’t
Goal: End Manual
Feature Checking
Let the tools do it and
free people to do testing please!
Testing doesn’t just
apply to existing
software, at the end
We test systems, not software.
We test early, we test often bu...
Testers Need to Learn
How to Code
Some, perhaps many will have to
Testing is about
measuring achievement,
not quality
Align systems delivery with project
goals and risks and you’ve got a s...
Testers don’t own
testing anymore
Be aTestMaster
"Assurance through the lifecycle"
New Model Testing
changes what testers &
developers need to
learn
Roles of developers/testers become blurred
Perhaps they ...
The phase after
development is
REWORK, not Testing
Testing must align with development,
not compete with it or rescue it
A New Model for Testing
http://dev.sp.qa/download/newModel
@paul_gerrard
Paul Gerrard
paul@gerrardconsulting.com
gerrardco...
Upcoming SlideShare
Loading in …5
×

New model

270 views

Published on

I believe that our existing models of testing are not fit for purpose – they are inconsistent, controversial, partial, proprietary and stuck in the past. They are not going to support us in the rapidly emerging technologies and approaches. The certification schemes that should represent the interests and integrity of our profession don’t, and we are left with schemes that are popular, but have low value, lower esteem and attract harsh criticism. My goal in proposing the New Model is to stimulate new thinking in this area.

eurostarconferences.com
testhuddle.com

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
270
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
5
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

New model

  1. 1. A New Model for Testing @paul_gerrard Paul Gerrard paul@gerrardconsulting.com gerrardconsulting.com Programme Chair 2014
  2. 2. Helping clients transform their testing through INNOVATION, COACHING and LEADERSHIP Our CLIENTS – Want to be agile rather than follow Agile dogma – Have a pragmatic approach and are focused on delivery – Want a solution that fits, not a badly fitting suit.
  3. 3. Typical tester job spec? (May 2014) • Skills & Requirements – Thorough C# knowledge including Linq – Good .Net experience including charting and winforms applications. – Experience building and maintaining automated test systems – Ability to interact with the developers to anticipate future needs – Ability to influence to ensure good test procedures are developed and implemented • Desirable Skills – Some knowledge of Java for mobile testing – An interest in photography.
  4. 4. Agenda • Pressure to Change • Three Innovations • Where is this Leading? • A New Model forTesting • Consequences • Shift-Left is More Important than Agile
  5. 5. What's happening? • Stampede to mobile computing • Big Data • Internet of (Every)Thing, Pervasive Computing • Continuous Delivery, DevOps • (Test) Analytics, Data-Driven Development • Shift-left, embedded testers, no test team • Did you really think Agile was the last word?
  6. 6. Three Innovations That Are Driving Change Being agile Shift Left Analytics
  7. 7. #1 Agile doesn’t work but being agile might Agile is no longer innovative. Challenging Agile and being agile might be Is continuous delivery for you?
  8. 8. Agile doesn’t work? • It works for some • But many organisations have difficulties • Mostly a cultural problem – Requirements take months to elicit and agree – Complex stakeholder relationships – Stakeholders with ‘all the knowledge’ can’t be spared to be the on-site customer/product owner – Lack of confidence => big system/acceptance tests • We are Agile but ...
  9. 9. Pragmatism rules (again) • Pragmatic companies use the things that work and discard those that don’t • Let's be honest: Most are doing some form of hybrid approach: – Chaotic interfaces between Agile teams and structured businesses – TDD, BDD,ATDD, SBE today… – Continuous Delivery, DevOps… tomorrow?
  10. 10. #2 Shift Left Developers in test? Testers in development? Testing (testers?) everywhere
  11. 11. TestingBDD/Test-Driven) Trusted Requirements + Stories Shift left (redistributed testing) Functional Requirements Build (+little testing) but lots of rework Testing (lots of checking + bugs) Trusted Requirements + Stories Testing SAVED TIME BDD/Test-Driven Who is going to do all this? BAs, Developers or Testers?
  12. 12. From traditional delivery… … to Continuous Delivery
  13. 13. Deployment into Production is not the Same as ‘Going Live’ Release to production and test environments on-demand Go Live weekly/monthly/quarterly…
  14. 14. #3 Test Analytics Analytics from production Analytics from test An opportunity for testing
  15. 15. Analysis Insight Decision Development and Testing Production Testing IntroducingTest Analytics “IntroducingTest Analytics”, Paul Gerrard: http://gerrardconsulting.com/index.php?q=node/630 Julian Harty, "Symbiotic Relationships betweenTesting and SMAC": http://blog.bettersoftwaretesting.com/2013/07/symbiotic-relationships-between- testing-and-social-mobile-analytics-cloud-computing/ “The capture, integration and analysis of test and production monitoring data to inform business and software development decision-making”.
  16. 16. Data silos for test analytics (an illustration) Stakeholder Requirements Assurance (Production) Application Monitoring Production Environment Monitoring  Stakeholders  Business Goals and measures  Stakeholder Involvement/ Engagement  Risk  Requirements  Stories/Feature Descriptions  Glossary of terms andTerm Usage  Processes  Process Paths (Workflows)  ManualTests  GeneratedTest Code (unit, integration, system level)  Application Instrumentation  AutomatedTest Execution History  Test Status  Feature Code Change History  Application Process Flows  User Accesses/ Activities  Feature Calls, Response times  Feature use cases (data usage)  Application Alerts/Failures  Database Accesses  Database Content  Production failures  System assets  Resource Usage Log  Performance data  System Events  System Alerts/ Failures/Incidents  Outages
  17. 17. Where is this leading? If I knew that, I wouldn't be here...
  18. 18. Three development patterns Structured Agile Continuous Autonomous
  19. 19. There aren't three patterns There are many
  20. 20. From high process to DIY process • In the past, high process was seen to be the technical solution to the software problem • But technical solutions cannot solve human problems • Agile promised flexibility, but a lot of Agile is really 'high process-in-the-small' • True agility means a team can adapt or select its process on-the-fly.
  21. 21. The old ways won't work in the future We need a New Model ofTesting (free from logistics)
  22. 22. A New Model for Testing Don't throw stones.Yet. I'm working on a refined version (with others). Want to help/review?
  23. 23. Forget Logistics (for the time being) Document or not? Agile v waterfall? Planned v exploratory? This technology v that technology?
  24. 24. ALLTesting is Exploratory We explore sources of knowledge ... ... to build test models ... ... that inform our testing.
  25. 25. AllTesting is Based on Models
  26. 26. Examples of models
  27. 27. Models are innate, essential, human
  28. 28. Judgement, exploring and testing Testing (the system) Our model(s) are adequate Our model(s) are not adequate Exploring (sources) Judgement Creates test models Uses test models We explore sources of knowledge to build test models that inform our testing BTW – Do Developers explore the same way? I think so.
  29. 29. Exploration process ExplorationDefinitions specs/stories Oracles Sources Design Models Test Models Enquiring Challenging Sources: People, documents, experience, system under test Modelling Test Models: Can be documented or mental models Predicting System under test
  30. 30. Testing process Testing System Under Test Refining Informing Applying Interpreting Test Models Revise the System More exploring Reporting
  31. 31. New Model Testing 29 page paper: http://dev.sp.qa/download/newModel
  32. 32. Some Consequences
  33. 33. Do developers and testers think the same way? Similar processes, not thinking the same Easier to embed testers in development Developer testing will improve
  34. 34. Covers functional and non-functional testing? Ignoring logistics means we can have the same foundational process for ALL testing (I think)
  35. 35. Relation to TDD and BDD? TDD is not testing BDD is modelling using stories
  36. 36. Testing v Checking? I submit that a Check is "a test that can be applied by a human or tool" But only humans can interpret
  37. 37. Test automation from a different perspective Automation uses different test models Is that why automation efforts fail so often?
  38. 38. DeveloperTesting The model works (for me)
  39. 39. Capabilities Enquiring, Modelling, Predicting, Challenging Informing,Applying, Interpreting, Refining Reporting and Logging
  40. 40. • Analysis, enquiry and elicitation • Modelling • Creation of custom models, using heuristics, guesses, brainstorming, ideation, creative thinking • Custom test design techniques • Comparison of models, value, advantages, disadvantages, compromises • Identification, validation and use of oracles • Predicate logic and proof • Hypothesis and inference • Socratic method • Rapid Review and Inspection techniques • Test case design • Test models and the meaning of coverage • Testing as controlled experiment • Observation, Note taking, recording A very different skillset • Basic data analysis and statistics • Decision-making with incomplete data • Computer forensics • Fault tree analysis • Failure diagnosis • Bug advocacy, triage processes and negotiation • Meaningful software and test metrics • Visual presentation of data • Reporting and presentation skills • Understanding stakeholders • Test analytics • Risk management, risk-based testing and decision-making • Critical Thinking • Interpersonal skills • Dealing with uncertainty/fallibility
  41. 41. Shift-Left is More Important than Agile
  42. 42. Developers write code; testers test because developers can’t No they don’t
  43. 43. Goal: End Manual Feature Checking Let the tools do it and free people to do testing please!
  44. 44. Testing doesn’t just apply to existing software, at the end We test systems, not software. We test early, we test often but it might not be called testing
  45. 45. Testers Need to Learn How to Code Some, perhaps many will have to
  46. 46. Testing is about measuring achievement, not quality Align systems delivery with project goals and risks and you’ve got a seat on the board
  47. 47. Testers don’t own testing anymore Be aTestMaster "Assurance through the lifecycle"
  48. 48. New Model Testing changes what testers & developers need to learn Roles of developers/testers become blurred Perhaps they are not different disciplines after all? Why have a tester certification?
  49. 49. The phase after development is REWORK, not Testing Testing must align with development, not compete with it or rescue it
  50. 50. A New Model for Testing http://dev.sp.qa/download/newModel @paul_gerrard Paul Gerrard paul@gerrardconsulting.com gerrardconsulting.com Programme Chair 2014

×