Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

"Exploring Testing from First Principles" Eurostar 2018 Keynote Alan Richardson EvilTester


Published on

What if we knew nothing about testing, how would we start? Does our every day life provides the tools we need to help us investigate a domain and develop the necessary skills we need to survive? This talk will start by assuming that about Software Testing and Technical software testing we know nothing. And then, using a process of questioning and exploration we will see how far we get in terms of building a model of Software Testing and Testing an application from a Technical perspective.

Over the course of the presentation, we will see a repeatable process of working from first principles to improve our knowledge, experience and skills. We will see that we can learn to perform complicated and technical software testing activities without having to adopt a complex understanding of software testing.

I plan to demonstrate that use of questions like "How do we know?" "How can I tell?" "What does that mean?" can lead us through a process of learning and mastery such that we can build our models of testing and our skill sets from first principles and that we do not need to overcomplicate our testing and we do not need to ask "why?". I will explain that "Why?" is a belief chain question, and we really want to focus on observable models of reality rather than 'beliefs' and 'common knowledge' about software testing.

e.g. I want to test a web application - what does that mean? What does test mean? What do I hope to achieve by testing it? How can I achieve those goals? I also have to understand some domain technology - "web", "application" which I learn from the 'web' domain rather than the 'testing' domain. When testing I ask "How do we know?" i.e. "How does it know I'm logged in?" because when we log in it creates a session which it puts in a cookie - which leads to other "What does that mean?" questions until we understand. And then we ask "How do I know it has created a cookie?" which leads us to identify how we can observe and interrogate the system e.g. "How can I see cookies". By exploring a web application from first principles with simple questions we will learn about browser developer tools, http proxies and web testing.

Published in: Software
  • Real Money Streams ~ Create multiple streams of wealth from your home! ★★★
    Are you sure you want to  Yes  No
    Your message goes here

"Exploring Testing from First Principles" Eurostar 2018 Keynote Alan Richardson EvilTester

  1. 1. #esconf | @eviltester 1
  2. 2. #esconf | @eviltester 2
  3. 3. #esconf | @eviltester 3
  4. 4. We over-complicate testing • Have you seen some of the definitions? • Have you tried to write your own definition? #esconf | @eviltester 4
  5. 5. Otherwise • it might appear too simple • it might appear easy #esconf | @eviltester 5
  6. 6. And if its too easy and simple • it might be undervalued • only junior staff get to do it • we might get paid less #esconf | @eviltester 6
  7. 7. Solution Cram in Everything #esconf | @eviltester 7
  8. 8. Software Testing is a process whereby we use a software system deployed in multiple configurations, via its GUI, API - and all exposed interfaces - to gather information about its behaviour, flaws and impact on the user and stakeholder needs and requirements; for the purpose of evaluating the quality, and fitness for purpose, in specific operational environments and conditions. A possible first draft definition #esconf | @eviltester 8
  9. 9. But we wouldn't stop there • refine our definition • simplify, chunk up • delete sections • change it over time • communicate it differently for different people But that's a communication. It isn't what testing 'is'. #esconf | @eviltester 9
  10. 10. Danger - over-complicate • How do other fields define themselves? e.g. complicated fields like Physics? #esconf | @eviltester 10
  11. 11. I should consult a dictionary or something. #esconf | @eviltester 11
  12. 12. What is Physics? "the branch of science concerned with the nature and properties of matter and energy." Google Dictionary Definition #esconf | @eviltester 12
  13. 13. What is Physics? #esconf | @eviltester 13
  14. 14. What about Rocket Science? #esconf | @eviltester 14
  15. 15. What is Rocket Science #esconf | @eviltester 15
  16. 16. Google Dictionary: What is Software Testing? "No definitions found. Search the web." #esconf | @eviltester 16
  17. 17. We have no choice We have to make this up for ourselves. #esconf | @eviltester 17
  18. 18. What is more important? Imagineedfusticating? or Working with a good Imagineedfusticator? #esconf | @eviltester 18
  19. 19. Imagineedfusticating "The art of getting exactly what you need when you need it" Definition from "The Associated Society of Imagineedfusticators for Advanced Imagineedfustication" #esconf | @eviltester 19
  20. 20. An Example Assume we know nothing about software testing. "No definitions found. Search the web." #esconf | @eviltester 20
  21. 21. How would we work from first principles? #esconf | @eviltester 21
  22. 22. Let's use a worked example #esconf | @eviltester 22
  23. 23. I've been tasked with testing this #esconf | @eviltester 23
  24. 24. I don't want anyone to know that I don't know how to test it, so I nod and then try to figure out what does software testing mean. So I look for definitions but we've seen how that turned out. So I go ask google "What is Software Testing" #esconf | @eviltester 24
  25. 25. What is Software Testing? Software testing is the process of executing a program or application with the intent of finding bugs in the software. It can also be described as the validation and verification of a software program or application or product or system. not text from a website that has high rankings for this web search #esconf | @eviltester 25
  26. 26. Validation & Verification • verification - noun “The process of establishing the truth, accuracy, or validity of something.” • validation - noun - “The action of checking or proving the validity or accuracy of something.” OK, so both are about validity, accuracy, proving and truth. Those words seem pretty similar, why would I do both? #esconf | @eviltester 26
  27. 27. OK, I'll forget that. #esconf | @eviltester 27
  28. 28. What have I learned? Software Testing is doing stuff to something to find out stuff like: • find bugs, • find things that are wrong, • find stuff that is right Warning: Be careful of certitude and absolutes. #esconf | @eviltester 28
  29. 29. Too much theory I have a job to do #esconf | @eviltester 29
  30. 30. What did they mean when they tasked me with "testing" this. • find bugs? • how will I know its a bug? • anything specific you want me to test it 'for'? • what do you want to 'do' with what I tell you? • is testing the input to something else? "What do you mean?" #esconf | @eviltester 30
  31. 31. OK... focus on the create author function and make sure it handles a range of input values "What values?" You know. Good name. No name. Long names. Errors for bad names. That stuff. "OK" "If it creates the user OK then we'll be fine." #esconf | @eviltester 31
  32. 32. We just learned: • stakeholder management • needs and requirement analysis • acceptance condition creation • risk based testing • test scope restriction #esconf | @eviltester 32
  33. 33. Why didn't I ask "Why do you want me to test this?" #esconf | @eviltester 33
  34. 34. Why? is a belief chain question #esconf | @eviltester 34
  35. 35. Why do you want me to test this? • because its your job • because I don' t want to test it • because testing sucks • because I want proof that it works • because I don't want the user finding any problems #esconf | @eviltester 35
  36. 36. That might be useful • we find out the organisation dynamics • we can 'educate' that testing does not 'prove that it works' • we could test from a user perspective #esconf | @eviltester 36
  37. 37. The temptation with "Why?" is to chain "Why?" • why is it my job? • why don't you want to test it? • why don't you want the user to find problems? Unproductive path / conflict. #esconf | @eviltester 37
  38. 38. But I care less about what they believe I care more about meeting their needs • What? • How? • When? • Where? #esconf | @eviltester 38
  39. 39. What do I do next? #esconf | @eviltester 39
  40. 40. Let me plan this out a bit • No Name - "" • Good Name - "Bob The Author" • Bad Name ? "Mr 123" #esconf | @eviltester 40
  41. 41. I can see there some validation when I just hit save What are the validation rules? I should check those. #esconf | @eviltester 41
  42. 42. #esconf | @eviltester 42
  43. 43. Is that what it is supposed to do? • How do I know what it is supposed to do? • How do I know that is the correct message? • Do I just decide if this works, or are there designs? • Should it be consistent? • "Create" -> "Added" #esconf | @eviltester 43
  44. 44. We just discovered • A need for a reference standard (oracle) • conformance to standard • requirements testing • acceptance condition validation • The need for exploration: planning, learning, revisiting plan • Consistency Heuristic #esconf | @eviltester 44
  45. 45. How do I tell people what I have done? • Better keep some notes to tell people what I've done. • I need to keep track of new things I want to do e.g. validation rules • I better keep track of the unanswered questions and concerns I have • How do people do that? #esconf | @eviltester 45
  46. 46. We just discovered • Exploratory Testing • Test Planning • Test Tools • Test Reporting #esconf | @eviltester 46
  47. 47. It said it added it, but did it? How can I see it was added? #esconf | @eviltester 47
  48. 48. Can I trust that? • How is the data stored? • What data is stored? • When is the data stored? • Where is the data stored? • How can I see the stored data? #esconf | @eviltester 48
  49. 49. We just discovered • A Testing Attitude Characteristic • Doubt • A Fundamental Testing Technique • Questioning • Create/Read flow #esconf | @eviltester 49
  50. 50. Then someone answers our questions And our world explodes with detail #esconf | @eviltester 50
  51. 51. Uhmm, OK "The validation rules are controlled by HTML using HTML5 required and pattern attributes. When the [Create] button is pressed an HTTP POST request is made to the server with the form details x-www-form-urlencoded. The author is stored in memory with an automatically generated id." #esconf | @eviltester 51
  52. 52. Maybe a diagram would help? #esconf | @eviltester 52
  53. 53. Now I have to learn about • HTML, HTML5 • required • pattern • HTTP • POST • form, form encoding #esconf | @eviltester 53
  54. 54. And how do I 'see' the HTML? #esconf | @eviltester 54
  55. 55. OMG I can change it too #esconf | @eviltester 55
  56. 56. What if I change the form so there is no validation? Then the server side validation should catch that. #esconf | @eviltester 56
  57. 57. So now I have to test the server side validation as well? "That will take quite a lot of time amending the form each time." "Test it at the HTTP level then" "How?" "Use an HTTP Debug Proxy." "A Wha?" #esconf | @eviltester 57
  58. 58. An HTTP Debug Proxy #esconf | @eviltester 58
  59. 59. What Questions Did I Ask? • How does this work? • What does this do? • How can I see that? How can I make that happen? • What should it do? • When should it do that? • Where should I see that? #esconf | @eviltester 59
  60. 60. How? Not Why? #esconf | @eviltester 60
  61. 61. What is Hard About This? Why do people find this tough? • don't like to be seen as 'not knowing' something • don't like to say "I don't know" #esconf | @eviltester 61
  62. 62. Let's see, what could we be? • Scientist • Skeptic • Believer • Seeker #esconf | @eviltester 62
  63. 63. Ken Campbell "Seeker!" #esconf | @eviltester 63
  64. 64. Robert Anton Wilson said... “Belief is the death of intelligence. As soon as one believes a doctrine of any sort, or assumes certitude, one stops thinking about that aspect of existence.” Robert Anton Wilson, Cosmic Trigger #esconf | @eviltester 64
  65. 65. Top Three Trouble-free Tactics To Tough Talk • Knowing nothing about something is good • You don't know nothing about it, what you know about it is nothing • Ask questions when you do know stuff #esconf | @eviltester 65
  66. 66. I know nothing about • Kardashians • Love Island • Big Brother That is good. #esconf | @eviltester 66
  67. 67. You don't know nothing about it What you know about it, you describe as 'nothing' • You ask questions to see if that description fits • You try to expand your model • You try to invalidate parts of your model Same as normal #esconf | @eviltester 67
  68. 68. As a Manager I ask questions when I do know stuff because: • I know other people in the room don't know • I know people have different understandings • It gives the speaker a chance to show their knowledge #esconf | @eviltester 68
  69. 69. Asking questions is good #esconf | @eviltester 69
  70. 70. The Tester! #esconf | @eviltester 70
  71. 71. Asking Questions is Good #esconf | @eviltester 71
  72. 72. End Credits • • @eviltester • • • #esconf | @eviltester 72
  73. 73. Follow • Linkedin - @eviltester • Twitter - @eviltester • Instagram - @eviltester • Facebook - @eviltester • Youtube - EvilTesterVideos • Pinterest - @eviltester #esconf | @eviltester 73
  74. 74. BIO Alan is a test consultant who enjoys testing at a technical level using techniques from psychotherapy and computer science. In his spare time Alan is currently programming a multi-user text adventure game and some buggy JavaScript games in the style of the Cascade Cassette 50. Alan is the author of the books "Dear Evil Tester", "Java For Testers" and "Automating and Testing a REST API". #esconf | @eviltester 74
  75. 75. Visit Alan's main website is and he blogs at #esconf | @eviltester 75