Successfully reported this slideshow.

CAST2011 Presenting a Compelling Software Testing Story

573 views

Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

CAST2011 Presenting a Compelling Software Testing Story

  1. 1. CAST 2011 Skills Track Presenting a Compelling Testing Story or An existentialist guide to test reporting Ben Kelly
  2. 2. Telling a Compelling Testing Story
  3. 3. http://www.youtube.com/watch?v=y7g6dKncO-I
  4. 4. Provided background and context Able to quickly clarify when the audience lost track Used auditory and visual aids to assist understanding Had extra information in reserve Note Jonsey's implicit demonstration that the information our tools give us is not the story.
  5. 5. He gave his audience a compelling reason to take specific action Did not presume to be the owner of this decision
  6. 6. <ul><li>A Story About </li></ul><ul><li>How Not to Tell a Compelling Testing Story </li></ul>
  7. 7. How Not to Tell a Compelling Testing Story <ul><li>On one project I submitted a weekly test report that was between 10-15 pages. </li></ul><ul><li>Had a nice little dashboard table that color-coded the various areas under test (shades of yellow and green early on) </li></ul><ul><li>About 2/3 into the allotted project time, a bunch of items turned red. </li></ul><ul><li>The PM demanded to know why. </li></ul><ul><li>I told him he should know already. It's all in the report and it's been coming for some time. </li></ul>
  8. 8. How Not to Tell a Compelling Testing Story <ul><li>His response was </li></ul><ul><li>'If you don't tell me about problems, I assume there are none.' </li></ul><ul><li>(...yeah, that's what I thought.) </li></ul><ul><li>All the information the PM needed was right there in the report. </li></ul><ul><li>I slaved to get that info together. Job done. </li></ul><ul><li>Right? </li></ul>
  9. 9. How Not to Tell a Compelling Testing Story <ul><li>Wrong. </li></ul><ul><li>Wrong medium </li></ul><ul><li>Wrong format </li></ul><ul><li>In short, because my test reporting was lazy, it didn't matter how good my testing was. </li></ul>
  10. 10. Reporting Methods <ul><li>Visual </li></ul><ul><li>Screen capture </li></ul><ul><li>Annotated images </li></ul><ul><li>Video recording </li></ul><ul><li>Charts/Graphs </li></ul><ul><li>Written </li></ul><ul><li>1-pager/1-slide </li></ul><ul><li>Email </li></ul><ul><li>Low-tech dash </li></ul><ul><li>Build-tool results </li></ul><ul><li>Test Session notes </li></ul><ul><li>Bug reports </li></ul><ul><li>Story notes </li></ul><ul><li>Instant Messenger </li></ul><ul><li>The Novel (don't do this) </li></ul><ul><li>Verbal </li></ul><ul><li>Standup </li></ul><ul><li>Pairing </li></ul><ul><li>Commentary </li></ul><ul><li>Bug repros </li></ul><ul><li>1-on-1 </li></ul><ul><li>MIP/Drive-by </li></ul>
  11. 11. Telling a Compelling Testing Story <ul><li>So how do you put a compelling testing story together? </li></ul>
  12. 12. Telling a Compelling Testing Story <ul><li>So how do you put a compelling testing story together? </li></ul><ul><li>I start with Modeling. </li></ul>
  13. 13. Modeling <ul><li>We create models for everything and everyone we interact with in the world. We filter all our experiences through them </li></ul><ul><li>Our experiences shape our models </li></ul><ul><li>Our models shape our experiences </li></ul>
  14. 14. Modeling <ul><li>This creates an interesting problem. </li></ul><ul><li>Not all models are created equal. </li></ul>
  15. 15. Modeling <ul><li>My model for this pen? </li></ul><ul><li>Blue </li></ul><ul><li>Generic </li></ul><ul><li>Often found chance </li></ul><ul><li>Easily lost </li></ul><ul><li>Shitty quality </li></ul>
  16. 16. Modeling – The Product <ul><li>In a given project, as you find out about the product, you create a model. </li></ul>
  17. 17. Modeling – The Product <ul><li>In a given project, as you find out about the product, you create a model. </li></ul><ul><li>Your audience does the same. </li></ul>
  18. 18. Modeling – The Audience <ul><li>Not only that, but you have a model of your audience </li></ul>
  19. 19. Modeling – The Audience <ul><li>Not only that, but you have a model of your audience </li></ul><ul><li>And they have a model of you. </li></ul>
  20. 20. Modeling – The User <ul><li>Where it gets even more fun is where you're reporting on test scenarios you've run for end users. You each have separate models for them too. </li></ul>
  21. 21. Modeling
  22. 22. Modeling Your Audience <ul><li>Ask questions: </li></ul><ul><li>Imagine the project is a wild success. Describe what that looks like to you. </li></ul><ul><li>What needs to happen in order for this project to be successful? </li></ul><ul><li>What are you afraid might go wrong? </li></ul><ul><li>What are the most important aspects of this project to you? </li></ul><ul><li>Ask for war stories about old projects. What did they take away from those? </li></ul><ul><li>What if you get pushback? </li></ul>
  23. 23. Modeling
  24. 24. Modeling
  25. 25. Meta-modeling
  26. 26. Meta-modeling <ul><li>(Making a model of another model) </li></ul><ul><li>How do we build a meta-model? </li></ul><ul><li>Speak to the audience </li></ul><ul><li>Where are they unsure or fuzzy? </li></ul><ul><li>Look at the product through their eyes </li></ul>
  27. 27. Meta-modeling <ul><li>Compare your product model to their meta-model </li></ul><ul><li>What's different? </li></ul><ul><ul><li>Are the differences important? (Do they conflict?) </li></ul></ul><ul><li>What can you tell them that they don't already know? </li></ul><ul><li>What might they tell you? </li></ul><ul><li>How can you tailor your reporting to fit their model? </li></ul><ul><li>How can you enrich the model they have (or vice versa)? </li></ul>
  28. 28. Reporting
  29. 29. Reporting <ul><li>I like to split reporting into two categories. </li></ul><ul><li>Pull reporting </li></ul><ul><li>& </li></ul><ul><li>Push reporting </li></ul>
  30. 30. Pull Reporting <ul><li>Pull reports are generally information gathering exercises from the viewpoint of the audience. </li></ul><ul><li>Your audience generally has enough context that they don't need to be pre framed </li></ul><ul><li>You may not have the same luxury </li></ul>
  31. 31. Pull Reporting <ul><li>Some examples of Pull reporting </li></ul><ul><li>Daily Stand-up </li></ul><ul><li>Drive-by sit-rep </li></ul><ul><li>Weekly meeting </li></ul><ul><li>Ambush </li></ul>
  32. 32. Pull Reporting <ul><li>Pull reports are often short. </li></ul><ul><li>They often have a sense of urgency (they are not always urgent) </li></ul><ul><li>Skills for Pull reporting </li></ul><ul><li>Remember to breathe. </li></ul><ul><li>If you're not sure, ask for clarification. </li></ul><ul><li>If you're not prepared, ask for time. </li></ul><ul><li>Some questions are better left unanswered (beware WHY) </li></ul><ul><li>Don't be afraid to speculate (but identify that you are) </li></ul><ul><li>Michael Kelly's MCOASTER heuristic </li></ul>
  33. 33. Pull Reporting <ul><li>MCOASTER </li></ul><ul><li>Mission </li></ul><ul><li>Coverage </li></ul><ul><li>Obstacles </li></ul><ul><li>AUDIENCE </li></ul><ul><li>Status </li></ul><ul><li>Techniques </li></ul><ul><li>Environment </li></ul><ul><li>Risk </li></ul>
  34. 34. Push Reporting
  35. 35. Push Reporting <ul><li>Initiated by you in order to share information you believe that a specific audience needs to hear. </li></ul><ul><li>This is where your story building skills come to the fore </li></ul><ul><li>Take the time to ensure your audience is briefed </li></ul><ul><li>Examples of Push Reporting: </li></ul><ul><li>Bug Reports </li></ul><ul><li>Critical bug discovery </li></ul><ul><li>Risk report </li></ul>
  36. 36. Push Reporting <ul><li>Skills for Push reporting </li></ul><ul><li>Preparation </li></ul><ul><li>Purpose. Story </li></ul><ul><li>Understand the model of your audience </li></ul><ul><li>Understand your audience's model of you </li></ul><ul><li>Execution </li></ul><ul><li>Pre-frame your audience </li></ul><ul><li>Have the information to back up your story </li></ul><ul><li>Have your story suggest your recommended course of action. Don't demand action. </li></ul><ul><li>Elicit Feedback </li></ul>
  37. 37. Push Reporting <ul><li>Skills for Push reporting </li></ul><ul><li>Consider the various media at your disposal use as much as you think is useful </li></ul><ul><li>e.g. A presentation with follow-up email outlining action points </li></ul><ul><li>(Beware of pushing too much) </li></ul>
  38. 38. Your Relationship with the Audience <ul><li>You will have your work cut out for you if your audience does not trust you. </li></ul><ul><li>At the other end of the scale, if your audience trusts you too much, they may fail to weigh other factors with sufficient gravity </li></ul>
  39. 39. Your Relationship with the Audience <ul><li>You can borrow credibility. </li></ul><ul><li>Find a trusted advisor – </li></ul><ul><li>recruit someone who has the ear of your audience if you do not. </li></ul><ul><li>You can have them </li></ul><ul><li>introduce your topic </li></ul><ul><li>report with you or </li></ul><ul><li>report for you. </li></ul><ul><li>You can use back-channels </li></ul><ul><li>If you are presenting to a hostile audience (or an indifferent one), consider an alternate audience to achieve the same outcome. </li></ul>
  40. 40. Your Relationship to the Audience <ul><li>You can build credibility. (It takes time, but in the short term) </li></ul>
  41. 41. Your Relationship to the Audience <ul><li>Speak the language of your audience </li></ul>
  42. 42. Your Relationship to the Audience <ul><li>Avoid hyperbole </li></ul>
  43. 43. Summary (tl;dr) <ul><li>Understand and practice modeling and meta-modeling </li></ul><ul><li>-Model your audience, meta-model their view of the product </li></ul><ul><li>-Align your presentation to meet their model or </li></ul><ul><li>-Reframe their model to match your story </li></ul><ul><li>Pull reporting (The audience comes to you) </li></ul><ul><li>-Take your time </li></ul><ul><li>-Practice off-the-cuff delivery (MCOASTER) </li></ul><ul><li>Push reporting (You go to the audience) </li></ul><ul><li>-Prepare </li></ul><ul><li>-Borrow credibility if yours is lacking </li></ul><ul><li>-Build a story that has meaning to your audience </li></ul>
  44. 44. Questions?
  45. 45. References/Further reading <ul><li>Please Understand Me (2) – David Kiersey </li></ul><ul><li>The Structure of Magic (Vol 1 & 2) – Richard Bandler, John Grinder </li></ul><ul><li>Seeing Like a State – James C. Scott </li></ul><ul><li>Bug Advocacy – Cem Kaner www.kaner.com/pdfs/BugAdvocacy.pdf </li></ul><ul><li>MCOASTER Heuristic – Michael Kelly http://www.informit.com/articles/article.aspx?p=457506 </li></ul><ul><li>Components of software testing image – James Bach </li></ul><ul><li>Thanks to Tim Coulter for corkboard.me </li></ul><ul><li>Images used in this presentation are copyright of their respective creators. No ownership is sought or intended. </li></ul>

×