Making testability our mission
•Download as PPTX, PDF•
0 likes•96 views
This is a call to action. On our cross functional teams and during our devops transformations we talk about how testing is for the whole team. Quality is everyone's responsibility. How much are we really doing to make this happen? Often we are working on systems that are hard to test for many reasons, but if we simply do more testing, write more automation we are neglecting what should be our main mission, advocating for increasing levels of testability, to truly get everyone involved in testing. We all have stories about how something is difficult to test, often never being tested or certainly left with the tester to figure it out. It doesn't have to be this way. During my talk, I want to introduce a set of principles for testability engineering. A new way to approach our work as testers. These principles will tackle how we make our systems more observable, controllable, how we share knowledge across teams and improve the testability of our dependencies. I believe it is time to create a new focus on testability, as it affects everything we do, what our teams do and beyond into how value is delivered for customers. I want you to take away from the talk: * Why a focus on testability can multiply your effectiveness as a tester * What the principles of testability engineering are and how to advocate for them * How you can make iterative changes to what you do in order to embrace testability New technology and complexity is rendering many software development techniques and paradigms obsolete at an increasing rate. We already exist in a space where an infinite number of tests of an array of different types can be performed. A new mission is needed, one that leverages the varied talents of all kinds of testers and culminates in a new focus on the exponential benefits that testability brings.
More Related Content
What's hot
What's hot (20)
Similar to Making testability our mission
Similar to Making testability our mission (20)
More from Ash Winter
More from Ash Winter (20)
Recently uploaded
Recently uploaded (20)
Making testability our mission
- 4. “Testing is a whole team activity” ~ lots of people we know @northern_tester Q449
- 5. “There’s a testing bottleneck” ~ all project managers @northern_tester Q449
- 6. “We can’t test that, its just a config change” - those devs @northern_tester Q449
- 7. “There is never enough time to do enough testing” ~ too many testers @northern_tester Q449
- 8. Lets drop some truth bombs @northern_tester Q449
- 9. If its hard to test… …it won’t get tested …you the tester will test it …your automation will be off target …your non-functional testing will add little value …your application will drive your ops people mad …your team will have a role based wedge FOREVER @northern_tester Q449
- 10. Breaking it down… •Reverse inspiration •Testable Architecture •Testability Engineering •What YOU can do @northern_tester Q449
- 11. My First Testing Role… • 2000 bugs in 2 years • Ticket communication • Long test cycles • Mastered “Quality Center” • Winning? @northern_tester Q449
- 12. A lack of testability warped what I thought testing was. And its happening to YOU @northern_tester Q449
- 13. Testability is a superset @northern_tester • Contains capabilities • Hard to define ”Easy to Test” • Co-opt • Testable architectures Q449
- 14. Observability @northern_tester Q449 • As it is over perceived to be • Oracles to evaluate problems • Tracing microservices
- 15. Controllability @northern_tester Q449 • Depth & Breadth • Delay to testing • Mocks
- 16. Decomposability @northern_tester Q449 • Isolation • Integration • Circuit breakers
- 17. Simplicity @northern_tester Q449 • Technologies • Problems • Are we done yet?
- 18. Introducing Testability Engineering* * Certification coming soon, only $10000 @northern_tester Q449
- 19. #1 Each layer of your architecture must have feedback loops @northern_tester Q449 • Without feedback, risk accumulates • Tests of different types, speeds & cadence
- 20. #2 Testable systems create the conditions for collaboration @northern_tester Q449 • Crossing divides • Integrated arguments Continuous Testing in DevOps, Dan Ashby, https://danashby.co.uk/2016/10/19/continuous- testing-in-devops
- 21. #3 Architectures designed for simplicity improve team flow @northern_tester Q449 • Architecture constraints manifest in team constraints • Complexity excludes contribution
- 22. #4 Testable systems open and expand organisational relationships @northern_tester Q449 • Flow of risk, fears and claims • Test what matters • When it matters
- 23. #5 Proactive instrumentation exposes information that matters @northern_tester Q449 • Judged on finding important problems • Tooling helps • Humans interpret Monitoring in the time of Cloud Native, Cindy Sridharan, https://medium.com/@copyconstruct/monitoring-in-the-time-of-cloud-native- c87c7a5bfa3e
- 24. #6 Testing is most effective when state can be controlled @northern_tester Q449 •No control = ineffective testing •Managing state •Time to start testing
- 25. #7 Good operability practices are synonymous with testability @northern_tester Q449 • Operability vs Features • To operate is to test Runbook Collaboration Template, Matthew Skelton, http://runbookcollab.info
- 26. #8 Enhancing adjacent testability improves the wider system @northern_tester Q449 • Winning again? • Adjacent systems constrain us • Systems thinking
- 27. What can YOU do for YOURSELF? @northern_tester Q449 • Learn source control deeply • Get into operability • Be able to draw your architecture • Building blocks of technology • Write some code • Talk about what people care about
- 28. What can YOU do for your team? @northern_tester Q449 Testing in Production, the safe way, Cindy Sridharan, https://medium.com/@copyconstruct/testing-in-production-the-safe- way-18ca102d0ef1
- 29. It almost always starts with a leap @northern_tester Show your team how you test, you may need to be vulnerable Q449
- 30. Think holistically @northern_tester Created by Rob Meaney, follow him @RobMeaney Q449
- 31. Tackle as a team @northern_tester https://github.com/ConfluxDigital/testability-questions Q449
- 32. Include the whole organisation @northern_tester Created by Ash Winter and Rob Meaney, in partnership with Ministry of Testing Q449
- 33. Testers as information providers is often a guarded position. Lets open up, become testability advocates instead. @northern_tester Q449
- 34. Outsourced? Cost centre? Systems thrown over the wall? Testability can be our DevOps @northern_tester Q449
- 35. I believe in testers as positive change agents for whole team testing cultures. I believe in testability to enact that change. I believe in all of you. @northern_tester Q449
- 36. Thank you for your attention. https://github.com/northern- tester/testability-engineering @northern_tester Q449
Editor's Notes
- Im not sure testing is really working out that well. We always seem to be in the middle of existential angst. Agile, DevOps, what shall we fail to embrace next? Lets change our footing.
- Lets add models here. Quadrants, THSM, Pyramid etc. Then a big cross. If its hard to test. Your strategy is for shit. Sorry.
- Some people say this, but it is slightly less vague than quality is everyone’s responsibility. Describes what a bit better and who might be doing it. Still hard to aim at while building hard to test systems.
- There is no bottleneck of the amount of testing, the real bottleneck is poor testability. Or to be more specific, the ability of the team to test and the testability of the system in question.
- Ah ha! Now we are getting to it. We’ve heard this one right? Its a config change. A no-op release to pick up a new setting. Don’t you worry about it. ‘Just’ is one of my favourite words in testing. This actually means with all the forms of testing we currently do and I know about, we can’t imagine how to test this. Ability to test and testability meet head on. Guess what happens next.
- I always enjoy this one. There is ALWAYS enough time to do any amount of testing. However, a better phrasing is the that in the time we have, we need to do the most effective testing for the risk at hand. Testability gives us this.
- We’ve all worked on hard to test systems. If you think you haven’t then you are or were in such denial that you were a captive reliant on their captor. Stockholm syndrome. So, lets drop some truth bombs on our candy asses.
- Simply won’t get tested - here’s a secret for you. IF IT HASN’T BEEN TESTED IT DOESN’T WORK, new stuff rarely does. You the tester will be burdened with it. Volunteers will be hard to find. Automation will target the areas that don’t break, or will just cover new stuff, rather than old fragile areas. Even if you can do performance and load testing, it will be brittle, late and on inappropriate environments. Probably mislead you more than lead you. One of the key indicators of poor testability is lack of diversity within your testing. IMPORTANT - your poor ops people. Sys Admin, DBA’s and App Support will be driven mad by your application. Hard to test means hard to operate in live, where it matters. Finally - your team will be divided by this system that you all hate, right down the lines of role. Unchecked, this will persist FOREVER.
- Reverse inspiration Core concepts Testability Engineering Its about YOU
- 2000 bugs in 2 years Communicated through tickets Longs test cycles against builds on long lived environments Mastered weirdly named tooling “Quality Centre” Left with a weird feeling - we did tons of testing, but we never got any faster, no one got what they wanted…
- I thought testing was raising bugs, communicating via tickets
- Superset as in, other ililities are contained within it Which makes it ethereal at times which is part of its problem, it his hard to describe but makes the world better. For me this is true of a lot of aspects of testing, where we co-opt other technologies to enhance our testing. One of the things that makes testability so intuitive as a direction for the craft of testing. But also makes it important, it is telling us to focus on the whole system, rather than making local optimisations. Let’s make this a bit more real, by talking about 4 core ilities of testability. In no particular order though.
- Observability allows us to understand the system as it actually is - we can explore and ask questions of the system Observability determines what problems we can detect and how we evaluate if they are problems observability tools and techniques are the lens to view and filter that information. Tracing through a micro service architecture is a great example of this. Seeing the whole transaction throughout a set of dependent services. Great for seeing effects and side effects of a behaviour.
- Controllability determines the dept and breath of our testing efforts - how deep you can go while still knowing what breadth you have covered. Without this you can go down the rabbit hole and miss the bigger picture. Without control testing is pushed later and later, Controllability determines what scenarios we can exercise - whether it be setting test data to the right state or ensuring a dependency returns a specific response.
- The ability to isolate components from one another - to truly know the effects that either being connected to (or not connected to will have an impact on the component you are testing) - more importantly knowing that you can develop and test wherever you want and isolate problematic areas. Being able to isolate problems easily speeds the development effort, moving from guessing where problems are, to isolating components and interactions, to chase problems to their origin https://martinfowler.com/bliki/CircuitBreaker.html
- The more complex the system, the harder it is to test. Sounds intuitive right? The harder it is to reason around a system, how many technology types, transport mechanisms, input, outputs, dependencies it has, the more problems can occur. Lots and lots of problems means lots of time spent testing, clarifying, checking, asking, exploring, re-exploring. You get the picture.
- Lets address the question of how do we make testing more meaningful for all team members, as usual we need some principles to work by, without those we are attempting to change the world from no platform. These should also be inclusive of roles, whether your focus is humans, technology or a mixture of both.
- A silent system is of limited use to anyone. We should be aware that for someone, somewhere, on some device your system is a pile of unusable rubbish that meets none of their needs. You just sleep well as you don’t know about it. Testability enables diverse testing activities, tests of differing speeds, testing when it matters most and giving actionable information. Lim ited forms of testing give you limit feedback. Risk accumulates with slow feedback right?
- Everyone needs to collaborate more. But that needs a basis, especially across roles and disciplines. Testabaility is a joint endevaour. If you have been forged in a culture where devs do dev and testers do testing. Breaking down hen it isn’t too surprising when that doesn’t happen naturally. How can testability assist with that basis? Imagine you have a problematic dependency and two teams argue over it. Your service is always down! No, your integration is poor! This conversation enriched with feedback( from both systems is a much better one. The same goes within team. Better testability == better feedback == better relationships between roles. We are providing critique on work, even done well it is a balance we need to strike.
- This is big. I’m going to make a statement here: Testable architectures are a leading indicator of quality. If an architecture is overly complex, has components that your team doesn’t control, you are heading for a fall. The flow of work through your team is dependent on architecture too. Supposed bottlenecks in testing are often bottleneck in architecture. Have a single team of DBA’s/Sys Admins responsible for your database/hosts? You will have a bottleneck that manifests in testing. Most systems have grown up over time, therefore serious diagnosis of the smells of hard to test architecture is needed. They don’t describe themselves as it.
- Testability means that you are testing what matters most, when it matters most. The key here is who is involved in that statement can really help with the what and the when. In order to truly engage with the risks and rewards feared and valued by organisations, relationships between stakeholders are needed Sounds obvious right? Often the world isn’t shaped like this though. It wouldn’t be a talk about testing without something about the dark art of communication. To improve a system, you must learn its interactions, organizational relationships assist the flow of information between people and teams.
- We must inherently accept that we cannot find all the problems right? Observability gives us this power to give our selves the best chance if finding the problems that matter. At a previous organization, we implemented New Relic, including its mobile front end agent to allow us to see client side errors. The avalanche of javascript errors was something to behold. Note the use of the word “helps” observability tools can see for us, but not interpret. The most common javascript error was mysterious. A stack trace is only a stack trace. The error was actually caused by losing network signal, which caused a much deeper error but the manifested error did not show that. An explorer was needed. But combined with the tooling, the real cause was found.
- Have you ever got the feeling you aren’t really in control of the system under test? You exercise it in one way and strange things happen elsewhere. When you talk about your testing without being in control of your state you are providing bad information. We understand that when your system state is not controllable the side effects of testing cannot be observed, which compromise the effectiveness of your testing. For testing to give the most relevant and timely information, controllability is required. Same tooling and mechanisms where sensible for test environments and production.
- If your system is hard to operate then it stands to reason that it will be hard to test. To be honest, its often where the problems really lie. As opposed to testing features over and over. I often hear this when asked to perform performance and load testing. Often systems are not operable enough to be tested in this way. They do not have logging, monitoring and tracing, unpatched servers, dodgy test environments. To operate, is to test. Solidarity with our friends in operations will go a long way. They have suffered.
- Your own testability is constrained by your dependencies. It has ever been so. I worked on the most beautiful API, acceptance test automation of my dreams, tracing, feature toggles, disposable environments. Apart from a steaming shit of a dependency where we had to raise a jira to add test data. Which would be arbitrarily deleted whenever this internal team fancied a change. Constant build failures. Ended up mocking it, then it was FUBAR in live. Bleh.
- Source control – leading indicator of quality Architecture FFS learn some code – its one of your teams primary artifacts. How can your team get interested in testing if you aren’t interested in THE PRIMARY FUCKING COLLABORATION ARTEFACT. How much is up to you. Build a basic version of your app. Talk about exploration/automation/perf/security/accessibility as though your team should care about it Talk external to your team about things stakeholders care about. Ops people want not to be woken up, Product people want to validate changes. STOP FUCKING TESTING EVERTHING – hold your nerve. Leave it in the column for a day or two. See who blinks.
- Show them the world of testing that is possible. The more testable, the ways become open.
- Everything starts with a leap from someone, it may as well be you. The simplest starting point is to show someone who may be able to help, how you test.
- Has the ability to observe and system has the characteristic of observability
- Has the ability to observe and system has the characteristic of observability
- Being an information provider is usually a position from which we defend ourselves against the largesse of organisations. How can we switch footing from defending to reaching out is the challenge, we need a new
- You know who else was a victim of this. Our friends in Ops
- I believe in testing as a force for focusing on value and risk. For me, this includes less fucking around with features and focusing where it really matters. I believe that testability can enable our teams to share this focus. I believe that it starts with us. Right here, today. All of us. You can do it.
- You will find the principles of testability engineering here