Recommended
More Related Content
What's hot
What's hot (20)
Similar to The automation break up: saying goodbye to full stack tests with task analysis
Similar to The automation break up: saying goodbye to full stack tests with task analysis (20)
Recently uploaded
Recently uploaded (20)
The automation break up: saying goodbye to full stack tests with task analysis
- 1. Automation in Testing - © Mark Winteringham - 2019Automation in Testing - © Mark Winteringham - 2019 The automation ‘breakup’ Saying goodbye to full stack tests with task analysis Mark Winteringham @2bittester
- 2. Automation in Testing - © Mark Winteringham - 2019Automation in Testing - © Mark Winteringham - 2019 How do we know if a check is valuable?
- 3. Automation in Testing - © Mark Winteringham - 2019 User can log into an application
- 4. Automation in Testing - © Mark Winteringham - 2019 Targeted?
- 5. Automation in Testing - © Mark Winteringham - 2019 Reliable?
- 6. Automation in Testing - © Mark Winteringham - 2019 Informative?
- 7. Automation in Testing - © Mark Winteringham - 2019 Maintainable?
- 8. Automation in Testing - © Mark Winteringham - 2019 Speedy?
- 9. Automation in Testing - © Mark Winteringham - 2019
- 10. Automation in Testing - © Mark Winteringham - 2019Automation in Testing - © Mark Winteringham - 2019 How do we identify and implement valuable checks?
- 11. Automation in Testing - © Mark Winteringham - 2019Automation in Testing - © Mark Winteringham - 2019 TASK ANALYSIS!
- 12. Automation in Testing - © Mark Winteringham - 2019 www.thunderboltkids.co.za
- 13. Automation in Testing - © Mark Winteringham - 2019 UI JS API SVC Click login link Build login page Interact with Login Send credentials to Auth API Parse credential s Check credentials / Create token Send token Flag as logged in Render room comp. View rooms Successful login flow Initial state: • Single page app code loaded into browser • Current state is not logged in Triggers Seam / LayerAction Key: JS/HTML HTTP JAVA USER
- 14. Automation in Testing - © Mark Winteringham - 2019Automation in Testing - © Mark Winteringham - 2019 Risk analysis
- 15. Automation in Testing - © Mark Winteringham - 2019 Parse credential s UI JS API SVC Click login link Build login page Interact with Login Send credentials to Auth API Check credentials / Create token Flag as logged in View rooms Successful login flow Triggers Seam / LayerAction Key: JS/HTML HTTP JAVA Send token USER 🔎 Initial state: • Single page app code loaded into browser • Current state is not logged in Check form is rendered correctly Visual check 🤖 Render room comp.
- 16. Automation in Testing - © Mark Winteringham - 2019 UI JS API SVC Click login link Build login page Interact with Login Send credentials to Auth API Check credentials / Create token Flag as logged in View rooms Successful login flow Triggers Seam / LayerAction Key: JS/HTML HTTP JAVA Parse credential s Send token 🤖 🔎 Initial state: • Single page app code loaded into browser • Current state is not logged in Check API handles request and response correctly API check USER Render room comp.
- 17. Automation in Testing - © Mark Winteringham - 2019 Parse credential s UI JS API SVC Click login link Build login page Interact with Login Send credentials to Auth API Check credentials / Create token Flag as logged in View rooms Successful login flow Triggers Seam / LayerAction Key: JS/HTML HTTP JAVA Send token Render room comp. 🤖 🔎 Initial state: • Single page app code loaded into browser • Current state is not logged in Check token is created correctly Unit check USER
- 18. Automation in Testing - © Mark Winteringham - 2019 Parse credential s UI JS API SVC Click login link Build login page Interact with Login Send credentials to Auth API Check credentials / Create token Flag as logged in View rooms Successful login flow Triggers Seam / LayerAction Key: JS/HTML HTTP JAVA Send token 🤖 🔎 Initial state: • Single page app code loaded into browser • Current state is not logged in Check login page HTML is correct Unit check USER Render room comp.
- 19. Automation in Testing - © Mark Winteringham - 2019 Visual checks Check form is rendered correctly JavaScript unit checks Backend unit checks Backend API checks Full stack smoke check Check API handles request and response correctly Check login page HTML is correct Check token is created correctly
- 20. Automation in Testing - © Mark Winteringham - 2019Automation in Testing - © Mark Winteringham - 2019 In Summary
- 21. Automation in Testing - © Mark Winteringham - 2019 Are your checks TRIMS? Apply task analysis Use task analysis to identify targeted checks Connect checks in a ‘chain of trust’
- 22. Automation in Testing - © Mark Winteringham - 2019 Mark Winteringham @2bittester www.mwtestconsultancy.co.uk www.automationintesting.com github.com/mwinteringham/restful-booker-platform
Editor's Notes
- In this talk I am going to explore two questions: How do we know if a check is valuable How do we identify and implement valuable checks
- So let’s start by working out whether a check is valuable or not. To demonstrate this we’re going to look at an End-to-end, full stack, UI driven, whatever you call it, check. So let’s take a look at our check in action…
- To demonstrate how we can use Trims to analyse a check, I’ve created an automated check that follows a typical pattern. It’s designed to check that a user can log into an application so let’s see it in action Demo and run End-to-End check
- So let’s analyse that E2E check Is it targeted? Not really. The check is doing everything on the UI layer, meaning if the intention is to check behavior in the backend it’s using the furthest layer from that implementations
- Is it reliable? Perhaps. This will be heavily influenced by the state of the application under test BUT this check is working against a complex system with many moving parts of which any one could fail on us. Although that maybe desirable.
- Is it informative? Let’s say something in the backend breaks causing our check to fail because it cannot find a specific element. That information is shallow at best and misleading at worst, meaning that it’s going to take considerable time to debug and react to. A risk can have many ways to impact a user
- Is it maintainable? Depends on how it was developed, if we are to assume that this check uses page object models, data builder patterns and DRY practices then we could say we’re happy that it’s maintanable
- Is it Speedy? Remember speed isn’t just time to run, but time to react to as well. Given that it’s not targeted and not very informative, then it’s fair to say that the feedback loop is going to be slow as we wade through the application and test code to work out what is exactly going on.
- We cannot rely on the scope of information from a check, but we can rely on checks if they are: Targeted Reliable Informative Maintainable Or have Speed Speed is not just how fast it runs, but how quickly we can process information and act upon it Lots of small checks that follow those valuable attributes will help us build a picture differently to how we test as humans but hopefully bring us to the same conclusion
- So it’s fair to say that that check isn’t up to muster. One of the main issues is that it’s trying to check multiple impacts
- What is task analysis? AiT is influenced by Rob Sabourin’s work on Task Analysis Breaking down a task into it’s component parts for analysis Cup of tea example
- Let’s see an example Show task analysis model and highlight Left hand side requires some knowledge of the system, but it can vary based on your knowledge Your first iteration could literally be frontend – backend We use task analysis to break down the different events that make up feature that our original risk is focused on So we have points on our model that represent each activity the system does Each action is an opportunity for a targeted check on that section And each action has it’s own inherent risks that may matter to us Finally the lines that cross boundaries between systems are integration risks to check for
- So it’s fair to say that that check isn’t up to muster. One of the main issues is that it’s trying to check multiple impacts
- This flow successfully captures the areas that could impact our application and realise the risk that a user can’t login. We can use that knowledge to identify targeted checks
- This flow successfully captures the areas that could impact our application and realise the risk that a user can’t login. We can use that knowledge to identify targeted checks
- This flow successfully captures the areas that could impact our application and realise the risk that a user can’t login. We can use that knowledge to identify targeted checks
- This flow successfully captures the areas that could impact our application and realise the risk that a user can’t login. We can use that knowledge to identify targeted checks
- So it’s fair to say that that check isn’t up to muster. One of the main issues is that it’s trying to check multiple impacts
- In summary Review checks and ask yourself are they TRIMS Break down unwieldy checks by applying task analysis to the system flow that is being covered Use the results of task analysis to determine checks that are more TRIMS Once you have checks in many different points build a ‘chain of trust’ by running them in a pipeline Task analysis can help identify dependencies for mocking