Dev Ops or Not - Testing Has To Change - Richard Hunter
•Download as PPTX, PDF•
1 like•805 views
Presentation by Richard Hunter, QA Manager at Connected Car at Jaguar Land Rover, on the topic of how Testing has to evolve in the same way as Software Development did over the last few decades. Given on 18 October 2017 on the occasion of SpotQA's FutureTest Series event #1 in London.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Dev Ops or Not - Testing Has To Change - Richard Hunter
Similar to Dev Ops or Not - Testing Has To Change - Richard Hunter (20)
Recently uploaded
Recently uploaded (20)
Dev Ops or Not - Testing Has To Change - Richard Hunter
- 2. DEV OPS OR NOT – TESTING HAS TO CHANGE RICHARD HUNTER DEV OPS OR NOT – TESTING HAS TO CHANGE OCTOBER 2017 R I C H AR D H U N T E R
- 3. CONTENT All about me Pace of change in development vs testing Complexity of delivery Why does this matter What do we need to do What skills are required to get there Why traditional test automation misses the point Testing in production. Why this matters Conclusions
- 4. ALL ABOUT ME • Career in software testing across many industries and clients • Currently working at Jaguar Land Rover as Global Quality Manager for Connected Car • Previously worked with Sainsbury's, BBC, ITV, Disney, Home Office, Animal Health, Skills Funding Agency, C4, C5, HRMC, Wythe Pharmaceuticals, EMI, Nuffield Health etc. • Worked in consultancy as well as real jobs DEV OPS OR NOT – TESTING HAS TO CHANGE RICHARD HUNTER
- 5. CHANGES IN DELIVERY – DEV V TEST Testing practices have stagnated • Most testing done at the end of development • Most testing is done without community led tools • Most testing is performed from a user point of view, using the UI to drive the testing Stats to back this up from the WQR Development practices have moved on, technology wise, hugely in the last 20 years Unit testing Component / headless testing Stubs / harnesses / Virtualization Community led tooling Release Management Code complexity Deployment CI DEV OPS OR NOT – TESTING HAS TO CHANGE RICHARD HUNTER 46% lack of facilities to manage environments 46% lack of right tools
- 6. COMPLEXITY • We live in a connected world. The customer requires the websites and apps they use integrated across everything I do. • A wonderful to use online recipe builder will not get the audience it needs if I can’t integrate to my email client, my calendar, my interactive calendar on my fridge or be able to send the ingredients into my weekly online shop. • It will also not take off if I can’t use it across iOS, Android, Windows, Mac, Chrome, IE(yes, still!), Firefox, different devices etc. • Too complex for a traditional manual approach or even a traditional automation approach • World Quality report - 85% internet of things part of their business operations • World Quality report – 68% of those 85% do not have a plan to test the internet of things • Now throw in the expectation of quick change to align to our businesses and customers DEV OPS OR NOT – TESTING HAS TO CHANGE RICHARD HUNTER
- 7. WHY DOES THIS MATTER? • World Quality Report 2012 Test Spend – 18% • World Quality Report 2018 (predicted) Test Spend – 40% • Top 7 Challenges in Application Development include: • Design Complexity (43%) • Difficulty Integrating with other and legacy systems (42%) • Reliance on Manual testing (40%) • Expectation of quick change RICHARD HUNTERDEV OPS OR NOT – TESTING HAS TO CHANGE
- 8. WHAT DO WE NEED TO DO • Micro and Macro approach (our own teams and the wider community) • Move down the application stack, test as early as we can in terms of development availability • API testing • Data testing • Integration testing • Headless functional testing • Aim to test as much without the UI as possible, build up to the UI by making sure that you do not have issues in the layers below. • Be braver in our community • Use open source tools. Contribute to the use of them (coding, feedback, feature requests) • Integrate your tool sets. Make it part of your development lifecycle that you take care of and develop your own tools. Share them. • Work with solution providers and tooling companies to develop their offerings. Tools are lagging behind the requirements – vendors will only change if we lead them. • Share across the community testing approaches that work for you. RICHARD HUNTERDEV OPS OR NOT – TESTING HAS TO CHANGE
- 9. WHAT SKILLS ARE REQUIRED TO GET THERE? • It will vary between companies, applications, landscapes, but generically we need to get our teams to be comfortable enough to: • Think about the data flow between components and systems. How do you prove an integration without integrating it? • Pick apart your application / design / landscape. How do you test at each level without need the whole? • Test with APIs – proving an API without the other system • Integration to other systems – JSON calls, SOAP calls etc. • Think about data integrity – how is your team currently testing that • Think about the tool gap – what is the ideal? What can you afford? How do you bridge the gap? • Get involved in your testing community and help develop and direct the tools that we use. RICHARD HUNTERDEV OPS OR NOT – TESTING HAS TO CHANGE
- 10. TRADITIONAL AUTOMATION – MISSING THE POINT? • Focus is on automation of your current manual testing • Doing what you do now, but quicker • With a tooling approach to testing, rather than an automation approach, so much more can be achieved. • Work on the application stack, building the testing up through the layers • Understand what is tested by develop and the technical testing, and build on that with traditional automation • Use UI tools to widen what you do – coverage and data inputs RICHARD HUNTERDEV OPS OR NOT – TESTING HAS TO CHANGE
- 11. TESTING IN PRODUCTION • Automate with production in mind • Integrated world means changes outside of your control can impact your product • How do you currently know that a new release has worked? • How do you currently know what your customer experience is? • Forgot about testing to get a product to production, focus on testing your customers experience. RICHARD HUNTERDEV OPS OR NOT – TESTING HAS TO CHANGE
- 12. CONCLUSION • Testing is dead (we should kill it) • Bums on seats model • Manual process throughout the lifecycle • Automation focussed on the execution of scripts • Using the code created to test the whole • Long live testing • Intelligent approaches to testing • Tool driven • Community led • Automation of tasks not execution RICHARD HUNTERDEV OPS OR NOT – TESTING HAS TO CHANGE
- 13. DEV OPS OR NOT – TESTING HAS TO CHANGE RICHARD HUNTER THANK YOU
Editor's Notes
- Slide 1 min Total – 1 min The purpose of the presentation is to cover a bit of background on the complexity of delivery to the modern customer, how testing is not moving with that complexity in the main, why traditional automation is not working, what we need to do and how that applies across the lifecyle. I called this presentation “Dev Ops or not – testing has to change” as my primary point is that the delivery methodology is not the only catalyst for change. The latest industry improvements is moving to dev ops and with it a slew of articles and industry talks about why dev ops may or may not kill testing as it is done now. Actually the complexity of the delivery landscape, when we are focused on “real” customers, is the reason why the industry has to change. There are many great teams and people out there who are miles ahead of what I am presenting today – however that is the top 1%. The vast majority of the test industry is still about manually testing the user interface and throwing people are it. This is a quick insight from my perspective on why the big hitters need to change or will die.
- Slide 30 seconds Total – 1.30 All company presentations have a bit about them, their customers, their sales growth etc. So I thought I would do the same to provide a little background that I have been around the block, so have put up some key facts. Let’s move on
- Slide 3 mins Total 4.30 This slide is unfair in many ways. What I am trying to say is that 20 years ago, unit testing in an automated way was not widely practiced, code reviews are done manually if at all, complexity of code reviewed by a single peer if at all, release management tools make life far easier, CI tools also have come on leaps and bounds. The reason – life is getting more and more complicated. What is delivered in today’s world can not cope without the toolsets used as normal practice everywhere I go. When I started at BT more years ago than I care to tell you, the development practices performed then would be rejected today as ludicrous. The testing practices are broadly recognizable across the industry. One of my previous employers a few years ago, a large organization, test team of 150+, had no test tool usage at all, other than an expensive test management tool, that was many years out of date. They are not in the minority. As an industry, the “test team” starts when the UI is available, and we drive the testing through that with the occasional delve into something more technical if we really have to. Data integrity testing, API testing, integration in the small as we used to call it are simply not done, assumed to be left to the development teams. One of the issues is that community test tools have not, broadly, taken off. We still pay huge licence fees for basic test management tools and automation tools – and get tied in forever. Expand on this if I get time.
- Slide 1 min Total –Slide 2 min Total – 6.30 mins At this point you could ask – so what? Testing might not have changed broadly, but the job is getting done. However – delivery in the connected world is getting complex. Gone are the days that you can attempt to launch a standalone app or website. It has to be integrated into everything else. The websites and apps that are not connected to the wider world will not make it. Talk through the example So if our test suppliers and industry in general do not upskill in their approach and tools, they will no longer be relevant in today’s testing landscape. The bums in seats model is rapidly losing credibility in this world. 1 min The purpose of the presentation is to cover a bit of background on the complexity of delivery to the modern customer, how testing is not moving with that complexity in the main, why traditional automation is not working, what we need to do and how that applies across the lifecyle. I called this presentation “Dev Ops or not – testing has to change” as my primary point is that the delivery methodology is not the only catalyst for change. The latest industry improvements is moving to dev ops and with it a slew of articles and industry talks about why dev ops may or may not kill testing as it is done now. Actually the complexity of the delivery landscape, when we are focused on “real” customers, is the reason why the industry has to change. There are many great teams and people out their that are miles ahead of what I am presenting today – however that is the top 1%. The vast majority of the test industry is still about manually testing the user interface and throwing people are it. This is a quick insight from my perspective on why the big hitters need to change or will die.
- Slide – 1 min. Total – 7.30 mins Cost of testing is increasing. Currently at 31% (down from 35% last year) but predicated to be 40% of the budget for a project by 2018. Can be tied directly into the top reported problems – design complexity, integration, manual testing, I will throw in lack of test environments as well. This is unsustainable from a pure delivery point of view. Through in the expectation of quick change from an agile or devops perspective, means more repeated testing, increasing the cost as we go forward (major driver behind the 40% expectation) From a business point of view, if the answer to reduce the spend is to reduce the integration or complexity you will not be competitive – you are left with how do you test.
- Slide – 2 min total – 9.30 mins To keep up with the connected world and the complicity that brings, and to start to reduce the test budget we need the test industry to evolve. We can do that in our own small ways and in a larger community driven approach as well. For our own teams, we need to move down the application stack, to be testing at the levels below the UI. Moving away from traditional “automation” to be using tools to help achieve our testing. Move to test the data, the APIs, the layers of transferring the data. Like the traditional automation triangle, the testing done on the UI should be the top of the triangle, not the body of our work. We should move away from thinking about manual testing (no tools) and automation testing (automate what we would have run manually) to think about the best way to test – using the right tools, right approach and testing at the correct levels to get to the end goal. This is not new – however our part of the industry is miles behind where it needs to be, and we in the industry need to drag it forward. The reason the development community has moved on so much, or one of the reasons, is that a technical community has come together and produced some great tools, that change and evolve as they go. We need that more in the test community. We can all name some tools out there, buzzilla for example, but they do not have the community behind them in the way a Jenkins type tool does. We need to start coming together behind tools and helping companies and groups of individuals to drive the direction and improvements of what those tools can do.
- Slide – 1 min 30 Total – 11 Mins So far, I have presented a negative picture of the testing industry. This is unfair – there are some great people in it, pushing us forward and have been, for longer than I have been in a career, talking about exactly the same things as we have covered here – making our teams skilled. We need to help our current teams skill up. Take your most technical resources and form working groups to pick apart your testing strategy from a technical point of view. If you have an ecommerce platform, what are you doing to test the data? What are you doing to test the APIs to your other systems? What are you doing in the integration layers? How are you simulating those parts of the system that are not available? Take an approach on to working out when is the first time you can test something and how you could do that as early as possible – what tools do you need? What techniques can be applied? Forget about automation – take a tooling approach and build up the layers – the automation will naturally follow. Look at what you need to integrate to? How do you know that will work? How do you test across your devices? Aim – how do you get to the point that you can change the UI and your testing regime will still work to a greater extent?
- Slide 2 mins – Total 13 mins I have mentioned in the previous slide to ignore traditional test automation. This is not quite true – but don’t focus on it as a starting point. Test Automation, as it currently stands, is such a small part of the whole of testing and can provide such little value. Go and have a look at Alan Richardson’s blog post on this, I read this week and I think it is fairly new – brilliantly explains in pictures how a traditional test automation approach looks at a small subsection of what could be achieved. So if I go to any number of test service providers and ask for test automation, they will take my current manual tests and automate them, in an expensive propriety tool, it will take a long time and very likely will never get to the point of a ROI. The problem is that it is looking to do what you do now, but quicker. And possibly against a wider variety of devices / browsers etc. Whilst this is valuable, there is so much more we can be doing first. Alan goes into, far better than I can, how you can automate the other parts of your testing process. Great read. What I would like to say here is to forget about test automation, as it is currently thought about. It should not be a way of doing what you already do, slightly quicker. What should be an aim, is how do you test at the right levels your application stack using the right tools to get you there in the most efficient way. If that means automating some data creation, and that is a one off process that you then throw away the scripts, then fine – if it is a way of automating your data inputs and outputs of your interfaces using SOAPUI (other tools are available), just as great. If it means creating your own tools – brilliant. Share them on github. Make them great for all to use. Then what you do end up automating, get into your CI tool. Then, and only then, do you attend to your UI. You build on what you have already tested, and now your focus is correct. You are not worried about data flow, or integration to other systems, you know that works. You are now interested in what the customer sees – how does it work for them? Does the functionality work from a UI perspective? Does it work across your target devices and browsers? This way you know (for a given value of know) that the underlying functionality will work, you are now interested in your customer and the issues will be in the UI layer or compatibility issues. – that statement is clearly not true. You will find other types of issues, but they should be reduced and when you do, find out why, improve your testing, rinse and repeat.
- Slide 1 min – Total – 14 mins One of my pet subjects is about widening the scope of testing. Traditionally we focus on getting the developed project to the customer. We then run off to the next project, or at best, support the current one with new releases, regression testing as we go. That is because most of our testing is manual, not tool based, and the parts that are automated are reliant on known data, can’t cope with change and not designed to do anything else. My view is that you should design automation to be able to be run against your production environment as well. How do you know a release worked? How do you know that the functionality is still working? (monitoring the servers will only do so much). What does your customer actually see? If you integrate to other systems out of your control, how do you know what they have changed does not impact you? In a devops driven connected world, a change elsewhere can easily look like your issue to a customer. If you design in your testing to both prove your functionality in your test environment as well as your production environment, you will get extra ROI of your work from monitoring production as well.