Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

No Devops Without Continuous Testing

772 views

Published on

Arthur Hicken Chief Evangelist of Parasoft @ PSQT 2016 discusses:
• What the shift from automated to
continuous means
• How disruption requires changes to how
we test software
• Addressing gaps between Dev and Ops
• Technologies that enable Continuous

Published in: Software
  • D0WNL0AD FULL ▶ ▶ ▶ ▶ http://1url.pw/Bib6o ◀ ◀ ◀ ◀
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

No Devops Without Continuous Testing

  1. 1. No DevOps Without Continuous Testing Arthur Hicken - Parasoft 1
  2. 2. Your Presenter Arthur Hicken is Chief Evangelist at Parasoft where he has been involved in automating various software development and testing practices for over 20 years. He has worked on projects including cybersecurity, database development, the software development lifecycle, web publishing and monitoring, and integration with legacy systems and maintains the IoT Hall-of-Shame http://bit.ly/iotshame and SQLi Hall-of-Shame http://bit.ly/sqlishame Follow him @codecurmudgeon Blog: http://codecurmudgeon.com Web: http://parasoft.com
  3. 3. Agenda • What the shift from automated to continuous means • How disruption requires changes to how we test software • Addressing gaps between Dev and Ops • Technologies that enable Continuous www.iist.org 3 www.psqtconference.com
  4. 4. Continuous 4 www.psqtconference.com • Build, Integration, Testing, Release, Delivery, Deployment • Builds on DevOps movement • Enables DevOps reality • High dependency on automation: – processes – assessment – decisions
  5. 5. DevOps “DevOps is the practice of operations and development engineers participating together in the entire service lifecycle, from design through the development process to production support.” – The Agile Admin • No more throwing code over the wall • Design for deployment 5 www.psqtconference.com
  6. 6. DevOps Pipeline 6 www.psqtconference.com
  7. 7. Automated vs Continuous • Just being automated isn’t continuous • Scheduling / frequency • Intermittent manual dependencies • Decisions points – Is it done? – Is it good? – Is it safe? – Can I push it out the door? 7 www.psqtconference.com
  8. 8. Complexity a Barrier to Testing 3rd Party System Evolving Component Mainframe Scheduled Access
  9. 9. API Economy Pieces/Parts come from everywhere Definitions changing (upgrades, new versions) Not always available Can’t test with real data without causing problems 9 www.psqtconference.com
  10. 10. We must Re-invent Test • The penalty for faulty software is increasing – Brand erosion – All industries at risk for total transformation • Technology is faster and more distributed – Containers – Microservices
  11. 11. Fixed Fixed Impact of Test in the Evolving SDLC Plan Develop Build Test Release Test as a time-boxed event is the prevailing method “Are we done testing?”
  12. 12. Impact of Test in Evolving SDLC Plan Develop Build Test Release Build Test Build Test Build Test Build Test Build Test Build Test Build Test Build Test Build Test Build Test “Does the release candidate have an acceptable level of risk?”
  13. 13. Gaps between Dev and Ops What is “done”? Fully testing with external dependencies Full environments and all their variants 13 www.psqtconference.com
  14. 14. Speed of Deployments Rapid deployments Deliver features and bug fixes Deployments outpace ability to test Too many changes Since we didn’t test enough, Devs are mired in bug fixes Glitches and outages can happen 14 www.psqtconference.com
  15. 15. QA and Testing • Cannot keep up with the rate of change • Quality lags behind • Do we even know what to test? • Complex technologies including message protocols and services 15 www.psqtconference.com
  16. 16. Enabling Technologies • Stubs • Service Virtualization • API testing • Test data management • Environment management • Self-service test environments 16 www.psqtconference.com
  17. 17. How to do it better ? • Give absolute priority to automated tests • Invest time into designing interfaces (API) • Use API (service) testing tools to cover interfaces • Measure the quality of the test
  18. 18. How to do it better ? • Use stubs and mocks to improve: – Facilitate test automation – Increase code penetration
  19. 19. How to do it better ? • Use service virtualization to improve testing automation – Isolate at the message layer – Simulate functional scenarios and performance conditions
  20. 20. How to do it better? Test management •Environments •Data •Dependencies 20 www.psqtconference.com
  21. 21. • The Challenge – Performance issues found late in the SDLC resulting in large rework and cost – Middleware component able to handle large load but available downstream dependent systems were unable to provide throughput needed Reduced testing time by 27-days for a major commercial bank Capacity Constrained staged environments Database Service Mainframe Application 1000 tps
  22. 22. Database Servic Mainfram e Applicatio n Reduced testing time by 27-days for a major commercial bank 1000 tps Virtual Asset Virtual Asset Virtual Asset Virtual Asset • The Solution – Identified key use-cases and emulate the behavior of dependent systems. Asset performance setup to match production response values – API and Load testing tools used to validate system under test. Virtual assets changed to model different scenarios • The Business Benefit – Able to predict performances issues and fix before full Perf testing. – Able to run different scenarios not easily achieved with production environment (excessive response times). – Setup for ‘performance test’ reduced to near zero
  23. 23. – The Challenge • Multiple teams using the same test database • Teams not respecting data integrity & others test data records • Regression tests consistently failing. Takes >1 hour to determine that it was due to “data changes”. • “Real problems” were getting lost in the noise Shared Databas e Eliminated 83% of the configuration time for a major telecom company Test data management for complex transactions ✔ ✖ ✖ Shared Database
  24. 24. – The Solution • Setup Virtual Assets to model the SQL queries and use API testing tool to manage automated nightly regressions against both virtual assets and live systems – The Business Benefit • Test teams able to focus on ‘real regressions’ and separate out data integrity issues from functional test failure Eliminated 83% of the configuration time for a major telecom company Virtual Asset Virtual Asset Virtual Asset ✔ ✔ ✔
  25. 25. Reduced wait time for test team by 60% for a major media conglomerate • The Challenge – Large agile development effort to adopt Service Oriented Architecture (SOA) – High risk project but the Test team “stuck waiting for the first build” – Development of functionality was not easy to coordinate as different teams had different schedules; not all finished at the same time Agile/Parallel development limited by system dependencies Iteration Iteration Iteration Team A Team B Team C Iteration Iteration Iteration Iteration Iteration Iteration Iteration Current Development/Testing dependencies
  26. 26. Reduced wait time for test team by 60% for a major media conglomerate • The Solution – Use descriptions of the new services (WSDL, XSD, example JSON payloads) to build Virtual assets prototyping the new functionality. – Test team builds tests with against the prototypes with API testing tool and the independent development tests use the prototypes to perform early stage Integration Testing • The Business Benefits – Met business goals and timelines, were able to test functionality “as soon as” it was available. Practiced TDD against prototypes to get a head-start on ‘full system testing’ Iteration Iteration Iteration Team A Team B Team C Iteration Iteration Iteration Iteration Iteration Iteration Iteration ✔✖✔✖✔✖
  27. 27. Extended testing hours for a stock trading broker • The Challenge – Leverages 3rd party (out sourced) system for trade exchanges – Many teams need access, so time is scheduled into 4 hour time windows, and could only test during market hours Capacity Constrained staged environments Database Servic e Mainfram e Applicatio n
  28. 28. Extended testing hours for a stock trading broker • The Solution – Use Test environment manager tool to setup ‘Test Silos’ where each team can do 24x7 testing against virtual assets Database Mainfram e Applicatio nEnvironment Manager
  29. 29. Conclusions Emulates dependencies for the Test Environment • Reduces complexity for early stage testing • Increases predictability On-Demand “Test Anytime, Anywhere” • Automated Provisioning for different use-cases • Automated Test Data Management/Simulation Reduce time to configure for test Realistic reusable assets Reduce hardware costs and system access fees
  30. 30. Q&A • Web – http://www.parasoft.com • Blog – http://alm.parasoft.com – http://codecurmudgeon.com •Social –Facebook: https://www.facebook.com/parasoftcorporation –Twitter: @Parasoft @CodeCurmudgeon –LinkedIn: http://www.linkedin.com/company/parasoft

×