Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Test Design for Fully Automated Build Architectures

352 views

Published on

Imagine this … As soon as any developed functionality is submitted into the code repository, it is automatically subjected to the appropriate battery of tests and then released straight into production. Well, setting up the pipeline capable of doing just that is becoming more and more common. But most organizations hit the same stumbling block—just what IS the appropriate battery of tests? Automated build architectures don't always lend themselves well to the traditional stages of testing. In this hands-on tutorial, Melissa Benua introduces you to key test design principles—applicable to organizations both large and small—that allow them to take full advantage of the pipeline's capabilities without introducing unnecessary bottlenecks. Learn how to make highly reliable tests that run fast and preserve just enough information to let testers and developers determine exactly what went wrong and how to reproduce the error locally. Explore ways to reduce overlap while still maintaining adequate test coverage. Take back ideas about which test areas could benefit from being combined into a single suite and which areas could benefit most from being broken out altogether.

https://starwest.techwell.com/program/tutorials/test-design-fully-automated-build-architectures-starwest-2017

Published in: Technology
  • D0WNL0AD FULL ▶ ▶ ▶ ▶ http://1lite.top/VQSEpj ◀ ◀ ◀ ◀
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Test Design for Fully Automated Build Architectures

  1. 1. 1 Test Design for Fully Automated Build Architectures Melissa Benua Senior Technical Lead, mParticle mbenua@gmail.com @queenofcode STARWEST 2017
  2. 2. 2 About Me Boeing Microsoft PlayFab mParticle
  3. 3. 3 About The Tutorial Source Control Track + Code Review Build + Test Deploy + Monitor
  4. 4. 4 Guiding Principles What are we doing here?
  5. 5. 5 G U I D I N G P R I N C I P L E S Key Test Features Important Reliable Specific
  6. 6. 6 G U I D I N G P R I N C I P L E S Importance Triage scenarios based on priority IMPOSSIBLE to cover every scenario Understand what failures can be tolerated Run the most important tests first
  7. 7. 7 G U I D I N G P R I N C I P L E S Importance Build Failure? UI Test Failure?
  8. 8. 8 G U I D I N G P R I N C I P L E S Reliability No flakiness No false-negatives or false-positives Repeatable without human intervention Cleans up after itself
  9. 9. 9 G U I D I N G P R I N C I P L E S Reliability
  10. 10. 10 G U I D I N G P R I N C I P L E S Specificity Clear answer to a clear question Have one main goal Don’t ‘boil the ocean’ Minimal overlapping coverage
  11. 11. 11 G U I D I N G P R I N C I P L E S Specificity
  12. 12. 12 G U I D I N G P R I N C I P L E S Proceed With Caution Shared resource reliance 1 Single- threading 2 Long duration test time 3 Caching 4
  13. 13. 13 G U I D I N G P R I N C I P L E S Exercise: Test Cases Photo Gallery Site UX FrontEnd (HTML5 + JS) Logic BackEnd (Java) Data Store (NoSQL) • Most important? • Easiest? • Most reliable?
  14. 14. 14 Automated Pipeline Structure What does it even look like?
  15. 15. 15 AU T O M AT E D P I P E L I N E S T R U C T U R E CI + CD Pipeline Source Control Track + Code Review Build + Test Deploy + Monitor
  16. 16. 16 AU T O M AT E D P I P E L I N E S T R U C T U R E Beyond Unit Tests Develop: Diff Build • Compile change against mainline • Execute unit tests Build: Continuous Integration • Compile change as a part of mainline submit • Execute functional tests Deploy: Continuous Deployment • Start staging environment • Deploy staging environment • Execute UI + load tests • Project structure? • Moving parts? • Functional boundaries? • Shared resources? • Mocking potential?
  17. 17. 17 AU T O M AT E D P I P E L I N E S T R U C T U R E Example Service Architecture UI Layer RESTful API Admin UI Web Frontend Backend Layer Auth Service Logic Service Cache Service Data Layer Database File Storage
  18. 18. 18
  19. 19. 19 AU T O M AT E D P I P E L I N E S T R U C T U R E The Matrix Component • Compile • Unit Package • Functional • Integration Release • Acceptance • UX • Load Product BE Machine Function Auth Logic FE Machine Function Web UI Cache
  20. 20. 20 AU T O M AT E D End to End
  21. 21. 21 AU T O M AT E D P I P E L I N E S T R U C T U R E Exercise: Mapping Categories to Stages Photo Gallery Site UX FrontEnd (HTML5 + JS) Logic BackEnd (Java) Data Store (NoSQL) • What should we run? • When should we run it? • How long should we wait?
  22. 22. 22 AU T O M AT E D P I P E L I N E S T R U C T U R E Key Takeaways Easy to conditionally run different categories 1 Find the balance between too many stages and too few 2 Machine time is MUCH cheaper than human time 3 Fail fast and fail often 4
  23. 23. 23 Monitoring and Reporting What just happened?
  24. 24. 24 M O N I T O R I N G AN D R E P O R T I N G Code Coverage 0 10 20 30 40 50 60 70 80 90 100 0 25 50 75 100 %ofBugsFound Code Coverage % Bugs vs Effort % of Bugs Found • Line vs branch coverage • Reports and readability • Data-driven decisions
  25. 25. 25 M O N I T O R I N G AN D R E P O R T I N G Flaky Test Handling Cost vs Value Net gain? Test Failure Fatigue Ignoring failures? Ease of Detection How do we know?
  26. 26. 26 M O N I T O R I N G AN D R E P O R T I N G Logging vs Counters Text Log File? Graph?
  27. 27. 27 M O N I T O R I N G AN D R E P O R T I N G Exercise: What Goes Where? Photo Gallery Site UX FrontEnd (HTML5 + JS) Logic BackEnd (Java) Data Store (NoSQL)
  28. 28. 28 P U T T I N G I T T O G E T H E R Overall Summary Follow natural system boundaries 1 Know where your cutline is and respect your time 2 Don’t try to boil the ocean 3 Use your data wisely 4
  29. 29. 29 Thank you! Melissa Benua mbenua@gmail.com @queenofcode http://www.queenofcode.net

×