Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Testistanbul 2016 - Keynote: "Why Automated Verification Matters" by Kristian Karl


Published on

Automated verification is becoming increasingly important. Getting a product from idea to customer as fast as possible in a Continuous Delivery, or a Deployment pipeline is crucial in more businesses than ever before. But how do we get a product through that pipe line, with high quality? Kristian will talk about how automated verification can get you there.

Published in: Technology
  • Login to see the comments

  • Be the first to like this

Testistanbul 2016 - Keynote: "Why Automated Verification Matters" by Kristian Karl

  1. 1. Why Automated Verification Matters Kristian Karl, Spotify @kristiankarl
  2. 2. 1983 -Started programming 1994 -Testing and automation 1998 -Consultant 2004 -Discovered Model-Based Testing 2010 -Test Manager at Spotify
  3. 3. Technology Product Design
  4. 4. Confidence Why does automated verification matter?
  5. 5. ● What can we check? ● What is important to check? ● How can we check it? ● How to involve everyone? ● How do we get ROI?
  6. 6. ● Dev environments ● Builds on pull requests and merge ● Static code analysis ● Code style checking ● Dynamic code analysis ● Verification ● Archiving artefacts ● Deployment What can be automated?
  7. 7. Business Logic Coverage Execution Time / Costs Code Coverage Test Type UI Integration Unit Test High Medium Low Low Medium High Long / High Medium Short / Low
  8. 8. A human using her intelligence and experience to critique and explore a system under test. Testing
  9. 9. Agile Testing
  10. 10. Check for the expected Find the unexpected Business Technology ● Functional testing ● BDD ● Specification by example ● Unit tests ● Component tests ● Exploratory testing ● Scenarios ● Usability testing ● Performance ● Security
  11. 11. Our approach at Spotify Disclaimer: There are 100+ teams at Spotify, and they all do things a bit different. Some teams do it very differently...
  12. 12. Code change Pre-merge build and tests Code and quality review Risk analysis Exploratory testing Continuous Delivery
  13. 13. Pre-merge ● Quick feedback loop ● The total time for the build needs to be time boxed ● Style checking ● Static code analysis ● Unit tests ● Stable functional tests ● Stable integration tests ● Packaging
  14. 14. Post-merge ● All the steps from the pre-review build ● Longer running functional and integration tests ● System tests ● Non-functional tests, like: - resource leaks - performance
  15. 15. What has helped Spotify doing automated verification?
  16. 16. The Test Data Service
  17. 17. The Test The Test Data Service Some Production Service Request: ● Premium user ● Not connected to Facebook ● Using feature flag X Query for user(s) matching The criterias List a matching user(s) Return selected user Pick one user, and lock it, Prevent any other test from using that user. Time
  18. 18. Model-based Testing à la GraphWalker
  19. 19. App not running Authentication form displayed Home page Start app Close Exit Valid credentials Bad credentials Logout
  20. 20. Our Test Interface(s)
  21. 21. Test automator role Developer role Defines a testing API Implements the testing API
  22. 22. Test Result Service
  23. 23. One test result index to rule them all - Many ways of visualizing
  24. 24. So what’s the lessons learned so far?
  25. 25. Backend teams ● Backend teams work with smallish micro services. ● High degree of automation on all parts of the pipeline. ● No one but the team touches the team’s code. ● They have operational responsibility. ● Few defects found after definition of done.
  26. 26. Frontend [mobile & desktop] teams ● Building monoliths ● 200+ people involved ● Low degree of automated verification ● Low testability ● High number of defects are found after definition of done. That hurts productivity...
  27. 27. Tragedy of the commons Individuals acting independently and rationally according to each other's self-interest, behave contrary to the best interests of the whole group
  28. 28. What can you do to improve? ● Ways of working are more important than tools and frameworks. ● Zero bug policy, instead of huge backlog of bugs. ● Pairing and embed ● Everyone writes and maintain automated verification. ● Testability will require non-functionality. ● Failed verifications must be fixed ASAP.
  29. 29. Automated verification matters, because it can ● Increase your productivity ● Give you confidence to deliver anytime ● Enable you work with defect prevention, rather than defect detection
  30. 30. References ● ● ● ● automation-pyramid ● ● ● ●