Testing in a continuous delivery environment

617 views

Published on

Talk at Iqnite conference for software quality and testing about how we have do continuous delivery at here.com

Together with Siddharth Somasundaram

http://www.iqnite-conferences.com/suisse/programme/programme.aspx

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
617
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
11
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Testing in a continuous delivery environment

  1. 1. Testing in a Continuous Delivery Environment Siddharth Somasundaram Stefan Verhoeff
  2. 2. HERE ● Formerly known as Nokia / NAVTEQ Maps ● Powers Bing, Facebook and Yahoo Maps ● 4 out of every 5 cars with in-dash navigation are powered by HERE
  3. 3. HERE ● Maps on the Web offering here.com
  4. 4. Testing Before Agile ● Testers are only involved in the later stage ● Long regression testing cycle before release ● Developers and Testers have separate work streams ○ Leads to conflicts between them ● Business pressure to release the software ○ Compromise on quality?
  5. 5. Importance of Agile ● Requirements change over time ● Testers are part of the planning process ● The entire team is responsible for Quality ● Faster release cycle ○ From once every 3 months to daily releases
  6. 6. Continuous Delivery ● Release early and often to production ● Automate testing and deployment ● Faster feedback, reduce risks and overhead for releasing ● Hot trend in the industry, especially web companies
  7. 7. But what about Quality Assurance? ● Lots of automation for regression testing ● Build enough trust in the test coverage ● Focus shifts from prevention of bugs to rapid discovery and resolution ● Exploratory testing can be decoupled from releasing
  8. 8. How do we do it? ● Team Culture and Technical practices we found to be useful
  9. 9. Team Culture Practices ● One agile Team, One Product and One Goal ○ Product Owner ○ Scrum Master ○ Developers / Testers ○ Offshore teams ● Communication solves most of the problems
  10. 10. Collaborate using Work Agreements The team formulates rules and hold each other accountable. For example: ● Running tests before committing ● No commits on red build ● First priority is to keep the build green
  11. 11. Build Monitor visible in every room
  12. 12. Pair Programming
  13. 13. Share knowledge with Tech Talks
  14. 14. Combine developer and operation teams
  15. 15. Technical Practices
  16. 16. Continuous Integration
  17. 17. Test Driven Development ● Code and tests are developed at the same time
  18. 18. Build Pipeline ●
  19. 19. Test Coverage at all levels © Alister Scott http://watirmelon.com/2011/06/10/yet-another-software-testing-pyramid/
  20. 20. Eliminate Flaky Tests ● Essential to keep the team trusting the test results ● Mocking dependencies
  21. 21. Feature Flags ● No branching ○ Prevent integration problems ○ Quality built in ● Turn features on in production when they are ready ● Quality boundaries for different release stages
  22. 22. Logging & Monitoring ● What happens after we go live? ● Production monitoring ensures edge cases are covered ● Real-time metrics for: system, application, business
  23. 23. Logging & Monitoring ● Quickly responding to alerts ● Production is the ultimate proof the code works Bug introduced We fixed it :-)
  24. 24. In Conclusion...
  25. 25. Is it Successful? The Metrics 2010 Now Release quarterly Release daily Mostly manual testing 7000+ unit tests, 400+ GUI tests Preparing release takes weeks Release package ready in 45 minutes Deployment takes days Deploy 3 sites in 10 minutes Management signoff required Team empowered to make changes
  26. 26. Deployments 2012 vs 2013
  27. 27. Business Benefits ● Reduce risk involved in big changes ● Reduce time to market ● Smaller more frequent feature rollouts ● Quickly learn from feedback on product changes
  28. 28. Business Benefits ● Predictable releases ● Improved team morale ○ Less time spend in stressful firefighting ○ Less bug fixing, more feature delivery ○ Satisfaction of delivering to the customer
  29. 29. Challenges ● Everybody agreeing on the process ● Discipline comes over time only ● Feedback needs to be given quickly ● High investment in test automation and keeping it running smoothly ● Need to react to incidents quickly
  30. 30. Future Plans ● Deploy every build to production ● Keep on investing in Automation and Monitoring ● Canary deployments to validate new releases before global rollout ● Leverage quick cycle time for A/B testing driven feature development
  31. 31. HERE Checklist Team Culture ● One Team ● Work Agreements ● Build Monitor in the room ● Pair Programming ● Tech Talks ● DevOps Technical ● Continuous Integration ● Test Driven Development ● Build Pipeline ● Test Coverage on all levels ● Feature Flags ● Logging & Monitoring
  32. 32. Merci Beaucoup Thank You Danke Schön Grazie

×