Talk at Iqnite conference for software quality and testing about how we have do continuous delivery at here.com
Together with Siddharth Somasundaram
http://www.iqnite-conferences.com/suisse/programme/programme.aspx
2. HERE
● Formerly known as Nokia / NAVTEQ Maps
● Powers Bing, Facebook and Yahoo Maps
● 4 out of every 5 cars with in-dash navigation are
powered by HERE
4. Testing Before Agile
● Testers are only involved in the later stage
● Long regression testing cycle before release
● Developers and Testers have separate work streams
○ Leads to conflicts between them
● Business pressure to release the software
○ Compromise on quality?
5. Importance of Agile
● Requirements change over time
● Testers are part of the planning process
● The entire team is responsible for Quality
● Faster release cycle
○ From once every 3 months to daily releases
6. Continuous Delivery
● Release early and often to production
● Automate testing and deployment
● Faster feedback, reduce risks and overhead for releasing
● Hot trend in the industry, especially web companies
7. But what about Quality Assurance?
● Lots of automation for regression testing
● Build enough trust in the test coverage
● Focus shifts from prevention of bugs to rapid discovery
and resolution
● Exploratory testing can be decoupled from releasing
8. How do we do it?
● Team Culture and Technical practices we found to be useful
9. Team Culture Practices
● One agile Team, One Product and One Goal
○ Product Owner
○ Scrum Master
○ Developers / Testers
○ Offshore teams
● Communication solves most of the problems
10. Collaborate using Work Agreements
The team formulates rules and hold each other
accountable. For example:
● Running tests before committing
● No commits on red build
● First priority is to keep the build green
20. Eliminate Flaky Tests
● Essential to keep the team
trusting the test results
● Mocking dependencies
21. Feature Flags
● No branching
○ Prevent integration problems
○ Quality built in
● Turn features on in production
when they are ready
● Quality boundaries for
different release stages
22. Logging & Monitoring
● What happens after we go live?
● Production monitoring ensures edge cases are covered
● Real-time metrics for: system, application, business
23. Logging & Monitoring
● Quickly responding to alerts
● Production is the ultimate proof the code works
Bug introduced We fixed it :-)
25. Is it Successful? The Metrics
2010 Now
Release quarterly Release daily
Mostly manual testing 7000+ unit tests, 400+
GUI tests
Preparing release takes
weeks
Release package ready
in 45 minutes
Deployment takes days Deploy 3 sites in 10
minutes
Management signoff
required
Team empowered to
make changes
27. Business Benefits
● Reduce risk involved in big changes
● Reduce time to market
● Smaller more frequent feature rollouts
● Quickly learn from feedback on product changes
28. Business Benefits
● Predictable releases
● Improved team morale
○ Less time spend in stressful firefighting
○ Less bug fixing, more feature delivery
○ Satisfaction of delivering to the customer
29. Challenges
● Everybody agreeing on the process
● Discipline comes over time only
● Feedback needs to be given quickly
● High investment in test automation and keeping it
running smoothly
● Need to react to incidents quickly
30. Future Plans
● Deploy every build to production
● Keep on investing in Automation and Monitoring
● Canary deployments to validate new releases before
global rollout
● Leverage quick cycle time for A/B testing driven feature
development
31. HERE Checklist
Team Culture
● One Team
● Work Agreements
● Build Monitor in the
room
● Pair Programming
● Tech Talks
● DevOps
Technical
● Continuous Integration
● Test Driven Development
● Build Pipeline
● Test Coverage on all
levels
● Feature Flags
● Logging & Monitoring