This document discusses how wego.here.com achieves continuous deployment with confidence through development rules, code reviews, feature flags, progressive rollouts, extensive testing at all levels, A/B testing, performance regression testing, blue-green deployments, analytics and monitoring, and a continuous deployment pipeline. Some key metrics include over 2000 commits and over 400 PR deployments in 2016, with deployments taking approximately 47 minutes from commit to production. Continuous deployment allows for lower risk releases, faster delivery of value to users, higher quality through automated testing and easy rollbacks, and a hypothesis-driven approach to product development through A/B testing.

1. How to continuously deploy with
confidence. Success story of
wego.here.com
Vadim Ivanov
Event Title | August 31, 2016

2. Source Control Server
… …
Year Commits Lines
added
Lines
removed
2016 2116 733408 326301
2015 4477 253713 254151

3. Agile
You're only as #agile as your ability to ship frequently, and without drama.

4. PR deployments
589 in 2015
721 in 2016

5. How do we do it
• Development rules + code review
• Feature flags + progressive rollouts
• Tests: FE UT, BE UT, Integration tests, UI tests, Smoke tests
• A/B tests
• KPI gate: performance regression tests
• 0 downtime blue green deployments
• Analytics + monitoring
• Continuous deployment

6. Development rules + code review
• New feature only under Feature flag
• Every new feature has a rollout schedule starting from 1%
• Feature can not be deployed without tests written
• Code style + pre-commit git hooks
• No code review for bug fixes without UT

7. Tests + performance regression
• FE UT: 3552, 87% code coverage
• BE UT: 690
• Integration tests: 195
• UI tests: 495 Scenarios
• Smoke tests: a subset of UI tests running against real API on dev, staging, pr
envs
• Web Page Performance: Load time, fully loaded, start render, speed index

8. A/B tests
• Currently running 15 A/B tests
• NPS scoring to evaluate test success

9. Deployment
• Zero downtime blue green deployment
• Continuous Deployment:
~47 mins from commit to production

10. Build pipeline (Jenkins Pipeline)

12. Analytics + monitoring
• Keep an eye after deployment
• Real-time metrics for: system, application, business
• Quality alerts
Bug introduced Bug fixed

13. Conclusion
• Stimulates a culture of quality, the mindset to always keep the product in
releasable state
• Low risk releases: small change -> small risk; zero-downtime deployments
that are undetectable to users
• Faster time to market: Value delivered immediately to the user when it's
ready; Quick feedback on product changes
• Higher quality: automated tools to discover a regression; easy to rollback;
easy to identify bugs
• Better products: A/B testing enable us to take a hypothesis-driven approach
to product development
• Happier teams

14. Thank you
Contact
Vadim Ivanov
vadim.ivanov@here.com
HERE