В докладе мы поговорим о концепции Feature Flags. Преимуществами этого подхода являются - Уменьшение количества веток (feature branches) - Постепенная выкатка изменений (gradual rollout) - Безопасный откат изменений (safe rollback) В докладе я расскажу о практическом опыте компании upwork.com, и о том как Feature Flags минимизируют наши риски при выпуске новой функциональности или рефакторинге. Введение в концепцию Feature Flags Feature Flags в Upwork Какие задачи решаются с помощью Feature Flags Как мы используем Feature Flags Детали реализации Общие рекомендации по использованию Ограничения подхода Доклад расчитан на широкий спектр аудитории

1. Feature Flags. Reducing risks
during shipping changes
Aleksandr Makhomet
Upwork
http://twitter.com/amahomet
https://www.facebook.com/alexander.mahomet

2. Feature Flags concept.

3. Feature Flags Concept
● The basic idea is to have a configuration that defines a bunch
of toggles for various features you have pending. The running
application then uses these flags in order to decide whether or
not to show the new feature. © Martin Fowler
● Other names: feature toggles, feature bits, flags, flippers,
conditional features
{% if feature_is_enabled('AwesomeRedButtons') %}
<p>Awesome Feature is enabled</p>
{% else %}
<p>Old boring blue buttons</p>
{% endif %}

4. Feature Flags ideas
● Gradual rollout
● Safe rollback
● Less feature branches

5. Gradual rollout
● Dark launching
● Easy to enable for a specific number of
users, even for yourself only

6. Gradual rollout
● Easy to revert if metrics are not good
● Automatic disabling in case of spikes

7. Less Feature Branches
● Less merge/integration hell
● Trunk based development
● Easy to mix branches
● Essential to Continuous Integration
approach

8. Who does use Feature Flags
● Google
● Flickr
● Asana
● Travis
● Rally
● More and more

9. Feature Flags in Upwork

10. What is Upwork.com
• Formerly odesk.com
• Upwork has nine million registered freelancers and
four million registered clients. Three million jobs
are posted annually, worth a total of $1 billion
USD, making it the world's largest freelancer
marketplace.
• Highload. Microservice architecture

11. Feature Flags. Our choise.
● Gradual rollout - Yes
● Graceful degradation - Yes
● Replace feature branches - Not yet

12. Implementation. Prerequisites
● As simple as possible
● Platform agnostic

13. Configuration file
● INI format
● Each feature has code name
● Naming convention
● GIT is used
● Each change should be reviewed
● Validation script
ShowMickeyMouseOnHomePage.enabled = false

14. Meta data
● Title
● Description
● Owner
● Responsible Team
● Reference Url
● Permanent

15. Constraints
● Internal users
● Percent of users (1%, 5%, 30%, 50%, 100%)
○ Random within 0 - 100% but constant
per user
● Percent of companies
Create any constraints useful for for your
project

16. How do we update flags
● False for start
● Gradual update using Etsy Deployinator
○ Staging -> Production
(including validation check)
● App servers update
● Placing configuration file to s3 storage

17. How do we use it in PHP
● Take from S3 Cache to APC and create a backup local
version.
● Service and Twig function
if ($this->container->get('feature')->isEnabled('Feature')) {
// do something ...
}
{% if feature_is_enabled('Feature') %}
<p>Feature is enabled</p>
{% endif %}
● For more information watch and read Benjamin Eberlei -
presentation “Feature Flags with Symfony”

18. Feature flags Audit
● Important to have visibility on changes
● Git history
● Write logs and send notification to
chat/email/etc when you deploy

19. Cleanup
● Cleanup as soon as feature is stable
● Keep number of conditions invasions
small
● Create follow-up ticket to not forget to
remove
● You can try to make cleanup fun :)

20. Testing
● Feature Flags could complicate testing
● Unit tests
○ Just mock dependencies
● Integration/Functional tests
We test combinations which should reasonably be expected to
happen in production

21. Feature Flags admin UI
● Important to have visibility on feature
flags statuses
● Such UI should provide readable list of
feature flags with possibility to search/
filter/sort etc
● We use it for enabling feature for current
session

22. FF lifecycle
● Add feature flag to config with disabled status
● Write code. Test your code in branch with
enabled feature flag.
● Release code
● Enable feature for your own user. Check it
● Enable for internal users. Check it
● Gradual enabling for group of users. Monitoring
● Enable feature for 100% of users. Monitoring
● If everything is fine - cleanup feature flag if it is
not permanent.

23. Feature Flags and A/B testing
● A/B testing is a kind of randomized
experiment with two variants, A and B.
● Feature flags approach is actually an A/B
testing
● A/B testing system could be much more
powerful
● We separated these systems

24. Side effects and
recommendations

25. Feature Flags problems
● Increase complexity and decrease the
readability of code
● Accidental exposure
● Increases complexity of testing

26. Feature Flags. Recommendations. Part 1
● Keep FF implementation simple and
modularized
● Use feature flags at minimum amount of
points, avoid very invasive cases
● Avoid mixing feature flags
● Do monitoring
● Be careful with partial enabling many
features at one time

27. Feature Flags. Recommendations. Part 2
● Reduce number of permanent feature flags
● Do cleanup as soon as possible
● Don’t leave code in the mainline just in case
you might need it again some day.
● Changes involving data migrations must be
handled with care
● Create clear guidelines, describing your
approach, including section like “when to use
toggles or not”

28. Conclusions
● Feature flags are very handy to roll out new
features gradually, but they also give you higher
control over your system running in production as
well.
● On top of that, they give you a means to disable
certain features during a production incident.
● At the same time this approach brings new
challenges. So decision is up to you, as always :)

29. More cool reading
General cool tech reading
● http://code.flickr.net/
● https://codeascraft.com (Etsy, Rasmus Lerdorf)
● https://www.rallydev.com/blog/engineering
● https://blog.asana.com/eng/
Feature flags related reading
● http://code.flickr.net/2009/12/02/flipping-out/
● http://martinfowler.com/bliki/FeatureToggle.html
● https://www.youtube.com/watch?v=KJKSCbYrNvY
(Benjamin Eberlei - Feature Flags with Symfony)
● http://blog.travis-ci.com/2014-03-04-use-feature-flags-to-ship-changes-with-confidence/
● https://blog.asana.com/2011/04/using-flags-to-ease-new-feature-development/
● https://www.rallydev.com/blog/authors/ryan-scott?page=1 (4 articles)
● http://swreflections.blogspot.com/2014/08/feature-toggles-are-one-of-worst-kinds.html

30. Le Fin

31. Questions?
Aleksandr Makhomet
http://twitter.com/amahomet
https://www.facebook.com/alexander.
mahomet