Over time, the software industry has come up with many ways to deliver code. Why is it so important to be in production as much as possible? What advantages and disadvantages do we have in rapid releases? Let’s talk about how to be faster, safer, and with better quality.

1. Release
Early, Often, Stable
Alexey Kupriyanenko, R&D Manager
WIX

2. RERO
Background
Release early. Release often.
And listen to your customers
Eliminate the risk of creating software that
no one will use
1997. Eric S. Raymond

3. Time-based
▪ Scope of release is defined in
advance
▪ Strict schedule for releases: every
week/month/year
▪ Predictability for contributors and
users
Background

4. source
Works really well with open source projects

5. Feature-based
▪ One release - one feature
▪ A developer is responsible for deploy
process
Background

6. In most cases, it’s the fastest feedback
loop between you and customers

7. I like the
Web
Background

8. What is often?
= amount of developers / 2
per day
but it can be more
Background

9. I know what I’m talking about
We have daily releases in production~500

10. It can be risky
Be prepared for the
production bugs
We all know that they are exists,
you must be honest with yourself
Risks

11. Therac-25
The Worst Computer Bugs in History:
Race conditions in Therac-25
These computer bugs left a significant
impact on the people who experienced
them
motivationRisks

12. Boeing 737
Boeing 737 cockpit screens go blank if
lands on specific runways:
with a 270-degree true heading
source
motivationRisks

13. Let’s talk about motivation

14. Eliminate the risk
of creating soft that no one is using
Motivation

15. Reduce code review complexity
Motivation

16. Higher quality software
Motivation

17. Reduce development friction
Motivation

18. Let’s talk about methodology

19. Continuous
Integration
Strategy
If it hurts, do it more
You could avoid merge conflicts by
integrating code more frequently

20. Continuous
Delivery
Strategy
Every time you commit some code
the commit makes its way to a server

21. Feature flags
Strategy
Putting an if condition around a new
feature, and having it related to something
external to the code
if (isFeatureEnabled) {
doSomething();
}

22. Separation of
delivery and release
Strategy
▪ Developing the feature bothers no one
▪ QA and other developers get the feature
as needed
▪ Deployment is low risk and uneventful
▪ Releasing a feature is a business
decision

23. Deployment is low risk
Strategy

24. QA and other developers get the feature as
needed
Strategy

25. Releasing a feature is a business decision
Strategy

26. Let’s talk about ingredients

27. Tests
Ingredients
You have to be sure that you didn't
break your or colleague's code

28. QA approach
Ingredients
It’s impossible to run tons of manual
regressions for your product
You have to find a way to have a
minimal amount of manual QA

29. Monitoring
Ingredients
It’s almost impossible to cover 100% of
all possible cases
You have to have monitoring of
production system

30. Version control
system
Ingredients
Get rid of all unnecessary steps and
branches in your git methodology
Challenge it as much as possible and
question every step

31. How it works in our projects

32. Microservices
Experience
One application can be deployed
granularly

33. Development
sprints
Experience
You don’t need to build an exceptional
process, work by scrum/kanban or
something that fits your needs

34. Experience
git
Most simplistic GitHub flow

35. Experience
CI

36. CI - under the hood
Experience

37. Apache Kafka
Open-source stream-processing
software. Platform for handling real-time
data feeds
Experience
https://www.wix.engineering/post/making-order-in-ci-cd-mess

38. git local
Experience
git remote
PUSHCommit
Repo
Update
Post ReceivePre ReceivePost CommitPost Commit

39. Git Event
Listener
Produce Kafka messages for every
commit push to any of Wix repositories,
this message is consumed by any
service (not only Lifecycle services that
what to be notified of such events).
Experience
https://www.wix.engineering/post/making-order-in-ci-cd-mess

40. Build Server - TeamCity
Experience

41. Build Server
Experience
We extended TeamCity capabilities and
added a plugin that produces a Kafka
event whenever a build starts /
completes / fails.
https://www.wix.engineering/post/making-order-in-ci-cd-mess

42. Repo
Descriptor
Service
Experience
Since we use several build tools and
technologies, we created a general
entity called “Artifact”
https://www.wix.engineering/post/making-order-in-ci-cd-mess

43. Build Descriptor
Service
Experience
Consumes TeamCity messages and
triggers a complete “build refresh”
process.
Once the task is completed, a message
is delivered to all consumers
https://www.wix.engineering/post/making-order-in-ci-cd-mess

44. Artifactory
Experience
Wix binary repository with a snapshot of
a successful build
https://www.wix.engineering/post/making-order-in-ci-cd-mess

45. RC service
(release candidate)
Experience
Manages the RC process, validates the
build can be RCed (no pending changes
to build in CI TC and CI build is green)
and monitors the progress of the RC
process
https://www.wix.engineering/post/making-order-in-ci-cd-mess

46. GA service
(global availability)
Experience
Manages the GA/Testbed/Rollback
processes, stores the current and past
GA versions of each build. It is also
responsible for the complete process
https://www.wix.engineering/post/making-order-in-ci-cd-mess

47. Production
Interface
Service
Experience
Responsible for reflecting the status of
an artifact in production.
It monitors the deployment progress
and updates the developers
https://www.wix.engineering/post/making-order-in-ci-cd-mess

48. Snapshot
Commit to master branch, triggers a build in dev TeamCity, a successful build produces
a SNAPSHOT version artifacts that are stored at Wix binary repository (Artifactory).
Experience

49. RC
The version is ready for deployment; it first needs to be promoted from Snapshot to RC,
this operation practically copied the last snapshot version from SNAPSHOT repository
in Artifactory to RC repository
Experience

50. RC
1 commit = 1 version
Each commit in master potentially can be released to production
Experience

51. RC
We deploy every RC to CDN
CDN: https://static.parastorage.com/services/editor-elements/1.221.0/dist/main.min.js
Experience

52. GA
Once an RC version is ready, the developer can deploy it to production by updating the
production with the new RC version.
Experience

53. GA
https://static.parastorage.com/services/editor-elements/1.221.0/dist/main.min.js
Experience
https://static.parastorage.com/services/editor-elements/1.223.0/dist/main.min.js

54. Environments
Experience
We have SINGLE prod environment
No staging/pre-prod or something else
On prod we are doing all testing
With the internal tool, we can override any
production artifact version with RC

55. Gradual rollout
Experience
We can open new features for
▪ specific group of instances in a cluster
▪ specific group of people
▪ specific group of GEO
▪ set % for every group

56. Gradual rollout makes sense
if you have enough traffic

57. A/B Experiment Infrastructure
https://github.com/wix/petri/
Experience

58. Monitoring
Experience
▪ kibana / Log monitoring
▪ sentry / JS runtime monitoring
▪ new relic / VM monitoring
▪ grafana / action monitoring
▪ pingdom / alive monitoring
▪ tableau / business monitoring
▪ opsgenie / on call management

59. Rollbacks
Experience
Easy to trace what has changed in
production and rollback if the new
version didn’t behave as expected

60. Enjoy
your Early, Ofte releasesearly, often, stable

61. Fast user feedback loop
Minimum development friction
High quality
Huge team motivation

62. Thank You
alexeyk@wix.com github.com/kupriyanenko

63. alexeyk@wix.com github.com/kupriyanenko
Q&A

64. GAs per Month
Experience