This slide covers Dark Launch and its Best practices.

1. Dark Launch
Best Practices
Giragadurai Vallirajan
@Girag

2. Agenda
● What is Dark Launching?
● Why we need it?
● Gitflow Workflow
● Best Practices
○ Deployment
○ Source Code Management - Branching Strategies
○ Architectural artifacts
● QA

3. What is Dark Launching?
A Dark launch is a one of practice where newly developer features and fixes are
deployed into production in the less user visible way. Major goals of dark
launch are to enable testing in production and minimise and contain the risk of
outages. Those features launched are tested with a lesser percentage of
user/traffic and then made available for wider audiences over the time. This
method was practiced and popularised by major companies like Facebook and
Google.

4. Why we need it?
Launching in dark addresses many teasing in production scenarios, however,
not limited to list, because there is no other environment like the production to
test features extensively.
1. Where the production traffic patterns not always reproduce in the
development environment.
2. Where real customer’s usage pattern needed.
3. Where features mostly backends or Infrastructural changes (Non-UI).
4. An easy way to load or stress test the feature.

5. Gitflow Workflow
GitFlow workflow is the best choice to implement the listed dark launches. has
strict branching strategies to manage large, complex project releases. GitFlow
workflow was first posted by Vincent Driessen at nvie. Its well suited for
projects that have scheduled release cycle and agile in nature. However, for
implementation it doesn’t need any additional fancy commands to learn.

6. Sample GitFlow Workflow

7. Best Practices
It covers most used Dark Launches, best practices of
● Blue Green Deployment
● A/B Testing
● Canary Deployment
And scope is,
● Application Deployment
● Source code management
● Data and dependency management

8. Blue Green Deployment
In this model, both versions of application deployed in two identical
infrastructure and tagged blue (v1) and green (v2) with switches on reverse
proxy. The new version of the application to be tested and qualified with smoke
test and other tests for production grade. Then environment ready to make
changes either v1 or v2 by diverting reverse proxy traffic. On the release day
switch the traffic to v2 and you are ready to go.

9. Deployment and Branching

10. A/B Testing
In this model, both versions of Application will be deployed and Pa% of people
will let to use Version A components and Pb% of people traffic directed to
Version B. The positive responses helps to choose the better feature for
production. This testing is meant for feature comparison on the application for
various reasons like ease of usability, clear noticeability, interesting design, etc
and its mostly targets UX selection.

11. Deployment and Branching

12. Canary Deployment
Normally, a weeks before the actual launch, certain features are deployed
silently and a subset of users traffic would be redirected to it. Its behaviour will
be analysed and it may expose, how the traffic absorbed by the new feature, is
there any performance, infrastructural capacity impacts and network
bottlenecks and etc. The faster feedback you get, the faster you can fail the
deployment, or proceed cautiously.

13. Deployment and Branching

14. Best Practices - Architecture
So far deployment and branching strategies were covered, however there are
various pain points while dark launching large scale and complex systems. In
this section, some of such pain points discussed with suggestive best
practices.

15. Pain Points #1...
Managing state and continuity of long-running process, it's one of hard pain
point and many batch data analytics will faces it. Dark launching such system
can be handled in two ways, easy way is holding new release until draining
running job, another way when release can’t hold, then have to store current
state of job, checkpoint of stages and data versioning to start the job with new
code where it left.

16. Pain Points #2...
Managing unsynchronised in-memory data consistency, it can be handled by
storing in-memory data in distributed data store like locally managed redis,
memcached with high performance or cloud-hosted AWS DynamoDB with high
availability.

17. Pain Points #3...
Manage necessary data(base) migration, capacity and infrastructural
changes, it has to be managed with well-planned migration and recovery
strategies. In case of database schema changes, previous version of data to be
migrated in to new schema and point updated code points the new table. In
case of increased capacity or infrastructural changes, start new version of code
with higher capacity infrastructure and resize later.

18. Pain Points #4...
Manage Non-sharable dependent services, time sharing based testing is one
of choice to solve such a problem.

19. Pain Points #5...
At times double the capacity of infrastructure needed especially for BlueGreen
deployment and AB Testing, well it’s more of business decision to consider
that they can afford without impacting calculated profit ratio.

20. Q&A