1.
Continuous Deployment
of Architectural Change
Matt Graham
Core Engineer @ Etsy
Continuous Deployer
#iasany
September 27, 2011

2.
A Brief History of
Deployment

3.
The Internet

4.
Web Applications

5.
Agility

6.
Continuous

7.
Database Deployment @ Etsy
Thursday

8.
What it's all about
● Broadly Applicable
● Tools Help
● Enables the Unfeasible

9.
MTTR vs MTTF

10.
MTTR vs MTTF
● Cheaper for
electrons

11.
MTTR vs MTTF
● Cheaper for ● Cheaper for
electrons protons & neutrons

12.
MTTR vs MTTF
● Cheaper for ● Cheaper for
electrons protons & neutrons
● Cost prohibitive
for humans

13.
MTTR vs MTTF
● Cheaper for ● Cheaper for
electrons protons & neutrons
● Cost prohibitive
for humans
● Necessitates low
MTTD

14.
The Case of Electrons
● Consider a project that has 6 bugs
● continuous deployment testing misses 4
& deploys fixes in 4 hours
● monthly release testing misses only 2
& deploys fixes in 24 hours
● 16 hours of broken < 48 hours of broken

15.
Good Excuses
● Infrequent Changes

16.
Good Excuses
● Infrequent Changes
● Infrequent Executions

17.
Good Excuses
● Infrequent Changes
● Infrequent Executions
● Life and Death

18.
Good Excuses
● Infrequent Changes
● Infrequent Executions
● Life and Death
● Physical Investment

19.
Deployment Tools @ Etsy

20.
Culture Before Tools
● No planned deploys; only on demand

21.
Culture Before Tools
● No planned deploys; only on demand
● A positive change is ready & tested

22.
Culture Before Tools
● No planned deploys; only on demand
● A positive change is ready & tested
● Software is stable & supported

23.
Jenkins
● Unit Tests

24.
Jenkins
● Unit Tests
● Functional Tests

25.
Jenkins
● Unit Tests
● Functional Tests
● Manual Testing

26.
Nagios & Naglite2
github.com/lozzd/Naglite2

27.
tail -f | grep

28.
github.com/etsy/deployinator

29.
IRC

30.
Key Tools
Communication Tools
Graphs
Feature Flags

31.
Feature Flags
Deploy != Product Launch

32.
Dark Launch
def get_feature_a_link():
return ...

33.
Dark Launch
def get_feature_a_link():
if enabled('NewFeatureA'):
return new_feature_a_link()
else:
return old_feature_a_link()

34.
Dark Launch
application_config
- NewFeatureA: admin
- NewFeatureB: off
- NewFeatureC: on

35.
Ramp Up
application_config
- NewFeatureA: 1%
- NewFeatureB: off
- NewFeatureC: on

36.
Ramp Up
application_config
- NewFeatureA: 5%
- NewFeatureB: off
- NewFeatureC: on

37.
Ramp Up
application_config
- NewFeatureA: 25%
- NewFeatureB: off
- NewFeatureC: on

38.
Ramp Up
application_config
- NewFeatureA: 100%
- NewFeatureB: off
- NewFeatureC: on

39.
AB Testing
● Prove success of interface changes
● Prove interest in new features

40.
Graphs!!!

41.
Ganglia

42.
Graphite

43.
Event Overlay

44.
StatsD
if ($success) {
StatsD::timing('feature.runtime', $time);
} else {
StatsD::increment('feature.failure');
}
github.com/etsy/statsd

45.
github.com/etsy/logster

46.
github.com/etsy/logster

47.
Community Communication

48.
Forums / Message Boards

49.
etsystatus.com

50.
twitter.com/etsystatus

51.
twitter.com/etsystatus

52.
Deployment is First Class
Deployment is a
First Class Feature

53.
Engineers are Users Too

54.
Enable Architectural Change

55.
Architecture is Relative

56.
Organic Architecture

57.
Premature Architecture

58.
Premature Architecture
What now?

59.
Passing Time => Change
● Scale
● Product
● Technology
● Engineering Team

60.
Passing Time => Change
● Scale
● Product
● Technology
● Engineering Team
● The Correct Architecture Changes

61.
Examples
● Images From Twisted to PHP
● From PostgreSQL to MySQL Shards

62.
From Twisted to PHP
● Run Apache/PHP on a new port

63.
From Twisted to PHP
● Run Apache/PHP on a new port
● Implement one service in PHP

64.
From Twisted to PHP
● Run Apache/PHP on a new port
● Implement one service in PHP
● Ramp up users on new service

65.
From Twisted to PHP
● Run Apache/PHP on a new port
● Implement one service in PHP
● Ramp up users on new service
● Repeat for remaining services

66.
From Twisted to PHP
● Run Apache/PHP on a new port
● Implement one service in PHP
● Ramp up users on new service
● Repeat for remaining services
● Shut down Twisted version

67.
PostgreSQL to MySQL Shards
● Migrate table by table

68.
PostgreSQL to MySQL Shards
● Migrate table by table
● Teed writes to MySQL

69.
PostgreSQL to MySQL Shards
● Migrate table by table
● Teed writes to MySQL
● Backfill old data from PostgreSQL

70.
PostgreSQL to MySQL Shards
● Migrate table by table
● Teed writes to MySQL
● Backfill old data from PostgreSQL
● Verify data matches in both DBs

71.
PostgreSQL to MySQL Shards
● Migrate table by table
● Teed writes to MySQL
● Backfill old data from PostgreSQL
● Verify data matches in both DBs
● Ramp up users reading from MySQL

72.
PostgreSQL to MySQL Shards
● Migrate table by table
● Teed writes to MySQL
● Backfill old data from PostgreSQL
● Verify data matches in both DBs
● Ramp up users reading from MySQL
● Stop Postgres writes & drop tables

73.
Continuous Deploy Pattern
● Change in small steps

74.
Continuous Deploy Pattern
● Change in small steps
● Dark launch placeholder via config

75.
Continuous Deploy Pattern
● Change in small steps
● Dark launch placeholder via config
● Deploy to production while dark

76.
Continuous Deploy Pattern
● Change in small steps
● Dark launch placeholder via config
● Deploy to production while dark
● Maintain old & new in parallel

77.
Continuous Deploy Pattern
● Change in small steps
● Dark launch placeholder via config
● Deploy to production while dark
● Maintain old & new in parallel
● Ramp up to new architecture

78.
Continuous Deploy Pattern
● Change in small steps
● Dark launch placeholder via config
● Deploy to production while dark
● Maintain old & new in parallel
● Ramp up to new architecture
● Remove old architecture

79.
Review
● Broadly applicable
● Key Tools: Feature Flags, Graphs,
Communication
● Enables Architectural Change
● It's really fun!!!

80.
Continuous Deployment
of Architectural Change
Matt Graham
http://twitter.com/lapsu
http://lapsu.tv
Core Engineer @ Etsy
Continuous Deployer
http://codeascraft.etsy.com
We're Hiring!
http://bit.ly/etsywantsawesome