Releasing To Production Every Week India

Releasing to Production Every Week Owen Rogers Twitter: @exortech http://exortech.com/blog

1 year

49 releases

49 releases (excluding patch releases)

~ 1 release / week

summary

lessons learned

process

practices

how long is your release cycle?

Vancouver

> 15 teams

Agile methods

Company

start up

real-time energy monitoring

building energy efficiency

save $

save kW

aggregate data

hosted solution

• 9 developers

• 9 developers • 1 product manager

• 9 developers • 1 product manager • 1 graphic designer

• 9 developers • 1 product manager • 1 graphic designer • 1 tester

what’s missing?

operations?

support?

project manager?

generalizing specialists

fungibility

rotating responsibility

daily support duty

few hand-offs

empowered

Process

Goal: • to surface risk as early as possible to keep problems out of production

maximize feedback

shrink time

Continuous Integration

Continuous Integration • 3-4 Commits/dev/day

Automated Build

Automated Build • 3 minute build

Automated Build • 3 minute build • 30-40 builds/day

Automated Test Deploy

Automated Test Deploy • 4-5 times per day

Automated Test Deploy • 4-5 times per day • Scheduled nightly at 11PM

Daily Standup

Daily Standup • Weekly company standup

Daily Review

Daily Review • Quick peer feature review

Weekly Demo

Weekly Demo • 15 minute company- wide user-driven demo

weekly cycle Mon Tues Wed Thurs Fri 9:30AM 10AM 11pm 5pm Planning User-driven Cut-off Release Meeting Demo Release Testing

Sustainable Pace

Sustainable Pace • regular release window

Sustainable Pace • regular release window • quality of life and code

Sustainable Pace • regular release window • quality of life and code • part of culture

Sustainable Pace • regular release window • quality of life and code • part of culture • company heartbeat

“agile”

Scrum?

what’s missing?

user stories?

story boards?

estimation?

estimation? fits or it doesn’t “muda”

pairing?

pairing? as needed

pairing? as needed code reviews

pairing? as needed code reviews open workspace

TDD? 52% coverage

TDD? 52% coverage + Selenium

retrospectives?

retrospectives? Release Review

retrospectives? Release Review 5 Whys

lightweight RCA

5 Whys

just-in-time

within 24 hours

1 Jira per question

Practices

Daily support rotation

5 Whys

Continuous Monitoring

Continuous Monitoring Socket App Server LogMaster Logger (async) WARN | ERROR

Continuous Monitoring • Proactive response to issues

Continuous Monitoring • Proactive response to issues • Clean logs

Continuous Monitoring • Proactive response to issues • Clean logs • Message throttling

Continuous Monitoring • Proactive response to issues • Clean logs • Message throttling • Gmail

Continuous Monitoring • Proactive response to issues • Clean logs • Message throttling • Gmail • Managing the signal-to-noise ratio

Continuous Monitoring App Server web app gmond gmetad UDP multicast RRD

Zero-downtime Deployment

Zero-downtime Deploys Load Balancer App Server App Server App Server Deploy

Zero-downtime Deploys

Zero-downtime Deploys • Fully automated deployment

Zero-downtime Deploys • Fully automated deployment • One button deploy from TeamCity

Zero-downtime Deploys • Fully automated deployment • One button deploy from TeamCity • Gracefully bring down servers

Zero-downtime Deploys What about the DB?

Zero-downtime Deploys Bering

Zero-downtime Deploys 012_rename_login_id_to_user_name ALTER TABLE USER RENAME COLUMN login_id user_name

Zero-downtime Deploys 012_rename_login_id_to_user_name 012_add_user_name_column 02_remove_login_id_column expansion contraction

Zero-downtime Deploys • expansion

Zero-downtime Deploys • expansion • preserves backwards compatibility

Zero-downtime Deploys • expansion • preserves backwards compatibility • contract

Zero-downtime Deploys • expansion • preserves backwards compatibility • contract • clean up

Zero-downtime Deploys • Typically safe expansion operations:

Zero-downtime Deploys • Typically safe expansion operations: • add table, add column, insert data

Zero-downtime Deploys • Typically safe expansion operations: • add table, add column, insert data • Typically unsafe expansion operations:

Zero-downtime Deploys • Typically safe expansion operations: • add table, add column, insert data • Typically unsafe expansion operations: • rename or remove columns or tables

Zero-downtime Deploys • Typically safe expansion operations: • add table, add column, insert data • Typically unsafe expansion operations: • rename or remove columns or tables • add constraint

Zero-downtime Deploys /expand /... /143 001_create_login_table 002_add_login_id_index 003_create_user_table 004_create_group_table /... /contract /... /143 001_add_inv_constraint 002_drop_alias_column /...

Zero-downtime Deploys • Database migration decoupled from the release

Zero-downtime Deploys • Database migration decoupled from the release • Simplified rollback

Zero-downtime Deploys • Database migration decoupled from the release • Simplified rollback • Some additional complexity in writing migrations

Release Options

Traditional Release

Traditional Release • high risk

Traditional Release • high risk • not scalable

Traditional Release • high risk • not scalable • prolonged release: “Don’t release until it’s all ready”

Release Options

Release Options • release to user subset (private beta)

Release Options • release to user subset (private beta) • split test (experimentation)

Release Options • release to user subset (private beta) • split test (experimentation) • incremental rollout

Release Options • release to user subset (private beta) • split test (experimentation) • incremental rollout • selective enabling/downgrading

Release Options • release to user subset (private beta) • split test (experimentation) • incremental rollout • selective enabling/downgrading • disable feature (feature darkmode)

Release Options • release to user subset (private beta) • split test (experimentation) • incremental rollout • selective enabling/downgrading • disable feature (feature darkmode) • defer commit

Release Options • release to user subset (private beta) • split test (experimentation) • incremental rollout • selective enabling/downgrading • disable feature (feature darkmode) • defer commit • defer release

Release Options

Release Options • Features running in production for weeks before fully available

Release Options • Features running in production for weeks before fully available • lots of data

Release Options • Features running in production for weeks before fully available • lots of data • “Release is a marketing term”

Release Options • Features running in production for weeks before fully available • lots of data • “Release is a marketing term” • Production levers

Release Options • Features running in production for weeks before fully available • lots of data • “Release is a marketing term” • Production levers • WANGMI: the discipline to defer

Use Production Data

Use Production Data • Upload Nightly backups

Use Production Data • Upload Nightly backups • Automated Test DB refresh after every release

Use Production Data • Upload Nightly backups • Automated Test DB refresh after every release • Local DB refresh on demand

Use Production Data • Upload Nightly backups • Automated Test DB refresh after every release • Local DB refresh on demand • Cleansed

Use Production Data • Upload Nightly backups • Automated Test DB refresh after every release • Local DB refresh on demand • Cleansed • Test data is a liability

Single Feature Release

• Daily Support Rotation • 5 Whys • Continuous Monitoring • Zero-downtime deployment • Release Options • Use Production Data • Single Feature Release

Questions? owen@pulseenergy.com

http://exortech.com	846
http://blog.codechef.com	476
http://www.codechef.com	385
http://blog.ww2.codechef.com	9
http://theoldreader.com	2
http://ww2.codechef.com	2
http://www.slideshare.net	1
http://staging.codechef.com	1

Releasing To Production Every Week India

by exortech on Sep 17, 2009

Accessibility

Categories

Upload Details

Usage Rights

8 Embeds 1,722

Statistics

Releasing To Production Every Week India Presentation Transcript