The document discusses the rise of DevOps as a new IT operations support model enabled by large cloud providers. It argues that DevOps will emerge because it is not solely driven by top-down implementations, and enables IT organizations to look for new models enabled by tools like Chef, Puppet, and Docker. DevOps adoption will be stimulated by the growing interest in tools that help align development and operations teams.

1. Production Begins in
Development
Lee Thompson Damon Edwards

2. • CTO Travel/Transportation
Products, Hewlett Packard
• Former Chief Technologist,
E*TRADE Financial
• Co-writer of “Fully Automated
Lee Thompson Provisioning” whitepaper
@stagr_lee

3. • Co-founder DTO Solutions
• DevOps Days organizer
• Co-writer of “Fully Automated
Provisioning” whitepaper
Damon Edwards
@damonedwards

4. • Co-founder DTO Solutions
• DevOps Days organizer
• Co-writer of “Fully Automated
Provisioning” whitepaper
Damon Edwards
@damonedwards

5. • Consumer-facing web applications
• Webtone utilities

6. Showing us what is possible...
Clouds
Continuous Delivery/Deployment
Lean Startup

7. What we want more of...
DevOps!

8. vOps. DevOps. DevOps. DevOps. DevO
The Rise of a New IT Operations
Support Model
!"#$%&'(#)*+,-.#/011#*+21+*#3425#6#7089*#.:46:*;"#*5-12"*<#
="#164;*#812><#-42+0<*4.#07:2#6#5607.:4*65#.:46:*;"#*5-12"*<#
="#$%?#23#@12=61#$%%%#24;670A6:027.C
=" $%? 23 @12=61 $%%% 24;670A6:027.
Why DevOps will not emerge: Why DevOps will emerge:
!B>1:>461 8967;*. 64* :9* 964<*.: :2 !)*+,-. 0. 72: >.>611" <40+*7 3425
B>1:>461#8967;*.#64*#:9*#964<*.:#:2# )*+,-.#0.#72:#>.>611"#<40+*7#3425#
05-1*5*7:(#67<#)*+,-.#4*F>04*.#6# :9*#:2-#<2/7#67<(#:9>.(#56"#=*#
.0;7030867:#4*:907G07;#23#DE# 524*#*6.01"#688*-:*<#="#DE#
2-*46:027.#827+*7:02761#/0.<25C 2-*46:027.#:*65.C
!E9*4*#0.#6#164;*#=2<"#23#/24G#/0:9# !DEDH#67<#2:9*4#=*.:#-468:08*.#
4*.-*8:#:2#DEDH#67<#2:9*4#=*.:# 3465*/24G.#64*#68G72/1*<;*<#:2#
-468:08*.#3465*/24G.#:96:#0.#614*6<"# 96+*#72:#<*10+*4*<#27#:9*04#;261.(#
688*-:*<#/0:907#:9*#07<>.:4"C
688*-:*< /0:907 :9* 07<>.:4" *76=107;#DE#24;670A6:027.#:2#122G#324#
*76=107; DE 24;670A6:027. :2 122G 324
!,-*7#.2>48*#J,KKL#5676;*5*7:# 7*/#52<*1.C
:221.(#/9089#64*#524*#610;7*<#/0:9# !E9*#;42/07;#07:*4*.:#07#:221.#.>89#
--
:90.#6--42689(#96+*#72:#.**7# -- -
6.#B9*3(#I>--*:(#*:8C(#/011#9*1-#
.0;7030867:#*7:*4-40.*#564G*:#.964*# .:05>16:*#<*567<#324#,KKM=6.*<#
:468:027C 5676;*5*7:

9. vOps. DevOps. DevOps. DevOps. DevO
The Rise of a New IT Operations
Support Model
!"#$%&'(#)*+,-.#/011#*+21+*#3425#6#7089*#.:46:*;"#*5-12"*<#
="#164;*#812><#-42+0<*4.#07:2#6#5607.:4*65#.:46:*;"#*5-12"*<#
="#$%?#23#@12=61#$%%%#24;670A6:027.C
=" $%? 23 @12=61 $%%% 24;670A6:027.
Why DevOps will not emerge: Why DevOps will emerge:
!B>1:>461 8967;*. 64* :9* 964<*.: :2 !)*+,-. 0. 72: >.>611" <40+*7 3425
B>1:>461#8967;*.#64*#:9*#964<*.:#:2# )*+,-.#0.#72:#>.>611"#<40+*7#3425#
05-1*5*7:(#67<#)*+,-.#4*F>04*.#6# :9*#:2-#<2/7#67<(#:9>.(#56"#=*#
.0;7030867:#4*:907G07;#23#DE# of a New IT524*#*6.01"#688*-:*<#="#DE#
The Rise Operations
Support Model
2-*46:027.#827+*7:02761#/0.<25C 2-*46:027.#:*65.C
!"#$%&'(#)*+,-.#/011#*+21+*#3425#6#7089*#.:46:*;"#*5-12"*<#
!E9*4*#0.#6#164;*#=2<"#23#/24G#/0:9# !DEDH#67<#2:9*4#=*.:#-468:08*.#
="#164;*#812><#-42+0<*4.#07:2#6#5607.:4*65#.:46:*;"#*5-12"*<#
4*.-*8:#:2#DEDH#67<#2:9*4#=*.:# $%? 23 @12=61 $%%% 24;670A6:027.
3465*/24G.#64*#68G72/1*<;*<#:2#
="#$%?#23#@12=61#$%%%#24;670A6:027.C
="
-468:08*.#3465*/24G.#:96:#0.#614*6<"# :296+*#72:#<*10+*4*<#27#:9*04#;261.(#
Why DevOps will not emerge:
B>1:>461 8967;*. 64* :9* 964<*.:
! !
Why DevOps will emerge:
B>1:>461#8967;*.#64*#:9*#964<*.:#:2# )*+,-. 0. 72: >.>611" <40+*7 3425
)*+,-.#0.#72:#>.>611"#<40+*7#3425#
688*-:*<#/0:907#:9*#07<>.:4"C
688*-:*< /0:907 :9* 07<>.:4"
05-1*5*7:(#67<#)*+,-.#4*F>04*.#6#*76=107; DE 24;670A6:027. :2 122G 324
*76=107;#DE#24;670A6:027.#:2#122G#324#
:9*#:2-#<2/7#67<(#:9>.(#56"#=*#
.0;7030867:#4*:907G07;#23#DE# 524*#*6.01"#688*-:*<#="#DE#
2-*46:027.#827+*7:02761#/0.<25C 7*/#52<*1.C
!,-*7#.2>48*#J,KKL#5676;*5*7:# 2-*46:027.#:*65.C
E9*4*#0.#6#164;*#=2<"#23#/24G#/0:9#
! ! DEDH#67<#2:9*4#=*.:#-468:08*.#
:221.(#/9089#64*#524*#610;7*<#/0:9# !E9*#;42/07;#07:*4*.:#07#:221.#.>89#
4*.-*8:#:2#DEDH#67<#2:9*4#=*.:# 3465*/24G.#64*#68G72/1*<;*<#:2#
-468:08*.#3465*/24G.#:96:#0.#614*6<"# 96+*#72:#<*10+*4*<#27#:9*04#;261.(#
--
:90.#6--42689(#96+*#72:#.**7# 07<>.:4"
688*-:*<#/0:907#:9*#07<>.:4"C
688*-:*< /0:907 :9* --
6.#B9*3(#I>--*:(#*:8C(#/011#9*1-#
*76=107;#DE#24;670A6:027.#:2#122G#324#
*76=107; DE 24;670A6:027. :2 122G 324 -
7*/#52<*1.C
.0;7030867:#*7:*4-40.*#564G*:#.964*#
:221.(#/9089#64*#524*#610;7*<#/0:9# .:05>16:*#<*567<#324#,KKM=6.*<#
,-*7#.2>48*#J,KKL#5676;*5*7:#
!
! E9*#;42/07;#07:*4*.:#07#:221.#.>89#
:468:027C --
:90.#6--42689(#96+*#72:#.**7#
.0;7030867:#*7:*4-40.*#564G*:#.964*#
5676;*5*7: --
6.#B9*3(#I>--*:(#*:8C(#/011#9*1-#
.:05>16:*#<*567<#324#,KKM=6.*<#
-
:468:027C 5676;*5*7:

10. Measuring success...
Alignment
Quality
Cycle Time

11. Know what kind of
business you are

12. What is your tolerance for...
Risk
Change

13. Consumer-facing applications...
"Move fast and break things.
Unless you are breaking stuff,
you are not moving fast enough."
-Mark Zuckerberg

14. Consumer-facing applications...
"Move fast and break things.
Unless you are breaking stuff,
you are not moving fast enough."
-Mark Zuckerberg

15. Webtone utilities...

16. Webtone utilities...

17. Webtone utilities...

18. Webtone utilities...

19. Webtone utilities...

20. What do you want...

21. What do you want...

22. What do you want...

23. What do you want...

24. What do you want...

25. What do you want...

26. What do you want...

27. What do you want...

28. Clouds
DevOps
Continuous Delivery/Deployment
Lean Startup
Alignment
Quality
Cycle Time

29. It all starts in Development

30. It all starts in Development
But what do we tell them to do
and how do we get them to do it?

31. Share ownership of availability
• Developers must wear pagers
• Incident command training so everyone
knows their roles
• Notification mechanism?
Access provisioning?

32. Non-functional requirements are
first class citizens
• All of the “-ility” requirements
• As much a part of your service as the
businessʼs functional requirements
• Manage through same process
• Make sure they are on the schedule
• Donʼt over commit

33. Strive for parity between dev & prod
• Develop test data fixtures for all environments
• Implement Mock Services for major
infrastructure pieces for Developer user.
• Continuous integration means integrate early
• Use the deployment, config, and packaging
tools in dev

34. Push config management
discipline back to Dev
• Dev is about creating variation, Ops is
about eliminating variation
• Augment deployment toolchain to support
the variation
• Do developers use the tools?
• Do developers like them?
• Devʼs need config too
• Accept config contributions and patches
from dev

35. Packaging... itʼs not just for the OS
• If you want to be a high performing web
operations organization, you need to take
change management seriously.
• Strict versioning is needed, and package
managers do that.
• Itʼs about being idempotent.
• Transfer packaging responsibility to dev
• Define the packaging constructs you will
support
http://webscalebook.com

36. Config is code
• if itʼs code it needs to be managed like
code...
• Should be transparent and identical SDLC
in both dev and ops
• Avoid or eliminate asymetric release
processes

37. Tailor release artifacts to roles
• “Small teams make better software”
• Make sure one team getting stuck doesnʼt
prevent another team to release (org
coupling)
• Large codebases suffer software entropy
effects
• Build an infrastructure that can reliably
manage lots of smaller artifacts
• Org conflict is a good time to suggest
breaking up a codebase into separate
concerns

38. Standard management vocabulary
• Consistent and expected management
behavior
• Across components and releases
• “start, stop, status, update, install...”

39. Rollback
• Rollback that works
• No, really works
• Seriously, really really works
• You say it will work? Test it right now

40. Standard metrics abstractions
• Dev surface metrics to Ops
• Avoid the back and forth conversations
• Use a standard framework
• Use standard types (gague, counter,
timer...)
• Ops knows what to expect and how to
visualize
https://github.com/codahale/metrics

41. Push test ownership to the edges
• QA is “Quality Assurance”
• QA writing tests = bottleneck
• QA writing tests = avoiding responsibility
• QA is architecture / tooling / assurance
• Test Driven Development
• Test Driven Operations (yes, you too)
• Bottom line: Everyone owns quality.

42. Test outside of the box
Crowd Test Simulate

43. Simulation
AUT

44. Build the right lifecycle...
Alignment
Quality
Cycle Time

45. Continuous Delivery...

46. Continuous Delivery...
Deployment...
Automated Automated
Testing Release

47. Donʼt be so dogmatic

48. Decoupled Release Model Example
Code, Config, Control
Code, Config, Control
Code, Config, Control
"Check-
Operations in"
"Check-
Rundeck
1 in"
"Check- Source
1 in" Repo
Source 1 Monitor
Rundeck Chef
1
"Check-
in"
"Check-
Source
Repo
"Auto-Deploy Repo
Instance Instance Develop in"
"Check-
New App Develop Package Manifest
in" "Check- Jobs &
Infrastructure
CI Server
Develop Package Manifest
Package Manifest
out"
"Check-
out"
"Check-
2 Adhoc
1 Instance Source
Repo
"Verify"
2 out" Operate
Commands
"Verify"
2
2
Watch
"Verify" Code
Monitoring Watch
Publish Code
& Testing Staging Results Watch
Build Rundeck
Release Publish Code
ResultsContinuous Build
Repo Publish Release Chef
Deployment
ResultsContinuous Build
Repo
Release
Continuous
Loop
Integration Deployment Unit RightScale
Testing Tests "Promote" ARelease
Repo
Prod Integration Deployment Unit
Loop
"Promote" Repo
A Prod
Release Testing Loop Tests
Repo
Dev Test
Automatic
Integration
Deployment
Testing
Unit
Tests 3
"Promote" A
Dev Test Automatic
Deployment 3
Dev Test
Dev Test
Automatic
Deployment 3
Bootstrap Develop Release Operate
Release
Repo Release
A Release Repo
Repo A
1 Repo
B
4 Promote Prod
Release
Repo
B
Audit Release
Job Repo Release
Repo Repo
C C
"Determine what packages "Promote release"
and versions are available"
2 "Check
-in"
Souce
Repo
Hudson
Prod
Release
Repo
Release
Package Manifest 5 Deploy
Job
"yum install"
Release
app code
Repo
A
Release
Rundeck
services Repo
3 Validate
Release
OS B
Release
Chef
Repo RightScale
C
"Check for conflicts and
Sandbox
Prod
run validation scripts"

49. Decoupled Release Model Example
Code, Config, Control
Code, Config, Control
Code, Config, Control
"Check-
Operations in"
"Check-
Rundeck
1 in"
"Check- Source
1 in" Repo
Source 1 Monitor
Rundeck Chef
1
"Check-
in"
"Check-
Source
Repo
"Auto-Deploy Repo
Instance Instance Develop in"
"Check-
New App Develop Package Manifest
in" "Check- Jobs &
Infrastructure
CI Server
Develop Package Manifest
Package Manifest
out"
"Check-
out"
"Check-
2 Adhoc
1 Instance Source
Repo
"Verify"
2 out" Operate
Commands
"Verify"
2
2
Watch
"Verify" Code
Monitoring Watch
Publish Code
& Testing Staging Results Watch
Build Rundeck
Release Publish Code
ResultsContinuous Build
Repo Publish Release Chef
Deployment
ResultsContinuous Build
Repo
Release
Continuous
Loop
Integration Deployment Unit RightScale
Testing Tests "Promote" ARelease
Repo
Prod Integration Deployment Unit
Loop
"Promote" Repo
A Prod
Release Testing Loop Tests
Repo
Dev Test
Automatic
Integration
Deployment
Testing
Unit
Tests 3
"Promote" A
Dev Test Automatic
Deployment 3
Dev Test
Dev Test
Automatic
Deployment 3
Bootstrap Develop Release Operate
Release
Repo Release
A Release Repo
Repo A
1 Repo
B
4 Promote Prod
Release
Repo
B
Audit Release
Job Repo Release
Repo Repo
C C
"Determine what packages "Promote release"
and versions are available"
2 "Check
-in"
Souce
Repo
Hudson
Prod
Release
Repo
Release
Package Manifest 5 Deploy
Job
"yum install"
Release
app code
Repo
A
Release
Rundeck
services Repo
3 Validate
Release
OS B
Release
Chef
Repo RightScale
C
"Check for conflicts and
Sandbox
Prod
run validation scripts"

50. Bootstrap
Operations
Rundeck
Rundeck Chef
"Auto-Deploy Instance Instance
New App
Infrastructure
CI Server
Source
1 Instance
Repo
Monitoring
& Testing Staging
Release
Repo
Prod
Release
Repo
Dev Test

51. Decoupled Release Model Example
Code, Config, Control
Code, Config, Control
Code, Config, Control
"Check-
Operations in"
"Check-
Rundeck
1 in"
"Check- Source
1 in" Repo
Source 1 Monitor
Rundeck Chef
1
"Check-
in"
"Check-
Source
Repo
"Auto-Deploy Repo
Instance Instance Develop in"
"Check-
New App Develop Package Manifest
in" "Check- Jobs &
Infrastructure
CI Server
Develop Package Manifest
Package Manifest
out"
"Check-
out"
"Check-
2 Adhoc
1 Instance Source
Repo
"Verify"
2 out" Operate
Commands
"Verify"
2
2
Watch
"Verify" Code
Monitoring Watch
Publish Code
& Testing Staging Results Watch
Build Rundeck
Release Publish Code
ResultsContinuous Build
Repo Publish Release Chef
Deployment
ResultsContinuous Build
Repo
Release
Continuous
Loop
Integration Deployment Unit RightScale
Testing Tests "Promote" ARelease
Repo
Prod Integration Deployment Unit
Loop
"Promote" Repo
A Prod
Release Testing Loop Tests
Repo
Dev Test
Automatic
Integration
Deployment
Testing
Unit
Tests 3
"Promote" A
Dev Test Automatic
Deployment 3
Dev Test
Dev Test
Automatic
Deployment 3
Bootstrap Develop Release Operate
Release
Repo Release
A Release Repo
Repo A
1 Repo
B
4 Promote Prod
Release
Repo
B
Audit Release
Job Repo Release
Repo Repo
C C
"Determine what packages "Promote release"
and versions are available"
2 "Check
-in"
Souce
Repo
Hudson
Prod
Release
Repo
Release
Package Manifest 5 Deploy
Job
"yum install"
Release
app code
Repo
A
Release
Rundeck
services Repo
3 Validate
Release
OS B
Release
Chef
Repo RightScale
C
"Check for conflicts and
Sandbox
Prod
run validation scripts"

52. Develop
Code, Config, Control
"Check-
in"
1 Source
Repo
"Check-
in"
Develop
Package Manifest "Check-
out"
"Verify"
2
Watch
Code
Publish
Results Build
Continuous
Release
Deployment
Repo
Integration Loop Unit
Testing Tests "Promote" A
Dev Test
Automatic
Deployment 3

53. Decoupled Release Model Example
Code, Config, Control
Code, Config, Control
Code, Config, Control
"Check-
Operations in"
"Check-
Rundeck
1 in"
"Check- Source
1 in" Repo
Source 1 Monitor
Rundeck Chef
1
"Check-
in"
"Check-
Source
Repo
"Auto-Deploy Repo
Instance Instance Develop in"
"Check-
New App Develop Package Manifest
in" "Check- Jobs &
Infrastructure
CI Server
Develop Package Manifest
Package Manifest
out"
"Check-
out"
"Check-
2 Adhoc
1 Instance Source
Repo
"Verify"
2 out" Operate
Commands
"Verify"
2
2
Watch
"Verify" Code
Monitoring Watch
Publish Code
& Testing Staging Results Watch
Build Rundeck
Release Publish Code
ResultsContinuous Build
Repo Publish Release Chef
Deployment
ResultsContinuous Build
Repo
Release
Continuous
Loop
Integration Deployment Unit RightScale
Testing Tests "Promote" ARelease
Repo
Prod Integration Deployment Unit
Loop
"Promote" Repo
A Prod
Release Testing Loop Tests
Repo
Dev Test
Automatic
Integration
Deployment
Testing
Unit
Tests 3
"Promote" A
Dev Test Automatic
Deployment 3
Dev Test
Dev Test
Automatic
Deployment 3
Bootstrap Develop Release Operate
Release
Repo Release
A Release Repo
Repo A
1 Repo
B
4 Promote Prod
Release
Repo
B
Audit Release
Job Repo Release
Repo Repo
C C
"Determine what packages "Promote release"
and versions are available"
2 "Check
-in"
Souce
Repo
Hudson
Prod
Release
Repo
Release
Package Manifest 5 Deploy
Job
"yum install"
Release
app code
Repo
A
Release
Rundeck
services Repo
3 Validate
Release
OS B
Release
Chef
Repo RightScale
C
"Check for conflicts and
Sandbox
Prod
run validation scripts"

54. Release
Release
Repo Release
A Release Repo
Repo A
1 Repo
B
4 Promote Prod
Release
Repo
B
Audit Release
Job Repo Release
Repo Repo
C C
"Determine what packages "Promote release"
and versions are available"
2 "Check
-in"
Souce
Repo
Hudson
Prod
Release
Repo
ease
Package Manifest 5 Deploy
Job
"yum install"
Release
app code
Repo
A
Release
Rundeck
services Repo
3 Validate
Release
OS B
Release
Chef
Repo RightScale
C
"Check for conflicts and
Sandbox
Prod
run validation scripts"