Your SlideShare is downloading. ×
Velocity 2011: Production Begins in Development
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Velocity 2011: Production Begins in Development

3,200

Published on

Velocity 2011 presentation by Damon Edwards and Lee Thompson

Velocity 2011 presentation by Damon Edwards and Lee Thompson

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,200
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
329
Comments
0
Likes
3
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Production Begins in DevelopmentLee Thompson Damon Edwards
  • 2. • CTO Travel/Transportation Products, Hewlett Packard • Former Chief Technologist, E*TRADE Financial • Co-writer of “Fully AutomatedLee Thompson Provisioning” whitepaper @stagr_lee
  • 3. • Co-founder DTO Solutions • DevOps Days organizer • Co-writer of “Fully Automated Provisioning” whitepaperDamon Edwards@damonedwards
  • 4. • Co-founder DTO Solutions • DevOps Days organizer • Co-writer of “Fully Automated Provisioning” whitepaperDamon Edwards@damonedwards
  • 5. • Consumer-facing web applications• Webtone utilities
  • 6. Showing us what is possible... Clouds Continuous Delivery/Deployment Lean Startup
  • 7. What we want more of... DevOps!
  • 8. vOps. DevOps. DevOps. DevOps. DevO The Rise of a New IT Operations Support Model !"#$%&(#)*+,-.#/011#*+21+*#3425#6#7089*#.:46:*;"#*5-12"*<# ="#164;*#812><#-42+0<*4.#07:2#6#5607.:4*65#.:46:*;"#*5-12"*<# ="#$%?#23#@12=61#$%%%#24;670A6:027.C =" $%? 23 @12=61 $%%% 24;670A6:027. Why DevOps will not emerge: Why DevOps will emerge: !B>1:>461 8967;*. 64* :9* 964<*.: :2 !)*+,-. 0. 72: >.>611" <40+*7 3425 B>1:>461#8967;*.#64*#:9*#964<*.:#:2# )*+,-.#0.#72:#>.>611"#<40+*7#3425# 05-1*5*7:(#67<#)*+,-.#4*F>04*.#6# :9*#:2-#<2/7#67<(#:9>.(#56"#=*# .0;7030867:#4*:907G07;#23#DE# 524*#*6.01"#688*-:*<#="#DE# 2-*46:027.#827+*7:02761#/0.<25C 2-*46:027.#:*65.C !E9*4*#0.#6#164;*#=2<"#23#/24G#/0:9# !DEDH#67<#2:9*4#=*.:#-468:08*.# 4*.-*8:#:2#DEDH#67<#2:9*4#=*.:# 3465*/24G.#64*#68G72/1*<;*<#:2# -468:08*.#3465*/24G.#:96:#0.#614*6<"# 96+*#72:#<*10+*4*<#27#:9*04#;261.(# 688*-:*<#/0:907#:9*#07<>.:4"C 688*-:*< /0:907 :9* 07<>.:4" *76=107;#DE#24;670A6:027.#:2#122G#324# *76=107; DE 24;670A6:027. :2 122G 324 !,-*7#.2>48*#J,KKL#5676;*5*7:# 7*/#52<*1.C :221.(#/9089#64*#524*#610;7*<#/0:9# !E9*#;42/07;#07:*4*.:#07#:221.#.>89# -- :90.#6--42689(#96+*#72:#.**7# -- - 6.#B9*3(#I>--*:(#*:8C(#/011#9*1-# .0;7030867:#*7:*4-40.*#564G*:#.964*# .:05>16:*#<*567<#324#,KKM=6.*<# :468:027C 5676;*5*7:
  • 9. vOps. DevOps. DevOps. DevOps. DevO The Rise of a New IT Operations Support Model !"#$%&(#)*+,-.#/011#*+21+*#3425#6#7089*#.:46:*;"#*5-12"*<# ="#164;*#812><#-42+0<*4.#07:2#6#5607.:4*65#.:46:*;"#*5-12"*<# ="#$%?#23#@12=61#$%%%#24;670A6:027.C =" $%? 23 @12=61 $%%% 24;670A6:027. Why DevOps will not emerge: Why DevOps will emerge: !B>1:>461 8967;*. 64* :9* 964<*.: :2 !)*+,-. 0. 72: >.>611" <40+*7 3425 B>1:>461#8967;*.#64*#:9*#964<*.:#:2# )*+,-.#0.#72:#>.>611"#<40+*7#3425# 05-1*5*7:(#67<#)*+,-.#4*F>04*.#6# :9*#:2-#<2/7#67<(#:9>.(#56"#=*# .0;7030867:#4*:907G07;#23#DE# of a New IT524*#*6.01"#688*-:*<#="#DE# The Rise Operations Support Model 2-*46:027.#827+*7:02761#/0.<25C 2-*46:027.#:*65.C !"#$%&(#)*+,-.#/011#*+21+*#3425#6#7089*#.:46:*;"#*5-12"*<# !E9*4*#0.#6#164;*#=2<"#23#/24G#/0:9# !DEDH#67<#2:9*4#=*.:#-468:08*.# ="#164;*#812><#-42+0<*4.#07:2#6#5607.:4*65#.:46:*;"#*5-12"*<# 4*.-*8:#:2#DEDH#67<#2:9*4#=*.:# $%? 23 @12=61 $%%% 24;670A6:027. 3465*/24G.#64*#68G72/1*<;*<#:2# ="#$%?#23#@12=61#$%%%#24;670A6:027.C =" -468:08*.#3465*/24G.#:96:#0.#614*6<"# :296+*#72:#<*10+*4*<#27#:9*04#;261.(# Why DevOps will not emerge: B>1:>461 8967;*. 64* :9* 964<*.: ! ! Why DevOps will emerge: B>1:>461#8967;*.#64*#:9*#964<*.:#:2# )*+,-. 0. 72: >.>611" <40+*7 3425 )*+,-.#0.#72:#>.>611"#<40+*7#3425# 688*-:*<#/0:907#:9*#07<>.:4"C 688*-:*< /0:907 :9* 07<>.:4" 05-1*5*7:(#67<#)*+,-.#4*F>04*.#6#*76=107; DE 24;670A6:027. :2 122G 324 *76=107;#DE#24;670A6:027.#:2#122G#324# :9*#:2-#<2/7#67<(#:9>.(#56"#=*# .0;7030867:#4*:907G07;#23#DE# 524*#*6.01"#688*-:*<#="#DE# 2-*46:027.#827+*7:02761#/0.<25C 7*/#52<*1.C !,-*7#.2>48*#J,KKL#5676;*5*7:# 2-*46:027.#:*65.C E9*4*#0.#6#164;*#=2<"#23#/24G#/0:9# ! ! DEDH#67<#2:9*4#=*.:#-468:08*.# :221.(#/9089#64*#524*#610;7*<#/0:9# !E9*#;42/07;#07:*4*.:#07#:221.#.>89# 4*.-*8:#:2#DEDH#67<#2:9*4#=*.:# 3465*/24G.#64*#68G72/1*<;*<#:2# -468:08*.#3465*/24G.#:96:#0.#614*6<"# 96+*#72:#<*10+*4*<#27#:9*04#;261.(# -- :90.#6--42689(#96+*#72:#.**7# 07<>.:4" 688*-:*<#/0:907#:9*#07<>.:4"C 688*-:*< /0:907 :9* -- 6.#B9*3(#I>--*:(#*:8C(#/011#9*1-# *76=107;#DE#24;670A6:027.#:2#122G#324# *76=107; DE 24;670A6:027. :2 122G 324 - 7*/#52<*1.C .0;7030867:#*7:*4-40.*#564G*:#.964*# :221.(#/9089#64*#524*#610;7*<#/0:9# .:05>16:*#<*567<#324#,KKM=6.*<# ,-*7#.2>48*#J,KKL#5676;*5*7:# ! ! E9*#;42/07;#07:*4*.:#07#:221.#.>89# :468:027C -- :90.#6--42689(#96+*#72:#.**7# .0;7030867:#*7:*4-40.*#564G*:#.964*# 5676;*5*7: -- 6.#B9*3(#I>--*:(#*:8C(#/011#9*1-# .:05>16:*#<*567<#324#,KKM=6.*<# - :468:027C 5676;*5*7:
  • 10. Measuring success... Alignment Quality Cycle Time
  • 11. Know what kind ofbusiness you are
  • 12. What is your tolerance for... Risk Change
  • 13. Consumer-facing applications... "Move fast and break things. Unless you are breaking stuff, you are not moving fast enough." -Mark Zuckerberg
  • 14. Consumer-facing applications... "Move fast and break things. Unless you are breaking stuff, you are not moving fast enough." -Mark Zuckerberg
  • 15. Webtone utilities...
  • 16. Webtone utilities...
  • 17. Webtone utilities...
  • 18. Webtone utilities...
  • 19. Webtone utilities...
  • 20. What do you want...
  • 21. What do you want...
  • 22. What do you want...
  • 23. What do you want...
  • 24. What do you want...
  • 25. What do you want...
  • 26. What do you want...
  • 27. What do you want...
  • 28. CloudsDevOpsContinuous Delivery/DeploymentLean Startup Alignment Quality Cycle Time
  • 29. It all starts in Development
  • 30. It all starts in Development But what do we tell them to do and how do we get them to do it?
  • 31. Share ownership of availability• Developers must wear pagers• Incident command training so everyone knows their roles• Notification mechanism? Access provisioning?
  • 32. Non-functional requirements are first class citizens• All of the “-ility” requirements• As much a part of your service as the businessʼs functional requirements• Manage through same process• Make sure they are on the schedule• Donʼt over commit
  • 33. Strive for parity between dev & prod• Develop test data fixtures for all environments• Implement Mock Services for major infrastructure pieces for Developer user.• Continuous integration means integrate early• Use the deployment, config, and packaging tools in dev
  • 34. Push config management discipline back to Dev• Dev is about creating variation, Ops is about eliminating variation• Augment deployment toolchain to support the variation• Do developers use the tools?• Do developers like them?• Devʼs need config too• Accept config contributions and patches from dev
  • 35. Packaging... itʼs not just for the OS • If you want to be a high performing web operations organization, you need to take change management seriously. • Strict versioning is needed, and package managers do that. • Itʼs about being idempotent. • Transfer packaging responsibility to dev • Define the packaging constructs you will support http://webscalebook.com
  • 36. Config is code• if itʼs code it needs to be managed like code...• Should be transparent and identical SDLC in both dev and ops• Avoid or eliminate asymetric release processes
  • 37. Tailor release artifacts to roles• “Small teams make better software”• Make sure one team getting stuck doesnʼt prevent another team to release (org coupling)• Large codebases suffer software entropy effects• Build an infrastructure that can reliably manage lots of smaller artifacts• Org conflict is a good time to suggest breaking up a codebase into separate concerns
  • 38. Standard management vocabulary• Consistent and expected management behavior• Across components and releases• “start, stop, status, update, install...”
  • 39. Rollback• Rollback that works• No, really works• Seriously, really really works• You say it will work? Test it right now
  • 40. Standard metrics abstractions• Dev surface metrics to Ops• Avoid the back and forth conversations• Use a standard framework• Use standard types (gague, counter, timer...)• Ops knows what to expect and how to visualize https://github.com/codahale/metrics
  • 41. Push test ownership to the edges• QA is “Quality Assurance”• QA writing tests = bottleneck• QA writing tests = avoiding responsibility• QA is architecture / tooling / assurance• Test Driven Development• Test Driven Operations (yes, you too)• Bottom line: Everyone owns quality.
  • 42. Test outside of the boxCrowd Test Simulate
  • 43. Simulation AUT
  • 44. Build the right lifecycle... Alignment Quality Cycle Time
  • 45. Continuous Delivery...
  • 46. Continuous Delivery... Deployment... Automated Automated Testing Release
  • 47. Donʼt be so dogmatic
  • 48. Decoupled Release Model Example Code, Config, Control Code, Config, Control Code, Config, Control "Check- Operations in" "Check- Rundeck 1 in" "Check- Source 1 in" Repo Source 1 Monitor Rundeck Chef 1 "Check- in" "Check- Source Repo "Auto-Deploy Repo Instance Instance Develop in" "Check- New App Develop Package Manifest in" "Check- Jobs & Infrastructure CI Server Develop Package Manifest Package Manifest out" "Check- out" "Check- 2 Adhoc 1 Instance Source Repo "Verify" 2 out" Operate Commands "Verify" 2 2 Watch "Verify" Code Monitoring Watch Publish Code & Testing Staging Results Watch Build Rundeck Release Publish Code ResultsContinuous Build Repo Publish Release Chef Deployment ResultsContinuous Build Repo Release Continuous Loop Integration Deployment Unit RightScale Testing Tests "Promote" ARelease Repo Prod Integration Deployment Unit Loop "Promote" Repo A Prod Release Testing Loop Tests Repo Dev Test Automatic Integration Deployment Testing Unit Tests 3 "Promote" A Dev Test Automatic Deployment 3 Dev Test Dev Test Automatic Deployment 3 Bootstrap Develop Release Operate Release Repo Release A Release Repo Repo A 1 Repo B 4 Promote Prod Release Repo B Audit Release Job Repo Release Repo Repo C C "Determine what packages "Promote release" and versions are available" 2 "Check -in" Souce Repo Hudson Prod Release Repo Release Package Manifest 5 Deploy Job "yum install" Release app code Repo A Release Rundeck services Repo 3 Validate Release OS B Release Chef Repo RightScale C "Check for conflicts and Sandbox Prod run validation scripts"
  • 49. Decoupled Release Model Example Code, Config, Control Code, Config, Control Code, Config, Control "Check- Operations in" "Check- Rundeck 1 in" "Check- Source 1 in" Repo Source 1 Monitor Rundeck Chef 1 "Check- in" "Check- Source Repo "Auto-Deploy Repo Instance Instance Develop in" "Check- New App Develop Package Manifest in" "Check- Jobs & Infrastructure CI Server Develop Package Manifest Package Manifest out" "Check- out" "Check- 2 Adhoc 1 Instance Source Repo "Verify" 2 out" Operate Commands "Verify" 2 2 Watch "Verify" Code Monitoring Watch Publish Code & Testing Staging Results Watch Build Rundeck Release Publish Code ResultsContinuous Build Repo Publish Release Chef Deployment ResultsContinuous Build Repo Release Continuous Loop Integration Deployment Unit RightScale Testing Tests "Promote" ARelease Repo Prod Integration Deployment Unit Loop "Promote" Repo A Prod Release Testing Loop Tests Repo Dev Test Automatic Integration Deployment Testing Unit Tests 3 "Promote" A Dev Test Automatic Deployment 3 Dev Test Dev Test Automatic Deployment 3 Bootstrap Develop Release Operate Release Repo Release A Release Repo Repo A 1 Repo B 4 Promote Prod Release Repo B Audit Release Job Repo Release Repo Repo C C "Determine what packages "Promote release" and versions are available" 2 "Check -in" Souce Repo Hudson Prod Release Repo Release Package Manifest 5 Deploy Job "yum install" Release app code Repo A Release Rundeck services Repo 3 Validate Release OS B Release Chef Repo RightScale C "Check for conflicts and Sandbox Prod run validation scripts"
  • 50. Bootstrap Operations Rundeck Rundeck Chef "Auto-Deploy Instance Instance New App Infrastructure CI Server Source 1 Instance Repo Monitoring & Testing Staging Release Repo Prod Release Repo Dev Test
  • 51. Decoupled Release Model Example Code, Config, Control Code, Config, Control Code, Config, Control "Check- Operations in" "Check- Rundeck 1 in" "Check- Source 1 in" Repo Source 1 Monitor Rundeck Chef 1 "Check- in" "Check- Source Repo "Auto-Deploy Repo Instance Instance Develop in" "Check- New App Develop Package Manifest in" "Check- Jobs & Infrastructure CI Server Develop Package Manifest Package Manifest out" "Check- out" "Check- 2 Adhoc 1 Instance Source Repo "Verify" 2 out" Operate Commands "Verify" 2 2 Watch "Verify" Code Monitoring Watch Publish Code & Testing Staging Results Watch Build Rundeck Release Publish Code ResultsContinuous Build Repo Publish Release Chef Deployment ResultsContinuous Build Repo Release Continuous Loop Integration Deployment Unit RightScale Testing Tests "Promote" ARelease Repo Prod Integration Deployment Unit Loop "Promote" Repo A Prod Release Testing Loop Tests Repo Dev Test Automatic Integration Deployment Testing Unit Tests 3 "Promote" A Dev Test Automatic Deployment 3 Dev Test Dev Test Automatic Deployment 3 Bootstrap Develop Release Operate Release Repo Release A Release Repo Repo A 1 Repo B 4 Promote Prod Release Repo B Audit Release Job Repo Release Repo Repo C C "Determine what packages "Promote release" and versions are available" 2 "Check -in" Souce Repo Hudson Prod Release Repo Release Package Manifest 5 Deploy Job "yum install" Release app code Repo A Release Rundeck services Repo 3 Validate Release OS B Release Chef Repo RightScale C "Check for conflicts and Sandbox Prod run validation scripts"
  • 52. Develop Code, Config, Control "Check- in" 1 Source Repo "Check- in" Develop Package Manifest "Check- out" "Verify" 2 Watch Code Publish Results Build Continuous Release Deployment Repo Integration Loop Unit Testing Tests "Promote" A Dev Test Automatic Deployment 3
  • 53. Decoupled Release Model Example Code, Config, Control Code, Config, Control Code, Config, Control "Check- Operations in" "Check- Rundeck 1 in" "Check- Source 1 in" Repo Source 1 Monitor Rundeck Chef 1 "Check- in" "Check- Source Repo "Auto-Deploy Repo Instance Instance Develop in" "Check- New App Develop Package Manifest in" "Check- Jobs & Infrastructure CI Server Develop Package Manifest Package Manifest out" "Check- out" "Check- 2 Adhoc 1 Instance Source Repo "Verify" 2 out" Operate Commands "Verify" 2 2 Watch "Verify" Code Monitoring Watch Publish Code & Testing Staging Results Watch Build Rundeck Release Publish Code ResultsContinuous Build Repo Publish Release Chef Deployment ResultsContinuous Build Repo Release Continuous Loop Integration Deployment Unit RightScale Testing Tests "Promote" ARelease Repo Prod Integration Deployment Unit Loop "Promote" Repo A Prod Release Testing Loop Tests Repo Dev Test Automatic Integration Deployment Testing Unit Tests 3 "Promote" A Dev Test Automatic Deployment 3 Dev Test Dev Test Automatic Deployment 3 Bootstrap Develop Release Operate Release Repo Release A Release Repo Repo A 1 Repo B 4 Promote Prod Release Repo B Audit Release Job Repo Release Repo Repo C C "Determine what packages "Promote release" and versions are available" 2 "Check -in" Souce Repo Hudson Prod Release Repo Release Package Manifest 5 Deploy Job "yum install" Release app code Repo A Release Rundeck services Repo 3 Validate Release OS B Release Chef Repo RightScale C "Check for conflicts and Sandbox Prod run validation scripts"
  • 54. Release Release Repo Release A Release Repo Repo A 1 Repo B 4 Promote Prod Release Repo B Audit Release Job Repo Release Repo Repo C C "Determine what packages "Promote release" and versions are available" 2 "Check -in" Souce Repo Hudson Prod Release Repoease Package Manifest 5 Deploy Job "yum install" Release app code Repo A Release Rundeck services Repo 3 Validate Release OS B Release Chef Repo RightScale C "Check for conflicts and Sandbox Prod run validation scripts"
  • 55. Decoupled Release Model Example Code, Config, Control Code, Config, Control Code, Config, Control "Check- Operations in" "Check- Rundeck 1 in" "Check- Source 1 in" Repo Source 1 Monitor Rundeck Chef 1 "Check- in" "Check- Source Repo "Auto-Deploy Repo Instance Instance Develop in" "Check- New App Develop Package Manifest in" "Check- Jobs & Infrastructure CI Server Develop Package Manifest Package Manifest out" "Check- out" "Check- 2 Adhoc 1 Instance Source Repo "Verify" 2 out" Operate Commands "Verify" 2 2 Watch "Verify" Code Monitoring Watch Publish Code & Testing Staging Results Watch Build Rundeck Release Publish Code ResultsContinuous Build Repo Publish Release Chef Deployment ResultsContinuous Build Repo Release Continuous Loop Integration Deployment Unit RightScale Testing Tests "Promote" ARelease Repo Prod Integration Deployment Unit Loop "Promote" Repo A Prod Release Testing Loop Tests Repo Dev Test Automatic Integration Deployment Testing Unit Tests 3 "Promote" A Dev Test Automatic Deployment 3 Dev Test Dev Test Automatic Deployment 3 Bootstrap Develop Release Operate Release Repo Release A Release Repo Repo A 1 Repo B 4 Promote Prod Release Repo B Audit Release Job Repo Release Repo Repo C C "Determine what packages "Promote release" and versions are available" 2 "Check -in" Souce Repo Hudson Prod Release Repo Release Package Manifest 5 Deploy Job "yum install" Release app code Repo A Release Rundeck services Repo 3 Validate Release OS B Release Chef Repo RightScale C "Check for conflicts and Sandbox Prod run validation scripts"
  • 56. Operate 1 Monitor Jobs & 2 Adhoc Commands Operate Rundeck Chef RightScale Prod
  • 57. Decoupled Release Model Example Code, Config, Control Code, Config, Control Code, Config, Control "Check- Operations in" "Check- Rundeck 1 in" "Check- Source 1 in" Repo Source 1 Monitor Rundeck Chef 1 "Check- in" "Check- Source Repo "Auto-Deploy Repo Instance Instance Develop in" "Check- New App Develop Package Manifest in" "Check- Jobs & Infrastructure CI Server Develop Package Manifest Package Manifest out" "Check- out" "Check- 2 Adhoc 1 Instance Source Repo "Verify" 2 out" Operate Commands "Verify" 2 2 Watch "Verify" Code Monitoring Watch Publish Code & Testing Staging Results Watch Build Rundeck Release Publish Code ResultsContinuous Build Repo Publish Release Chef Deployment ResultsContinuous Build Repo Release Continuous Loop Integration Deployment Unit RightScale Testing Tests "Promote" ARelease Repo Prod Integration Deployment Unit Loop "Promote" Repo A Prod Release Testing Loop Tests Repo Dev Test Automatic Integration Deployment Testing Unit Tests 3 "Promote" A Dev Test Automatic Deployment 3 Dev Test Dev Test Automatic Deployment 3 Bootstrap Develop Release Operate Release Repo Release A Release Repo Repo A 1 Repo B 4 Promote Prod Release Repo B Audit Release Job Repo Release Repo Repo C C "Determine what packages "Promote release" and versions are available" 2 "Check -in" Souce Repo Hudson Prod Release Repo Release Package Manifest 5 Deploy Job "yum install" Release app code Repo A Release Rundeck services Repo 3 Validate Release OS B Release Chef Repo RightScale C "Check for conflicts and Sandbox Prod run validation scripts"

×