Production Begins in
        Development



Lee Thompson       Damon Edwards
• CTO Travel/Transportation
                 Products, Hewlett Packard


               • Former Chief Technologist,
                 E*TRADE Financial


               • Co-writer of “Fully Automated
Lee Thompson     Provisioning” whitepaper
  @stagr_lee
• Co-founder DTO Solutions
                • DevOps Days organizer
                • Co-writer of “Fully Automated
                  Provisioning” whitepaper
Damon Edwards
@damonedwards
• Co-founder DTO Solutions
                • DevOps Days organizer
                • Co-writer of “Fully Automated
                  Provisioning” whitepaper
Damon Edwards
@damonedwards
• Consumer-facing web applications
• Webtone utilities
Showing us what is possible...

         Clouds



         Continuous Delivery/Deployment




         Lean Startup
What we want more of...




               DevOps!
vOps. DevOps. DevOps. DevOps. DevO




            The Rise of a New IT Operations
            Support Model
            !"#$%&'(#)*+,-.#/011#*+21+*#3425#6#7089*#.:46:*;"#*5-12"*<#
            ="#164;*#812><#-42+0<*4.#07:2#6#5607.:4*65#.:46:*;"#*5-12"*<#
                        ="#$%?#23#@12=61#$%%%#24;670A6:027.C
                        =" $%? 23 @12=61 $%%% 24;670A6:027.
          Why DevOps will not emerge:           Why DevOps will emerge:
          !B>1:>461 8967;*. 64* :9* 964<*.: :2 !)*+,-. 0. 72: >.>611" <40+*7 3425
           B>1:>461#8967;*.#64*#:9*#964<*.:#:2# )*+,-.#0.#72:#>.>611"#<40+*7#3425#
           05-1*5*7:(#67<#)*+,-.#4*F>04*.#6#     :9*#:2-#<2/7#67<(#:9>.(#56"#=*#
           .0;7030867:#4*:907G07;#23#DE#         524*#*6.01"#688*-:*<#="#DE#
           2-*46:027.#827+*7:02761#/0.<25C       2-*46:027.#:*65.C
          !E9*4*#0.#6#164;*#=2<"#23#/24G#/0:9#  !DEDH#67<#2:9*4#=*.:#-468:08*.#
           4*.-*8:#:2#DEDH#67<#2:9*4#=*.:#       3465*/24G.#64*#68G72/1*<;*<#:2#
           -468:08*.#3465*/24G.#:96:#0.#614*6<"# 96+*#72:#<*10+*4*<#27#:9*04#;261.(#
           688*-:*<#/0:907#:9*#07<>.:4"C
           688*-:*< /0:907 :9* 07<>.:4"          *76=107;#DE#24;670A6:027.#:2#122G#324#
                                                 *76=107; DE 24;670A6:027. :2 122G 324
          !,-*7#.2>48*#J,KKL#5676;*5*7:#         7*/#52<*1.C
           :221.(#/9089#64*#524*#610;7*<#/0:9# !E9*#;42/07;#07:*4*.:#07#:221.#.>89#
                 --
           :90.#6--42689(#96+*#72:#.**7#                     --                 -
                                                 6.#B9*3(#I>--*:(#*:8C(#/011#9*1-#
           .0;7030867:#*7:*4-40.*#564G*:#.964*#  .:05>16:*#<*567<#324#,KKM=6.*<#
           :468:027C                             5676;*5*7:
vOps. DevOps. DevOps. DevOps. DevO
       The Rise of a New IT Operations
       Support Model
       !"#$%&'(#)*+,-.#/011#*+21+*#3425#6#7089*#.:46:*;"#*5-12"*<#
       ="#164;*#812><#-42+0<*4.#07:2#6#5607.:4*65#.:46:*;"#*5-12"*<#
                   ="#$%?#23#@12=61#$%%%#24;670A6:027.C
                   =" $%? 23 @12=61 $%%% 24;670A6:027.
    Why DevOps will not emerge:                             Why DevOps will emerge:
    !B>1:>461 8967;*. 64* :9* 964<*.: :2 !)*+,-. 0. 72: >.>611" <40+*7 3425
     B>1:>461#8967;*.#64*#:9*#964<*.:#:2# )*+,-.#0.#72:#>.>611"#<40+*7#3425#
     05-1*5*7:(#67<#)*+,-.#4*F>04*.#6#                          :9*#:2-#<2/7#67<(#:9>.(#56"#=*#
     .0;7030867:#4*:907G07;#23#DE# of a New IT524*#*6.01"#688*-:*<#="#DE#
                            The Rise                             Operations
                            Support Model
     2-*46:027.#827+*7:02761#/0.<25C                            2-*46:027.#:*65.C
                             !"#$%&'(#)*+,-.#/011#*+21+*#3425#6#7089*#.:46:*;"#*5-12"*<#
    !E9*4*#0.#6#164;*#=2<"#23#/24G#/0:9#                    !DEDH#67<#2:9*4#=*.:#-468:08*.#
                             ="#164;*#812><#-42+0<*4.#07:2#6#5607.:4*65#.:46:*;"#*5-12"*<#
     4*.-*8:#:2#DEDH#67<#2:9*4#=*.:# $%? 23 @12=61 $%%% 24;670A6:027.
                                                                3465*/24G.#64*#68G72/1*<;*<#:2#
                                             ="#$%?#23#@12=61#$%%%#24;670A6:027.C
                                             ="

     -468:08*.#3465*/24G.#:96:#0.#614*6<"# :296+*#72:#<*10+*4*<#27#:9*04#;261.(#
                          Why DevOps will not emerge:
                           B>1:>461 8967;*. 64* :9* 964<*.:
                                !                          !
                                                                  Why DevOps will emerge:
                           B>1:>461#8967;*.#64*#:9*#964<*.:#:2# )*+,-. 0. 72: >.>611" <40+*7 3425
                                                                   )*+,-.#0.#72:#>.>611"#<40+*7#3425#
     688*-:*<#/0:907#:9*#07<>.:4"C
     688*-:*< /0:907 :9* 07<>.:4"
                           05-1*5*7:(#67<#)*+,-.#4*F>04*.#6#*76=107; DE 24;670A6:027. :2 122G 324
                                                                *76=107;#DE#24;670A6:027.#:2#122G#324#
                                                                   :9*#:2-#<2/7#67<(#:9>.(#56"#=*#
                           .0;7030867:#4*:907G07;#23#DE#           524*#*6.01"#688*-:*<#="#DE#
                           2-*46:027.#827+*7:02761#/0.<25C 7*/#52<*1.C
    !,-*7#.2>48*#J,KKL#5676;*5*7:#                                 2-*46:027.#:*65.C
                           E9*4*#0.#6#164;*#=2<"#23#/24G#/0:9#
                                !                          !       DEDH#67<#2:9*4#=*.:#-468:08*.#
     :221.(#/9089#64*#524*#610;7*<#/0:9# !E9*#;42/07;#07:*4*.:#07#:221.#.>89#
                           4*.-*8:#:2#DEDH#67<#2:9*4#=*.:#         3465*/24G.#64*#68G72/1*<;*<#:2#
                           -468:08*.#3465*/24G.#:96:#0.#614*6<"# 96+*#72:#<*10+*4*<#27#:9*04#;261.(#
           --
     :90.#6--42689(#96+*#72:#.**7# 07<>.:4"
                           688*-:*<#/0:907#:9*#07<>.:4"C
                           688*-:*< /0:907 :9*                                       --
                                                                6.#B9*3(#I>--*:(#*:8C(#/011#9*1-#
                                                                   *76=107;#DE#24;670A6:027.#:2#122G#324#
                                                                   *76=107; DE 24;670A6:027. :2 122G 324  -
                                                                   7*/#52<*1.C
     .0;7030867:#*7:*4-40.*#564G*:#.964*#
                           :221.(#/9089#64*#524*#610;7*<#/0:9# .:05>16:*#<*567<#324#,KKM=6.*<#
                           ,-*7#.2>48*#J,KKL#5676;*5*7:#
                                !
                                                           !       E9*#;42/07;#07:*4*.:#07#:221.#.>89#
     :468:027C                   --
                           :90.#6--42689(#96+*#72:#.**7#
                           .0;7030867:#*7:*4-40.*#564G*:#.964*#
                                                                5676;*5*7:     --
                                                                   6.#B9*3(#I>--*:(#*:8C(#/011#9*1-#
                                                                   .:05>16:*#<*567<#324#,KKM=6.*<#
                                                                                                  -

                                 :468:027C                   5676;*5*7:
Measuring success...

         Alignment

         Quality

         Cycle Time
Know what kind of
business you are
What is your tolerance for...


             Risk
             Change
Consumer-facing applications...
            "Move fast and break things.
           Unless you are breaking stuff,
          you are not moving fast enough."
                 -Mark Zuckerberg
Consumer-facing applications...
            "Move fast and break things.
           Unless you are breaking stuff,
          you are not moving fast enough."
                 -Mark Zuckerberg
Webtone utilities...
Webtone utilities...
Webtone utilities...
Webtone utilities...
Webtone utilities...
What do you want...
What do you want...
What do you want...
What do you want...
What do you want...
What do you want...
What do you want...
What do you want...
Clouds


DevOps


Continuous Delivery/Deployment


Lean Startup




   Alignment

   Quality

   Cycle Time
It all starts in Development
It all starts in Development
   But what do we tell them to do
  and how do we get them to do it?
Share ownership of availability

• Developers must wear pagers
• Incident command training so everyone
  knows their roles

• Notification mechanism?
  Access provisioning?
Non-functional requirements are
       first class citizens

• All of the “-ility” requirements
• As much a part of your service as the
  businessʼs functional requirements

• Manage through same process
• Make sure they are on the schedule
• Donʼt over commit
Strive for parity between dev & prod

• Develop test data fixtures for all environments
• Implement Mock Services for major
  infrastructure pieces for Developer user.

• Continuous integration means integrate early
• Use the deployment, config, and packaging
  tools in dev
Push config management
  discipline back to Dev
• Dev is about creating variation, Ops is
  about eliminating variation
• Augment deployment toolchain to support
  the variation
• Do developers use the tools?
• Do developers like them?
• Devʼs need config too
• Accept config contributions and patches
  from dev
Packaging... itʼs not just for the OS
  • If you want to be a high performing web
    operations organization, you need to take
    change management seriously.

  • Strict versioning is needed, and package
    managers do that.

  • Itʼs about being idempotent.
  • Transfer packaging responsibility to dev
  • Define the packaging constructs you will
    support
                        http://webscalebook.com
Config is code
• if itʼs code it needs to be managed like
  code...

• Should be transparent and identical SDLC
  in both dev and ops

• Avoid or eliminate asymetric release
  processes
Tailor release artifacts to roles
• “Small teams make better software”
• Make sure one team getting stuck doesnʼt
  prevent another team to release (org
  coupling)
• Large codebases suffer software entropy
  effects
• Build an infrastructure that can reliably
  manage lots of smaller artifacts
• Org conflict is a good time to suggest
  breaking up a codebase into separate
  concerns
Standard management vocabulary

• Consistent and expected management
  behavior

• Across components and releases
• “start, stop, status, update, install...”
Rollback
• Rollback that works
• No, really works
• Seriously, really really works
• You say it will work? Test it right now
Standard metrics abstractions
• Dev surface metrics to Ops
• Avoid the back and forth conversations
• Use a standard framework
• Use standard types (gague, counter,
  timer...)

• Ops knows what to expect and how to
  visualize

              https://github.com/codahale/metrics
Push test ownership to the edges
• QA is “Quality Assurance”
• QA writing tests = bottleneck
• QA writing tests = avoiding responsibility
• QA is architecture / tooling / assurance
• Test Driven Development
• Test Driven Operations (yes, you too)
• Bottom line: Everyone owns quality.
Test outside of the box




Crowd Test       Simulate
Simulation




             AUT
Build the right lifecycle...

            Alignment

            Quality

            Cycle Time
Continuous Delivery...
Continuous Delivery...
           Deployment...
                 Automated   Automated
                  Testing     Release
Donʼt be so dogmatic
Decoupled Release Model Example
                                                                           Code, Config, Control
                                                                            Code, Config, Control
                                                                              Code, Config, Control

                                                                                                             "Check-
     Operations                                                                                                in"
                                                                                                               "Check-
      Rundeck
                                                                                                         1        in"
                                                                                                                   "Check-               Source
                                                                                                          1           in"                 Repo
                                                                                                                                           Source                                         1        Monitor
                       Rundeck       Chef
                                                                                                              1
                                                                                                             "Check-
                                                                                                               in"
                                                                                                               "Check-
                                                                                                                                             Source
                                                                                                                                            Repo
    "Auto-Deploy                                                                                                                              Repo
                       Instance    Instance            Develop                                                    in"
                                                                                                                   "Check-
      New App                                           Develop              Package Manifest
                                                                                                                      in"        "Check-                                                          Jobs &
    Infrastructure
                      CI Server
                                                          Develop              Package Manifest
                                                                                 Package Manifest
                                                                                                                                   out"
                                                                                                                                   "Check-
                                                                                                                                     out"
                                                                                                                                      "Check-
                                                                                                                                                                                          2       Adhoc
                  1   Instance     Source
                                    Repo
                                                          "Verify"
                                                                       2                                                                out"                                   Operate
                                                                                                                                                                                                Commands
                                                            "Verify"
                                                                        2
                                                                             2
                                                                                                         Watch
                                                                "Verify"                                 Code
                      Monitoring                                                                           Watch
                                                                                        Publish             Code
                      & Testing    Staging                                              Results                Watch
                                                                                                                Build                                                                             Rundeck
                                   Release                                                Publish              Code
                                                                                          ResultsContinuous       Build
                                    Repo                                                      Publish                                                    Release                                        Chef
                                                                                                 Deployment
                                                                                              ResultsContinuous       Build
                                                                                                                                                          Repo
                                                                                                                                                          Release
                                                                                                           Continuous
                                                                                                         Loop
                                                                                        Integration Deployment Unit                                                                             RightScale
                                                                                          Testing                       Tests             "Promote"         ARelease
                                                                                                                                                            Repo
                                    Prod                                                   Integration     Deployment Unit
                                                                                                            Loop
                                                                                                                                            "Promote"         Repo
                                                                                                                                                              A                                                              Prod
                                   Release                                                   Testing           Loop        Tests
                                    Repo
                                                       Dev Test
                                                                                                        Automatic
                                                                                               Integration
                                                                                                       Deployment
                                                                                                 Testing
                                                                                                                               Unit
                                                                                                                              Tests             3
                                                                                                                                               "Promote"        A
                      Dev Test                                                                             Automatic
                                                                                                          Deployment                             3
                                                        Dev Test
                                                          Dev Test
                                                                                                              Automatic
                                                                                                             Deployment                              3


                  Bootstrap                                           Develop                                                                   Release                                   Operate

                                                                                               Release
                                                                                                Repo                                                                                          Release
                                                                                                  A        Release                                                                             Repo
                                                                                                            Repo                                                                                 A

                                              1         Repo
                                                                                                              B
                                                                                                                                                  4          Promote            Prod
                                                                                                                                                                                                         Release
                                                                                                                                                                                                          Repo
                                                                                                                                                                                                            B
                                                        Audit                                                        Release
                                                                                                                                                               Job              Repo                               Release
                                                                                                                      Repo                                                                                          Repo
                                                                                                                        C                                                                                             C
                                                  "Determine what packages                                                                               "Promote release"
                                                  and versions are available"



                                              2                            "Check
                                                                             -in"
                                                                                               Souce
                                                                                               Repo
                                                                                                                  Hudson
                                                                                                                                                                                                     Prod
                                                                                                                                                                                                    Release
                                                                                                                                                                                                     Repo

                              Release
                                                    Package Manifest                                                                              5          Deploy
                                                                                                                                                              Job
                                                                                                                                                                                                 "yum install"

                                                                                                               Release
                                                                                    app code
                                                                                                                Repo
                                                                                                                  A
                                                                                                                           Release
                                                                                                                                                                             Rundeck
                                                                                    services                                Repo
                                              3        Validate
                                                       Release
                                                                                      OS                                      B
                                                                                                                                      Release
                                                                                                                                                                               Chef

                                                                                                                                       Repo                                  RightScale
                                                                                                                                         C
                                                  "Check for conflicts and
                                                                                 Sandbox
                                                                                                                                                                                                   Prod
                                                   run validation scripts"
Decoupled Release Model Example
                                                                           Code, Config, Control
                                                                            Code, Config, Control
                                                                              Code, Config, Control

                                                                                                             "Check-
     Operations                                                                                                in"
                                                                                                               "Check-
      Rundeck
                                                                                                         1        in"
                                                                                                                   "Check-               Source
                                                                                                          1           in"                 Repo
                                                                                                                                           Source                                         1        Monitor
                       Rundeck       Chef
                                                                                                              1
                                                                                                             "Check-
                                                                                                               in"
                                                                                                               "Check-
                                                                                                                                             Source
                                                                                                                                            Repo
    "Auto-Deploy                                                                                                                              Repo
                       Instance    Instance            Develop                                                    in"
                                                                                                                   "Check-
      New App                                           Develop              Package Manifest
                                                                                                                      in"        "Check-                                                          Jobs &
    Infrastructure
                      CI Server
                                                          Develop              Package Manifest
                                                                                 Package Manifest
                                                                                                                                   out"
                                                                                                                                   "Check-
                                                                                                                                     out"
                                                                                                                                      "Check-
                                                                                                                                                                                          2       Adhoc
                  1   Instance     Source
                                    Repo
                                                          "Verify"
                                                                       2                                                                out"                                   Operate
                                                                                                                                                                                                Commands
                                                            "Verify"
                                                                        2
                                                                             2
                                                                                                         Watch
                                                                "Verify"                                 Code
                      Monitoring                                                                           Watch
                                                                                        Publish             Code
                      & Testing    Staging                                              Results                Watch
                                                                                                                Build                                                                             Rundeck
                                   Release                                                Publish              Code
                                                                                          ResultsContinuous       Build
                                    Repo                                                      Publish                                                    Release                                        Chef
                                                                                                 Deployment
                                                                                              ResultsContinuous       Build
                                                                                                                                                          Repo
                                                                                                                                                          Release
                                                                                                           Continuous
                                                                                                         Loop
                                                                                        Integration Deployment Unit                                                                             RightScale
                                                                                          Testing                       Tests             "Promote"         ARelease
                                                                                                                                                            Repo
                                    Prod                                                   Integration     Deployment Unit
                                                                                                            Loop
                                                                                                                                            "Promote"         Repo
                                                                                                                                                              A                                                              Prod
                                   Release                                                   Testing           Loop        Tests
                                    Repo
                                                       Dev Test
                                                                                                        Automatic
                                                                                               Integration
                                                                                                       Deployment
                                                                                                 Testing
                                                                                                                               Unit
                                                                                                                              Tests             3
                                                                                                                                               "Promote"        A
                      Dev Test                                                                             Automatic
                                                                                                          Deployment                             3
                                                        Dev Test
                                                          Dev Test
                                                                                                              Automatic
                                                                                                             Deployment                              3


                  Bootstrap                                           Develop                                                                   Release                                   Operate

                                                                                               Release
                                                                                                Repo                                                                                          Release
                                                                                                  A        Release                                                                             Repo
                                                                                                            Repo                                                                                 A

                                              1         Repo
                                                                                                              B
                                                                                                                                                  4          Promote            Prod
                                                                                                                                                                                                         Release
                                                                                                                                                                                                          Repo
                                                                                                                                                                                                            B
                                                        Audit                                                        Release
                                                                                                                                                               Job              Repo                               Release
                                                                                                                      Repo                                                                                          Repo
                                                                                                                        C                                                                                             C
                                                  "Determine what packages                                                                               "Promote release"
                                                  and versions are available"



                                              2                            "Check
                                                                             -in"
                                                                                               Souce
                                                                                               Repo
                                                                                                                  Hudson
                                                                                                                                                                                                     Prod
                                                                                                                                                                                                    Release
                                                                                                                                                                                                     Repo

                              Release
                                                    Package Manifest                                                                              5          Deploy
                                                                                                                                                              Job
                                                                                                                                                                                                 "yum install"

                                                                                                               Release
                                                                                    app code
                                                                                                                Repo
                                                                                                                  A
                                                                                                                           Release
                                                                                                                                                                             Rundeck
                                                                                    services                                Repo
                                              3        Validate
                                                       Release
                                                                                      OS                                      B
                                                                                                                                      Release
                                                                                                                                                                               Chef

                                                                                                                                       Repo                                  RightScale
                                                                                                                                         C
                                                  "Check for conflicts and
                                                                                 Sandbox
                                                                                                                                                                                                   Prod
                                                   run validation scripts"
Bootstrap


       Operations
        Rundeck



                         Rundeck       Chef
      "Auto-Deploy       Instance    Instance
        New App
      Infrastructure
                        CI Server
                                     Source
                    1   Instance
                                      Repo

                        Monitoring
                        & Testing    Staging
                                     Release
                                      Repo


                                      Prod
                                     Release
                                      Repo
                        Dev Test
Decoupled Release Model Example
                                                                           Code, Config, Control
                                                                            Code, Config, Control
                                                                              Code, Config, Control

                                                                                                             "Check-
     Operations                                                                                                in"
                                                                                                               "Check-
      Rundeck
                                                                                                         1        in"
                                                                                                                   "Check-               Source
                                                                                                          1           in"                 Repo
                                                                                                                                           Source                                         1        Monitor
                       Rundeck       Chef
                                                                                                              1
                                                                                                             "Check-
                                                                                                               in"
                                                                                                               "Check-
                                                                                                                                             Source
                                                                                                                                            Repo
    "Auto-Deploy                                                                                                                              Repo
                       Instance    Instance            Develop                                                    in"
                                                                                                                   "Check-
      New App                                           Develop              Package Manifest
                                                                                                                      in"        "Check-                                                          Jobs &
    Infrastructure
                      CI Server
                                                          Develop              Package Manifest
                                                                                 Package Manifest
                                                                                                                                   out"
                                                                                                                                   "Check-
                                                                                                                                     out"
                                                                                                                                      "Check-
                                                                                                                                                                                          2       Adhoc
                  1   Instance     Source
                                    Repo
                                                          "Verify"
                                                                       2                                                                out"                                   Operate
                                                                                                                                                                                                Commands
                                                            "Verify"
                                                                        2
                                                                             2
                                                                                                         Watch
                                                                "Verify"                                 Code
                      Monitoring                                                                           Watch
                                                                                        Publish             Code
                      & Testing    Staging                                              Results                Watch
                                                                                                                Build                                                                             Rundeck
                                   Release                                                Publish              Code
                                                                                          ResultsContinuous       Build
                                    Repo                                                      Publish                                                    Release                                        Chef
                                                                                                 Deployment
                                                                                              ResultsContinuous       Build
                                                                                                                                                          Repo
                                                                                                                                                          Release
                                                                                                           Continuous
                                                                                                         Loop
                                                                                        Integration Deployment Unit                                                                             RightScale
                                                                                          Testing                       Tests             "Promote"         ARelease
                                                                                                                                                            Repo
                                    Prod                                                   Integration     Deployment Unit
                                                                                                            Loop
                                                                                                                                            "Promote"         Repo
                                                                                                                                                              A                                                              Prod
                                   Release                                                   Testing           Loop        Tests
                                    Repo
                                                       Dev Test
                                                                                                        Automatic
                                                                                               Integration
                                                                                                       Deployment
                                                                                                 Testing
                                                                                                                               Unit
                                                                                                                              Tests             3
                                                                                                                                               "Promote"        A
                      Dev Test                                                                             Automatic
                                                                                                          Deployment                             3
                                                        Dev Test
                                                          Dev Test
                                                                                                              Automatic
                                                                                                             Deployment                              3


                  Bootstrap                                           Develop                                                                   Release                                   Operate

                                                                                               Release
                                                                                                Repo                                                                                          Release
                                                                                                  A        Release                                                                             Repo
                                                                                                            Repo                                                                                 A

                                              1         Repo
                                                                                                              B
                                                                                                                                                  4          Promote            Prod
                                                                                                                                                                                                         Release
                                                                                                                                                                                                          Repo
                                                                                                                                                                                                            B
                                                        Audit                                                        Release
                                                                                                                                                               Job              Repo                               Release
                                                                                                                      Repo                                                                                          Repo
                                                                                                                        C                                                                                             C
                                                  "Determine what packages                                                                               "Promote release"
                                                  and versions are available"



                                              2                            "Check
                                                                             -in"
                                                                                               Souce
                                                                                               Repo
                                                                                                                  Hudson
                                                                                                                                                                                                     Prod
                                                                                                                                                                                                    Release
                                                                                                                                                                                                     Repo

                              Release
                                                    Package Manifest                                                                              5          Deploy
                                                                                                                                                              Job
                                                                                                                                                                                                 "yum install"

                                                                                                               Release
                                                                                    app code
                                                                                                                Repo
                                                                                                                  A
                                                                                                                           Release
                                                                                                                                                                             Rundeck
                                                                                    services                                Repo
                                              3        Validate
                                                       Release
                                                                                      OS                                      B
                                                                                                                                      Release
                                                                                                                                                                               Chef

                                                                                                                                       Repo                                  RightScale
                                                                                                                                         C
                                                  "Check for conflicts and
                                                                                 Sandbox
                                                                                                                                                                                                   Prod
                                                   run validation scripts"
Develop

                   Code, Config, Control



                                           "Check-
                                             in"
                                      1                           Source
                                                                   Repo
                                           "Check-
                                             in"
   Develop
                    Package Manifest                         "Check-
                                                               out"

    "Verify"
               2
                                       Watch
                                       Code
                          Publish
                          Results                    Build
                                      Continuous
                                                                              Release
                                      Deployment
                                                                               Repo
                          Integration    Loop         Unit
                           Testing                   Tests        "Promote"      A


   Dev Test
                                      Automatic
                                     Deployment                        3
Decoupled Release Model Example
                                                                           Code, Config, Control
                                                                            Code, Config, Control
                                                                              Code, Config, Control

                                                                                                             "Check-
     Operations                                                                                                in"
                                                                                                               "Check-
      Rundeck
                                                                                                         1        in"
                                                                                                                   "Check-               Source
                                                                                                          1           in"                 Repo
                                                                                                                                           Source                                         1        Monitor
                       Rundeck       Chef
                                                                                                              1
                                                                                                             "Check-
                                                                                                               in"
                                                                                                               "Check-
                                                                                                                                             Source
                                                                                                                                            Repo
    "Auto-Deploy                                                                                                                              Repo
                       Instance    Instance            Develop                                                    in"
                                                                                                                   "Check-
      New App                                           Develop              Package Manifest
                                                                                                                      in"        "Check-                                                          Jobs &
    Infrastructure
                      CI Server
                                                          Develop              Package Manifest
                                                                                 Package Manifest
                                                                                                                                   out"
                                                                                                                                   "Check-
                                                                                                                                     out"
                                                                                                                                      "Check-
                                                                                                                                                                                          2       Adhoc
                  1   Instance     Source
                                    Repo
                                                          "Verify"
                                                                       2                                                                out"                                   Operate
                                                                                                                                                                                                Commands
                                                            "Verify"
                                                                        2
                                                                             2
                                                                                                         Watch
                                                                "Verify"                                 Code
                      Monitoring                                                                           Watch
                                                                                        Publish             Code
                      & Testing    Staging                                              Results                Watch
                                                                                                                Build                                                                             Rundeck
                                   Release                                                Publish              Code
                                                                                          ResultsContinuous       Build
                                    Repo                                                      Publish                                                    Release                                        Chef
                                                                                                 Deployment
                                                                                              ResultsContinuous       Build
                                                                                                                                                          Repo
                                                                                                                                                          Release
                                                                                                           Continuous
                                                                                                         Loop
                                                                                        Integration Deployment Unit                                                                             RightScale
                                                                                          Testing                       Tests             "Promote"         ARelease
                                                                                                                                                            Repo
                                    Prod                                                   Integration     Deployment Unit
                                                                                                            Loop
                                                                                                                                            "Promote"         Repo
                                                                                                                                                              A                                                              Prod
                                   Release                                                   Testing           Loop        Tests
                                    Repo
                                                       Dev Test
                                                                                                        Automatic
                                                                                               Integration
                                                                                                       Deployment
                                                                                                 Testing
                                                                                                                               Unit
                                                                                                                              Tests             3
                                                                                                                                               "Promote"        A
                      Dev Test                                                                             Automatic
                                                                                                          Deployment                             3
                                                        Dev Test
                                                          Dev Test
                                                                                                              Automatic
                                                                                                             Deployment                              3


                  Bootstrap                                           Develop                                                                   Release                                   Operate

                                                                                               Release
                                                                                                Repo                                                                                          Release
                                                                                                  A        Release                                                                             Repo
                                                                                                            Repo                                                                                 A

                                              1         Repo
                                                                                                              B
                                                                                                                                                  4          Promote            Prod
                                                                                                                                                                                                         Release
                                                                                                                                                                                                          Repo
                                                                                                                                                                                                            B
                                                        Audit                                                        Release
                                                                                                                                                               Job              Repo                               Release
                                                                                                                      Repo                                                                                          Repo
                                                                                                                        C                                                                                             C
                                                  "Determine what packages                                                                               "Promote release"
                                                  and versions are available"



                                              2                            "Check
                                                                             -in"
                                                                                               Souce
                                                                                               Repo
                                                                                                                  Hudson
                                                                                                                                                                                                     Prod
                                                                                                                                                                                                    Release
                                                                                                                                                                                                     Repo

                              Release
                                                    Package Manifest                                                                              5          Deploy
                                                                                                                                                              Job
                                                                                                                                                                                                 "yum install"

                                                                                                               Release
                                                                                    app code
                                                                                                                Repo
                                                                                                                  A
                                                                                                                           Release
                                                                                                                                                                             Rundeck
                                                                                    services                                Repo
                                              3        Validate
                                                       Release
                                                                                      OS                                      B
                                                                                                                                      Release
                                                                                                                                                                               Chef

                                                                                                                                       Repo                                  RightScale
                                                                                                                                         C
                                                  "Check for conflicts and
                                                                                 Sandbox
                                                                                                                                                                                                   Prod
                                                   run validation scripts"
Release


                                                    Release
                                                     Repo                                                                             Release
                                                       A      Release                                                                  Repo
                                                               Repo                                                                      A

       1         Repo
                                                                 B
                                                                                                 4       Promote            Prod
                                                                                                                                                Release
                                                                                                                                                 Repo
                                                                                                                                                   B
                 Audit                                                  Release
                                                                                                           Job              Repo                          Release
                                                                         Repo                                                                              Repo
                                                                           C                                                                                 C
           "Determine what packages                                                                  "Promote release"
           and versions are available"



       2                        "Check
                                  -in"
                                                    Souce
                                                    Repo
                                                                   Hudson
                                                                                                                                             Prod
                                                                                                                                            Release
                                                                                                                                             Repo

ease
             Package Manifest                                                                    5       Deploy
                                                                                                          Job
                                                                                                                                         "yum install"

                                                                 Release
                                         app code
                                                                  Repo
                                                                    A
                                                                             Release
                                                                                                                         Rundeck
                                         services                             Repo
       3        Validate
                Release
                                           OS                                   B
                                                                                       Release
                                                                                                                           Chef

                                                                                        Repo                             RightScale
                                                                                          C
           "Check for conflicts and
                                      Sandbox
                                                                                                                                           Prod
            run validation scripts"
Decoupled Release Model Example
                                                                           Code, Config, Control
                                                                            Code, Config, Control
                                                                              Code, Config, Control

                                                                                                             "Check-
     Operations                                                                                                in"
                                                                                                               "Check-
      Rundeck
                                                                                                         1        in"
                                                                                                                   "Check-               Source
                                                                                                          1           in"                 Repo
                                                                                                                                           Source                                         1        Monitor
                       Rundeck       Chef
                                                                                                              1
                                                                                                             "Check-
                                                                                                               in"
                                                                                                               "Check-
                                                                                                                                             Source
                                                                                                                                            Repo
    "Auto-Deploy                                                                                                                              Repo
                       Instance    Instance            Develop                                                    in"
                                                                                                                   "Check-
      New App                                           Develop              Package Manifest
                                                                                                                      in"        "Check-                                                          Jobs &
    Infrastructure
                      CI Server
                                                          Develop              Package Manifest
                                                                                 Package Manifest
                                                                                                                                   out"
                                                                                                                                   "Check-
                                                                                                                                     out"
                                                                                                                                      "Check-
                                                                                                                                                                                          2       Adhoc
                  1   Instance     Source
                                    Repo
                                                          "Verify"
                                                                       2                                                                out"                                   Operate
                                                                                                                                                                                                Commands
                                                            "Verify"
                                                                        2
                                                                             2
                                                                                                         Watch
                                                                "Verify"                                 Code
                      Monitoring                                                                           Watch
                                                                                        Publish             Code
                      & Testing    Staging                                              Results                Watch
                                                                                                                Build                                                                             Rundeck
                                   Release                                                Publish              Code
                                                                                          ResultsContinuous       Build
                                    Repo                                                      Publish                                                    Release                                        Chef
                                                                                                 Deployment
                                                                                              ResultsContinuous       Build
                                                                                                                                                          Repo
                                                                                                                                                          Release
                                                                                                           Continuous
                                                                                                         Loop
                                                                                        Integration Deployment Unit                                                                             RightScale
                                                                                          Testing                       Tests             "Promote"         ARelease
                                                                                                                                                            Repo
                                    Prod                                                   Integration     Deployment Unit
                                                                                                            Loop
                                                                                                                                            "Promote"         Repo
                                                                                                                                                              A                                                              Prod
                                   Release                                                   Testing           Loop        Tests
                                    Repo
                                                       Dev Test
                                                                                                        Automatic
                                                                                               Integration
                                                                                                       Deployment
                                                                                                 Testing
                                                                                                                               Unit
                                                                                                                              Tests             3
                                                                                                                                               "Promote"        A
                      Dev Test                                                                             Automatic
                                                                                                          Deployment                             3
                                                        Dev Test
                                                          Dev Test
                                                                                                              Automatic
                                                                                                             Deployment                              3


                  Bootstrap                                           Develop                                                                   Release                                   Operate

                                                                                               Release
                                                                                                Repo                                                                                          Release
                                                                                                  A        Release                                                                             Repo
                                                                                                            Repo                                                                                 A

                                              1         Repo
                                                                                                              B
                                                                                                                                                  4          Promote            Prod
                                                                                                                                                                                                         Release
                                                                                                                                                                                                          Repo
                                                                                                                                                                                                            B
                                                        Audit                                                        Release
                                                                                                                                                               Job              Repo                               Release
                                                                                                                      Repo                                                                                          Repo
                                                                                                                        C                                                                                             C
                                                  "Determine what packages                                                                               "Promote release"
                                                  and versions are available"



                                              2                            "Check
                                                                             -in"
                                                                                               Souce
                                                                                               Repo
                                                                                                                  Hudson
                                                                                                                                                                                                     Prod
                                                                                                                                                                                                    Release
                                                                                                                                                                                                     Repo

                              Release
                                                    Package Manifest                                                                              5          Deploy
                                                                                                                                                              Job
                                                                                                                                                                                                 "yum install"

                                                                                                               Release
                                                                                    app code
                                                                                                                Repo
                                                                                                                  A
                                                                                                                           Release
                                                                                                                                                                             Rundeck
                                                                                    services                                Repo
                                              3        Validate
                                                       Release
                                                                                      OS                                      B
                                                                                                                                      Release
                                                                                                                                                                               Chef

                                                                                                                                       Repo                                  RightScale
                                                                                                                                         C
                                                  "Check for conflicts and
                                                                                 Sandbox
                                                                                                                                                                                                   Prod
                                                   run validation scripts"
Operate




               1    Monitor



                     Jobs &
               2     Adhoc
                   Commands
     Operate

                   Rundeck
                     Chef
                   RightScale
                                Prod
Decoupled Release Model Example
                                                                           Code, Config, Control
                                                                            Code, Config, Control
                                                                              Code, Config, Control

                                                                                                             "Check-
     Operations                                                                                                in"
                                                                                                               "Check-
      Rundeck
                                                                                                         1        in"
                                                                                                                   "Check-               Source
                                                                                                          1           in"                 Repo
                                                                                                                                           Source                                         1        Monitor
                       Rundeck       Chef
                                                                                                              1
                                                                                                             "Check-
                                                                                                               in"
                                                                                                               "Check-
                                                                                                                                             Source
                                                                                                                                            Repo
    "Auto-Deploy                                                                                                                              Repo
                       Instance    Instance            Develop                                                    in"
                                                                                                                   "Check-
      New App                                           Develop              Package Manifest
                                                                                                                      in"        "Check-                                                          Jobs &
    Infrastructure
                      CI Server
                                                          Develop              Package Manifest
                                                                                 Package Manifest
                                                                                                                                   out"
                                                                                                                                   "Check-
                                                                                                                                     out"
                                                                                                                                      "Check-
                                                                                                                                                                                          2       Adhoc
                  1   Instance     Source
                                    Repo
                                                          "Verify"
                                                                       2                                                                out"                                   Operate
                                                                                                                                                                                                Commands
                                                            "Verify"
                                                                        2
                                                                             2
                                                                                                         Watch
                                                                "Verify"                                 Code
                      Monitoring                                                                           Watch
                                                                                        Publish             Code
                      & Testing    Staging                                              Results                Watch
                                                                                                                Build                                                                             Rundeck
                                   Release                                                Publish              Code
                                                                                          ResultsContinuous       Build
                                    Repo                                                      Publish                                                    Release                                        Chef
                                                                                                 Deployment
                                                                                              ResultsContinuous       Build
                                                                                                                                                          Repo
                                                                                                                                                          Release
                                                                                                           Continuous
                                                                                                         Loop
                                                                                        Integration Deployment Unit                                                                             RightScale
                                                                                          Testing                       Tests             "Promote"         ARelease
                                                                                                                                                            Repo
                                    Prod                                                   Integration     Deployment Unit
                                                                                                            Loop
                                                                                                                                            "Promote"         Repo
                                                                                                                                                              A                                                              Prod
                                   Release                                                   Testing           Loop        Tests
                                    Repo
                                                       Dev Test
                                                                                                        Automatic
                                                                                               Integration
                                                                                                       Deployment
                                                                                                 Testing
                                                                                                                               Unit
                                                                                                                              Tests             3
                                                                                                                                               "Promote"        A
                      Dev Test                                                                             Automatic
                                                                                                          Deployment                             3
                                                        Dev Test
                                                          Dev Test
                                                                                                              Automatic
                                                                                                             Deployment                              3


                  Bootstrap                                           Develop                                                                   Release                                   Operate

                                                                                               Release
                                                                                                Repo                                                                                          Release
                                                                                                  A        Release                                                                             Repo
                                                                                                            Repo                                                                                 A

                                              1         Repo
                                                                                                              B
                                                                                                                                                  4          Promote            Prod
                                                                                                                                                                                                         Release
                                                                                                                                                                                                          Repo
                                                                                                                                                                                                            B
                                                        Audit                                                        Release
                                                                                                                                                               Job              Repo                               Release
                                                                                                                      Repo                                                                                          Repo
                                                                                                                        C                                                                                             C
                                                  "Determine what packages                                                                               "Promote release"
                                                  and versions are available"



                                              2                            "Check
                                                                             -in"
                                                                                               Souce
                                                                                               Repo
                                                                                                                  Hudson
                                                                                                                                                                                                     Prod
                                                                                                                                                                                                    Release
                                                                                                                                                                                                     Repo

                              Release
                                                    Package Manifest                                                                              5          Deploy
                                                                                                                                                              Job
                                                                                                                                                                                                 "yum install"

                                                                                                               Release
                                                                                    app code
                                                                                                                Repo
                                                                                                                  A
                                                                                                                           Release
                                                                                                                                                                             Rundeck
                                                                                    services                                Repo
                                              3        Validate
                                                       Release
                                                                                      OS                                      B
                                                                                                                                      Release
                                                                                                                                                                               Chef

                                                                                                                                       Repo                                  RightScale
                                                                                                                                         C
                                                  "Check for conflicts and
                                                                                 Sandbox
                                                                                                                                                                                                   Prod
                                                   run validation scripts"

Velocity 2011: Production Begins in Development

  • 1.
    Production Begins in Development Lee Thompson Damon Edwards
  • 2.
    • CTO Travel/Transportation Products, Hewlett Packard • Former Chief Technologist, E*TRADE Financial • Co-writer of “Fully Automated Lee Thompson Provisioning” whitepaper @stagr_lee
  • 3.
    • Co-founder DTOSolutions • DevOps Days organizer • Co-writer of “Fully Automated Provisioning” whitepaper Damon Edwards @damonedwards
  • 4.
    • Co-founder DTOSolutions • DevOps Days organizer • Co-writer of “Fully Automated Provisioning” whitepaper Damon Edwards @damonedwards
  • 5.
    • Consumer-facing webapplications • Webtone utilities
  • 6.
    Showing us whatis possible... Clouds Continuous Delivery/Deployment Lean Startup
  • 7.
    What we wantmore of... DevOps!
  • 8.
    vOps. DevOps. DevOps.DevOps. DevO The Rise of a New IT Operations Support Model !"#$%&'(#)*+,-.#/011#*+21+*#3425#6#7089*#.:46:*;"#*5-12"*<# ="#164;*#812><#-42+0<*4.#07:2#6#5607.:4*65#.:46:*;"#*5-12"*<# ="#$%?#23#@12=61#$%%%#24;670A6:027.C =" $%? 23 @12=61 $%%% 24;670A6:027. Why DevOps will not emerge: Why DevOps will emerge: !B>1:>461 8967;*. 64* :9* 964<*.: :2 !)*+,-. 0. 72: >.>611" <40+*7 3425 B>1:>461#8967;*.#64*#:9*#964<*.:#:2# )*+,-.#0.#72:#>.>611"#<40+*7#3425# 05-1*5*7:(#67<#)*+,-.#4*F>04*.#6# :9*#:2-#<2/7#67<(#:9>.(#56"#=*# .0;7030867:#4*:907G07;#23#DE# 524*#*6.01"#688*-:*<#="#DE# 2-*46:027.#827+*7:02761#/0.<25C 2-*46:027.#:*65.C !E9*4*#0.#6#164;*#=2<"#23#/24G#/0:9# !DEDH#67<#2:9*4#=*.:#-468:08*.# 4*.-*8:#:2#DEDH#67<#2:9*4#=*.:# 3465*/24G.#64*#68G72/1*<;*<#:2# -468:08*.#3465*/24G.#:96:#0.#614*6<"# 96+*#72:#<*10+*4*<#27#:9*04#;261.(# 688*-:*<#/0:907#:9*#07<>.:4"C 688*-:*< /0:907 :9* 07<>.:4" *76=107;#DE#24;670A6:027.#:2#122G#324# *76=107; DE 24;670A6:027. :2 122G 324 !,-*7#.2>48*#J,KKL#5676;*5*7:# 7*/#52<*1.C :221.(#/9089#64*#524*#610;7*<#/0:9# !E9*#;42/07;#07:*4*.:#07#:221.#.>89# -- :90.#6--42689(#96+*#72:#.**7# -- - 6.#B9*3(#I>--*:(#*:8C(#/011#9*1-# .0;7030867:#*7:*4-40.*#564G*:#.964*# .:05>16:*#<*567<#324#,KKM=6.*<# :468:027C 5676;*5*7:
  • 9.
    vOps. DevOps. DevOps.DevOps. DevO The Rise of a New IT Operations Support Model !"#$%&'(#)*+,-.#/011#*+21+*#3425#6#7089*#.:46:*;"#*5-12"*<# ="#164;*#812><#-42+0<*4.#07:2#6#5607.:4*65#.:46:*;"#*5-12"*<# ="#$%?#23#@12=61#$%%%#24;670A6:027.C =" $%? 23 @12=61 $%%% 24;670A6:027. Why DevOps will not emerge: Why DevOps will emerge: !B>1:>461 8967;*. 64* :9* 964<*.: :2 !)*+,-. 0. 72: >.>611" <40+*7 3425 B>1:>461#8967;*.#64*#:9*#964<*.:#:2# )*+,-.#0.#72:#>.>611"#<40+*7#3425# 05-1*5*7:(#67<#)*+,-.#4*F>04*.#6# :9*#:2-#<2/7#67<(#:9>.(#56"#=*# .0;7030867:#4*:907G07;#23#DE# of a New IT524*#*6.01"#688*-:*<#="#DE# The Rise Operations Support Model 2-*46:027.#827+*7:02761#/0.<25C 2-*46:027.#:*65.C !"#$%&'(#)*+,-.#/011#*+21+*#3425#6#7089*#.:46:*;"#*5-12"*<# !E9*4*#0.#6#164;*#=2<"#23#/24G#/0:9# !DEDH#67<#2:9*4#=*.:#-468:08*.# ="#164;*#812><#-42+0<*4.#07:2#6#5607.:4*65#.:46:*;"#*5-12"*<# 4*.-*8:#:2#DEDH#67<#2:9*4#=*.:# $%? 23 @12=61 $%%% 24;670A6:027. 3465*/24G.#64*#68G72/1*<;*<#:2# ="#$%?#23#@12=61#$%%%#24;670A6:027.C =" -468:08*.#3465*/24G.#:96:#0.#614*6<"# :296+*#72:#<*10+*4*<#27#:9*04#;261.(# Why DevOps will not emerge: B>1:>461 8967;*. 64* :9* 964<*.: ! ! Why DevOps will emerge: B>1:>461#8967;*.#64*#:9*#964<*.:#:2# )*+,-. 0. 72: >.>611" <40+*7 3425 )*+,-.#0.#72:#>.>611"#<40+*7#3425# 688*-:*<#/0:907#:9*#07<>.:4"C 688*-:*< /0:907 :9* 07<>.:4" 05-1*5*7:(#67<#)*+,-.#4*F>04*.#6#*76=107; DE 24;670A6:027. :2 122G 324 *76=107;#DE#24;670A6:027.#:2#122G#324# :9*#:2-#<2/7#67<(#:9>.(#56"#=*# .0;7030867:#4*:907G07;#23#DE# 524*#*6.01"#688*-:*<#="#DE# 2-*46:027.#827+*7:02761#/0.<25C 7*/#52<*1.C !,-*7#.2>48*#J,KKL#5676;*5*7:# 2-*46:027.#:*65.C E9*4*#0.#6#164;*#=2<"#23#/24G#/0:9# ! ! DEDH#67<#2:9*4#=*.:#-468:08*.# :221.(#/9089#64*#524*#610;7*<#/0:9# !E9*#;42/07;#07:*4*.:#07#:221.#.>89# 4*.-*8:#:2#DEDH#67<#2:9*4#=*.:# 3465*/24G.#64*#68G72/1*<;*<#:2# -468:08*.#3465*/24G.#:96:#0.#614*6<"# 96+*#72:#<*10+*4*<#27#:9*04#;261.(# -- :90.#6--42689(#96+*#72:#.**7# 07<>.:4" 688*-:*<#/0:907#:9*#07<>.:4"C 688*-:*< /0:907 :9* -- 6.#B9*3(#I>--*:(#*:8C(#/011#9*1-# *76=107;#DE#24;670A6:027.#:2#122G#324# *76=107; DE 24;670A6:027. :2 122G 324 - 7*/#52<*1.C .0;7030867:#*7:*4-40.*#564G*:#.964*# :221.(#/9089#64*#524*#610;7*<#/0:9# .:05>16:*#<*567<#324#,KKM=6.*<# ,-*7#.2>48*#J,KKL#5676;*5*7:# ! ! E9*#;42/07;#07:*4*.:#07#:221.#.>89# :468:027C -- :90.#6--42689(#96+*#72:#.**7# .0;7030867:#*7:*4-40.*#564G*:#.964*# 5676;*5*7: -- 6.#B9*3(#I>--*:(#*:8C(#/011#9*1-# .:05>16:*#<*567<#324#,KKM=6.*<# - :468:027C 5676;*5*7:
  • 10.
    Measuring success... Alignment Quality Cycle Time
  • 11.
    Know what kindof business you are
  • 12.
    What is yourtolerance for... Risk Change
  • 13.
    Consumer-facing applications... "Move fast and break things. Unless you are breaking stuff, you are not moving fast enough." -Mark Zuckerberg
  • 14.
    Consumer-facing applications... "Move fast and break things. Unless you are breaking stuff, you are not moving fast enough." -Mark Zuckerberg
  • 15.
  • 16.
  • 17.
  • 18.
  • 19.
  • 20.
    What do youwant...
  • 21.
    What do youwant...
  • 22.
    What do youwant...
  • 23.
    What do youwant...
  • 24.
    What do youwant...
  • 25.
    What do youwant...
  • 26.
    What do youwant...
  • 27.
    What do youwant...
  • 28.
  • 29.
    It all startsin Development
  • 30.
    It all startsin Development But what do we tell them to do and how do we get them to do it?
  • 31.
    Share ownership ofavailability • Developers must wear pagers • Incident command training so everyone knows their roles • Notification mechanism? Access provisioning?
  • 32.
    Non-functional requirements are first class citizens • All of the “-ility” requirements • As much a part of your service as the businessʼs functional requirements • Manage through same process • Make sure they are on the schedule • Donʼt over commit
  • 33.
    Strive for paritybetween dev & prod • Develop test data fixtures for all environments • Implement Mock Services for major infrastructure pieces for Developer user. • Continuous integration means integrate early • Use the deployment, config, and packaging tools in dev
  • 34.
    Push config management discipline back to Dev • Dev is about creating variation, Ops is about eliminating variation • Augment deployment toolchain to support the variation • Do developers use the tools? • Do developers like them? • Devʼs need config too • Accept config contributions and patches from dev
  • 35.
    Packaging... itʼs notjust for the OS • If you want to be a high performing web operations organization, you need to take change management seriously. • Strict versioning is needed, and package managers do that. • Itʼs about being idempotent. • Transfer packaging responsibility to dev • Define the packaging constructs you will support http://webscalebook.com
  • 36.
    Config is code •if itʼs code it needs to be managed like code... • Should be transparent and identical SDLC in both dev and ops • Avoid or eliminate asymetric release processes
  • 37.
    Tailor release artifactsto roles • “Small teams make better software” • Make sure one team getting stuck doesnʼt prevent another team to release (org coupling) • Large codebases suffer software entropy effects • Build an infrastructure that can reliably manage lots of smaller artifacts • Org conflict is a good time to suggest breaking up a codebase into separate concerns
  • 38.
    Standard management vocabulary •Consistent and expected management behavior • Across components and releases • “start, stop, status, update, install...”
  • 39.
    Rollback • Rollback thatworks • No, really works • Seriously, really really works • You say it will work? Test it right now
  • 40.
    Standard metrics abstractions •Dev surface metrics to Ops • Avoid the back and forth conversations • Use a standard framework • Use standard types (gague, counter, timer...) • Ops knows what to expect and how to visualize https://github.com/codahale/metrics
  • 41.
    Push test ownershipto the edges • QA is “Quality Assurance” • QA writing tests = bottleneck • QA writing tests = avoiding responsibility • QA is architecture / tooling / assurance • Test Driven Development • Test Driven Operations (yes, you too) • Bottom line: Everyone owns quality.
  • 42.
    Test outside ofthe box Crowd Test Simulate
  • 43.
  • 44.
    Build the rightlifecycle... Alignment Quality Cycle Time
  • 45.
  • 46.
    Continuous Delivery... Deployment... Automated Automated Testing Release
  • 47.
    Donʼt be sodogmatic
  • 48.
    Decoupled Release ModelExample Code, Config, Control Code, Config, Control Code, Config, Control "Check- Operations in" "Check- Rundeck 1 in" "Check- Source 1 in" Repo Source 1 Monitor Rundeck Chef 1 "Check- in" "Check- Source Repo "Auto-Deploy Repo Instance Instance Develop in" "Check- New App Develop Package Manifest in" "Check- Jobs & Infrastructure CI Server Develop Package Manifest Package Manifest out" "Check- out" "Check- 2 Adhoc 1 Instance Source Repo "Verify" 2 out" Operate Commands "Verify" 2 2 Watch "Verify" Code Monitoring Watch Publish Code & Testing Staging Results Watch Build Rundeck Release Publish Code ResultsContinuous Build Repo Publish Release Chef Deployment ResultsContinuous Build Repo Release Continuous Loop Integration Deployment Unit RightScale Testing Tests "Promote" ARelease Repo Prod Integration Deployment Unit Loop "Promote" Repo A Prod Release Testing Loop Tests Repo Dev Test Automatic Integration Deployment Testing Unit Tests 3 "Promote" A Dev Test Automatic Deployment 3 Dev Test Dev Test Automatic Deployment 3 Bootstrap Develop Release Operate Release Repo Release A Release Repo Repo A 1 Repo B 4 Promote Prod Release Repo B Audit Release Job Repo Release Repo Repo C C "Determine what packages "Promote release" and versions are available" 2 "Check -in" Souce Repo Hudson Prod Release Repo Release Package Manifest 5 Deploy Job "yum install" Release app code Repo A Release Rundeck services Repo 3 Validate Release OS B Release Chef Repo RightScale C "Check for conflicts and Sandbox Prod run validation scripts"
  • 49.
    Decoupled Release ModelExample Code, Config, Control Code, Config, Control Code, Config, Control "Check- Operations in" "Check- Rundeck 1 in" "Check- Source 1 in" Repo Source 1 Monitor Rundeck Chef 1 "Check- in" "Check- Source Repo "Auto-Deploy Repo Instance Instance Develop in" "Check- New App Develop Package Manifest in" "Check- Jobs & Infrastructure CI Server Develop Package Manifest Package Manifest out" "Check- out" "Check- 2 Adhoc 1 Instance Source Repo "Verify" 2 out" Operate Commands "Verify" 2 2 Watch "Verify" Code Monitoring Watch Publish Code & Testing Staging Results Watch Build Rundeck Release Publish Code ResultsContinuous Build Repo Publish Release Chef Deployment ResultsContinuous Build Repo Release Continuous Loop Integration Deployment Unit RightScale Testing Tests "Promote" ARelease Repo Prod Integration Deployment Unit Loop "Promote" Repo A Prod Release Testing Loop Tests Repo Dev Test Automatic Integration Deployment Testing Unit Tests 3 "Promote" A Dev Test Automatic Deployment 3 Dev Test Dev Test Automatic Deployment 3 Bootstrap Develop Release Operate Release Repo Release A Release Repo Repo A 1 Repo B 4 Promote Prod Release Repo B Audit Release Job Repo Release Repo Repo C C "Determine what packages "Promote release" and versions are available" 2 "Check -in" Souce Repo Hudson Prod Release Repo Release Package Manifest 5 Deploy Job "yum install" Release app code Repo A Release Rundeck services Repo 3 Validate Release OS B Release Chef Repo RightScale C "Check for conflicts and Sandbox Prod run validation scripts"
  • 50.
    Bootstrap Operations Rundeck Rundeck Chef "Auto-Deploy Instance Instance New App Infrastructure CI Server Source 1 Instance Repo Monitoring & Testing Staging Release Repo Prod Release Repo Dev Test
  • 51.
    Decoupled Release ModelExample Code, Config, Control Code, Config, Control Code, Config, Control "Check- Operations in" "Check- Rundeck 1 in" "Check- Source 1 in" Repo Source 1 Monitor Rundeck Chef 1 "Check- in" "Check- Source Repo "Auto-Deploy Repo Instance Instance Develop in" "Check- New App Develop Package Manifest in" "Check- Jobs & Infrastructure CI Server Develop Package Manifest Package Manifest out" "Check- out" "Check- 2 Adhoc 1 Instance Source Repo "Verify" 2 out" Operate Commands "Verify" 2 2 Watch "Verify" Code Monitoring Watch Publish Code & Testing Staging Results Watch Build Rundeck Release Publish Code ResultsContinuous Build Repo Publish Release Chef Deployment ResultsContinuous Build Repo Release Continuous Loop Integration Deployment Unit RightScale Testing Tests "Promote" ARelease Repo Prod Integration Deployment Unit Loop "Promote" Repo A Prod Release Testing Loop Tests Repo Dev Test Automatic Integration Deployment Testing Unit Tests 3 "Promote" A Dev Test Automatic Deployment 3 Dev Test Dev Test Automatic Deployment 3 Bootstrap Develop Release Operate Release Repo Release A Release Repo Repo A 1 Repo B 4 Promote Prod Release Repo B Audit Release Job Repo Release Repo Repo C C "Determine what packages "Promote release" and versions are available" 2 "Check -in" Souce Repo Hudson Prod Release Repo Release Package Manifest 5 Deploy Job "yum install" Release app code Repo A Release Rundeck services Repo 3 Validate Release OS B Release Chef Repo RightScale C "Check for conflicts and Sandbox Prod run validation scripts"
  • 52.
    Develop Code, Config, Control "Check- in" 1 Source Repo "Check- in" Develop Package Manifest "Check- out" "Verify" 2 Watch Code Publish Results Build Continuous Release Deployment Repo Integration Loop Unit Testing Tests "Promote" A Dev Test Automatic Deployment 3
  • 53.
    Decoupled Release ModelExample Code, Config, Control Code, Config, Control Code, Config, Control "Check- Operations in" "Check- Rundeck 1 in" "Check- Source 1 in" Repo Source 1 Monitor Rundeck Chef 1 "Check- in" "Check- Source Repo "Auto-Deploy Repo Instance Instance Develop in" "Check- New App Develop Package Manifest in" "Check- Jobs & Infrastructure CI Server Develop Package Manifest Package Manifest out" "Check- out" "Check- 2 Adhoc 1 Instance Source Repo "Verify" 2 out" Operate Commands "Verify" 2 2 Watch "Verify" Code Monitoring Watch Publish Code & Testing Staging Results Watch Build Rundeck Release Publish Code ResultsContinuous Build Repo Publish Release Chef Deployment ResultsContinuous Build Repo Release Continuous Loop Integration Deployment Unit RightScale Testing Tests "Promote" ARelease Repo Prod Integration Deployment Unit Loop "Promote" Repo A Prod Release Testing Loop Tests Repo Dev Test Automatic Integration Deployment Testing Unit Tests 3 "Promote" A Dev Test Automatic Deployment 3 Dev Test Dev Test Automatic Deployment 3 Bootstrap Develop Release Operate Release Repo Release A Release Repo Repo A 1 Repo B 4 Promote Prod Release Repo B Audit Release Job Repo Release Repo Repo C C "Determine what packages "Promote release" and versions are available" 2 "Check -in" Souce Repo Hudson Prod Release Repo Release Package Manifest 5 Deploy Job "yum install" Release app code Repo A Release Rundeck services Repo 3 Validate Release OS B Release Chef Repo RightScale C "Check for conflicts and Sandbox Prod run validation scripts"
  • 54.
    Release Release Repo Release A Release Repo Repo A 1 Repo B 4 Promote Prod Release Repo B Audit Release Job Repo Release Repo Repo C C "Determine what packages "Promote release" and versions are available" 2 "Check -in" Souce Repo Hudson Prod Release Repo ease Package Manifest 5 Deploy Job "yum install" Release app code Repo A Release Rundeck services Repo 3 Validate Release OS B Release Chef Repo RightScale C "Check for conflicts and Sandbox Prod run validation scripts"
  • 55.
    Decoupled Release ModelExample Code, Config, Control Code, Config, Control Code, Config, Control "Check- Operations in" "Check- Rundeck 1 in" "Check- Source 1 in" Repo Source 1 Monitor Rundeck Chef 1 "Check- in" "Check- Source Repo "Auto-Deploy Repo Instance Instance Develop in" "Check- New App Develop Package Manifest in" "Check- Jobs & Infrastructure CI Server Develop Package Manifest Package Manifest out" "Check- out" "Check- 2 Adhoc 1 Instance Source Repo "Verify" 2 out" Operate Commands "Verify" 2 2 Watch "Verify" Code Monitoring Watch Publish Code & Testing Staging Results Watch Build Rundeck Release Publish Code ResultsContinuous Build Repo Publish Release Chef Deployment ResultsContinuous Build Repo Release Continuous Loop Integration Deployment Unit RightScale Testing Tests "Promote" ARelease Repo Prod Integration Deployment Unit Loop "Promote" Repo A Prod Release Testing Loop Tests Repo Dev Test Automatic Integration Deployment Testing Unit Tests 3 "Promote" A Dev Test Automatic Deployment 3 Dev Test Dev Test Automatic Deployment 3 Bootstrap Develop Release Operate Release Repo Release A Release Repo Repo A 1 Repo B 4 Promote Prod Release Repo B Audit Release Job Repo Release Repo Repo C C "Determine what packages "Promote release" and versions are available" 2 "Check -in" Souce Repo Hudson Prod Release Repo Release Package Manifest 5 Deploy Job "yum install" Release app code Repo A Release Rundeck services Repo 3 Validate Release OS B Release Chef Repo RightScale C "Check for conflicts and Sandbox Prod run validation scripts"
  • 56.
    Operate 1 Monitor Jobs & 2 Adhoc Commands Operate Rundeck Chef RightScale Prod
  • 57.
    Decoupled Release ModelExample Code, Config, Control Code, Config, Control Code, Config, Control "Check- Operations in" "Check- Rundeck 1 in" "Check- Source 1 in" Repo Source 1 Monitor Rundeck Chef 1 "Check- in" "Check- Source Repo "Auto-Deploy Repo Instance Instance Develop in" "Check- New App Develop Package Manifest in" "Check- Jobs & Infrastructure CI Server Develop Package Manifest Package Manifest out" "Check- out" "Check- 2 Adhoc 1 Instance Source Repo "Verify" 2 out" Operate Commands "Verify" 2 2 Watch "Verify" Code Monitoring Watch Publish Code & Testing Staging Results Watch Build Rundeck Release Publish Code ResultsContinuous Build Repo Publish Release Chef Deployment ResultsContinuous Build Repo Release Continuous Loop Integration Deployment Unit RightScale Testing Tests "Promote" ARelease Repo Prod Integration Deployment Unit Loop "Promote" Repo A Prod Release Testing Loop Tests Repo Dev Test Automatic Integration Deployment Testing Unit Tests 3 "Promote" A Dev Test Automatic Deployment 3 Dev Test Dev Test Automatic Deployment 3 Bootstrap Develop Release Operate Release Repo Release A Release Repo Repo A 1 Repo B 4 Promote Prod Release Repo B Audit Release Job Repo Release Repo Repo C C "Determine what packages "Promote release" and versions are available" 2 "Check -in" Souce Repo Hudson Prod Release Repo Release Package Manifest 5 Deploy Job "yum install" Release app code Repo A Release Rundeck services Repo 3 Validate Release OS B Release Chef Repo RightScale C "Check for conflicts and Sandbox Prod run validation scripts"