ELIMINATING
“OVER THE FENCE”
Blurring the lines between developers and operations
Eric Tucker
Maritime DevCon
ABOUT ME
● DevOps Engineer at Blue Spurs
● AWS Certified DevOps Engineer
● Specialize in infrastructure automation in AWS
● Previously a developer, moved to operations
team, then strived towards engaging both sides
THE “NO AGENDA” AGENDA
This will not be a talk where I sell you a product.
This will not be a talk about a cool new technology.
This will not be a talk focused on any single topic.
This will be a talk about:
○ lessons learned
○ technology & people
○ evolution of work culture
○ personal experience
○ pain points
○ mistakes
WHAT IS “DEVOPS”?
● You’ve probably heard it before
● It has countless meanings
● None of correct or incorrect
Any improvements done to move faster through the
software development lifecycle count as DevOps.
- Rob Brigham, Senior Product Manager, Amazon Web Services
- -
TRADITIONAL ORGANIZATIONAL CULTURE
● No communication between developers and
operations team members
● Developers have no visibility into environments
● Code “hand-off”
● Lots of things can be missed:
○ Configuration
○ New package dependencies
○ Other special instructions
● README:
○ Install this package
○ Copy this file, edit it to your settings
○ Run these commands
○ Copy the code here
○ Run these other commands
○ If this doesn’t work, try this…
● Oh, and those instructions have nothing
to do with deployment to test / prod
environments!
TRADITIONAL DEVELOPER SETUP
TRADITIONAL OPERATIONS SETUP
● Test/Prod server setup once:
○ Manually
○ Oh, and that person doesn’t work here anymore
○ And… it was never documented
● Deployment is a set of steps in a text
document
● Developer setup instructions don’t help:
○ Server is Redhat
○ Developers use Ubuntu (or worse, Windows)
SOLUTIONS
Changes to:
● Technology
● People
TECHNOLOGY
TECHNICAL #1: DON’T BLACK BOX
● Test and production environments
should not be a mystery
● Developers should know how these are configured
○ a document of manual instructions does not
count
● Let developers run the same setup scripts to
configure an environment (VM, Vagrant Box, etc)
○ Challenge with expensive software licenses
○ Don’t use Windows based software!
TECHNICAL #2: AUTOMATE
● Treat servers like cattle, not pets
● Infrastructure as code
● Use automation tools to configure servers:
○ Ex: Chef, Ansible, Puppet
○ Not a text document of instructions to manually
run!
● Automate infrastructure creation where possible
○ AWS CloudFormation & AWS Elastic Beanstalk
TECHNICAL #3: VIRTUALIZE
● Physical servers on a rack is no longer a viable provisioning
strategy.
● Every server should be virtualized
● Benefits:
○ Cost savings (higher host utilization)
○ Easy to provision more capacity (within host limits)
○ Snapshot backups
○ OS standardization
TECHNICAL #4: CONTAINERS
● You should be using containers
● Developers can run multi-tier application
on their machine
● Behaviour and deployment is the same
across environments - no mysteries or surprises
● Docker runs everywhere
TECHNICAL #5: MICROSERVICES
● Monolithic deployments are obsolete:
○ Blast radius
○ Downtime during deployment
○ Difficult to troubleshoot and/or rollback
● Build on the container mantra:
○ One container
○ One purpose
○ One deployment
○ All intercommunication done via APIs
TECHNICAL #6: CONTINUOUS INTEGRATION
New code should always be doing something:
● Constantly running builds
● Constantly running tests
● Constantly deploying
● Code pipelines
● Feedback loops (monitoring and alerting)
TECHNICAL #7: AUTOMATED TESTING
● Manual user testing slows down development and delivery
● Nearly all tests can be performed by a machine:
○ Unit testing
○ Integration testing
○ UI testing
○ Automated mobile testing
● Provide ways for developers to run all tests:
○ Earlier feedback loop reduces costs and errors
○ Developers catching issues earlier means fewer
erroneous commits
TECHNICAL CHANGES REVIEW
1. No black box servers
2. Automate EVERYTHING
3. Virtualize
4. Use containers
5. Microservices
6. Continuous integration
7. Automated testing
PEOPLE
PEOPLE #1: TWO PIZZA TEAMS
● Amazon team size rule
● If you can’t feed the team with two
pizzas, it’s too big
● If your project is too big for that
team, the work must be divided
● Smaller teams means more
collaboration
PEOPLE #2: ENGAGE EVERYONE
● Operations should be part of initial design phase
○ Aids in preparation for infrastructure provisioning
● Operations involved with [major] feature design
○ Fewer surprises
○ Fewer impossible promises
● Developers should be part of architecture design
○ Eliminate black box through knowledge sharing
○ Experience designing highly performant systems
● Developers should design monitoring solutions
○ They code it - they define failure
○ Work with operations to determine escalation procedures
PEOPLE #3: COMMUNICATION
● Operations should attend sprint planning / standups
● Developers should be involved with deployment
planning meetings
● Train developers in advanced concepts - ignorance leads
to over the fence or worse, resentment
● Operations should be aware of upcoming releases
(with more than 24 hours notice!)
PEOPLE #4: ROTATING TEAM MEMBERS
● When possible, rotate team members into and out of
both sides:
○ A developer can join operations for two weeks to get a
feel for how the wheels turn
○ An operations member can join development for two
weeks to understand the challenges they face
delivering client features
● Knowledge grows
● Greater respect for what the other goes through
● Team becomes more nimble
PEOPLE #5: NO SPECIALIZATION
● If things stop when one person takes vacation that’s a skill gap!
● No silo coding
● Shadowing
● Regular code reviews & pull requests
● Discuss any and all changes with other team members
PEOPLE #6: RESPONSIBILITY
● Perfect scenario: no developer and operations split
● By fostering a DevOps mantra you encourage every team
member to be responsible for every phase of delivery
● Friday night deployment?
○ You’re on the hook
○ Can’t throw it over the fence and say:
“it worked in development”
○ Better code (hopefully)
Questions?

Eric tucker - Eliminating "Over the Fence"

  • 1.
    ELIMINATING “OVER THE FENCE” Blurringthe lines between developers and operations Eric Tucker Maritime DevCon
  • 2.
    ABOUT ME ● DevOpsEngineer at Blue Spurs ● AWS Certified DevOps Engineer ● Specialize in infrastructure automation in AWS ● Previously a developer, moved to operations team, then strived towards engaging both sides
  • 3.
    THE “NO AGENDA”AGENDA This will not be a talk where I sell you a product. This will not be a talk about a cool new technology. This will not be a talk focused on any single topic. This will be a talk about: ○ lessons learned ○ technology & people ○ evolution of work culture ○ personal experience ○ pain points ○ mistakes
  • 4.
    WHAT IS “DEVOPS”? ●You’ve probably heard it before ● It has countless meanings ● None of correct or incorrect Any improvements done to move faster through the software development lifecycle count as DevOps. - Rob Brigham, Senior Product Manager, Amazon Web Services - -
  • 5.
    TRADITIONAL ORGANIZATIONAL CULTURE ●No communication between developers and operations team members ● Developers have no visibility into environments ● Code “hand-off” ● Lots of things can be missed: ○ Configuration ○ New package dependencies ○ Other special instructions
  • 6.
    ● README: ○ Installthis package ○ Copy this file, edit it to your settings ○ Run these commands ○ Copy the code here ○ Run these other commands ○ If this doesn’t work, try this… ● Oh, and those instructions have nothing to do with deployment to test / prod environments! TRADITIONAL DEVELOPER SETUP
  • 7.
    TRADITIONAL OPERATIONS SETUP ●Test/Prod server setup once: ○ Manually ○ Oh, and that person doesn’t work here anymore ○ And… it was never documented ● Deployment is a set of steps in a text document ● Developer setup instructions don’t help: ○ Server is Redhat ○ Developers use Ubuntu (or worse, Windows)
  • 8.
  • 9.
  • 10.
    TECHNICAL #1: DON’TBLACK BOX ● Test and production environments should not be a mystery ● Developers should know how these are configured ○ a document of manual instructions does not count ● Let developers run the same setup scripts to configure an environment (VM, Vagrant Box, etc) ○ Challenge with expensive software licenses ○ Don’t use Windows based software!
  • 11.
    TECHNICAL #2: AUTOMATE ●Treat servers like cattle, not pets ● Infrastructure as code ● Use automation tools to configure servers: ○ Ex: Chef, Ansible, Puppet ○ Not a text document of instructions to manually run! ● Automate infrastructure creation where possible ○ AWS CloudFormation & AWS Elastic Beanstalk
  • 12.
    TECHNICAL #3: VIRTUALIZE ●Physical servers on a rack is no longer a viable provisioning strategy. ● Every server should be virtualized ● Benefits: ○ Cost savings (higher host utilization) ○ Easy to provision more capacity (within host limits) ○ Snapshot backups ○ OS standardization
  • 13.
    TECHNICAL #4: CONTAINERS ●You should be using containers ● Developers can run multi-tier application on their machine ● Behaviour and deployment is the same across environments - no mysteries or surprises ● Docker runs everywhere
  • 14.
    TECHNICAL #5: MICROSERVICES ●Monolithic deployments are obsolete: ○ Blast radius ○ Downtime during deployment ○ Difficult to troubleshoot and/or rollback ● Build on the container mantra: ○ One container ○ One purpose ○ One deployment ○ All intercommunication done via APIs
  • 15.
    TECHNICAL #6: CONTINUOUSINTEGRATION New code should always be doing something: ● Constantly running builds ● Constantly running tests ● Constantly deploying ● Code pipelines ● Feedback loops (monitoring and alerting)
  • 16.
    TECHNICAL #7: AUTOMATEDTESTING ● Manual user testing slows down development and delivery ● Nearly all tests can be performed by a machine: ○ Unit testing ○ Integration testing ○ UI testing ○ Automated mobile testing ● Provide ways for developers to run all tests: ○ Earlier feedback loop reduces costs and errors ○ Developers catching issues earlier means fewer erroneous commits
  • 17.
    TECHNICAL CHANGES REVIEW 1.No black box servers 2. Automate EVERYTHING 3. Virtualize 4. Use containers 5. Microservices 6. Continuous integration 7. Automated testing
  • 18.
  • 19.
    PEOPLE #1: TWOPIZZA TEAMS ● Amazon team size rule ● If you can’t feed the team with two pizzas, it’s too big ● If your project is too big for that team, the work must be divided ● Smaller teams means more collaboration
  • 20.
    PEOPLE #2: ENGAGEEVERYONE ● Operations should be part of initial design phase ○ Aids in preparation for infrastructure provisioning ● Operations involved with [major] feature design ○ Fewer surprises ○ Fewer impossible promises ● Developers should be part of architecture design ○ Eliminate black box through knowledge sharing ○ Experience designing highly performant systems ● Developers should design monitoring solutions ○ They code it - they define failure ○ Work with operations to determine escalation procedures
  • 21.
    PEOPLE #3: COMMUNICATION ●Operations should attend sprint planning / standups ● Developers should be involved with deployment planning meetings ● Train developers in advanced concepts - ignorance leads to over the fence or worse, resentment ● Operations should be aware of upcoming releases (with more than 24 hours notice!)
  • 22.
    PEOPLE #4: ROTATINGTEAM MEMBERS ● When possible, rotate team members into and out of both sides: ○ A developer can join operations for two weeks to get a feel for how the wheels turn ○ An operations member can join development for two weeks to understand the challenges they face delivering client features ● Knowledge grows ● Greater respect for what the other goes through ● Team becomes more nimble
  • 23.
    PEOPLE #5: NOSPECIALIZATION ● If things stop when one person takes vacation that’s a skill gap! ● No silo coding ● Shadowing ● Regular code reviews & pull requests ● Discuss any and all changes with other team members
  • 25.
    PEOPLE #6: RESPONSIBILITY ●Perfect scenario: no developer and operations split ● By fostering a DevOps mantra you encourage every team member to be responsible for every phase of delivery ● Friday night deployment? ○ You’re on the hook ○ Can’t throw it over the fence and say: “it worked in development” ○ Better code (hopefully)
  • 26.