DEVOPS IS A VERB
DevOps UMN 2017
THIS IS AWESOME…..THANK YOU!!
INTRODUCTION
ANDY DOMEIER
• 12+ Years @ SPS Commerce
• 24x7 System Operations
• Reliability Engineering
• Performance Engineering
• DevOps Days & DevOps Meetup Organizer
• Optimistic by nature
• “The disk is half empty”
• Twitter: @ajdomie
• My wife thinks she’s winning
@ajdomie
@ajdomie
YOUR JOB IS NOT YOUR JOB:
IN ONE SENTENCE, WHAT’S YOUR JOB?
Fred Kofman - https://www.youtube.com/watch?v=6OI7REyatq4
DO YOU………
• Developer:
• Write Code
• Build applications that improve the educational experience for UofM students
• Systems Engineer:
• Build and Operate Servers
• Provide a stable environment for applications to operate effectively
• Database Engineer:
• Build and Operate Databases
• Provide a highly performing data store for applications
@ajdomie
@ajdomie
@ajdomie
@ajdomie
THAT’S DEVOPS!
“IT@UMN IS A COMMUNITY OF IT PROFESSIONALS
ACROSS THE UNIVERSITY WORKING TOGETHER TO
SUPPORT AND ADVANCE THE ACADEMIC MISSION OF
THE UNIVERSITY OF MINNESOTA.”
Guiding Principals - https://it.umn.edu/about/guiding-principles
SUBD. 1. MISSION. THE UNIVERSITY OF MINNESOTA
(UNIVERSITY), FOUNDED IN THE BELIEF THAT ALL PEOPLE ARE
ENRICHED BY UNDERSTANDING, IS DEDICATED TO THE
ADVANCEMENT OF LEARNING AND THE SEARCH FOR TRUTH;
TO THE SHARING OF THIS KNOWLEDGE THROUGH EDUCATION
FOR A DIVERSE COMMUNITY; AND TO THE APPLICATION OF
THIS KNOWLEDGE TO BENEFIT THE PEOPLE OF THE STATE, THE
NATION, AND THE WORLD.
UMN Mission Statement - http://regents.umn.edu/sites/regents.umn.edu/files/policies/Mission_Statement.pdf
DOES THIS CHANGE
• The way we communicate
• The way we solve problems
• The way we use technology
• What we build
• How we build
• When we build
• …….Everything
@ajdomie
WHY NOW?
SOFTWARE IS EATING
THE WORLD!
@ajdomie
“DIGITAL TRANSFORMATION”
@ajdomie
COMMON TRAITS OF WINNERS
• Ship Fast, Ship Often
• Fail Fast & Iterate
• Prove theories & learn faster
• They Win Together
• Tesla:
• Tesla software is no good without the physical car
• No one will buy the car if it doesn’t look good
• Battery life is critical
• Airbnb & Lyft:
• No available & secure payment solution, no drivers or hosts
• No drivers or hosts, no platforms needed
@ajdomie
THE CORE OF DEVOPS
•Culture
•Automation
•Measurement
@ajdomie
CULTURE:
WINNING AS
A TEAM
@ajdomie
@ajdomie
@ajdomie
YAY DEVOPS!
@ajdomie
CULTURE
• Ok, win as a team… got it.
@ajdomie
THE END!
QUESTIONS?
THIS IS REALLY HARD
• Aligning to a greater goal
• Team goals & incentives
• Individual goals & incentives
• Triple Down on Communication
• “You mean I have to physically talk to them?”
• (yes, that’s what I mean)
• Effective communication takes time
• Listening is usually more important
• Safety in Failure
• Fail fast & iterate
• The perception of failure
@ajdomie
PRIORITY CONFLICT
• Feature v. Bug
• New Server Now v. Automate
• Patch v. Build New
• Prod is Broke v. Life
@ajdomie
TRANSPARENCY
@ajdomie
WHERE TO START….
• Build empathy
• Between teams
• Between individuals
• Sometimes offense needs to play defense
• Build trust
• Between teams
• Between individuals
• Service teams
• Work as if everything your team builds is a service
• Align your service with the mission
@ajdomie
Jeff Sussna Empathy the essence of DevOps:
http://blog.ingineering.it/post/72964480807/empathy-the-essence-of-devops
RESPOND IN ISOLATION
• Hey Danielle, It looks like the site is acting up and when looking around the only outlier
I have found so far is a cpu spike on the DB. Can you help me investigate this a bit
more?
RESPOND AS A TEAM
WHAT CAN YOU DO TOMORROW THAT
CAN MAKE A DIFFERENCE?
AUTOMATE:
BE BETTER,
FASTER &
ALSO BETTER
@ajdomie
TECHNOLOGY MAKES SOMETHING BETTER
EVERY DAY!
HOW DO WE BUILD IT?
@ajdomie
HOW DO WE SECURE IT?
@ajdomie
HOW DO WE SHIP IT?
@ajdomie
HOW DO WE OPERATE IT?
@ajdomie
SMARTER NOT HARDER*
• If something is hard (but important), do it often
• Provision servers
• Provision DB’s
• Test your code
• Ship your code
• Decrease Friction
• Make building & shipping more consistent (not easy)
• Limit Risk
• Automated testing
• Reduce the impact of failure
• Phased deployments
• Feature flags
• Monitoring
*Note: please continue to work hard (And smart  )@ajdomie
DIGITAL TRANSFORMATION
• Teams that ship fastest win
• Theories are proven (or disproven), learn quickly
• Fail fast (and safely) & iterate
• Competitive advantage
• Win together
• Automate to deliver a service
• No single service wins the game
• Share and you learn faster
@ajdomie
WHAT CAN YOU DO TOMORROW THAT
CAN MAKE A DIFFERENCE?
MEASURE:
VALIDATE
& ITERATE
@ajdomie
ALIGN YOUR GOALS
• Favor action and move forward
• Fail fast and iterate
• Time to value for your customer
• Encourage technical pride
• Seeing your efforts providing value in production is good for the soul
• Limit risk to enable failing fast
• Foster innovation
@ajdomie
DELIVER DELIGHTFUL SERVICE
• If it moves, graph it! - @etsy
• If it doesn’t move, graph it anyways! – also @etsy
• Performance to promises
• Incidents
• Cause
• Monitoring
• Recover efforts
@ajdomie
SHARE YOUR DATA
@ajdomie
WHAT CAN YOU DO TOMORROW THAT
CAN MAKE A DIFFERENCE?
RELIABILITY @ SPS
@ajdomie
RELIABILITY’S JOURNEY
• 2 years ago:
• Single monitoring stack
• Manual setup
• Lacked proper variety of solutions
• The Goal: Expand service offerings
• Logs
• User experience & availability
• Application performance
• Notification platform
@ajdomie
RELIABILITY’S JOURNEY
• 1 year ago:
• Basic monitoring stacks
• Manual setup
• Custom monitors were time consuming
• Consistency was a fun idea
• The Goal: Monitoring as a Service
• Establish a baseline that’s automated
• Custom was easier
• Consistency in solution types
• Established experts of the service “monitoring”
@ajdomie
RELIABILITY’S JOURNEY
• Today:
• Monitoring Service
• Event based serverless monitoring setup
• Prescribed reusable patterns for various situations
• Single pane of glass & improved collaboration
• The Goal: Operations Automation Framework
• Query-able documentation (Dependencies)
• Beta: Advanced Visualization - Vizceral
• Event based automation improving communication & transparency
• Event based automation improving time to remediation
• Encourage shared learnings and efforts, reusable functions
@ajdomie
RELIABILITY’S JOURNEY
• What hasn’t gone well?
• Under communicated motivation and goals (yr1)
• Generated a “Who are you to tell me” response
@ajdomie
RELIABILITY’S JOURNEY
• What has gone well?
• Reduced friction by simplifying serverless deployments
• Limited risk with architecture (getting harder)
• Failed fast & iterated
• Double downed on transparency
• Internal presentations
• Blog posts
• Advocates
• All PRs welcomed
• Invested in empathy & collaboration
@ajdomie
RELIABILITY’S RESULTS
• Reduced incident rates & time to resolution
• Positively impacted:
• Customer experience
• Team moral
• Engineering productivity
• Cross team collaboration
• Technical pride
• SPS Mission: To be the most chosen and relied upon retail network.
@ajdomie
WHAT NOW?
@ajdomie
TO DO LIST
• Deliver Services (regardless of org)
• Encourage reuse, care about customers
• Solve problems at scale with tools
• Leverage tools consistently
• Be Empathetic
• Create transparency to priorities & goals
• Build trust
• Fail Fast & Iterate
• Change is a good thing, get good at it
• Improve agility & velocity
• Share
• Knowledge
• Experiences
@ajdomie
BIGGEST BLOCKERS
@ajdomie
• Cultural
• Technical
• Measurements
DEVOPS IS A VERB
• It’s collaborating
• It’s automating
• It’s communicating
• It’s iterating
• It’s trusting each other
• It’s winning
@ajdomie
WHAT CAN YOU DO TOMORROW THAT CAN MAKE A
DIFFERENCE? (X3)
“
”
DEVOPS ISN’T A GAME YOU PLAY UNTIL YOU WIN, IT’S
A THING YOU DO THAT RESULTS IN WINNING.
THANK YOU!!
QUESTIONS, DEMOS, COMPLAINTS, ETC…
@ajdomie
ARTICLES
• Your job is not your job - Fred Kofman
• https://www.youtube.com/watch?v=6OI7REyatq4
• Empathy the essence of DevOps - Jeff Sussna
• http://blog.ingineering.it/post/72964480807/empathy-the-essence-of-devops
• Promise Theory – Mark Burgess
• “Scaling cooperation with Promises, not obligations”
• http://markburgess.org/blog_cooperate.html
• DevOps as an example of promise-oriented cooperation, without money
@ajdomie

Devops is a Verb

  • 1.
    DEVOPS IS AVERB DevOps UMN 2017
  • 2.
  • 3.
    INTRODUCTION ANDY DOMEIER • 12+Years @ SPS Commerce • 24x7 System Operations • Reliability Engineering • Performance Engineering • DevOps Days & DevOps Meetup Organizer • Optimistic by nature • “The disk is half empty” • Twitter: @ajdomie • My wife thinks she’s winning @ajdomie
  • 4.
  • 5.
    YOUR JOB ISNOT YOUR JOB: IN ONE SENTENCE, WHAT’S YOUR JOB? Fred Kofman - https://www.youtube.com/watch?v=6OI7REyatq4
  • 6.
    DO YOU……… • Developer: •Write Code • Build applications that improve the educational experience for UofM students • Systems Engineer: • Build and Operate Servers • Provide a stable environment for applications to operate effectively • Database Engineer: • Build and Operate Databases • Provide a highly performing data store for applications @ajdomie
  • 7.
  • 8.
  • 9.
  • 10.
  • 11.
    “IT@UMN IS ACOMMUNITY OF IT PROFESSIONALS ACROSS THE UNIVERSITY WORKING TOGETHER TO SUPPORT AND ADVANCE THE ACADEMIC MISSION OF THE UNIVERSITY OF MINNESOTA.” Guiding Principals - https://it.umn.edu/about/guiding-principles
  • 12.
    SUBD. 1. MISSION.THE UNIVERSITY OF MINNESOTA (UNIVERSITY), FOUNDED IN THE BELIEF THAT ALL PEOPLE ARE ENRICHED BY UNDERSTANDING, IS DEDICATED TO THE ADVANCEMENT OF LEARNING AND THE SEARCH FOR TRUTH; TO THE SHARING OF THIS KNOWLEDGE THROUGH EDUCATION FOR A DIVERSE COMMUNITY; AND TO THE APPLICATION OF THIS KNOWLEDGE TO BENEFIT THE PEOPLE OF THE STATE, THE NATION, AND THE WORLD. UMN Mission Statement - http://regents.umn.edu/sites/regents.umn.edu/files/policies/Mission_Statement.pdf
  • 13.
    DOES THIS CHANGE •The way we communicate • The way we solve problems • The way we use technology • What we build • How we build • When we build • …….Everything @ajdomie
  • 14.
    WHY NOW? SOFTWARE ISEATING THE WORLD! @ajdomie
  • 15.
  • 16.
    COMMON TRAITS OFWINNERS • Ship Fast, Ship Often • Fail Fast & Iterate • Prove theories & learn faster • They Win Together • Tesla: • Tesla software is no good without the physical car • No one will buy the car if it doesn’t look good • Battery life is critical • Airbnb & Lyft: • No available & secure payment solution, no drivers or hosts • No drivers or hosts, no platforms needed @ajdomie
  • 17.
    THE CORE OFDEVOPS •Culture •Automation •Measurement @ajdomie
  • 18.
  • 19.
  • 20.
  • 21.
  • 22.
    CULTURE • Ok, winas a team… got it. @ajdomie
  • 23.
  • 24.
    THIS IS REALLYHARD • Aligning to a greater goal • Team goals & incentives • Individual goals & incentives • Triple Down on Communication • “You mean I have to physically talk to them?” • (yes, that’s what I mean) • Effective communication takes time • Listening is usually more important • Safety in Failure • Fail fast & iterate • The perception of failure @ajdomie
  • 25.
    PRIORITY CONFLICT • Featurev. Bug • New Server Now v. Automate • Patch v. Build New • Prod is Broke v. Life @ajdomie
  • 26.
  • 27.
    WHERE TO START…. •Build empathy • Between teams • Between individuals • Sometimes offense needs to play defense • Build trust • Between teams • Between individuals • Service teams • Work as if everything your team builds is a service • Align your service with the mission @ajdomie Jeff Sussna Empathy the essence of DevOps: http://blog.ingineering.it/post/72964480807/empathy-the-essence-of-devops
  • 28.
  • 29.
    • Hey Danielle,It looks like the site is acting up and when looking around the only outlier I have found so far is a cpu spike on the DB. Can you help me investigate this a bit more? RESPOND AS A TEAM
  • 30.
    WHAT CAN YOUDO TOMORROW THAT CAN MAKE A DIFFERENCE?
  • 31.
  • 32.
    TECHNOLOGY MAKES SOMETHINGBETTER EVERY DAY!
  • 33.
    HOW DO WEBUILD IT? @ajdomie
  • 34.
    HOW DO WESECURE IT? @ajdomie
  • 35.
    HOW DO WESHIP IT? @ajdomie
  • 36.
    HOW DO WEOPERATE IT? @ajdomie
  • 37.
    SMARTER NOT HARDER* •If something is hard (but important), do it often • Provision servers • Provision DB’s • Test your code • Ship your code • Decrease Friction • Make building & shipping more consistent (not easy) • Limit Risk • Automated testing • Reduce the impact of failure • Phased deployments • Feature flags • Monitoring *Note: please continue to work hard (And smart  )@ajdomie
  • 38.
    DIGITAL TRANSFORMATION • Teamsthat ship fastest win • Theories are proven (or disproven), learn quickly • Fail fast (and safely) & iterate • Competitive advantage • Win together • Automate to deliver a service • No single service wins the game • Share and you learn faster @ajdomie
  • 39.
    WHAT CAN YOUDO TOMORROW THAT CAN MAKE A DIFFERENCE?
  • 40.
  • 41.
    ALIGN YOUR GOALS •Favor action and move forward • Fail fast and iterate • Time to value for your customer • Encourage technical pride • Seeing your efforts providing value in production is good for the soul • Limit risk to enable failing fast • Foster innovation @ajdomie
  • 42.
    DELIVER DELIGHTFUL SERVICE •If it moves, graph it! - @etsy • If it doesn’t move, graph it anyways! – also @etsy • Performance to promises • Incidents • Cause • Monitoring • Recover efforts @ajdomie
  • 43.
  • 44.
    WHAT CAN YOUDO TOMORROW THAT CAN MAKE A DIFFERENCE?
  • 45.
  • 46.
    RELIABILITY’S JOURNEY • 2years ago: • Single monitoring stack • Manual setup • Lacked proper variety of solutions • The Goal: Expand service offerings • Logs • User experience & availability • Application performance • Notification platform @ajdomie
  • 47.
    RELIABILITY’S JOURNEY • 1year ago: • Basic monitoring stacks • Manual setup • Custom monitors were time consuming • Consistency was a fun idea • The Goal: Monitoring as a Service • Establish a baseline that’s automated • Custom was easier • Consistency in solution types • Established experts of the service “monitoring” @ajdomie
  • 48.
    RELIABILITY’S JOURNEY • Today: •Monitoring Service • Event based serverless monitoring setup • Prescribed reusable patterns for various situations • Single pane of glass & improved collaboration • The Goal: Operations Automation Framework • Query-able documentation (Dependencies) • Beta: Advanced Visualization - Vizceral • Event based automation improving communication & transparency • Event based automation improving time to remediation • Encourage shared learnings and efforts, reusable functions @ajdomie
  • 49.
    RELIABILITY’S JOURNEY • Whathasn’t gone well? • Under communicated motivation and goals (yr1) • Generated a “Who are you to tell me” response @ajdomie
  • 50.
    RELIABILITY’S JOURNEY • Whathas gone well? • Reduced friction by simplifying serverless deployments • Limited risk with architecture (getting harder) • Failed fast & iterated • Double downed on transparency • Internal presentations • Blog posts • Advocates • All PRs welcomed • Invested in empathy & collaboration @ajdomie
  • 51.
    RELIABILITY’S RESULTS • Reducedincident rates & time to resolution • Positively impacted: • Customer experience • Team moral • Engineering productivity • Cross team collaboration • Technical pride • SPS Mission: To be the most chosen and relied upon retail network. @ajdomie
  • 52.
  • 53.
    TO DO LIST •Deliver Services (regardless of org) • Encourage reuse, care about customers • Solve problems at scale with tools • Leverage tools consistently • Be Empathetic • Create transparency to priorities & goals • Build trust • Fail Fast & Iterate • Change is a good thing, get good at it • Improve agility & velocity • Share • Knowledge • Experiences @ajdomie
  • 54.
  • 55.
    DEVOPS IS AVERB • It’s collaborating • It’s automating • It’s communicating • It’s iterating • It’s trusting each other • It’s winning @ajdomie
  • 56.
    WHAT CAN YOUDO TOMORROW THAT CAN MAKE A DIFFERENCE? (X3)
  • 57.
    “ ” DEVOPS ISN’T AGAME YOU PLAY UNTIL YOU WIN, IT’S A THING YOU DO THAT RESULTS IN WINNING.
  • 58.
    THANK YOU!! QUESTIONS, DEMOS,COMPLAINTS, ETC… @ajdomie
  • 59.
    ARTICLES • Your jobis not your job - Fred Kofman • https://www.youtube.com/watch?v=6OI7REyatq4 • Empathy the essence of DevOps - Jeff Sussna • http://blog.ingineering.it/post/72964480807/empathy-the-essence-of-devops • Promise Theory – Mark Burgess • “Scaling cooperation with Promises, not obligations” • http://markburgess.org/blog_cooperate.html • DevOps as an example of promise-oriented cooperation, without money @ajdomie

Editor's Notes

  • #6 Fred Kaufman – 5 seconds – One sentence what’s your job? Your job is not what you do… your job is the goal you pursue How many people defined your job as what you do? Help the university win
  • #13 So if I read this right, for everyone in this room, winning is simply to assist in advancing the the search for truth, and to share this knowledge in a way that can make the entire world a better place…… No big deal… :D
  • #15 This is just a word cloud of the top few DevOps Definitions available when googling the internet. It’s not prescriptive on HOW you should do these things, just that you should do them well. This is really the key.