Where did the term come from?
Dev

Ops
@ablythe
Velocity Conference 2009
“10+ Deploys Per Day”
“Dev and Ops Cooperation at Flickr”
See:
• http://www.youtube.com/watch?v=LdOe18KhtT4

• Slides
@ablythe
Take Aways…
• Automated Infrastructure
• Shared Version Control
• One Step Build (and Deploy)
– They used Hudson, “Small Frequent Changes”

• Feature Flags
– AKA Branching in Code

• Shared Metrics/Shared Goal
• IRC and IM Robots
Patrick Debois
• http://www.jedi.be/blog/
• @patrickdebois
CAMS

• Culture
• Automation
• Measure
• Sharing
CAMS

• Culture (the hardest one)
• Automation
• Measure
• Sharing
Culture (Allspaw & Hammond)
•
•
•
•

Respect
Trust
Healthy Attitude about failure
Avoiding Blame
Michael Nygard

@mtnygard
http://thinkrelevance.com/blog/2014/02/17/culture-is-defined-by-what-you-cannot-say
Things You Cannot Say
•
•
•
•
•
•
•
•
•
•
•
•

That's not my job.
(From a developer) Well, the tests all pass, so the problem must be in Operations.
(From operations) I did everything right, it must be a code problem.
Oh, production is down? I didn't know.
I built the production environment by hand. There's no reason to automate it
because we're only doing it once.
How close is QA to production? I don't know.
You need a new environment? Put in a service request and we'll see.
Your service request was approved. You should have a new machine in a few
weeks.
Why do you need another terabyte of storage? You're already using a terabyte!
Our DBAs won't let you install that server. We're a SQL Server shop.
I don't know what our slowest query is.
I don't know how many cores our production environment has.
John Willis
• http://www.getchef.com/blog/2010/07/16/w
hat-devops-means-to-me/
• @botchagalupe
• Maintains: http://devopscafe.org/
Devops is not a plan, it’s a reaction
• Passionate people who get together
• We talk about Skunkworks projects and their
success
• Desire to do things better
• Donating time
Devops is not a judgment
• Talking about deploying 20 times a day is
because people are excited that they can do it
• This doesn’t mean that you have to do it!
• But if you chat with the dude/dudette who
can/is doing it, then I bet she is willing to
share insight you can benefit from.
Devops is not meant to be an
exclusive club
• There is not one place that you will find a
definition of DevOps
• All are invited, there is plenty of work to do…
Devops is not just a bunch of really
smart people
• Sure, there are some shining stars
• But mostly we’re just Dudes/Dudettes trying
to do our craft better every day
Devops is not a product
• You cannot just buy DevOps and install it…
• You cannot just instruct your team to go
DevOps… but you may encourage it…
Devops is not a run around traditional
IT
• DevOps should work within the system you
already have.
• I have no idea what ITIL is…
Lean/Kanban
CAMS

•Culture
•Automation
•Measuring
•Sharing
Lean Flow
Configuration Management
• Not just installation…
• State Management
• Idempotence
CAMS

•Culture
•Automation
•Measure
•Sharing
Will he talk about Etsy here?
• Measure Anything, Measure Everything
• http://codeascraft.com/2011/02/15/measureanything-measure-everything/
Will he talk about Etsy here?
• No…
• It’s a great blog… go read it.
Gene Kim

@RealGeneKim
http://www.realgenekim.me/it-controls-benchmark/
3 key questions…
• To what extent does the organization define,
monitor and enforce some standardized
configuration strategy?
• To what extent does the organization define,
monitor and enforce a process culture?
• To what extent is restricted access to
production?
From Gene Kim
• On the one hand, it's astonishing that if you ask
those three questions and put them into a
spreadsheet, you can predict compliance
performance, security performance, IT
operational performance and project due date
performance.
• On the other hand, most people would agree that
it is common sense that these three controls are
prerequisites to good
performance. Unfortunately, common sense is
not common practice.
CAMS

•Culture
•Automation
•Measure
•Sharing
Carl Chesser
• beecham + blaze + splunk real-time search +
validation testing = win
• “each environment deployment of beecham was
around 16 min in execution time”
• “It was awesome to watch how beecham rolled
through the whole upgrade, seeing the splunk
real-time logs show as each service was
restarting, kicking off a series of tests with
minimal effort and having splunk bring back any
issues.”
Schon Thacker
• Linux: Directory Structure and "Best
Practices"
• https://connect.ucern.com/docs/DOC-176416
• Refer to this often
Kalin Hicks
• Comment on: Automated Alerting Strategy ?
• Basically laying out how to get text messages
from Zabbix for all KC Carriers.
Kyle McGovern
• LUKS Hadoop performance
• The cost savings to Cerner can be huge from
this.
Great… But what do you think DevOps
is?
• For me…
• It is the tight feedback loop.
When is the best time to plant a tree?
When is the best time to plant a tree?
Culture
• Be Positive and celebrate each others success
– :thumbsup:
– :shipit:
– :+1:

• Understand each others constraints
On Opsinfra… We Believe…
• A minimal number of associates need to directly
touch nodes (approaching zero)
• These tools matter – improve the lives of our
users
•
• Tools should be hardened - so that we can find
the root cause and drive corrective action
Need for Automation of Configuration
46K VM’s as of Q3 2013

simple exponential trend in Excel
75K by end of 2014
Measure
• Using Splunk on Millennium+ over 240 defects
were fixed or prevented last year
• Measure
– Downtimes length
– MTBF
– Performance
– Successes
– Everything…
Sharing
• Devops is a Movement
• Meetups like this are run on a shoe string
budget
• They are about sharing ideas, attend
them, blog, and most importantly have fun.
Sharing
DevOps is a Movement

What is a Movement?

@ablythe
One thing I didn’t say…
That it is/will be easy
But the alternative is to leave here and go back
to your silo…

What is devops

  • 5.
    Where did theterm come from? Dev Ops @ablythe
  • 6.
    Velocity Conference 2009 “10+Deploys Per Day” “Dev and Ops Cooperation at Flickr” See: • http://www.youtube.com/watch?v=LdOe18KhtT4 • Slides @ablythe
  • 7.
    Take Aways… • AutomatedInfrastructure • Shared Version Control • One Step Build (and Deploy) – They used Hudson, “Small Frequent Changes” • Feature Flags – AKA Branching in Code • Shared Metrics/Shared Goal • IRC and IM Robots
  • 8.
  • 12.
  • 13.
    CAMS • Culture (thehardest one) • Automation • Measure • Sharing
  • 14.
    Culture (Allspaw &Hammond) • • • • Respect Trust Healthy Attitude about failure Avoiding Blame
  • 15.
  • 16.
    Things You CannotSay • • • • • • • • • • • • That's not my job. (From a developer) Well, the tests all pass, so the problem must be in Operations. (From operations) I did everything right, it must be a code problem. Oh, production is down? I didn't know. I built the production environment by hand. There's no reason to automate it because we're only doing it once. How close is QA to production? I don't know. You need a new environment? Put in a service request and we'll see. Your service request was approved. You should have a new machine in a few weeks. Why do you need another terabyte of storage? You're already using a terabyte! Our DBAs won't let you install that server. We're a SQL Server shop. I don't know what our slowest query is. I don't know how many cores our production environment has.
  • 17.
  • 18.
    Devops is nota plan, it’s a reaction • Passionate people who get together • We talk about Skunkworks projects and their success • Desire to do things better • Donating time
  • 19.
    Devops is nota judgment • Talking about deploying 20 times a day is because people are excited that they can do it • This doesn’t mean that you have to do it! • But if you chat with the dude/dudette who can/is doing it, then I bet she is willing to share insight you can benefit from.
  • 20.
    Devops is notmeant to be an exclusive club • There is not one place that you will find a definition of DevOps • All are invited, there is plenty of work to do…
  • 21.
    Devops is notjust a bunch of really smart people • Sure, there are some shining stars • But mostly we’re just Dudes/Dudettes trying to do our craft better every day
  • 22.
    Devops is nota product • You cannot just buy DevOps and install it… • You cannot just instruct your team to go DevOps… but you may encourage it…
  • 23.
    Devops is nota run around traditional IT • DevOps should work within the system you already have. • I have no idea what ITIL is…
  • 24.
  • 25.
  • 26.
  • 27.
    Configuration Management • Notjust installation… • State Management • Idempotence
  • 28.
  • 29.
    Will he talkabout Etsy here? • Measure Anything, Measure Everything • http://codeascraft.com/2011/02/15/measureanything-measure-everything/
  • 30.
    Will he talkabout Etsy here? • No… • It’s a great blog… go read it.
  • 31.
  • 34.
    3 key questions… •To what extent does the organization define, monitor and enforce some standardized configuration strategy? • To what extent does the organization define, monitor and enforce a process culture? • To what extent is restricted access to production?
  • 35.
    From Gene Kim •On the one hand, it's astonishing that if you ask those three questions and put them into a spreadsheet, you can predict compliance performance, security performance, IT operational performance and project due date performance. • On the other hand, most people would agree that it is common sense that these three controls are prerequisites to good performance. Unfortunately, common sense is not common practice.
  • 36.
  • 37.
    Carl Chesser • beecham+ blaze + splunk real-time search + validation testing = win • “each environment deployment of beecham was around 16 min in execution time” • “It was awesome to watch how beecham rolled through the whole upgrade, seeing the splunk real-time logs show as each service was restarting, kicking off a series of tests with minimal effort and having splunk bring back any issues.”
  • 38.
    Schon Thacker • Linux:Directory Structure and "Best Practices" • https://connect.ucern.com/docs/DOC-176416 • Refer to this often
  • 39.
    Kalin Hicks • Commenton: Automated Alerting Strategy ? • Basically laying out how to get text messages from Zabbix for all KC Carriers.
  • 40.
    Kyle McGovern • LUKSHadoop performance • The cost savings to Cerner can be huge from this.
  • 41.
    Great… But whatdo you think DevOps is? • For me… • It is the tight feedback loop.
  • 42.
    When is thebest time to plant a tree?
  • 43.
    When is thebest time to plant a tree?
  • 44.
    Culture • Be Positiveand celebrate each others success – :thumbsup: – :shipit: – :+1: • Understand each others constraints
  • 45.
    On Opsinfra… WeBelieve… • A minimal number of associates need to directly touch nodes (approaching zero) • These tools matter – improve the lives of our users • • Tools should be hardened - so that we can find the root cause and drive corrective action
  • 46.
    Need for Automationof Configuration 46K VM’s as of Q3 2013 simple exponential trend in Excel 75K by end of 2014
  • 47.
    Measure • Using Splunkon Millennium+ over 240 defects were fixed or prevented last year • Measure – Downtimes length – MTBF – Performance – Successes – Everything…
  • 48.
    Sharing • Devops isa Movement • Meetups like this are run on a shoe string budget • They are about sharing ideas, attend them, blog, and most importantly have fun.
  • 49.
  • 50.
    DevOps is aMovement What is a Movement? @ablythe
  • 51.
    One thing Ididn’t say… That it is/will be easy But the alternative is to leave here and go back to your silo…

Editor's Notes

  • #6 Where did the term come from? <click>Velocity Conference 2009 <click>Two Men from Flickr – Everyone familiar with Flickr?... Photo sharing site? Who has a Flickr account?One a Dev… John Allspaw <click>One an Ops… Paul Hammond <click>
  • #7 John Allspaw and Paul Hammond (Flickr) “In the last week there were 67 deploys of 496 changes by 18 people” – Flickr Dev Blog, December 17th 2008.http://velocityconf.com/velocity2009/public/schedule/detail/7641
  • #15 Remember the
  • #42 When I started at Cerner 11 years ago, we pressed CD’s and mailed them to clients…Today Carl was able to push his code to production and test and monitor in 18 minutes…
  • #46 Fill out rest of We Believe Statements
  • #51 I’m going to take a tangent for 3 minutes to Let Derek Sivers, creator of CD Baby explain to you what a movement is…http://www.ted.com/talks/derek_sivers_how_to_start_a_movement.html