6. Velocity Conference 2009
“10+ Deploys Per Day”
“Dev and Ops Cooperation at Flickr”
See:
• http://www.youtube.com/watch?v=LdOe18KhtT4
• Slides
@ablythe
7. Take Aways…
• Automated Infrastructure
• Shared Version Control
• One Step Build (and Deploy)
– They used Hudson, “Small Frequent Changes”
• Feature Flags
– AKA Branching in Code
• Shared Metrics/Shared Goal
• IRC and IM Robots
16. Things You Cannot Say
•
•
•
•
•
•
•
•
•
•
•
•
That's not my job.
(From a developer) Well, the tests all pass, so the problem must be in Operations.
(From operations) I did everything right, it must be a code problem.
Oh, production is down? I didn't know.
I built the production environment by hand. There's no reason to automate it
because we're only doing it once.
How close is QA to production? I don't know.
You need a new environment? Put in a service request and we'll see.
Your service request was approved. You should have a new machine in a few
weeks.
Why do you need another terabyte of storage? You're already using a terabyte!
Our DBAs won't let you install that server. We're a SQL Server shop.
I don't know what our slowest query is.
I don't know how many cores our production environment has.
18. Devops is not a plan, it’s a reaction
• Passionate people who get together
• We talk about Skunkworks projects and their
success
• Desire to do things better
• Donating time
19. Devops is not a judgment
• Talking about deploying 20 times a day is
because people are excited that they can do it
• This doesn’t mean that you have to do it!
• But if you chat with the dude/dudette who
can/is doing it, then I bet she is willing to
share insight you can benefit from.
20. Devops is not meant to be an
exclusive club
• There is not one place that you will find a
definition of DevOps
• All are invited, there is plenty of work to do…
21. Devops is not just a bunch of really
smart people
• Sure, there are some shining stars
• But mostly we’re just Dudes/Dudettes trying
to do our craft better every day
22. Devops is not a product
• You cannot just buy DevOps and install it…
• You cannot just instruct your team to go
DevOps… but you may encourage it…
23. Devops is not a run around traditional
IT
• DevOps should work within the system you
already have.
• I have no idea what ITIL is…
34. 3 key questions…
• To what extent does the organization define,
monitor and enforce some standardized
configuration strategy?
• To what extent does the organization define,
monitor and enforce a process culture?
• To what extent is restricted access to
production?
35. From Gene Kim
• On the one hand, it's astonishing that if you ask
those three questions and put them into a
spreadsheet, you can predict compliance
performance, security performance, IT
operational performance and project due date
performance.
• On the other hand, most people would agree that
it is common sense that these three controls are
prerequisites to good
performance. Unfortunately, common sense is
not common practice.
37. Carl Chesser
• beecham + blaze + splunk real-time search +
validation testing = win
• “each environment deployment of beecham was
around 16 min in execution time”
• “It was awesome to watch how beecham rolled
through the whole upgrade, seeing the splunk
real-time logs show as each service was
restarting, kicking off a series of tests with
minimal effort and having splunk bring back any
issues.”
38. Schon Thacker
• Linux: Directory Structure and "Best
Practices"
• https://connect.ucern.com/docs/DOC-176416
• Refer to this often
39. Kalin Hicks
• Comment on: Automated Alerting Strategy ?
• Basically laying out how to get text messages
from Zabbix for all KC Carriers.
40. Kyle McGovern
• LUKS Hadoop performance
• The cost savings to Cerner can be huge from
this.
41. Great… But what do you think DevOps
is?
• For me…
• It is the tight feedback loop.
44. Culture
• Be Positive and celebrate each others success
– :thumbsup:
– :shipit:
– :+1:
• Understand each others constraints
45. On Opsinfra… We Believe…
• A minimal number of associates need to directly
touch nodes (approaching zero)
• These tools matter – improve the lives of our
users
•
• Tools should be hardened - so that we can find
the root cause and drive corrective action
46. Need for Automation of Configuration
46K VM’s as of Q3 2013
simple exponential trend in Excel
75K by end of 2014
47. Measure
• Using Splunk on Millennium+ over 240 defects
were fixed or prevented last year
• Measure
– Downtimes length
– MTBF
– Performance
– Successes
– Everything…
48. Sharing
• Devops is a Movement
• Meetups like this are run on a shoe string
budget
• They are about sharing ideas, attend
them, blog, and most importantly have fun.
50. DevOps is a Movement
What is a Movement?
@ablythe
51. One thing I didn’t say…
That it is/will be easy
But the alternative is to leave here and go back
to your silo…
Editor's Notes
Where did the term come from? <click>Velocity Conference 2009 <click>Two Men from Flickr – Everyone familiar with Flickr?... Photo sharing site? Who has a Flickr account?One a Dev… John Allspaw <click>One an Ops… Paul Hammond <click>
John Allspaw and Paul Hammond (Flickr) “In the last week there were 67 deploys of 496 changes by 18 people” – Flickr Dev Blog, December 17th 2008.http://velocityconf.com/velocity2009/public/schedule/detail/7641
Remember the
When I started at Cerner 11 years ago, we pressed CD’s and mailed them to clients…Today Carl was able to push his code to production and test and monitor in 18 minutes…
Fill out rest of We Believe Statements
I’m going to take a tangent for 3 minutes to Let Derek Sivers, creator of CD Baby explain to you what a movement is…http://www.ted.com/talks/derek_sivers_how_to_start_a_movement.html