The most important DevOps things I’ve learned over the last 4 years. I presented this at Agile 2013 in Nashville, TN. This talk is the talk and story referenced in Gene Kim's Devops Handbook (https://www.amazon.com/DevOps-Handbook-World-Class-Reliability-Organizations/dp/1942788002)
#agile #devops #automation #culture #distributedTeams #measurement #sharing #bestPractices
2. Howdy!
• I’m Karthik Gaekwad
• Senior Web Engineer
• Mentor Graphics Embedded
• From Austin,TX
• We have great BBQ,
chill people and
brutal summers!
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
3. My Background
• National Instruments (NI)
• Ecommerce Team (Agile)
• R&D Cloud Team (DevOps)
• Mentor Graphics Embedded
• Cloud Team (DevOps)
• Organizer: CloudAustin, DevOpsDays
Austin, TheAgileAdmin
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
5. The most important DevOps things I’ve
learned over the last 4 years
Let’s not waste time!
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
6. First, we have to talk about agile...
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
21. Results
Better products being delivered, at a faster
pace.
Methodology expanded to all other web
teams...
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
22. Results
I love my job! #ForReal
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
23. Agile: Retrospective
• Scrum brought team together.
• Building products on time, which were
actually needed.
• Superior collaboration between business
and IT.
• Organization shift -> Scrum
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
24. Ops conversations
“Agile is cool! but it’s
hard for ops to be
agile because it takes
months to buy and
provision servers.”
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
27. But in the community...
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
28. But in the community...
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
29. “We’re making cloud
products now”
“Go build us
cloud products
now”
Sincerely,
Management
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
30. “We’re making cloud
products now”
“Cloud is
AWESOME!”
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
31. “We’re making cloud
products now”
“Cloud is
OSSM”
-Dave Nielson
-NIST
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
32. “We’re making cloud
products now”
“Cloud is AWESOME
OSSM”
On-demand
Self-service
Scalable
Measurable
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
33. “Give me API or
give me death”
@littleidea
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
34. “We’re making cloud
products now”
“Agile DevOps
Cloud Team”
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
35. DevOps
“A cultural and professional movement”
Adam Jacobs
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
36. DevOps
“System administrators participating in an agile
development process alongside developers and using many
of the same agile techniques for their systems work.”
The Agile Admin Blog
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
37. Alison Chaiken
Embedded Linux,Thought Leader
“Get rid of the devs;
get rid of the ops; let’s
just find people who can
solve problems”
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
43. C
U
LT
U
R
E
But we suffered from
“Wall of confusionitis”
Devs: “REST Services!”
Ops: “System Stability!”
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
44. C
U
LT
U
R
E
Not seeing eye to eye
#Frustrating #Angry
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
46. C
U
LT
U
R
E
Really, this is “Platform Architecture”, but I’m too lazy to redo the image
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
47. C
U
LT
U
R
E
Devs and ops understand good architecture.
#GoodPlaceToStart
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
49. • Everyone understood architecture
• Helped everyone speak same language
• Reduced friction
• Called it “The System Model”
A
U
T
O
M
A
T
IO
N
#AgileDevOps@iteration1 Agile 2013
1st win:Architecture
Wednesday, August 7, 13
50. 1st win:Architecture
• “Model Driven Automation”
• Championed by ops
• Everyone needs to understand the model
• Figure out how to deploy the model
A
U
T
O
M
A
T
IO
N
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
51. 1st win:Architecture
• Programmable Infrastructure Environment
(PIE)
• Based on version controlled architecture
diagram
• Provisioned cloud boxes
• Ran deploys
• Set standard naming for boxes
• On demand deploy for apps
A
U
T
O
M
A
T
IO
N
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
52. Outcomes
• Ops learns Java!
• Ops learns version control!
• Developers get comfortable with
production environments!
• ...Now we’re speaking many common
languages!
A
U
T
O
M
A
T
IO
N
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
53. No Chef? No Puppet?
A
U
T
O
M
A
T
IO
N
Are y’all crazy???
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
54. A
U
T
O
M
A
T
IO
N
Everybody be cool
• We believe the cleanest abstraction layer is
starting with the entire system you’re trying
to model
• In 2009, chef and puppet were younger
• We were scared of Ruby
• We needed windows cloud support
• Plus dawg, cloud was a wild wild world then!
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
62. Tools Philosophy
D
EV
O
PS
• Needed to work for cloud
• Facilitate collaboration
• Charts and graphs
• Everyone wants to use it
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
63. Tools Philosophy
M
EA
SU
R
E
• Monitoring = Cloudkick
• Monitor servers
• Monitor code (custom synthetic checks)
We use #datadog now
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
64. Tools Philosophy
M
EA
SU
R
E
Teach your devs how to monitor their code,
and they will love the tool
Example: #NewRelic
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
65. Tools Philosophy
M
EA
SU
R
E
If it measures business metrics, management
will love you too!
#statsd #dogstatsd
HINT: Management pays you. Make them happy
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
67. Tools Philosophy
M
EA
SU
R
E
Get a logging solution.
Plethora of choices!
#Logstash #Graylog #Splunk #Sumologic
Which one? CloudAustin July Meetup
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
68. SH
A
R
IN
G
You’ll need more tools
Find them or write them!
You’re devops now!
Tools Philosophy
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
82. D
EV
O
PS
Test Everything
Test the code
Test the infrastructure
It’s all the same now
#infrastructureAsCode
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
87. D
EV
O
PS
Manual Changes
Don’t do this.
Do this:
1. Make manual changes in dev
2.Add to your chef/puppet script
3. Release to test/prod
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
88. DevOps 301
MOAR MOAR Tips
For when your team grows
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
89. Security in the loop
Add security audits as a part of your
sprint.
We do an audit periodically
#AgileDevOps@iteration1 Agile 2013
@Gauntlt @RuggedDevOps
Wednesday, August 7, 13
90. Security in the loop
Hire an infosec pro to your devops
team
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
91. Security in the loop
No budget?
Try a cross functional working group
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
92. Expanding Teams?
• Rotate times for meetings (status/planning)
• Scrum standups were recorded and dropbox(ed)
• “OK to call” philosophy
• Campfire chat (saves history)
• Ownership of new features
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
93. Expanding Teams?
• No secret: Face to face is always better
• Prioritize travels for visits to teammates
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
95. Everyone asking
for DevOps?
Make sure your team culture is strong
enough so you don’t lose person
permanently :)
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
97. DevOps 401
MOAR MOAR MOAR Ideas
For next time. I don’t have any.
Tweet #agiledevops for your best ideas!
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
98. STILL STUCK?
Just ask!
We’re devops, and this is real
(even though we can’t define it)
Twitter: #devopsama
http://devopsweekly.com/
Go to a devopsdays
https://groups.google.com/forum/#!forum/devops
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
99. DevOps: Retrospective
• DevOps and CAMS
bring team together.
• Building stable
products on time,
which were actually
needed.
• Superior collaboration
between devs and ops
• Organization shift ->
DevOps
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13
100. DevOps: Retrospective
• DevOps and CAMS
bring team together.
• Building stable
products on time,
which were actually
needed.
• Superior collaboration
between devs and ops
• Organization shift ->
DevOps
#AgileDevOps@iteration1 Agile 2013
• Scrum brought team
together.
• Building products on
time, which were
actually needed.
• Superior
collaboration between
business and IT.
• Organization shift ->
Scrum
Wednesday, August 7, 13
101. Thank you!
Let’s work together, and
solve the problems that
our business wants us
to!
#DevOps
#AgileDevOps@iteration1 Agile 2013
Wednesday, August 7, 13