Let's set aside the buzzwords for a moment and have an honest discussion about DevOps. There is the idea of putting more Dev into Ops, but just as crucial (if not more crucial) is getting your Devs to think more like Ops. Most developers have little to no experience dealing with production environments, and helping them add value to DevOps efforts can be difficult. This talk will cover practical ways of mentoring Devs into more DevOps skills and responsibilities. Ultimately, the goal is to help your Devs gain the skills leading to better production health, application performance and uptime. Of course, we'll also consider how SaltStack can help.

1. Mentoring Devs
Into DevOps
Justin Carmony
Director of Development
Deseret Digital Media

3. Lets Measure
The Audience
• Who here is a…
• System Administrator?
• Developer?
• Manager / Management?
• “DevOp?”

4. Confession:
I’m a Developer

6. Self-Taught Ops
Because There Was No One Else To Do It

7. About Me
• Director of Development
for Deseret Digital Media
• Utah PHP Usergroup
President
• I Make (and Break)
Web Stuff (10 years)
• Salt User in Production
since 0.8
(I <3 Salt)

8. This Presentation
• Lessons learned at DDM & previous jobs
• Insight into our process of increasing “DevOps”
• We’re still learning, but this what we’ve found.
• Slides will be posted online, so don’t worry about
copying slide content.
• Feel free to ask on-topic questions during, and
we’ll have questions at the end.

9. About DDM
• Deseret Digital Media runs local website like
KSL.com, DeseretNews.com
• Running National and International Websites like
OK.com, familia.com.br, etc.
• ~10 million pageviews a day across sites.
• ~150 internal VMs, a few dozen physical
machines, some AWS sprinkled around.

10. Lets Start With a Story!

11. You Work for an
Awesome Tech Company

12. Team Is Working Hard to
Build New Things!

13. You launch your
awesome product!

14. A Few More Features…

15. … and next thing you know…

17. Awesome Job
Team, We Rock!

19. We Need !
Real-Time XYZ Feature!
ASAP!

20. We Need !
Real-Time XYZ Feature!
ASAP!

21. We Need !
Real-Time XYZ Feature!
ASAP!
&#$%!

24. “Huh, it works if you !
just turn off caching…”!
- Dev @ 80th Hour This Week

26. “I’m sure this !
will work…”

30. “Our servers are melting!”

37. “We Need a Better Solution!”

38. So…

39. Where Do We Start?

44. We Have This Problem

45. Challenges We Faced
• Giant mesh-up of technologies
• Tightly-coupled & fragile infrastructure
• Debugging production only bugs was difficult
• Bugs that were part code, part environment were
a nightmare to track down.

48. So One Day…
We Had A Genius Idea!

49. Lets Hire a DevOp!

50. I’m Not Joking
We Actually Said This

51. Two Problems
with this “Idea”

52. Problem #1 - We Didn’t Understand
What We Really Wanted

53. Step 1: Hire a DevOp!
Step 2: ????????????!
Step 3: Profit!

54. Step 1: Hire a DevOp!
Step 2: ????????????!
Step 3: Profit! Everything Works !
Perfectly!

55. Problem #2 - People Who Are Great
At Dev & Ops Are Hard To Find

56. Expectation:

57. Reality:

58. Honest Team Discussion:
What is it we’re really looking for?

59. We Discovered a Few Things

60. What does DevOps
Mean To Us?
• DevOps: Dev & Ops, a Culture of Collaboration
• Our Goal: “10 deploys a day without issues”
• Everyone shares the goal of quick development of
features AND a stable system that stays up.

61. Team Structure
Devs: 30 Ops: 2

62. Team Structure
Devs: 30 Ops: 2
DevOps: 1

63. Team Structure
Devs: 30 Ops: 2
DevOps: 1

64. Team Structure
Devs: 30 Ops: 2
DevOps: 1
Hiring one person won’t just solve all our problems!

65. Team Realizations
• Hardest problem already solved: awesome team
• No foreseeable rapid expansion, must operate at
our current scale
• Each Project’s Director of Development was
acting as the bridge between Dev and Ops, but
would become a bottleneck.

66. Teams Already Had Some
Ad-Hoc DevOps Tools
- Real-time Logging
- Capistrano Deploys
- Nagios Alerts
- Server Metrics
- Puppet for File Mgmt
- App Stats w/ Graphite
- Graphite Dashboards
- Salt for Cfg Management
- Homebrewed Metrics Sys.
- Homebrewed Alert System

67. Teams Already Had Some
Ad-Hoc DevOps Tools
- Real-time Logging
- Capistrano Deploys
- Nagios Alerts
- Server Metrics
- Puppet for File Mgmt
- App Stats w/ Graphite
- Graphite Dashboards
- Salt for Cfg Management
- Homebrewed Metrics Sys.
- Homebrewed Alert System

68. Step 1: Hire a DevOp!
Step 2: ????????????!
Step 3: Profit! Everything Works !
Perfectly!

69. Step 1: Hire a DevOp!
Step 2: ????????????!
Step 3: Profit! Everything Works !
Perfectly!

70. We Formed A Strategy

71. Step #1: Promote Dev
to DevOp Role

72. WAIT!
Isn’t that the advice you just
said was a bad idea?!

73. DevOp Engineer
• Well Defined Role:
• Ownership over the TOOLS to
improve DevOps efforts.
• Resource for other teams to
help use DevOps Tools.
• Easy to work with, aptitude for
systems & ops, likes to try
new things.

74. Promoting From Within
• A seasoned dev for your team already knows:
• Your Pain Points
• Your System’s Quirks
• How the “Chaos Works”
• Knows the people & personalities on your team

75. Step #2: Change
Team Structure

76. Team Structure
Devs: 30 Ops: 2

77. Team Structure
Devs: 30 Ops: 2

78. Team Structure
Goal: Spread Out Expertise By Increasing
Ops Experience & Skills Among Devs
Dev Ops

79. Team Structure
Goal: Spread Out Expertise By Increasing
Ops Experience & Skills Among Devs
Dev Ops

80. Team Structure
Dev Ops

81. Team Structure
Dev Ops

82. Increasing Ops Among Devs
• Identify Devs who liked “Ops” & wanted to Learn
• Pair Dev with Op / Director
• Learning Dev works on things, not Op /Director.
• Pair program if needed.

83. Step #3: Increase
Everyone’s Insight

84. Step #3: Increase
Everyone’s Insight

86. Metrics
• Everyone has access to Network, Server, and
Application Metrics.
• Consolidate & reduce places to look. We try to
pipe everything to StatsD / Graphite
• Each developer trained to add & track metrics in
production.
• We’re okay with 98% uptime of stats to avoid
complexity.

88. Real-Time Logging

89. Real-Time Logging
• Harder & more complicated at scale
• Still trying to solve well, we have lots of logs.
• Start with small window of data (i.e. 48 hours) and
start to expand window.
• We’re trying Logstash, ElasticSearch, and Kibana
right now.
• Generate Statistics off our Logs

90. Tracking Changes
• Everything, everything, everything in git
(we use GitHub)
• Everyone has access to all repos
• Everyone does work through Pull Requests
• Everyone has their work code reviewed *
* - Your can merge w/o a review, but must be
willing to defend your choice

91. Deploys

92. Everyone Can Deploy
• Automated our deployment process to a single
step.
• Everyone can deploy, deployments are logged
• Easy rollback is a requirement!
• Implementing feature flags to turn off single parts
of our application.

93. Tests
Unit
Functional
Integration
Acceptance
etc

94. Automated Tests
• If you want to trust your Devs, you need tests
• Legacy apps we wrote Integration Tests
• New Apps & Refactored Legacy Parts have Unit
Tests
• Continuous Integration to make sure tests run

96. So, where’s the salt?

97. Step #4: Devs
Use The Ops Tools

98. Devs can grok salt

99. Safe Environment
For Devs to Learn

100. Safe Environment
For Devs to Learn
salt * cmd.run "rm -rf /tmp /*"

101. Safe Environment
For Devs to Learn
salt * cmd.run "rm -rf /tmp /*"
Salt is awesome, but it can’t !
recover from that

102. Dev Salt Master
Devs Can Look Into Every Server

103. Dev Salt Master
• Every server has two minions:
• Admin Salt (aka root)
• Dev Salt (aka bob)
• Each connect to different master server:
• All Devs have access to Dev Salt Master
• Trusted Devs get access to Admin Salt Master

104. Everything Salty in Git
Reminder:

105. Dev Environment
• Developers own the Dev Environment
• Dev Teams manage the Salt States for their Env
• Vagrant + Salt for their Env
• Who makes changes? Developers
• DevOp helps advise & offer support

106. Team Structure
Dev Ops

107. Stage Environment
• Stage & Production use same salt repos, different
branches
• Developers make all the changes for Application
Servers
• All Changes through Pull Requests
• We’ll worry about env changes before code
• Small changes we quickly release, large or long
running branches are scary & dangerous

108. Production Environment
• Merge change to Production Branch
• salt * state.highstate
• Reminder: Small quick changes over time, never
a large change at once.

109. Environment Caveats
• Ops & DevOps Manage VM Hosts, Physical Load
Balancers, FireWalls, etc
• Ops & DevOps manage servers that deal with
data:
• MySQL
• MongoDB
• etc

110. Mentoring Devs

111. Mentoring Devs
• Not every Dev will become an amazing DevOp
• Thats okay!

112. Level of “DevOps” Skills
• Thinks about their impact on Ops: Everyone
• Able to debug issues with production: Most
• Able to make changes to environments: Many
• “Awesome DevOp”: Some

113. Team’s “DevOps” Skills
0
25
50
75
100
Think Debug Change DevOp
Current Goal

114. So Everything Is
Awesome for us, right?

115. Honesty Slide: We Have
Skeletons In Our Closets

116. Where We Are At
• All Dev Environments using Vagrant + Salt
• All New Stage & Prod Environments are Salty
• Some Legacy Stage & Production Envs are Salty
• Continuously working on getting out stuff salty.

117. Making This Work For Your Team

118. Honest Introspection
• Determine for your
team what are your…
• Strengths
• Weaknesses
• Problems
• Goals

119. Increase Team’s Insight
• Make sure devs can see
& understand how their
code performs
• Increase responsibility of
team for those metrics.
• If they break it, they fix it.
Do not always bail them
out.
• Everyone can see
everything.

120. Increase Team’s Insight
• Make sure devs can see
& understand how their
code performs
• Increase responsibility of
team for those metrics.
• If they break it, they fix it.
Do not always bail them
out.
• Everyone can see
everything.

121. Mentor Those With
Desire / Aptitude
• Give Developers Safe
Environment to Learn
• Let them submit code-reviewed
changes for Stage & Production
• When teaching / mentoring, let
the learner drive, kindly offer
advice and help.
• It takes time, but worth the
investment.

122. A Few Final Thoughts

123. Team Culture Matters

124. Positive Influence

125. Questions?

126. Thank You
Justin Carmony
Email: justin@justincarmony.com
Twitter: @JustinCarmony
IRC: carmony #salt #uphpu
Website: justin@justincarmony.com

127. p.s. we’re hiring, email / pm / tweet me