Justin Carmony discusses the journey of integrating DevOps practices at Deseret Digital Media, emphasizing the importance of collaboration between developers and operations teams. He outlines challenges faced, such as a fragmented infrastructure, and the company's strategy to promote from within and spread DevOps knowledge among developers. The presentation also highlights the need for enhanced metrics, automated deployments, and a culture of continuous learning and responsibility for code performance.

1. Mentoring Devs
Into DevOps
Justin Carmony
Director of Development
Deseret Digital Media

3. Lets Measure
The Audience
• Who here is a…
• System Administrator?
• Developer?
• Manager / Management?
• “DevOp?”

4. Confession:
I’m a Developer

6. Self-Taught Ops
Because There Was No One Else To Do It

7. About Me
• Director of Development
for Deseret Digital Media
• Utah PHP Usergroup
President
• I Make (and Break)
Web Stuff (10 years)
• Salt User in Production
since 0.8
(I <3 Salt)

8. This Presentation
• Lessons learned at DDM & previous jobs
• Insight into our process of increasing “DevOps”
• We’re still learning, but this what we’ve found.
• Slides will be posted online, so don’t worry about
copying slide content.
• Feel free to ask on-topic questions during, and
we’ll have questions at the end.

9. About DDM
• Deseret Digital Media runs local website like
KSL.com, DeseretNews.com
• Running National and International Websites like
OK.com, familia.com.br, etc.
• ~10 million pageviews a day across sites.
• ~150 internal VMs, a few dozen physical
machines, some AWS sprinkled around.

10. Lets Start With a Story!

11. You Work for an
Awesome Tech Company

12. Team Is Working Hard to
Build New Things!

13. You launch your
awesome product!

14. A Few More Features…

15. … and next thing you know…

17. Awesome Job
Team, We Rock!

19. We Need !
Real-Time XYZ Feature!
ASAP!

20. We Need !
Real-Time XYZ Feature!
ASAP!

21. We Need !
Real-Time XYZ Feature!
ASAP!
&#$%!

24. “Huh, it works if you !
just turn off caching…”!
- Dev @ 80th Hour This Week

26. “I’m sure this !
will work…”

30. “Our servers are melting!”

37. “We Need a Better Solution!”

38. So…

39. Where Do We Start?

44. We Have This Problem

45. Challenges We Faced
• Giant mesh-up of technologies
• Tightly-coupled & fragile infrastructure
• Debugging production only bugs was difficult
• Bugs that were part code, part environment were
a nightmare to track down.

48. So One Day…
We Had A Genius Idea!

49. Lets Hire a DevOp!

50. I’m Not Joking
We Actually Said This

51. Two Problems
with this “Idea”

52. Problem #1 - We Didn’t Understand
What We Really Wanted

53. Step 1: Hire a DevOp!
Step 2: ????????????!
Step 3: Profit!

54. Step 1: Hire a DevOp!
Step 2: ????????????!
Step 3: Profit! Everything Works !
Perfectly!

55. Problem #2 - People Who Are Great
At Dev & Ops Are Hard To Find

56. Expectation:

57. Reality:

58. Honest Team Discussion:
What is it we’re really looking for?

59. We Discovered a Few Things

60. What does DevOps
Mean To Us?
• DevOps: Dev & Ops, a Culture of Collaboration
• Our Goal: “10 deploys a day without issues”
• Everyone shares the goal of quick development of
features AND a stable system that stays up.

61. Team Structure
Devs: 30 Ops: 2

62. Team Structure
Devs: 30 Ops: 2
DevOps: 1

63. Team Structure
Devs: 30 Ops: 2
DevOps: 1

64. Team Structure
Devs: 30 Ops: 2
DevOps: 1
Hiring one person won’t just solve all our problems!

65. Team Realizations
• Hardest problem already solved: awesome team
• No foreseeable rapid expansion, must operate at
our current scale
• Each Project’s Director of Development was
acting as the bridge between Dev and Ops, but
would become a bottleneck.

66. Teams Already Had Some
Ad-Hoc DevOps Tools
- Real-time Logging
- Capistrano Deploys
- Nagios Alerts
- Server Metrics
- Puppet for File Mgmt
- App Stats w/ Graphite
- Graphite Dashboards
- Salt for Cfg Management
- Homebrewed Metrics Sys.
- Homebrewed Alert System

67. Teams Already Had Some
Ad-Hoc DevOps Tools
- Real-time Logging
- Capistrano Deploys
- Nagios Alerts
- Server Metrics
- Puppet for File Mgmt
- App Stats w/ Graphite
- Graphite Dashboards
- Salt for Cfg Management
- Homebrewed Metrics Sys.
- Homebrewed Alert System

68. Step 1: Hire a DevOp!
Step 2: ????????????!
Step 3: Profit! Everything Works !
Perfectly!

69. Step 1: Hire a DevOp!
Step 2: ????????????!
Step 3: Profit! Everything Works !
Perfectly!

70. We Formed A Strategy

71. Step #1: Promote Dev
to DevOp Role

72. WAIT!
Isn’t that the advice you just
said was a bad idea?!

73. DevOp Engineer
• Well Defined Role:
• Ownership over the TOOLS to
improve DevOps efforts.
• Resource for other teams to
help use DevOps Tools.
• Easy to work with, aptitude for
systems & ops, likes to try
new things.

74. Promoting From Within
• A seasoned dev for your team already knows:
• Your Pain Points
• Your System’s Quirks
• How the “Chaos Works”
• Knows the people & personalities on your team

75. Step #2: Change
Team Structure

76. Team Structure
Devs: 30 Ops: 2

77. Team Structure
Devs: 30 Ops: 2

78. Team Structure
Goal: Spread Out Expertise By Increasing
Ops Experience & Skills Among Devs
Dev Ops

79. Team Structure
Goal: Spread Out Expertise By Increasing
Ops Experience & Skills Among Devs
Dev Ops

80. Team Structure
Dev Ops

81. Team Structure
Dev Ops

82. Increasing Ops Among Devs
• Identify Devs who liked “Ops” & wanted to Learn
• Pair Dev with Op / Director
• Learning Dev works on things, not Op /Director.
• Pair program if needed.

83. Step #3: Increase
Everyone’s Insight

84. Step #3: Increase
Everyone’s Insight

86. Metrics
• Everyone has access to Network, Server, and
Application Metrics.
• Consolidate & reduce places to look. We try to
pipe everything to StatsD / Graphite
• Each developer trained to add & track metrics in
production.
• We’re okay with 98% uptime of stats to avoid
complexity.

88. Real-Time Logging

89. Real-Time Logging
• Harder & more complicated at scale
• Still trying to solve well, we have lots of logs.
• Start with small window of data (i.e. 48 hours) and
start to expand window.
• We’re trying Logstash, ElasticSearch, and Kibana
right now.
• Generate Statistics off our Logs

90. Tracking Changes
• Everything, everything, everything in git
(we use GitHub)
• Everyone has access to all repos
• Everyone does work through Pull Requests
• Everyone has their work code reviewed *
* - Your can merge w/o a review, but must be
willing to defend your choice

91. Deploys

92. Everyone Can Deploy
• Automated our deployment process to a single
step.
• Everyone can deploy, deployments are logged
• Easy rollback is a requirement!
• Implementing feature flags to turn off single parts
of our application.

93. Tests
Unit
Functional
Integration
Acceptance
etc

94. Automated Tests
• If you want to trust your Devs, you need tests
• Legacy apps we wrote Integration Tests
• New Apps & Refactored Legacy Parts have Unit
Tests
• Continuous Integration to make sure tests run

96. So, where’s the salt?

97. Step #4: Devs
Use The Ops Tools

98. Devs can grok salt

99. Safe Environment
For Devs to Learn

100. Safe Environment
For Devs to Learn
salt * cmd.run "rm -rf /tmp /*"

101. Safe Environment
For Devs to Learn
salt * cmd.run "rm -rf /tmp /*"
Salt is awesome, but it can’t !
recover from that

102. Dev Salt Master
Devs Can Look Into Every Server

103. Dev Salt Master
• Every server has two minions:
• Admin Salt (aka root)
• Dev Salt (aka bob)
• Each connect to different master server:
• All Devs have access to Dev Salt Master
• Trusted Devs get access to Admin Salt Master

104. Everything Salty in Git
Reminder:

105. Dev Environment
• Developers own the Dev Environment
• Dev Teams manage the Salt States for their Env
• Vagrant + Salt for their Env
• Who makes changes? Developers
• DevOp helps advise & offer support

106. Team Structure
Dev Ops

107. Stage Environment
• Stage & Production use same salt repos, different
branches
• Developers make all the changes for Application
Servers
• All Changes through Pull Requests
• We’ll worry about env changes before code
• Small changes we quickly release, large or long
running branches are scary & dangerous

108. Production Environment
• Merge change to Production Branch
• salt * state.highstate
• Reminder: Small quick changes over time, never
a large change at once.

109. Environment Caveats
• Ops & DevOps Manage VM Hosts, Physical Load
Balancers, FireWalls, etc
• Ops & DevOps manage servers that deal with
data:
• MySQL
• MongoDB
• etc

110. Mentoring Devs

111. Mentoring Devs
• Not every Dev will become an amazing DevOp
• Thats okay!

112. Level of “DevOps” Skills
• Thinks about their impact on Ops: Everyone
• Able to debug issues with production: Most
• Able to make changes to environments: Many
• “Awesome DevOp”: Some

113. Team’s “DevOps” Skills
0
25
50
75
100
Think Debug Change DevOp
Current Goal

114. So Everything Is
Awesome for us, right?

115. Honesty Slide: We Have
Skeletons In Our Closets

116. Where We Are At
• All Dev Environments using Vagrant + Salt
• All New Stage & Prod Environments are Salty
• Some Legacy Stage & Production Envs are Salty
• Continuously working on getting out stuff salty.

117. Making This Work For Your Team

118. Honest Introspection
• Determine for your
team what are your…
• Strengths
• Weaknesses
• Problems
• Goals

119. Increase Team’s Insight
• Make sure devs can see
& understand how their
code performs
• Increase responsibility of
team for those metrics.
• If they break it, they fix it.
Do not always bail them
out.
• Everyone can see
everything.

120. Increase Team’s Insight
• Make sure devs can see
& understand how their
code performs
• Increase responsibility of
team for those metrics.
• If they break it, they fix it.
Do not always bail them
out.
• Everyone can see
everything.

121. Mentor Those With
Desire / Aptitude
• Give Developers Safe
Environment to Learn
• Let them submit code-reviewed
changes for Stage & Production
• When teaching / mentoring, let
the learner drive, kindly offer
advice and help.
• It takes time, but worth the
investment.

122. A Few Final Thoughts

123. Team Culture Matters

124. Positive Influence

125. Questions?

126. Thank You
Justin Carmony
Email: justin@justincarmony.com
Twitter: @JustinCarmony
IRC: carmony #salt #uphpu
Website: justin@justincarmony.com

127. p.s. we’re hiring, email / pm / tweet me