SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps

927 views

Published on

Let's set aside the buzzwords for a moment and have an honest discussion about DevOps. There is the idea of putting more Dev into Ops, but just as crucial (if not more crucial) is getting your Devs to think more like Ops. Most developers have little to no experience dealing with production environments, and helping them add value to DevOps efforts can be difficult. This talk will cover practical ways of mentoring Devs into more DevOps skills and responsibilities. Ultimately, the goal is to help your Devs gain the skills leading to better production health, application performance and uptime. Of course, we'll also consider how SaltStack can help.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
927
On SlideShare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
38
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps

  1. 1. Mentoring Devs Into DevOps Justin Carmony Director of Development Deseret Digital Media
  2. 2. Lets Measure The Audience • Who here is a… • System Administrator? • Developer? • Manager / Management? • “DevOp?”
  3. 3. Confession: I’m a Developer
  4. 4. Self-Taught Ops Because There Was No One Else To Do It
  5. 5. About Me • Director of Development
 for Deseret Digital Media • Utah PHP Usergroup
 President • I Make (and Break) 
 Web Stuff (10 years) • Salt User in Production since 0.8 (I <3 Salt)
  6. 6. This Presentation • Lessons learned at DDM & previous jobs • Insight into our process of increasing “DevOps” • We’re still learning, but this what we’ve found. • Slides will be posted online, so don’t worry about copying slide content. • Feel free to ask on-topic questions during, and we’ll have questions at the end.
  7. 7. About DDM • Deseret Digital Media runs local website like KSL.com, DeseretNews.com • Running National and International Websites like OK.com, familia.com.br, etc. • ~10 million pageviews a day across sites. • ~150 internal VMs, a few dozen physical machines, some AWS sprinkled around.
  8. 8. Lets Start With a Story!
  9. 9. You Work for an Awesome Tech Company
  10. 10. Team Is Working Hard to Build New Things!
  11. 11. You launch your awesome product!
  12. 12. A Few More Features…
  13. 13. … and next thing you know…
  14. 14. Awesome Job Team, We Rock!
  15. 15. We Need ! Real-Time XYZ Feature! ASAP!
  16. 16. We Need ! Real-Time XYZ Feature! ASAP!
  17. 17. We Need ! Real-Time XYZ Feature! ASAP! &#$%!
  18. 18. “Huh, it works if you ! just turn off caching…”! - Dev @ 80th Hour This Week
  19. 19. “I’m sure this ! will work…”
  20. 20. “Our servers are melting!”
  21. 21. “We Need a Better Solution!”
  22. 22. So…
  23. 23. Where Do We Start?
  24. 24. We Have This Problem
  25. 25. Challenges We Faced • Giant mesh-up of technologies • Tightly-coupled & fragile infrastructure • Debugging production only bugs was difficult • Bugs that were part code, part environment were a nightmare to track down.
  26. 26. So One Day… We Had A Genius Idea!
  27. 27. Lets Hire a DevOp!
  28. 28. I’m Not Joking We Actually Said This
  29. 29. Two Problems with this “Idea”
  30. 30. Problem #1 - We Didn’t Understand What We Really Wanted
  31. 31. Step 1: Hire a DevOp! Step 2: ????????????! Step 3: Profit!
  32. 32. Step 1: Hire a DevOp! Step 2: ????????????! Step 3: Profit! Everything Works ! Perfectly!
  33. 33. Problem #2 - People Who Are Great At Dev & Ops Are Hard To Find
  34. 34. Expectation:
  35. 35. Reality:
  36. 36. Honest Team Discussion: What is it we’re really looking for?
  37. 37. We Discovered a Few Things
  38. 38. What does DevOps Mean To Us? • DevOps: Dev & Ops, a Culture of Collaboration • Our Goal: “10 deploys a day without issues” • Everyone shares the goal of quick development of features AND a stable system that stays up.
  39. 39. Team Structure Devs: 30 Ops: 2
  40. 40. Team Structure Devs: 30 Ops: 2 DevOps: 1
  41. 41. Team Structure Devs: 30 Ops: 2 DevOps: 1
  42. 42. Team Structure Devs: 30 Ops: 2 DevOps: 1 Hiring one person won’t just solve all our problems!
  43. 43. Team Realizations • Hardest problem already solved: awesome team • No foreseeable rapid expansion, must operate at our current scale • Each Project’s Director of Development was acting as the bridge between Dev and Ops, but would become a bottleneck.
  44. 44. Teams Already Had Some Ad-Hoc DevOps Tools - Real-time Logging - Capistrano Deploys - Nagios Alerts - Server Metrics - Puppet for File Mgmt - App Stats w/ Graphite - Graphite Dashboards - Salt for Cfg Management - Homebrewed Metrics Sys. - Homebrewed Alert System
  45. 45. Teams Already Had Some Ad-Hoc DevOps Tools - Real-time Logging - Capistrano Deploys - Nagios Alerts - Server Metrics - Puppet for File Mgmt - App Stats w/ Graphite - Graphite Dashboards - Salt for Cfg Management - Homebrewed Metrics Sys. - Homebrewed Alert System
  46. 46. Step 1: Hire a DevOp! Step 2: ????????????! Step 3: Profit! Everything Works ! Perfectly!
  47. 47. Step 1: Hire a DevOp! Step 2: ????????????! Step 3: Profit! Everything Works ! Perfectly!
  48. 48. We Formed A Strategy
  49. 49. Step #1: Promote Dev to DevOp Role
  50. 50. WAIT! Isn’t that the advice you just said was a bad idea?!
  51. 51. DevOp Engineer • Well Defined Role: • Ownership over the TOOLS to improve DevOps efforts. • Resource for other teams to help use DevOps Tools. • Easy to work with, aptitude for systems & ops, likes to try new things.
  52. 52. Promoting From Within • A seasoned dev for your team already knows: • Your Pain Points • Your System’s Quirks • How the “Chaos Works” • Knows the people & personalities on your team
  53. 53. Step #2: Change Team Structure
  54. 54. Team Structure Devs: 30 Ops: 2
  55. 55. Team Structure Devs: 30 Ops: 2
  56. 56. Team Structure Goal: Spread Out Expertise By Increasing Ops Experience & Skills Among Devs Dev Ops
  57. 57. Team Structure Goal: Spread Out Expertise By Increasing Ops Experience & Skills Among Devs Dev Ops
  58. 58. Team Structure Dev Ops
  59. 59. Team Structure Dev Ops
  60. 60. Increasing Ops Among Devs • Identify Devs who liked “Ops” & wanted to Learn • Pair Dev with Op / Director • Learning Dev works on things, not Op /Director. • Pair program if needed.
  61. 61. Step #3: Increase Everyone’s Insight
  62. 62. Step #3: Increase Everyone’s Insight
  63. 63. Metrics • Everyone has access to Network, Server, and Application Metrics. • Consolidate & reduce places to look. We try to pipe everything to StatsD / Graphite • Each developer trained to add & track metrics in production. • We’re okay with 98% uptime of stats to avoid complexity.
  64. 64. Real-Time Logging
  65. 65. Real-Time Logging • Harder & more complicated at scale • Still trying to solve well, we have lots of logs. • Start with small window of data (i.e. 48 hours) and start to expand window. • We’re trying Logstash, ElasticSearch, and Kibana right now. • Generate Statistics off our Logs
  66. 66. Tracking Changes • Everything, everything, everything in git 
 (we use GitHub) • Everyone has access to all repos • Everyone does work through Pull Requests • Everyone has their work code reviewed * * - Your can merge w/o a review, but must be willing to defend your choice
  67. 67. Deploys
  68. 68. Everyone Can Deploy • Automated our deployment process to a single step. • Everyone can deploy, deployments are logged • Easy rollback is a requirement! • Implementing feature flags to turn off single parts of our application.
  69. 69. Tests Unit Functional Integration Acceptance etc
  70. 70. Automated Tests • If you want to trust your Devs, you need tests • Legacy apps we wrote Integration Tests • New Apps & Refactored Legacy Parts have Unit Tests • Continuous Integration to make sure tests run
  71. 71. So, where’s the salt?
  72. 72. Step #4: Devs Use The Ops Tools
  73. 73. Devs can grok salt
  74. 74. Safe Environment For Devs to Learn
  75. 75. Safe Environment For Devs to Learn salt * cmd.run "rm -rf /tmp /*"
  76. 76. Safe Environment For Devs to Learn salt * cmd.run "rm -rf /tmp /*" Salt is awesome, but it can’t ! recover from that
  77. 77. Dev Salt Master Devs Can Look Into Every Server
  78. 78. Dev Salt Master • Every server has two minions: • Admin Salt (aka root) • Dev Salt (aka bob) • Each connect to different master server: • All Devs have access to Dev Salt Master • Trusted Devs get access to Admin Salt Master
  79. 79. Everything Salty in Git Reminder:
  80. 80. Dev Environment • Developers own the Dev Environment • Dev Teams manage the Salt States for their Env • Vagrant + Salt for their Env • Who makes changes? Developers • DevOp helps advise & offer support
  81. 81. Team Structure Dev Ops
  82. 82. Stage Environment • Stage & Production use same salt repos, different branches • Developers make all the changes for Application Servers • All Changes through Pull Requests • We’ll worry about env changes before code • Small changes we quickly release, large or long running branches are scary & dangerous
  83. 83. Production Environment • Merge change to Production Branch • salt * state.highstate • Reminder: Small quick changes over time, never a large change at once.
  84. 84. Environment Caveats • Ops & DevOps Manage VM Hosts, Physical Load Balancers, FireWalls, etc • Ops & DevOps manage servers that deal with data: • MySQL • MongoDB • etc
  85. 85. Mentoring Devs
  86. 86. Mentoring Devs • Not every Dev will become an amazing DevOp • Thats okay!
  87. 87. Level of “DevOps” Skills • Thinks about their impact on Ops: Everyone • Able to debug issues with production: Most • Able to make changes to environments: Many • “Awesome DevOp”: Some
  88. 88. Team’s “DevOps” Skills 0 25 50 75 100 Think Debug Change DevOp Current Goal
  89. 89. So Everything Is Awesome for us, right?
  90. 90. Honesty Slide: We Have Skeletons In Our Closets
  91. 91. Where We Are At • All Dev Environments using Vagrant + Salt • All New Stage & Prod Environments are Salty • Some Legacy Stage & Production Envs are Salty • Continuously working on getting out stuff salty.
  92. 92. Making This Work For Your Team
  93. 93. Honest Introspection • Determine for your team what are your… • Strengths • Weaknesses • Problems • Goals
  94. 94. Increase Team’s Insight • Make sure devs can see & understand how their code performs • Increase responsibility of team for those metrics. • If they break it, they fix it. 
 Do not always bail them out. • Everyone can see everything.
  95. 95. Increase Team’s Insight • Make sure devs can see & understand how their code performs • Increase responsibility of team for those metrics. • If they break it, they fix it. 
 Do not always bail them out. • Everyone can see everything.
  96. 96. Mentor Those With Desire / Aptitude • Give Developers Safe Environment to Learn • Let them submit code-reviewed changes for Stage & Production • When teaching / mentoring, let the learner drive, kindly offer advice and help. • It takes time, but worth the investment.
  97. 97. A Few Final Thoughts
  98. 98. Team Culture Matters
  99. 99. Positive Influence
  100. 100. Questions?
  101. 101. Thank You Justin Carmony Email: justin@justincarmony.com Twitter: @JustinCarmony IRC: carmony #salt #uphpu Website: justin@justincarmony.com
  102. 102. p.s. we’re hiring, email / pm / tweet me

×