Building Bridges: A DevOps Story
•Download as PPTX, PDF•
1 like•640 views
Join me as I recount my journey in helping introduce DevOps at D2L. Like many enterprise companies we had strong divides between Ops and Dev. Over the years the walls had been built up and the animosity grew. It wasn't until we decided to fully embrace AWS that we had to start building bridges instead of walls. In this talk I will take you through our journey and the tools, tips, and techniques that helped us heal the divide and begin working together instead of against each other. I will also talk about common pitfalls and mistakes that people make as they start out on the road to DevOps. This talk will help anyone working at a company where the developers and operations are not working together optimally.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (7)
Similar to Building Bridges: A DevOps Story
Similar to Building Bridges: A DevOps Story (20)
More from Matthew Campbell, OCT
Recently uploaded
Recently uploaded (20)
Building Bridges: A DevOps Story
- 1. Building Bridges: A DevOps Story Matt Campbell matthew.campbell@d2l.com @beardedcoder
- 2. Matt Campbell Engineering Director with D2L Leading project to achieve proper web-scale in AWS Previously lead move to monthly deployments matthew.campbell@d2l.com @beardedcoder
- 5. Millions of logins daily at peak TBs of aggregate data PBs of aggregate content Clients with Multi- TB DBs
- 19. Things you can neither change nor influence
- 21. Things you can change Things you can influence
- 29. Things you can change Things you can influence
- 30. Set a clear, achievable vision
- 35. Re-org to bring people together
- 47. Everyone has skin in the game
- 53. Mentorship
- 54. Be brave
- 61. Be empathetic
- 63. Regardless of what we discover, we understand and truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand. Norm Kerth
- 68. You don’t have to turn this into something. It doesn’t have to upset you Marcus Aurelius
- 70. 1. Set a clear, achievable vision 2. Re-Org to bring people together • Task Forces • Join in on an RCA and go deep • Physically co-locate • Grab lunch • Some people may not be with you when you are done, have those conversations early 3. Everyone has skin in the game • “Do it with me” versus “Do it to me” 4. Premortems • Delegation poker • Have a neutral party run these
- 71. 5. Mentorship 6. Be brave • Address problems early • Walk and talk • Appreciate that change is hard 7. Be more child-like • Challenge the status quo • Ask “Why?” • “That’s interesting … tell me more about that” • But also pick your battles 8. Be empathetic • Humanize the problem • Retro Prime Diretive • Embrace failure
- 72. Things you can change Things you can influence
- 73. Building Bridges: A DevOps Story Matt Campbell matthew.campbell@d2l.com @beardedcoder
Editor's Notes
- https://www.soswildlifecontrol.com/wp-content/uploads/2017/01/qtq80-vB3b6f.jpeg
- Who is D2L? What is it that we do? Image Copyright 2016 D2L Inc.
- We’re global https://upload.wikimedia.org/wikipedia/commons/0/09/BlankMap-World-v2.png
- We’re data intensive <See if you can get stats on logins, amount of data, size of DBs>
- When the company was small this worked just fine. Every had access to everything and people worked close (as in physically) together.
- As the company grew, we became larger. Becoming larger means you must become more specialized. Without someone guiding the growth of the company carefully you end up with various groups (factions) all trying to solve their own problem (often the same problem) in their own way.
- This causes walls to go up between the factions as we attempt to maintain control over our pocket of the solution. We also begin generating sub-cultures with their own dialects that are focused on what is important to them
- This leads to animosity between the groups as it seems as simple as “if they only knew the truth, they would work the way I do” https://pbs.twimg.com/media/DDNN8J4WAAElgZp.jpg
- In this talk we are going to focus on the fluffy stuff. A lot of DevOps talks about how to improve the tech to make everything smoother, but much of what doesn’t work involves people or processes.
- In my past project, I assumed that tech could solve all the problems
- However, CD project we still had many gaps between Dev and Ops Although we made huge improvements, much of it was done through back channels
- Didn’t cause the org changes we were hoping for, instead created localized pockets of streamlined behavior http://student.sbc.edu.hk/~111112/images/Streamlined%202.jpg
- Need to focus on people, process, and tech https://media.licdn.com/mpr/mpr/AAEAAQAAAAAAAAflAAAAJDE0ZjBmY2I5LWU3OGUtNGNkMC04MDVjLWViM2I4OGYxNzI0Mw.jpg
- But what do you pick?
- But what do you pick?
- But what do you pick?
- But what do you pick?
- But what do you pick?
- But what do you pick?
- There are no silver bullets
- Can’t I just DevOps it?
- No. It’s not that simple https://i.ytimg.com/vi/6w0jakILepk/maxresdefault.jpg
- However, CD project we still had many gaps between Dev and Ops Although we made huge improvements, much of it was done through back channels
- http://blog.crisp.se/wp-content/uploads/2016/05/Misalignment.png
- But what do you pick?
- https://pbs.twimg.com/media/CJ-wJPwWUAE5JNk.jpg
- I’ve heard a lot of people say our vision is “dealing great client experiences”. However that is not clear nor achievable
- We all come into things with different Experiences, Skills, Beliefs, and Motivations. Ops will view that goal as “maximize uptime” whereas Dev will view it as “Deliver features fast”
- http://static.wixstatic.com/media/88dc79_291d010ff0fa4439aac8be78fe27b33b~mv2.jpg
- https://static.deathandtaxesmag.com/uploads/2014/02/S4_E21_High_Five.png
- http://electric-cloud.com/wp-content/uploads/use-case-graphic_continuous-delivery.png
- ShadowOps - We were unable to bring our success rates to our goal via tech - Brought together the two teams - That month we hit the goal … without code changes to tie it to https://vignette.wikia.nocookie.net/warcommander/images/9/9e/SpecialOps-HeaderLogo-3.png/revision/latest/scale-to-width-down/300?cb=20140627010700
- If a permanent re-org isn’t possible, can you assemble a Task Force to dig into a problem that is currently plaguing your org?
- If even that isn’t possible, can you hijack the next RCA and bring Ops and Dev together to discuss in it? - Go blameless - Find where the actual problems are - Focus on How not Why - War Rooms are not Task Forces - Talk about NetApp perf issue https://www.thetreecenter.com/wp-content/uploads/tree-roots.jpg
- Ideally, you want people to tell you their stories. I like timelining as an approach for recording this https://fogsmoviereviews.files.wordpress.com/2011/11/20111126-170845.jpg
- If a re-org is outside of your control, can you physically move desks to sit closer together? If you can’t as a team, can you as an invidiual? http://i.imgur.com/9uO5owC.jpg
- Now is as good as anytime to talk about this. Not everyone will make it through this transformation. The early you can have these tough conversations the better it will be for everyone
- Do it with me versus do it to me http://adsolutions.yp.com/sites/default/files/styles/widescreen_large/public/iStock_000010606864Large.jpg?itok=o-0QkiQt
- Releasing Machinator as a server-prep tool - But we’re freeing you up to do more interesting things?!?!? - Basically telling this person their job is useless
- We created an initiative called Automate Joe. We did … and he ended up quitting (but for different reasons)
- https://i.kinja-img.com/gawker-media/image/upload/s--ccAitUxJ--/c_scale,fl_progressive,q_80,w_800/svwa9adt5co36kuoy3mh.jpg
- http://3.bp.blogspot.com/-_lIIz3N6r_k/VmelbEabuRI/AAAAAAAAA-E/QshK5u3yby0/s1600/07.%2Bpiggyback.png
- http://images.mentalfloss.com/sites/default/files/411363.jpg?resize=1100x740
- Address negative behavior immediately. Also be brave in addressing when you do not live up to intentions/commitments http://25.media.tumblr.com/a9dc2aaf291beb323a8c9f155b469bf6/tumblr_mvpcge6obA1r9hvapo3_1280.png
- https://quotefancy.com/quote/1243810/Grace-Murray-Hopper-Humans-are-allergic-to-change-They-love-to-say-We-ve-always-done-it
- Kids have an inate ability to ask “Why”. They also don’t care what others think about them
- Good technique is to say “That’s interesting, tell me more about that” http://s2.quickmeme.com/img/9d/9d56d82da6ff8d9d338e4c2918ab18bfee1f5c8372bee1be2d3cfbcced5b57f4.jpg
- Pick your battles (“It is safer to deploy slower”). Some things are ready to be won yet. http://media.fakeposters.com/results/2013/02/03/gwmg4o9vcm.jpg
- We are all afraid of failing. Help people embrace and learn from failure
- While you may not be able to bring down the walls, hopefully you can at least find better ways to work with them
- http://i.telegraph.co.uk/multimedia/archive/02209/clifton_2209817k.jpg
- So what did you pick?
- https://www.soswildlifecontrol.com/wp-content/uploads/2017/01/qtq80-vB3b6f.jpeg