Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Why don'we just say "No" - Devopsdays Silicon Valley Jun 2014

1,870 views

Published on

Getting things done doesn't always start with "Yes".

Published in: Technology, Business

Why don'we just say "No" - Devopsdays Silicon Valley Jun 2014

  1. 1. Why don’t We just say no? ! ! ! Ge$ng!things!done!doesn't!always! start!with!"Yes". @dominicad! www.ddegrandis.com!
  2. 2. @dominicad! www.ddegrandis.com! For a number of reasons, it’s hard to say no.!
  3. 3. @dominicad! www.ddegrandis.com! #1.!!We!do!stuff!for!people!we!like.!!! !
  4. 4. @dominicad! www.ddegrandis.com! Ask!your!neighbor!this!quesCon….! !! “Why!do!you!take!on!more!work!than! you!have!capacity!to!do?”!!!
  5. 5. @dominicad! www.ddegrandis.com! Reason!#!2!–!We!are!team!players!
  6. 6. @dominicad! www.ddegrandis.com! Reason!#!3!–!Fear!from!those!in!posiCon!of!power!
  7. 7. @dominicad! www.ddegrandis.com! Reason!#!4!–!People!pleaser!
  8. 8. @dominicad! www.ddegrandis.com!
  9. 9. Why do Development teams seem to be able to say no more than Ops teams? !
  10. 10. Stuff Ops deals with that Dev traditionally doesn’t: •  Live prod issues •  DDOS attacks •  100% disk utilization •  access requests at 2:00 am •  opening up firewalls
  11. 11. Over the last 11 years, a lot of Dev teams have been given permission to say, “We’ve already done the planning for this week. Your next opportunity to get your request considered is next week.” Saying “No” has been institutionalized by using timeframes.
  12. 12. This doesn’t work for Ops very well… “Sorry - that live production fix will have to wait until next week” What if instead of using timeframes to say “No”, we used limits to say “No”?
  13. 13. A!look!at!one!team’s!evoluCon! 40 Ops Engineers (SysAdmin, DBA, Network, Mon, Sec) Tasked to build out/retrofit 6 data-centers across 6 diff countries. AND…. •  keep the lights on four existing data centers •  build out a new platform architecture •  support live issues (on-call) •  roll out a new configuration management tool •  deploy a crap load of new features •  deal with 3 reorgs over a 6 month period !! ! !
  14. 14. !!Problems: •  Conflicting priorities •  risky dependencies •  over-commitments •  interrupt-driven context switching resulting in missed commitments. •  Confusion around the new org structure !! ! !
  15. 15. How!we!got!started!on!road!to! improving!–!looked!at!3!data!pts! !! ! open closed #1 Can we keep up with the demand?
  16. 16. 0! 10! 20! 30! 40! 50! 60! 1! 2! 3! 4! 5! 6! 7! 8! 9! 10! 20! 30! 40! 50! 60! 70! 80! 90! 100! 110! 120! 130! 140! 150! 160! 170! 180! 190! 200! 250! 300! More! #"#ckets" # days it took for ticket to go from created to closed #2 Lead time – how long does it take to get work done?
  17. 17. ValidateDesign Deliver Done Implement doing Maint" Projects Implement done !! 96 #3!Where!is!work!stuck?!!
  18. 18. Customer!Mtg!Cme! !! Invited!customers!to!a!meeCng!and!showed!them!the!data.!! –  showed!them!the!demand!and!what!was!ge$ng!done!!! –  showed!them!the!Lead!Cme!! –  showed!!where!work!got!stuck! !! Customers!appreciated!the!visibility!into!Ops! !We!took!advantage!of!that!by!humbly!asking!for!their!help.! !beginning!with!all!of!those!Cckets!si$ng!in!the!validate!state.!!
  19. 19. Next!steps! The!introducCon!of!a!work`in`progress!(wip)! limit.!!Some!of!these!guys!had!20!–!40!Cckets!in! their!queue.!!We!asked!them,!“Does!this!seem! reasonable?”! ! How!about!10?!!! Let’s!head!in!that!direcCon!and!see!what! happens.!!! !
  20. 20. and!then!there!were!reorgs! The!1st!!org!restructure!change!created!the!A!Team!!`!to! focus!on!compleCng!projects!close!to!being!done,!but!sCll! hanging!on.! ! •  This!team!didn’t!have!to!respond!to!one!off!requests! and!wasn’t!supposed!to!be!on`call.! !! •  2nd!org!structure!change!split!Ops!into!3!teams!(live! Ops,!Build,!architecture)! •  Live`Ops!with!25%!of!team!and!60%!of!the!work!! ! !
  21. 21. •  access requests for systems, non-Zabbix monitor •  hardware investigation/verification/fixes -  vlan/port changes -  data retrieval (i.e. logs, network stats, etc) •  configuration triage - firewalls, load balancers, OS settings •  (small) capacity expansion •  verification of configs/services across shards •  database development consultation •  security compliance mitigation Live Ops tasks
  22. 22. Live!Ops!SRE!Changes!   •  Socialized wip limit idea over 6 months and gradually lowered it from 10 to 7 – out of 18 guys, average is 5-7. •  Hired 4 more people, although 2 got stolen by another team •  Closed out all tickets with no activity > 90 days •  Started saying “No” to last minute requests. !
  23. 23. Hi D.C., Team SRE has a very large number of changes scheduled for today already, and an even larger number of requests in our backlog that this request will displace if moved to the front of the queue. It would not be fair to other teams if we jumped on this immediately while planned work is pushed off. Monitoring should be a requirement for a service to go live, not a last minute addition. For us to fully support a live service, please implement monitoring before going live. For future requests, please give us as much notice as possible, and make sure to create a ticket (xxx.com) so we can prioritize and schedule the changes as necessary. Here's the ticket for this work…. Respectfully, A.H !
  24. 24. Live!Ops!SRE!changes!con’t!   •  Took time during standups to focus on kaizen improvements. •  Reduced validate state from 7 to 5 to 3 days. •  Found creative way to deal with walkups, and work done via personal relationships •  15 min daily sync up at 3pm instead of interrupting. •  5 min videos to present Ops review data to customers
  25. 25. A few simple rules
  26. 26. Clear definitions of done between queues
  27. 27. kaikzen/ ! noun, continuous improvement A business philosophy or system based on making positive changes on a regular basis.
  28. 28. “Disney thinks we can’t do it. Let’s prove them wrong.” -Steve Jobs
  29. 29. “Asking this much of people, even when they wanted to give it, was not acceptable.” ! !! ! ! ! !!`!Ed!Catmull!
  30. 30. “WIP it down to the personal safety level”!
  31. 31. “No” is an honorable reply to someone asking too much from you.
  32. 32. “The!takeaway!for!leaders!is! realizing!the!power!you!have! over!other!people!when!you!! ask!something!of!them.”! !!! !!!!!!!!!!!!!!!!!!!!!!!!!!`!Vaness!Bohns
  33. 33. ! But,!what!about!when!the!show!must! go!on?! ! World!Cup!finals!`!!Sunday,!July!13! !! fifa regulations ensure they have capacity. “pg 8 e) ensuring the presence of a sufficient number of ground staff and security stewards to guarantee safety.” Let’s not expect day shift workers to also cover the night shift.
  34. 34. @dominicad! www.ddegrandis.com! Honey-do list rules for saying no to the spouse…
  35. 35. @dominicad! www.ddegrandis.com! Credits! ! hqp://www.psychologytoday.com/blog/caveman`logic/201403/why`so` many`people`just`cant`say`no! ! The!Book!of!NO:!250!Ways!to!Say!It!—!and!Mean!It!—!And!Stop!People` Pleasing!Forever. hqp://www.amazon.com/The`Book`No`People`pleasing`Forever/dp/ 0071460780! ! Vanessa!Bohns!!`!!social!psychologist!and!professor!of!management! sciences!at!the!University!of!Waterloo!in!Ontario! hqp://hereandnow.wbur.org/2014/03/31/saying`no`psychology! ! eBay!arCcle!!hqp://www.uie.com/arCcles/death_of_relaunch/! ! Fifa!regulaCons! http://resources.fifa.com/mm/document/tournament/competition/01/47/38/17/ regulationsfwcbrazil2014_update_e_neutral.pdf !

×