Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

2

Share

Download to read offline

DevOps - It's About How We Work

Download to read offline

DevOps is far more about culture and organization than it is about technology and tooling. This talk will discuss the speaker's experiences leading high-performing engineering teams at Google, eBay, and Stitch Fix, and will offer suggestions for other organizations to level up their DevOps game.

https://www.meetup.com/SV-ELC/events/240087808/

Modern software-service models take advantage of the great benefits in having the same team both build the software as well as operate it in production -- "You Build It; You Run It" is the Amazon mantra. What does this mean in practice?

Organizationally, it means small teams with well-defined areas of responsibility, directly aligned with the business. The teams are cross-functional, meaning that each team has all the skill sets it requires to do its job, while at the same time relying on other teams for supporting services, tools, and libraries.

Process-wise, it means doubling down on practices like test-driven development and continuous delivery. Using continuous delivery practices, high-performing teams can and do release their applications and services multiple times a day. This enables them to iterate rapidly, experiment courageously, and fail more quickly.

Culturally, it means end-to-end ownership. Each team owns its software end-to-end, from design to development to deployment to retirement. The same engineers who are responsible for the features are responsible for quality, performance, operations, and maintenance. This ownership puts incentives in the right place to encourage building maintainable, observable, and operable systems from the start.

All these techniques and approaches are available to everyone, and practical examples in this talk will help other organizations on their journey.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

DevOps - It's About How We Work

  1. 1. DevOps It’s About How We Work Randy Shoup @randyshoup linkedin.com/in/randyshoup
  2. 2. Background • VP Engineering at Stitch Fix o Combining “Art and Science” to revolutionize apparel retail • Consulting “CTO as a service” o Helping companies scale engineering organizations and technology • Director of Engineering for Google App Engine o World’s largest Platform-as-a-Service • Chief Engineer / Distinguished Architect at eBay o Multiple generations of eBay’s infrastructure @randyshoup linkedin.com/in/randyshoup
  3. 3. Stitch Fix @randyshoup linkedin.com/in/randyshoup
  4. 4. Stitch Fix @randyshoup linkedin.com/in/randyshoup
  5. 5. Stitch Fix @randyshoup linkedin.com/in/randyshoup
  6. 6. Stitch Fix @randyshoup linkedin.com/in/randyshoup
  7. 7. Combining Art and [Data] Science • 1:1 Ratio of Data Science to Engineering o Almost 100 software engineers o Almost 100 data scientists and algorithm developers o Unique in our industry • Apply intelligence to *every* part of the business o Buying o Inventory management o Logistics optimization o Styling recommendations o Demand prediction • Humans and machines augmenting each other @randyshoup linkedin.com/in/randyshoup
  8. 8. Styling at Stitch Fix Personal styling Inventory @randyshoup linkedin.com/in/randyshoup
  9. 9. Personalized Recommendations Inventory Algorithmic recommendations Machine learning @randyshoup linkedin.com/in/randyshoup
  10. 10. Expert Human Curation Human curation Algorithmic recommendations @randyshoup linkedin.com/in/randyshoup
  11. 11. How do we work, and why does it work?
  12. 12. Modern Software Development Practices CultureTechnology Organization
  13. 13. Modern Software Development TDD and Continuous Delivery DevOpsMicroservices Small Teams
  14. 14. Modern Software Development TDD and Continuous Delivery DevOpsMicroservices Small Teams
  15. 15. Conway’s Law • Organization determines architecture o Design of a system will be a reflection of the communication paths within the organization • Modular system requires modular organization o Small, independent teams lead to more flexible, composable systems o Larger, interdependent teams lead to larger systems • We can engineer the system we want by engineering the organization @randyshoup linkedin.com/in/randyshoup
  16. 16. Small “Service” Teams • Amazon “2 Pizza” Teams o No team should be larger than can be fed by 2 large pizzas o Typically 4-6 people o Mix of junior and senior people • Aligned to Business Domains o Clear, well-defined area of responsibility o Single service or set of related services o Minimal, well-defined “interface” @randyshoup linkedin.com/in/randyshoup
  17. 17. Full-Stack Teams • All disciplines required for the team’s function o Design o Development o Quality and Performance o Maintenance o Operations • Symphony, not a Factory o Diversity of skills and talents o Working together more important than individual talent • Depend on other teams for supporting services, libraries, and tools @randyshoup linkedin.com/in/randyshoup
  18. 18. End-to-End Ownership • Teams own their roadmap • No separate maintenance or sustaining engineering team • Teams are long-term o Team owns service from design to deployment to retirement @randyshoup linkedin.com/in/randyshoup
  19. 19. Stable Points in Organization Size • ~5 o Everyone fits around a conference table o Single team, no structure o High bandwidth communication between individuals o Fluid roles • ~20 o Very difficult to manage as a single team, but possible o Need to introduce structure, but can be challenging to make it optimal / efficient o Potential trough of productivity and motivation • ~50-100+ o Requires shift from coordinating individuals to coordinating teams o High-bandwidth within teams, loose coupling between teams o Focus on team structure and responsibilities @randyshoup linkedin.com/in/randyshoup
  20. 20. Modern Software Development TDD and Continuous Delivery DevOpsMicroservices Small Teams
  21. 21. Test-Driven Development • Tests help you go faster o Tests “have your back” o Development velocity • Tests make better code o Confidence to break things o Courage to refactor mercilessly • Tests make better systems o Catch bugs earlier, fail faster @randyshoup linkedin.com/in/randyshoup
  22. 22. “Do you have time to do it twice?” “We don’t have time to do it right!”
  23. 23. Test-Driven Development • Do it right (enough) the first time o The more constrained you are on time and resources, the more important it is to build solid features o Build one great thing instead of two half-finished things • Right ≠ Perfect (80 / 20 Rule) •  Basically no bug tracking system (!) o Bugs are fixed as they come up o Backlog contains features we want to build o Backlog contains technical debt we want to repay @randyshoup linkedin.com/in/randyshoup
  24. 24. Transitioning to Testing • Write functional tests around a component o If you can only have a few tests, they should be meaningful ones • Fail any build that breaks a test • Opportunistically add tests o For every new bug, add a test that reproduces the bug and verifies the fix o For every new feature, add tests for that feature @randyshoup linkedin.com/in/randyshoup
  25. 25. Transitioning to Testing • Keep ratcheting up the level o E.g., Compiler warnings in eBay search Flickr user smurfie_77
  26. 26. Continuous Delivery • Most applications deployed multiple times per day • More solid systems o Release smaller units of work o Smaller changes to roll back or roll forward o Faster to repair, easier to understand, simpler to diagnose • Rapid experimentation o Small experiments and rapid iteration are cheap @randyshoup linkedin.com/in/randyshoup
  27. 27. Continuous Delivery • Enabled by o API-driven infrastructure (“cloud”) o PaaS o Containers • eBay 2-week trains vs. today @randyshoup linkedin.com/in/randyshoup
  28. 28. Triangle of Technical Tradeoffs • When you choose date and features, you implicitly choose a level of quality • Changing one changes the others • Be open and honest when you are making these tradeoffs Date QualityFeatures
  29. 29. Vicious Cycle of Technical Debt Technical Debt “No time to do it right” Quick- and-dirty
  30. 30. Virtuous Cycle of Investment Solid Foundation Confidence Faster and Better Testing
  31. 31. Modern Software Development TDD and Continuous Delivery DevOpsMicroservices Small Teams
  32. 32. Culture eats strategy for breakfast. -- Peter Drucker
  33. 33. Culture eats strategy and organization and technology and process and … for breakfast. -- me
  34. 34. Cross-Functional Collaboration • Open communication o Individuals encouraged to work directly with each other o Prefer informal cooperation over formal channels • Best decisions made through partnership o Agreement on goals and priorities makes it easier to agree on tactics o Given common context, well-meaning people will generally agree o “Disagree and Commit” • Solve problems instead of pointing fingers o Otherwise, playing strategy instead of solving the problem o Otherwise, avoiding blame and hiding the ball @randyshoup linkedin.com/in/randyshoup
  35. 35. None of us is as smart as all of us. -- Japanese proverb, as quoted by Bob Taylor
  36. 36. Goals of a Service Owner • Meet the needs of my clients … • Functionality • Quality • Performance • Stability and reliability • Constant improvement over time • … at minimum cost and effort • Leverage common tools and infrastructure • Leverage other services • Automate building, deploying, and operating my service • Optimize for efficient use of resources @randyshoup linkedin.com/in/randyshoup
  37. 37. Responsibilities of a Service Owner • End-to-end Ownership o Team owns service from design to deployment to retirement o No separate maintenance or sustaining engineering team • Autonomy and Accountability o Freedom to choose technology, methodology, working environment o Responsibility for the results of those choices @randyshoup linkedin.com/in/randyshoup
  38. 38. You Build It, You Run It. -- Werner Vogels
  39. 39. Service Relationships • Vendor – Customer Relationship o Friendly and cooperative, but structured o Clear ownership and division of responsibility • Customer Focus o Value of service comes from its value to its customers • Customer can choose to use service or not (!) o Must be strictly better than the alternatives of build, buy, borrow @randyshoup linkedin.com/in/randyshoup
  40. 40. Service-Service Relationships • Service-Level Agreement (SLA) o Promise of service levels by the provider o Customer needs to be able to rely on the service, like a utility • Charging and Cost Allocation o Charge customers for *usage* of the service o Aligns economic incentives of customer and provider o Motivates both sides to optimize for efficiency o (+) Pre- / post-allocation at Google
  41. 41. Blameless Post-Mortems • Post-mortem After Every Incident o Document exactly what happened o What went right o What went wrong • Open and Honest Discussion o What contributed to the incident? o What could we have done better?  Engineers compete to take personal responsibility (!)  “Finally we can fix that broken system”  @randyshoup linkedin.com/in/randyshoup
  42. 42. Blameless Post-Mortems • Action Items o How will we change process, technology, documentation, etc. o How could we have automated the problems away? o How could we have diagnosed more quickly? o How could we have restored service more quickly? • Follow up (!) @randyshoup linkedin.com/in/randyshoup
  43. 43. Failure is not falling down, but refusing to get back up. -- Theodore Roosevelt
  44. 44. Modern Software Development TDD and Continuous Delivery DevOpsMicroservices Small Teams
  45. 45. Thanks! • Stitch Fix is hiring! o www.stitchfix.com/careers o Based in San Francisco o Hiring everywhere! o More than half remote, all across US o Application development, Platform engineering, Data Science • Please contact me o @randyshoup o linkedin.com/in/randyshoup
  46. 46. Appendices Randy Shoup @randyshoup linkedin.com/in/randyshoup
  47. 47. Sharing Specialty Skills • Specialty Skills o Security, User Experience, Compliance, DBA, etc. o Often quite difficult to hire o Rarely need a full-time person on each team • Approach 1: Service Model o Make domain teams as self-sufficient as possible o Encode best practices into service / tool / library o Own those specialty services just like domain teams do @randyshoup linkedin.com/in/randyshoup
  48. 48. Sharing Specialty Skills • Approach 2: Shared Model o Single person shared among multiple teams • Approach 3: Coaching / Advisory Model o Specialty team is a pool of advisors o Provide special expertise as needed o Goal is to make domain teams self-sufficient @randyshoup linkedin.com/in/randyshoup
  49. 49. Team Anti-Patterns • Skill-based teams o Based around tiers or technologies (e.g., front-end team, application team, DBA team, Ops team) o (-) Every project crosses many team boundaries o (-) No end-to-end ownership of the system o (-) No end-to-end ownership of the customer experience • Project-based teams o Form ad-hoc team for a particular project, then disband o (-) No long-term ownership of code, product, service o (-) Encourages short-term approach instead of sustainable technical debt
  50. 50. Team Anti-Patterns • Large teams o (-) Teams larger than 8-10 should be split o (-) Communication and coordination overhead makes it increasingly difficult to sustain velocity
  51. 51. Effective Global Teams • Local Ownership o Well-defined area of responsibility o Clean interface with the rest of the organization • Individual teams are co-located o High-bandwidth communication within a team o Minimal coordination across teams
  52. 52. Global Team Anti-Patterns • Anti-Pattern: Split Teams Over Geographies o (-) Constant need for coordination over time zones o (-) Local conversations become disruptive rather than helpful o (-) No local pride of ownership • Anti-Pattern: Remote Team as Job Shop o (-) Constant need for management and task assignment o (-) Resentment between first-tier and second-tier sites o (-) No local pride of ownership o Ex. eBay remote offices vs. Google remote offices
  53. 53. Remote Teams • Fully remote *OR* fully co-located o Remote teams rely on virtual proximity (chat, hangouts, IRC) o Co-located teams rely on physical proximity (co-working) • Anti-Pattern: “Mostly” co-located o (-) Co-located majority ends up determining communication methods o (-) Remote individuals left out, less able to contribute, less productive
  54. 54. Feature Flags • Configuration “flag” to enable / disable a feature for a particular set of users o Independently discovered at eBay, Facebook, Google, etc. • More solid systems o Decouple feature delivery from code delivery o Rapid on and off o Develop / test / verify in production o Dark launches • Enables experimentation o A | B testing
  • whilpert

    Jun. 9, 2020
  • kikicarbonell

    Nov. 9, 2017

DevOps is far more about culture and organization than it is about technology and tooling. This talk will discuss the speaker's experiences leading high-performing engineering teams at Google, eBay, and Stitch Fix, and will offer suggestions for other organizations to level up their DevOps game. https://www.meetup.com/SV-ELC/events/240087808/ Modern software-service models take advantage of the great benefits in having the same team both build the software as well as operate it in production -- "You Build It; You Run It" is the Amazon mantra. What does this mean in practice? Organizationally, it means small teams with well-defined areas of responsibility, directly aligned with the business. The teams are cross-functional, meaning that each team has all the skill sets it requires to do its job, while at the same time relying on other teams for supporting services, tools, and libraries. Process-wise, it means doubling down on practices like test-driven development and continuous delivery. Using continuous delivery practices, high-performing teams can and do release their applications and services multiple times a day. This enables them to iterate rapidly, experiment courageously, and fail more quickly. Culturally, it means end-to-end ownership. Each team owns its software end-to-end, from design to development to deployment to retirement. The same engineers who are responsible for the features are responsible for quality, performance, operations, and maintenance. This ownership puts incentives in the right place to encourage building maintainable, observable, and operable systems from the start. All these techniques and approaches are available to everyone, and practical examples in this talk will help other organizations on their journey.

Views

Total views

1,327

On Slideshare

0

From embeds

0

Number of embeds

37

Actions

Downloads

39

Shares

0

Comments

0

Likes

2

×