Successfully reported this slideshow.
Your SlideShare is downloading. ×

Move Fast and Decide Things - Datadog Dash 2022.pdf

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad

Check these out next

1 of 22 Ad

Move Fast and Decide Things - Datadog Dash 2022.pdf

Download to read offline

When Greenlight had 30 engineers and a single monolith, it was possible for everyone to understand the entire context of the product. Engineers could get together to discuss a potential change and be confident in their understanding of the consequences and potential blast radius if things went wrong. Now, at around 300 engineers and 30+ microservices in production, it’s challenging for any one engineer to feel confident in either of those things. Tracking down who has knowledge of what can quickly lead to a “design by committee” failure case. How do we make sure that the decisions we are making won’t negatively affect other systems while continuing moving forward fast?

At Greenlight, we’ve implemented a framework of Decision Logs and Technical Specs that, coupled with strong principles on ownership, increases cross-team awareness of impactful decisions that are being made across the engineering organization.

In this talk we will explain how the framework works, how it was first introduced by the Infrastructure and Operations group, its benefits, and some of the challenges we found while implementing it and getting it adopted in other parts of the engineering organization.

By the end of the talk attendees will know whether incorporating some of these patterns into their organizations would help with scaling their engineering teams and how to get started, learning from our wins and our mistakes.

When Greenlight had 30 engineers and a single monolith, it was possible for everyone to understand the entire context of the product. Engineers could get together to discuss a potential change and be confident in their understanding of the consequences and potential blast radius if things went wrong. Now, at around 300 engineers and 30+ microservices in production, it’s challenging for any one engineer to feel confident in either of those things. Tracking down who has knowledge of what can quickly lead to a “design by committee” failure case. How do we make sure that the decisions we are making won’t negatively affect other systems while continuing moving forward fast?

At Greenlight, we’ve implemented a framework of Decision Logs and Technical Specs that, coupled with strong principles on ownership, increases cross-team awareness of impactful decisions that are being made across the engineering organization.

In this talk we will explain how the framework works, how it was first introduced by the Infrastructure and Operations group, its benefits, and some of the challenges we found while implementing it and getting it adopted in other parts of the engineering organization.

By the end of the talk attendees will know whether incorporating some of these patterns into their organizations would help with scaling their engineering teams and how to get started, learning from our wins and our mistakes.

Advertisement
Advertisement

More Related Content

Recently uploaded (20)

Advertisement

Move Fast and Decide Things - Datadog Dash 2022.pdf

  1. 1. Move Fast and Decide Things Datadog Dash 2022 Matt Farmer (he/him), Principal Engineer - Infrastructure & Operations matt.farmer@greenlight.me 10.13
  2. 2. Agenda: Move Fast and Decide Things ● Introduction ● Agenda ● Why are we talking about this? ● Decision Logs & Technical Specs ○ Overview of format and how it answers growing pains ○ How we implemented it ○ What went well & what didn’t ● Discussion / Q&A All stock imagery in this talk is from unsplash. Citations on final slide.
  3. 3. Documentation
  4. 4. Decision Rights
  5. 5. Meetings about the wrong things
  6. 6. Decision Logs & Technical Specs
  7. 7. Decision Logs & Technical Specs Tech Specs Decision Logs Decision Logs are directional. They plot a desired end state for a process, architecture, or technology choice and explain why. Tech Specs are documents that describe a solution to a specific problem. They don’t necessarily endorse the solution for other, related problems but serve as a collaboration point on the particular problem.
  8. 8. Common Elements Insert Section Subtitle Here The color green is the foundation of our brand. It is in our name, it is part of our brand DNA and it is the color the brand will lead with. The color green is the foundation of our brand. It is in our name, it is part of our brand DNA and it is the color the brand will lead with. Page Properties / Metadata Searchable metadata about each document. Things like: ● Author ● Date ● Scope of impact ● Current status (e.g. draft, review, accepted, rejected, replaced, etc) Objective & Problem Statement What is the thing we’re actually trying to solve? How bad is it affecting us? For technical specs: list out goals, non-goals, and anti-goals.
  9. 9. Insert Section Subtitle Here The color green is the foundation of our brand. It is in our name, it is part of our brand DNA and it is the color the brand will lead with. The color green is the foundation of our brand. It is in our name, it is part of our brand DNA and it is the color the brand will lead with. Page Properties / Metadata Searchable metadata about each document. Things like: ● Author ● Date ● Scope of impact ● Current status (e.g. draft, review, accepted, rejected, replaced, etc) Objective & Problem Statement
  10. 10. Problem Statement We need to assemble some cereal and need to decide how we’re going to do it. Objective Goals ● A delicious bowl of cereal. Non-Goals ● Coffee to go with the cereal. Anti-Goals ● Milk spilled all over the counter.
  11. 11. Common Elements Insert Section Subtitle Here The color green is the foundation of our brand. It is in our name, it is part of our brand DNA and it is the color the brand will lead with. The color green is the foundation of our brand. It is in our name, it is part of our brand DNA and it is the color the brand will lead with. Context What technical and business context is relevant to the decision making process. Depending on the scope this document this could include macroeconomic climate or minuta about code formatting. Assumptions & Constraints What are we taking to be true as a given as we go about proposing this decision or solution? What are some restrictions on the future impact of this decision or solution?
  12. 12. Context We are hungry. Cereal is a delicious breakfast food that we would like to eat. Assumptions and Constraints Assumptions ● You already have a clean bowl, cereal, and milk. Constraints ● This does not handle the case that your milk has gone bad.
  13. 13. Tech Spec Elements Insert Section Subtitle Here Risks Evaluation of how this solution could impact us from various perspectives. E.g. We consider: ● Data ● Scale ● Security ● And others Insert Section Subtitle Here Proposed Solution The centerpiece of the tech spec: the proposed solution. Use lots of diagrams and charts! Insert Section Subtitle Here Rejected Alternatives What did we also evaluate and then ultimately decide not to pursue?
  14. 14. Proposed Solution The proposed approach is to do the following: 1. Find a bowl. 2. Put cereal in the bowl. 3. Pour milk into the bowl with cereal.
  15. 15. Risks and Considerations Security This spec assumes you’ve locked the doors on your house and a bear will not enter your house and steal your cereal. Scale This approach should work for all quantities of cereal and milk provided that you have a big enough bowl.
  16. 16. Rejected Alternatives Not Eating Cereal We don’t have anything else to eat before lunch and we don’t want to be hangry for our morning meetings. Pouring Milk into the Bowl Before the Cereal We are skeptical that we could reliably achieve the correct cereal-to-milk ratio following this approach. If our priority was to have a ton of milk we’d just drink a glass of milk.
  17. 17. Decision Log Elements Insert Section Subtitle Here Options Considered What other things did we consider when making this decision, if anything. Insert Section Subtitle Here Decision What are we going to do. The centerpiece of the document. Insert Section Subtitle Here Consequences What has to change as a result of doing this?
  18. 18. Our Implementation ● Heavy emphasis on lazy consensus ● Confluence as the system-of-record ● Active communication about the introduction of these formats ● Published communication standards for communicating around these things ● Utilization of document formats wasn’t a requirement, but strong recommendation ● Lead by example: technical leadership corps starts using these heavily
  19. 19. Wins ● Good adoption within Infrastructure, Architecture, and several other engineering groups. ● Fewer surprises from the discussion of a topic to its implementation ● Meetings that could be an email were now an email (or a series of Slack messages and Confluence comments - tomato tomahto) ● Audit trail of point-in-time documents - automatic documentation about why we’re doing what we’re doing ● Lazy consensus keeps us moving forward
  20. 20. Misses ● Communicated a lot up front; haven’t done a great job of ensuring continuing education on the topic as folks join the organization. ● Some groups leaned a bit too far into the philosophy - sometimes creating documents when a JIRA ticket would have sufficed. ● No “owner” - we didn’t really designate ownership for re-evaluation of these document formats and ensuring that they remain useful to the entire organization.
  21. 21. Thank You greenlight.com #greenlight Feedback? Comments? Funny gifs? matt.farmer@greenlight.me
  22. 22. Unsplash Citations ● Iceberg by Alexander Hafemann on Unsplash ● Tax Forms by Markus Spiske on Unsplash ● Teamwork by Dylan Gillis on Unsplash

×