High-performing engineering teams regularly dedicate time on measuring the performance & quality of the systems and applications they’re building or on measuring & improving the various aspects of the development lifecycle. High-performing product companies are also data-driven when it comes to measuring the impact of new features & products in terms of business KPIs and Northstar metrics. Can a data-driven approach be applied to measuring the performance, maturity and continuous improvement of an engineering team or the whole engineering organisation? In this discussion we’ll cover various important topics related to quantifying the performance of an engineering organisation

1. Следете актуалните обяви за IT Management
Measuring the Productivity of
Your Engineering Organisation -
the Good, the Bad and the Ugly
Marin Dimitrov (Craft Docs)

2. Следете актуалните обяви за IT Management
Agenda
● Why it matters
● How to approach engineering productivity
● Popular frameworks
● Taking the first steps
About me
Director of Engineering
Previously with

3. Следете актуалните обяви за IT Management
Does It Matter?

4. Следете актуалните обяви за IT Management
Current Environment

5. Следете актуалните обяви за IT Management
Current Environment

6. Следете актуалните обяви за IT Management
Current Environment

7. Следете актуалните обяви за IT Management
How to Approach
Engineering Productivity

8. Следете актуалните обяви за IT Management
Measuring Productivity
● Why?
● Set clear goals for the process
○ visibility / awareness
○ alignment
○ continuous improvement
○ prioritisation of improvements
● What is your hypothesis?
● What will you do differently based on
the learnings?
● Who?
● Who is the target audience for the
learnings?
● Different level of details & language
for engineers, managers, senior eng
leadership, non-eng stakeholders
● Who is the decision maker on the next
steps?

9. Следете актуалните обяви за IT Management
Measuring Productivity
● How?
● 4 key dimensions
○ Velocity
○ Quality
○ Impact
○ Satisfaction
● quantitative vs qualitative approach
● What?
● code metrics (e.g. code churn / rework)
● flow metrics (e.g. cycle time)
● throughput metrics (e.g. number of
deploys)
● system metrics (e.g. MTTR)
● qualitative metrics (e.g. NPS)
● impact / outcome metrics

10. Следете актуалните обяви за IT Management
(mis)Measuring Productivity

11. Следете актуалните обяви за IT Management
Common Mistakes
● Not taking action on the issues &
inefficiencies
● Measuring for learning vs
performance (mis)management
● Creating incentives to “game the
system” (Goodhart’s Law)
● Not sharing the data / learnings with
eng teams
● Over-indexing on outputs (instead of
outcomes) or velocity (instead of value)
● Vanity metrics & success theatrics.
Measuring what’s easy vs what’s
important
● Measuring too many things
● Delaying measurement until
everything can be automated

12. Следете актуалните обяви за IT Management
Popular Frameworks

13. Следете актуалните обяви за IT Management
DORA
● https://dora.dev/
● measure 2 key aspects of the SDLC:
throughput vs stability
● may be extended with availability
metrics
● annual industry benchmark:
https://cloud.google.com/devops/stat
e-of-devops
● Deployment Frequency: frequency of
successful releases to production
● Lead Time for Changes: time between a
code merge and its deployable state
● Mean Time to Recovery: time between
the service interruption and its recovery
● Change Failure Rate: how often new
changes or hotfixes lead to failures after
deployment

14. Следете актуалните обяви за IT Management
DORA
https://dora.dev/research/

15. Следете актуалните обяви за IT Management
DORA
https://dora.dev/quickcheck/

16. Следете актуалните обяви за IT Management
DORA

17. Следете актуалните обяви за IT Management
SPACE
● “The SPACE of Developer Productivity:
There's more to it than you think”,
ACM Queue 03/2021
● Myths about developer productivity:
○ it’s about activity
○ it’s about individual performance
○ one metric can tell us everything
○ only managers care about it
○ it’s only about systems & tools
● Satisfaction & well being
● Performance
● Activity
● Collaboration & communication
● Efficiency & flow

18. Следете актуалните обяви за IT Management
SPACE
● SPACE is more of a “meta framework”
● the metrics are only suggested - you
can add new metrics to the
dimensions
● DORA & DevEx can be considered as
specific instances of SPACE

19. Следете актуалните обяви за IT Management
QUANTS (Google)
● Quality of the code
○ readability; test cases; evolution
● Attention from engineers
○ distractions & context switching
● Intellectual complexity
○ cognitive load; essential vs
accidental complexity
● Tempo and velocity
○ speed, throughput
● Satisfaction

20. Следете актуалните обяви за IT Management
DevEx
● “DevEx: What Actually Drives
Productivity”, ACM Queue 05/2023
● Feedback loops - speed, quality,
throughput, no hand-offs
● Flow State - focused time vs
disruptions, no context switching
● Cognitive Load - complexity, code
quality, documentation, self-serve
tools

21. Следете актуалните обяви за IT Management
Framework Adoption

22. Следете актуалните обяви за IT Management
Qualitative Feedback
● Provides a more holistic view on
SDLC challenges
● Can capture valuable unstructured
feedback & ideas
● Can provide context for explaining
or challenging quantitative data
● Human aspect - measuring the
satisfaction of the organisation
● Feedback may be biased (overly
positive / negative, skewed)
● Less regular cadence (to avoid the
“survey fatigue”)
● Limited areas can be covered by a
survey (in order to keep it short)

23. Следете актуалните обяви за IT Management
Qualitative Feedback
https://github.com/
TeamTopologies/Te
am-Cognitive-
Load-Assessment

24. Следете актуалните обяви за IT Management
Qualitative Feedback
https://getdx.com/guide/developer-experience-surveys/

25. Следете актуалните обяви за IT Management
Zooming Out

26. Следете актуалните обяви за IT Management
Agile Fluency Model
● https://www.agilefluency.org/
● developed by ThoughtWorks
● applicable at various levels: team,
group, organisation
● expected benefits: risk↘️, continuous
improvement, alignment↗️, ROI↗️,
productivity↗️
● Focussing - base level agile practices,
focus, transparency, alignment
● Delivering - delivery at speed, high
quality, ability to pivot
● Optimising - data driven,
experimentation, fast learning
● Strengthening - organisational design,
cross-team learning, shared practices

27. Следете актуалните обяви за IT Management
Agile Fluency Model

28. Следете актуалните обяви за IT Management
Agile Fluency Model

29. Следете актуалните обяви за IT Management
Agile Fluency Model

30. Следете актуалните обяви за IT Management
Taking the First Steps

31. Следете актуалните обяви за IT Management
Next Steps
1. Define the goals & your hypothesis -
what will you change, based on the
findings?
2. Start simple - pick a prescriptive
framework (like DORA) and run the
process e2e (measure - learn - change)
3. Be curious and don’t jump to
conclusions
4. Be transparent and make the team
co-own this process (instead of being
concerned about finger pointing /
performance mis-management)
5. Explain to stakeholders how developer
productivity impacts biz outcomes and
why changes may be needed
6. Augment the quantitative data with a
qualitative survey

32. Следете актуалните обяви за IT Management
Next Steps
7. Define the actions, based on the
findings. Focus on a couple of high-
impact / low-effort ideas first, and
don’t try to “boil the ocean“
8. Adapt your actions based on the
complexity of the context - simple vs
complicated vs complex
https://hbr.org/2007/11/a-leaders-
framework-for-decision-making

33. Следете актуалните обяви за IT Management
Next Steps
9. Don’t wait till everything is automated
10. Consider extending your
measurement approach, based on
descriptive frameworks (SPACE,
DevEX) or your own metrics
11. Zoom out to cultural, collaboration,
market understanding aspects (Agile
Fluency Model stages)

34. Следете актуалните обяви за IT Management
Next Steps
https://newsletter.pragmaticengineer.com/p/measuring-developer-productivity-bae

35. Thank you!
СЛЕДВАЩО СЪБИТИЕ
Лектор Дата Език
Следете актуалните обяви за IT Management
Онлайн събитие
Understanding infrastructure and availability
Боян Кроснов 15.02.24 Български език
Contacts:
https://linkedin.com/in/MarinDimitrov