Agile Kolkata 2023 I EEBO Metrics in the Times of GenAI - Dinker Charak

© 2023 Thoughtworks | eebo.org | agilekolkata.com | 1
About This Deck for Agile Kolkata, 2023
Meta
Synopsis
The talk session explores the growth of EEBO Metrics
(Engineering Excellence to Business Outcomes) as a
way to underscores the value of Engineering
Excellence as an enabler of business outcomes.
However, as GenAI exploded into the scene and
disrupted many processes, the question we asked
ourselves was, is EEBO Metrics as a framework still
relevant? So we started inspecting the new SDLC from
the lens of EEBO Metrics and see if it does bring forth
the value GenAI brings to Engineering Excellence.
Key Takeaway
1. Measuring Engineering Excellence: The session emphasizes the
importance of fitness metrics to ensure technology investments
contribute to desired outcomes.
2. Correlation for Success: Using metrics to establish correlations
between engineering excellence and business outcomes provides
valuable insights for achieving success.
3. Bridge to Excellence: EEBO Metrics serve as a bridge between
engineering excellence and ROI, offering a clear path for measuring
and improving technology investments.
4. Glimpse of new SDLC in times of GenAI: Where and how can
GenAI be used in SDLC to accelerate time-to-market and journey
towards excellence.
5. Does GenAI lead to Better Outcomes: Using EEBO Metrics to
establish a before and after scenario can help establish the value
GenAI can bring to complex product development .

EEBO Metrics in
the times of GenAI

A banks's technology capabilities were hampering its growth.
The return on technology investment lagged its competitors.
The management invested in people, processes and product.
The metrics started looking great indicating improvements.
But the business was still not happy!

Speed is irrelevant,
if you are going in the wrong direction.

“We need a tool”
Focus is on Tool

AgileCraft Fortify SCA lizard SonarCloud
Azure DevOps GitHub LoadRunner SonarLint
Checkmarx GitLab LoadView SonarQube
Checkstyle HotJar Madge SpotBugs
Clang tidy
IBM AppScan
Sourcemeter
NeoLoad Targetprocess
Cloc IBM Rational AppScan NS-Metric Toxicity
Cobertura include: owasp Trello
Codacy JaCoCo
OWASP
Dependency-Check
Userlytics
Code Climate JDepend Pivotal Tracker UserTesting
CodeNarc Jira PMD UserZoom
Coverity Scan JMeter QAC VersionOne
Cppcheck jscpd Riscure WebLoad
Crazyegg Klocwork simian
FindBugs LeanKit SONAR

“We need to track this
metric”
Focus is on Metrics
Focus is on Tool

# of prod defects Code quality Design and toxicity of frequently used files No of defects found in production Story flow efficiency
% capacity aligned to strategic work Code review Speed Design Defect Density number of compliance audits passed TDD deviation
% late stage defect discovery Code smells Dev start to deployment ready - Story Cycle Time number of features developed per unit of time
Team Satisfaction Survey about the
Candidate
Actual Cost Code smells (new code) Duplications number of security vulnerabilities discovered and fixed Tech Debt
Actual vs Target hiring rate Code style adherence effectiveness of security training programs Offer-Joining Ratio Technical debt (new code)
Actual vs Target onboarding rate Coding Cycle time End to end tests Outage duration Test Automation Ratio
ADRs per repository Cognitive complexity Escaped Defect Rate per Release Outages Test case pass rate
Attribute Match Leading Indicator Commit hygiene Escaped defects P0-P2 Defect Leakage. -- Baseline P0-P1 Test coverage
Attrition rate By Exp, Skills, Dept,
Gender
Commit vs No. of build failures Failure statistics for tests Passing % Test distribution
Automated functional tests Conditional coverage Feature Velocity Variance Pen testing results Test execution time
Automated unit tests Conditional coverage (new code) Find root cause for defect Percentage of pull requests approved after code review Test run time for integration
Avg Builds Per day Consecutive focus hours per day Fix Cycle Pipeline hygiene Test-less commits
Avg Effort spent on Defects per
Sprint
Cost Performance Future Sensing Pipeline run / queue times & failures Throughput (TPS/TPM)
Avg Satisfaction Rating for New Hires Cost Variance Generated Value PR merge times Time taken to run tests
Avg time for defect resolution Coverage Get answers to an internal Information / technical query QA cycle time Time to deploy into production
BA cycle time Coverage (new code) Git contributions Quality of Onboarding Program Toxicity metrics
Backlog Health Customer reported Defects (Based on JIRA) Trainer Pool Creation Rate of adoption of paved paths & standards Traceability to Requirement
Branching Strategy Cycle time Improvement in Customer Experience Requirement Change Rate Training Coverage
Build Failure Rate Cycle time to roll out 1 offer Improvement in Efficiency and Effectiveness Requirement Stability Index Training Effectiveness
Build Time Cyclomatic complexity Increase in Influence Resiliency static code analysis UAT cycle time
Candidate Experience
Decrease COPQ (effort on Defects per
sprint)
Integration tests Response time Unfinished in flight stories
Capacity Defect Category Ratio Issues reported per (Module, Priority, Severity) rollback Unit Test coverage
Capacity loss Defect Density L3 incidents Run frequency of tests Unit tests duration
Change Cycle Defect Detection rate Latency Schedule Variance Validate a local code change works
Change fail % Defect fix vs New feature effort Launch a new service in production Scope creep Value Delivered per Release
Change Failure Rate Degree of encapsulation Layering rule adherence Scope Creep Impact Variation
CI Build time Degree of Pair Programming Lead Time Scope Volatility per sprint Work in progress
Code Complexity Dependency drift Level 1 Tech Interview rejection Security warnings
Code coverage Deployment Efficiency Mean Time to Recovery (MTTR) Skipped unit tests
Code duplication Deployment Frequency New Joinees per month Spillover per Sprint

“We need to tell a cohesive
story”
metric”
Focus is on Narration
Focus is on Metrics
Focus is on Tool

“We are not able to tell which
teams is delivering value and
which needs better alignment”
“We are not able to tell which
teams are doing well and
which need help & guidance”
“We are not able to decide
which metric to use to
showcase value to business”

“We need to relate to
business outcomes”
“We need to tell a cohesive
story”
metric”
Focus is on Purpose
Focus is on Narration
Focus is on Metrics
Focus is on Tool

Using Metrics to establish a correlation
Metrics reflecting
excellence in software
development
Metrics reflecting
excellence in
production deployment
Metrics categories
reflecting progress
towards desired
business outcomes

Characteristics of a good fitness metrics
Fitness Metric is
multivariate (calculated
from multiple data points)
Fitness Metric measures
an activity against its
desired goal
Fitness Metrics should
make success & failure
criteria clear
Fitness Metrics
are Fully Modeled
& agreed upon

Industry Reference
Remediation Plan
Ready
Threshold of Failure
Success Marker
Baselining Period Time
Monitoring Period
Face
Value Ownership Matrix
Fully Modelled Metrics: More than a number
Measurement

3 key metrics of excellence in
software development
4 key categories for business
outcomes
Build failure rate
Security warnings
Tech debt
Deployment frequency
Mean time to restore
Change fail percentage
Lead time
Market sensing
Improvement in efficiency &
effectiveness
Improvement in experience
Increase in influence
Using Metrics to establish a correlation

And
then
ChatGPT
happened,
changing everything.

GenAI driven SDLC
Maintenance
Deployment
Testing
Development
Design
Analysis
Planning
Research
● User research
● Market research
● Regulatory research
● Requirement analysis
● Security and compliance
● UI & process analysis
● Coding and implementation
● Unit and integration testing
● API development
● Data annotation & pipelines
● Project planning
● Data & migration planning
● Security & risk planning
● User centric design
● Policy and procedure design
● Solution Architecture
● Database design
● Functional test, E2E tests
● Load testing
● Compliance testing
● Deployment planning
● Monitoring and observability
● Automated deployment
● RCA and data
processing
● Bug fixing and
testing
● Technical debt
addressal
Quick Win HIgh Leverage

software development
4 key categories for business
outcomes
Build failure rate
Security warnings
Tech debt
Deployment frequency
Mean time to restore
Change fail percentage
Lead time
Market sensing
Improvement in efficiency &
effectiveness
Improvement in experience
Increase in influence
Using Metrics to measure GenAI driven
excellence

LEAD
TIME
TECH
DEBT
● Code quality
● Managing tech debt
● Optimal Branching strategy
● Design flexibility
● Automated test coverage
● Shift left in security and
performance
● Continuous Integration
● UAT
● Continuous delivery
● Configuration Management/
Provisioning
● Story prioritization
● Early feedback and
adaptation
● AB testing
● Outdated Codebase
● Deficient Initial Design
● Complex Dependencies
● Excessive Module Coupling
● Divergence from Intended
Architecture
● High Code Complexity
● Non-uniform Coding Practices
● Neglected Code Refactoring
● Increased Cognitive
Complexity
● Elevated Cyclomatic
Complexity
● Excessive Code Modifications
● Accelerated Development
Pace
● Absence of CI/CD Processes
● Inconsistent Configuration
Management
● Inefficient Branching
Strategies
● Deficient Documentation
● Concentrated Knowledge
Domains
● Incomplete Documentation
● Accumulating Bug/Issue
Reports
● Improper Feature Prioritization
● Costly Technical Debt
Remediation
● Significant Cost of Delay
● Delayed Security Integration
● Unresolved Security Issues
● Backlogged Performance
Issues
● Postponed Performance
Optimization
● Inadequate Test Coverage
● Manual Testing Over-reliance
● Insufficient Code Coverage
Code Quality Architecture/Design Testability
Security &
Performance
Delivery Infra & DevOps
Engineering
Excellence
GenAI improves Lead Time and Tech Debt

EEBO Metrics
are telling
a fascinating story.
For now, let me leave
you with this.

2023 Developer Survey
“77% feel favorable towards
using AI tools as part of their
development workflow”
StackOverﬂow
Survey
“70% see a benefit to using
AI coding tools at work”
GitHub

Dinker Charak
dinkerc@thoughtworks.com | www.eebo.org | www.ddiinnxx.com

1. Developer expertise is essential for evaluating the validity and accuracy
2. Prompting expertise is critical for generating high-quality prompts
3. Limitations in first-generation LLM tools can increase cognitive load due to
maintain context across interactions
Limitations

Agile Kolkata 2023 I EEBO Metrics in the Times of GenAI - Dinker Charak

Recommended

Recommended

More Related Content

Similar to Agile Kolkata 2023 I EEBO Metrics in the Times of GenAI - Dinker Charak

Similar to Agile Kolkata 2023 I EEBO Metrics in the Times of GenAI - Dinker Charak (20)

More from AgileNetwork

More from AgileNetwork (20)

Recently uploaded

Recently uploaded (20)

Agile Kolkata 2023 I EEBO Metrics in the Times of GenAI - Dinker Charak