Presentation about a new NEN initiative to create a Practice Guideline for output based performance management of Agile teams

1. NEN NPR 5333
Richard Sweer – Richard@finidy.nl – www.finidy.nl

2. Agenda
• Why a (new) standard
• Objective of NEN NPR 5333
• Scope
• What really matters the business
• Participant and approach

3. Why (new) standard - devil’s triangle/square
Fixed
Variable
Scope
Duration Costs
Quality

4. Why (new) standard – Agile a challenge for contract mngt.
In a ‘Time & Material
contract’, the client agrees
to pay the (internal)
supplier based on the time
spent by the supplier's
employees &
subcontractors to perform
the work and materials
used.
In a ‘Time & Material with
KPIs contract’, the client
agrees to pay the (internal)
supplier based on the time
spent by the (internal)
supplier's employees to
perform the work and
materials used, with the
condition that certain pre-
determined conditions for
success (KPIs) are met (e.g.
DoD).
An ‘output-based contract’ is an
agreement between a customer and the
(internal) supplier, which creates a
relationship for the delivery of services
or products. The driving force behind
the contract is that it focuses on what
the deliverables are in business terms
rather than how they should be
delivered.
Payment is per output delivered (e.g.
Function Points). The (internal) supplier
is not paid if the quality requirements
are not met (Definition of Done). Higher
productivity means more throughput
and more profit (win-win scenario).
With ‘outcome based’
contracting’, an agreement
is made that a (internal)
supplier or provider of
services must achieve
specific goals and is paid
only when those
objectives are met.
Payment is tied to % of
business benefits (reward
sharing).
Time & Material T&M with KPIs Output-based Outcome-based
Evolution of Agile contracting, enabled by 4KPIs and underlying metrics
low Maturity of Client and (internal) Supplier & risk of (internal) Supplier high
high Risk of Client regarding achieving quality, duration, costs and happiness of business users low

5. Why a (new) standard – SAFe did not help us!
Reference - Customer Stories: Putting SAFe to Work | Scaled Agile

6. I started with a better overview and coherence!
Scope
(Epics, Features, User Stories, …)
ISO 24765 Vocabulary
ISO 12207 Software life cycle processes (ISO 15288 System life cycle processes)
ISO 15289 Content of life-cycle information items (documentation) for ISO 12207/15288)
ISO 15939 Systems and software engineering — Measurement process
ISO 2501n Quality Model Division
ISO 25010 Software Product Quality model - “old” ISO 9126
ISO 25019 Quality in use model
ISO 2502n Quality Measurement Division
ISO 25021 Quality measure elements
ISO 25022 Measurement of quality in use - “old” ISO 9126-4
ISO 25023 Measurement of system and software product quality “old” ISO 9126-2 and -3
ISO 5055 Automated source code quality measures (cwe.mitre.org - 205 Weaknesses)
ISO 14143 Functional Size Measurements
ISO 24570 Nesma - Functional Size Measurement
ISO 20926 IFPUG - Functional Size Measurement
ISO 19761 COSMIC - Functional Size Measurement
ISO 25021 Quality measure elements
4KPIs - Better, Faster, Cheaper and Happier
- with 40 underlying metrics -
• Hours and costs per Function Point • Committed Story Points
• Realized Story Points • Defect Density
• Defect Removal Efficiency • Mean Time To Repair (MTTR)
• Mean Time Between Failures (MTBF) • Happiness score per Sprint
• Happiness score per release • Compiler warnings
• Abstract Interpretation • McCabe/Cyclomatic complexity
• Dead Code • Code and branch coverage
• Mutation test coverage • …
Costs
Duration
Scope
Quality
Happiness
General
model is expandable with the use of
these ISO standards
nominated to the NEN standard committee Software
and Systems engineering as a NEN NPR and ISO TR
Duration
(Deadlines, Planning,
Dependencies, …)
Costs
(People, Software,
Hardware, Data feeds, …)
Quality

7. Objective NEN NPR 5333
▪ Objectively and transparently measure the output and improvements
of Agile software development teams.
▪ By “objective” we mean that it refers to something that is based on facts, independent of
personal preferences or opinions and free of prejudice.
▪ By “transparent” we mean complete transparency in the metrics (measurement prescription,
measurement method, measurement value, traceability to source data, etc.).
▪ By “output” we mean the amount of business functionality that is delivered by the agile team
each period (e.g. sprint/iteration/Program Increment (PI)).
▪ By “improvement” here we mean the incremental improvement of the output (contributing to the
outcome) for the proposed/used metrics
▪ This model can then be used for contracting of Agile software development based on time
and material or output-based1)
1) With output-based contracting, the internal/external supplier is not paid per hour, but per delivered business functionality.

8. Target audience
• Guideline for Agile teams, user- and support organization
• Scrum en Kanban teams.
• Product Owner.
• Release train engineers (SAFe).
• Supplier management.
• Project managers.
• IT-controllers.
• (C-level) Management.

9. Scope
• Especially for custom software development.
• Primarily for agile software development
• Based on the quality characteristics according to ISO 25010.
• But also for:
• Standard package implementations (without the source code quality metrics)
• A waterfall way or a hybrid form of this (e.g. Water-Scrum-Fall).
• Contracting (agile) software development activities, both for time and material
and for output-based contracts.

10. What really matters the business!
Mean Time To Repair/Resolve
(MTTR)
Defect Density (DD)
How much, how agile, how fast? - delivered business functionality
(best indicator for business value)
Happiness
scores
Dead Code
Compiler warnings
McCabe/Cyclomatic complexity
Code and branch coverage
Committed & Realized Story Points
Hours per Function Point
Defect Removal Efficiency (DRE)
Abstract Interpretation
Mutation test coverage
Technical debt
Mean Time Between Failures
(MTBF)
% realized Story Points Function Points versus
Virtual Function Points
First time right
# Story points refined (# sprints)

11. More than 30 companies more than 60 participants
Software Secrets Uw specialist voor Functie Punt analyse

12. Agile approach
WG-4: Agile Contracting 3-4 months
WG-1: Metrics “Better” - (#20– 2-3 months)
WG-3: Metrics “Happier” - (#5 – 1-2 months)
WG-2 Metrics “Faster” en “Cheaper (#15 – 2-3 months )
Project plan
go/no-go decision
Start quotation process
1-2 months
W9
Sprint 1 - 4 weeks
W11
W10 W12 W13
Sprint 2 - 4 weeks
W14 W16
Sprint 3 - 4 weeks
W20 W21 tot W23
Sprint 4- 4 weeks
W24 W25 tot W27
Sprint 5 - 4 weeks
W28 W29 until W31
Sprint 6 - 4 weeks
W32
4th week of each sprint
• Monday to Thursday processing review, finishing sprint goal
and preparing presentation
• Retrospective per workgroup Sprint planning per workgroup
Presentation workgroups (5x 15 minutes)
3e week van elke sprint
Thursday and Friday
review by reviewers
Week 16
Prepare backlog
per workgroup and
sprint planning
PVA
V0.1
PVA
V0.2
PVA
V0.3
PVA
V0.4
PVA
V0.9
PVA
V1.0
Week 12
Final
participation
Week 15
Explain ‘best
practices’
Week 14
Final
Project plan
WG-0 Overarching activities
w15 W17 tot W19

13. Become a Nesma member and
get connected at
https://nesma.org/members/registration-form/
office@nesma.org
Nesma | LinkedIn