Metrics for Mofel-Based Systems Development

®
IBM Software Group
© 2013 IBM CorporationInnovation for a smarter planet
Metrics
You can’t control what you don’t measure
Dr. Bruce Powel Douglass, Ph.D.
Chief Evangelist
Global Technology Ambassador
Bruce.Douglass@us.ibm.com
Twitter: @BruceDouglass
Yahoo: http://tech.groups.yahoo.com/group/RT-UML

IBM Software Group | Rational software
Innovation for a smarter planet
Agenda
 Introductions
 On the importance of being metric
 Types of metrics
Design Metrics
Quality Metrics
Project Metrics
 Q&A

Dr. Bruce Powel Douglass
Chief Evangelist
Global Technology Ambassador
IBM Rational

Innovation for a smarter planet 4
On the Importance of Being Metric
Companies that
measure
Companies that don’t
measure
On-time projects 75% 45%
Late projects 20% 40%
Cancelled projects 5% 15%
Defect removal >95% Unknown
Cost estimates Accurate Optimistic
User satisfaction High Low
Software status High Low
Staff morale High Low
Source: Applied Software Measurement 3rd Edition by Capers Jones 2008
Results improve when measurements are used

On the Importance of Being Metric
 A metric is a measurement of something important to you
 A metric includes
 A scalar value which is measured
 Range
 Units
 Consistent technique for measurement acquisition
 Standard analysis of outcome
 Metrics should be
 Easy to measure
 Correlate with the actual information desired
 Have easy to understand appropriate interpretation for decision makers
 Example: BMI
 BMI = Weight in Kilograms / ( Height in Meters)2
 What is the BMI for a body builder?
 Example: LOC
 Lines of code as a measure of work performed
 If I optimized from 700K lines to 500K lines have I done negative work?

On the Importance of Metrics
 Metrics are used
To understand an aspect of a project or system
 Static metric measures a static property of system (e.g.
size, complexity, defect density)
 Dynamic metric measures something that changes (e.g.
velocity, iteration burndown, defect rate)
To reduce risk
To answer a question
To enable informed decision making
 Metrics are best used to change from open-loop (ballistic)
decision making to closed loop (dynamic evidence-based)
decision making
Plan
Act

Goal-Based versus Plan-Based Metrics
 A goal is a designed outcome
 Plans try to achieve goals but may themselves but suboptimal or in error
 Ergo, Goal-based metrics are generally preferred to plan-based metrics
 Plan-based metrics
On planned schedule
 Metric: Number of hours worked on project
 Metric: Number of lines of code generated
 Metric: Weight lost
 Goal based metrics
Progressing towards system delivery
 Metric: Requirements delivered/designed/implemented/verified/validated
 Metric: Amount of verified functionality delivered
 Metric: body fat percentage

When are metrics useful?
When there is
a consensus
on what to
measure
When all
relevant
measurements
are made
When the
measurement
is timely
When the
measurement
correlates to the
desired
information
When the
measurement is
precise and
accurate
enough
When it allows
you to make an
informed correct
decision
When all
performers
know how to
make the
measurement
When the
measurement is
properly
analyzed
When the
analysis leads
to proper action

Key Metric Success Factors
Clear and easy
performance of
measurement
Training for
relevant
personnel for
measurement
performance
Support for
metrics at all
project levels
Governance and
enforcement of
metric policies
Clear assignment
of roles and
responsibilities
Outcomes
displayed in ways
meaningful to
consumers
Retention of
metrics for
historical
reference
Consistent use of
outcomes for
decision making
Early
establishment of
metrics

Breaking Bad (metrics)
Lack of
consensus and
buy in
Measurements
inconsistently
gathered
Measurements
inaccurate or
lack consistent
accuracy
Measurements
gathered too late
Analysis
inappropriate for
metric
Analysis not
applied or is
ignored
Metric not
retained or
referenced
Metric not highly
correlated with
desired
information

Use dashboards to provide summary views
11

Best Metric Practices
 Explicitly link metrics to goals
 Define what will be done with the results
 Train staff in metrics gathering, analysis, and interpretation (as appropriate)
 Prefer trends over statics
 Use short plan  act  measure  analyze  plan… cycles
 Audit metric acquisition to ensure consistency of data
source, methods, frequency, and analysis
 Change metrics when they stop driving improvements
 Automate metric acquisition and analysis where possible
 Render the analytic results in a form useful to the consumers of the metric
 Apply intelligence to interpretation, don’t just blindly accept the obvious conclusion
 Use previously captured metrics as guidelines for future planning but don’t
slavishly follow them

Type of metrics
Project
Quality
Design

Project Metrics
 Project metrics measure project concerns
Earned Value
Schedule variance
Velocity
Iteration burndown
Release burndown
Requirements churn / enhancement request trend
Age of enhancement (responsiveness metric)
Cost per unit work
Project
Quality
Design

Earned Value
 Planned value (AKA Budgeted Cost of Work Scheduled BCWS)
 Earned Value is how much value has been created so far:
 Application
Define the work as a set of mutually exclusive tasks
Assign a Planned Value to each task (when complete)
Define earning rules (0/100 simplest rule)
 Schedule Variance
 Schedule Performance Index (>1 is ahead of plan)
Project
Quality
Design

Velocity
 A team’s velocity is the amount of functionality it completed in the previous
iteration
If something isn’t done then its points isn’t counted towards a teams velocity that iteration.
Therefore a 5 point story that is 80% done is counted as 0 and not as 4 points that
iteration.
 Velocity is typically measured in a point system unique to the individual team
You cannot compare teams using points because Team A measures using a different
point system than Team B.
Points are typically called story points or user story points by teams that have adopted a
user-story driven approach to development
Project
Quality
Design

Iteration burndown
 Shows amount of work taken on by the team for a single iteration and
how much work is left to do
 Usage:
 Enables the team to identify where to adjust scope or resources
to finish the iteration successfully
 Provides delivery progress for the iteration
Project
Quality
Design

Release burndown
 Shows the estimated functionality remaining to complete the current release
 Usage:
Track actual progress
Estimate release date based on remaining work/velocity
Project
Quality
Design

Enhancement Request Trend
 Many agile teams work on a new release of a
solution which is already deployed into production
 Shows the trend of enhancement requests received,
approved, and closed during the project lifecycle
Usage:
Few enhancement requests may indicate lack of
interest in the current production release OR may
indicate satisfaction in the current release
A high number of enhancement requests can
indicate that the current production release is not
functioning as stakeholders expect
A growing backlog of enhancement requests may
indicate an inability of the team to respond to
changing requirements
Project
Quality
Design

Age of enhancement requests
 Tracks the length of time stakeholder enhancement requests remain open
 Usage:
Indicates responsiveness of delivery team
Unaddressed requests can impact the stakeholders' perception of value
Project
Quality
Design

Cost per unit of work
 Tracks the cost of delivering a single unit of work (such as a user story
point or use case point) across iterations.
 Usage:
Used to monitor costs throughout the project lifecycle based on the cost of the
team and their velocity.
Monitoring this metric in each iteration helps the team understand if their
spending is sustainable.
Project
Quality
Design

Design Metrics
 Organizational Complexity
 Requirements Complexity
 Model Architectural Complexity
 Model Semantic Complexity
 Model Design Complexity
Project
Quality
Design
These metrics are instrumented in a Rhapsody wizard that is available at Merlin’s Cave:
http://merlinscave.info/Merlins_Cave/Wizards/Entries/2010/1/8_Model_Metrics.html

Design Metrics
 Organizational
Project
Quality
Design
Name Purpose Description
NP Number of Packages Identifies the number of “work” or “configuration”
units in the model
DPC Depth of Package
Containment
A measure of the depth of the model organization
unit
EP Number of Elements
In Package
The number of classes and other elements (such
as use cases and types) in a specified package, a
measure of the size and granularity of the package
AEP Average Number of
Elements Per
Package
A measure of the overall granularity of the model
organization
MEPP Maximum Number of
Elements Per
Package
A measure of the maximum complexity of model
organizational elements
PU Package Utility Number of developers the number of people who
have read or usage access to a package / the
number of developers who write element of a
package

Project
Quality
DesignRequirements Complexity
NUC Number of Use Cases A measure of the number of independent capabilities of
the system
FP
UCP
Function Points
Use Case Points
An estimate of the complexity of the problem to be
solved, maps well to the NUC metric
NA Number of Actors The number of actors associated with a given use case
NUCA Number of Use Cases
per Actor
Given an actor, the number of use cases associated
with the it
NUCSD Number of Use Case
Sequence Diagrams
The total number of black-box sequence diagrams used
as exemplars for use cases
AUCSD Average number of Use
Case Sequence
Diagrams
NUCSD / NUC. This is a measure of the average scope
of a use case
NUCS Number of Use Case
States
Total number of states + activities used to specify the
use cases
UCDC Use Case
Decomposition
Complexity
The number of use cases derived from a single use
case – this includes generalization and dependencies
(both «includes», and «extends» relations)
IIC Information Item Count Total number of Information Items in the use case model
IICUC Information Item Count
per Use Case
IIC / NUC

Project
Quality
DesignModel Architecture Complexity
NS Number of
Subsystems
The number of large-scale architectural units of a
system.
NT Number of Tasks Number of «active» objects in a system
NAS Number of Address
Spaces
A measure of the scope of the distribution of a
model across address spaces or computers.
CASC Cross-Address Space
Coupling
A measure of the cohesion within address spaces
versus cohesion across address spaces.
RAS Redundant
Architecture Scope
Number of redundant architectural units for use in
the Safety and Reliability Architecture (either
homogeneous or heterogeneous)
NP Number of Processors Number of processor nodes in the system. This
may (or may not) be identical with the NAS metric
NCP Number of
Components per
Processor
Measures the cohesion of functionality within a
processor node, assuming that a component
provides a coherent set of functionality.
NUCS Number of Use Cases
per Subsystem
For systems that decompose system use cases into
subsystem level use cases

Project
Quality
DesignModel Semantic Complexity
NoC Number of Classes A measure of the number of model size
CC Class Coupling Measures the cohesion of the classes by computing the
number of associations a class has with its peers
TSC Total number of
subclasses
Measures the global use of generalization within a model
CID Class Inheritance Depth The maximum length of a given class generalization
taxonomy
NC Number of Children Number of direct descendent (subclasses), a measure of
the class reuse
NM Number of Methods Number of methods within a class
NP Number of ports Number of unique identifiable connection points of a
class
EF Encapsulation Factor Number of class features (attributes, methods, and event
receptions) publicly visible divided by the total number of
such features – a measure of the degree to which the
internal structure of a class is encapsulated
SF Specialization Factor The number of operations and statechart action sets
which are specialized in subclasses

Project
Quality
DesignBehavioral Semantics Complexity
SDS Sequence Diagram size Number of messages x number of lifelines
DCC Douglass Cyclomatic
Complexity
Modified McCabe cyclomatic complexity to account for
nesting and and-states
WMC Weighted Methods per
Class
A measure of (non-reactive) class complexity = sum of
methods x complexity for all methods. For classes
without activity diagrams, method complexity can be
estimated by Lines of Code in the method.
ND Nesting Depth State and activity nesting depth – number of levels of
nesting
NE Nesting Encapsulation Number of transitions (other than default) that cross
levels of nesting
NAS Number of And-States Total number of and-states within a statechart
SCN Statechart
completeness
Number of events received by a statemachine / number
of transitions
NPS Number of pseudostates Number of non-default pseudostates such as history,
conditional, fork, join, junction, and stubs

Quality Metrics
 Quality metrics focus on
Compliance to plans
Deviation of expected functionality and correctness
 Compliance to standards
AKA “Syntactic correctness”
Audit / Review Performance Percentage
Audit / Review Pass Percentage
 Correctness
AKA “Semantic correctness”
Defect Density
Defect Trend
Requirements Coverage
Trace Completeness
Project
Quality
Design

Defect Density
 Tracks the number of defects found, fixed, and still remaining during a given period
of time per thousand source lines of code (KSLOC), per model, or design unit
 Usage:
Indication of the quality of the product
Indication of the effectiveness of testing efforts
Project
Quality
Design

Defect Trend Chart
 Shows defect arrival and closure rates, indicates the remaining defect
backlog, projects the future defect arrival/close rate up to and post-ship
 Usage:
– Indication of the quality of the product
– Indication of the effectiveness of testing efforts
Project
Quality
Design

Test coverage of requirements
 Indicates the percentage of requirements linked to validating tests
 For agile teams, this is often the percentage of user stories which have one or
more acceptance tests associated with them
 Usage:
When the coverage isn’t 100% it indicates that the solution isn’t fully tested
When the coverage is 100% we need other metrics to determine sufficiency of testing
Project
Quality
Design

Requirements-based Code Coverage
 Indicates the completeness of the coverage of the code versus the requirements
 This metric is required for evidence by some safety standards (e.g. DO-178)
 Usage:
When the coverage isn’t 100% it indicates that the solution isn’t fully tested or that there
is code for which there are no requirements
Project
Quality
Design

Trace Completeness
 Indicates the degree of consistency among engineering work products
 Typically traces among elements from
requirements, architecture, design, code, test, safety analysis (required by some
safety standards e.g. DO-178, ISO26262)
 Usage
When it is important to ensure consistency of the design due to high consequences of
failure
Project
Quality
Design
Design / Implementation
Elements
D1 D2 D3 D4 D5
Requirements
R1 x x
R2
R3 x
R4 x
Gold plating?
Unimplemented
requirement

Dimensions
Team
(In Process)
Middle Management
(Development Mgmt.)
Development Executive
(VP Development)
Time-to-Delivery /
Schedule
User Story Points / Use Case Points
Iteration Burndown, Blocking Work Item
Release Burndown
Product Value: Iteration Velocity
Stakeholder Feedback, # of Enhancement Request, Age of Enhancement Request
Tested and Delivered Requirements, Business Value
Velocity, Customer Satisfaction
Product Cost /
Expense
Effort (Man-hours)
Cost / Unit of work
Development / Maintenance
Costs
Product Quality Technical Debt (Defect trend, defect density)
Test Status, Test Coverage of Requirement, Test Execution Status
Quality at Ship
Predictability User Story Points / Use Case Points
Planned/Actual Cost and Velocity
Trend Variance. Likelihood of on-time delivery
Note: Bold indicates that there is Out-Of-The-Box report supported by Rational tools
From In Process (Team) To Executive Value: Appropriate Metrics for Each Management level

Adding Metrics into your process: Define your metrics
Concerns
• Identify your concerns
Goals
• Specify your goals
Select
• Define your metrics
Implement
• Define when and how metrics will be gathered
Specify
analytics
• Define how metrics outcomes will be used

Adding Metrics into your process: Update your process
Gathering
• Identify how the metric data will be captured
Instrumenting
• Tool up for metrics
Process
update
• Add into your process definition
Train
• Train workers to properly capture and/or use metric data
Deploy
• Instrument your project

Summary: Metric Guidance
 Clearly define goals and explicitly link metrics to them
 Prefer goal-based over plan-based metrics
 Obtain buy in and consensus
 Train and allocate resources
 Use short enough cycles that metrics can positively affect outcomes
 Collect only a few key measurements
 Automate where possible
 Metrics that do not result in changed actions are worse than useless
 Understand
Metrics are not the goal
Most issues require more than a single metric
Metrics augment but do not replace intelligent judgment

Final Thoughts

Metrics for Mofel-Based Systems Development

More Related Content

What's hot

Similar to Metrics for Mofel-Based Systems Development

Recently uploaded

Metrics for Mofel-Based Systems Development

Editor's Notes