Which Development Metrics Should I Watch?

© COPYRIGHT 2017 COVEROS, INC. ALL RIGHTS RESERVED. 1@CoverosGene
Agility. Security. Delivered.
Which Development Metrics
Should I Watch?
Gene Gotimer
@CoverosGene

About Coveros
• Coveros builds security-critical applications using agile methods.
• Coveros Services
• Agile transformations
• Agile development and testing
• DevOps and continuous integration
• Application security analysis
• Agile & Security training
• Government qualifications
• DCAA approved rates and accounting
• TS facility clearance
Areas of Expertise

Select Clients

Why Metrics?
“Without data you’re just
another person with an
opinion.”
– W. Edwards Deming

Metrics as Targets
“People with targets and
jobs dependent upon
meeting them will probably
meet the targets –
even if they have to destroy
the enterprise to do it.”
– W. Edwards Deming

Goodhart’s Law
“When a measure becomes
a target, it ceases to be a
good measure.”
– Charles Goodhart

Managing to Metrics
“Managers who don't know
how to measure what they want
settle for wanting what they can
measure.”
– Russell Ackoff

What Makes a Good Metric?
S Specific
M Measurable
A Achievable
R Relevant
T Timely

How Many Metrics?
to
at least to actively pay attention to

SonarQube
• Software quality dashboard
• Gives visibility into results of
• Unit tests
• Static analysis
• Duplicate code
• Quality problems
SMARTSM T

SMARTA

Which metrics?
Get on with it already…
SMARTR

WTFs per Minute
SMARTXXX

Lines of Code
• Used for estimating costs, for example COCOMO
• Measures developer productivity

Lines of Code
“Measuring programming progress
by lines of code is like
measuring aircraft building
progress by weight.”
– Bill Gates

Lines of Code
• Used for estimating costs, for example COCOMO
• Measures Doesn’t measure developer productivity
• Rewards build-it-yourself instead of using existing solutions
• What if I’m deleting code?
• Using lambdas?
SMARTX

Spell It All Out
1 boolean boolResult = false;
2 if (hashResult > 0)
3 boolResult = true;
4 else
5 boolResult = false;
6 assertEquals("HashCode returned was not returned",
true, boolResult);
as opposed to
1 assertTrue("HashCode should be a positive integer",
hashResult > 0);

Number of Bugs Fixed
• Fixing bugs is good
• When does it become a bug?
• What if you write bug-free code to start with?
SMART/ / X

Code Complexity
public List<Double> getAngles() {
return Collections.unmodifiableList(angles);
}
public double getPerimeter() {
double perimeter = 0.0d;
for (double length : lengths) {
perimeter += length;
}
return perimeter;
}
1.0
2.0
SMARTRTAMS

Number of Unit Tests
• Unit tests document the developer’s intent
• We need enough
• One test per path
• Can use code complexity as guide
• But don’t usually have it per method
• Leads to guessing
SMARTXX

Testivus on Code Coverage
One morning, a programmer asked the master
“I am ready to write some unit tests.
What code coverage should I aim for?”
The master replied,
“Don’t worry about coverage, just write some good tests.”

Later, a second programmer asked the master the same question.
The master pointed to a pot of boiling water and asked
“How many grains of rice should I put in that pot?”
The programmer replied,
“How can I possibly tell you? It depends on how many people you
need to feed, how hungry they are, what other food you are serving,
how much rice you have available, and so on.”
“Exactly,” replied the master.

Towards the end of the day, a third programmer asked
the same question about code coverage.
The master said sternly
“Eighty percent and no less!”

The apprentice asked the master why he gave the programmers
three different answers to the same question.
The master began to answer.
“The first programmer is new and just getting started with testing.
Right now he has a lot of code and no tests.
He has a long way to go; focusing on code coverage at this time would
be depressing and quite useless.
He’s better off just getting used to writing and running some tests.
He can worry about coverage later.”

“The second programmer, on the other hand, is quite
experienced both at programming and testing.
When I replied by asking her how many grains of rice
I should put in a pot, I helped her realize that the amount of testing
necessary depends on a number of factors, and she knows those
factors better than I do – it’s her code after all.
There is no single, simple, answer, and she’s smart enough to handle
the truth and work with that.”

“I see,” said the apprentice.
“But if there is no single simple answer, why did you tell
the third programmer Eighty percent and no less?”
The master laughed.
“The third programmer wants only simple answers –
even when there are no simple answers…
and then does not follow them anyway.”
Alberto Savoia
http://www.artima.com/forums/flat.jsp?forum=106&thread=204677

Code Coverage
• Measures code executed when unit tests run
• NOT amount of code tested
• Good tool to find untested code
• Not covered == not tested
• Covered == possibly tested

Test All The Things, Literally
for (Method method : object.getClass().getMethods()) {
if (method.getName.startsWith("set")) {
if (method.getParameterTypes().length > 0) {
Class paramClass = method.getParameterTypes()[0];
if (paramClass.getName.equals("java.lang.String")) {
method.invoke(object, "JUNIT TEST");
} else if (paramClass.getName.equals("java.lang.Object")) {
method.invoke(object, new Object());
} else if (paramClass.getName.equals("java.util.Date")) {
method.invoke(object, new java.util.Date());
} else if (paramClass.getName.equals("java.math.BigDecimal")) {
method.invoke(object, new BigDecimal(100));

Code Coverage
• Measures code executed when unit tests run
• NOT amount of code tested
• Good tool to find untested code
• Not covered == not tested
• Covered == possibly tested
SMARTX

Mutation Testing
• Reruns unit tests against modified versions of your code
• If tests still pass, code isn’t tested
• Tests quality of tests
public int foo(int i) {
i--;
return i;
}
public int foo(int i) {
i++;
return i;
}
SMARTRTAMS

Number of Failing Unit Tests
• Unit tests document the developer’s intent
• What is an acceptable number of failing tests?
SMARTRTAMS

Legacy Code
• Greenfield code = your mess
• Legacy code = someone else’s mess that
you have inherited
• Consider applying
• Quality gates to new code
• Watch trends on existing code
• As you modify existing code
• Leave the code cleaner than it was
when you found it

Technical Debt
“You can have things early in a development and
gain experience and that’s a good strategy, as
long as you have a plan to pay it back.
If you don’t pay it back then you
get a compounding – learning that
you’ve consciously avoided comes
back to hurt you. ”
– Ward Cunningham

Qualities of Good Code
Cohesive
non-Redundant
Encapsulated
Assertive
Testable
Explicit
David Bernstein
@ToBeAgile
https://tobeagile.com/

Technical Debt
• Bugs and potential bugs
• Coding standards violations
• Duplications
• Lack of unit tests
• Bad distribution of complexity
• Not enough or too many comments
• Spaghetti design
← redundant
← not testable
← not cohesive
← not cohesive
← not explicit
not assertive
not encapsulated
SonarQube measures technical debt as:

Technical Debt
• Bugs and potential bugs
• Coding standards violations
• Duplications
• Lack of unit tests
• Bad distribution of complexity
• Not enough or too many comments
• Spaghetti design
SonarQube measures technical debt as:
SMARTRTAMS

Normalization of Deviance
“The gradual process through
which unacceptable practice or
standards become acceptable.
As the deviant behavior is
repeated without catastrophic
results, it becomes the social
norm for the organization.”
– Diane Vaughn

Continuous Integration
• The act of frequently integrating different developer’s
code, building, and testing each commit to find any
problems quickly
• The goal is that software is always
in a working state
• The developer codes, commits, and then
waits for the automated build, unit tests,
and static analysis to run
• If anything fails, the developer is on hand
to fix it before anyone else is disrupted

Build Time
• Time to complete cycle
• Automated build
• Unit tests
• Static analysis
• CI cycle should be no more than 10 minutes
• Keep it short to encourage more frequent exercise

Build Time
• Time to complete cycle
• Automated build
• Unit tests
• Static analysis
• CI cycle should be no more than 10 minutes
• Keep it short to encourage more frequent exercise
SMARTRTAMS

Escaped Defects
• Defects that are found after “done”
• Could be
• Bugs
• Functional defects
• Security issues
• Usability problems
• Unacceptable performance
• What is an acceptable number of escaped defects?
SMARTX

Escaped Defects
• Zero isn’t realistic
• Zero may not be desirable
• Watch the trend
• Hold even or decrease
• Adjust process and definition of done
• As you improve
• In order to improve
• Best measure of team quality
SMARTRTAMS

Retrospectives
• Regularly reevaluate
• Metrics
• Are they still relevant?
• Are we missing metrics we should be watching?
• Quality gates
• Are they achievable?
• Could we be stricter? Would that add value?
• Use escaped defects as a guide

Wrap Up

#Coveros5
• Metrics should be SMART
Specific, Measurable, Achievable, Relevant, and Timely
• Use metrics to set quality gates
If you can't commit to the metric, don't bother tracking it
• Code coverage doesn't tell the whole story
use mutation testing too
• Trends are usually more important than absolute values
except for 0 failing unit tests
• Escaped defects is the most important metric
no matter where your boundary of “escape” is

Development Metrics
Good
Escaped defects
0 failing unit tests
Technical debt
Mutation testing coverage
Build time
BAD
Lines of code
Number of bugs fixed
Number of unit tests
Code coverage (alone)

Questions?
Gene Gotimer
gene.gotimer@coveros.com
@CoverosGene
Come see my Lightning Talk
Thursday at 3:45pm in Wekiwa 3&4 for
Creative Solutions to Already Solved Problems

Which Development Metrics Should I Watch?

Recommended

Recommended

More Related Content

Similar to Which Development Metrics Should I Watch?

Similar to Which Development Metrics Should I Watch? (20)

More from Coveros, Inc.

More from Coveros, Inc. (11)

Recently uploaded

Recently uploaded (20)

Which Development Metrics Should I Watch?