During a recent webinar, Kevin Nassery, Software Security Practice Lead at Synopsys Software Integrity Group spoke to attendees about using metrics to drive their software security initiative. ntuition can take you quite far at the beginning of your application security journey. But even the most experienced leaders will eventually need data to guide them through a decision or justify their investments. Well-designed software security metrics provide that compass. For more information, please visit our website at https://www.synopsys.com/BSIMM

1. CONFIDENTIAL© 2019 Synopsys, Inc.1
Using Metrics to Drive Your Software Security
Initiative
June 18, 2019
Kevin Nassery

2. CONFIDENTIAL© 2019 Synopsys, Inc.2
Intro
Me:
• Lead BSIMM ops & SSI practice at Synopsys Software Integrity Group
• 20+ years (50/50 in consulting) across infrastructure security, software
security, and program consulting
• 50+ BSIMM assessments
Metrics talk:
• The good
• The bad
• The must-haves
• But…not in that order
• Real stories (but anonymized
to protect the guilty)

3. CONFIDENTIAL© 2019 Synopsys, Inc.3
Metrics vs. measures
Used interchangeably in business—can’t assume what someone is talking
about when they use either term. However:
–A measure is a numerical observation independent of the process defining
how it was taken
–A metric is a numerical observation based on standard systems, methods,
calculations, and data sources
Metrics provide better comparative value over collections of measures:
– Consistency in observation process
– Consistency in meaning

4. CONFIDENTIAL© 2019 Synopsys, Inc.4
What you need to know about SSI metrics in 2 minutes
Must
• Measure SDL
compliance
• Get telemetry
from primary
gates
• Create feedback
loop to SSDL
enhancement
• Look at the data!
• Consider factors
of the program
execution and risk
Should
• Correlate data
• Strive for good
time series data
• Compare value of
efforts
• Test your theories/
intuition
• Tell your AppSec
story
• Fix your metrics
when you know
they’re broken
• Maximize the
security impact of
spend
Should avoid
• Accidentally
complicating the
numbers
• Intentionally
complicating the
numbers
• Rushing metrics
development
Must not
• Make up numbers
• Make up stories
about made-up
numbers

5. CONFIDENTIAL© 2019 Synopsys, Inc.5
What does BSIMM tell us about metrics?
[SM2.1] Publish data about software security internally.
• Just 39% of firms in BSIMM9 are publishing data about their SSI within the organization,
compared to 84% that have identified gate locations (SM1.4).
[SM3.3] Identify metrics and use them to drive budgets.
• Only 15% of firms are using metrics formally enough to drive fiscal decisions about their
software security initiatives.

6. CONFIDENTIAL© 2019 Synopsys, Inc.6
Misstep #1: Not enough context
CEO to CISO:
“I see January was a real
setback in our open issues from
penetration testing. Let’s not let
that happen again.”
CISO to CEO:
“Yes, ma’am.”
CISO to AppSec director:
“Cancel that new pen test
contract ASAP!”

7. CONFIDENTIAL© 2019 Synopsys, Inc.7
Misstep #1: Not enough context (cont.)
Date % PT # PT open H # PT open M # PT open L
6/2/19 66 1438 796 2148
5/2/19 66 1778 856 2540
4/2/19 66 1619 844 2320
3/2/19 66 1720 905 2450
2/2/19 66 2861 1435 4160
1/2/19 66 2841 1496 4211
12/2/18 66 1381 739 1950
11/2/18 66 1307 651 1880
10/2/18 66 1270 632 1875
9/2/18 66 927 447 1240
8/2/18 47 758 451 1100
7/2/18 47 748 405 1000
6/2/18 48 731 387 950
5/2/18 52 655 388 900
4/2/18 52 627 360 925
3/2/18 52 678 403 933
2/2/18 50 655 364 850
1/2/18 50 606 284 800
12/2/17 50 683 315 900
11/2/17 50 832 414 1200
10/2/17 10 97 35 50
9/2/17 10 105 69 150
8/2/17 10 228 75 200
7/2/17 10 101 48 50
0
10
20
30
40
50
60
70
3/6/17 6/14/17 9/22/17 12/31/17 4/10/18 7/19/18 10/27/18 2/4/19 5/15/19 8/23/19
% Portfolio Tested in Last 90 days
AppSec director to CISO:
“In January 2019, we began authenticated application testing, which gave us new
visibility into vulnerabilities we previously weren’t aware of. Most of these issues
were present long before they were discovered, and we’ve started the remediation
progress with our business and technical owners. Before that, in August, we were
able to bring our testing coverage across our portfolio up from 50% to 66%.”
Issues like this are best managed by ensuring executives are familiar with the
concepts of “managed” vs. “unmanaged” risk and providing visibility into both areas
wherever possible.

8. CONFIDENTIAL© 2019 Synopsys, Inc.8
Misstep #2: Bad definition & calculation
0
50
100
150
200
250
300
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
MTTR (days)
Total
AppSec director to CISO on January 1:
“Our most important application security metric
is our mean time to remediate, which
represents our organization’s ability to
remediate our known findings.”
October 1:
“Good news: We were able to finally close out
a critical design flaw discovered in January!”
CISO to AppSec director:
“That’s great. Unfortunately, your bonus has
been canceled due to the September MTTR.”

9. CONFIDENTIAL© 2019 Synopsys, Inc.9
Misstep #2: Bad definition & calculation (cont.)
Row Labels Sum of TTR (days)
Jan 13
Feb 11
Mar 124
Apr 16
May 10
Jun 20
Jul 73
Aug 63
Sep 253
Oct 17
Nov 12
Dec 15
Finding Open Date TTR (days) EOMonth-Remed
4006 1/7/18 240 9/30/18
2360 3/23/18 100 3/31/18
4726 8/7/18 60 8/31/18
1372 7/9/18 50 7/31/18
3345 1/13/18 7 1/31/18
0
50
100
150
200
250
300
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
MTTR (days)
Total
By not representing “open” issues in the MTTR calculation, the most “difficult” problems were
unaccounted for until they were closed. These types of lagging issues are quite common, unless
great care is taken in metrics development. Metrics, like software, often contain bugs.

10. CONFIDENTIAL© 2019 Synopsys, Inc.10
Misstep #3: Hidden factors, hidden risk
“Our severity scale reduces risk to low, where CVSS indicated the issue is not exploitable.”
• Misinterpreting “Not Defined” as “Not
Exploitable” hid hundreds of critical
security flaws from the remediation
efforts.
• The original intent was to facilitate
“prioritization” of risk, but this
carelessness resulted in major program
setbacks and provoked leadership
changes.
Image source: CVSS v3.0: Specification Document

11. CONFIDENTIAL© 2019 Synopsys, Inc.11
Misstep #4: Breaking the incentive model
“Because I’m a penetration tester, my compensation is driven by making
applications more secure. The key metric we use is if the applications
tested have fewer findings over time.”
Every published metric may have an unintended influence on stakeholder
incentives. Pressure check how your organization is using data to ensure
the incentives align with the overall goals of the organization and security
initiative.

12. CONFIDENTIAL© 2019 Synopsys, Inc.12
Highlight #1: Using metrics to drive vendor selection
“We provided the same application, artifacts, and source
code to multiple vendors and our internal testing team
with the same allotted time. The complete superset of
results was then aggregated and deduplicated, and each
vendor was given a percentage score against the
baseline. This score was combined with other
comparative factors, such as quality of guidance,
reporting factors, and cost, and was used to drive our
vendor schedule for the next 24 months.”
Designing metrics that can inform budgetary decisions is
a key behavior of successful SSI leaders.

13. CONFIDENTIAL© 2019 Synopsys, Inc.13
Highlight #2: Using a security activity to measure another
“We weren’t sure if our secure development training was
effective, so we analyzed our defects from multiple discovery
sources against our developer curriculum content. Each high
and critical issue was cross-referenced with our training
content, which led to us identifying both areas where we didn’t
have any coverage and areas where we did have coverage but
developers were still making mistakes.
This gave us clear direction on expanding the security
development curriculum and refreshing problematic existing
content.”

14. CONFIDENTIAL© 2019 Synopsys, Inc.14
Highlight #3: Making the case for SSI growth
“After our acquisition, the value of our SSI
investment was unclear to our new executive
leadership team. Fortunately, we had the tools and
automation in place to sample a number of
applications not managed by our SSI and compare
against the SSI’s portfolio. This demonstrated to
our leadership teams the importance of our
upstream program elements such as training, IDE
integrated secure code review, and threat
modeling.”

15. CONFIDENTIAL© 2019 Synopsys, Inc.15
Fun resources to inspire a new mindset
A fun primer on how to look at data differently, and a fun
introduction to thinking about incentives
A documented approach that may help you break
some of the institutional inertia regarding “bad”
numbers
How objectives and key results (OKRs) has
helped tech giants from Intel to Google achieve
explosive growth—and how it can help any
organization thrive

16. CONFIDENTIAL© 2019 Synopsys, Inc.16
Discussion

17. CONFIDENTIAL© 2019 Synopsys, Inc.17
Interested in learning more?
Our fourth annual FLIGHT Boston 2019 conference is just around the corner. On Sept. 17–19,
Synopsys will bring together leading experts from around the world to help you take your
software security/development practice to new heights. You’ll learn about the latest insights and
best practices in application security, DevOps and the cloud, and open source license
compliance.
Visit https://snps.sw-sec.co/FLIGHTBoston for more information and use code FLIGHT19 for
50% off your registration.

18. Thank You