Software companies can have hundreds of software products in-market at any one time, all requiring support and security fixes with tight release timelines or no releases planned at all. At the same time, the velocity of open source vulnerabilities that rapidly become public or vulnerabilities found within internally written code can challenge the best intentions of any SDLC. How do you prioritize publicly known vulnerabilities against internally found vulnerabilities? When do you hold a release to update that library for a critical vulnerability fix when it's already slipped? How do you track unresolved vulnerabilities that are considered security debt? You ARE reviewing the security posture of your software releases, right? As a software developer, product owner, or business leader being able to prioritize software security fixes against revenue-generating features and customer expectations is a critical function of any development team. Dealing with the reality of increased security fix pressure and expectations of immediate security fixes on tight timelines are becoming the norm. This presentation looks at the real world process of the BlackBerry Product Security team. In partnership with product owners, developers, and senior leaders, they've spent many years developing and refining a software defect tracking system and a risk-based release evaluation process that provides an effective software 'security gate.' Working with readily available tools and longer-term solutions including automation, we will provide solutions attendees can take away and implement immediately. • Tips on how to document, prioritize, tag, and track security vulnerabilities, their fixes, and how to prioritize them into release targets • Features of common tools [JIRA, Bugzilla, and Excel] you may not know of and examples of simple automation you can use to verify ticket resolution. • A guide to building a release review process, when to escalate to gate a release, who to inform, and how to communicate.

1. STOP THAT RELEASE,
THERE'S A VULNERABILITY!

2. Christine Gadsby
Director - Product Security Operations
Diahann Gooden
Senior Operations Program Manager
Simran Sidhu
Social Media Specialist
Tyler Townes
Manager – Product Security Response

4. Meet Lucy the Whoodle
• Service for Dog Autism

5. Lucy needs maintenance

6. You are either here........... OR you are here......................
Why are software releases important?

7. Why is this important to BlackBerry?

8. Enterprise-scale
Vulnerability Management
• 100s products to manage
• 100s of sources of threat intel
• 1000s of vulnerabilities to investigate
• ..and many strained relationships

9. Requirements Design Development Testing Deployment
What DEV teams think ......
Threat Modeling+
Design Review
Secure
Architecture
and Hardening
requirements
Security
Testing + Code
Review
Static Analysis Guidance
What Product and Software Security Does …...
That’s it, right?.
SDLC – Bringing a secure product to market

10. AND then....
Open hunting
season begins
It's Launch Day, YAY!

11. What should we be doing?
Software Readiness Review Program
Adding security review to release criteria
• Mitigating risk on behalf of your customers
• Multiple software versions of the same product are in market concurrently
• Know the security posture of your products
• Customers don't like upgrading! It's expensive and time consuming and is often a
double-edged sword
• Ensure you have a ship vehicle for all your patches!
A FIX IN THE BUILD IS BETTER THAN TWO IN THE REPOSITORY!!!!!!!

12. So now
what?
¯_( )_/¯

13. Step ONE: GET SUPPORT
Step TWO: define a vulnerability
• Define based on risk to your
customers, stakeholders, partners
and brand.
• Assess risk level definitions – Agree
on what "critical" really means.
• Ensure security and development are
able to agree with prioritization to
fixes... and what happens when they
don’t. (We fail them....!)
Identify a Common language

14. Create your own Software Readiness
Review
Step THREE: Create standards
• Establish leadership support to use a SRR program
as a security control
• Understand the security posture of each software
release
• Tag vulnerabilities for ease of identification and
tracking
• Define your risk threshold (SRR pass/fail criteria)
• Outline exception process (waiver)
• You need templates and standardization!

16. SWSI Calculator (Should We Ship
It?)Case #: 2896478 Scoring Rating
Base CVSS Score: 5.2
SWSI Score
REVENUE IMPACT
Tier 1 (< $100,000)
Tier 2 ($100,000 - $9999,999)
Tier 3 ($1MM+)
1 2 3 2 .52
EASE OF DISCOVERY
Tier 1 (Hard - Requires complex reverse engineering)
Tier 2 (Moderate – Pen tester would find during an audit)
Tier 3 (Easy – Automated tools could find)
1 2 3 1 1.04
MEDIA / PUBLICITY
Tier 1 (obscure blog/ twitter user)
Tier 2 (industry website)
Tier 3 (MSM, Direct inquiry)
1 2 3 1 2.08
IMPACT TO THE BUSINESS
Tier 1 (customer loses confidence in the business)
Tier 2 (Frustrates customer with high value contract)
Tier 3 (Prevents deal from closing)
1 2 3 2 1.04
RESEARCH TRENDS
Tier 1 (New focus on a subsystem that hasn’t faced rigorous testing)
Tier 2 (new platform with research expected)
Tier 3 (new area of research w/ high likelihood of further discovery)
1 2 3 2 1.04
Total SWSI Rating 5.2

17. SWSI Calculator (Should We Ship
It?)Case #: 2896478 Scoring Rating
Base CVSS Score: 5.2
SWSI Score
REVENUE IMPACT
Tier 1 (< $100,000)
Tier 2 ($100,000 - $9999,999)
Tier 3 ($1MM+)
1 2 3 2 .52
EASE OF DISCOVERY
Tier 1 (Hard - Requires complex reverse engineering)
Tier 2 (Moderate – Pen tester would find during an audit)
Tier 3 (Easy – Automated tools could find)
1 2 3 1 1.04
MEDIA / PUBLICITY
Tier 1 (obscure blog/ twitter user)
Tier 2 (industry website)
Tier 3 (MSM, Direct inquiry)
1 2 3 1 2.08
IMPACT TO THE BUSINESS
Tier 1 (customer loses confidence in the business)
Tier 2 (Frustrates customer with high value contract)
Tier 3 (Prevents deal from closing)
1 2 3 2 1.04
RESEARCH TRENDS
Tier 1 (New focus on a subsystem that hasn’t faced rigorous testing)
Tier 2 (new platform with research expected)
Tier 3 (new area of research w/ high likelihood of further discovery)
1 2 3 2 1.04
Total SWSI Rating 5.2

18. It's not that easy...!
• Threat landscape is unpredictable – There's no Patch Tuesday for
OSS!
• Difficulties with multi-party disclosure
• Weighing business priorities and technical risk
• Who will own the liability?
• Tracking fix commitments – keeping business units honest
• Standardized Process between business units
• Managing relationships
So, what happens when you don't agree on what to release?

19. We need a plan to escalate!

20. Technical Assessment
- Escalation -
Issue ID Date created Severity Public (Y/N) Remediation
schedule
Missed release
vehicles
Risk level Additional
details

21. Issue backlog characteristics
1. List unresolved issues by severity.
2. Highlight lingering issues based on issue filing date (making sure to flag any publicly known issue)
3. Provide details causing delays in mitigation.
Technical reviews and recommendations
1. Provide remediation schedule as documented in the defect management system.
2. Highlight all missed release opportunities.
3. Summarize technical assessment findings and release recommendations.
Release Escalation

22. Things to
Remember!

23. But it's worth it.
Risk landscape is ALWAYS changing
Numbers from 2017:
ü On average, 8 potential vulnerabilities investigated against our product
versions daily
ü Reviewed a total of 515 releases
• Discovery rate / public announcements are unpredictable
• This is an on-going process; don't get hung up on each release not
being perfect
• Focus on making progress
• Be a good partner, you're here to support the business

24. Thanks for listening!
Christine Gadsby
• cgadsby@blackberry.com
• @BBSIRT
BlackBerry Careers - blackberry.com/company/careers
Github - https://github.com/ProductSecurity
BBSIRT - blackberry.com/enterprise/security/incident-response-team

25. Questions?