Commercial software products rely on formal test strategies to describe who will perform testing, when testing will occur, the process that will be followed, the depth of testing, and more. Test strategies are extended by test plans that detail specific tests that will be executed and how success will be measured. Test strategies and plans support objectively evaluating that software meets requirements and functions properly. Conversely, security teams think about where security gates should be in the SDLC and deploy SAST, DAST, IAST, manual testing, or a combination. Rarely is it considered what level of coverage these methods provide, and output from security testing is not mapped back to requirements. Compared to other teams involved in the SDLC, security seems to just be winging their test strategies and plans. Especially in a DevOps environment where silos are broken and responsibilities are shared across dev, ops, test, and security, use of common methodologies will help to reduce confusion and improve pipeline throughput. During this talk we will discuss: - What are test strategies and how are they used by product teams to provide consistency in testing (something security generally lacks)? - What are test plans and how are they used by product teams to enable visibly strong test coverage (something security also lacks)? - In a DevOps environment, what is security’s role in existing test strategies? - How can security teams leverage test plans to provide better visibility on test coverage and map findings back to requirements to reduce confusion and demonstrate security value to stakeholders throughout the value stream? - What other lessons can we learn from how dev, ops, and test support quality deliveries that can enable more effective and efficient security (e.g. security as code)?

1. A Stratagem on Strategy
Rolling Security Testing into Product
Testing
October 24, 2019

2. • 11 years in AppSec
• 6th LASCON presentation
• <3 BBQ
• Writes lots of code
• Apps and infrastructure
• <3 DevOps
• Always looking to learn from other domains
• Believes in unicorns
• Shocked I like WoW Classic (not 60 yet..)
• 15 years relevant experience
• 8 years dev
• 7 years security
• 4 years AppSec consulting
• Passionate about DevSecOps and changing
the way organizations perform security
testing
• 1 wife, 3 children (ages 14, 12, 9)
• Archeage is better than WoW (fight me)
Kevin Fealey Josh Wallace
The people up front
24 October 2019Page 2

3. A Stratagem on Strategy - Overview
1. Comparing Quality and Security Assurance
2. Vulnerabilities vs Requirements
3. Security Test Strategy
4.
Practical Example: Requirements, Test cases,
Reporting
5. Compatibility with DevOps
6. Next Steps
Page 3
Key Takeaways
• QA is a valued, mature function in most organizations,
is well-understood by developers, and clearly
demonstrates test coverage.
• Security generally does not provide a measure of test
coverage, across security domains (most common),
application functions, or lines of code.
Thesis: There is an opportunity for security to better
integrate into QA processes by adopting their
methodologies, which may improve consistency and
relationships, while reducing confusion
24 October 2019

4. • Risk-based approach
• Higher risk applications receive more
(broader & deeper) security attention
• Security governance and general processes
for the following activities:
• User story reviews
• Security architecture assessments
• Frequent threat modeling
• Manual security code review
• Manual penetration test
• SAST, DAST, IAST integrated into CI
• Supply chain security (third party / FOSS)
• Most/all applications treated the same by
security
• SAST, DAST, IAST integrated into CI
• Supply chain security (FOSS)
• Manual pen tests as required for compliance
Mature programs Less mature programs
Security assurance program strategy
Page 4 24 October 2019

5. • Full set of functional and nonfunctional requirements
• Formalized test strategy that details:
• System architecture
• Test scope
• Dependencies and constraints
• Milestones and activities
• Governance model
• Roles and responsibilities
• Assumptions
• Test approach & methodology for unit, integration, system
and user acceptance testing
• Defect management
• Requirement traceability
• Tooling
• Formalized test plans for each test phase detailing
specific tests for a particular application
• Test case management platform and processes to
standardize and centralize test information
• Same as above
Mature programs Less mature programs
Quality Assurance (QA) program strategy
Page 5
Same as above
24 October 2019

6. Security vs quality assurance summary
Page 6 24 October 2019

7. Stages of testing
Page 7
Requirements
Gathering
Test Planning
Test
Development
Test
Execution
Test
Reporting
Requirements feed reporting
QA Testing
QA and
Security
Testing
Legend:
24 October 2019

8. Delivering vulnerabilities vs pass/fail requirements
Page 8
Finding 104: Clickjacking
It was observed that the application does not explicitly
prevent or deny “Clickjacking” style attacks.
“Clickjacking”, also known as a "UI redress attack", is
an attack that aims at “hijacking” mouse clicks from
unaware end-users in order to perform a set of
malicious actions on the target site.
What the hell is
Clickjacking??
Test Fail: Requirement 104
X-Frame-Options header shall be set to “deny” on all
HTTP responses. It is recommended this be performed
by the load balancer.
I’m 12 and totally
understand what I
need to do!
• Implemented right the first time and clear guidance
when not
• Very few false positives• Lack of a Clickjacking finding != clickjacking is not a
real risk – it may not have been tested
24 October 2019

9. Test strategies
Page 9
► Consolidated document (often 100+ pages, without security)
that sets expectations with developers and other stakeholders
What is a test strategy?
How is it used?
► How often will tests occur? What are the triggers? How do developers
pass (acceptance criteria)?
► Provides centralized knowledge of the application to be used by testers
► Enables consistency in test approach
► Provides clarity on dependencies, constraints, and roles & responsibilities
including escalation path
► Defines the product to be tested, core functionality and interfaces
► Defines the test objectives - does the application need to meet certain
requirements or have a “clean scan?”
Does your security team contribute to your test strategy?
24 October 2019

10. • Scope
• Approach
• Automation strategy
• Test schedule
• Tooling
• Risks and Impacts
• Review and approvals
Security should collaborate on test strategy sections, such as:
Integrating security teams into the test strategy
Page 10
A good place to establish security-related taxonomy (let’s all speak the same language)
24 October 2019

11. “QA is a valued, mature function in most
organizations, is well-understood by developers, and
clearly demonstrates test coverage.”
• Security generally does not provide a measure of test coverage,
across security domains (most common), application functions, or
lines of code.
• Security is confusing to those outside of security.
• Thesis: There is an opportunity for security to better integrate into
QA processes by adopting their methodologies, which may improve
consistency and relationships, while reducing confusion.
Page 11
- Kevin
24 October 2019

12. Let’s get practical
Turning theory into action
Page 12 24 October 2019

13. Example application description
Page 13
• Imagine that we work for a large financial services organization (or maybe you do)
• Our CEO observes that Bitcoin is taking the world by storm and forms a team to develop a
strategy to allow users to make transactions with crypto-currencies
• Inside of our online banking application, we want to include a new feature that will allow
users to exchange US Dollars to Bitcoin within their accounts
24 October 2019

14. Example app requirements (user stories)
Page 14
Feature: Exchange USD for BTC
• ID: FCRY-1
• Story: As a user, I want to exchange US Dollars
(USD) in my account to Bitcoin (BTC)
Acceptance criteria
• Conversion webpage and form exist
• Conversion endpoint exists
• Swagger API YAML file exists
• Conversion endpoint is able to convert USD to BTC
• Conversion webpage is able to invoke conversion
endpoint with user-provided values
24 October 2019

15. Example app QA test case
Description: Exchanging USD to BTC correctly adjusts BTC and USD balances in the user’s
account
Related requirement and/or test: FCRY-1: As a user, I want to exchange US Dollars (USD)
in my account to Bitcoin (BTC)
Remarks/Notes: N/A
Gherkin Script:
• Given the USD to BTC conversion webpage
• And a user account containing 0 BTC and $9,000 USD
• When the user types “9000” into the USD box and clicks “Convert”
• Then the user’s account page should be displayed
• AND the user’s account should contain $0 USD and 1 BTC
Manual or Automated: Automated
Page 15 24 October 2019

16. Example app security test case (TLS)
Description: Utilize TLS connections for all content requiring authenticated access and for transfer of
all sensitive information
Related requirement and/or test: FCRY-1: As a user, I want to exchange US Dollars (USD) in my
account to Bitcoin (BTC)
Remarks/Notes: Additional test cases are required to validate other TLS requirements, such as
allowed ciphers, disallowed SSL/TLS versions, and use of valid certificates
Gherkin Script:
•Given the USD to BTC conversion webpage
•And a user account containing 0 BTC and $9,000 USD
•When the user types “9000” into the USD box and clicks “Convert”
•Then the HTTP request should be sent with TLS 1.2
Manual or Automated: Automated
Page 16 24 October 2019

17. Example app security test case (input validation)
Description: Validate user input matches defined whitelist criteria
Related requirement and/or test: FCRY-1: As a user, I want to exchange US Dollars (USD) in my account to Bitcoin
(BTC)
Remarks/Notes: POST parameter “USD” allowed values are whole numbers in the range of 0 to 25000
Gherkin Script:
•Given the USD to BTC conversion webpage
•And a user account containing 0 BTC and $9,000 USD
•When the user types <valToTransfer> into the USD box and clicks “Convert”
•Then the USD to BTC conversion webpage should be displayed
•And “Please transfer a valid quantity of USD” should be displayed in the “errorMsg” field
Manual or Automated: Automated
Page 17 24 October 2019

18. Leverage existing QA reporting
Page 18 24 October 2019

19. Is this compatible with DevOps?
• Doing this now as part of an Agile transformation, heavily leveraging DevOps
• Adds more work for security up front, but increases compliance with security requirements
due to transparency, consistency, and delivering more actionable output (i.e. requirements,
not vulnerabilities)
• Improves collaboration between security and product teams and provides better insight into
risks during design and development processes
Page 19 24 October 2019

20. Improved testing workflow implementing security
Page 20
Requirements
Gathering
Test Planning
Test
Development
Test
Execution
Test
Reporting
Requirements feed reporting
QA Testing
QA and
Security
Testing
Legend:
24 October 2019

21. Next steps
• Identify the QA leads within your organization
• Find out about their test processes and artifacts (strategies, test cases, reports, dashboards,
etc.)
• Work with QA to more formally integrate security into existing testing frameworks
• Build a repository of reusable, actionable security requirements
• Sit in sprint planning and add security requirements to user stories
• Write your first security test case
Page 21 24 October 2019

22. EY | Assurance | Tax | Transactions | Advisory
About EY
EY is a global leader in assurance, tax, transaction and
advisory services. The insights and quality services we deliver
help build trust and confidence in the capital markets and in
economies the world over. We develop outstanding leaders
who team to deliver on our promises to all of our stakeholders.
In so doing, we play a critical role in building a better working
world for our people, for our clients and for our communities.
EY refers to the global organization, and may refer to one or
more, of the member firms of Ernst & Young Global Limited,
each of which is a separate legal entity. Ernst & Young Global
Limited, a UK company limited by guarantee, does not provide
services to clients. Information about how EY collects and
uses personal data and a description of the rights individuals
have under data protection legislation are available via
ey.com/privacy. For more information about our organization,
please visit ey.com.
© 2019 EYGM Limited.
All Rights Reserved.
This material has been prepared for general informational purposes only and is not
intended to be relied upon as accounting, tax or other professional advice. Please
refer to your advisors for specific advice.
ey.com