This session addresses the technology challenges of continuous security testing to “deliver securely,” and discusses best practices and tooling based on first hand experience in both enterprise and startup environment.

1. © 2016 by RayLai, Confidential and Proprietary© 2016 by RayLai, Confidential and Proprietary
Continuous Security
Testing
Ray Lai
Dec 7, 2016 (Version 0.2)

2. © 2016 by RayLai, Confidential and Proprietary
Synopsis
This session addresses the technology
challenges of continuous security
testing to “deliver securely,” and
discusses best practices and tooling
based on first hand experience in both
enterprise and startup environment.

3. © 2016 by RayLai, Confidential and Proprietary
The Problem
How to complete security testing whenever
a build is initiated
Develop
Security testDeploy
Impact
US$10,000 / bug fixed
US$4M data breach (2016)
Damage to brand

4. © 2016 by RayLai, Confidential and Proprietary
Security Testing: Now and Then
People
• Professional/consultants
Technology
• Commercial tools
Process
• Central governance
• Testing before SW release
Before/Now Desired State
People
• Engineering community
Technology
• Open source tools
• UI and API security testing
Process
• Self-serve
• Continuous: at build/deploy
time

5. © 2016 by RayLai, Confidential and Proprietary
How

6. © 2016 by RayLai, Confidential and Proprietary
What is “Continuous Security Testing”?
Continuous Security Testing = New Tools?

7. © 2016 by RayLai, Confidential and Proprietary
Continuous Security Testing: Point of View
App Source
Codes
Objectives
• Efficiency
• Feedback loop for developers
Security
Metrics
Automated
Security
Testing
Web App
Vulnerability
Scanning
Threat
Modeling
Secure Build
Build PipelineInput Output
Stakeholders
• DevOps
• Developers
• Quality engineers
• Security
• Static code analysis
• Vuln scanning
• App assessment
• Pen test
• Risk assessment
• Compliance
• Operations review
• Defect reports
• Threat modeling
Challenges:false positives/noise,long build time

8. © 2016 by RayLai, Confidential and Proprietary
Continuous Security Testing: Paradigm Shift
Testing approach
• Network vs app security
• Front-end vs backend;
integrated test
Tools
• Enterprise (integrated), On-
premise
Execution
• Testing separated from build
pipeline
Old New
Testing approach
• Testing how apps and
systems are connected
Tools
• Cloud-based, per pay use
• Single-purpose, open source
Execution
• DevOps: Test in build phase

9. © 2016 by RayLai, Confidential and Proprietary
How to Deliver Securely?
Exercise 1
You have just completed
functional and security
testing for your flagship app
and platform. However, a
security researcher informs
you that they can reset user
password via SMS, and
take over a user’s account.
Options
1. Extend test coverage
(resources)
2. Enforce pen test before
release (professionals)
3. Educate engineers (training)
How would you do differently to deliver
securely?

10. © 2016 by RayLai, Confidential and Proprietary
How to Deliver Securely?
Exercise 2
A security researcher
reports a Cross-site
Scripting (XSS) issue in
your portal, with XSS
examples in existing static
Web pages. Your dev team
argues that XSS on static
Web pages is not harmful,
and any fix will take them at
least 1 month.
Options
1. Validate if this is a fire drill
(risk management, education)
2. Fix any XSS defect
(resources)
3. Enforce pen test before
release (professionals)
How would you do differently to deliver
securely?

11. © 2016 by RayLai, Confidential and Proprietary
How to Deliver Securely?
Exercise 3
Amazon AWS notifies your
security team that your
standalone test VM (AMI)
has been compromised.
While your app team argues
that it is an isolated
instance, AWS finds traces
that it was used to access
other AMI instances without
permission, and also
becomes a DDOS agent.
Options
1. Introduce security controls
and security testing for cloud
apps (technology, resources)
2. Enforce pen test before
release (professionals)
3. Raise awareness (education)
How would you do differently to deliver
securely?

12. © 2016 by RayLai, Confidential and Proprietary
How to Deliver Securely?
Recommendation (Past) Continuous Security Testing
Exercise 1
(password
reset)
Threat modeling
Test for trusted boundaries
(connected security)
In build pipeline, spin an app instance
to test for end-to-end flow.
Exercise 2
(XSS)
Resolve any XSS defects
Raise awareness
Include UI testing for XSS in
automated test suite (build process).
Exercise 3
(compromised
AMI)
Cloud security testing –
production pen test
Cloud security controls
In build pipeline, spin an AMI instance
to test AWS security policies and
controls.
This is what I would do differently...

13. © 2016 by RayLai, Confidential and Proprietary
Recommendation
• Continuous security testing is a journey (various maturity levels)
– Identify your low hanging fruits and priorities
– Building community is an art (e.g. contents, outcome and appropriate
engagement)
• Be open mind about changes in execution approach
– DevOps -> build pipeline for security
– Gradual shift from commercial, integrated tools to open source tools
• Be prepared for the next big wave
– Emerging security challenges in IoT and block chain technologies

14. © 2016 by RayLai, Confidential and Proprietary
Appendix: Best Practices

15. © 2016 by RayLai, Confidential and Proprietary
What to Prioritize
https://techcrunch.com/2015/01/22/security-for-startups-in-10-steps/
Prioritize
Security culture
Secure platform
Your Website is front-door
Secure coding
Security controls
Plan for failure
Physical security
Be open with public
Email with multi-factor auth
Threats

16. © 2016 by RayLai, Confidential and Proprietary
Which Tools to Use
brakerman
Static Code Analysis /
Vulnerability Scanners
Cloud / Infrastructure Security
Data SecurityConnected Systems
WhatWeb Web Scanner
w3af
OSSEC Arachni
findBugs
FlawFinder
PMD

17. © 2016 by RayLai, Confidential and Proprietary
What to Test
Major Security Vulnerabilities Static Code
Analysis
Web App Scan Pen Test
OWASP Top 10 Partial Partial Partial
HeartBleed-type SSL Issues No No Partial
DOS, e.g. due to memory leakage Partial
(Memory
leakage)
Partial Partial
DDOS, e.g. bot attack No No Partial
Cloud hosting security issues No Limited Yes
NoSQL database security issues No No Partial
IoT security issues No Varies Varies
Blockchain security issues No No Varies

18. © 2016 by RayLai, Confidential and Proprietary
What to Test (cont’d)
OWASP Top 10 Static Code
Analysis
Web App Scan Pen Test
A1 Injection X X X
A2 Broken Auth and Session Management X X
A3 Cross-site Scripting (XSS) Partial X X
A4 Insecure Direct Object References Partial X
A5 Security Misconfiguration X X
A6 Sensitive Data Exposure Partial X
A7 Missing Function Level Access Control X Partial
A8 Cross-Site Request Forgery (CSRF) X X
A9 Using Components with Known
Vulnerabilities
Partial
A10 Unvalidated Redirects and Forwards Partial X

19. © 2016 by RayLai, Confidential and Proprietary
How to Execute (Build Pipeline for Security)
Web App
Scan
Deploy to
Production
Security
Assessment
Deploy to
Production
Build
Pipeline
Penetration
Test
Runtime
Container
Security
Test Client
Web App
Scan
BeforeNew
Separate, serial events
Continuous integration process to create temporary containers for security testing

20. © 2016 by RayLai, Confidential and Proprietary
Security Build Pipeline (Example)
App
Container
Jenkins Server
App
Container
Docker Swarm
App
Container
Security
Scanner
(Docker)
Shared Data Container / Data Volume
AppsData
Pre-requisites
• Apps deployed in docker containers
• App data (synthetic or scrubbed
production data) in shared data
containers
• Parallelize testing tasks in Jenkins
Jenkins Setup
• Create docker swarm (apps and data)
• Launch docker swarm
• Create security test client (security
scanner)
• Launch security test client
• Post test results in shared data
container or data volume
• Shut down all docker containers
Gauntlt, ClamAV, Brakerman,
OWASP ZAP

21. © 2016 by RayLai, Confidential and Proprietary
What to Look For: Web App Security Testing
Session Management
• HTTPOnly and SECURE
flags (cookies)
UI and App
• XSS, CSRF
Data
• Injection vulnerabilities
Traditional Approach What’s New
API
• API token robustness;
playback and Man-in-the-
middle attacks
Data theft
• Data privacy issues (in transit
or at rest)
NoSQL database
• Access control issues

22. © 2016 by RayLai, Confidential and Proprietary
What to Look For: Platform and Cloud
Platform Infrastructure
• Network security, e.g. port
scan, mis-configuration
• Default passwords and
settings
Identity management
• SSO; access control
Compliance
• Security certification, e.g. PCI
Traditional Approach What’s New
Platform Stack
• Timely patches
• Core security, e.g.
Heartbleed bug
• DDOS
Data security
• Storage security, e.g. S3
Compliance
• Public cloud (e.g. AWS)
security policies and controls

23. © 2016 by RayLai, Confidential and Proprietary
How to Measure Success
Defect
Density
Threat Model
Custom Risk
Scores
OWASP Top
10
WhiteHat WSI
Security Index
Cenzic HARM
Scores
HP Fortify
Scores
Internal Metrics
Financial management
app has a defect density
of 15.23 (5M LOC)
25 attack vectors Biz risks: 5.2
Tech risks: 7.9
Tech risks reflect # of
total vulnerabilities
from scanning/testing
Industry
Example:
For benchmarking

24. © 2016 by RayLai, Confidential and Proprietary
How Do I Get Trained
Traditional Channels What’s New
Security Certification
• CEH, CISSP, etc.
Conferences
• RSA, OWASP AppSec,
BlackHat, DEFCON, etc.
Product training
• E.g. Fortify
Online Resources for the
Impatient
• Youtube
• Udemy, Coursera, etc.
Community
• Meetups or user groups, e.g.
OWASP