4. CONTEXT
Cloud-First
• All new applications
and services
designed with focus
on maximising cloud
usage and technical
excellence.
Risk Appetite
• Low resistance to
adoption of new
technologies,
SaaS/PaaS solutions
and developer
autonomy.
Build It, Run It
• Teams own
development and
operations of their
solutions, enabling
technical excellence
and encouraging
risk ownership.
Greenfields
• Early days with full
autonomy to run
application security
program as seen fit
for organisation.
7. EARLY DAYS (~18 MONTHS AGO)
• Shift-left approach already
agreed upon
• Vendor-centric perspective on
application security and
DevSecOps
People
• Trials on a few tools done in
collaboration with contractors
• Only one (open source) tool
deployed by SRE team for
container image scanning
Tools
• Penetration testing as change
management control for
releases
• Security only involved during
design and delivery
Process
10. EARLY DAYS
Challenges in identifying gaps and mapping tools
Significant number of repositories (600+) and AWS accounts (100+)
Multitude of programming languages, including Java, JavaScript, Go,
Kotlin, Swift, Scala, Python and more
Sophisticated development environment and stacks
Providing security scan results as early as possible in CI environments
Leveraging SaaS solutions while ensuring security and confidentiality of
data
13. LESSONS LEARNED
Not every repository is relevant, but no single individual or team knows
which ones are
Licensing per unique contributor (SCA) can get expensive if you have
turnover and/or outsource
Licensing per repository (SAST) can get expensive with highly
compartmentalized architectures (e.g., microservices)
15. LESSONS LEARNED
Automation of security scans is not enough…
SAST reports are useless in text/log output formats
SAST scans can take a long time to complete, resulting in slow builds if
done inline
SCA scans are much better when done inline, but hooking to
dependency management tools can get tricky
16. STARTING LEFT OF THE SDLC
• Individual developers
interested in testing security
tools
• Informal security champions
identified
People
• Checkmarx (SAST) and Snyk
(SCA) chosen as first tools
• Manually triggered scans
across all repositories
Tools
• On demand triage of findings
• No mandatory scans for
change requests
• JIRA tickets created for
confirmed findings
Process
23. SHORTENING THE FEEDBACK LOOP
Scan
• Run security tools via
scripts in CI
01
Submit
• Publish scan results to
OWASP DefectDojo
02
Annotate
• Annotate GitHub pull
request with link to
DefectDojo scan results
03
26. APPSEC HUB FLOW
1. GitHub PR webhooks trigger
build in CI and notify hub
2. Hub requests SAST and SCA
scans
3. SCA and SAST pull code from
GitHub
4. CI scans container using Clair
5. Clair scan log published as
artefact to CI
6. SCA and SAST scan results
fetched and published to PR
7. Clair scan results published to PR
29. APPSEC HUB:
LIMITATIONS
Debugging failed scans is a nightmare
Inconsistent APIs across tools
Asynchronous APIs with no notification (e.g.,
webhooks) require polling
Extensive sanity checks to prevent triggering
actions on unmanaged repositories and CI
pipelines
Side effect: explosive license usage due to
scans on irrelevant code
Implementation architecture misaligned with
other services/applications in organisation
30.
31. EAT YOUR OWN DOG
FOOD
BUILD AND OPERATE
SYSTEMS THE SAME
WAY THE TEAMS
YOU ARE
SUPPORTING DO
32.
33.
34. EAT YOUR OWN DOG
FOOD
Converging on stack used by development
teams enabled…
Better understanding of user experience
Early identification of common false positives
Support from development teams
Credibility and respect
35. LESSONS LEARNED
Scanning code and artefacts with security tools is the bare minimum
Security orchestration > security automation
Security checks and gates must be aligned with existing development
practices and flows
⚠ Security checks should only block if 100% accurate
⚠ Do not create long-lived issues automatically, developers might
address issues as soon as they are reported via pull request annotations
Automation might expose broken or questionable patterns and
processes
Engineering decisions might impact ability to perform security checks (e.g., Snyk automated pull
requests)
37. APPSECD HUB
Lambda orchestration (using
AWS Step Functions)
Full traceability via state
machine
Monitoring and alarms with
real time notifications in Slack
Aligned with best practices
from development teams
40. BEYOND TECHNOLOGY
Most developers consider security a distraction
or a roadblock – often for a good reason
“Security Ivory Tower” is still alive in many
organizations
Encouraging engagement requires giving back
and knowing when to compromise
41. BEYOND
TECHNOLOGY
Providing security results inline
with pull requests encourages
risk ownership
Approving PR = approving
associated issues found
Do not underestimate the impact
of ❌ instead of ✅
Full audit trail via configuration
management – who submitted the
pull request, who approved, who
merged
Shared accountability for
security issues
45. CONCLUSIONS
Tools are means of achieving
alignment against security policies
Scanning profiles must adhere to expectations
and have optimal signal-to-noise ratios
Working reference implementations
over reporting violations
Libraries and code patterns provide security
make the developer’s work easier
Remove all ambiguity
Any room for ambiguous interpretation on
policies and expectations might cause problems
46. CONCLUSIONS
Where are we today?
200+ GitHub collaborators
1,200 GitHub repositories
Supported by a team of 5 people – 3 security architects and 2 security
engineers
DevSecOps..?
Ongoing journey
Left side of SDLC (mostly) taken care of
Enable other parts of security capability operate at DevOps speeds
Editor's Notes
While most of our story focuses on a specific organization, this is collation of experiences in multiple organizations of different sizes and cultures in implementing an application security program.
It is intentionally told in a linear fashion to facilitate story telling but happened in a much more tortuous way in real life.
Our not-so-fictional organization and the point in time in their journey to a devsecops/shared responsibility model
Support for languages/tech stacks limits products, which limits ability to shop around and exposes us to licensing issues
We believed our experience as developers could be leveraged to deliver an implementation that is technically excellent in a short timeframe, and we could iterate from that. We enabled security scans in commits via an API we built, which allowed our pipeline tool to trigger scans by having security tools pull code from GitHub
Why not move even further left and provide feedback via IDE (i.e., ”code change”?) – we do not have a standard IDE in the organization, and some artefacts on which we report (like containers) are only built as part of the CI pipeline
Use OWASP as a reference – OWASP AppSec pipeline project
Why remove defect tracker? Because we believe tickets created by systems are perceived as less valuable by development teams, and because those create a permanent record of what could potentially be an ephemeral/temporary vulnerable state during the development of a feature