This document discusses the journey of an application security engineer in implementing an effective application security program within a large organization. It describes early challenges around tool selection, process definition, and gaining developer buy-in. It outlines the creation of an "AppSec Hub" to automate security scans and feedback, but limitations led to building tools aligned with how development teams work. Lessons learned include the importance of relationships, compromise, and embedding security seamlessly into development flows and tools. The conclusions emphasize that tools should focus on alignment, scanning profiles should have optimal signal-to-noise, and removing ambiguity to make development easier.

1. APPLICATION SECURITY
FROM THE INSIDE OUT
TALES FROM THE TRENCHES

2. DISCLAIMER

3. $ WHOAMI
Ulisses Albuquerque
Application Security Engineer
@urma
ulissesalbuquerque
• Open-source advocate
• Software engineer turned security professional
• Builder of breaking tools

4. CONTEXT
Cloud-First
• All new applications
and services
designed with focus
on maximising cloud
usage and technical
excellence.
Risk Appetite
• Low resistance to
adoption of new
technologies,
SaaS/PaaS solutions
and developer
autonomy.
Build It, Run It
• Teams own
development and
operations of their
solutions, enabling
technical excellence
and encouraging
risk ownership.
Greenfields
• Early days with full
autonomy to run
application security
program as seen fit
for organisation.

6. IDENTIFYING GAPS
AND LAYING A
PLAN

7. EARLY DAYS (~18 MONTHS AGO)
• Shift-left approach already
agreed upon
• Vendor-centric perspective on
application security and
DevSecOps
People
• Trials on a few tools done in
collaboration with contractors
• Only one (open source) tool
deployed by SRE team for
container image scanning
Tools
• Penetration testing as change
management control for
releases
• Security only involved during
design and delivery
Process

8. AWS SHARED
RESPONSIBILITY
MODEL

9. Security information mixed
with regular CI output in build
logs

10. EARLY DAYS
 Challenges in identifying gaps and mapping tools
 Significant number of repositories (600+) and AWS accounts (100+)
 Multitude of programming languages, including Java, JavaScript, Go,
Kotlin, Swift, Scala, Python and more
 Sophisticated development environment and stacks
 Providing security scan results as early as possible in CI environments
 Leveraging SaaS solutions while ensuring security and confidentiality of
data

11. STACK DIVERSITY
DETERMINES
TOOLSET
TOOLS MUST SUPPORT ALL LANGUAGES
AND TECH STACK COMPONENTS USED

12. STARTING
LEFT OF THE
SDLC

13. LESSONS LEARNED
Not every repository is relevant, but no single individual or team knows
which ones are
Licensing per unique contributor (SCA) can get expensive if you have
turnover and/or outsource
Licensing per repository (SAST) can get expensive with highly
compartmentalized architectures (e.g., microservices)

14. STEALTH MODE

15. LESSONS LEARNED
Automation of security scans is not enough…
SAST reports are useless in text/log output formats
SAST scans can take a long time to complete, resulting in slow builds if
done inline
SCA scans are much better when done inline, but hooking to
dependency management tools can get tricky

16. STARTING LEFT OF THE SDLC
• Individual developers
interested in testing security
tools
• Informal security champions
identified
People
• Checkmarx (SAST) and Snyk
(SCA) chosen as first tools
• Manually triggered scans
across all repositories
Tools
• On demand triage of findings
• No mandatory scans for
change requests
• JIRA tickets created for
confirmed findings
Process

17. PROVIDING FRIENDLY
SECURITY TOOLING
OUTPUT
SECURITY SCAN RESULTS
CAN BE MUCH EASIER TO
INTERPRET WHEN NOT
BURIED IN BUILD LOGS

19. HOW TO PROVIDE
SECURITY
FEEDBACK AS
EARLY AS
POSSIBLE?

20. Code
Change
Pull
Request
CI Build
Artefacts
Cloud
Resources
SHORTENING THE FEEDBACK LOOP

21. OWASP TO THE
RESCUE

23. SHORTENING THE FEEDBACK LOOP
Scan
• Run security tools via
scripts in CI
01
Submit
• Publish scan results to
OWASP DefectDojo
02
Annotate
• Annotate GitHub pull
request with link to
DefectDojo scan results
03

25. OUR SOLUTION:
APPSEC HUB

26. APPSEC HUB FLOW
1. GitHub PR webhooks trigger
build in CI and notify hub
2. Hub requests SAST and SCA
scans
3. SCA and SAST pull code from
GitHub
4. CI scans container using Clair
5. Clair scan log published as
artefact to CI
6. SCA and SAST scan results
fetched and published to PR
7. Clair scan results published to PR

27. APPSEC HUB

28. APPSEC HUB: FUNCTIONAL, NOT PRACTICAL
“BUILD IT, RUN IT” PROVEN EASIER SAID THAN DONE

29. APPSEC HUB:
LIMITATIONS
 Debugging failed scans is a nightmare
 Inconsistent APIs across tools
 Asynchronous APIs with no notification (e.g.,
webhooks) require polling
 Extensive sanity checks to prevent triggering
actions on unmanaged repositories and CI
pipelines
 Side effect: explosive license usage due to
scans on irrelevant code
 Implementation architecture misaligned with
other services/applications in organisation

31. EAT YOUR OWN DOG
FOOD
BUILD AND OPERATE
SYSTEMS THE SAME
WAY THE TEAMS
YOU ARE
SUPPORTING DO

34. EAT YOUR OWN DOG
FOOD
 Converging on stack used by development
teams enabled…
 Better understanding of user experience
 Early identification of common false positives
 Support from development teams
 Credibility and respect

35. LESSONS LEARNED
Scanning code and artefacts with security tools is the bare minimum
Security orchestration > security automation
Security checks and gates must be aligned with existing development
practices and flows
⚠ Security checks should only block if 100% accurate
⚠ Do not create long-lived issues automatically, developers might
address issues as soon as they are reported via pull request annotations
Automation might expose broken or questionable patterns and
processes
Engineering decisions might impact ability to perform security checks (e.g., Snyk automated pull
requests)

36. APPSECD HUB
IMPROVING ON THE
ORIGINAL DESIGN

37. APPSECD HUB
 Lambda orchestration (using
AWS Step Functions)
 Full traceability via state
machine
 Monitoring and alarms with
real time notifications in Slack
 Aligned with best practices
from development teams

38. BUILDING
RELATIONSHIPS
ACROSS TEAMS

40. BEYOND TECHNOLOGY
Most developers consider security a distraction
or a roadblock – often for a good reason
“Security Ivory Tower” is still alive in many
organizations
Encouraging engagement requires giving back
and knowing when to compromise

41. BEYOND
TECHNOLOGY
 Providing security results inline
with pull requests encourages
risk ownership
 Approving PR = approving
associated issues found
 Do not underestimate the impact
of ❌ instead of ✅
 Full audit trail via configuration
management – who submitted the
pull request, who approved, who
merged
 Shared accountability for
security issues

42. https://speakerdeck.com/chriseng/time-to-grow-up-counterproductive-security-behaviors-that-must-end

43. Security Approval
Security Endorsement
Security Advice

44. https://www.slideshare.net/wickett/how-to-effect-change-in-the-epistemological-wasteland-of-application-security

45. CONCLUSIONS
Tools are means of achieving
alignment against security policies
Scanning profiles must adhere to expectations
and have optimal signal-to-noise ratios
Working reference implementations
over reporting violations
Libraries and code patterns provide security
make the developer’s work easier
Remove all ambiguity
Any room for ambiguous interpretation on
policies and expectations might cause problems

46. CONCLUSIONS
 Where are we today?
 200+ GitHub collaborators
 1,200 GitHub repositories
 Supported by a team of 5 people – 3 security architects and 2 security
engineers
 DevSecOps..?
 Ongoing journey
 Left side of SDLC (mostly) taken care of
 Enable other parts of security capability operate at DevOps speeds