1. If CI/CD have time for
security, so do you
Software development is speeding up; Waterfall to Agile to Continuous
Integration to Continuous Deployment. Do we still have time for security?

2. Who’s afraid of CI/CD?
• I hear from clients that they don’t have time for security because
they’re using Agile or CI/CD
• Is CI more hostile to security?
• If so, where does security fit?

3. Tenants of CI/CD
• Fast - Make automation, builds, setups, deploys, fast and automated
• Early - Do experiments, enable A/B testing, reduce sunk costs
• Often - Build and test all the time
• Responsive - Be reactive to your customers, know your changes won’t break the
app
This is not new!
Originally discussed in Grady Booch’s book
“Object Oriented Design” in 1991 pg. 209

4. CI/CD in a nutshell
• From Thoughtworks (http://www.thoughtworks.com/continuous-integration)
• Check in frequently
• Don’t check in broken code
• Don’t check in untested code
• Don’t check in when the build is broken
• Don’t go home after checking in until the system builds

5. Detect errors quickly
• Make changes all the time
• Build and run your tests quickly
• Trust your tests and know you didn’t break anything critical
• (You do have tests for everything critical, right?)
• Integrate quickly and often so you know where things break, when
you break them!
MartinFowler.com/bliki/FeatureBranch.html

6. Where does security fit?
• Fast - Make automation, builds, setups, deploys, fast and
automated
• Make security fast, automate, match security
testing to time available
• Early - Do experiments, enable A/B testing, reduce sunk
costs
• Get security into epics, stories, Threat Model, use
Training
• Often - Build and test all the time
• Don’t let security issues through, this breaks the build
• Responsive - Be reactive to your customers, know your changes won’t break the app
• Respond to security issues from your customers and team

7. Fast
• Security can’t hold back integration (deploys, testing, etc.)
• Match security assessments to time available
• Static Analysis on dev’s machines, during commit and integration
• Automated scanning on integration and test deployment
• Manual Penetration testing on major changes or periodically
(monthly, quarterly)

8. • Ask users about security
• Get security into stories, epics, requirements, use cases & misuse
cases
• Perform Threat Modeling and Threat Exercises so the team
understands attack surface and assets
• Train the team so they can participate in meaningful security
discussions
Early

9. Early - Threat Modeling
• Can help prioritize what to focus on and what to protect
• Helps ask the right questions
• Get everybody on the same page
• Align and prioritize assets and components
• Enumerates roles and attackers

10. Often
• In CI/CD “everything happens all the time” so too must security
• Don’t break the build
• Add security automation at every layer to help identify breaking builds
• Don’t introduce new security vulnerabilities!
• Trust the tools and training
• For God’s sake don’t introduce vulnerabilities that have been previously
fixed/reported
• Write regression tests for each issue that you can

11. Responsive
• Respond to security threats quickly
• Understand what an attack looks like and plug
the hole quickly
• Disclosure!
Listen to and respond to security researcher keep
them in the loop and fix the issue fast
more info at:
http://blog.securityinnovation.com/blog/2014/06/the-importance-of-vulnerability-disclosure-programs-and-bug-bounties.html

12. Frustratingly Fast and Responsive
• We deliver vulnerability information as we find it
• With one client using CI/CD we were getting fixes for our issues in
sub 1hr time
• They were also pushing out new features, and other bug fixes
• Screenshots on vulns as well as repro steps and videos were
paramount

15. Architect for security
• To go fast, it must be habit
• Centralize security components
• Reduce the likelihood of side effects
when changing code
• Increases confidence in fix
• Decreases downtime due to bugs and integration
• Fix an issue once, don’t see it again

16. Good initial candidates
• Some components to be centralized
• Input validation
• Authentication/authorization
• Data access (SQL and non-sql)
• Encoding
• Encryption
• Key Business Logic
• Code Complete (2nd Edition) still holds up amazingly well
http://www.amazon.com/Code-Complete-Practical-Handbook-Construction/dp/0735619670

17. Automation
• You must know what you’re doing won’t break your build or
deployment
• Automation is key - if it can be automated, it should be
• Build Fast (near real-time) static analysis on dev’s machines to
help identify basic issues early
• SQLi detection (string concatenation)
• XSS detection (failure to encode on output)
• Command injection (concatenation/dangerous functions)
• Remote Code Execution (dangerous functions, known insecure libraries)

18. What does Security Testing look like?
• If you have the tools use automated testing (web app scanner)
as frequently as it will allow
• Scan at least monthly, weekly is better, daily is better still
• Pay attention to your results, consider them breaking the
build
• Tune your automation to reduce all false positives
• Do a rapid (manual) assessment frequently (monthly/quarterly), document new code and
focus testing on new components
• Do a deep (manual) assessment when appropriately (quarterly/annually), this will test
everything

19. What Does Work in Agile
• Allows you to react quickly to security vulnerabilities
• Tighten the feedback loop between you and your users
• React to vulnerabilities in your software and 3rd party software
• Quickly cut out vulnerable systems if necessary
• Have confidence in your solutions (with automated testing in place)

20. What Doesn't
• Cowboy coders with a license to check in and deploy
• Mistakes happen, checks aren’t always performed
• Code may be deployed for weeks or
months before a deep analysis is
performed
• Gives the Audit teams heart attacks
"Debugging is twice as hard as writing the code in the
first place. Therefore, if you write the code as cleverly
as possible, you are, by definition, not smart enough to
debug it." --Brian Kernighan

21. Story Time!
• One client checked in debug code which disabled CC# filtering
• Was discovered when a customer reported they could see their entire CC# instead of **** ****
**** 1234
• Fixed quickly, hours after discovery
• A test case was added to their suite and the issue hasn’t arisen again
• However! The numbers were stored in logs
• Luckily the developer who was at fault remembered this and the logs were flushed
• Another test case was added to check for this

22. Conclusions
• There is always time for security
• You can match your security tasks to the time you have available
• Trusting your developers to “do the right thing” isn’t as scary as you
might think
• Amazingly people react well when you trust them!

23. Contact me!
Kevin Poniatowski
Senior Security Instructor
Security Innovation
kponiatowski@securityinnovation.com
https://securityinnovation.com