Development methodologies like Agile teach developers to make small and quick improvements to code to get features into production quickly. SAAS and web scale companies now push updated code to production using tools such as Jenkins and Chef more than ten times a day. This means that while we get updated features and software delivered faster, there is a side effect of security issues being introduced into production quicker than ever. In this talk we’ll explore why security professionals should embrace the very same DevOps tools like Jenkins, Chef, Puppet, etc. to automate security checks and remediation on code and automated infrastructure.

1. DevOps, Agile, $buzzword
and
How the hell to secure
them
By:
Francisco Donoso

2. What are you talking about?
● What hell are Devops, Agile, and
$INSERT_BUZZWORD_HERE
● What kind of challenges do these methodologies present to
security folks
● How to use these tools to your advantage
○ Config management suites 101
○ Automated security scans when code is committed
● Cats

3. Who are you anyway?
● Have done the security analyst / security engineer thing
● Have done the security consulting thing
● MSS Architect leading a team of Security Engineers and Software Devs
● DevOps Security Engineer at:
We’re hiring!

4. Days of old

5. Dev VS Ops

6. 1. Weeks writing code
2. Weeks “testing” code
Dev VS Ops

7. Ship it!
1. Weeks writing code
2. Weeks “testing” code
3. Hand it over the ops team for deployment
Dev VS Ops

8. 1. Weeks writing code
2. Weeks “testing” code
3. Hand it over the ops team for deployment
Dev VS Ops
4. Get out of the office as quickly as possible

9. 1. Weeks writing code
2. Weeks “testing” code
3. Hand it over the ops team for deployment
4. Get out of the office as quickly as possible
Dev VS Ops
5. Deployment is a complete train wreck

10. Dev VS Ops
1. Weeks writing code
2. Weeks “testing” code
3. Hand it over the ops team for deployment
4. Get out of the office as quickly as possible
5. Deployment is a complete train wreck
6. Blame

11. Dev VS Ops
1. Weeks writing code
2. Weeks “testing” code
3. Hand it over the ops team for deployment
4. Get out of the office as quickly as possible
5. Deployment is a complete train wreck
6. Blame
7. Roll back….

12. What is Agile?

13. Agile Development Methodology

14. Agile Development
Methodology
● Frequent changes to production
● New features (new code!!!) being introduced every sprint (often
weekly)
● Less time for QA / Security review

15. Intro to DevOps

16. DevOps 101
DevOps is the practice of operations and development engineers participating
together in the entire service lifecycle, from inception to deployment.
Operations staff making use many of the same techniques & tools as
developers for their systems work.

17. DevOps 101

18. DevOps Tool Belt

19. Continuous Integration

20. Continuous Delivery
Dev Stage “Canary” Prod

21. Server management

22. Days of old
Billy Susie Evan

23. Horizontal scaling

24. Config Management / Orchestration

25. “Infrastructure as code”
Easy to identify changes in
production configurations

26. “Infrastructure as code”
Version-able, Repeatable, Testable
Consistency - Ensure that your server builds meet security
requirements.
Makes life easier - Automate repeated tasks
Change Management - “Code” review and be less scared of
change! (patching anyone..? Bueller? Bueller?)
DR is easier!

27. How the big boys do it
Etsy: Average 25 deploys per day Amazon: Well….

28. Agile + DevOps = ?

29. How to secure this stuff

30. Config Management for good
● Base security configs
● Centralized logging & auth
● Host based firewall rules
● Automated Vuln Scans
● Easier Patching

31. Taking a step back..Config Management
Roles:
● Web Server
● Database Server
● Mail Server
Environments:
● Production
● Staging
● Development

32. Automate security
Configure a base hardened config that is used everywhere
Configure centralized auth
Configure centralized logging FOR ALL THE THINGS
Configure host based firewalls per role & environment
Quickly mitigate insecure configurations

33. Example Time
Write a recipe to install and configure rsyslog to
send auth logs to centralized logging servers

34. Automatically logging all the things
1.Automatically install rsyslog
2.Automatically write a configuration file with custom
“destination” log server based on the “nodes”
environment
3.Start the service
4.Profit (from logs...yeah...logs)

35. Node Attributes
Attribute Name
Attribute Value

36. Installing rsyslog with Chef
1
2
1.For the package named “rsyslog”:
2.Install it

37. 1.Write the rsyslog config file
2.Use the ruby template file as the source
3.Replace the “logserver” variable in the template with the proper
log server based on environment
Write a custom rsyslog config file
1
2
3

38. Enable and start the service
1
2
1.For the service named “rsyslog”:
2.Enable it during boot & start it immediately

39. Putting it all together (recipe)

40. Continuing with our
base_server recipe

41. Attribute precedence
1.Role
2.Environment
3.Recipe

42. Node Attributes
Attribute Name
Attribute Value
Environment
Attribute

43. Remove unused packages
1
2
3
1.List of packages to remove saved as a “node”
attribute
2.For each package in the list:
3.Remove the package

44. Make it easier for later
1
2
3
1.List of packages to upgrade and their versions
saved as a “node” attribute
2.Upgrade the package:
3.To the specified version

45. Making security convenient

46. Put Security on ALL THE THINGS
NodeJS_server
base_server
$APP_server
MySQL_server
NGINX_server

47. Demo Time

48. Firewalling in a
DevOps World

49. Database VLANDMZ
Traditional Firewalling

50. Database VLANDMZ
Traditional Firewalling

51. DMZ
Traditional Firewalling
Database VLAN

52. Networks @ Scale

53. Datacenter network
Networks @ Scale

54. 1.Find all nodes with a role of “web_server”
2.For each node in the “webnodes” variable:
3.Add a simple iptables rule to allow inbound to MySQL
4.Replace the source with the value for of that web servers IP Address
Automatically generate firewall rules
3
1
2
4

55. Let’s take a step back
Testing:
● Unit
● Integration
● Regression
● Security
CI System

56. How CI does its thing
Dev Cat
Code Repo
CI System
Did you commit yet?
Did you commit yet?
Did you commit yet?

57. How CI does its thing
CI System ohhh a
commit!
1.Compile the source (if needed)
I’m a
VM!
Build Agent
Run tests
2.Spin up VM / container
3.Deploy code
4.Run tests
5.Profit!

58. 1.NMAP
2.Vulnerability scans
3.OWASP ZAP scans
4.Static code analysis
Automatic Security testing

59. Infrastructure is code
Test Like It

60. Chef Recipe Testing
I’m a
VM!
Test Kitchen

61. 1
2
3
Chef Recipe
Testing

62. Chef Recipe Testing
I’m a
Cheff’d
VM!
Test Kitchen
Vulnerability
Scanner1
2

63. What we covered
● Intro to Agile & Devops
● How we NEED to automate to properly secure our
infrastructure & code
● Config management suites 101
○ Build a base server config that is used everywhere
● Automated security tests

64. THANKS FOR LISTENING
Q A&
I’ll try to answer questions now
Q&A Session
@Francisckrs Francisco Donoso Francisck