Introduction to DevSecOps on AWS
•
19 likes•5,883 views
Learn the processes that Amazon’s engineers use to practice DevOps and discuss how you can bring these processes to your company by using AWS tools.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Introduction to DevSecOps on AWS
Similar to Introduction to DevSecOps on AWS (20)
More from Amazon Web Services
More from Amazon Web Services (20)
Recently uploaded
Recently uploaded (19)
Introduction to DevSecOps on AWS
- 1. © 2016 AWS and affiliates, all rights reserved Introduction to DevSecOps on AWS Chuck Meyer Security Solutions Architect
- 2. © 2016 AWS and affiliates, all rights reserved What is DevOps? Cultural Philosophy Practices Tools
- 3. © 2016 AWS and affiliates, all rights reserved Competing Forces Business Development Operations Build it faster Keep it stable Security Make it secure
- 4. © 2016 AWS and affiliates, all rights reserved Who is DevSecOps DevSecOps is • Team/Community effort, not a person • Automated and autonomous security • Security at scale DevSecOps role • Not there to audit code • Implement the control segments to validate and audit code and artifacts as part of the CI/CD process Security OperationsDevelopment
- 5. © 2016 AWS and affiliates, all rights reserved Terminology DevSecOps / Security Automation / Security at Scale Make up your mind…
- 6. © 2016 AWS and affiliates, all rights reserved Why security automation Reduce risk of human error - Automation is effective - Automation is reliable - Automation is scalable
- 7. © 2016 AWS and affiliates, all rights reserved Why security automation Reduce risk of human error - Automation is effective - Automation is reliable - Automation is scalable Don’t worry…we still need humans
- 8. © 2016 AWS and affiliates, all rights reserved Why security automation High pace of innovation is great
- 9. © 2016 AWS and affiliates, all rights reserved Why security automation We also want high pace of: • Detection • Alerting • Remediation • Countermeasures • Forensics
- 10. © 2016 AWS and affiliates, all rights reserved What is DevSecOps Three flavors - Security of the CI/CD Pipeline - Automated IAM roles, Jenkins server hardening, etc. - Security in the CI/CD Pipeline - Automated security tests, code analysis, etc. - Security Automation - Automated Incident Response Remediation, forensics, etc.
- 11. © 2016 AWS and affiliates, all rights reserved Security in/of the CI/CD Pipeline
- 12. © 2016 AWS and affiliates, all rights reserved What is DevSecOps DevOps = Efficiencies that speed up this lifecycle DevSecOps = Validate building blocks without slowing lifecycle developers customers releasetestbuild plan monitor delivery pipeline feedback loop Software development lifecycle Security
- 13. © 2016 AWS and affiliates, all rights reserved CI/CD for DevOps Version Control CI Server Package Builder Deploy Server Commit to Git/masterDev Get / Pull Code AMIs Send Build Report to Dev Stop everything if build failed Distributed Builds Run Tests in parallel Staging Env Test Env Code Config Tests Prod Env Push Config Install Create Repo CloudFormation Templates for Environment Generate
- 14. © 2016 AWS and affiliates, all rights reserved Version Control CI Server Package Builder Promote Process Validate Git-SecretsDev Get / Pull Code AMIs Log for audit Staging Env Test Env Code Config Tests Prod Env Audit/Validate Config Checksum Continuous Scan CI/CD for DevSecOps Send Build Report to Security Stop everything if audit/validation failed CloudFormation Templates for Environment
- 15. © 2016 AWS and affiliates, all rights reserved What Does DevSecOps CI/CD Give Us? • Confidence that our code is validated against corporate security policies. • Avoid infrastructure/application failure in a later deployment due to different security configuration • Match DevOps pace of innovation • Audit and alert • Security at scale!
- 16. © 2016 AWS and affiliates, all rights reserved Security Automation
- 17. © 2016 AWS and affiliates, all rights reserved AWS CloudFormation primer Infrastructure is code
- 18. © 2016 AWS and affiliates, all rights reserved AWS CloudFormation Primer Allows you to define a “template” • Composed of different “resources” • Provision that template into repeatable, live, “stacks”. CloudFormation (CFn) provides a single service interface • Let CFn perform state changes and govern who calls CFn Treat as Code • Check in your templates CFn templates can hook into external configuration management frameworks • Jenkins/Chef/Puppet/etc.
- 19. © 2016 AWS and affiliates, all rights reserved AWS CloudFormation Stacks JSON Template Stack Stack Stack Dev Test Staging Prod Demos Regions
- 20. © 2016 AWS and affiliates, all rights reserved Split Ownership Configurations Who knows your solution best? • Dev, Infra, Sec…? • Delegate ownership • Infra – VPC design, IGW Deployment, Subnets, etc • DevOps – EC2, Elastic BeanStalk, RDS, DynamoDB, etc • OS Patching, Security Agent Deployments, IAM Roles, etc Use Yaml and split file into chunks or functions • Separate file sources with access control – Use IAM/VPC-E/etc. • Push files -> Validate -> Merge files -> Validate -> Deploy -> Validate AWS CodePipeline or Jenkins for deployment • Promotion flows • Move from manual to Automation based on validation quality • Excellent for merging jobs of split configurations
- 21. © 2016 AWS and affiliates, all rights reserved Merging From single file or multiple files • Maintain access control using policies • Use different source stores if needed Based on function/state Reusable patterns Maintain order, especially of validation • Security validation last to execute • Security should always win
- 22. © 2016 AWS and affiliates, all rights reserved Validation Keep track of what section you are validating • Stage vs Prod • Merged vs separated Validate often and log/alert • Validate part and end result • Run-time validation Tools • AWS CodeCommit • AWS Lambda • Config / Config Rules • CloudWatch Logs / CloudWatch Events • Etc.
- 23. © 2016 AWS and affiliates, all rights reserved Where else can this be applied? CloudFormation Template Task Definition Application Specification File (AppSpec file) …and more. AWS CloudFormation AWS CodeDeployAmazon EC2 Container Service
- 24. © 2016 AWS and affiliates, all rights reserved AWS Tooling Execution • Lambda Tracking • AWS Config Rules Amazon CloudWatch Events • AWS CloudTrail • AWS Inspector Track/Log • Amazon CloudWatch Logs • Amazon DynamoDB Alert • SNS Third party Open Source
- 25. © 2016 AWS and affiliates, all rights reserved Other resources / Open Source • Some of the projects out there: – ThreatResponse.cloud https://threatresponse.cloud – Cloud Custodian https://github.com/capitalone/cloud-custodian – Security Monkey https://github.com/Netflix/security_monkey – FIDO https://github.com/Netflix/Fido • And many more…
- 26. © 2016 AWS and affiliates, all rights reserved Automatic Incident Response Remediation
- 27. © 2016 AWS and affiliates, all rights reserved Creating a blueprint Continuous / Event based Config Rules CloudWatch Events Is it region specific Will action risk breaking something Yes: Call human No: Lambda Will enable add cost Yes: Based on possible cost limit call human No/Minor: Set rules Is there a source of truth Config Rules: Check previous •Caution on multiple events CWE: Check tag/DDB •Have default value Action Revert change based on above Forensic Is it human (or unknown source) or machine (CI/CD) CI/CD: Create ticket (Jira etc) Human: Should we countermeasur e/prevent? Are they using MFA •No: Add MFA (external Lambda) First occurrence (check DDB) •Yes: Disable account/Keys Alert High: SMS/Page Low: Email/tracking system Logging Is it sensitive Yes: Encrypt (KMS) No: Cleartext Always: Access control
- 28. © 2016 AWS and affiliates, all rights reserved The anatomy of security automation Mode Section Actions Initiate React Config Rules / CloudWatch Events / Log Parsing Trigger Lambda Learn Lambda / CloudWatch Logs Execution Priority Action Restart service, delete user, etc. Forensics Discover: Who/where/when, allowed to execute? Countermeasure Disable access keys, isolate instance, etc. Alert Text/Page, email, ticket system Logging Database, ticket system, encrypt data?
- 29. © 2016 AWS and affiliates, all rights reserved How do I know what happened - Config
- 30. © 2016 AWS and affiliates, all rights reserved The key to Custom Rules response = client.put_evaluations( Evaluations=[ { 'ComplianceResourceType': 'string', 'ComplianceResourceId': 'string', 'ComplianceType': 'COMPLIANT'|'NON_COMPLIANT'|'NOT_APPLICABLE'|'INSUFFICIENT_DATA', 'Annotation': 'string', 'OrderingTimestamp': datetime(2015, 1, 1) }, ], ResultToken='string’ )
- 31. © 2016 AWS and affiliates, all rights reserved How do I know what happened – CloudWatch Events { ”account”: “111111111111”, ”region”: “us-east-1”, ”detail”: { ”eventVersion”: “1.02”, ”eventID”: “c78ce8de-46ee-4fea-bcf4-0e889d419f2f”, ”eventTime”: “2016-01-18T03:32:18Z”, ”requestParameters”: { ”userName”: “trigger” }, ”eventType”: “AwsApiCall”, ”responseElements”: { ”user”: { ”userName”: “trigger”, ”path”: “/”, ”createDate”: “Jan 18, 2016 3:32:18 AM”, ”userId”: “AIDAIKL7LKTAUFPNJQ3LY”, ”arn”: “arn:aws:iam::111111111111:user/trigger” } }, ”awsRegion”: “us-east-1”, ”eventName”: “CreateUser”, ”userIdentity”: { ”userName”: “IAM-API-RW”, ”principalId”: “AIDAI5SJPHVGH1WK7HTQS”, ”accessKeyId”: “AKIAIGYEYSX4EVED52YA”, ”type”: “IAMUser”, ”arn”: “arn:aws:iam::111111111111:user/IAM-API-RW”, ”accountId”: “111111111111” }, ”eventSource”: “iam.amazonaws.com”, ”requestID”: “13bb2739-bd94-11e5-9abd-af4e7ff9090f”, ”userAgent”: “aws-cli/1.9.20 Python/2.7.10 Darwin/15.2.0 botocore/1.3.20”, ”sourceIPAddress”: “111.112.113.114” }, ”detail-type”: “AWS API Call via CloudTrail”, ”source”: “aws.iam”, ”version”: “0”, ”time”: “2016-01-18T03:32:18Z”, ”id”: “d818DD19-7b16-4e1d-a491-794a26b51657”,
- 32. © 2016 AWS and affiliates, all rights reserved Different sources have different event ”eventName”: “CreateUser”, ”userIdentity”: { ”userName”: “IAM-API-RW”, ”principalId”: “AIDAI5RTPJGHE43K7GEQS”, ”accessKeyId”: “AKIADSJGHSXRKVDM52DA”, ”type”: “IAMUser”, ”arn”: “arn:aws:iam::111111111111:user ”accountId”: “111111111111” ”eventName”: “CreateUser”, "userIdentity": { "principalId": "AROGKTYFTCBFKTESCEVK:henrikj", "accessKeyId": ”GFSHKUOLZG53JE5DHKRC", "sessionContext": { "sessionIssuer": { "userName": ”AssumeAdministrator", "type": "Role", "arn": "arn:aws:iam::111111111111:role/Administrator", "principalId": "AROSKTRDFTXBUFLSKCEVK", "accountId": " 111111111111 " }, "attributes": { "creationDate": "2016-01-18T16:50:04Z", "mfaAuthenticated": "false" } }, "type": "AssumedRole", "arn": "arn:aws:sts::111111111111:assumed- role/Administrator/henrikj", "accountId": "111111111111"
- 33. © 2016 AWS and affiliates, all rights reserved How can I get the different events? import json def lambda_handler(event, context): eventdump = json.dumps(event, indent=2) print("Received event: " + json.dumps(event, indent=2)) return eventdump
- 34. © 2016 AWS and affiliates, all rights reserved Risks • You can now automatically mess up your approved changes • No proper alerting and follow-up on automatic events • Over/under complicated scripts • No info on desired state • Race the hacker…automation wars!
- 35. © 2016 AWS and affiliates, all rights reserved Best practices Implement “Compliance Status” for easy overview • Use pre defined checks • Create extended custom checks • Fix the issue while checking Evaluate/remediate changes/events in your account • Doesn’t replace log analysis (Machine Learning FTW) • Protect against changes made by (un)authorized accounts • Automatic remediation for critical events • Do forensic on the fly Always Log and Alert!
Editor's Notes
- Lets start looking at what DevOps is… Three things: Cultural philosophy on how changes and deployments are handled within the organization Practices around this What tools you have/use/develop to perform this
- Developers are paid to change things i.e. write code, Ops folks are paid to NOT change things and keep things stable Adding security to the mix we can say that security is paid to make sure what development is doing is not introducing insecurity that ops then will make public
- The perimeter is no longer an option… Security, now more than ever, is an arms race… The only way to win: - Customer focus - Open and transparent - Iteration over perfection - Hunting over reaction
- Doesn’t matter if you need to run it one time or 1000 times…
- Don’t worry…we still need humans, just focus
- - Lets look at a software development lifecycle - here's the general development lifecycle for an application or service - every new feature or bug fix goes through this process - developer writes code, code is built and unit tested, app is deployed to a testing environment for deeper testing, finally given a thumbs up and deployed to production where customers can use it - after that happens, the company can collect feedback from customers, make decisions, and continue to iterate and improve the product - the faster you can complete that loop, the faster you can innovate Where does security come in here? It’s important to understand that we cannot implement one point of presence We need to integrate with the flow of events in order to be agile and elastic and not be a blocker for pace of innovation
- Example of a CI/CD flow
- Add Git-Secrets Looking at the same flow using our secret DevSec goggles we see a different picture We need to inject ourselves in all parts and sections of the flow Highlight promotion process and manual vs automated process as segway to next slide Mention git-secrets to look for keys on commits
- One of the key components in this is treating your infrastructure as code.
- You may want to stage that application stack through Dev/Test/Prod. You may want to have a stripped down version of the stack for demo purposes. You may want to have the stack in multiple regions. You may want to package the configuration and ship it to a customer. You can also version control the template you have carefully designed. Templatization and replication is useful in other scenarios as well. If you want everyone in your company to use a standard VPC configuration, you can capture the standard configuration in a CloudFormation template and have everyone use it. If you have an IT service catalog that needs to stamp out copies of services to multiple users, you can use CloudFormation as a building block.
- Poll: Who knows what Yaml stands for (Yaml Aint Markup Language) Important to validate post merging to get the correct build/State Notice I mentioned Validate…it’s important!
- If security fail, the flow fails
- Keeping track of target is important when working on checked in resources thru for example jenkins jobs. Validating the wrong file/state can cause instant failures or insecure deployments We will show more on Run-time validation shortly… Script is validating that all of the IP addresses within the CFn are from the corporate CIDR block
- Where this becomes really important is when you look at the actual components and what you can do in terms of securing and validating the workload. CFN - Explain CloudFormation template and how you can use it to enforce certain AMIs, network config, SG, etc. ECS - Explain Task Definition and how this allows you to enforce port mappings, CPU/RAM usage, etc. CodeDeploy - Explain AppSpec.yml and how you can run validation scripts and fail deployment if they fail – define what software to install and what lifecycle hooks to take action upon In general, you have a software artifact that you can deploy with an AWS service, and these services can be configured to enforce your security processes Validation is key! Task Definition Which Docker images to use with the containers in your task How much CPU and memory to use with each container Whether containers are linked together in a task What (if any) ports from the container are mapped to the host container instance Whether the task should continue to run if the container finishes or fails The command the container should run when it is started What (if any) environment variables should be passed to the container when it starts Any data volumes that should be used with the containers in the task
- !30s slide! Make new slide for best practices if needed We are not going to focus on the tools but quick roundup. Pen is mightier then the sword…but Lambda is running circles around that pen...
- Config Rules only in us-east-1 atm Region specific, IAM = us-east-1
- Priced based on number of active rules per month $2.00 per active rule per month with account-level allowance of 20,000 evaluations per active rule. Overage of $0.0001 per evaluation Evaluation: Single result reported for the rule/resource. Evaluations are shared across rules in account. Active Rule: Rule with at least one evaluation that month Customer Managed Rules may incur an additional charges from AWS Lambda
- Why is this important (logging)
- First step for any new function…Introducing…the dump function Easy, just tie to the trigger and dump
- Pre defined checks have a growing library