Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

(SEC308) Wrangling Security Events In The Cloud


Published on

Have you prepared your AWS environment for detecting and managing security-related events? Do you have all the incident response training and tools you need to rapidly respond to, recover from, and determine the root cause of security events in the cloud? Even if you have a team of incident response rock stars with an arsenal of automated data acquisition and computer forensics capabilities, there is likely a thing or two you will learn from several step-by-step demonstrations of wrangling various potential security events within an AWS environment, from detection to response to recovery to investigating root cause. At a minimum, show up to find out who to call and what to expect when you need assistance with applying your existing, already awesome incident response runbook to your AWS environment.

Published in: Technology

(SEC308) Wrangling Security Events In The Cloud

  1. 1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Don “Beetle” Bailey, AWS Security Josh Du Lac, AWS Professional Services October 2015 SEC308 Wrangling Security Events in The Cloud
  2. 2. What to expect from this session • Tactical follow-on to previous talks • Concrete examples of potential events and how you can handle them • Ideas for increasing security agility • Specific AWS mechanisms to leverage • More than 1 way to catch a cat burglar, so reinvent as needed • Relevant resources, including docs, code, and partners
  3. 3. “Intrusion Detection in the Cloud” redux • AWS-specific areas to monitor for security-concerning events • Prerequisites • Key concepts, such as security role, write-once storage • Key services to leverage, events and behaviors to look for • Example detection of key configuration changes, resource usage anomalies • YouTube search “Intrusion Detection in the Cloud”
  4. 4. “Incident Response (IR) in the Cloud” redux • Ensuring your existing IR process considers AWS • More prerequisites • Mechanisms for mitigation and investigation • Tactics specific to AWS IR, such as constraining exposed AWS credentials • Tactics analogous to traditional IR, modified for AWS, such as Amazon EC2 instance memory dumping, analysis • YouTube search “Incident Response in the Cloud”
  5. 5. Security event wrangling = Response in depth • Types of security events • Detect -> Recover • Investigate -> Protect • Leveraging AWS mechanisms for increased security agility
  6. 6. Example events of concern, signatures • Configuration changes that impact ability to detect or understand events • Activities that are inconsistent with expectations • Activities that violate policy • Resources no longer available • Resources more available than desired • Event detection signatures != commercial product, and may require careful thought vs. operations to develop
  7. 7. Protect, detect, react, recover, etc. Protect Detect Recover Investigate
  8. 8. AWS = Agility for security geeks • Ability to programmatically inventory environment— knowing what you need to protect is key • Awareness of what’s happening, what’s changing, from AWS API activity to application behavior • Detection and alerting mechanisms, freedom to create and flexibility to configure and tune what’s appropriate for YOU • Analysis and response, via the same platform, natively or with AWS partner solutions
  9. 9. AWS CloudTrail • Records AWS API calls for your account and delivers log files to you. • Turn it ON! de/cloudtrail-user-guide.html
  10. 10. CloudTrail events • A record in JSON format that contains information about requests for resources in your account. • Describes which service was accessed, what action was performed, and any parameters for the action. • Helps you determine who made the request. • The event data is enclosed in a Records array. de/send-cloudtrail-events-to-cloudwatch-logs.html
  11. 11. Example CloudTrail event "Records": [{ "eventVersion": "1.0", "userIdentity": { "type": "IAMUser", "principalId": "EX_PRINCIPAL_ID", "arn": "arn:aws:iam::123456789012:user/Alice", "accountId": "123456789012", "accessKeyId": "EXAMPLE_KEY_ID", "userName": "Alice" }, "eventTime": "2015-03-24T21:11:59Z", "eventSource": "", "eventName": "CreateUser", "awsRegion": "us-east-1", "sourceIPAddress": ”", "userAgent": "aws-cli/1.3.2 Python/2.7.5 Windows/7", "requestParameters": { "userName": "Bob" }, "responseElements": { "user": { "createDate": "Mar 24, 2015 9:11:59 PM", "userName": "Bob", "arn": "arn:aws:iam::123456789012:user/Bob", "path": "/", "userId": "EXAMPLEUSERID" } ....
  12. 12. CloudTrail OFF "userIdentity": { "type": "IAMUser", "principalId": "AIDAI5WIMUDR2UZUI62VO", "arn": "arn:aws:iam::000123456789:user/reinvent-sec308", "accountId": "000123456789", "accessKeyId": "AKIAIRAHHRD3PHLUFJLQ", "userName": "reinvent-sec308" }, "eventTime": "2015-09-23T00:41:45Z", "eventSource": "", "eventName": "StopLogging", "awsRegion": "us-west-2", "sourceIPAddress": “", "userAgent": "aws-cli/1.7.25 Python/2.7.5 Darwin/13.4.0", "requestParameters": { "name": "CloudTrail-Default" }, "responseElements": null, ....
  13. 13. Amazon CloudWatch Logs • Monitor, store, and access your log files from Amazon EC2 instances, AWS CloudTrail, or other sources. • Enable in the AWS Management Console, CLI, or via AWS CloudFormation. • Monitor and alarm for specific phrases, values, or patterns. DeveloperGuide/WhatIsCloudWatchLogs.html
  14. 14. CloudFormation -> CloudWatch alarms • Downloadable and editable example CloudFormation template from AWS • Contains predefined CloudWatch metric filters and alarms that enable you to receive email notifications when certain security- related API calls are made in your AWS account • Amazon S3 bucket events, network events, Amazon EC2 events, AWS CloudTrail, and AWS Identity and Access Management (IAM) events cloudformation-template-to-create-cloudwatch-alarms.html
  15. 15. CloudTrail OFF event – Detect "CloudTrailStopMetricFilter": { "Type": "AWS::Logs::MetricFilter", "Properties": { "LogGroupName": { "Ref" : "LogGroupName" }, "FilterPattern": ”{ ($.eventName = StopLogging) }", "MetricTransformations": [ { "MetricNamespace": "CloudTrailMetrics", "MetricName": "CloudTrailEventCount", "MetricValue": "1" } ] } },
  16. 16. CloudTrail OFF event – Detect "CloudTrailStoppedAlarm": { "Type": "AWS::CloudWatch::Alarm", "Properties": { "AlarmName" : ”CloudTrailStoppedAlarm", "AlarmDescription" : "Alarms when StopLogging API call is made", "AlarmActions" : [{ "Ref" : "AlarmNotificationTopic" }], "MetricName" : "CloudTrailEventCount", "Namespace" : "CloudTrailMetrics", "ComparisonOperator" : "GreaterThanOrEqualToThreshold", "EvaluationPeriods" : "1", "Period" : "300", "Statistic" : "Sum", "Threshold" : "1" } },
  17. 17. CloudTrail OFF event – Recover
  18. 18. CloudTrail OFF event – Investigate "userIdentity": { "type": "IAMUser", "principalId": "AIDAI5WIMUDR2UZUI62VO", "arn": "arn:aws:iam::000123456789:user/reinvent-sec308", "accountId": "000123456789", "accessKeyId": "AKIAIRAHHRD3PHLUFJLQ", "userName": "reinvent-sec308" }, "eventTime": "2015-09-23T00:41:45Z", "eventSource": "", "eventName": "StopLogging", "awsRegion": "us-west-2", "sourceIPAddress": "", "userAgent": "aws-cli/1.7.25 Python/2.7.5 Darwin/13.4.0", "requestParameters": { "name": "CloudTrail-Default" }, "responseElements": null, ....
  19. 19. CloudTrail OFF event – Protect Deny permissions for CloudTrail in IAM groups or roles { "Sid": "Stmt0001", "Effect": "Deny", "Action": [ "cloudtrail:DeleteTrail", "cloudtrail:StopLogging" ], "Resource": [ "*" ] }
  20. 20. Multi-Factor Authentication (MFA) • Require unique authentication codes to access AWS websites or services • Hardware or virtual authentication device generates codes • Enter codes manually via AWS Management Console or accompany API requests • Configure via IAM edentials_mfa.html
  21. 21. MFA Deactivate Event ..... "eventTime": "2015-09-20T18:53:02Z", "eventSource": "", "eventName": "DeactivateMFADevice", "awsRegion": "us-east-1", "sourceIPAddress": ”", "userAgent": "", "requestParameters": { "userName": ”bob", "serialNumber": "arn:aws:iam::000019241430:mfa/bob" }, "responseElements": null, "requestID": "d1a9ebf8-5fc8-11e5-9d8f-1bc7c6757e61", .....
  22. 22. MFA Deactivate Event – Detect "MFADeactivateMetricFilter": { "Type": "AWS::Logs::MetricFilter", "Properties": { "LogGroupName": { "Ref" : "LogGroupName" }, "FilterPattern": "{ ($.eventName=DeactivateMFADevice) }”, "MetricTransformations": [ { "MetricNamespace": "CloudTrailMetrics", "MetricName": "MFADeactivateEventCount", "MetricValue": "1" } ] } },
  23. 23. MFA Deactivate Event – Recover Reconfigure the MFA device
  24. 24. MFA Deactivate Event – Investigate ..... "eventTime": "2015-09-20T18:53:02Z", "eventSource": "", "eventName": "DeactivateMFADevice", "awsRegion": "us-east-1", "sourceIPAddress": ”", "userAgent": "", "requestParameters": { "userName": ”bob", "serialNumber": "arn:aws:iam::000019241430:mfa/bob" }, "responseElements": null, "requestID": "d1a9ebf8-5fc8-11e5-9d8f-1bc7c6757e61", .....
  25. 25. MFA Deactivate Event – Protect Use AWS Identity & Access Management to require MFA Management-of-Multi-Factor-Authentication-to-AWS-IAM-Users
  26. 26. S3 object versioning
  27. 27. S3 object deletion event – Detect • Bucket logging? Check. • Bucket versioning? Check. • Continuously reviewing logs …? NO • We can enable push notifications for S3 events that might concern us (for example, deletions) • Configure S3 to detect events like ObjectRemoved • S3 sends alert to the Amazon SNS topic of your choosing • SNS topic sends message to subscribers, such as an email to your
  28. 28. S3 object deletion event – Recover • Restore deleted file from previous version. • Via AWS Management Console, just a couple clicks to download/upload deleted version. • Via CLI/API, just an S3 copy object request, specifying version ID with copy source. • If you enabled versioning AFTER initial object put, version ID will be “NULL”. OK, you can still specify “NULL” as a version to restore from.
  29. 29. Recover deleted S3 object – AWS CLI aws s3api list-object-versions --bucket reinvent2015-sec308 --prefix prod aws s3api copy-object --bucket reinvent2015-sec308 - -copy-source reinvent2015- sec308/prod/important.txt?versionId=null --key prod/important.txt
  30. 30. Recover deleted S3 object (from backup) – AWS CLI aws s3api copy-object --bucket reinvent2015-sec308 - -copy-source reinvent2015- sec308/backup/important.txt?versionId=null --key prod/important.txt
  31. 31. S3 object deletion event – Investigate
  32. 32. S3 object deletion event – Protect • Bucket versioning protects against inadvertent delete or overwrite of objects. • Consider more restrictive policies for credentials, such as specifically disallow S3 object removal. • Additional layer of protection; enable MFA Delete on a versioned S3 bucket. oning.html#MultiFactorAuthenticationDelete
  33. 33. Log-in anomaly event – Detect "ConsoleSignInAnomalyMetricFilter": { "Type": "AWS::Logs::MetricFilter", "Properties": { "LogGroupName": { "Ref" : "LogGroupName" }, "FilterPattern": "{ ($.eventName = ConsoleLogin) && ($.sourceIPAddress != 55.55.*) }", "MetricTransformations": [ { "MetricNamespace": "CloudTrailMetrics", "MetricName": "ConsoleSignInAnomalyCount", "MetricValue": "1" } ] } },
  34. 34. Log-in anomaly event – Recover Add null IAM policy to the user (Deny all permissions): { "Version": "2012-10-17", "Statement": [ { "Effect": "Deny", "Action": [ "*" ], "Resource": [ "*" ] } ] }
  35. 35. Log-in anomaly event – Investigate Look in CloudTrail – Determine what events happened after the ConsoleLogin.
  36. 36. Log-in anomaly event – Protect Add Condition statements to IAM "Condition" : { "IpAddress" : { "aws:SourceIp" : [””] } }
  37. 37. Open security group • ingress has limited validity, but commonly used. • Web server = Likely OK for the Internet to access 80/443. • All of the web server’s OTHER ports? Likely NOT OK to access the Internet. • Policies can vary. No admin ports open to the world? OK. • Creation and change velocity among security groups should be LOW.
  38. 38. AWS Config • AWS resource inventory, configuration history, and configuration change notifications • Discover existing AWS resources • Export inventory of your AWS resources with all configuration details • Determine how a resource was configured at any point in time • Security geeks should LOVE it!
  39. 39. Open security group event – Detect • Subscribe to AWS Config notification topic. • Filter notifications for creation of security groups that might be concerning. You could look for the following, individually or combined: • “SecurityGroup” and “Created” within subject • changeType : “CREATE” within body • resourceType: "AWS::EC2::SecurityGroup” within body
  40. 40. Open security group event – Detect "groupId": "sg-7dc0d21a", ... "ipPermissions": [ { "ipProtocol": "-1", "fromPort": null, "toPort": null, "userIdGroupPairs": [], "ipRanges": [ "" ], "prefixListIds": [] } ], ...
  41. 41. Open security group event – Recover • If responding soon enough to the creation of a new security group and no instances, simply delete the security group. • Otherwise, assign running instances to another security group, and then delete the offending security group. • You can’t delete a default security group, but you can change its rules back to something sane, including no rules.
  42. 42. Delete open security group – AWS CLI aws ec2 delete-security-group --no-dry-run --group- id sg-d3bda2b4
  43. 43. Open security group event – Investigate • Revisit the AWS Config change notification. • Note time, action, and security group ID to correlate to principal and source IP of EC2 API call via AWS CloudTrail. • If possible, engage principal to understand intent or determine if unexplained, such as by external actor and potentially malicious.
  44. 44. Open security group event – Protect • Appropriately constrain or deactivate associated credentials as warranted. • Security group changes, particularly within production, should not be a frequent event, so maintain high vigilance.
  45. 45. Unapproved AMIs Amazon Machine Images • Public AMI • Marketplace AMI • Private AMI • Approved AMIs/“Golden” AMIs
  46. 46. Unapproved AMI event – Detect • Compare launched EC2 instances against a whitelist. • What is a good method to compare against a whitelist?
  47. 47. Let’s use AWS Lambda! • Runs your code in response to events. • Automatically manages compute resources for you. • Create new back-end services where compute resources are automatically triggered based on custom requests. • You can read CloudTrail events with AWS Lambda.
  48. 48. Unapproved AMI event - Recover matchingRecords, function(record, complete) { var params = { InstanceIds: [] }; // List each instance ID for (var i = 0; i < record.responseElements.instancesSet.items.length; i++){ params.InstanceIds.push(record.responseElements.instancesSet.items[i].instanceId); } // Terminate the enumerated instances ec2.terminateInstances(params, complete);
  49. 49. Unapproved AMI event – Investigate Interrogate CloudTrail logs as before • Who launched it? • Where did the request come from? • Which subnet was it being launched into?
  50. 50. Unapproved AMI event – Protect Restrict access in IAM to specific AMIs IDs
  51. 51. Automate IR? • Most, if not all, of the pieces to automate IR exist in AWS • Automated IR = Even greater security agility • Detect -> Protect programmatically • Lambda-fy your IR!
  52. 52. Detecting events in Lambda … var EVENT_SOURCE_TO_TRACK = /; var EVENT_NAME_TO_TRACK = /StopLogging/; var matchingRecords = records .Records .filter(function(record) { return record.eventSource.match(EVENT_SOURCE_TO_TRACK) && record.eventName.match(EVENT_NAME_TO_TRACK); }); … Source: adminuser.html
  53. 53. Responding to events in Lambda … if (matchingRecords.length >= 1) { console.log(’StopLogging detected! Reverting...'); cloudtrail.startLogging(cloudtrailParams, function(err, data) { ….
  54. 54. Responding to events in Lambda
  55. 55. Building a “Lambda Responder” CloudTrail S3 Lambda Lambda SNS
  56. 56. Building a “Lambda Responder” 1. Turn on AWS CloudTrail – Choose an S3 bucket. 2. Create an SNS topic. 3. Update the topic policy to allow event notifications from your S3 bucket. 4. Configure your S3 bucket to send event notifications to the SNS topic. 5. Create an IAM role for the Lambda functions. 6. Create the Lambda functions and process SNS messages. notifications-to-multiple-endpoints/ by John Stamper
  57. 57. Building a “Lambda Responder” • What could you automatically respond to?
  58. 58. Lambda – Automated S3 object recovery ... var bucket = event.Records[0]; var key = event.Records[0].s3.object.key; var backup = ’your-backup-bucket/' + key; var params = { Bucket: bucket, CopySource: backup, Key: key, }; s3.copyObject(params, function(err, data) { // removed for brevity }); ...
  59. 59. Lambda – Automated open security group delete var snsMsgString = JSON.stringify(event.Records[0].Sns.Message); var snsMsgObject = getSNSMessageObject(snsMsgString); if (snsMsgObject.configurationItemDiff.changeType == 'CREATE' && snsMsgObject.configurationItem.resourceType == 'AWS::EC2::SecurityGroup' && snsMsgObject.configurationItem.configuration.ipPermissions[0].ipProtocol == '-1' && snsMsgObject.configurationItem.configuration.ipPermissions[0].ipRanges == '') { var params = { DryRun: false, GroupId: snsMsgObject.configurationItem.resourceId, }; ec2.deleteSecurityGroup(params, function(err, data) { context.succeed(snsMsgObject); }); }
  60. 60. AWS Config -> Lambda … IR aaS? AWS Config Rules! • Extends AWS Config with a powerful new rule system • Use existing rules from AWS and from partners • You can also define your own custom rules • SEC314 - NEW LAUNCH! AWS Config/Config Rules: Use AWS Config Rules to Improve Governance over Configuration Changes to Your Resources
  61. 61. Practice makes perfect • IR game day…YAY! • Tabletop first…yay? • See SEC316 – Harden Your Architecture with Security Incident Response Simulations (SIRS), Jon Miller and Armando Leite
  62. 62. AWS Partner, Dell SecureWorks, IR Support • Customer IR case example • Our IR preparedness “Wish List” for AWS customers • How to contact us
  63. 63. IR Case Example – Background, Event • Dell SecureWorks contacted by an AWS customer, a provider of cloud- based collaboration software • Customer investigated abnormally high CPU usage on Internet-facing servers hosting their customers’ applications • Customer’s review of system logs identified unauthorized logins from a wide array of IP addresses using compromised credentials • Threat actors leveraged the Customer’s compromised web app credentials to gain unauthorized entry and propagate to a multitude of connected resources within the Customer’s AWS environment • Dell SecureWorks performed digital forensics on the Customer’s web applications, AWS instances and snapshots, AWS CloudTrail logs, and suspected on-premise systems
  64. 64. IR Case Example - Response • Dell SecureWorks prepared forensic analysis environment: • Launched forensic EC2 instances within Dell SecureWorks’ VPC • Created S3 bucket for event data storage and transfer of forensic artifacts • Using IAM, Customer provided appropriate access for Dell SecureWorks to: • Acquire snapshots of the affected Customer’s EC2 instances • Transfer snapshots to Dell SecureWorks’ S3 bucket for forensic analysis • Receive access to Customer’s CloudTrail logs for forensic analysis • Using rapidly-deployed forensic toolsets, Dell SecureWorks conducted forensic exam of: • File systems of the Customer’s Internet-facing EC2 instances • Customer’s AMIs • Customer’s AWS CloudTrail logs • Dell SecureWorks provided comprehensive analysis of the incident and affected AWS resources
  65. 65. IR Case Example - Takeaways • AWS enables shorter response times for security events vs. on-premise • Time between engagement kickoff and commencing analysis was drastically reduced • Security event data can be rapidly acquired, staged, and analyzed all within AWS • Appropriate access can be quickly granted to security event responders via AWS IAM • The ability to collaborate on configuration activities directly within AWS minimized time taken for troubleshooting • Creating effective environments for sharing incident response resources and data within AWS is straight-forward • Versus traditional IR, cost savings are also realized via IR within AWS through reduction of the investigation timeline (minimized time to data acquisition, resource setup, and initial analysis)
  66. 66. Our IR Prep “Wish List” for AWS Customers • Take snapshots of all affected or suspected instances • Collect network and instance metadata • Create a restricted-access VPC, Security Group, and/or separate AWS account • Be ready to create temporary users / credentials via IAM • Enable and centralize CloudTrail and CloudWatch logs • Create a dedicated S3 bucket for sharing incident response artifacts
  67. 67. How to Contact Dell SecureWorks • Incident Response Hotline (24x7x365) 1-877-884-1110 • Website • Booth: #446 (next to Docker) Flag me down and/or visit our booth to learn more about Dell SecureWorks’ experience and capabilities and how we are partnered with AWS to provide Incident Response for AWS customers!
  68. 68. AWS Security Best Practices whitepaper • Help for designing security infrastructure and configuration of your AWS environment • High-level guidance for: • Managing accounts, users, groups, and roles • Managing OS-level access to instances • Securing your data, OS, apps, and infrastructure • Managing security monitoring, auditing, alerting, and incident response
  69. 69. External resources – Reading, training • SANS Reading Room, Incident Response • FIRST • CERT, Incident Management
  70. 70. External resources – IR tools, frameworks • Mozilla Investigator (MIG) • Netflix Fully Integrated Defense Operations (FIDO) automated-security.html
  71. 71. Other relevant talks this week • SEC403 - Timely Security Alerts and Analytics: Diving into AWS CloudTrail Events by Using Apache Spark on Amazon EMR, Will Kruse • SEC303 – Architecting for End-to-End Security in the Enterprise, Hart Rossman and Bill Shinn • If you miss(ed) any of them live, they will be on YouTube, just like this talk. • Don’t forget last year’s “Intrusion Detection in the Cloud” and “Incident Response in the Cloud” that are already on YouTube!
  72. 72. AWS Support for security concerns • AWS Support is the one-stop shop for AWS customers, for any concerns, including security related. • If AWS Support cannot immediately address your concerns, they will escalate internally to the appropriate technical team, AWS Security included.
  73. 73. AWS security resources • AWS Security Blog • AWS Security Center • Contact the AWS security team
  74. 74. Summary • Security agility with AWS • Threat vs. policy-driven concerns, enumerate, create signatures, detection mechanisms • Automate IR where you can • Two ways to get more practice: you only get to choose one • We (AWS and our technology partners) are here to help!
  75. 75. Remember to complete your evaluations!
  76. 76. Thank you!