Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

You Can’t Protect What You Can’t See: AWS Security Monitoring & Compliance Validation from Adobe (SAC309)


Published on

Ensuring security and compliance across a globally distributed, large-scale AWS deployment requires a scalable process and a comprehensive set of technologies. In this session, Adobe will deep-dive into the AWS native monitoring and security services and some Splunk technologies leveraged globally to perform security monitoring across a large number of AWS accounts. You will learn about Adobe’s collection plumbing including components of S3, Kinesis, CloudWatch, SNS, Dynamo DB and Lambda, as well as the tooling and processes used at Adobe to deliver scalable monitoring without managing an unwieldy number of API keys and input stanzas. Session sponsored by Splunk.

AWS Competency Partner

Published in: Technology

You Can’t Protect What You Can’t See: AWS Security Monitoring & Compliance Validation from Adobe (SAC309)

  1. 1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Randy Young, Splunk Scott Pack, Adobe November 29, 2016 SAC309 You Can’t Protect What You Can’t See AWS Security Monitoring & Compliance Validation
  2. 2. What to expect from the session •Learn how to automate data collection for security monitoring and validate compliance for large numbers of AWS accounts. •Learn how Splunk & the Splunk App for AWS can enable you to managing your AWS environment.
  3. 3. Presenters • Scott Pack • Security Engineer @ Adobe • SLC, UT • 2 Year AWS User • 4 Year Splunker • Proudly DQd at 3 Pinewood Derbies • Randy Young • Principal Product Manager @ Splunk • Bezerkly, CA • 8 Year AWS User • 3 ½ Years a Splunker • Proud Dubs Season Ticket Holder R
  4. 4. The background Digital Marketing ~55k physical hosts across 30 sites Collection of ~20 admin teams. • Different tech stacks, but mostly *nix Monitoring Toolset: • Netflow, FPC, IDS, Network Transaction S
  5. 5. Security monitoring 5 Security Engineering: • Build & Maintain Monitoring Toolset • Define (w/ SOC) “Security Notables” • Work with Internal Audit to gauge compliance Security Operations: • Event Analysis • “Hunting” • Investigation • Incident Response S
  6. 6. What is Splunk? Platform for Machine Data Correlation & Enrichment Field Extraction Reporting & Alerting Data Collection & Field Extraction Multiple use cases across one platform R
  7. 7. What can Splunk do for your AWS environment? 7 Splunk App for AWS EC2 EMR Amazon Kinesis Route 53 VPC ELB S3 CloudFront CloudTrail CloudWatch Amazon Redshift SNS API Gateway Config RDS CF IAM Lambda Explore Analyze Dashboard Alert Act AWS Data Sources R
  8. 8. Shift to the cloud 8 Lots of accounts … > 200 Dozens of teams, thousands of instances Missing data to: • Detect/respond to incidents • Making assurances to Compliance We received a mandate: Fix this • Get whatever visibility you can • Minimize risk of operations impact • Be cost sensitive S
  9. 9. AWS security incidents 9 1. Infrastructure Impact Baddie impacts the infrastructure as an external user (DDOS) 2. Host Compromise Baddie has some control of a host. (Command Injection) 3. Account Compromise Baddie interacts as an authenticated AWS user. (Account Takeover) S
  10. 10. Initiative goals Identify & collect security relevant data Analysis the same as on-premises Data -> Splunk ES -> SOC Minimize operations impact Limit IAM users No risk to services Quick setup 10 S
  11. 11. Data sources S
  12. 12. AWS native sources 11/30/201612 CloudTrail API Usage & Logging VPC Flow Logs Virtual Interface Connectivity AWS Config Account Configuration & Inventory ELB Access Logs Load Balancer Logging Trusted Advisor Security Practice Checks Identity & Access Management Credential Report R
  13. 13. Data examples 13 CloudTrail VPC Flow Logs ELB Access Logs Config Credential Report R
  14. 14. Cross-account authentication 14 IAM users • Use API Keys directly Roles • AWS Security Token Service • Can be “assumed” by a specified principal • Authenticate to an aggregation account user • Assume the cross-account role • Retrieve temporary access keys • Make calls with temporary keys Tutorial: Delegating Access using IAM Roles - Shon Sha re:Invent 2014 - S
  15. 15. A few more AWS services 15 S3 – File/Object Storage Lambda – Code without Instances Amazon Kinesis – Data Streaming CloudWatch Logs SNS – Notification Service DynamoDB – NoSQL Database S
  16. 16. Collection plumbing: S3 S3 Buckets: • ELB (1 per region) • Permit PutObject from ELB IAM Roles • Config • Permit PutObject from • Config Parsed • CloudTrail • Permit PutObject from • Trusted Advisor Results • Permit PutObject from Lambda execution IAM role 11/30/201616 AWS ELB Account IDs for Log Delivery: S
  17. 17. Collection plumbing: VPC flows Amazon Kinesis stream: • 1 per region CloudWatch log destinations • 1 per region • Directs to region-local Amazon Kinesis stream 17 S
  18. 18. 18 Aggregation 18 CloudTrail VPC Flow Logs Config ELB Access Logs Trusted Advisor IAM Amazon S3 Per Region CloudWatch Per Region CloudWatch Destination Monitored Account Aggregation Account S
  19. 19. Registration S
  20. 20. 20 CloudFormation Resources: Config Role FlowLogs Role SecEng Role SNS Notification Role’s Done! Inputs: Description Jira Queue Registration Lambda Registration DynamoDB Monitoring registration S
  21. 21. Registration through web UI 11/30/201621 S
  22. 22. 22 Scheduled delivery enforcement Distributor Handler Config STS Config Handler IAM Credential Report STS Distributor CloudWatch CloudWatch Scheduled retrieval & storage S
  23. 23. Dashboards & analysis S
  24. 24. Splunk apps & add-ons • Input Methods: S3 • Input Sourcetypes: CloudTrail, VPC Flows, ELB Access Logs • Parsing Handler: GZIPMessageHandler 11/30/201624 Aggregation reduces amount of Splunk inputs: 26 Total Inputs • S3: 14 • Amazon Kinesis Inputs: 10 • Additional Logging: 2 Currently running on a dedicated Heavy Forwarder. • If needed, split regions to different forwarders. S
  25. 25. Sourcetypes, lookups, and other fun 25 Sourcetypes: Cheated off the Splunk App for AWS. • Set JSON KV format and check line-breaks Use HTTP Event Collector FOR DynamoDB Registrations • Scheduled lookup-generating search • Auto lookups on each sourcetype Tagging into Enterprise Security data models • ELB Access Logs & VPC Flow Logs right out of the box S
  26. 26. Onboarding dashboard 26 S
  27. 27. Account overview S
  28. 28. Compliance checks Inspect Config + Credential Reports + Bunches more Query per Standard/Compliance Requirement S
  29. 29. Resource lookup S
  30. 30. Example ES correlation rules 30 • Console logins from outside org IP space • Flows to/from threat actors • Instance increase by X% within 24-hours • AMI sharing to non-org AWS account • URI/user agent web application attacks • Multiple service API denies for 1 API key within X mins • (Nimbostratus – Andres Riancho, BlackHat 2014) S
  31. 31. Things that can go wrong: S
  32. 32. Splunk hints 32 Amazon Kinesis Modular Input* • Can chew up memory. • /opt/splunk/etc/apps/kinesis_ta/bin java_args = [ JAVA_EXECUTABLE, "-classpath",CLASSPATH,"- Xms512m","-Xmx512m", "- Dsplunk.securetransport.protocol="+SECURE_TRANSPORT,JAVA_MAIN _CLASS] Config snapshots are jsonormous • Use Lambda to split out the resources. * You can now use the Splunk TA for Kinesis Inputs S
  33. 33. AWS hints ELB permission granularity restrictions • ModifyAttributes Keep an eye on capacity. Watch: • DynamoDB read capacity • Amazon Kinesis shard usage AWS internal actions • Auto Scaling • EMR S
  34. 34. Where we’re at right now • 57 AWS accounts currently enrolled • ~3 TB/day • Haven’t broken any accounts yet! • Finding more data sources • Config Rules • Amazon Inspector • Automating our AWS security policy audit • Written a handful of Splunk Enterprise correlation rules • Actioned by SOC • Automated Jira ticketing for remediation 11/30/201634 S
  35. 35. Make machine data accessible, actionable and valuable to everyone. 35 R
  36. 36. Splunk and AWS – Customer value 36 “Customers love the agility of AWS together with the end-to-end visibility of Splunk.” Andy Jassy, AWS CEO R
  37. 37. Operational Intelligence Security Intelligence - Etc. AWS data leveraged across multiple use cases Financial Intelligence R
  38. 38. Operations Intelligence - What is my EBS footprint and posture across all my accounts and all my regions? - Who started/stopped/restarted what instances and when? - What EC2 instances are underutilized and perhaps overprovisioned? - What is the traffic volume into my VPC and where is it originating from? - Why are certain resources unreachable from certain subnets/VPCs? - List resources with missing or non- conforming tags? - Etc. Security Intelligence - Who added that rule in the security group that protects our application servers? - Where is the blocked traffic into that VPC coming from? - What was the activity trail of a particular user before and after that incident? - Alert me when a user imports key pairs or when a security group allows all ports - What instances are provisioned outside of a VPC, by whom and when? - What security groups are defined but not attached to ay resource? - Etc. - Etc. Sample use cases for AWS data Financial Intelligence - How many instances are you running? - What Reserved Instances have you purchased in the past? - What is your Reserved Instance utilization? - How much are you paying per account? - How much are you using per service across all accounts? - How many Reserved Instances should I buy based on usage? - Is this account within budget this month, and how have they tracked in the last year? - Etc. R
  39. 39. Now you have all this data… what do you do with it? HR Director: Good afternoon… You: (smile nervously) HR Director: Joe was let go today. Can you close his account. I want to get an email if his account does anything strange this weekend. You: (nod) And create an alert. R
  40. 40. sourcetype=aws:cloudtrail userIdentity.userName=joe|table _time event* user* Save as alert > Email action R
  41. 41. Now you have all this data… what do you do with it? CFO: Good Afternoon… You: (smile nervously) CFO: Our production account’s spending is on track, but I need YOU to cut our development account spend by 1/3. You: No problem! R
  42. 42. AWS tag-based instance auto start/stop 43 Weekends Non-Working Hours 1. Create IAM user ‘robot’ 2. Install AWS CLI on splunk host 3. Define tag: PowerSave=LongRun/ RareRun/Normal on each instances 4. Create splunk alert • CRON, run in morning/night • SPL to search instances by tag • Alert action to call AWS CLI to batch start/stop instances And save 40% Development cost! R
  43. 43. Now you have all this data… what do you do with it? Developer: I am going to cut out early. By the way, I ran a script and created a bunch of untagged EC2 instances. Can you help me find them? Have a great weekend! You: What the #*$%! R
  44. 44. Tag AWS resource properly Find untagged EC2 instances • sourcetype=aws:description source="*:ec2_instances" NOT "tags.Name"=*| table region id instance_type ip_address key_name Define a naming conventions for EC2 instance and enforce it • DLA_Jove_testEC2Cmd. D: Dev, L: Linux, A: AWS project • <Role><OS><Project>_<Owner><Note> • sourcetype=aws:description source="*:ec2_instances" (NOT "tags.Name"=*) OR ("tags.Name"=* tags.Name!=Q* tags.Name!=D* tags.Name!=P* tags.Name!=U*) R
  45. 45. Just use the “Name” tag 4R
  46. 46. 48 Splunk app for AWS demo R
  47. 47. Splunk runs on and with AWS SOC2 Type II Certified Cloud Services Apps Splunk Add-on for AWS Splunk App for AWS Specific Integrations Config, CloudTrail, CloudWatch, VPC Flow Logs, Lambda: AWS IoT, Amazon Kinesis: AWS CloudFormation Splunk Core + Enterprise Security & ITSI available Enterprise on AWS For small IT teams, starts $3/day Software Apps and Integrations As a Service on AWS Delivery Models For small IT teams, starts $75/mo R
  48. 48. Launched: Splunk Light w/ app for AWS Multiple use cases across one platform Splunk Light AMI on AWS Marketplace Free 20GB License 6 Month Term = $6,000 Value Bundled with App for AWS Go To: & Search “Splunk Light” Demos available at AWS Re:Invent Booth #206
  49. 49. Thank you! 51 Contact: Twitter: @scottjpack Contact: Twitter: @drandallyoung
  50. 50. Remember to complete your evaluations!