Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved
AWS Logging, Analysis and Alerting
Brian Wagner
So...
My Application
Why monitor?
What are we looking for?
Billing
API activity
Changes to resources
Application activity
Network activity
Detailed Billing
Billing Information logged Daily in S3
Also Visible in the Billing Console
Alarms can be set on Billing I...
Sample Records
ItemDescription
UsageSta
rtDate
UsageE
ndDate
UsageQua
ntity
Curren
cyCode
CostBef
oreTax
Cre
dits
TaxA
mou...
AWS CloudTrail
CloudTrail can help you achieve many
tasks
Security analysis
Track changes to AWS resources, for
example VP...
AWS CloudTrail logs can be delivered cross-account
CloudTrail can help achieve many tasks
Accounts can send their trails t...
AWS Config
AWS Config is a fully managed service that provides you
with an inventory of your AWS resources, lets you audit...
©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved
Continuous ChangeRecordingChanging 

Resources
AWS...
Am I safe?
Properly configured resources
are critical to security
AWS Config enables you to
continuously monitor the
confi...
Where is the evidence?
Many compliance audits
require access to the state of
your systems at arbitrary times
(i.e. PCI, HI...
Resource
A resource is an AWS object
you can create, update or
delete on AWS
Examples include Amazon
EC2 instances, Securi...
Resources
Resource Type Resource
Amazon EC2 EC2 Instance
EC2 Elastic IP (VPC only)
EC2 Security Group
EC2 Network Interfac...
Relationships

• Bi-directional map of
dependencies
automatically assigned
• Change to a resource
propagates to create
Con...
Relationships

Resource Relationship Related Resource
CustomerGateway is attached to VPN Connection
Elastic IP (EIP) is at...
Configuration Item
All AWS API configuration attributes for a given resource
at a given point in time, captured on every c...
Component Description Contains
Metadata Information about this configuration
item
Version ID, Configuration item ID,
Time ...
Essentially, “Lambda Integration for Config”
Apply detailed checks to the state of your configuration, at the point
when i...
Full visibility of your AWS environment
CloudTrail will record access to API calls and save logs in your S3
buckets, no ma...
Managing, Monitoring & Processing Logs
CloudWatch Logs Features
‣ Near real-time, aggregate, monitor, store, and search
Am...
Firewall Requirements
Based on NIST SP-800, PCI-DSS and others
‣ Anti-Spoofing
‣ Packet-Filtering (minimum) stateful/state...
VPC Flow Logs
CloudWatch

Logs
LogGroup
ENI-LogStream
ENI-LogStream
ENI-LogStream
ENI-LogStream
ENI-LogStream
ENI-LogStrea...
VPC Flow Logs in Context
route restrictively
lock down on network level
isolate concerns
lock down on instance level
Flows
Flow Log Record Structure
Event-Version
Account Number
ENI-ID
Source-IP
Destination-IP
SourcePort
Destination-Port
Protoco...
Flow Log Sampling
Flow Logs are statistical reports of activity over a window of time
Start-Time Window End-Time Window
Nu...
Statistical Sampling and Spikes
Time
Src/Dst IP/Port Tuple
?
Example
Logs→metrics→alerts→actions
AWS Config
CloudWatch /
CloudWatch Logs
CloudWatch
alarms
AWS CloudTrail
Amazon EC2 OS logs
Am...
©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved
Brian Wagner
Solutions Architect
AWS Germany
Thank...
Upcoming SlideShare
Loading in …5
×

Monitoring and Alerting

4,789 views

Published on

Monitoring and Alerting

Published in: Business

Monitoring and Alerting

  1. 1. ©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved AWS Logging, Analysis and Alerting Brian Wagner Solutions Architect AWS Germany
  2. 2. My Application Why monitor?
  3. 3. What are we looking for? Billing API activity Changes to resources Application activity Network activity
  4. 4. Detailed Billing Billing Information logged Daily in S3 Also Visible in the Billing Console Alarms can be set on Billing Info to Alert on Unexpected Activity
  5. 5. Sample Records ItemDescription UsageSta rtDate UsageE ndDate UsageQua ntity Curren cyCode CostBef oreTax Cre dits TaxA mount Tax Typ e TotalC ost $0.000 per GB - regional data transfer under the monthly global free tier 01.04.14 00:00 30.04.14 23:59 0.0000067 5 USD 0.00 0.0 0.000 000 Non e 0.0000 00 $0.05 per GB-month of provisioned storage - US West (Oregon) 01.04.14 00:00 30.04.14 23:59 1.126.666. 554 USD 0.56 0.0 0.000 000 Non e 0.5600 00 First 1,000,000 Amazon SNS API Requests per month are free 01.04.14 00:00 30.04.14 23:59 10.0 USD 0.00 0.0 0.000 000 Non e 0.0000 00 First 1,000,000 Amazon SQS Requests per month are free 01.04.14 00:00 30.04.14 23:59 4153.0 USD 0.00 0.0 0.000 000 Non e 0.0000 00 $0.00 per GB - EU (Ireland) data transfer from US West (Northern California) 01.04.14 00:00 30.04.14 23:59 0.0000329 2 USD 0.00 0.0 0.000 000 Non e 0.0000 00 $0.000 per GB - data transfer out under the monthly global free tier 01.04.14 00:00 30.04.14 23:59 0.0231101 9 USD 0.00 0.0 0.000 000 Non e 0.0000 00 First 1,000,000 Amazon SNS API Requests per month are free 01.04.14 00:00 30.04.14 23:59 88.0 USD 0.00 0.0 0.000 000 Non e 0.0000 00 $0.000 per GB - data transfer out under the monthly global free tier 01.04.14 00:00 30.04.14 23:59 3.3E-7 USD 0.00 0.0 0.000 000 Non e 0.0000 00
  6. 6. AWS CloudTrail CloudTrail can help you achieve many tasks Security analysis Track changes to AWS resources, for example VPC security groups and NACLs Compliance – log and understand AWS API call history Prove that you did not: Use the wrong region Use services you don’t want Troubleshoot operational issues – quickly identify the most recent changes to your environment
  7. 7. AWS CloudTrail logs can be delivered cross-account CloudTrail can help achieve many tasks Accounts can send their trails to a central account Central account can then do analytics Central account can: ‣ Redistribute the trails ‣ Grant access to the trails ‣ Filter and reformat Trails (to meet privacy requirements)
  8. 8. AWS Config AWS Config is a fully managed service that provides you with an inventory of your AWS resources, lets you audit the resource configuration history and notifies you of resource configuration changes.
  9. 9. ©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved Continuous ChangeRecordingChanging Resources AWS Config History Stream Snapshot (ex. 2014-11-05) AWS Config
  10. 10. Am I safe? Properly configured resources are critical to security AWS Config enables you to continuously monitor the configurations of your resources at AWS API level, and evaluate these configurations for potential security weaknesses
  11. 11. Where is the evidence? Many compliance audits require access to the state of your systems at arbitrary times (i.e. PCI, HIPAA) A complete inventory of all resources and their configuration attributes at AWS API level is available for any point in time
  12. 12. Resource A resource is an AWS object you can create, update or delete on AWS Examples include Amazon EC2 instances, Security Groups, Network ACLs, VPCs and subnets Amazon EC2 Instance, ENI... Amazon EBS Volumes AWS CloudTrail Log Amazon VPC VPC, Subnet...
  13. 13. Resources Resource Type Resource Amazon EC2 EC2 Instance EC2 Elastic IP (VPC only) EC2 Security Group EC2 Network Interface Amazon EBS EBS Volume Amazon VPC VPCs Network ACLs Route Table Subnet VPN Connection Internet Gateway Customer Gateway VPN Gateway AWS CloudTrail Trail
  14. 14. Relationships
 • Bi-directional map of dependencies automatically assigned • Change to a resource propagates to create Configuration Items for related resources Example: Security Group sg-10dk8ej and EC2 instance i-123a3d9 are “associated with” each other
  15. 15. Relationships
 Resource Relationship Related Resource CustomerGateway is attached to VPN Connection Elastic IP (EIP) is attached to Network Interface is attached to Instance Instance contains Network Interface is attached to ElasticIP (EIP) is contained in Route Table is associated with Security Group is contained in Subnet is attached to Volume is contained in Virtual Private Cloud (VPC) InternetGateway is attached to Virtual Private Cloud (VPC) … …. …..
  16. 16. Configuration Item All AWS API configuration attributes for a given resource at a given point in time, captured on every configuration change.
  17. 17. Component Description Contains Metadata Information about this configuration item Version ID, Configuration item ID, Time when the configuration item was captured, State ID indicating the ordering of the configuration items of a resource, MD5Hash, etc. Common Attributes Resource attributes Resource ID, tags, Resource type. Amazon Resource Name (ARN) Availability Zone, etc. Relationships How the resource is related to other resources associated with the account EBS volume vol-1234567 is attached to an EC2 instance i- a1b2c3d4 Current Configuration Information returned through a call to the Describe or List API of the resource e.g. for EBS Volume State of DeleteOnTermination flag Type of volume. For example, gp2, io1, or standard Related Events The AWS CloudTrail events that are related to the current configuration of the resource AWS CloudTrail event ID Configuration Item
  18. 18. Essentially, “Lambda Integration for Config” Apply detailed checks to the state of your configuration, at the point when it changes Raise alerts if anything is outside compliance with your defined policy ‣ Eg if there’s unencrypted non-root EBS volumes ‣ …or eg if any taggable resources aren’t tagged appropriately We have a library of pre-built rules – or build your own See also Re:Invent (SEC308) “Wrangling Security Events in the Cloud” (https://www.youtube.com/watch?v=uc1Q0XCcCv4) Feature is available right now Introducing Config Rules
  19. 19. Full visibility of your AWS environment CloudTrail will record access to API calls and save logs in your S3 buckets, no matter how those API calls were made Who did what and when and from where (IP address) CloudTrail support for many AWS services and growing - includes EC2, EBS, VPC, RDS, IAM and RedShift Easily Aggregate all instance log information – CloudWatch Logs agent scrapes files from EC2 instances and sends them to S3 Also enables alerting with SNS on “strings of interest”, just like regular CloudWatch CloudWatch Logs used as delivery mechanism for Flow Logging Out of the box integration with log analysis tools from AWS partners including Splunk, AlertLogic and SumoLogic Monitoring: Get consistent visibility of logs
  20. 20. Managing, Monitoring & Processing Logs CloudWatch Logs Features ‣ Near real-time, aggregate, monitor, store, and search Amazon Elasticsearch Service Integration ‣ Analytics and Kibana interface AWS Lambda & Amazon Kinesis Integration ‣ Custom processing with your code Export to S3 ‣ SDK & CLI batch export of logs
  21. 21. Firewall Requirements Based on NIST SP-800, PCI-DSS and others ‣ Anti-Spoofing ‣ Packet-Filtering (minimum) stateful/stateless ‣ Segregation of Duties at the management side ‣ Logging/Audit capabilities on the management side ‣ Event-Logging on processed traffic Security Group IAM AWS Config CloudTrail FlowLogs
  22. 22. VPC Flow Logs CloudWatch Logs LogGroup ENI-LogStream ENI-LogStream ENI-LogStream ENI-LogStream ENI-LogStream ENI-LogStream ENI-LogStream
  23. 23. VPC Flow Logs in Context route restrictively lock down on network level isolate concerns lock down on instance level Flows
  24. 24. Flow Log Record Structure Event-Version Account Number ENI-ID Source-IP Destination-IP SourcePort Destination-Port Protocol Number Number of Packets Number of Bytes Start-Time Window End-Time Window Action State 2 123456789 eni-31607853 172.16.0.10 172.16.0.172 80 41707 6 1 40 1440402534 1440402589 ACCEPT OK
  25. 25. Flow Log Sampling Flow Logs are statistical reports of activity over a window of time Start-Time Window End-Time Window Number of Packets Number of Bytes Action
  26. 26. Statistical Sampling and Spikes Time Src/Dst IP/Port Tuple ?
  27. 27. Example
  28. 28. Logs→metrics→alerts→actions AWS Config CloudWatch / CloudWatch Logs CloudWatch alarms AWS CloudTrail Amazon EC2 OS logs Amazon VPC Flow Logs Amazon SNS email notification HTTP/S notification SMS notifications Mobile push notifications API calls from most services Monitoring data from AWS services Custom metrics
  29. 29. ©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved Brian Wagner Solutions Architect AWS Germany Thank You

×