This document provides an overview of monitoring Azure and AWS cloud environments. It discusses why monitoring is important for threat detection, hunting and response. It outlines what aspects should be monitored, including operating systems, applications, network traffic, and cloud service logs. Specific AWS and Azure monitoring options are described, such as CloudTrail, VPC Flow Logs, and Azure Audit Logs. Integrating cloud logs with SIEMs and threat intelligence feeds is also covered. Endpoint monitoring tools are suggested to record process, file, registry and network activity on virtual machines.

1. Join the conversation #devseccon
By Alfredo Reino
Monitoring AWS and
Azure

2. Agenda
• Who am I
• Why monitor anything
• What to monitor
• How to monitor Azure
• How to monitor AWS
• Integrating with SIEMs and MSSPs

3. Who am I
[insert something funny and slightly self-deprecating here]
Alfredo Reino / @areino
alfredo@aebura.co.uk
Security Architect

4. Why monitor anything
• Threat detection (reactive)
• Threat hunting (proactive)
• Incident response and forensics
• Data mining, anomaly detection
• Reporting and dashboards
• Regulatory and policy requirements
• Troubleshooting and root cause analysis

5. What to monitor
• (Easy answer) Everything and anything!
• But
• is it possible to log?
• is it cost effective to do it?
• do you have the storage?
• can you make sense of it?
• do you have the tools/skillset/capability/time to use it?
• Priorities!

6. What to monitor
• Using the kill-chain

7. Generic kill chain
• Recon
• Passive recon
• Active recon
• Delivery
• Internet-facing services
• Inbound email
• Web browsing
• Removable media
• Insider / Third-party access
• Exploitation
• Internet-facing servers
• User endpoint
• Installation
• Lateral movement
• Elevation of privilege
• Persistence
• Command and Control
• Actions on target
• Access to internal system
and data
• Exfiltration
• Attack third-party

8. IaaS/PaaS kill chain
• Recon
• Passive recon
• Active recon
• Delivery
• Internet-facing services
• Inbound email
• Web browsing
• Removable media
• Insider / Third-party access
• Exploitation
• Internet-facing servers
• User endpoint
• Installation
• Lateral movement
• Elevation of privilege
• Persistence
• Command and Control
• Actions on target
• Access to internal system
and data
• Exfiltration
• Attack third-party

9. What to monitor – Shared responsibility
Azure Shared responsibility model
https://aka.ms/sharedresponsibility
AWS Shared responsibility model
https://aws.amazon.com/compliance/shared-responsibility-model/

10. What to monitor (IaaS/PaaS)
• Operating System logs from IaaS virtual machines
• Application/service logs (webserver, database, etc.)
• Performance metrics (CPU, memory, data in/out, filesystem, …)
• Network traffic (at interface or across boundaries)
• Other cloud security solutions (WAF, AV, FIM, etc.)
• IaaS/PaaS service fabric logs
• Audit/management logs (cloud resource access and management)
• Blob Storage/S3

11. AWS Options
• CloudTrail
• Records AWS API calls (usage of Management Console, SDKs, command line tools, and higher-level AWS services such
as AWS CloudFormation).
• The recorded information includes the identity of the API caller, the time of the API call, the source IP address of the API
caller, the request parameters, and the response elements returned by the AWS service.
• Logs to S3 bucket (possibility of aggregating multiple region or multiple account CloudTrail logs in one S3 bucket)
• CloudWatch
• Collects and tracks metrics, collects and monitors log files, sets alarms.
• Monitor EC2 instances, WAF, DynamoDB tables, and Amazon RDS DB instances, as well as custom metrics generated by
applications and services, and any log files applications generate.
• S3 Server Access Logging
• Track requests for access to S3 bucket.
• Each access log record provides details about a single access request, such as the requester, bucket name, request time,
request action, response status, and error code, if any.
• VPC Flow Logs
• Log traffic flow in Virtual Private Cloud (VPC), subnets or Elastic Network Interfaces (ENI).
• Captures accepted and rejected traffic.
• Logs to CloudWatch.

12. AWS CloudTrail

13. AWS VPC Flow Logs

14. Log Management solutions for AWS
• SumoLogic
• logs from CloudTrail, VPC Flow, ELB, S3, etc
• Splunk add-on for AWS
• logs from AWS Config, Config Rules, CloudWatch,
CloudTrail, Billing, S3, VPC Flow Log, Amazon
Inspector, Metadata inputs, etc
• ELK (ElasticSearch+LogStash+Kibana)
• logs from applications, OS, ELB, CloudTrail, VPC Flow,
CloudFront, S3, etc

15. Log Management solutions for Azure
• SumoLogic
• logs from Azure Audit Logs, AD access, etc.
• Splunk add-on for Microsoft Cloud
• logs from Storage Tables, Storage Blobs, Azure
Service Management APIs and Office 365
Management API.
• ELK (ElasticSearch+LogStash+Kibana)
• logs from applications, OS, Storage Blobs, Service
Management APIs, etc.
• Azure Log Analytics
• Part of OMS (Operations Management Suite).
• Collect logs from agents (Win/Linux), storage,
performance, IIS logs, syslog, etc.

16. Connecting AWS logs to a SIEM
• Connectors by SIEM vendors
• HP ArcSight SmartConnector
• Need to allow inbound SSL to ESM 

17. Connecting AWS logs to a SIEM
• Connectors by SIEM vendors
• IBM Qradar
• Native support for AWS CloudTrail
using S2 REST API
• Need to import the SSL cert first

18. Connecting AWS logs to a SIEM
• Connectors by SIEM vendors
• Splunk
• Requires “Splunk for AWS” app and
“Splunk Add-on for Amazon Web
Services”.
• Requires appropriate permissions
created on IAM.
• Collects events from Simple Queue
Service (SQS) that subscribes to a
Simple Notification Service (SNS)
events from AWS Config.

19. Connecting AWS logs to a SIEM
• Connectors by SIEM vendors
• ELK Stack (logz.io)

20. Azure SIEM Integrator
• Integrate with on-premises SIEM (or MSSP)
• Logs supported
• VM logs
• Azure Audit Logs
• Azure Security Center alerts

21. Azure SIEM Integrator
• How to deploy
• Install Azlog Integrator on Windows server (on-premises)
• https://www.microsoft.com/en-us/download/details.aspx?id=53324
• Needs access to Azure Storage
• Install SIEM log collection agent on same server
• Splunk Universal Forwarder
• HP ArcSight Windows Event Collector
• IBM QRadar WinCollect
• …
• Configure SIEM agents for collection
• https://blogs.msdn.microsoft.com/azuresecurity/2016/08/23/azure-log-siem-configuration-steps/
• Scalability
• On a 8 proc machine – 1 instance of Azlog can process about 277 EPS
• On a 4 proc machine – 1 instance of Azlog can process about 17 EPS
• Multiple instances of the SIEM Integrators can be run if event volume is high

22. Azure SIEM Integrator example

23. Azure SIEM Integrator example

24. Azure SIEM Integrator example

25. Threat intel feeds
• Good feeds of IOCs can be invaluable
• Integrate threat intel feeds in SIEM/Log Management solutions
• tagging of events
• quick searches for malicious activity
• For increased value, maintain your OWN threat intel feed and
repository

26. Endpoint activity monitoring
• Endpoint process activity monitoring tool
• such as Carbon Black
• Deploy to IaaS instances
• Agent-based blackbox-type recording for
• process activity (creation, termination,
child processes)
• filesystem and registry activity
• inbound and outbound network
connections
• Integration with threat intel feeds
• Can integrate (using API) with log
retention solutions
• “if log event X then find process tree at the
time for endpoint Y”

27. Join the conversation #devseccon
Thanks!