Download as PDF, PPTX
Uploaded bySteamhaus
KELK Stack on AWS
Sean Clerkin, a senior site reliability engineer at Steamhaus, discusses the challenges of centralized logging on AWS and introduces a serverless logging solution using ELK stack components. The setup utilizes AWS services like Kinesis, Elasticsearch, and Lambda for logging and analysis and is automated with Terraform. The project is not production-ready and aims for readability, with emphasis on the importance of IAM and bucket policies.
KELK Stack on AWS
- 2. SteamhausKELK ON AWS Whoam I? Sean Clerkin Senior Site Reliability Engineer
- 3.
- 4. No centralised logging Userneeds OS knowledge Distribution Of keys Enemy of autoscaling Log rotation Users download logs unnecessarily Doesn’t scale To many servers Slow to find issues Alerting is hard Sshing to servers :(
- 5. SteamhausKELK ON AWS ELKis awesomE
- 6. SteamhausKELK ON AWS ELKon ec2
- 7. SteamhausKELK ON AWS KELKon AWS • Low maintenance - No ec2, Uses entirely AWS serverless technologies and services • ALB, Cloudfront and Cloudtrail logs are ingested as well as EC2 logs • Logs are archived in S3 for long term storage, and indexed in Elasticsearch for short term analytics • Automated with Terraform • Open source Kinesis: buffering and delivering instance logs Elasticsearch: Indexing and log storage Lambda: processing and delivering S3 logs Kibana: Search and analytics
- 8. SteamhausKELK ON AWS Howdoes it work?
- 9.
- 10.
- 11.
- 12.
- 13.
- 14.
- 15.
- 16.
- 17.
- 18.
- 19.
- 20.
- 21. SteamhausKELK ON AWS Automation code Sample WebStack VPC ALB EC2 Logging Stack Kinesis Elasticsearch Service Lambda S3 CloudfrontPython Terraform Do try this at home! github.com/steamhaus/kelk-example
- 22. SteamhausKELK ON AWS Calloutsfrom the build • It’s not production ready, built for readability • Nailing iam and bucket policies can take a while! • Testing lambda - create a test event in the UI • Use Terraform, rinse and repeat
- 23. SteamhausKELK ON AWS AnyQuestions..?
- 24. Thank you :) Contactus hello@steamhaus.co.uk 0161 820 2020 @steamhausmcr Locate us Fourways House 57 Hilton Street Manchester M1 2EJ