Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

KELK Stack on AWS

265 views

Published on

The KELK Stack - Kinesis, Elasticsearch, Lambda, Kibana. An example of a low maintenance logging pattern using AWS services.

Published in: Internet
  • Be the first to comment

  • Be the first to like this

KELK Stack on AWS

  1. 1. SteamhausKELK ON AWS Who am I? Sean Clerkin Senior Site Reliability Engineer
  2. 2. Logging is difficult
  3. 3. No centralised logging User needs OS knowledge Distribution Of keys Enemy of autoscaling Log rotation Users download logs unnecessarily Doesn’t scale To many servers Slow to find issues Alerting
 is hard Sshing to servers :(
  4. 4. SteamhausKELK ON AWS ELK is awesomE
  5. 5. SteamhausKELK ON AWS ELK on ec2
  6. 6. SteamhausKELK ON AWS KELK on AWS • Low maintenance - No ec2, Uses entirely AWS serverless technologies and services • ALB, Cloudfront and Cloudtrail logs are ingested as well as EC2 logs • Logs are archived in S3 for long term storage, and indexed in Elasticsearch for short term analytics • Automated with Terraform • Open source Kinesis: buffering and delivering instance logs Elasticsearch: Indexing and log storage Lambda: processing and delivering S3 logs Kibana: Search and analytics
  7. 7. SteamhausKELK ON AWS How does it work?
  8. 8. SteamhausKELK ON AWS
  9. 9. SteamhausKELK ON AWS
  10. 10. SteamhausKELK ON AWS
  11. 11. SteamhausKELK ON AWS
  12. 12. SteamhausKELK ON AWS
  13. 13. SteamhausKELK ON AWS
  14. 14. SteamhausKELK ON AWS
  15. 15. SteamhausKELK ON AWS
  16. 16. SteamhausKELK ON AWS
  17. 17. SteamhausKELK ON AWS
  18. 18. SteamhausKELK ON AWS
  19. 19. SteamhausKELK ON AWS
  20. 20. SteamhausKELK ON AWS Automation
 code Sample Web Stack VPC ALB EC2 Logging
 Stack Kinesis Elasticsearch
 Service Lambda S3 CloudfrontPython Terraform Do try this at home! github.com/steamhaus/kelk-example
  21. 21. SteamhausKELK ON AWS Callouts from the build • It’s not production ready, built for readability • Nailing iam and bucket policies can take a while! • Testing lambda - create a test event in the UI • Use Terraform, rinse and repeat
  22. 22. SteamhausKELK ON AWS Any Questions..?
  23. 23. Thank you :) Contact us
 
 hello@steamhaus.co.uk
 0161 820 2020
 @steamhausmcr Locate us Fourways House
 57 Hilton Street 
 Manchester M1 2EJ

×