Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Creating a Serverless AMI Certification Pipeline


Published on

Presented at Seattle AWS Architects & Engineers meetup on January 31, 2017. Discussion on building a serverless CI/CD pipeline for AMIs.

Published in: Internet
  • Be the first to comment

Creating a Serverless AMI Certification Pipeline

  1. 1. Creating a Serverless AMI Certification Pipeline SEATTLE AWS ARCHITECTS & ENGINEERS MEETUP DAVID F. SEVERSKI - JANUARY 31, 2017
  2. 2. Background I’m in information security Most of my time is in the risk space Started trying to figure out this cloud thing 5 years ago AWS Full 5 Certified Drinker of the DevOps Kool-Aid 2
  3. 3. Genesis Jim Fink’s November AWS Inspector Talk 3
  4. 4. Lambda Already in Use AMI S3 Cleaner Controlling AWS ML Endpoints SNS-Slack Bridge Periodic RDS Extractions 4
  5. 5. Enter Step Functions and Lambda Environment Variables Announced at re:invent 2016 5 Step Functions and Environment Variables?!
  6. 6. Real World Architecture “Sometimes, it’s messy” 6 Image © Valve Software
  7. 7. Materials Will Be Provided GitHub, Slideshare, etc. 7
  8. 8. Problem Statement How can I ensure that my AMIs are built without known vulnerabilities? No persistent hosts Rapidly changing images (Packer built AMIs) Building a known good image verification system without statically running hosts (Jenkins, etc.) 8
  9. 9. Quick Lambda Refresher Functions deployed without servers (that you manage anyways) Maximum runtime of five minutes Auto-scaled by AWS Super cheap to run Choice of languages Using Python 2.6 and Boto for this example Can use IAM roles to access other AWS services Wide support for triggering via other AWS Services 9
  10. 10. Basic Concepts of Step Functions Collection of tasks ◦ Worker resources – can be on premise! ◦ Simple deciders Steps ◦ Pass State ◦ Task State ◦ Choice State ◦ Wait State ◦ Succeed State ◦ Fail State Multiple independent executions can run at once JSON-based language - 10
  11. 11. Services Involved (Partial) CloudTrail CloudWatch Lambda Inspector Step Functions EC2 11
  12. 12. Caution! Cutting, and sometimes bleeding, edge stuff here 12
  13. 13. Basic Flow Approve/Reject New Image Test the Image Launch Test Instance Detect New Image 13
  14. 14. Detect 1. Track API calls - CloudTrail setup on account 2. CloudWatch Log event filter subscription tracks any image being built ◦ RegisterImage (instance images) ◦ CreateImage (EBS images) ◦ CopyImage (copying public instances into this account) 3. CloudWatch triggers Lambda function to start Step Function ◦ Input: AMI ID 4. Check Image State 14 Detect Launch Test Approve
  15. 15. Launch 1. Launch Instance of candidate AMI ◦ Launch Instance ◦ Uses an EC2 Spot Instance 2. Install Inspector Agent ◦ UserData with simple “curl | bash” 3. Check Instance State 4. Tag Instance for Scanning ◦ Assigns a batch ID to identify the instance(s) to scan 15 Detect Launch Test Approve
  16. 16. Test 1. Start Inspector Run 2. Wait 15 Minutes 3. Poll Inspector Status 4. Try and Repeat with Timeout 16 Detect Launch Test Approve
  17. 17. Approve/Reject 1. Terminate Instance 2. Parse results ◦ Arbitrary criteria, are there any CVEs rated with a CVSS of 6 or above? ◦ If so, FAIL ◦ Else PASS 3. Tag AMI 17 Detect Launch Test Approve
  18. 18. What We’ve Built So Far 18
  20. 20. Enter Terraform 20
  21. 21. Terraform – AMI-Security-Validator Creates the IAM roles for Lambda functions ◦ Launch EC2 Instance ◦ Check Instance Ready ◦ Tag EC2 Resources ◦ Start Inspector Assessment Run ◦ Check Inspector Assessment Run Complete ◦ Parse Inspector Assessment Run Findings ◦ Terminate Instances ◦ Start Step Functions Create Lambda Functions from S3 Artifacts CloudTrail event trigger 21
  22. 22. Continuous Deployment with Travis 22 GitHub Commit Travis CI Build and Push to S3
  23. 23. Using Terraform Set up infrastructure VPC VPC flow logs (use the VPC module) Subnets in all three AZs Private subnets Internet gateways Security group which allows access from home IP 23
  24. 24. Future Directions •Present a user interface that provides only tagged AMIs for launch by users •Service Catalog? 24
  25. 25. Lessons Learned 25
  26. 26. Gripes and Lessons Learned – Step Functions Step Functions cannot be edited once created Eventual consistency is a very annoying problem “Programing” in JSON Triggering state machines…how? Result processing is limited Execution ID is not available to the state machine Clearly defining the APIs of your functions is sanity saving statelint tool is ruby based AWS Config not supported 26
  27. 27. More Gripes – Lambda and Inspector Lambda ◦ Python environment only recently received updates to work with Step Functions – No release notes! ◦ AWS Config not supported ◦ Python 3.5 & Tagging  Inspector ◦ API is clunky – best suited for static target list ◦ ResourceGroup entity required by TargetGroup, but no ability to describe, modify, or delete them after creation…are they gone when the TargetGroup is deleted? Don't know. ◦ Ubuntu 16.04 LTS support launched only recently (Jan 5) available ◦ Preview-agents API takes a preview-agents-arn, but that's really an assessment-target-arn ◦ Strange convergence issues between resource groups, targets, and runs 27
  28. 28. Would You Like Some More Gripes? Management ◦ No Terraform or CloudFormation support for Step Functions ◦ Terraform support coming soon! ◦ No CloudFormation support for AWS Inspector ◦ Terraform now supports AWS Inspector as of v0.8.5 IAM Roles ◦ Pathing strategy for IAM roles is really helpful ◦ Placing roles under a /service-roles path makes filtering and sorting a dream 28
  29. 29. Thanks! David F. Severski @DSeverski 29
  30. 30. References Lambda Code Image Scanner Step Function Definition & Terraform Setup Terraform Support for Step Functions AWS Step Language Documentation Packer Terraform 30