(DEV301) Automating AWS with the AWS CLI

15,279 views

Published on

In this talk, you’ll learn how you can use the AWS CLI to automate common administrative tasks in AWS.  We’ll cover several scenarios including EBS snapshot management and S3 backups and see how to combine AWS CLI features to create powerful tools for automation.  You’ll see how to develop, debug, and deploy these examples in several live, end to end examples.

Published in: Technology

(DEV301) Automating AWS with the AWS CLI

  1. 1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. James Saryerwinnie October 2015 Automating AWS with the AWS CLI DEV301
  2. 2. AWS Command Line Interface Unified tool to manage your AWS services
  3. 3. $ git show 1.0.0 tag 1.0.0 Tagger: James Saryerwinnie <js@jamesls.com> Date: Mon Sep 2 18:38:51 2013 -0700 Tagging 1.0.0 release. commit 7fbdac0b19ea9dcc0380242068af20ef425ac5c3 Merge: 6f76721 469bab6 Author: James Saryerwinnie <js@jamesls.com> Date: Mon Sep 2 18:38:51 2013 -0700 Merge branch 'release-1.0.0' * release-1.0.0: Bumping version to 1.0.0
  4. 4. Interactive
  5. 5. Shell scripts#!/bin/bash import_key_pair() { echo -n "Would you like to import ~/.ssh/id_rsa.pub? [y/N]: " read confirmation if [[ "$confirmation" != "y" ]] then return fi aws ec2 import-key-pair --key-name id_rsa --public-key-material file://~/.ssh/id_rsa.pub } create_instance_profile() { echo -n "Would you like to create an IAM instance profile? [y/N]: " read confirmation if [[ "$confirmation" != "y" ]]; then return fi aws iam create-role --role-name dev-ec2-instance --assume-role-policy-document "$TRUST_POLICY" || errexit "Could not create Role" # Use a managed policy policies=$(aws iam list-policies --scope AWS) admin_policy_arn=$(jp -u "Policies[?PolicyName=='AdministratorAccess'].Arn | [0]" <<< "$policies") aws iam attach-role-policy --role-name dev-ec2-instance --policy-arn "$admin_policy_arn" || errexit "Could not attach role policy" # Then we need to create an instance profile from the role. aws iam create-instance-profile --instance-profile-name dev-ec2-instance || errexit "Could not create instance profile." # And add it to the role aws iam add-role-to-instance-profile --role-name dev-ec2-instance --instance-profile-name dev-ec2-instance || errexit "Could not add role to instance profile." } compute_key_fingerprint() { # Computes the fingerprint of a public SSH key given a private # RSA key. This can be used to compare against the output given # from aws ec2 describe-key-pair. openssl pkey -in ~/.ssh/id_rsa -pubout -outform DER | openssl md5 -c | cut -d = -f 2 | tr -d '[:space:]' } do_setup() { echo "Checking for required resources..." echo "" # 1. Check if a security group is found for # both windows and non-windows tags. # If not, we'll eventually give the option to # configure this. if resource_exists "aws ec2 describe-security-groups --filter Name=tag:dev-ec2-instance,Values=non-windows"; then echo "Security groups exists." else echo "Security group not found." fi echo "" # 2. Make sure the keypair is imported. if [[ ! -f ~/.ssh/id_rsa ]]; then echo "Missing ~/.ssh/id_rsa key pair." elif has_new_enough_openssl; then fingerprint=$(compute_key_fingerprint ~/.ssh/id_rsa) if resource_exists "aws ec2 describe-key-pairs --filter Name=fingerprint,Values=$fingerprint"; then echo "Key pair exists." else echo "~/.ssh/id_rsa key pair does not appear to be imported." import_key_pair fi else echo "Can't check if SSH key has been imported." echo "You need at least openssl 1.0.0 that has a "pkey" command." echo "Please upgrade your version of openssl." fi echo "" # 3. Check that they have an IAM role called dev-ec2-instance. # There is no server side filter for this like we have with the EC2 # APIs, so we have to manually use a --query option for us. if resource_exists "aws iam list-instance-profiles" "length(InstanceProfiles[?InstanceProfileName=='dev-ec2-instance'])"; then echo "Instance profile exists." else echo "Missing IAM instance profile 'dev-ec2-instance'" create_instance_profile fi echo "" } do_setup
  6. 6. aws s3 ls Application with an AWS SDK
  7. 7. aws s3 ls Application with an AWS SDK Shell Scripts – what we’ll cover
  8. 8. A good shell script • < 100 SLOC • Commands in sequence • Piping input/output • Simple domain logic
  9. 9. • Familiar with AWS CLI • Familiar with bash • Familiar with AWS Prerequisites
  10. 10. $ aws ec2 describe-instances --filters Name=architecture,Values=x86_64 $ du $(ls | grep cli) | sort -nr | tail -2 Amazon S3Amazon EC2 Prerequisites
  11. 11. Getting up to speed • Becoming a Command Line Expert with the AWS CLI • Advanced Usage of the AWS CLI • Installation • Configuration
  12. 12. $ We will be in the terminal…
  13. 13. $ We will be in the terminal…a lot
  14. 14. Common patterns
  15. 15. Main patterns • Single output to a single command • Map list output to N AWS CLI commands • Storing JSON documents and querying later • Resource exists check
  16. 16. Main patterns • Single output to a single command • Map list output to N AWS CLI commands • Storing JSON documents and querying later • Resource exists check
  17. 17. Output to a single parameter
  18. 18. Output to a single parameter $ aws ec2 describe… { "Reservations": [ { "Groups": [ { "GroupId": "sg-abc", "GroupName": "Windows" } ], "Instances": [ { "AmiLaunchIndex": 0, "Architecture": "x86_64", "BlockDeviceMappings": [ { "DeviceName": "/dev/sda1", "Ebs": { "AttachTime": "2015-01-29T21:22:17.000Z", "DeleteOnTermination": true, "Status": "attached", $ aws ec2 run… --foo <here>
  19. 19. Output to a single parameter aws ec2 create-tags --tags Key=purpose,Value=dev --resources $(aws ec2 run-instances --image-id ami-12345 --query Instances[].InstanceId --output text)
  20. 20. Output to a single parameter aws ec2 create-tags --tags Key=purpose,Value=dev --resources $(aws ec2 run-instances --image-id ami-12345 --query Instances[].InstanceId --output text) Error handling
  21. 21. Output to a single parameter instance_ids=$(aws ec2 run-instances --image-id ami-12345 --query Instances[].InstanceId --output text) || errexit "Could not run instance" aws ec2 create-tags --tags Key=purpose,Value=dev --resources "$(instance_ids)" || errexit "<errmsg>"
  22. 22. Output to a single parameter instance_ids=$(aws ec2 run-instances --image-id ami-12345 --query Instances[].InstanceId --output text) || errexit "Could not run instance" aws ec2 create-tags --tags Key=purpose,Value=dev --resources "$(instance_ids)" || errexit "<errmsg>"
  23. 23. Output to a single parameter instance_ids=$(aws ec2 run-instances --image-id ami-12345 --query Instances[].InstanceId --output text) || errexit "Could not run instance" aws ec2 create-tags --tags Key=purpose,Value=dev --resources "$(instance_ids)" || errexit "<errmsg>"
  24. 24. Output to a single parameter • Used to map one part of the output to one part of the input • Use --query and --output text • Make sure to handle the error case before using the value
  25. 25. Main Patterns • Single output to a single command • Map list output to N AWS CLI commands • Storing JSON documents and querying later • Resource exists check
  26. 26. Mapping one output to N AWS calls
  27. 27. Mapping one output to N AWS calls $ aws iam list… [ "a", "b", ”c", "d", "e" ] $ aws iam … --foo <here> $ aws iam … --foo <here> $ aws iam … --foo <here> $ aws iam … --foo <here> $ aws iam … --foo <here>
  28. 28. Mapping one output to N AWS calls for name in $(aws iam list-users --query "Users[].[UserName]" --output text); do aws iam delete-user --user-name "$name" done
  29. 29. Mapping one output to N AWS calls for name in $(aws iam list-users --query "Users[].[UserName]" --output text); do aws iam delete-user --user-name "$name" done User1 User2 User3 User4 User5
  30. 30. Mapping one output to N AWS calls for name in $(aws iam list-users --query "Users[].[UserName]" --output text); do aws iam delete-user --user-name "$name" done
  31. 31. Mapping one output to N AWS calls aws iam list-users --query "Users[].[UserName]" --output text | xargs -I {} -P 10 aws iam delete-user --user-name "{}"
  32. 32. Mapping one output to N AWS calls aws iam list-users --query "Users[].[UserName]" --output text | xargs -I {} -P 10 aws iam delete-user --user-name "{}"
  33. 33. Mapping one output to N AWS calls aws iam list-users --query "Users[].[UserName]" --output text | xargs -I {} -P 10 aws iam delete-user --user-name "{}" Parallel
  34. 34. Mapping one output to N AWS calls -+= 72730 james -bash -+- 76343 james xargs -I {} –P 9 aws iam delete-user --user-name {} |--- 76348 james aws iam delete-user --user-name user1 |--- 76349 james aws iam delete-user --user-name user2 |--- 76350 james aws iam delete-user --user-name user3 |--- 76351 james aws iam delete-user --user-name user4 |--- 76352 james aws iam delete-user --user-name user5 |--- 76353 james aws iam delete-user --user-name user6 |--- 76354 james aws iam delete-user --user-name user7 |--- 76355 james aws iam delete-user --user-name user8 --- 76356 james aws iam delete-user --user-name user9
  35. 35. Mapping one output to N AWS calls • Use a for loop for functions • If you’re using xargs, use -I {} • Use xargs –P N parallel execution • Use “.[UserName]” instead of “.UserName” to get newline separated output.. • This works because nothing is written to stdout on error
  36. 36. Demo
  37. 37. Main patterns  Single output to a single command  Map list output to N AWS CLI commands • Storing JSON documents and querying later • Resource exists check
  38. 38. Saving the entire output for querying later { "AmiLaunchIndex": 0, "Architecture": "x86_64", "BlockDeviceMappings": [ { "DeviceName": "/dev/sda1", "Ebs": { "AttachTime": "2015-01-29T21:22:17.000Z", "DeleteOnTermination": true, "Status": "attached", "VolumeId": "vol-12345" } }, { "DeviceName": "xvdg", "Ebs": { "AttachTime": "2015-09-17T19:14:27.000Z", "DeleteOnTermination": false, "Status": "attached", "VolumeId": "vol-12345" } } ], "ClientToken": "12345", "EbsOptimized": false, "Hypervisor": "xen", "IamInstanceProfile": { "Arn": "arn:aws:iam::12345", "Id": "12345" }, "ImageId": "ami-12345", "InstanceId": "i-12345", "InstanceType": "m3.large", "KeyName": "id_rsa", "LaunchTime": "2015-09-17T17:24:13.000Z", "Monitoring": { "State": "disabled" }, } instance =
  39. 39. Saving the entire output for querying later { "AmiLaunchIndex": 0, "Architecture": "x86_64", "BlockDeviceMappings": [ { "DeviceName": "/dev/sda1", "Ebs": { "AttachTime": "2015-01-29T21:22:17.000Z", "DeleteOnTermination": true, "Status": "attached", "VolumeId": "vol-12345" } }, { "DeviceName": "xvdg", "Ebs": { "AttachTime": "2015-09-17T19:14:27.000Z", "DeleteOnTermination": false, "Status": "attached", "VolumeId": "vol-12345" } } ], "ClientToken": "12345", "EbsOptimized": false, "Hypervisor": "xen", "IamInstanceProfile": { "Arn": "arn:aws:iam::12345", "Id": "12345" }, "ImageId": "ami-12345", "InstanceId": "i-12345", "InstanceType": "m3.large", "KeyName": "id_rsa", "LaunchTime": "2015-09-17T17:24:13.000Z", "Monitoring": { "State": "disabled" }, } instance = query $instance BlockDeviceMappings[0].VolumeId
  40. 40. Saving the entire output for querying later { "AmiLaunchIndex": 0, "Architecture": "x86_64", "BlockDeviceMappings": [ { "DeviceName": "/dev/sda1", "Ebs": { "AttachTime": "2015-01-29T21:22:17.000Z", "DeleteOnTermination": true, "Status": "attached", "VolumeId": "vol-12345" } }, { "DeviceName": "xvdg", "Ebs": { "AttachTime": "2015-09-17T19:14:27.000Z", "DeleteOnTermination": false, "Status": "attached", "VolumeId": "vol-12345" } } ], "ClientToken": "12345", "EbsOptimized": false, "Hypervisor": "xen", "IamInstanceProfile": { "Arn": "arn:aws:iam::12345", "Id": "12345" }, "ImageId": "ami-12345", "InstanceId": "i-12345", "InstanceType": "m3.large", "KeyName": "id_rsa", "LaunchTime": "2015-09-17T17:24:13.000Z", "Monitoring": { "State": "disabled" }, } instance = query $instance ImageId query $instance BlockDeviceMappings[0].VolumeId
  41. 41. Saving the entire output for querying later { "AmiLaunchIndex": 0, "Architecture": "x86_64", "BlockDeviceMappings": [ { "DeviceName": "/dev/sda1", "Ebs": { "AttachTime": "2015-01-29T21:22:17.000Z", "DeleteOnTermination": true, "Status": "attached", "VolumeId": "vol-12345" } }, { "DeviceName": "xvdg", "Ebs": { "AttachTime": "2015-09-17T19:14:27.000Z", "DeleteOnTermination": false, "Status": "attached", "VolumeId": "vol-12345" } } ], "ClientToken": "12345", "EbsOptimized": false, "Hypervisor": "xen", "IamInstanceProfile": { "Arn": "arn:aws:iam::12345", "Id": "12345" }, "ImageId": "ami-12345", "InstanceId": "i-12345", "InstanceType": "m3.large", "KeyName": "id_rsa", "LaunchTime": "2015-09-17T17:24:13.000Z", "Monitoring": { "State": "disabled" }, } instance = query $instance InstanceId query $instance ImageId query $instance BlockDeviceMappings[0].VolumeId
  42. 42. jp – The JMESPath CLI • Same language as --query • Cross platform executable • https://github.com/jmespath/jp
  43. 43. jp – The JMESPath CLI • Same language as --query • Cross platform executable • https://github.com/jmespath/jp Mac OS X brew tap jmespath/jmespath brew install jp
  44. 44. Saving the entire output for querying later instance=$(aws run-instance --image-id ami-1234 --query Instances[0]) query() { jp -u "$2" <<<"$1" } instance_id=$(query "$instance" InstanceId) state_name=$(query "$instance" State.Name) instance_name=$(query "$instance" "Tags[?Key=='Name'].Value | [0]")
  45. 45. Saving the entire output for querying later instance=$(aws run-instance --image-id ami-1234 --query Instances[0]) query() { jp -u "$2" <<<"$1" } instance_id=$(query "$instance" InstanceId) state_name=$(query "$instance" State.Name) instance_name=$(query "$instance" "Tags[?Key=='Name'].Value | [0]") instance= {"InstanceId": ”i-12345", ... }
  46. 46. Saving the entire output for querying later instance=$(aws run-instance --image-id ami-1234 --query Instances[0]) query() { jp -u "$2" <<<"$1" } instance_id=$(query "$instance" InstanceId) state_name=$(query "$instance" State.Name) instance_name=$(query "$instance" "Tags[?Key=='Name'].Value | [0]")
  47. 47. Saving the entire output for querying later instance=$(aws run-instance --image-id ami-1234 --query Instances[0]) query() { jp -u "$2" <<<"$1" } instance_id=$(query "$instance" InstanceId) state_name=$(query "$instance" State.Name) instance_name=$(query "$instance" "Tags[?Key=='Name'].Value | [0]")
  48. 48. Saving the entire output for querying later instance=$(aws run-instance --image-id ami-1234 --query Instances[0]) query() { jp -u "$2" <<<"$1" } instance_id=$(query "$instance" InstanceId) state_name=$(query "$instance" State.Name) instance_name=$(query "$instance" "Tags[?Key=='Name'].Value | [0]")
  49. 49. Saving the entire output for querying later 1. Use JSON output, not text, --output json 2. Store the entire thing as a variable, or write to a temp file if output size is a concern. 3. Use the “jp” command to filter down results as needed.
  50. 50. Main Patterns  Single output to a single command  Map list output to N AWS CLI commands  Storing JSON documents and querying later • Resource exists check
  51. 51. resource_exists() { } 1. Determine command to run with query 2. Run command and check errors 3. Check length of query result
  52. 52. resource_exists() { local cmd local num_matches if [[ -z "$2" ]]; then cmd="$1 --query length(*[0])" else cmd="$1 --query $2" fi } 2. Run command and check errors 3. Check length of query result
  53. 53. resource_exists() { num_matches=$($command) if [[ "$?" -ne 0 ]]; then echo "Could not check if resource exists, exiting." exit 2 fi } 3. Check length of query result 1. Determine command to run with query
  54. 54. resource_exists() { if [[ "$num_matches" -gt 0 ]]; then # RC of 0 mean the resource exists, success. return 0 else return 1 fi } 1. Determine command to run with query 2. Run command and check errors
  55. 55. Usage: Server-side filtering describe_groups="aws ec2 describe-security-groups" if resource_exists "$describe_groups --filter Name=tag:dev-ec2-instance,Values=linux"; then echo "Security groups exists." else echo "Security group not found." fi
  56. 56. Usage: Client-side filtering list_profiles="aws iam list-instance-profiles" if resource_exists "$list_profiles" "InstanceProfiles[?InstanceProfileName=='dev-ec2-instance']" then echo "Instance profile exists." else echo "Missing IAM instance profile 'dev-ec2-instance'" create_instance_profile fi
  57. 57. Resource exists • Check if a resource exists by filtering down a list and checking if the length is greater than 0. • Use server side filtering when possible • Use JMESPath query for client side filtering
  58. 58. Demo
  59. 59. Common Patterns  Single output to a single command  Map list output to N AWS CLI commands  Storing JSON documents and querying later  Resource exists check
  60. 60. Deployment tips
  61. 61. # * * * * * command to execute # │ │ │ │ │ # │ │ │ │ │ # │ │ │ │ └───── day of week (0 – 6) # │ │ │ └────────── month (1 - 12) # │ │ └─────────────── day of month (1 - 31) # │ └──────────────────── hour (0 - 23) # └───────────────────────── min (0 - 59) Cron
  62. 62. $ whoami $HOME/.aws/config $HOME/.aws/credentials export AWS_CONFIG_FILE export AWS_SHARED_CREDENTIALS_FILE
  63. 63. $ whoami $HOME/.aws/config $HOME/.aws/credentials export AWS_CONFIG_FILE export AWS_SHARED_CREDENTIALS_FILE • Explicitly set AWS_CONFIG_FILE/AWS_SHARED_CREDENTIALS_FILE • Ensure user running cron job has ~/.aws/config ~/.aws/credentials
  64. 64. aws configure set region $(curl -s http://169.254.169.254/latest/dynamic/instance-identity/document | jp -u 'region') Configure a region
  65. 65. Summary • Common patterns • Shell script examples • Deployment • Tools
  66. 66. Remember to complete your evaluations!
  67. 67. Thank you! • User guide: http://docs.aws.amazon.com/cli/latest/userguide/ • Reference docs: http://docs.aws.amazon.com/cli/latest/reference/ • JMESPath: http://jmespath.org/ • re:Invent 2014 - https://www.youtube.com/watch?v=vP56l7qThNs • re:Invent 2013 - https://www.youtube.com/watch?v=qiPt1NoyZm0

×