(DEV301) Advanced Usage of the AWS CLI | AWS re:Invent 2014
•
26 likes•10,399 views
The AWS CLI provides an easy-to-use command line interface to AWS and allows you to create powerful automation scripts. In this session, you learn advanced techniques that open up new scenarios for using the CLI. We demonstrate how to filter and transform service responses, how to chain and script commands, and how to write custom plugins.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to (DEV301) Advanced Usage of the AWS CLI | AWS re:Invent 2014
Similar to (DEV301) Advanced Usage of the AWS CLI | AWS re:Invent 2014 (20)
More from Amazon Web Services
More from Amazon Web Services (20)
Recently uploaded
Recently uploaded (20)
(DEV301) Advanced Usage of the AWS CLI | AWS re:Invent 2014
- 1. Advanced Usage of the AWS CLI James Saryerwinnie, Amazon Web Services November 12, 2014 | Las Vegas © 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
- 2. Crash Course Foundation Advanced Scenarios Intro to the AWS CLI Exploring Key Functionality Looking at Advanced CLI Features
- 3. Crash Course Advanced Proficient Novice Intro to the AWS CLI
- 4. AWS Command Line Interface Unified tool to manage your AWS services
- 5. MSI (Windows) Bundled (cross platform) pip (cross platform)
- 7. $ aws ec2 describe-‐instances service (command) operation (subcommand)
- 8. $ aws iam list-‐access-‐keys service (command) operation (subcommand)
- 9. --output json { "Places": [ { "City": "Seattle", "State": "WA" }, { "City": "Las Vegas", "State": "NV" } ] }
- 10. --output text PLACES Seattle WA PLACES Las Vegas NV
- 11. --output table -‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐ | SomeOperationName | +-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+ || Places || |+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐+| || City | State || |+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐+| || Seattle | WA || || Las Vegas | NV || |+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐+|
- 12. All Outputs { JSON Text Table "Places": [ { "City": "Seattle", "State": "WA" }, { "City": "Las Vegas", "State": "NV" } ] } PLACES Seattle WA PLACES Las Vegas NV -‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐ | SomeOperationName | +-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+ || Places || |+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐+| || City | State || |+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐+| || Seattle | WA || || Las Vegas | NV || |+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐+|
- 13. Basic AWS CLI Usage DEMO
- 14. Foundation Advanced Exploring Key Functionality Proficient Novice
- 15. Configuration
- 16. aws configure
- 17. aws configure AWS Access Key ID [**ABCD]: AWS Secret Access Key [****************EFGH]: Default region name [us-‐west-‐2]: Default output format [None]:
- 19. aws configure <subcommand> list - List common configuration sources! get - Get the value of a single config var! set - Set the value of a single config var
- 20. aws configure get region
- 21. aws configure set profile.prod.region us-‐west-‐2
- 22. A profile is a group of configuration values
- 23. aws configure -‐-‐profile prod
- 24. Configuration Files ~/.aws/credentials ~/.aws/config • supported by all AWS SDKs! • only contains credentials • Used only by the CLI! • Can contain credentials (but not the default behavior)
- 26. aws configure set profile.prod.aws_access_key_id foo ~/.aws/credentials ~/.aws/config
- 27. aws configure set profile.prod.aws_access_key_id foo ~/.aws/credentials ~/.aws/config [prod] aws_access_key_id = foo
- 28. aws configure set profile.prod.aws_secret_access_key bar ~/.aws/credentials ~/.aws/config [prod] aws_access_key_id = foo
- 29. aws configure set profile.prod.aws_secret_access_key bar ~/.aws/credentials ~/.aws/config [prod] aws_access_key_id = foo aws_secret_access_key = bar
- 30. aws configure set profile.prod.region us-‐west-‐2 ~/.aws/credentials ~/.aws/config [prod] aws_access_key_id = foo aws_secret_access_key = bar
- 31. aws configure set profile.prod.region us-‐west-‐2 ~/.aws/credentials ~/.aws/config [profile prod] region = us-west-2 [prod] aws_access_key_id = foo aws_secret_access_key = bar
- 32. aws configure set profile.prod.output text ~/.aws/credentials ~/.aws/config [profile prod] region = us-west-2 [prod] aws_access_key_id = foo aws_secret_access_key = bar
- 33. aws configure set profile.prod.output text ~/.aws/credentials ~/.aws/config [profile prod] region = us-west-2 output = text [prod] aws_access_key_id = foo aws_secret_access_key = bar
- 34. create-new-user.sh #!/bin/bash # Create a new user and create a new profile. aws iam create-‐user -‐-‐user-‐name reinvent-‐user credentials=$(aws iam create-‐access-‐key -‐-‐user-‐name reinvent-‐user -‐-‐query 'AccessKey.[AccessKeyId,SecretAccessKey]' -‐-‐output text) access_key_id=$(echo $credentials | cut -‐d' ' -‐f 1) secret_access_key=$(echo $credentials | cut -‐d' ' -‐f 2) aws configure set profile.reinvent.aws_access_key_id "$access_key_id" aws configure set profile.reinvent.secret_access_key "$secret_access_key"
- 35. create-new-user.sh #!/bin/bash # Create a new user and create a new profile. aws iam create-‐user -‐-‐user-‐name reinvent-‐user credentials=$(aws iam create-‐access-‐key -‐-‐user-‐name reinvent-‐user -‐-‐query 'AccessKey.[AccessKeyId,SecretAccessKey]' -‐-‐output text) access_key_id=$(echo $credentials | cut -‐d' ' -‐f 1) secret_access_key=$(echo $credentials | cut -‐d' ' -‐f 2) aws configure set profile.reinvent.aws_access_key_id "$access_key_id" aws configure set profile.reinvent.secret_access_key "$secret_access_key"
- 36. create-new-user.sh #!/bin/bash # Create a new user and create a new profile. aws iam create-‐user -‐-‐user-‐name reinvent-‐user credentials=$(aws iam create-‐access-‐key -‐-‐user-‐name reinvent-‐user -‐-‐query 'AccessKey.[AccessKeyId,SecretAccessKey]' -‐-‐output text) access_key_id=$(echo $credentials | cut -‐d' ' -‐f 1) secret_access_key=$(echo $credentials | cut -‐d' ' -‐f 2) aws configure set profile.reinvent.aws_access_key_id "$access_key_id" aws configure set profile.reinvent.secret_access_key "$secret_access_key"
- 37. create-new-user.sh #!/bin/bash # Create a new user and create a new profile. aws iam create-‐user -‐-‐user-‐name reinvent-‐user credentials=$(aws iam create-‐access-‐key -‐-‐user-‐name reinvent-‐user -‐-‐query 'AccessKey.[AccessKeyId,SecretAccessKey]' -‐-‐output text) access_key_id=$(echo $credentials | cut -‐d' ' -‐f 1) secret_access_key=$(echo $credentials | cut -‐d' ' -‐f 2) aws configure set profile.reinvent.aws_access_key_id "$access_key_id" aws configure set profile.reinvent.secret_access_key "$secret_access_key"
- 38. create-new-user.sh #!/bin/bash # Create a new user and create a new profile. aws iam create-‐user -‐-‐user-‐name reinvent-‐user credentials=$(aws iam create-‐access-‐key -‐-‐user-‐name reinvent-‐user -‐-‐query 'AccessKey.[AccessKeyId,SecretAccessKey]' -‐-‐output text) access_key_id=$(echo $credentials | cut -‐d' ' -‐f 1) secret_access_key=$(echo $credentials | cut -‐d' ' -‐f 2) aws configure set profile.reinvent.aws_access_key_id "$access_key_id" aws configure set profile.reinvent.secret_access_key "$secret_access_key"
- 39. create-new-user.sh #!/bin/bash # Create a new user and create a new profile. aws iam create-‐user -‐-‐user-‐name reinvent-‐user credentials=$(aws iam create-‐access-‐key -‐-‐user-‐name reinvent-‐user -‐-‐query 'AccessKey.[AccessKeyId,SecretAccessKey]' -‐-‐output text) access_key_id=$(echo $credentials | cut -‐d' ' -‐f 1) secret_access_key=$(echo $credentials | cut -‐d' ' -‐f 2) aws configure set profile.reinvent.aws_access_key_id "$access_key_id" aws configure set profile.reinvent.secret_access_key "$secret_access_key"
- 40. Use the aws configure suite of subcommands
- 41. Query
- 42. -‐-‐query (string) ! A JMESPath query to use in filtering the response data.
- 43. Implementation Details --query Processing aws ec2 ... parse params HTTP request parse response format response display response Retry Pagination
- 44. Implementation Details --query Processing aws ec2 ... parse params HTTP request format response display response Retry Pagination parse response parse response body apply --query
- 45. Implementation Details --query Processing <ListUsersResponse xmlns="..."> <ListUsersResult> <Users> <member> <UserId>userid</UserId> <Path>/</Path> <UserName>james</UserName> <Arn>arn:aws:iam::..:user/james</Arn> <CreateDate>2013-‐03-‐09T23:36:32Z</CreateDate> </member> <Users> </ListUsersResult> <ListUsersResponse> !!
- 46. Implementation Details --query Processing { "Users": [ { "Arn": "arn:aws:iam::...:user/james", "UserId": "userid", "CreateDate": "2013-‐03-‐09T23:36:32Z", "Path": "/", "UserName": "james" } ] }
- 47. Implementation Details --query Processing -‐-‐query Users[0].[UserName,Path,UserId] { "Users": [ { "Arn": "arn:aws:iam::...:user/james", "UserId": "userid", "CreateDate": "2013-‐03-‐09T23:36:32Z", "Path": "/", "UserName": "james" } ] }
- 48. Implementation Details --query Processing -‐-‐query Users[0].[UserName,Path,UserId] [ [ "james", "/", "id" ], ]
- 49. Implementation Details --query Processing -‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐ | ListUsers | +-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+ | james | / | userid | +-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+
- 50. Implementation Details --query Processing aws ec2 ... parse params HTTP request parse response format response display response Retry Pagination
- 51. http://jmespath.org A Query Language For JSON
- 52. Waiters
- 53. Amazon EC2 Instance State Transitions pending rebooting running Stop stopping stopped Terminate shutting-down terminated Reboot Terminate Start
- 54. ec2-instance-running.sh #!/bin/bash instance_id=$(aws ec2 run-‐instances -‐-‐image-‐id ami-‐12345 -‐-‐query Reservations[].Instances[].InstanceId -‐-‐output text) instance_state=$(aws ec2 describe-‐instances -‐-‐instance-‐ids $instance_id -‐-‐query 'Reservations[].Instances[].State.Name') while [ "$instance_state" != "running" ] do sleep 1 instance_state=$(aws ec2 describe-‐instances -‐-‐instance-‐ids $instance_id -‐-‐query 'Reservations[].Instances[].State.Name') done
- 55. ec2-instance-running.sh #!/bin/bash instance_id=$(aws ec2 run-‐instances -‐-‐image-‐id ami-‐12345 -‐-‐query Reservations[].Instances[].InstanceId -‐-‐output text) instance_state=$(aws ec2 describe-‐instances -‐-‐instance-‐ids $instance_id -‐-‐query 'Reservations[].Instances[].State.Name') while [ "$instance_state" != "running" ] do sleep 1 instance_state=$(aws ec2 describe-‐instances -‐-‐instance-‐ids $instance_id -‐-‐query 'Reservations[].Instances[].State.Name') done
- 56. ec2-instance-running.sh #!/bin/bash instance_id=$(aws ec2 run-‐instances -‐-‐image-‐id ami-‐12345 -‐-‐query Reservations[].Instances[].InstanceId -‐-‐output text) instance_state=$(aws ec2 describe-‐instances -‐-‐instance-‐ids $instance_id -‐-‐query 'Reservations[].Instances[].State.Name') while [ "$instance_state" != "running" ] do sleep 1 instance_state=$(aws ec2 describe-‐instances -‐-‐instance-‐ids $instance_id -‐-‐query 'Reservations[].Instances[].State.Name') done
- 57. ec2-instance-running.sh #!/bin/bash instance_id=$(aws ec2 run-‐instances -‐-‐image-‐id ami-‐12345 -‐-‐query Reservations[].Instances[].InstanceId -‐-‐output text) instance_state=$(aws ec2 describe-‐instances -‐-‐instance-‐ids $instance_id -‐-‐query 'Reservations[].Instances[].State.Name') while [ "$instance_state" != "running" ] do sleep 1 instance_state=$(aws ec2 describe-‐instances -‐-‐instance-‐ids $instance_id -‐-‐query 'Reservations[].Instances[].State.Name') done
- 58. ec2-instance-running.sh #!/bin/bash instance_id=$(aws ec2 run-‐instances -‐-‐image-‐id ami-‐12345 -‐-‐query Reservations[].Instances[].InstanceId -‐-‐output text) instance_state=$(aws ec2 describe-‐instances -‐-‐instance-‐ids $instance_id -‐-‐query 'Reservations[].Instances[].State.Name') while [ "$instance_state" != "running" ] do sleep 1 instance_state=$(aws ec2 describe-‐instances -‐-‐instance-‐ids $instance_id -‐-‐query 'Reservations[].Instances[].State.Name') done
- 59. ec2-instance-running.sh #!/bin/bash instance_id=$(aws ec2 run-‐instances -‐-‐image-‐id ami-‐12345 -‐-‐query Reservations[].Instances[].InstanceId -‐-‐output text) instance_state=$(aws ec2 describe-‐instances -‐-‐instance-‐ids $instance_id -‐-‐query 'Reservations[].Instances[].State.Name') while [ "$instance_state" != "running" ] do sleep 1 instance_state=$(aws ec2 describe-‐instances -‐-‐instance-‐ids $instance_id -‐-‐query 'Reservations[].Instances[].State.Name') done • No timeouts • Failure states • Hand-written code
- 60. ec2-waiters.sh instance_id=$(aws ec2 run-‐instances -‐-‐image-‐id ami-‐12345 -‐-‐query Reservations[].Instances[].InstanceId -‐-‐output text) aws ec2 wait instance-‐running -‐-‐instance-‐ids $instance_id
- 61. ec2-waiters.sh instance_id=$(aws ec2 run-‐instances -‐-‐image-‐id ami-‐12345 -‐-‐query Reservations[].Instances[].InstanceId -‐-‐output text) aws ec2 wait instance-‐running -‐-‐instance-‐ids $instance_id subcommand waiter name describe-instances options
- 62. Advanced Scenarios Advanced Looking at advanced AWS CLI features Proficient Novice
- 63. Templates
- 64. The AWS CLI is data driven
- 65. Implementation Details JSON Models "RunInstancesRequest":{ "type":"structure", "required":[ "ImageId", "MinCount", "MaxCount" ], "members":{ "ImageId":{"shape":"String"}, "MinCount":{"shape":"Integer"}, "MaxCount":{"shape":"Integer"}, "KeyName":{"shape":"String"}, "SecurityGroups":{ "shape":"SecurityGroupStringList", "locationName":"SecurityGroup" },
- 66. Implementation Details JSON Models "RunInstancesRequest":{ "type":"structure", "required":[ "ImageId", "MinCount", "MaxCount" ], "members":{ "ImageId":{"shape":"String"}, "MinCount":{"shape":"Integer"}, "MaxCount":{"shape":"Integer"}, "KeyName":{"shape":"String"}, "SecurityGroups":{ "shape":"SecurityGroupStringList", "locationName":"SecurityGroup" }, -‐-‐image-‐id -‐-‐min-‐count -‐-‐max-‐count -‐-‐key-‐name -‐-‐security-‐groups
- 67. Implementation Details JSON Models "RunInstancesRequest":{ "type":"structure", "required":[ "ImageId", "MinCount", "MaxCount" ], "members":{ "ImageId":{"shape":"String"}, "MinCount":{"shape":"Integer"}, "MaxCount":{"shape":"Integer"}, "KeyName":{"shape":"String"}, "SecurityGroups":{ "shape":"SecurityGroupStringList", "locationName":"SecurityGroup" }, arguments.json { "ImageId": "ami-‐12345", "MinCount": 1, "MaxCount": 1, "KeyName": "id_rsa", "SecurityGroups": ["SSH", "web"], }
- 68. aws ec2 run-‐instances -‐-‐cli-‐input-‐json file://arguments.json
- 69. What else can we do?
- 70. aws ec2 run-‐instances -‐-‐generate-‐cli-‐skeleton
- 71. Creating and using JSON templates DEMO
- 73. export AWS_ACCESS_KEY_ID=... ~/.aws/credentials CredentialLocator ~/.aws/config Amazon EC2 Instance Metadata
- 74. CredentialLocator export AWS_ACCESS_KEY_ID=... ~/.aws/credentials ~/.aws/config Amazon EC2 Instance Metadata
- 75. CredentialLocator export AWS_ACCESS_KEY_ID=... ~/.aws/credentials ~/.aws/config Amazon EC2 Instance Metadata
- 76. CredentialLocator export AWS_ACCESS_KEY_ID=... ~/.aws/credentials ~/.aws/config Amazon EC2 Instance Metadata
- 77. CredentialLocator export AWS_ACCESS_KEY_ID=... ~/.aws/credentials ~/.aws/config Amazon EC2 Instance Metadata
- 78. export AWS_ACCESS_KEY_ID=... ~/.aws/credentials ~/.aws/config Amazon EC2 Instance Metadata CredentialLocator AssumeRole
- 79. Delegate access to AWS resources using AWS Identity and Access Management (IAM) roles
- 81. Production Development Role Policy Trust Policy
- 85. configure-role.sh aws configure set profile.prodrole.role_arn arn:aws:iam... aws configure set profile.prodrole.source_profile dev
- 86. ~/.aws/credentials ~/.aws/config [dev] aws_access_key_id = foo aws_secret_access_key = bar [profile prodrole] role_arn = arn:aws:iam source_profile = dev
- 87. ~/.aws/credentials ~/.aws/config [dev] aws_access_key_id = foo aws_secret_access_key = bar [profile prodrole] role_arn = arn:aws:iam source_profile = dev
- 88. Using roles with the AWS CLI DEMO
- 90. aws s3 cp
- 91. We want to avoid disk
- 92. aws s3 cp -‐ s3://bucket/key
- 93. aws s3 cp s3://bucket/key -‐
- 94. Compress
- 95. s3-compress.sh aws s3 cp s3://bucket/key -‐ | bzip2 -‐-‐best | aws s3 cp -‐ s3://bucket/key.bz2
- 96. Amazon S3 Streaming DEMO
- 97. botocore
- 98. Implementation Details Architecture AWS CLI botocore Providers Serializers Parsers HTTP
- 99. Implementation Details Architecture AWS CLI botocore Providers Serializers Parsers HTTP boto3
- 100. list_buckets.py import botocore ! s = botocore.session.get_session() s3 = s.create_client('s3', region_name='us-‐west-‐2') ! for bucket in s3.list_buckets()['Buckets']: print("Bucket: {bucket}".format(bucket=bucket['Name']))
- 101. Use botocore in the python REPL DEMO
- 102. Wrapping Up • Configuration! • Waiters! • Query • Templates! • Credential Providers! • Amazon S3 Streaming
- 103. For More Information • https://github.com/aws/aws-cli! • http://docs.aws.amazon.com/cli/latest/userguide/! • https://forums.aws.amazon.com/forum.jspa?forumID=150! • http://docs.aws.amazon.com/cli/latest/reference/! • http://jmespath.org/! • Boto3 Talk
- 104. Please give us your feedback on this presentation Join the conversation on Twitter with #reinvent DEV301 © 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.