Are you using the AWS CLI to manage your AWS services and want to do more? In this session learn how to utilize the latest features of the AWS CLI to improve your current workflows in interacting with and managing your AWS resources from the command line. It is recommended that audience members have familiarity with the AWS CLI as this talk will focus on its newer, more advanced features. Come hear from the core AWS CLI development team on how to leverage these features for 2017 and beyond!

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:INVENT
AWS CLI: 2017 and Beyond
K y l e K n a p p , A m a z o n W e b S e r v i c e s
N o v e m b e r 2 9 , 2 0 1 7

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Unified tool to manage AWS services
AWS Command Line Interface

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Improvements in 2017
 Debugging
 Providing credentials
 Amazon S3

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Improvements in 2017
 Debugging
 Providing credentials
 Amazon S3

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS CLI

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS CLI
$ aws s3api list-buckets

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS CLI
$ aws s3api list-buckets
{
"Owner": {...},
"Buckets": [
{
"CreationDate": "2017-01-31T22:34:19.000Z",
"Name": “mybucket"
}
]
}

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS CLI
botocore
$ aws s3api list-buckets

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS CLI
CLI_ARGUMENTS
botocore
$ aws s3api list-buckets
CLI arguments: [s3api, list-buckets]

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS CLI
CLI_ARGUMENTS
botocore
$ aws s3api list-buckets
Client call:
# Python code
response = s3_client.list_buckets()
API_CALL

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
HTTP_REQUEST
AWS CLI
CLI_ARGUMENTS
botocore
$ aws s3api list-buckets
Endpoint URL: https://s3-us-west-2.amazonaws.com
HTTP request method: GET
HTTP request headers:
X-Amz-Content-SHA256: e3b0c44298fc1c149afb…
X-Amz-Date: 20171112T194250Z
Authorization: AWS4-HMAC-SHA256 Credential=…
HTTP request body: None
API_CALL

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
HTTP_REQUEST
AWS CLI
HTTP_RESPONSE
CLI_ARGUMENTS
botocore
$ aws s3api list-buckets
HTTP response status code: 200
HTTP resonse headers:
x-amz-id-2: Fu6qoG5k/iJvkz…
server: AmazonS3
transfer-encoding: chunked
x-amz-request-id: 9E682628048A343652
date: Sun, 12 Nov 2017 19:42:50 GMT
content-type': application/xml
API_CALL

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
HTTP_REQUEST
AWS CLI
HTTP_RESPONSE
CLI_ARGUMENTS
botocore
$ aws s3api list-buckets
HTTP response body:
<?xml version="1.0" encoding="UTF-8"?>
<ListAllMyBucketsResult xmlns="http://s3.amazonaws...">
<Owner>...</Owner>
<Buckets>
<Bucket>...</Bucket>
</Buckets>
</ListAllMyBucketsResult>
API_CALL

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
HTTP_REQUEST
AWS CLI
PARSED_RESPONSE HTTP_RESPONSE
CLI_ARGUMENTS
botocore
$ aws s3api list-buckets
Parsed response:
{
"Owner": {...},
"Buckets": [{...}]
}
API_CALL

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
HTTP_REQUEST
AWS CLI
PARSED_RESPONSE HTTP_RESPONSE
CLI_ARGUMENTS
botocore
CLI_RC
$ aws s3api list-buckets
Return code: 0
API_CALL

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
HTTP_REQUEST
AWS CLI
PARSED_RESPONSE HTTP_RESPONSE
CLI_ARGUMENTS
botocore
CLI_RC
API_CALL
$ aws s3api list-buckets
{
"Owner": {...},
"Buckets": [
{
"CreationDate": "2017-01-31T22:34:19.000Z",
"Name": “mybucket"
}
]
}

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
HTTP_REQUEST
AWS CLI
PARSED_RESPONSE HTTP_RESPONSE
SIGNATURE_DATAAPI_CALLCLI_ARGUMENTS
CLI_RC
--debug

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
HTTP_REQUEST
AWS CLI
PARSED_RESPONSE HTTP_RESPONSE
CLI_ARGUMENTS
botocore
CLI_RC
--debug
API_CALL

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Demo

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Improvements in 2017
 Debugging
 cli_history
 aws history show
 aws history list
 Providing credentials
 Amazon S3

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Improvements in 2017
 Debugging
 Providing credentials
 Amazon S3

22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Credential chain
export AWS_ACCESS_KEY_ID=...
~/.aws/credentials
~/.aws/config
Amazon EC2 Instance
Metadata
Amazon ECS
container
credentials
Credential chain
AWS CLI

23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Credential chain
Credential chain
AWS credentials AWS CLI
$ aws s3 ls
export AWS_ACCESS_KEY_ID=...
~/.aws/credentials
~/.aws/config
Amazon EC2 Instance
Metadata
Amazon ECS
container
credentials

24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Credential chain
export AWS_ACCESS_KEY_ID=...
~/.aws/credentials
~/.aws/config
Amazon EC2 Instance
Metadata
Amazon ECS
container
credentials
Credential chain
AWS SDK for Python
Custom credential provider

25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Credential chain
export AWS_ACCESS_KEY_ID=...
~/.aws/credentials
~/.aws/config
Amazon EC2 Instance
Metadata
Amazon ECS
container
credentials
Credential chain
AWS CLI
Custom credential provider

26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Credential chain
export AWS_ACCESS_KEY_ID=...
~/.aws/credentials
~/.aws/config
Amazon EC2 Instance
Metadata
Amazon ECS
container
credentials
Credential chain
AWS CLI

27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
~ / . a w s / c o n f i g
[default]
aws_access_key_id = AKIDF...
aws_secret_access_key = 1rHl...
[profile dev]
credential_process = /usr/local/bin/awscreds-custom
AWS CLI
$ aws s3 ls --profile dev
stdout (rc=0)
{
“AccessKeyId”: “ADIF...”,
“SecretAccessKey”: “2rFt...”,
“Version”: 1
}
InvokeProcess(/usr/local/bin/awscreds-custom)
awscreds-custom
AWS
credentials

28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Demo

29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
SAML
AssumeRoleWithSAML
LDAP identity store Identity provider (IdP)
role

30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
SAML
AssumeRoleWithSAML
LDAP identity store Identity provider (IdP)
role
AWS CLI
saml-creds

31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Demo

32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Multiple account structure
org account

33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Multiple account structure
org account
team account team account team account

34. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Multiple account structure
org account
team account team account team account
role
AssumeRole

35. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Assume role
[profile admin-org-account]
aws_access_key_id = AKIDF...
aws_secret_access_key = 1rHl...
[profile admin-team-account]
source_profile = admin-org-account
role_arn = arn:...:role/admin-team-account
~/.aws/config
$ aws s3 ls --profile admin-team-account
admin-team-account role
AssumeRole

36. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Multiple account structure
org account
team account team account team account

37. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Multiple account structure
org account
team account team account team account
project account project account

38. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Assume role
[profile admin-org-account]
aws_access_key_id = AKIDF...
aws_secret_access_key = 1rHl...
[profile admin-team-account]
source_profile = admin-org-account
role_arn = arn:...:role/admin-team-account
[profile admin-project-account]
source_profile = admin-team-account
role_arn = arn:...admin-project-account
~/.aws/config
$ aws s3 ls --profile admin-project-account
admin-team-account role
admin-project-account role
AssumeRole
AssumeRole

39. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Assume role
[profile admin-team-account]
credential_source = Ec2InstanceMetadata
role_arn = arn:...role/admin-team-account
~/.aws/config
$ aws s3 ls --profile admin-team-account
admin-team-account role
Amazon EC2 Instance
Metadata
AssumeRole

40. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Assume role
[profile admin-team-account]
credential_source = EcsContainer
role_arn = arn:...role/admin-team-account
~/.aws/config
AssumeRole
Amazon ECS
container
credentials
admin-team-account role
$ aws s3 ls --profile admin-team-account

41. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Assume role
[profile admin-team-account]
credential_source = Environment
role_arn = arn:...role/admin-team-account
~/.aws/config
AssumeRole
export AWS_ACCESS_KEY_ID=...
admin-team-account role
$ aws s3 ls --profile admin-team-account

42. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Demo

43. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Improvements in 2017
 Debugging
 Providing credentials
 credential_process
 source_profile
 credential_source
 Amazon S3

44. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Improvements in 2017
 Debugging
 Providing credentials
 Amazon S3

45. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
aws s3 cp
Amazon S3 bucket
AWS CLI
Thread pool
Disk
Thread 1
Thread 2
Thread 3
IO Thread
IO Queue

46. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
aws s3 cp
Amazon S3 bucket
AWS CLI
Thread pool
Disk
Thread 1
Thread 2
Thread 3
IO Thread
IO Queue

47. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
aws s3 cp
Amazon S3 bucket
AWS CLI
Thread pool
Disk
Thread 1
Thread 2
Thread 3
IO Thread
IO Queue

48. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
aws s3 cp
Amazon S3 bucket
AWS CLI
Thread pool
Disk
Thread 1
Thread 2
Thread 3
IO Thread
IO Queue

49. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
aws s3 cp
Amazon S3 bucket
AWS CLI
Thread pool
Disk
Thread 1
Thread 2
Thread 3
IO Thread
IO Queue

50. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
aws s3 cp
Amazon S3 bucket
AWS CLI
Thread pool
Disk
Thread 1
Thread 2
Thread 3
IO Thread
IO Queue

51. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
aws s3 cp
Amazon S3 bucket
AWS CLI
Thread pool
Disk
Thread 1
Thread 2
Thread 3
IO Thread
IO Queue

52. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
aws s3 cp
Amazon S3 bucket
AWS CLI
Thread pool
Disk
Thread 1
Thread 2
Thread 3
IO Thread
IO Queue

53. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
[default]
region = us-east-1
~/.aws/config

54. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
[default]
region = us-east-1
s3 =
max_concurrent_requests = 20
multipart_chunksize = 16MB
multipart_threshold = 64MB
max_queue_size = 10000
~/.aws/config

55. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
[default]
region = us-east-1
s3 =
max_concurrent_requests = 20
multipart_chunksize = 16MB
multipart_threshold = 64MB
max_queue_size = 10000
~/.aws/config

56. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
max_concurrent_requests = 3
Amazon S3 bucket
AWS CLI
Thread pool
Disk
Thread 1
Thread 2
Thread 3
IO Thread
IO Queue

57. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
max_concurrent_requests = 4
Amazon S3 bucket
AWS CLI
Thread pool
Disk
Thread 1
Thread 2
Thread 3
IO Thread
IO Queue
Thread 4

58. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
max_concurrent_requests = 2
Amazon S3 bucket
AWS CLI
Thread pool
Disk
Thread 1
Thread 2
IO Thread
IO Queue

59. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Demo

60. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Improvements in 2017
 Debugging
 Providing credentials
 Amazon S3
 max_bandwidth

61. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Improvements in 2017
 Debugging
 cli_history
 aws history show
 aws history list
 Providing credentials
 credential_process
 source_profile
 credential_source
 Amazon S3
 max_bandwidth

62. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Helpful Links
• AWS CLI repository: https://github.com/aws/aws-cli
• Process providers: https://github.com/awslabs/awsprocesscreds/
• re:Invent scripts: https://github.com/awslabs/awscli-reinvent-examples
• AWS CLI Blog: https://aws.amazon.com/blogs/developer/category/cli/

63. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you!
@ t h e k y l e k n a p p