This document discusses automating a "break-glass" solution for granting temporary emergency access to critical AWS resources. The solution uses AWS Identity and Access Management (IAM), Amazon Simple Notification Service (SNS), AWS Lambda, AWS Systems Manager, and AWS Service Catalog to provision temporary access. When a request is approved, the helpdesk agent gains one-time access to provision the required resources through a service catalog. The solution ensures access is removed at the end of the approved duration and auditing of all actions is performed.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Perform Diagnostics on Running Instances without
Affecting Availability & Reliability
Nirav Kothari
Principal Consultant – AWS Professional Services
AWS
A R C 3 1 3

3. Break-glass procedure: Granting emergency access to
critical systems
Break glass (which draws its name from breaking the glass to pull a
fire alarm) refers to a quick means for a person who does not have
access privileges to certain information to gain access when necessary

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
“Break-glass” solution automation overview
Solution deep dive
Part 1: Pre-requisites
Part 2: AWS Identity and Access Management (IAM)
Part 3: Amazon Simple Notification Service (Amazon SNS)
Part 4: Amazon CloudWatch
Part 5: AWS Lambda
Part 6: AWS Service Catalog
Part 7: AWS Systems Manager (Systems Manager)

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Breakout repeats
Monday, November 26
ARC313-R - [REPEAT] Perform Diagnostics on Running Instances
without Affecting Availability/Reliability
04:00 p.m. – 05:00 p.m. | Aria West, Level 3, Starvine 10, Table 5, T1
Tuesday, November 27
ARC313-R1 - [REPEAT 1] Perform Diagnostics on Running Instances
without Affecting Availability/Reliability
10:00 a.m. – 11:00 a.m. | Aria West, Level 3, Starvine 3, Table 2, T1
Wednesday, November 28
ARC313-R2 - [REPEAT 2] Perform Diagnostics on Running
Instances without Affecting Availability/Reliability
02:30 p.m. – 03:30 p.m. | Mirage, Grand Ballroom B, Table 8, T1

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What are we solving?
We are solving two separate use cases
1. On-demand, temporary remote access (SSH) for Amazon Elastic Compute Cloud (Amazon EC2)
instances running in a regulated production environment; if your requirement is basic shell access
to an Amazon EC2 instance, you can use AWS Systems Manager; more information here:
https://aws.amazon.com/blogs/aws/new-session-manager/
2. On-demand, temporary higher-privilege access to AWS APIs, for instance network super user

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
“Break-glass” solution overview
virtual private cloud
VPC subnet
VPC subnet
EC2 instance
Bastion
host
Target
hosts
Target
hosts
Users / requestor
Helpdesk Agent
Approver
Lambda
function
Amazon
CloudWatch
Event
AWS
Service Catalog
IAM
AWS
Management
Console
Process flow
User has a need for privileged access or remote access (SSH)
to a target host; user calls Helpdesk to request access
Helpdesk gathers required information to create emergency
change request
Helpdesk reviews information with approver and gets the
request approved. Upon approval Helpdesk receives MFA code
for GlassBreak user
Helpdesk agent logs on to AWS Management Console using
this MFA code
Upon successful login, CloudWatch rule generates an event
Event publishes notification using Amazon SNS, sends email
notification to Helpdesk agent, and triggers Lambda function
Lambda function acts as a login script and adds the
GlassBreak user to a group that has access to an AWS Service
Catalog
Helpdesk agent gets end user’s username and public key
(from the user) and inputs that as a parameter and duration
of access to provision AWS Service Catalog product
AWS Service Catalog uses Systems Manager to create a user
in the bastion host, create a *kill* Lambda function based on
the duration of access requested, and remove the GlassBreak
user from group that has access to AWS Service Catalog

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s break it down . . .
Part 1: Pre-requisites
These components are required to support break-glass
automation
1. A bastion host with no users allowed and AWS Systems Manager installed;
Amazon AMIs have SSM agent pre-installed
2. Security group that allows remote access to bastion host with pre-approved IP
addresses in source
3. An EC2 instance role that allows Systems Manager and some additional access
for the automation to work, we will also add network admin functional role
policy for this use case. I’m using SSM, network admin policy, and a few more
permissions for this use case; we will be using bastion host at the end of duration
to delete automation
4. Target host(s) that allows remote access from bastion host security group

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s break it down . . .
Part 2: IAM
These users and roles are required
1. A pre-designated GlassBreak user with no permissions; this user account will
inherit permissions later at run time
2. A group with appropriate permissions to execute AWS CloudFormation
template, run a Lambda function, and access to an existing AWS Service
Catalog that will be used at run time

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s break it down . . .
Part 3: Amazon SNS
We need Amazon SNS topic for automation to work
1. SNS topic with two subscriptions
1. An email address, Helpdesk email address, to notify that the GlassBreak
user has access to AWS Service Catalog that will run the automation
2. A Lambda function that will detect successful login of the GlassBreak user
and add it to a group that has access to AWS Service Catalog

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s break it down . . .
Part 4: Amazon CloudWatch Events
Create a rule that will monitor successful login by GlassBreak
user and publish on an existing SNS topic
1. Imagine executing a login script

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s break it down . . .
Part 5: AWS Lambda
Two separate Lambda functions are used in this solution
1. The first Lambda function will be pre-installed and will be triggered by an
event. This is an event generated by CloudWatch Events rule upon detecting a
successful login from GlassBreak (pre-designated) user  Think of this as a
login script
2. The second Lambda function will be deployed by AWS Service Catalog that will
accept end user’s public key and duration of the access. Lambda function will
be scheduled based on the duration, and it would perform a clean-up of all the
resources deployed by this automation  Think of this as a clean-up script

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s break it down . . .
Part 6: AWS Service Catalog
AWS Service Catalog to execute process automation
1. AWS Service Catalog will accept requestor public key and duration for the
privileged access
2. Take these parameters and create user with end-user provided public key on
bastion host; this will allow the requestor to log in to bastion host using their
private key
3. AWS Service Catalog will take the duration of access parameter and create a
Lambda function that will perform a clean-up at the end of duration
4. Upon successful completion, this AWS Service Catalog will remove GlassBreak
user from the group that has access to this AWS Service Catalog, run one time

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s break it down . . .
Part 7: AWS Systems Manager
Systems Manager is used for two functions
1. When AWS Service Catalog is executed, it uses Systems Manager to insert user-
provided public key in the bastion host’s *authorized_keys* file; this will allow
user to log in to bastion host using their own private keys
2. A shell script deployed through AWS Service Catalog to delete (self-delete) the
AWS CloudFormation template (that is, “Service Catalog”)

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Additional details
Given the nature of temporary access required, automation has these features
1. The designated user will never have any permissions; upon successful login, governed by a two-factor
authentication, they will get one-time run access to automation
2. Automation will accept duration of access as a mandatory field; this is to ensure that users have privileged access
only for the duration that was approved; if they need to extend their session, they will have to go through this
entire process again
3. Automation ensures that every time a new user is added to bastion host, a log file is updated for audit
4. Run once access to GlassBreak user is provided through an AWS Service Catalog, additional auditing and tracking
capabilities
5. AWS Service Catalog will remove the GlassBreak user from group to ensure that the Helpdesk agent can run this
automation only one time
6. AWS Service Catalog creates a self-destructing Lambda function to ensure everything deployed with this
automation self-cleans at the end of the duration
7. To allow multiple users access at the same time and to ensure that there might not be a first in-first out, all
function names are dynamically assigned in the code

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Optimization options
Break-glass solution and this lab were designed to use AWS native services.
1. If your organization has SSH key management solution, you can update the AWS Service Catalog to ask for duration
only; you can automate key vending from your current key management implementation
2. Duration of access is parameterized to allow any duration between 5 and 60 mins; update it as per your
organization requirement
3. Solution assumes that end user will use AWS Command Line Interface (AWS CLI) for privilege access; after access to
bastion host is granted, if they prefer UI access, consider workspaces to replace bastion host or grant role to the
userID
4. You can add Amazon SNS notification in the clean-up function to notify you when the end-user access is revoked
5. This solution assumes that you have a bastion host that’s always running, with specific functional roles such as EC2
admins, network admins, etc., you can create bastion host with individual functional roles for your environment, or
automate the solution to accept privilege required as a parameter and provision bastion host at run time
6. You can enable session logging on bastion host for additional logging and auditing purposes; please refer to
https://aws.amazon.com/blogs/security/how-to-record-ssh-sessions-established-through-a-bastion-host/

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
“Break glass” summary slide
Request routed
for approval
Approver shares
MFA code
Lambda adds user
to group
Helpdesk logs in as
GlassBreak user
User
request
CWE detects
login
Helpdesk notified and
Lambda triggered
Helpdesk runs AWS
Service Catalog
Accepts end user’s
pub key and duration
Systems Manager
creates user on
bastion host
Clean up Lambda
function scheduled
based on duration
User gains access
to AWS Service
Catalog
Approved?
Check duration?GlassBreak user
removed from group
Run Cleanup
function
Terminate
GlassBreak user’s
SSH session
Systems Manager
deletes end user
from bastion host
Delete AWS Service
Catalog provisioned
product
End
Helpdesk gathers
information
1
1
Wait
2
2 3
3

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Acronyms used in this document
Acronym Description
Requestor An end user/user that is requesting temporary privilege access
Helpdesk or Helpdesk
agent
Support staff that receives the request routes it through organizational approval process and runs AWS
Service Catalog that would grant temporary access to end user
Approver A designated approver in your organization authorized to approve an emergency change request
EC2 Amazon EC2 service; more information can be found here https://aws.amazon.com/ec2/
IAM
AWS Identity and Access Management (IAM) service; more information can be found here
https://aws.amazon.com/iam/
Systems Manager
AWS Systems Manager service; more information can be found here
https://aws.amazon.com/systems-manager/?nc2=h_m1
AWS Service Catalog
AWS Service Catalog service; more information can be found here
https://aws.amazon.com/servicecatalog/?nc2=h_m1
CloudWatch Events
Part of Amazon CloudWatch service; more information can be found here
https://aws.amazon.com/cloudwatch/?nc2=h_m1
Lambda
AWS Lambda service; more information can be found here
https://aws.amazon.com/lambda/?nc2=h_m1

20. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Nirav Kothari
kotharin@amazon.com

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.