AWS CloudFormation provides many features to automate the provisioning of infrastructure for all types of complex applications. In this workshop, you will learn how to build AWS CloudFormation templates using proven methods and best practices. You will also deploy a fully functional data lake architecture, which uses AWS services like Amazon RDS and open source components like Apache Zeppelin. The labs will demonstrate the capabilities of AWS CloudFormation to stand up infrastructure in a modular way, walk through the deployment of a complex end-to-end application, and validate that all components of the application are working.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:INVENT
Luis Colon, Sr. Developer Advocate, AWS Cloud Formation
Chuck Meyer, Sr. Developer Advocate, AWS Cloud Formation
D E V 3 3 7
N o v e m b e r 2 8 , 2 0 1 7
Deploy a Data Lake
with AWS CloudFormation

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CLOUDFORMATION AT A GLANCE
Code in YAML or
JSON directly or use
sample templates
Upload local
files or from
an S3 bucket
Create stack
using console,
API or CLI
Stacks and
resources are
provisioned

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS CLOUDFORMATION
Enables the provisioning and management of your
infrastructure as code (IaC)
Supports YAML and JSON formats
Allows for reusable component design strategies
Understands dependencies, supports rollbacks and
versioning

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CLOUDFORMATION STACKS
A template is a file listing the resources of a stack in a
declarative way: tell CloudFormation what you need created,
rather than how to create it
• When you change a template, you are declaring the need
to update your stack
• You can do so directly, or with an update plan called a
changeset
Virtually every kind of resource can be managed with stacks

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
APPLICATION ARCHITECTURES
CDN
AppServer
Database
Static Objects
Multi-tier
Transactional
(Java, LAMP)
Cache
Containers
NoSQL Cluster
Orchestration
Microservices
Transactional
(Docker, K8s)
Endpoints
State Machine
IoT
Functions
Serverless
Transactional
(Lambda, APIs, IoT)
Event Streams
Spark Clusters
Warehouse
ETL
Big Data
Analytical
(Data Lakes, Hubs)
…

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Build the data lake Add safety guardrails Add test pipeline (TBD?)
OUR WORKSHOP PLAN

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
LAB 1
Allotted time: 30–35 min
Objectives:
Implement first version of data lake
Verify all components are working

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ADDING SAFETY GUARDRAILS
STACK TERMINATION PROTECTION
RESOURCE DELETION POLICIES
STACK POLICIES
IAM POLICIES
ROLLBACK TRIGGERS
P A R T I I

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
STACK TERMINATION PROTECTION
Prevent a stack from being accidentally deleted

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
RESOURCE DELETION POLICY
Control deletion on a resource-by-resource basis

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
STACK POLICIES
Prevent operations on groups of resources

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
IAM POLICIES
Prevent users, groups, and roles from executing risky operations

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ROLLBACK TRIGGERS
Set up alarms
while updating
stacks
If alarms fire,
automatically
roll back
Revert changes impacting performance

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
LAB 2
Allotted time: 30–35 min
Objectives:
Implement safety guardrail controls
Verify controls are working

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ADD A VALIDATION PIPELINE
P A R T I I I

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
VALIDATION PIPELINE
Run a set of customizable tests for logical and
functional integrity against CloudFormation
templates
• Designed to integrate with an existing AWS
CodeCommit repo
• Automatically provisions and configures necessary
services
• AWS CodePipeline
• AWS CodeBuild
• AWS Lambda

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
VALIDATION PIPELINE
Triggered by a commit on your repo
1. Run logical pre-create tests,
including syntax checks
2. Launch test stacks in multiple
regions
3. Runs functional post-create sets on
the test stacks
4. If tests are successful, an email is
sent to indicate template is ready for
approval
5. Final template is deployed to an S3
bucket, also storing CloudWatch
data for each Lambda function
6. Deploy stack directly from the S3
bucket

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
LAB 3
Allotted time: 30–35 min
Objectives:
Implement validation pipeline
Verify all components are working

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CloudFormation works
for virtually all
application and resource
types, including data
lakes
It also allows
implementation of
several layers of safety
guardrails for critical
resources
Treat your templates as
you would any other code,
including implementing
versioning and automated
testing
WRAP UP

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
THANK YOU!
Luis Colon, Sr. Developer Advocate, AWS Cloud Formation (licolon@amazon.com)
Chuck Meyer, Sr. Developer Advocate, AWS Cloud Formation (cmmeyer@amazon.com)
D E V 3 3 7 D E P L O Y A D A T A L A K E W I T H C L O U D F O R M A T I O N