Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Martin Sirull, AWS Professional Services
Mirza B...
On today’s show…
Martin’s gonna talk about why we deployed an application
in production without SSH keys. And then dive in...
Reference application
• Experian.com
• 10+ million users
• 100,000+ requests per hour
• PCI-compliant environment
What are the network security threats?
Open Ports
DDOS
SQL Injection
XSS
CSRFPoodle
Heartbleed
Challenges of SSH
SSH tunnels
• Forward tunneling
• Reverse SSH tunneling
• Easy to circumvent firewall rules
Key manageme...
Did you know?
Immutable infrastructure possible?
What’s truly immutable infrastructure?
What’s practically immutable infrastructure?
What do we want?
Photo by Jurvetson (flickr)
AUTOMATE
EVERYTHING!
Key goals
• No humans in production
• Everything has to be automated
• No SSH back doors into production
• Development has...
Ask 2 questions Instead
How are we going to get changes into the pipeline?
How are we going to automatically get the data ...
What does our target environment need?
How are we going to automate?
AMI (image) baking!
The pipeline
AWS
CodeCommit
Amazon ECS
Build/test
Deploy
Redeploy to next
environments
Git clone
What is AWS CloudFormation?
CloudFormation
template
CloudFormation
stack AWS resources
What is AWS CloudFormation?
What goes in AWS CloudFormation?
• Amazon S3 buckets
• Amazon DynamoDB tables
• Amazon SQS
• Amazon RDS databases
• Amazon...
What is AWS CloudFormation?
How do we make it easier for developers?
{
"ServiceName": ”MyAwesomeService",
"DeploymentSystem": ”ECS",
"DeploymentType":...
How do we make it easier for developers?
{
"Resources": {
"KMS": [
{
"logical_id": "DefaultKey"
}
],
"S3": [
{
"logical_id...
What does our target environment need?
Base instance configuration: cfn-init
{
"Resources": {
"MyInstance": {
"Type": "AWS::EC2::Instance",
"Metadata": {
"AWS::C...
Implications on development
The initial reaction
So you’re telling me that
we are rolling a brand new
platform out to production,
with 100s of instanc...
What does our target environment need?
App-specific instance configuration: AWS CodeDeploy
Developer view of AWS CodeDeploy
How to debug code deployments?
How do we configure the application?
The road to self-discovery – Step 1
The road to self-discovery – Step 2
The road to self-discovery – Step 3
Configuration properties
• Feature flags
• Thread pool sizing
• ListenPort
Secure conf...
How about a developer’s config?
Challenges with instance bootstrapping?
• Dependency issues with package installation at runtime
• Potential vector for ma...
Can we combine these layers?
What is Docker?
How to get started?
FROM ubuntu:trusty
EXPOSE 80
RUN apt-get update
RUN apt-get install -y python3-setuptools
RUN easy_ins...
How to get started?
FROM ubuntu:trusty
EXPOSE 80
RUN apt-get update
RUN apt-get install -y python3-setuptools
RUN easy_ins...
How about the external environment?
Implications on development – Environment
configuration
What do we typically need to know about the outside world?
• Datab...
The road to self-discovery – Step 2 ( repeat )
The road to self-discovery – Step 3B
aws cloudformation list-stack-resources –stack-name receiptservice-prod-87287ASD0
• S3 buckets
• DynamoDB tables
• SQS
• RDS* databases
• KMS keys
What about credentials
IAM
What about after the application is up?
A GOOD day in production
A BAD day in production
Instances down?!
NO SSH!
Keep Calm
And Turn Debug On
Keep calm and turn debug on
Production monitoring – Keeping your cool
All logs are immediately shipped off of the box
• Logstash, ELK, Splunk, etc
• W...
Production monitoring – Keeping your cool
Proactive monitoring
• CloudWatch metrics
• Leveraging APM solutions such as New...
Other implications on development
Instances must be ephemeral
Fits the microservices paradigm
• No application state writt...
What happens when….?
I REALLY need access to the disk for forensics, etc.?
• No change from existing best practice
• Snaps...
Securing code pipelines
All changes are versioned
• All ability to deploy changes are managed through IAM roles
• AWS Clou...
Break glass in case of emergency?
Ask 2 questions Instead
How are we going to get changes into the pipeline?
How are we going to automatically get the data ...
How many times have we had to log in?
0
2 years
Thank you!
Remember to complete
your evaluations!
AWS re:Invent 2016: Life Without SSH: Immutable Infrastructure in Production (SAC318)
AWS re:Invent 2016: Life Without SSH: Immutable Infrastructure in Production (SAC318)
AWS re:Invent 2016: Life Without SSH: Immutable Infrastructure in Production (SAC318)
AWS re:Invent 2016: Life Without SSH: Immutable Infrastructure in Production (SAC318)
AWS re:Invent 2016: Life Without SSH: Immutable Infrastructure in Production (SAC318)
Upcoming SlideShare
Loading in …5
×

AWS re:Invent 2016: Life Without SSH: Immutable Infrastructure in Production (SAC318)

2,742 views

Published on

This session covers what a real-world production deployment of a fully automated deployment pipeline looks like with instances that are deployed without SSH keys. By leveraging AWS CloudFormation along with Docker and AWS CodeDeploy, we show how we achieved semi-immutable and fully immutable infrastructures, and what the challenges and remediations were.

Published in: Technology

AWS re:Invent 2016: Life Without SSH: Immutable Infrastructure in Production (SAC318)

  1. 1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Martin Sirull, AWS Professional Services Mirza Baig, Experian Consumer Services December 1, 2016 SAC318 Life Without SSH Immutable Infrastructure in Production
  2. 2. On today’s show… Martin’s gonna talk about why we deployed an application in production without SSH keys. And then dive into how it got deployed. Mirza’s gonna talk about how Martin’s points above impacted (or didn’t) development and then how the production environment was monitored.
  3. 3. Reference application • Experian.com • 10+ million users • 100,000+ requests per hour • PCI-compliant environment
  4. 4. What are the network security threats? Open Ports DDOS SQL Injection XSS CSRFPoodle Heartbleed
  5. 5. Challenges of SSH SSH tunnels • Forward tunneling • Reverse SSH tunneling • Easy to circumvent firewall rules Key management • Where do you store them? Can you control storage? • Rotation of keys? • Federation? (Centrify, etc)
  6. 6. Did you know?
  7. 7. Immutable infrastructure possible?
  8. 8. What’s truly immutable infrastructure?
  9. 9. What’s practically immutable infrastructure?
  10. 10. What do we want?
  11. 11. Photo by Jurvetson (flickr) AUTOMATE EVERYTHING!
  12. 12. Key goals • No humans in production • Everything has to be automated • No SSH back doors into production • Development has to be: Easy, fast, secure. Pick three
  13. 13. Ask 2 questions Instead How are we going to get changes into the pipeline? How are we going to automatically get the data we need off the box?
  14. 14. What does our target environment need?
  15. 15. How are we going to automate? AMI (image) baking!
  16. 16. The pipeline AWS CodeCommit Amazon ECS Build/test Deploy Redeploy to next environments Git clone
  17. 17. What is AWS CloudFormation? CloudFormation template CloudFormation stack AWS resources
  18. 18. What is AWS CloudFormation?
  19. 19. What goes in AWS CloudFormation? • Amazon S3 buckets • Amazon DynamoDB tables • Amazon SQS • Amazon RDS databases • Amazon ElastiCache instances • AWS KMS keys • IAM roles • IAM policies • Amazon CloudFront • Amazon VPC • Internet gateway • Routes • Route tables • Network ACL • Front-end router/ELB • Internal ELB • Auto Scaling group and metrics
  20. 20. What is AWS CloudFormation?
  21. 21. How do we make it easier for developers? { "ServiceName": ”MyAwesomeService", "DeploymentSystem": ”ECS", "DeploymentType": "Python", "Port": 8080, "RootDir": ”helloworld”, "APIGateway": "True" }
  22. 22. How do we make it easier for developers? { "Resources": { "KMS": [ { "logical_id": "DefaultKey" } ], "S3": [ { "logical_id": "StandardBucket" } ], "Dynamo": [ { "logical_id": "table", "hash": "hash", "range": "range" } }
  23. 23. What does our target environment need?
  24. 24. Base instance configuration: cfn-init { "Resources": { "MyInstance": { "Type": "AWS::EC2::Instance", "Metadata": { "AWS::CloudFormation::Init": { "config": { "packages": {}, "groups": {}, "users": {}, "sources": {}, "files": {}, "commands": {}, "services": {} }}}}}}
  25. 25. Implications on development
  26. 26. The initial reaction So you’re telling me that we are rolling a brand new platform out to production, with 100s of instances, and we can’t log in to a single one?
  27. 27. What does our target environment need?
  28. 28. App-specific instance configuration: AWS CodeDeploy
  29. 29. Developer view of AWS CodeDeploy
  30. 30. How to debug code deployments?
  31. 31. How do we configure the application?
  32. 32. The road to self-discovery – Step 1
  33. 33. The road to self-discovery – Step 2
  34. 34. The road to self-discovery – Step 3 Configuration properties • Feature flags • Thread pool sizing • ListenPort Secure configuration repository • Consul • Spring cloud config • Custom solution • DynamoDB • Amazon S3
  35. 35. How about a developer’s config?
  36. 36. Challenges with instance bootstrapping? • Dependency issues with package installation at runtime • Potential vector for malicious code injection? • Automatic scaling slower with a full bootstrap
  37. 37. Can we combine these layers?
  38. 38. What is Docker?
  39. 39. How to get started? FROM ubuntu:trusty EXPOSE 80 RUN apt-get update RUN apt-get install -y python3-setuptools RUN easy_install3 pip RUN pip3 install flask ADD . /home/root CMD python3 /home/root/hello_world.py
  40. 40. How to get started? FROM ubuntu:trusty EXPOSE 80 RUN apt-get update RUN apt-get install -y python3-setuptools RUN easy_install3 pip RUN pip3 install flask ADD . /home/root CMD python3 /home/root/hello_world.py
  41. 41. How about the external environment?
  42. 42. Implications on development – Environment configuration What do we typically need to know about the outside world? • Database tables • Amazon SQS queues • Encryption keys • Amazon S3 buckets • Amazon SNS topics • Amazon Kinesis streams • Amazon ElastiCache endpoints
  43. 43. The road to self-discovery – Step 2 ( repeat )
  44. 44. The road to self-discovery – Step 3B aws cloudformation list-stack-resources –stack-name receiptservice-prod-87287ASD0
  45. 45. • S3 buckets • DynamoDB tables • SQS • RDS* databases • KMS keys What about credentials IAM
  46. 46. What about after the application is up?
  47. 47. A GOOD day in production
  48. 48. A BAD day in production
  49. 49. Instances down?! NO SSH!
  50. 50. Keep Calm And Turn Debug On Keep calm and turn debug on
  51. 51. Production monitoring – Keeping your cool All logs are immediately shipped off of the box • Logstash, ELK, Splunk, etc • Writing directly to Amazon CloudWatch Logs and subscriptions • http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Subs criptions.html
  52. 52. Production monitoring – Keeping your cool Proactive monitoring • CloudWatch metrics • Leveraging APM solutions such as NewRelic, AppDynamics, etc • Advanced health checks • SpringBoot ACTUATOR – Health – Metrics – Service information – Thread dumps – Environment
  53. 53. Other implications on development Instances must be ephemeral Fits the microservices paradigm • No application state written to disk • Key for automatic scaling • Cheap to manufacture ( CloudFormation templates )
  54. 54. What happens when….? I REALLY need access to the disk for forensics, etc.? • No change from existing best practice • Snapshot volume and connect to forensics EC2 instance I need to do a thread dump? • Standardized logging on startup/shutdown sequences Other Implications on development
  55. 55. Securing code pipelines All changes are versioned • All ability to deploy changes are managed through IAM roles • AWS CloudTrail auditing Source code is sanitized • Clean package dependencies • OWASP dependency check Static analysis • Parasoft, Fortify, Veracode, etc
  56. 56. Break glass in case of emergency?
  57. 57. Ask 2 questions Instead How are we going to get changes into the pipeline? How are we going to automatically get the data we need off the box?
  58. 58. How many times have we had to log in? 0 2 years
  59. 59. Thank you!
  60. 60. Remember to complete your evaluations!

×