This session covers what a real-world production deployment of a fully automated deployment pipeline looks like with instances that are deployed without SSH keys. By leveraging AWS CloudFormation along with Docker and AWS CodeDeploy, we show how we achieved semi-immutable and fully immutable infrastructures, and what the challenges and remediations were.

1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Martin Sirull, AWS Professional Services
Mirza Baig, Experian Consumer Services
December 1, 2016
SAC318
Life Without SSH
Immutable Infrastructure in Production

4. On today’s show…
Martin’s gonna talk about why we deployed an application
in production without SSH keys. And then dive into how it
got deployed.
Mirza’s gonna talk about how Martin’s points above
impacted (or didn’t) development and then how the
production environment was monitored.

5. Reference application
• Experian.com
• 10+ million users
• 100,000+ requests per hour
• PCI-compliant environment

6. What are the network security threats?
Open Ports
DDOS
SQL Injection
XSS
CSRFPoodle
Heartbleed

7. Challenges of SSH
SSH tunnels
• Forward tunneling
• Reverse SSH tunneling
• Easy to circumvent firewall rules
Key management
• Where do you store them? Can you control storage?
• Rotation of keys?
• Federation? (Centrify, etc)

8. Did you know?

9. Immutable infrastructure possible?

10. What’s truly immutable infrastructure?

11. What’s practically immutable infrastructure?

12. What do we want?

13. Photo by Jurvetson (flickr)
AUTOMATE
EVERYTHING!

14. Key goals
• No humans in production
• Everything has to be automated
• No SSH back doors into production
• Development has to be: Easy, fast, secure. Pick three

15. Ask 2 questions Instead
How are we going to get changes into the pipeline?
How are we going to automatically get the data we need off the box?

16. What does our target environment need?

17. How are we going to automate?
AMI (image) baking!

18. The pipeline
AWS
CodeCommit
Amazon ECS
Build/test
Deploy
Redeploy to next
environments
Git clone

19. What is AWS CloudFormation?
CloudFormation
template
CloudFormation
stack AWS resources

20. What is AWS CloudFormation?

21. What goes in AWS CloudFormation?
• Amazon S3 buckets
• Amazon DynamoDB tables
• Amazon SQS
• Amazon RDS databases
• Amazon ElastiCache
instances
• AWS KMS keys
• IAM roles
• IAM policies
• Amazon CloudFront
• Amazon VPC
• Internet gateway
• Routes
• Route tables
• Network ACL
• Front-end router/ELB
• Internal ELB
• Auto Scaling group
and metrics

22. What is AWS CloudFormation?

23. How do we make it easier for developers?
{
"ServiceName": ”MyAwesomeService",
"DeploymentSystem": ”ECS",
"DeploymentType": "Python",
"Port": 8080,
"RootDir": ”helloworld”,
"APIGateway": "True"
}

24. How do we make it easier for developers?
{
"Resources": {
"KMS": [
{
"logical_id": "DefaultKey"
}
],
"S3": [
{
"logical_id": "StandardBucket"
}
],
"Dynamo": [
{
"logical_id": "table",
"hash": "hash",
"range": "range"
}
}

25. What does our target environment need?

26. Base instance configuration: cfn-init
{
"Resources": {
"MyInstance": {
"Type": "AWS::EC2::Instance",
"Metadata": {
"AWS::CloudFormation::Init": {
"config": {
"packages": {},
"groups": {},
"users": {},
"sources": {},
"files": {},
"commands": {},
"services": {}
}}}}}}

27. Implications on development

28. The initial reaction
So you’re telling me that
we are rolling a brand new
platform out to production,
with 100s of instances,
and we can’t log in to a
single one?

29. What does our target environment need?

30. App-specific instance configuration: AWS CodeDeploy

31. Developer view of AWS CodeDeploy

32. How to debug code deployments?

33. How do we configure the application?

35. The road to self-discovery – Step 1

36. The road to self-discovery – Step 2

37. The road to self-discovery – Step 3
Configuration properties
• Feature flags
• Thread pool sizing
• ListenPort
Secure configuration repository
• Consul
• Spring cloud config
• Custom solution
• DynamoDB
• Amazon S3

38. How about a developer’s config?

39. Challenges with instance bootstrapping?
• Dependency issues with package installation at runtime
• Potential vector for malicious code injection?
• Automatic scaling slower with a full bootstrap

40. Can we combine these layers?

42. What is Docker?

43. How to get started?
FROM ubuntu:trusty
EXPOSE 80
RUN apt-get update
RUN apt-get install -y python3-setuptools
RUN easy_install3 pip
RUN pip3 install flask
ADD . /home/root
CMD python3 /home/root/hello_world.py

44. How to get started?
FROM ubuntu:trusty
EXPOSE 80
RUN apt-get update
RUN apt-get install -y python3-setuptools
RUN easy_install3 pip
RUN pip3 install flask
ADD . /home/root
CMD python3 /home/root/hello_world.py

45. How about the external environment?

46. Implications on development – Environment
configuration
What do we typically need to know about the outside world?
• Database tables
• Amazon SQS queues
• Encryption keys
• Amazon S3 buckets
• Amazon SNS topics
• Amazon Kinesis streams
• Amazon ElastiCache endpoints

47. The road to self-discovery – Step 2 ( repeat )

48. The road to self-discovery – Step 3B
aws cloudformation list-stack-resources –stack-name receiptservice-prod-87287ASD0

49. • S3 buckets
• DynamoDB tables
• SQS
• RDS* databases
• KMS keys
What about credentials
IAM

50. What about after the application is up?

51. A GOOD day in production

52. A BAD day in production

53. Instances down?!
NO SSH!

54. Keep Calm
And Turn Debug On
Keep calm and turn debug on

55. Production monitoring – Keeping your cool
All logs are immediately shipped off of the box
• Logstash, ELK, Splunk, etc
• Writing directly to Amazon CloudWatch Logs and subscriptions
• http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Subs
criptions.html

56. Production monitoring – Keeping your cool
Proactive monitoring
• CloudWatch metrics
• Leveraging APM solutions such as NewRelic, AppDynamics, etc
• Advanced health checks
• SpringBoot ACTUATOR
– Health
– Metrics
– Service information
– Thread dumps
– Environment

57. Other implications on development
Instances must be ephemeral
Fits the microservices paradigm
• No application state written to disk
• Key for automatic scaling
• Cheap to manufacture ( CloudFormation templates )

58. What happens when….?
I REALLY need access to the disk for forensics, etc.?
• No change from existing best practice
• Snapshot volume and connect to forensics EC2 instance
I need to do a thread dump?
• Standardized logging on startup/shutdown sequences
Other Implications on development

59. Securing code pipelines
All changes are versioned
• All ability to deploy changes are managed through IAM roles
• AWS CloudTrail auditing
Source code is sanitized
• Clean package dependencies
• OWASP dependency check
Static analysis
• Parasoft, Fortify, Veracode, etc

60. Break glass in case of emergency?

61. Ask 2 questions Instead
How are we going to get changes into the pipeline?
How are we going to automatically get the data we need off the box?

63. How many times have we had to log in?
0
2 years

64. Thank you!

65. Remember to complete
your evaluations!