The document discusses best practices for architecting infrastructure on AWS including: 1) Using infrastructure as code with CloudFormation and version control for all changes. 2) Designing infrastructure to be automated through auto scaling groups and configuration management with no manual instance launches. 3) Implementing monitoring and alerting to detect issues and only use paging for critical problems.

1. AWS Loft: Behind the scenes with Cotap
Architecting for the Cloud:
Hoping for the best, prepared for the worst.

2. Infrastructure as Code

3. Infrastructure as Code
● Current state
● Past decisions
● Tracking the evolution

4. ● CloudFormation
● Design -> JSON
● Version Control!
Infrastructure as Code

5. Infrastructure as Code

6. Infrastructure as Code

7. Infrastructure as Code

10. Rule #1
All changes have to be under Version Control

11. Design for automation

12. Design for automation
● AutoScalingGroups
● Hardware: CloudFormation
● Software: Configuration management
● Cattle not Cats

13. Rule #2
No instances should be launched manually.

14. Monitoring & Alerting

15. Monitoring & Alerting
● Cost of
○ Interruptions
○ Waking somebody up
● Channels
● Self-healing infrastructure
● External monitoring
● Page only when critical

16. Monitoring & Alerting
Situation Channel Page
Disk full 60% Chat, Email ✗
Disk full 90% Chat, Email, PagerDuty ✓
Chef not running for > 30m Chat, Email ✗
Redis not running for > 3 x 5s Chat, Email, PagerDuty ✓
ElasticSearch N-1 Chat, Email ✗
ElasticSearch N-2 Chat, Email, PagerDuty ✓

17. Monitoring & Alerting
● Cost of
○ Interruptions
○ Waking somebody up
● Channels
● Self-healing infrastructure
● External monitoring
● Page only when critical

18. Platform to fail

19. Platform to fail
● Easy creation of temporary “Stacks”
● Branches can get their own hardware
● Clients can talk to a branch
● QA happens on Sandbox
● Exact copy of Production
● Scale up/down based on needs
● Different Region (us-east-1)

20. Platform to fail

21. Platform to fail
● Easy creation of temporary “Stacks”
● Branches can get their own hardware
● Clients can talk to a branch
● QA happens on Sandbox
● Exact copy of Production
● Scale up/down based on needs
● Different Region (us-east-1)

22. All changes have to go through Sandbox.
Rule #3

23. Rule #4
Production is just a more powerful Sandbox

24. Disaster Recovery

25. Disaster Recovery
● Multi-AZs
● Traffic routing
● Multi-Regions (S3 too)
● AutoScalingGroups Min:1 Max:1
● Off-site backups (VPN + Disks)
● RPO + RTO

26. Security

27. Security
● MFA
● Public key distribution
● Root key rotation
● Private/Public Subnets
● ACLs/Security Groups
● Update AMIs
● Trusted Advisor!

28. Security

29. Scaling

30. Scaling
● Preemptive
● Automatic
● Vertically
● Horizontally
● Bottlenecks

31. Scaling

32. Cost Control

33. Cost Control
● Tags
○ Role
○ Environment
● Cost explorer
● Threshold alerting
● Share monthly
● Export to CSV
● Right-Scale (ASG)

34. Cost Control

35. Cost Control

36. Cost Control
● Tags
○ Role
○ Environment
● Cost explorer
● Threshold alerting
● Share monthly
● Export to CSV
● Right-Scale (ASG)

37. 4 rules of 5 nines.
● All changes have to be under VC
● No instance should be launched manually
● All changes are deployed to Sandbox first
● Production is just a more powerful Sandbox

38. Questions?
t: @martincozzi
e: martin@cotap.com
engineering.cotap.com