Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

AWS Blackbelt NINJA Dojo

1,845 views

Published on

40, 1173 & 516. What do these numbers mean? Since inception AWS has introduced more than 40 major new services, released over 1173 new services and features, with 516 new features and services announced in 2014 alone. How you use the AWS platform last year may be very different to how you utilise it today to maximize innovation, outcomes and remaining competitive. In this advanced technical session an AWS Solution Architect will address technical requirements for successfully deploying and managing applications on the AWS platform, how solutions were potentially architected previously, both off-cloud and on-cloud, and some of the best practice recommendations on AWS today.

Speaker: Dean Samuels, Solutions Architect, Amazon Web Services

Published in: Technology

AWS Blackbelt NINJA Dojo

  1. 1. AWS Black Belt Ninja Dojo Dean Samuels, Solutions Architect Amazon Web Services
  2. 2. Business 101 Technical 201 Technical 301 Technical 401 Technical Session Grading
  3. 3. Your Application Stacks Availability Zone A Availability Zone B Private subnetPrivate subnet Public subnetPublic subnet Private subnetPrivate subnet CloudFront Glacier S3 DynamoDB Route 53 CloudWatch CloudFormation NAT Stacks for: Amazon SQS Auto Scaling groups AWS Region SNS
  4. 4. Your Application Stacks Availability Zone A Availability Zone B Private subnetPrivate subnet Public subnetPublic subnet Private subnetPrivate subnet CloudFront Glacier S3 DynamoDB Route 53 CloudWatch CloudFormation NAT Stacks for: VPC Amazon SQS Auto Scaling groups AWS Region SNS
  5. 5. Your Application Stacks Availability Zone A Availability Zone B Private subnetPrivate subnet Public subnetPublic subnet Private subnetPrivate subnet CloudFront Glacier S3 DynamoDB Route 53 CloudWatch CloudFormation NAT Stacks for: VPC Edge Services Amazon SQS Auto Scaling groups AWS Region SNS
  6. 6. Your Application Stacks Availability Zone A Availability Zone B Private subnetPrivate subnet Public subnetPublic subnet Private subnetPrivate subnet CloudFront Glacier S3 DynamoDB Route 53 CloudWatch CloudFormation NAT Stacks for: VPC Edge Services Amazon SQS Auto Scaling groups AWS Region SNS
  7. 7. Your Application Stacks Availability Zone A Availability Zone B Private subnetPrivate subnet Public subnetPublic subnet Private subnetPrivate subnet CloudFront Glacier S3 DynamoDB Route 53 CloudWatch CloudFormation NAT Stacks for: VPC Edge Services Datastores Amazon SQS Auto Scaling groups AWS Region SNS
  8. 8. Your Application Stacks Availability Zone A Availability Zone B Private subnetPrivate subnet Public subnetPublic subnet Private subnetPrivate subnet CloudFront Glacier S3 DynamoDB Route 53 CloudWatch CloudFormation NAT Stacks for: VPC Edge Services Datastores Amazon SQS Auto Scaling groups AWS Region SNS
  9. 9. Your Application Stacks Availability Zone A Availability Zone B Private subnetPrivate subnet Public subnetPublic subnet Private subnetPrivate subnet CloudFront Glacier S3 DynamoDB Route 53 CloudWatch CloudFormation NAT Stacks for: VPC Edge Services Datastores Applications Amazon SQS Auto Scaling groups AWS Region SNS
  10. 10. Your Application Stacks Availability Zone A Availability Zone B Private subnetPrivate subnet Public subnetPublic subnet Private subnetPrivate subnet CloudFront Glacier S3 DynamoDB Route 53 CloudWatch CloudFormation NAT Stacks for: VPC Edge Services Datastores Applications Presentation Amazon SQS Auto Scaling groups AWS Region SNS
  11. 11. Your Application Stacks Availability Zone A Availability Zone B Private subnetPrivate subnet Public subnetPublic subnet Private subnetPrivate subnet CloudFront Glacier S3 DynamoDB Route 53 CloudWatch CloudFormation NAT Stacks for: VPC Edge Services Datastores Applications Presentation Amazon SQS Auto Scaling groups AWS Region SNS
  12. 12. How can I optimise the performance of these AWS services
  13. 13. Your Application Stacks Availability Zone A Availability Zone B Private subnetPrivate subnet Public subnetPublic subnet Private subnetPrivate subnet CloudFront Glacier S3 DynamoDB Route 53 CloudWatch CloudFormation NAT Stacks for: VPC Edge Services Datastores Applications Presentation Amazon SQS Auto Scaling groups AWS Region SNS
  14. 14. Your Application Stacks Availability Zone A Availability Zone B Private subnetPrivate subnet Public subnetPublic subnet Private subnetPrivate subnet CloudFront Glacier S3 DynamoDB Route 53 CloudWatch CloudFormation NAT Stacks for: VPC Edge Services Datastores Applications Presentation Amazon SQS Auto Scaling groups AWS Region SNS
  15. 15. Amazon EBS – Larger & Faster Volumes
  16. 16. Amazon EBS – Larger & Faster Volumes GP2 1GB-16TB
  17. 17. Amazon EBS – Larger & Faster Volumes GP2 PIOPS/ io2 1GB-16TB 4GB-16TB
  18. 18. Amazon EBS – Larger & Faster Volumes GP2 PIOPS/ io2 MAG/STD 1GB-16TB 4GB-16TB 1GB-1TB
  19. 19. Amazon EBS – Larger & Faster Volumes GP2 PIOPS/ io2 MAG/STD 10,000 IOPS (<1TB – 3000 IOPS) 1GB-16TB 4GB-16TB 1GB-1TB
  20. 20. Amazon EBS – Larger & Faster Volumes GP2 PIOPS/ io2 MAG/STD 10,000 IOPS (<1TB – 3000 IOPS) 160MB/s (<1TB – 128MB/s) 1GB-16TB 4GB-16TB 1GB-1TB
  21. 21. Amazon EBS – Larger & Faster Volumes GP2 PIOPS/ io2 MAG/STD 10,000 IOPS (<1TB – 3000 IOPS) 160MB/s (<1TB – 128MB/s) 20,000 IOPS 1GB-16TB 4GB-16TB 1GB-1TB
  22. 22. Amazon EBS – Larger & Faster Volumes GP2 PIOPS/ io2 MAG/STD 10,000 IOPS (<1TB – 3000 IOPS) 160MB/s (<1TB – 128MB/s) 20,000 IOPS 320MB/s (<1TB – 128MB/s) 1GB-16TB 4GB-16TB 1GB-1TB
  23. 23. Amazon EBS – Larger & Faster Volumes GP2 PIOPS/ io2 MAG/STD 10,000 IOPS (<1TB – 3000 IOPS) 160MB/s (<1TB – 128MB/s) 20,000 IOPS 320MB/s (<1TB – 128MB/s) ~100 IOPS 1GB-16TB 4GB-16TB 1GB-1TB
  24. 24. Amazon EBS – Larger & Faster Volumes GP2 PIOPS/ io2 MAG/STD 10,000 IOPS (<1TB – 3000 IOPS) 160MB/s (<1TB – 128MB/s) 20,000 IOPS 320MB/s (<1TB – 128MB/s) ~100 IOPS 50-90MB/s 1GB-16TB 4GB-16TB 1GB-1TB
  25. 25. Amazon EBS – Larger & Faster Volumes GP2 PIOPS/ io2 MAG/STD 10,000 IOPS (<1TB – 3000 IOPS) 160MB/s (<1TB – 128MB/s) 20,000 IOPS 320MB/s (<1TB – 128MB/s) ~100 IOPS 50-90MB/s 1GB-16TB 4GB-16TB 1GB-1TB EC2 48,000 IOPS @ 16K IO 800MB/s^ ^Amazon EC2 *.8xlarge instances support 10Gb/s network
  26. 26. Amazon EBS – Larger & Faster Volumes GP2 PIOPS/ io2 MAG/STD 10,000 IOPS (<1TB – 3000 IOPS) 160MB/s (<1TB – 128MB/s) 20,000 IOPS 320MB/s (<1TB – 128MB/s) ~100 IOPS 50-90MB/s 1GB-16TB 4GB-16TB 1GB-1TB EC2 48,000 IOPS @ 16K IO 800MB/s^ EBS-Optimized @ 500Mb, 1Gb, 2Gb^ ^Amazon EC2 *.8xlarge instances support 10Gb/s network 1-2ms
  27. 27. Amazon EBS – Larger & Faster Volumes GP2 PIOPS/ io2 MAG/STD 10,000 IOPS (<1TB – 3000 IOPS) 160MB/s (<1TB – 128MB/s) 20,000 IOPS 320MB/s (<1TB – 128MB/s) ~100 IOPS 50-90MB/s 1GB-16TB 4GB-16TB 1GB-1TB EC2 48,000 IOPS @ 16K IO 800MB/s^ EBS-Optimized @ 500Mb, 1Gb, 2Gb^ ^Amazon EC2 *.8xlarge instances support 10Gb/s network 1-2ms 48,000 IOPS @ 16K IO 800MB/s^ EC2
  28. 28. Amazon EBS – Larger & Faster Volumes GP2 PIOPS/ io2 MAG/STD 10,000 IOPS (<1TB – 3000 IOPS) 160MB/s (<1TB – 128MB/s) 20,000 IOPS 320MB/s (<1TB – 128MB/s) ~100 IOPS 50-90MB/s 1GB-16TB 4GB-16TB 1GB-1TB EC2 48,000 IOPS @ 16K IO 800MB/s^ EBS-Optimized @ 500Mb, 1Gb, 2Gb^ EBS-Optimized @ 500Mb, 1Gb, 2Gb^ ^Amazon EC2 *.8xlarge instances support 10Gb/s network 1-2ms 1-2ms 48,000 IOPS @ 16K IO 800MB/s^ EC2
  29. 29. Amazon EBS – Larger & Faster Volumes GP2 PIOPS/ io2 MAG/STD 10,000 IOPS (<1TB – 3000 IOPS) 160MB/s (<1TB – 128MB/s) 20,000 IOPS 320MB/s (<1TB – 128MB/s) ~100 IOPS 50-90MB/s 1GB-16TB 4GB-16TB 1GB-1TB EC2 48,000 IOPS @ 16K IO 800MB/s^ EBS-Optimized @ 500Mb, 1Gb, 2Gb^ EBS-Optimized @ 500Mb, 1Gb, 2Gb^ ^Amazon EC2 *.8xlarge instances support 10Gb/s network 1-2ms 1-2ms ~2-40ms 48,000 IOPS @ 16K IO 800MB/s^ EC2 48,000 IOPS @ 16K IO 800MB/s^ EC2
  30. 30. Amazon EBS – Larger & Faster Volumes GP2 PIOPS/ io2 MAG/STD 10,000 IOPS (<1TB – 3000 IOPS) 160MB/s (<1TB – 128MB/s) 20,000 IOPS 320MB/s (<1TB – 128MB/s) ~100 IOPS 50-90MB/s 1GB-16TB 4GB-16TB 1GB-1TB EC2 48,000 IOPS @ 16K IO 800MB/s^ EBS-Optimized @ 500Mb, 1Gb, 2Gb^ EBS-Optimized @ 500Mb, 1Gb, 2Gb^ ^Amazon EC2 *.8xlarge instances support 10Gb/s network 1-2ms 1-2ms ~2-40ms 48,000 IOPS @ 16K IO 800MB/s^ EC2 48,000 IOPS @ 16K IO 800MB/s^ EC2 Optimal queue depth to achieve lower latency and highest IOPS is ~1 QD per 200 IOPS
  31. 31. Amazon EBS
  32. 32. Amazon EBS Cost Optimisation
  33. 33. Amazon EBS Cost Optimisation • 1TB PIOPS volume with 4K IOPS – $429.32* per month per volume _________________________________ *Pricing for AWS Sydney region – ap-southeast-2
  34. 34. Amazon EBS Cost Optimisation • 1TB PIOPS volume with 4K IOPS – $429.32* per month per volume _________________________________ • GP2 1TB volume with 3000 IOPS – $122.88* *Pricing for AWS Sydney region – ap-southeast-2
  35. 35. Amazon EBS Cost Optimisation • 1TB PIOPS volume with 4K IOPS – $429.32* per month per volume _________________________________ • GP2 1TB volume with 3000 IOPS – $122.88* • GP2 2 x 500GB volumes at 3K, burst to 6K – $122.88* ~70% Cost Savings. 50% more peak I/O with *Pricing for AWS Sydney region – ap-southeast-2
  36. 36. Amazon EBS Cost Optimisation • 1TB PIOPS volume with 4K IOPS – $429.32* per month per volume _________________________________ • GP2 1TB volume with 3000 IOPS – $122.88* • GP2 2 x 500GB volumes at 3K, burst to 6K – $122.88* ~70% Cost Savings. 50% more peak I/O with General Purpose (SSD) *Pricing for AWS Sydney region – ap-southeast-2
  37. 37. Amazon EBS Cost Optimisation • 1TB PIOPS volume with 4K IOPS – $429.32* per month per volume _________________________________ • GP2 1TB volume with 3000 IOPS – $122.88* • GP2 2 x 500GB volumes at 3K, burst to 6K – $122.88* ~70% Cost Savings. 50% more peak I/O with General Purpose (SSD) Management Optimisation *Pricing for AWS Sydney region – ap-southeast-2
  38. 38. Amazon EBS Cost Optimisation • 1TB PIOPS volume with 4K IOPS – $429.32* per month per volume _________________________________ • GP2 1TB volume with 3000 IOPS – $122.88* • GP2 2 x 500GB volumes at 3K, burst to 6K – $122.88* ~70% Cost Savings. 50% more peak I/O with General Purpose (SSD) Management Optimisation • Leverage tags to add metadata to snapshots – Application stack – Instance Id – Volume Id – Version – Type (daily, weekly) *Pricing for AWS Sydney region – ap-southeast-2 Use together with new AMI creation date
  39. 39. Amazon EC2 • Next Generation Instance Types – C4 & C3: Compute Optimized – R3: Memory Optimized – I2: High IO – D2: Dense-storage
  40. 40. Amazon EC2 • Next Generation Instance Types – C4 & C3: Compute Optimized – R3: Memory Optimized – I2: High IO – D2: Dense-storage • Hardware Assisted Virtualization (HVM)
  41. 41. Amazon EC2 • Next Generation Instance Types – C4 & C3: Compute Optimized – R3: Memory Optimized – I2: High IO – D2: Dense-storage • Hardware Assisted Virtualization (HVM) • Enhanced Networking
  42. 42. Virtualization layer eth0 eth1 Instance Virtual NICs Physical NIC VIF Amazon EC2 – Enhanced Networking
  43. 43. Virtualization layer eth0 eth1 Instance Virtual NICs Physical NIC Virtualization layer eth0 Instance Physical NIC VF Driver eth1 VF VIF SR-IOV Amazon EC2 – Enhanced Networking
  44. 44. Virtualization layer eth0 eth1 Instance Virtual NICs Physical NIC Virtualization layer eth0 Instance Physical NIC VF Driver eth1 VF VIF SR-IOV Amazon EC2 – Enhanced Networking Instance 1 Instance 2 ........
  45. 45. Demo
 EC2 & EBS Optimisation
  46. 46. Amazon S3 – Distributing Key Names <my_bucket>/2013_11_13-164533125.jpg <my_bucket>/2013_11_13-164533126.jpg <my_bucket>/2013_11_13-164533127.jpg <my_bucket>/2013_11_13-164533128.jpg <my_bucket>/2013_11_12-164533129.jpg <my_bucket>/2013_11_12-164533130.jpg <my_bucket>/2013_11_12-164533131.jpg <my_bucket>/2013_11_12-164533132.jpg <my_bucket>/2013_11_11-164533133.jpg
  47. 47. Amazon S3 – Distributing Key Names <my_bucket>/2013_11_13-164533125.jpg <my_bucket>/2013_11_13-164533126.jpg <my_bucket>/2013_11_13-164533127.jpg <my_bucket>/2013_11_13-164533128.jpg <my_bucket>/2013_11_12-164533129.jpg <my_bucket>/2013_11_12-164533130.jpg <my_bucket>/2013_11_12-164533131.jpg <my_bucket>/2013_11_12-164533132.jpg <my_bucket>/2013_11_11-164533133.jpg
  48. 48. Amazon S3 – Distributing Key Names <my_bucket>/2013_11_13-164533125.jpg <my_bucket>/2013_11_13-164533126.jpg <my_bucket>/2013_11_13-164533127.jpg <my_bucket>/2013_11_13-164533128.jpg <my_bucket>/2013_11_12-164533129.jpg <my_bucket>/2013_11_12-164533130.jpg <my_bucket>/2013_11_12-164533131.jpg <my_bucket>/2013_11_12-164533132.jpg <my_bucket>/2013_11_11-164533133.jpg Don’t Do This!
  49. 49. Amazon S3 – Distributing Key Names <my_bucket>/2013_11_13-164533125.jpg <my_bucket>/2013_11_13-164533126.jpg <my_bucket>/2013_11_13-164533127.jpg <my_bucket>/2013_11_13-164533128.jpg <my_bucket>/2013_11_12-164533129.jpg <my_bucket>/2013_11_12-164533130.jpg <my_bucket>/2013_11_12-164533131.jpg <my_bucket>/2013_11_12-164533132.jpg <my_bucket>/2013_11_11-164533133.jpg Don’t Do This! You end up with this
  50. 50. Amazon S3 – Distributing Key Names <my_bucket>/2013_11_13-164533125.jpg <my_bucket>/2013_11_13-164533126.jpg <my_bucket>/2013_11_13-164533127.jpg <my_bucket>/2013_11_13-164533128.jpg <my_bucket>/2013_11_12-164533129.jpg <my_bucket>/2013_11_12-164533130.jpg <my_bucket>/2013_11_12-164533131.jpg <my_bucket>/2013_11_12-164533132.jpg <my_bucket>/2013_11_11-164533133.jpg 1 2 N 1 2 N Don’t Do This! You end up with this
  51. 51. Amazon S3 – Distributing Key Names <my_bucket>/2013_11_13-164533125.jpg <my_bucket>/2013_11_13-164533126.jpg <my_bucket>/2013_11_13-164533127.jpg <my_bucket>/2013_11_13-164533128.jpg <my_bucket>/2013_11_12-164533129.jpg <my_bucket>/2013_11_12-164533130.jpg <my_bucket>/2013_11_12-164533131.jpg <my_bucket>/2013_11_12-164533132.jpg <my_bucket>/2013_11_11-164533133.jpg 1 2 N 1 2 N Partition Partition Partition Partition Don’t Do This! You end up with this
  52. 52. Amazon S3 – Distributing Key Names <my_bucket>/2013_11_13-164533125.jpg <my_bucket>/2013_11_13-164533126.jpg <my_bucket>/2013_11_13-164533127.jpg <my_bucket>/2013_11_13-164533128.jpg <my_bucket>/2013_11_12-164533129.jpg <my_bucket>/2013_11_12-164533130.jpg <my_bucket>/2013_11_12-164533131.jpg <my_bucket>/2013_11_12-164533132.jpg <my_bucket>/2013_11_11-164533133.jpg 1 2 N 1 2 N Partition Partition Partition Partition Don’t Do This! You end up with this
  53. 53. Amazon S3 – Distributing Key Names <my_bucket>/2013_11_13-164533125.jpg <my_bucket>/2013_11_13-164533126.jpg <my_bucket>/2013_11_13-164533127.jpg <my_bucket>/2013_11_13-164533128.jpg <my_bucket>/2013_11_12-164533129.jpg <my_bucket>/2013_11_12-164533130.jpg <my_bucket>/2013_11_12-164533131.jpg <my_bucket>/2013_11_12-164533132.jpg <my_bucket>/2013_11_11-164533133.jpg 1 2 N 1 2 N Partition Partition Partition Partition Don’t Do This! You end up with this
  54. 54. Amazon S3 – Distributing Key Names <my_bucket>/2013_11_13-164533125.jpg <my_bucket>/2013_11_13-164533126.jpg <my_bucket>/2013_11_13-164533127.jpg <my_bucket>/2013_11_13-164533128.jpg <my_bucket>/2013_11_12-164533129.jpg <my_bucket>/2013_11_12-164533130.jpg <my_bucket>/2013_11_12-164533131.jpg <my_bucket>/2013_11_12-164533132.jpg <my_bucket>/2013_11_11-164533133.jpg 1 2 N 1 2 N Partition Partition Partition Partition If you want a bucket capable of routinely exceeding 100 TPS Don’t Do This! You end up with this
  55. 55. Amazon S3 – Distributing Key Names <my_bucket>/2013_11_13-164533125.jpg <my_bucket>/2013_11_13-164533126.jpg <my_bucket>/2013_11_13-164533127.jpg <my_bucket>/2013_11_13-164533128.jpg <my_bucket>/2013_11_12-164533129.jpg <my_bucket>/2013_11_12-164533130.jpg <my_bucket>/2013_11_12-164533131.jpg <my_bucket>/2013_11_12-164533132.jpg <my_bucket>/2013_11_11-164533133.jpg 1 2 N 1 2 N Partition Partition Partition Partition If you want a bucket capable of routinely exceeding 100 TPS Note: 100 TPS is A LOT! Don’t Do This! You end up with this
  56. 56. Amazon S3 – Distributing Key Names <my_bucket>/2013_11_13-164533125.jpg <my_bucket>/2013_11_13-164533126.jpg <my_bucket>/2013_11_13-164533127.jpg <my_bucket>/2013_11_13-164533128.jpg <my_bucket>/2013_11_12-164533129.jpg <my_bucket>/2013_11_12-164533130.jpg <my_bucket>/2013_11_12-164533131.jpg <my_bucket>/2013_11_12-164533132.jpg <my_bucket>/2013_11_11-164533133.jpg <my_bucket>/521335461-2013_11_13.jpg <my_bucket>/465330151-2013_11_13.jpg <my_bucket>/987331160-2013_11_13.jpg <my_bucket>/465765461-2013_11_13.jpg <my_bucket>/125631151-2013_11_13.jpg <my_bucket>/934563160-2013_11_13.jpg <my_bucket>/532132341-2013_11_13.jpg <my_bucket>/565437681-2013_11_13.jpg <my_bucket>/234567460-2013_11_13.jpg 1 2 N 1 2 N Partition Partition Partition Partition If you want a bucket capable of routinely exceeding 100 TPS Note: 100 TPS is A LOT! Don’t Do This! You end up with this
  57. 57. Amazon S3 – Distributing Key Names <my_bucket>/2013_11_13-164533125.jpg <my_bucket>/2013_11_13-164533126.jpg <my_bucket>/2013_11_13-164533127.jpg <my_bucket>/2013_11_13-164533128.jpg <my_bucket>/2013_11_12-164533129.jpg <my_bucket>/2013_11_12-164533130.jpg <my_bucket>/2013_11_12-164533131.jpg <my_bucket>/2013_11_12-164533132.jpg <my_bucket>/2013_11_11-164533133.jpg <my_bucket>/521335461-2013_11_13.jpg <my_bucket>/465330151-2013_11_13.jpg <my_bucket>/987331160-2013_11_13.jpg <my_bucket>/465765461-2013_11_13.jpg <my_bucket>/125631151-2013_11_13.jpg <my_bucket>/934563160-2013_11_13.jpg <my_bucket>/532132341-2013_11_13.jpg <my_bucket>/565437681-2013_11_13.jpg <my_bucket>/234567460-2013_11_13.jpg 1 2 N 1 2 N Partition Partition Partition Partition If you want a bucket capable of routinely exceeding 100 TPS Note: 100 TPS is A LOT! Don’t Do This! You end up with this
  58. 58. Amazon S3 – Distributing Key Names <my_bucket>/2013_11_13-164533125.jpg <my_bucket>/2013_11_13-164533126.jpg <my_bucket>/2013_11_13-164533127.jpg <my_bucket>/2013_11_13-164533128.jpg <my_bucket>/2013_11_12-164533129.jpg <my_bucket>/2013_11_12-164533130.jpg <my_bucket>/2013_11_12-164533131.jpg <my_bucket>/2013_11_12-164533132.jpg <my_bucket>/2013_11_11-164533133.jpg <my_bucket>/521335461-2013_11_13.jpg <my_bucket>/465330151-2013_11_13.jpg <my_bucket>/987331160-2013_11_13.jpg <my_bucket>/465765461-2013_11_13.jpg <my_bucket>/125631151-2013_11_13.jpg <my_bucket>/934563160-2013_11_13.jpg <my_bucket>/532132341-2013_11_13.jpg <my_bucket>/565437681-2013_11_13.jpg <my_bucket>/234567460-2013_11_13.jpg 1 2 N 1 2 N Partition Partition Partition Partition If you want a bucket capable of routinely exceeding 100 TPS Note: 100 TPS is A LOT! Don’t Do This! You end up with this Do this…
  59. 59. Amazon S3 – Distributing Key Names <my_bucket>/2013_11_13-164533125.jpg <my_bucket>/2013_11_13-164533126.jpg <my_bucket>/2013_11_13-164533127.jpg <my_bucket>/2013_11_13-164533128.jpg <my_bucket>/2013_11_12-164533129.jpg <my_bucket>/2013_11_12-164533130.jpg <my_bucket>/2013_11_12-164533131.jpg <my_bucket>/2013_11_12-164533132.jpg <my_bucket>/2013_11_11-164533133.jpg <my_bucket>/521335461-2013_11_13.jpg <my_bucket>/465330151-2013_11_13.jpg <my_bucket>/987331160-2013_11_13.jpg <my_bucket>/465765461-2013_11_13.jpg <my_bucket>/125631151-2013_11_13.jpg <my_bucket>/934563160-2013_11_13.jpg <my_bucket>/532132341-2013_11_13.jpg <my_bucket>/565437681-2013_11_13.jpg <my_bucket>/234567460-2013_11_13.jpg 1 2 N 1 2 N Partition Partition Partition Partition 1 2 N 1 2 N Partition Partition Partition Partition If you want a bucket capable of routinely exceeding 100 TPS Note: 100 TPS is A LOT! Don’t Do This! You end up with this Do this… You end up with this
  60. 60. Amazon S3 – Distributing Key Names <my_bucket>/2013_11_13-164533125.jpg <my_bucket>/2013_11_13-164533126.jpg <my_bucket>/2013_11_13-164533127.jpg <my_bucket>/2013_11_13-164533128.jpg <my_bucket>/2013_11_12-164533129.jpg <my_bucket>/2013_11_12-164533130.jpg <my_bucket>/2013_11_12-164533131.jpg <my_bucket>/2013_11_12-164533132.jpg <my_bucket>/2013_11_11-164533133.jpg <my_bucket>/521335461-2013_11_13.jpg <my_bucket>/465330151-2013_11_13.jpg <my_bucket>/987331160-2013_11_13.jpg <my_bucket>/465765461-2013_11_13.jpg <my_bucket>/125631151-2013_11_13.jpg <my_bucket>/934563160-2013_11_13.jpg <my_bucket>/532132341-2013_11_13.jpg <my_bucket>/565437681-2013_11_13.jpg <my_bucket>/234567460-2013_11_13.jpg 1 2 N 1 2 N Partition Partition Partition Partition 1 2 N 1 2 N Partition Partition Partition Partition If you want a bucket capable of routinely exceeding 100 TPS Note: 100 TPS is A LOT! Don’t Do This! You end up with this Do this… You end up with this
  61. 61. Amazon S3 – Distributing Key Names <my_bucket>/2013_11_13-164533125.jpg <my_bucket>/2013_11_13-164533126.jpg <my_bucket>/2013_11_13-164533127.jpg <my_bucket>/2013_11_13-164533128.jpg <my_bucket>/2013_11_12-164533129.jpg <my_bucket>/2013_11_12-164533130.jpg <my_bucket>/2013_11_12-164533131.jpg <my_bucket>/2013_11_12-164533132.jpg <my_bucket>/2013_11_11-164533133.jpg 1 2 N 1 2 N Partition Partition Partition Partition 1 2 N 1 2 N Partition Partition Partition Partition If you want a bucket capable of routinely exceeding 100 TPS Note: 100 TPS is A LOT! Don’t Do This! You end up with this Do this… You end up with this
  62. 62. Amazon S3 – Distributing Key Names <my_bucket>/2013_11_13-164533125.jpg <my_bucket>/2013_11_13-164533126.jpg <my_bucket>/2013_11_13-164533127.jpg <my_bucket>/2013_11_13-164533128.jpg <my_bucket>/2013_11_12-164533129.jpg <my_bucket>/2013_11_12-164533130.jpg <my_bucket>/2013_11_12-164533131.jpg <my_bucket>/2013_11_12-164533132.jpg <my_bucket>/2013_11_11-164533133.jpg 1 2 N 1 2 N Partition Partition Partition Partition 1 2 N 1 2 N Partition Partition Partition Partition If you want a bucket capable of routinely exceeding 100 TPS Note: 100 TPS is A LOT! Don’t Do This! You end up with this Do this… You end up with this <my_bucket>/images/521335461-2013_11_13.jpg <my_bucket>/images/465330151-2013_11_13.jpg <my_bucket>/images/987331160-2013_11_13.jpg <my_bucket>/movies/465765461-2013_11_13.jpg <my_bucket>/movies/125631151-2013_11_13.jpg <my_bucket>/thumbs-small/934563160-2013_11_13.jpg <my_bucket>/thumbs-small/532132341-2013_11_13.jpg <my_bucket>/thumbs-small/565437681-2013_11_13.jpg <my_bucket>/thumbs-small/234567460-2013_11_13.jpg
  63. 63. Amazon S3 – Distributing Key Names <my_bucket>/2013_11_13-164533125.jpg <my_bucket>/2013_11_13-164533126.jpg <my_bucket>/2013_11_13-164533127.jpg <my_bucket>/2013_11_13-164533128.jpg <my_bucket>/2013_11_12-164533129.jpg <my_bucket>/2013_11_12-164533130.jpg <my_bucket>/2013_11_12-164533131.jpg <my_bucket>/2013_11_12-164533132.jpg <my_bucket>/2013_11_11-164533133.jpg 1 2 N 1 2 N Partition Partition Partition Partition 1 2 N 1 2 N Partition Partition Partition Partition If you want a bucket capable of routinely exceeding 100 TPS Note: 100 TPS is A LOT! Don’t Do This! You end up with this Do this… You end up with this <my_bucket>/images/521335461-2013_11_13.jpg <my_bucket>/images/465330151-2013_11_13.jpg <my_bucket>/images/987331160-2013_11_13.jpg <my_bucket>/movies/465765461-2013_11_13.jpg <my_bucket>/movies/125631151-2013_11_13.jpg <my_bucket>/thumbs-small/934563160-2013_11_13.jpg <my_bucket>/thumbs-small/532132341-2013_11_13.jpg <my_bucket>/thumbs-small/565437681-2013_11_13.jpg <my_bucket>/thumbs-small/234567460-2013_11_13.jpg This is also ok
  64. 64. Amazon S3 – Secondary Lists
 Restrict Use of S3 LIST DynamoDB RDS CloudSearch EC2 S3 ObjectCreated Notification Lambda SQS Workers
  65. 65. Amazon S3 – Secondary Lists
 Restrict Use of S3 LIST DynamoDB RDS CloudSearch EC2 S3 ObjectCreated Notification Lambda SQS Workers
  66. 66. Amazon S3 – Secondary Lists
 Restrict Use of S3 LIST DynamoDB RDS CloudSearch EC2 S3 ObjectCreated Notification Lambda SQS Workers
  67. 67. Demo
 S3 Optimisation
  68. 68. How can I simplify encryption for data in transit and data at rest?
  69. 69. Your Application Stacks Availability Zone A Availability Zone B Private subnetPrivate subnet Public subnetPublic subnet Private subnetPrivate subnet CloudFront Glacier S3 DynamoDB Route 53 CloudWatch CloudFormation NAT Stacks for: VPC Edge Services Datastores Applications Presentation Amazon SQS Auto Scaling groups AWS Region SNS
  70. 70. Your Application Stacks Availability Zone A Availability Zone B Private subnetPrivate subnet Public subnetPublic subnet Private subnetPrivate subnet CloudFront Glacier S3 DynamoDB Route 53 CloudWatch CloudFormation NAT Stacks for: VPC Edge Services Datastores Applications Presentation Amazon SQS Auto Scaling groups AWS Region SNS Elastic Load Balancer with SSL Termination (Announced 2010)
  71. 71. Your Application Stacks Availability Zone A Availability Zone B Private subnetPrivate subnet Public subnetPublic subnet Private subnetPrivate subnet CloudFront Glacier S3 DynamoDB Route 53 CloudWatch CloudFormation NAT Stacks for: VPC Edge Services Datastores Applications Presentation Amazon SQS Auto Scaling groups AWS Region SNS Elastic Load Balancer with SSL Termination (Announced 2010) CloudFront with HTTPS Access With Custom Domain Names (Announced 2013)
  72. 72. Your Application Stacks Availability Zone A Availability Zone B Private subnetPrivate subnet Public subnetPublic subnet Private subnetPrivate subnet CloudFront Glacier S3 DynamoDB Route 53 CloudWatch CloudFormation NAT Stacks for: VPC Edge Services Datastores Applications Presentation Amazon SQS Auto Scaling groups AWS Region SNS Elastic Load Balancer with SSL Termination (Announced 2010) CloudFront with HTTPS Access With Custom Domain Names (Announced 2013) RDS with SSL (MySQL - 2010) (SQL Server – 2012) (Oracle/NNE – 2013) (PostgreSQL – 2013)
  73. 73. Your Application Stacks Availability Zone A Availability Zone B Private subnetPrivate subnet Public subnetPublic subnet Private subnetPrivate subnet CloudFront Glacier S3 DynamoDB Route 53 CloudWatch CloudFormation NAT Stacks for: VPC Edge Services Datastores Applications Presentation Amazon SQS Auto Scaling groups AWS Region SNS Elastic Load Balancer with SSL Termination (Announced 2010) CloudFront with HTTPS Access With Custom Domain Names (Announced 2013) RDS with SSL (MySQL - 2010) (SQL Server – 2012) (Oracle/NNE – 2013) (PostgreSQL – 2013)
  74. 74. Simplifying encryption in AWS – Today Amazon S3 Object Amazon EBS Volume Amazon RDS
 or Redshift Custom
 Application AWS KMS Store Data with Envelope Encryption Client Application Announced 2014
  75. 75. Simplifying encryption in AWS – Today Amazon S3 Object Amazon EBS Volume Amazon RDS
 or Redshift Custom
 Application AWS KMS 1) User creates Customer Master Keys (CMK) Store Data with Envelope Encryption Client Application Announced 2014
  76. 76. Simplifying encryption in AWS – Today Amazon S3 Object Amazon EBS Volume Amazon RDS
 or Redshift Custom
 Application AWS KMS 2) User associates resource with CMK Store Data with Envelope Encryption Client Application Announced 2014
  77. 77. Simplifying encryption in AWS – Today Amazon S3 Object Amazon EBS Volume Amazon RDS
 or Redshift Custom
 Application AWS KMS Obj 3) Request to store data & context for encryption Data Data Data Requests Store Data with Envelope Encryption Client Application Announced 2014
  78. 78. Simplifying encryption in AWS – Today Amazon S3 Object Amazon EBS Volume Amazon RDS
 or Redshift Custom
 Application AWS KMS Obj Data Data Data 4) Service requests encryption key with context Store Data with Envelope Encryption Client Application Announced 2014
  79. 79. Simplifying encryption in AWS – Today Amazon S3 Object Amazon EBS Volume Amazon RDS
 or Redshift Custom
 Application AWS KMS Obj Data Data Data 5) AWS KMS returns an encryption (data) key + an encrypted version of the key + + + +Store Data with Envelope Encryption Client Application Announced 2014
  80. 80. Simplifying encryption in AWS – Today Amazon S3 Object Amazon EBS Volume Amazon RDS
 or Redshift Custom
 Application AWS KMS 6) Service encrypts the data with the encryption key then deletes the key from memory Store Data with Envelope Encryption Client Application Announced 2014
  81. 81. Simplifying encryption in AWS – Today Amazon S3 Object Amazon EBS Volume Amazon RDS
 or Redshift Custom
 Application AWS KMS 7) Service stores the data along with the encrypted key Store Data with Envelope Encryption Client Application Announced 2014
  82. 82. Simplifying encryption in AWS – Today Amazon S3 Object Amazon EBS Volume Amazon RDS
 or Redshift Custom
 Application AWS KMS Client Application Retrieve Data with Envelope Encryption Announced 2014
  83. 83. Simplifying encryption in AWS – Today Amazon S3 Object Amazon EBS Volume Amazon RDS
 or Redshift Custom
 Application AWS KMS Client Application Request Request Request Request 1) Request to retrieve data Retrieve Data with Envelope Encryption Announced 2014
  84. 84. Simplifying encryption in AWS – Today Amazon S3 Object Amazon EBS Volume Amazon RDS
 or Redshift Custom
 Application AWS KMS Client Application Request Request Request Request 2) Service retrieves the encrypted data & encrypted key. Retrieve Data with Envelope Encryption Announced 2014
  85. 85. Simplifying encryption in AWS – Today Amazon S3 Object Amazon EBS Volume Amazon RDS
 or Redshift Custom
 Application AWS KMS Client Application 3) Service sends the encrypted key and the UserID to KMS. Retrieve Data with Envelope Encryption Announced 2014
  86. 86. Simplifying encryption in AWS – Today Amazon S3 Object Amazon EBS Volume Amazon RDS
 or Redshift Custom
 Application AWS KMS Client Application 4) AWS KMS unencrypts the encryption key and returns the key to the service Retrieve Data with Envelope Encryption Announced 2014
  87. 87. Simplifying encryption in AWS – Today Amazon S3 Object Amazon EBS Volume Amazon RDS
 or Redshift Custom
 Application AWS KMS Client Application 5) Service decrypts the data with the encryption key, then deletes the key from memory Data Data DataObj Retrieve Data with Envelope Encryption Announced 2014
  88. 88. 6) Service returns the data to the user Simplifying encryption in AWS – Today Amazon S3 Object Amazon EBS Volume Amazon RDS
 or Redshift Custom
 Application AWS KMS Client Application Data Data Data Obj Retrieve Data with Envelope Encryption Announced 2014
  89. 89. Demo
 Integrating KMS
  90. 90. I’ve hit some obstacles with my VPC in terms of integration and performance, what are some of my options
  91. 91. Your Application Stacks Availability Zone A Availability Zone B Private subnetPrivate subnet Public subnetPublic subnet Private subnetPrivate subnet CloudFront Glacier S3 DynamoDB Route 53 CloudWatch CloudFormation NAT Stacks for: VPC Edge Services Datastores Applications Presentation Amazon SQS Auto Scaling groups AWS Region SNS
  92. 92. Your Application Stacks Availability Zone A Availability Zone B Private subnetPrivate subnet Public subnetPublic subnet Private subnetPrivate subnet CloudFront Glacier S3 DynamoDB Route 53 CloudWatch CloudFormation NAT Stacks for: VPC Edge Services Datastores Applications Presentation Amazon SQS Auto Scaling groups AWS Region SNS This is a bottleneck & SPOF!
  93. 93. Your Application Stacks Availability Zone A Availability Zone B Private subnetPrivate subnet Public subnetPublic subnet Private subnetPrivate subnet CloudFront Glacier S3 DynamoDB Route 53 CloudWatch CloudFormation NAT Stacks for: VPC Edge Services Datastores Applications Presentation Amazon SQS Auto Scaling groups AWS Region SNS This is a bottleneck & SPOF! These are bandwidth- intensive for Internet egress
  94. 94. Your Application Stacks Availability Zone A Availability Zone B Private subnetPrivate subnet Public subnetPublic subnet Private subnetPrivate subnet CloudFront Glacier S3 DynamoDB Route 53 CloudWatch CloudFormation NAT Stacks for: VPC Edge Services Datastores Applications Presentation Amazon SQS Auto Scaling groups AWS Region SNS This is a bottleneck & SPOF! These are bandwidth- intensive for Internet egress Applications with legacy network reqs
  95. 95. 10.4.0.0/16 10.0.0.0/16 172.16.0.0/16 192.168.0.0/16 172.17.0.0/16 10.1.0.0/16 10.2.0.0/1610.3.0.0/16 company data center 10.10.0.0/16 VPC Peering
  96. 96. 10.4.0.0/16 10.0.0.0/16 172.16.0.0/16 192.168.0.0/16 172.17.0.0/16 10.1.0.0/16 10.2.0.0/1610.3.0.0/16 company data center 10.10.0.0/16 VPC Peering
  97. 97. 10.4.0.0/16 10.0.0.0/16 172.16.0.0/16 192.168.0.0/16 172.17.0.0/16 10.1.0.0/16 10.2.0.0/1610.3.0.0/16 company data center 10.10.0.0/16 VPC Peering
  98. 98. 10.1.0.0/16 10.0.0.0/16 10.0.0.0/16 Taking VPC Peering to the next Level
  99. 99. 10.1.0.0/16 10.0.0.0/16 10.0.0.0/16 Taking VPC Peering to the next Level
  100. 100. 10.1.0.0/16 10.0.0.0/16 10.0.0.0/16 ✔ Taking VPC Peering to the next Level
  101. 101. 10.1.0.0/16 10.0.0.0/16 10.0.0.0/16 ✔ Taking VPC Peering to the next Level Overlapping IP is not a dead end
  102. 102. 10.0.0.0/16 10.0.0.0/16 10.1.0.0/16 A B C Taking VPC Peering to the next Level PCX-1 PCX-2
  103. 103. 10.0.0.0/16 10.0.0.0/16 Subnet 1 10.1.1.0/24 Subnet 2 10.1.2.0/24 10.1.0.0/16 A B C Taking VPC Peering to the next Level PCX-1 PCX-2
  104. 104. 10.0.0.0/16 10.0.0.0/16 Subnet 1 10.1.1.0/24 Subnet 2 10.1.2.0/24 10.1.0.0/16Route Table Subnet 1 Destination Target 10.1.0.0/16 local 10.0.0.0/16 PCX-1 A B C Taking VPC Peering to the next Level PCX-1 PCX-2
  105. 105. 10.0.0.0/16 10.0.0.0/16 Subnet 1 10.1.1.0/24 Subnet 2 10.1.2.0/24 10.1.0.0/16Route Table Subnet 1 Destination Target 10.1.0.0/16 local 10.0.0.0/16 PCX-1 Route Table Subnet 2 Destination Target 10.1.0.0/16 local 10.0.0.0/16 PCX-2 A B C Taking VPC Peering to the next Level PCX-1 PCX-2
  106. 106. 10.0.0.0/16 10.0.0.0/16 Subnet 1 10.1.1.0/24 Subnet 2 10.1.2.0/24 10.1.0.0/16Route Table Subnet 1 Destination Target 10.1.0.0/16 local 10.0.0.0/16 PCX-1 Route Table Subnet 2 Destination Target 10.1.0.0/16 local 10.0.0.0/16 PCX-2 A B C Taking VPC Peering to the next Level PCX-1 PCX-2 Route Table Subnet # Destination Target 10.0.0.0/16 local 10.1.1.0/24 PCX-1 Route Table Subnet # Destination Target 10.0.0.0/16 local 10.1.2.0/24 PCX-1
  107. 107. 10.0.0.0/16 10.0.0.0/16 Subnet 1 10.1.1.0/24 Subnet 2 10.1.2.0/24 10.1.0.0/16Route Table Subnet 1 Destination Target 10.1.0.0/16 local 10.0.0.0/16 PCX-1 Route Table Subnet 2 Destination Target 10.1.0.0/16 local 10.0.0.0/16 PCX-2 A B C Taking VPC Peering to the next Level Floating NAT Network PCX-1 PCX-2 Route Table Subnet # Destination Target 10.0.0.0/16 local 10.1.1.0/24 PCX-1 Route Table Subnet # Destination Target 10.0.0.0/16 local 10.1.2.0/24 PCX-1
  108. 108. 10.0.0.0/16 10.0.0.0/16 Subnet 1 10.1.1.0/24 Subnet 2 10.1.2.0/24 10.1.0.0/16Route Table Subnet 1 Destination Target 10.1.0.0/16 local 10.0.0.0/16 PCX-1 Route Table Subnet 2 Destination Target 10.1.0.0/16 local 10.0.0.0/16 PCX-2 A B C Taking VPC Peering to the next Level Floating NAT Network 10.0.0.58 PCX-1 PCX-2 Route Table Subnet # Destination Target 10.0.0.0/16 local 10.1.1.0/24 PCX-1 Route Table Subnet # Destination Target 10.0.0.0/16 local 10.1.2.0/24 PCX-1
  109. 109. 10.0.0.0/16 10.0.0.0/16 Subnet 1 10.1.1.0/24 Subnet 2 10.1.2.0/24 10.1.0.0/16Route Table Subnet 1 Destination Target 10.1.0.0/16 local 10.0.0.0/16 PCX-1 Route Table Subnet 2 Destination Target 10.1.0.0/16 local 10.0.0.0/16 PCX-2 A B C Taking VPC Peering to the next Level Floating NAT Network 10.0.0.58 10.0.0.105 PCX-1 PCX-2 Route Table Subnet # Destination Target 10.0.0.0/16 local 10.1.1.0/24 PCX-1 Route Table Subnet # Destination Target 10.0.0.0/16 local 10.1.2.0/24 PCX-1
  110. 110. 10.0.0.0/16 10.0.0.0/16 Subnet 1 10.1.1.0/24 Subnet 2 10.1.2.0/24 10.1.0.0/16Route Table Subnet 1 Destination Target 10.1.0.0/16 local 10.0.0.0/16 PCX-1 Route Table Subnet 2 Destination Target 10.1.0.0/16 local 10.0.0.0/16 PCX-2 A B C Taking VPC Peering to the next Level Floating NAT Network 10.0.0.58 10.0.0.105 PCX-1 PCX-210.1.1.105 10.1.2.105 Route Table Subnet # Destination Target 10.0.0.0/16 local 10.1.1.0/24 PCX-1 Route Table Subnet # Destination Target 10.0.0.0/16 local 10.1.2.0/24 PCX-1
  111. 111. 10.0.0.0/16 10.0.0.0/16 Subnet 1 10.1.1.0/24 Subnet 2 10.1.2.0/24 10.1.0.0/16Route Table Subnet 1 Destination Target 10.1.0.0/16 local 10.0.0.0/16 PCX-1 Route Table Subnet 2 Destination Target 10.1.0.0/16 local 10.0.0.0/16 PCX-2 A B C Taking VPC Peering to the next Level Floating NAT Network SRC: 10.0.0.58 DST: 10.1.1.105 SRC: 10.1.2.105 DST: 10.0.0.105 10.0.0.58 10.0.0.105 PCX-1 PCX-210.1.1.105 10.1.2.105 Route Table Subnet # Destination Target 10.0.0.0/16 local 10.1.1.0/24 PCX-1 Route Table Subnet # Destination Target 10.0.0.0/16 local 10.1.2.0/24 PCX-1
  112. 112. 10.0.0.0/16 10.0.0.0/16 Subnet 1 10.1.1.0/24 Subnet 2 10.1.2.0/24 10.1.0.0/16Route Table Subnet 1 Destination Target 10.1.0.0/16 local 10.0.0.0/16 PCX-1 Route Table Subnet 2 Destination Target 10.1.0.0/16 local 10.0.0.0/16 PCX-2 A B C Taking VPC Peering to the next Level Floating NAT Network SRC: 10.0.0.58 DST: 10.1.1.105 SRC: 10.1.2.105 DST: 10.0.0.105 10.0.0.58 10.0.0.105 PCX-1 PCX-210.1.1.105 10.1.2.105 Route Table Subnet # Destination Target 10.0.0.0/16 local 10.1.1.0/24 PCX-1 Route Table Subnet # Destination Target 10.0.0.0/16 local 10.1.2.0/24 PCX-1
  113. 113. 10.0.0.0/16 10.0.0.0/16 Subnet 1 10.1.1.0/24 Subnet 2 10.1.2.0/24 10.1.0.0/16Route Table Subnet 1 Destination Target 10.1.0.0/16 local 10.0.0.0/16 PCX-1 Route Table Subnet 2 Destination Target 10.1.0.0/16 local 10.0.0.0/16 PCX-2 A B C Taking VPC Peering to the next Level Floating NAT Network SRC: 10.0.0.58 DST: 10.1.1.105 SRC: 10.1.2.105 DST: 10.0.0.105 10.0.0.58 10.0.0.105 PCX-1 PCX-210.1.1.105 10.1.2.105 Route Table Subnet # Destination Target 10.0.0.0/16 local 10.1.1.0/24 PCX-1 Route Table Subnet # Destination Target 10.0.0.0/16 local 10.1.2.0/24 PCX-1
  114. 114. 10.0.0.0/16 10.0.0.0/16 Subnet 1 10.1.1.0/24 Subnet 2 10.1.2.0/24 10.1.0.0/16Route Table Subnet 1 Destination Target 10.1.0.0/16 local 10.0.0.0/16 PCX-1 Route Table Subnet 2 Destination Target 10.1.0.0/16 local 10.0.0.0/16 PCX-2 A B C Taking VPC Peering to the next Level Floating NAT Network SRC: 10.0.0.58 DST: 10.1.1.105 SRC: 10.1.2.105 DST: 10.0.0.105 10.0.0.58 10.0.0.105 PCX-1 PCX-210.1.1.105 10.1.2.105 Route Table Subnet # Destination Target 10.0.0.0/16 local 10.1.1.0/24 PCX-1 Route Table Subnet # Destination Target 10.0.0.0/16 local 10.1.2.0/24 PCX-1
  115. 115. 10.0.0.0/16 10.0.0.0/16 Subnet 1 10.1.1.0/24 Subnet 2 10.1.2.0/24 10.1.0.0/16Route Table Subnet 1 Destination Target 10.1.0.0/16 local 10.0.0.0/16 PCX-1 Route Table Subnet 2 Destination Target 10.1.0.0/16 local 10.0.0.0/16 PCX-2 A B C Taking VPC Peering to the next Level Floating NAT Network SRC: 10.0.0.58 DST: 10.1.1.105 SRC: 10.1.2.105 DST: 10.0.0.105 10.0.0.58 10.0.0.105 PCX-1 PCX-210.1.1.105 10.1.2.105 Route53 Private Hosted Zone Route53 Private Hosted Zone Route Table Subnet # Destination Target 10.0.0.0/16 local 10.1.1.0/24 PCX-1 Route Table Subnet # Destination Target 10.0.0.0/16 local 10.1.2.0/24 PCX-1
  116. 116. Demo
 VPC to VPC Communication
  117. 117. Availability Zone A Private Subnet Availability Zone B Private Subnet Internet AWS region Public Subnet Public Subnet NAT • Use Auto Scaling for NAT availability • Create 1 NAT per Availability Zone • All private subnet route tables to point to same zone NAT • 1 Auto Scaling group per NAT with min and max size set to 1 • Let Auto Scaling monitor the health and availability of your NATs • NAT bootstrap script updates route tables programmatically • Latest version of script – uses tags: https://github.com/ralex-aws/vpc Auto scale HA NAT Dynamo DB Scaling Internet egress capacity NAT ASG min=1 max=1 ASG min=1 max=1 SQS SNS
  118. 118. Availability Zone A Private Subnet Availability Zone B Private Subnet Internet AWS region Public Subnet Public Subnet NAT • Use Auto Scaling for NAT availability • Create 1 NAT per Availability Zone • All private subnet route tables to point to same zone NAT • 1 Auto Scaling group per NAT with min and max size set to 1 • Let Auto Scaling monitor the health and availability of your NATs • NAT bootstrap script updates route tables programmatically • Latest version of script – uses tags: https://github.com/ralex-aws/vpc Auto scale HA NAT Dynamo DB Scaling Internet egress capacity NAT ASG min=1 max=1 ASG min=1 max=1 SQS SNS
  119. 119. Availability Zone A Private Subnet Availability Zone B Private Subnet Internet AWS region Public Subnet Public Subnet NAT • Use Auto Scaling for NAT availability • Create 1 NAT per Availability Zone • All private subnet route tables to point to same zone NAT • 1 Auto Scaling group per NAT with min and max size set to 1 • Let Auto Scaling monitor the health and availability of your NATs • NAT bootstrap script updates route tables programmatically • Latest version of script – uses tags: https://github.com/ralex-aws/vpc Auto scale HA NAT Dynamo DB Scaling Internet egress capacity NAT ASG min=1 max=1 ASG min=1 max=1 SQS SNS
  120. 120. Availability Zone A Private Subnet(s) Private Subnet(s) AWS region VPN connection Customer data center Intranet AppsIntranet Apps Availability Zone B Internal customers Controlling the border Internal Load balancer Elastic Load Balancing Private Subnet Elastic Load Balancing Private Subnet S3 Scaling Internet egress capacity Direct Connect DynamoDBSQS
  121. 121. Availability Zone A Private Subnet(s) Private Subnet(s) AWS region VPN connection Customer data center Intranet AppsIntranet Apps Availability Zone B Internal customers Controlling the border Internal Load balancer Elastic Load Balancing Private Subnet Elastic Load Balancing Private Subnet S3 Scaling Internet egress capacity Direct Connect DynamoDBSQS
  122. 122. Availability Zone A Private Subnet(s) Private Subnet(s) AWS region VPN connection Customer data center Intranet AppsIntranet Apps Availability Zone B Internal customers Controlling the border Internal Load balancer Elastic Load Balancing Private Subnet Elastic Load Balancing Private Subnet • Squid Proxy layer deployed between internal load balancer and the IGW border. Public Subnet Public Subnet S3 Scaling Internet egress capacity Direct Connect DynamoDBSQS
  123. 123. Availability Zone A Private Subnet(s) Private Subnet(s) AWS region VPN connection Customer data center Intranet AppsIntranet Apps Availability Zone B Internal customers Controlling the border Internal Load balancer Elastic Load Balancing Private Subnet Elastic Load Balancing Private Subnet • Squid Proxy layer deployed between internal load balancer and the IGW border. Public Subnet Public Subnet S3 • Only proxy subnets have route to IGW. Scaling Internet egress capacity # CIDR AND Destination Domain based Allow # CIDR Subnet blocks for Internal ELBs acl int_elb_cidrs src 10.1.3.0/24 10.1.4.0/24 # Destination domain for target S3 bucket acl aws_v2_endpoints dstdomain .amazonaws.com # Squid does AND on both ACLs for allow match http_access allow int_elb_cidrs aws_v2_endpoints # Deny everything else http_access deny all Direct Connect DynamoDBSQS
  124. 124. Availability Zone A Private Subnet(s) Private Subnet(s) AWS region VPN connection Customer data center Intranet AppsIntranet Apps Availability Zone B Internal customers Controlling the border Internal Load balancer Elastic Load Balancing Private Subnet Elastic Load Balancing Private Subnet • Squid Proxy layer deployed between internal load balancer and the IGW border. Public Subnet Public Subnet S3 • Only proxy subnets have route to IGW. Scaling Internet egress capacity # CIDR AND Destination Domain based Allow # CIDR Subnet blocks for Internal ELBs acl int_elb_cidrs src 10.1.3.0/24 10.1.4.0/24 # Destination domain for target S3 bucket acl aws_v2_endpoints dstdomain .amazonaws.com # Squid does AND on both ACLs for allow match http_access allow int_elb_cidrs aws_v2_endpoints # Deny everything else http_access deny all Direct Connect DynamoDBSQS
  125. 125. Availability Zone A Private Subnet(s) Private Subnet(s) AWS region VPN connection Customer data center Intranet AppsIntranet Apps Availability Zone B Internal customers Controlling the border Internal Load balancer Elastic Load Balancing Private Subnet Elastic Load Balancing Private Subnet • Squid Proxy layer deployed between internal load balancer and the IGW border. Public Subnet Public Subnet S3 • Only proxy subnets have route to IGW. • Proxy security group allows inbound only from Elastic Load Balancing security group. Scaling Internet egress capacity # CIDR AND Destination Domain based Allow # CIDR Subnet blocks for Internal ELBs acl int_elb_cidrs src 10.1.3.0/24 10.1.4.0/24 # Destination domain for target S3 bucket acl aws_v2_endpoints dstdomain .amazonaws.com # Squid does AND on both ACLs for allow match http_access allow int_elb_cidrs aws_v2_endpoints # Deny everything else http_access deny all Direct Connect DynamoDBSQS
  126. 126. Availability Zone A Private Subnet(s) Private Subnet(s) AWS region VPN connection Customer data center Intranet AppsIntranet Apps Availability Zone B Internal customers Controlling the border Internal Load balancer Elastic Load Balancing Private Subnet Elastic Load Balancing Private Subnet • Squid Proxy layer deployed between internal load balancer and the IGW border. Public Subnet Public Subnet S3 HTTP/S • Only proxy subnets have route to IGW. • Proxy security group allows inbound only from Elastic Load Balancing security group. • Proxy restricts which URLs may pass. In this example, *.amazonaws.com is allowed. Scaling Internet egress capacity # CIDR AND Destination Domain based Allow # CIDR Subnet blocks for Internal ELBs acl int_elb_cidrs src 10.1.3.0/24 10.1.4.0/24 # Destination domain for target S3 bucket acl aws_v2_endpoints dstdomain .amazonaws.com # Squid does AND on both ACLs for allow match http_access allow int_elb_cidrs aws_v2_endpoints # Deny everything else http_access deny all Direct Connect DynamoDBSQS
  127. 127. Availability Zone A Private Subnet(s) Private Subnet(s) AWS region VPN connection Customer data center Intranet AppsIntranet Apps Availability Zone B Internal customers Controlling the border Internal Load balancer Elastic Load Balancing Private Subnet Elastic Load Balancing Private Subnet • Squid Proxy layer deployed between internal load balancer and the IGW border. Public Subnet Public Subnet S3 HTTP/S • Only proxy subnets have route to IGW. • Proxy security group allows inbound only from Elastic Load Balancing security group. • Proxy restricts which URLs may pass. In this example, *.amazonaws.com is allowed. • Egress NACLs on proxy subnets enforce HTTP/S only. Scaling Internet egress capacity # CIDR AND Destination Domain based Allow # CIDR Subnet blocks for Internal ELBs acl int_elb_cidrs src 10.1.3.0/24 10.1.4.0/24 # Destination domain for target S3 bucket acl aws_v2_endpoints dstdomain .amazonaws.com # Squid does AND on both ACLs for allow match http_access allow int_elb_cidrs aws_v2_endpoints # Deny everything else http_access deny all Direct Connect DynamoDBSQS
  128. 128. Availability Zone A Private Subnet(s) Private Subnet(s) AWS region VPN connection Customer data center Intranet AppsIntranet Apps Availability Zone B Internal customers Controlling the border Internal Load balancer Elastic Load Balancing Private Subnet Elastic Load Balancing Private Subnet • Squid Proxy layer deployed between internal load balancer and the IGW border. Public Subnet Public Subnet S3 HTTP/S • Only proxy subnets have route to IGW. • Proxy security group allows inbound only from Elastic Load Balancing security group. • Proxy restricts which URLs may pass. In this example, *.amazonaws.com is allowed. • Egress NACLs on proxy subnets enforce HTTP/S only. Scaling Internet egress capacity # CIDR AND Destination Domain based Allow # CIDR Subnet blocks for Internal ELBs acl int_elb_cidrs src 10.1.3.0/24 10.1.4.0/24 # Destination domain for target S3 bucket acl aws_v2_endpoints dstdomain .amazonaws.com # Squid does AND on both ACLs for allow match http_access allow int_elb_cidrs aws_v2_endpoints # Deny everything else http_access deny all Direct Connect DynamoDBSQS
  129. 129. Availability Zone A Private Subnet(s) Private Subnet(s) AWS region VPN connection Customer data center Intranet AppsIntranet Apps Availability Zone B Internal customers Controlling the border Internal Load balancer Elastic Load Balancing Private Subnet Elastic Load Balancing Private Subnet • Squid Proxy layer deployed between internal load balancer and the IGW border. Public Subnet Public Subnet S3 HTTP/S • Only proxy subnets have route to IGW. • Proxy security group allows inbound only from Elastic Load Balancing security group. • Proxy restricts which URLs may pass. In this example, *.amazonaws.com is allowed. • Egress NACLs on proxy subnets enforce HTTP/S only. Scaling Internet egress capacity # CIDR AND Destination Domain based Allow # CIDR Subnet blocks for Internal ELBs acl int_elb_cidrs src 10.1.3.0/24 10.1.4.0/24 # Destination domain for target S3 bucket acl aws_v2_endpoints dstdomain .amazonaws.com # Squid does AND on both ACLs for allow match http_access allow int_elb_cidrs aws_v2_endpoints # Deny everything else http_access deny all Direct Connect DynamoDBSQS
  130. 130. Availability Zone A Private Subnet(s) Private Subnet(s) AWS region VPN connection Customer data center Intranet AppsIntranet Apps Availability Zone B Internal customers Controlling the border Internal Load balancer Elastic Load Balancing Private Subnet Elastic Load Balancing Private Subnet • Squid Proxy layer deployed between internal load balancer and the IGW border. Public Subnet Public Subnet S3 HTTP/S • Only proxy subnets have route to IGW. • Proxy security group allows inbound only from Elastic Load Balancing security group. • Proxy restricts which URLs may pass. In this example, *.amazonaws.com is allowed. • Egress NACLs on proxy subnets enforce HTTP/S only. Scaling Internet egress capacity # CIDR AND Destination Domain based Allow # CIDR Subnet blocks for Internal ELBs acl int_elb_cidrs src 10.1.3.0/24 10.1.4.0/24 # Destination domain for target S3 bucket acl aws_v2_endpoints dstdomain .amazonaws.com # Squid does AND on both ACLs for allow match http_access allow int_elb_cidrs aws_v2_endpoints # Deny everything else http_access deny all Direct Connect DynamoDBSQS • Could also have HA NATs NATNAT
  131. 131. Availability Zone A Private Subnet(s) Private Subnet(s) AWS region VPN connection Customer data center Intranet AppsIntranet Apps Availability Zone B Internal customers Controlling the border Internal Load balancer Elastic Load Balancing Private Subnet Elastic Load Balancing Private Subnet • Squid Proxy layer deployed between internal load balancer and the IGW border. Public Subnet Public Subnet S3 HTTP/S • Only proxy subnets have route to IGW. • Proxy security group allows inbound only from Elastic Load Balancing security group. • Proxy restricts which URLs may pass. In this example, *.amazonaws.com is allowed. • Egress NACLs on proxy subnets enforce HTTP/S only. Scaling Internet egress capacity # CIDR AND Destination Domain based Allow # CIDR Subnet blocks for Internal ELBs acl int_elb_cidrs src 10.1.3.0/24 10.1.4.0/24 # Destination domain for target S3 bucket acl aws_v2_endpoints dstdomain .amazonaws.com # Squid does AND on both ACLs for allow match http_access allow int_elb_cidrs aws_v2_endpoints # Deny everything else http_access deny all Direct Connect DynamoDBSQS • Could also have HA NATs NATNAT
  132. 132. Multicast on AWS
  133. 133. Multicast on AWS • Not directly supported
  134. 134. Multicast on AWS • Not directly supported 10.0.0.54 10.0.0.79 10.0.1.132 Subnet 10.0.0.0/24 Subnet 10.0.1.0/24 10.0.1.18310.0.0.41
  135. 135. Multicast on AWS • Not directly supported • Can be implemented with an overlay network – GRE or L2TP tunnels, Ntop’s N2N 10.0.0.54 10.0.0.79 10.0.1.132 Subnet 10.0.0.0/24 Subnet 10.0.1.0/24 10.0.1.18310.0.0.41
  136. 136. Multicast on AWS • Not directly supported • Can be implemented with an overlay network – GRE or L2TP tunnels, Ntop’s N2N 10.0.0.54 10.0.0.79 10.0.1.132 Subnet 10.0.0.0/24 Subnet 10.0.1.0/24 Tunnel 10.0.1.18310.0.0.41
  137. 137. Multicast on AWS • Not directly supported • Can be implemented with an overlay network – GRE or L2TP tunnels, Ntop’s N2N 10.0.0.54 10.0.0.79 10.0.1.132192.16.0.10 192.168.0.13 Subnet 10.0.0.0/24 Subnet 10.0.1.0/24 Tunnel 10.0.1.18310.0.0.41
  138. 138. Multicast on AWS • Not directly supported • Can be implemented with an overlay network – GRE or L2TP tunnels, Ntop’s N2N • GRE configuration can be automated – Multicast configuration stored in tags 10.0.0.54 10.0.0.79 10.0.1.132192.16.0.10 192.168.0.12 192.168.0.13 Subnet 10.0.0.0/24 Subnet 10.0.1.0/24 Tunnel 10.0.1.18310.0.0.41 192.168.0.12 192.168.0.0/24 Overlay
  139. 139. Multicast on AWS • Not directly supported • Can be implemented with an overlay network – GRE or L2TP tunnels, Ntop’s N2N • GRE configuration can be automated – Multicast configuration stored in tags 10.0.0.54 10.0.0.79 10.0.1.132192.16.0.10 192.168.0.12 192.168.0.13 Subnet 10.0.0.0/24 Subnet 10.0.1.0/24 Tunnel 10.0.1.18310.0.0.41 192.168.0.12 192.168.0.0/24 Overlay TAG: multicast App1,192.168.0.13/24 TAG: multicast App1,192.168.0.12/24 TAG: multicast App1,192.168.0.10/24
  140. 140. Multicast on AWS • Not directly supported • Can be implemented with an overlay network – GRE or L2TP tunnels, Ntop’s N2N • GRE configuration can be automated – Multicast configuration stored in tags 10.0.0.54 10.0.0.79 10.0.1.132192.16.0.10 192.168.0.12 192.168.0.13 Subnet 10.0.0.0/24 Subnet 10.0.1.0/24 Tunnel 10.0.1.18310.0.0.41 192.168.0.12 192.168.0.0/24 Overlay TAG: multicast App1,192.168.0.13/24 TAG: multicast App1,192.168.0.12/24 TAG: multicast App1,192.168.0.10/24 Setup Guide: http://bit.ly/aws-multi
  141. 141. Multicast on AWS • Not directly supported • Can be implemented with an overlay network – GRE or L2TP tunnels, Ntop’s N2N • GRE configuration can be automated – Multicast configuration stored in tags • Periodically check for new members (60 seconds) 10.0.0.54 10.0.0.79 10.0.1.132192.16.0.10 192.168.0.12 192.168.0.13 Subnet 10.0.0.0/24 Subnet 10.0.1.0/24 Tunnel 10.0.1.18310.0.0.41 192.168.0.12 192.168.0.0/24 Overlay TAG: multicast App1,192.168.0.13/24 TAG: multicast App1,192.168.0.12/24 TAG: multicast App1,192.168.0.10/24 Setup Guide: http://bit.ly/aws-multi
  142. 142. Demo
 Scalable & HA Internet Egress
  143. 143. I’ve automated my deployments but what about responding to events?
  144. 144. Your Application Stacks Availability Zone A Availability Zone B Private subnetPrivate subnet Public subnetPublic subnet Private subnetPrivate subnet CloudFront Glacier S3 DynamoDB Route 53 CloudWatch CloudFormation NAT Stacks for: VPC Edge Services Datastores Applications Presentation Amazon SQS Auto Scaling groups AWS Region SNS
  145. 145. Your Application Stacks Availability Zone A Availability Zone B Private subnetPrivate subnet Public subnetPublic subnet Private subnetPrivate subnet CloudFront Glacier S3 DynamoDB Route 53 CloudWatch CloudFormation NAT Stacks for: VPC Edge Services Datastores Applications Presentation Amazon SQS Auto Scaling groups AWS Region SNS What about services with no native CloudWatch integration
  146. 146. Your Application Stacks Availability Zone A Availability Zone B Private subnetPrivate subnet Public subnetPublic subnet Private subnetPrivate subnet CloudFront Glacier S3 DynamoDB Route 53 CloudWatch CloudFormation NAT Stacks for: VPC Edge Services Datastores Applications Presentation Amazon SQS Auto Scaling groups AWS Region SNS What about services with no native CloudWatch integration Managing non- CloudFormation supported resources/events
  147. 147. Your Application Stacks Availability Zone A Availability Zone B Private subnetPrivate subnet Public subnetPublic subnet Private subnetPrivate subnet CloudFront Glacier S3 DynamoDB Route 53 CloudWatch CloudFormation NAT Stacks for: VPC Edge Services Datastores Applications Presentation Amazon SQS Auto Scaling groups AWS Region SNS What about services with no native CloudWatch integration Collecting and analysing non-EC2 logs Managing non- CloudFormation supported resources/events
  148. 148. Your Application Stacks Availability Zone A Availability Zone B Private subnetPrivate subnet Public subnetPublic subnet Private subnetPrivate subnet CloudFront Glacier S3 DynamoDB Route 53 CloudWatch CloudFormation NAT Stacks for: VPC Edge Services Datastores Applications Presentation Amazon SQS Auto Scaling groups AWS Region SNS What about services with no native CloudWatch integration Collecting and analysing non-EC2 logs Managing non- CloudFormation supported resources/events
  149. 149. Advanced uses of CloudWatch – Custom Metrics #!/usr/bin/python import boto.ec2.cloudwatch import boto.vpc AWS_Regions=["us-east-1","us-west-2","us-west-1","eu-west-1"] CloudWatch_Region="us-east-1" cw = boto.ec2.cloudwatch.connect_to_region(CloudWatch_Region) for region in AWS_Regions: vpcconn = boto.vpc.connect_to_region(region) vpns = vpcconn.get_all_vpn_connections() for vpn in vpns: if vpn.state == "available": active_tunnels = 0 if vpn.tunnels[0].status == "UP": active_tunnels+=1 if vpn.tunnels[1].status == "UP": active_tunnels+=1 print vpn.id+" has "+str(active_tunnels)+" active tunnels!” cw.put_metric_data("VPNStatus", vpn.id, value=active_tunnels, dimensions={'VGW':vpn.vpn_gateway_id, 'CGW':vpn.customer_gateway_id})
  150. 150. Advanced uses of CloudWatch – Custom Metrics #!/usr/bin/python import boto.ec2.cloudwatch import boto.vpc AWS_Regions=["us-east-1","us-west-2","us-west-1","eu-west-1"] CloudWatch_Region="us-east-1" cw = boto.ec2.cloudwatch.connect_to_region(CloudWatch_Region) for region in AWS_Regions: vpcconn = boto.vpc.connect_to_region(region) vpns = vpcconn.get_all_vpn_connections() for vpn in vpns: if vpn.state == "available": active_tunnels = 0 if vpn.tunnels[0].status == "UP": active_tunnels+=1 if vpn.tunnels[1].status == "UP": active_tunnels+=1 print vpn.id+" has "+str(active_tunnels)+" active tunnels!” cw.put_metric_data("VPNStatus", vpn.id, value=active_tunnels, dimensions={'VGW':vpn.vpn_gateway_id, 'CGW':vpn.customer_gateway_id})
  151. 151. Advanced uses of CloudWatch – Custom Metrics #!/usr/bin/python import boto.ec2.cloudwatch import boto.vpc AWS_Regions=["us-east-1","us-west-2","us-west-1","eu-west-1"] CloudWatch_Region="us-east-1" cw = boto.ec2.cloudwatch.connect_to_region(CloudWatch_Region) for region in AWS_Regions: vpcconn = boto.vpc.connect_to_region(region) vpns = vpcconn.get_all_vpn_connections() for vpn in vpns: if vpn.state == "available": active_tunnels = 0 if vpn.tunnels[0].status == "UP": active_tunnels+=1 if vpn.tunnels[1].status == "UP": active_tunnels+=1 print vpn.id+" has "+str(active_tunnels)+" active tunnels!” cw.put_metric_data("VPNStatus", vpn.id, value=active_tunnels, dimensions={'VGW':vpn.vpn_gateway_id, 'CGW':vpn.customer_gateway_id})
  152. 152. Advanced uses of CloudWatch – Custom Metrics #!/usr/bin/python import boto.ec2.cloudwatch import boto.vpc AWS_Regions=["us-east-1","us-west-2","us-west-1","eu-west-1"] CloudWatch_Region="us-east-1" cw = boto.ec2.cloudwatch.connect_to_region(CloudWatch_Region) for region in AWS_Regions: vpcconn = boto.vpc.connect_to_region(region) vpns = vpcconn.get_all_vpn_connections() for vpn in vpns: if vpn.state == "available": active_tunnels = 0 if vpn.tunnels[0].status == "UP": active_tunnels+=1 if vpn.tunnels[1].status == "UP": active_tunnels+=1 print vpn.id+" has "+str(active_tunnels)+" active tunnels!” cw.put_metric_data("VPNStatus", vpn.id, value=active_tunnels, dimensions={'VGW':vpn.vpn_gateway_id, 'CGW':vpn.customer_gateway_id}) And Not Just For AWS Resources!
  153. 153. Advanced uses of CloudWatch – Logs CloudWatch Logs
  154. 154. Advanced uses of CloudWatch – Logs EC2 CloudWatch Logs OS Agent-based
  155. 155. Advanced uses of CloudWatch – Logs EC2 Traditional Server CloudWatch Logs OS Agent-based OS Agent-based
  156. 156. Advanced uses of CloudWatch – Logs CloudTrail EC2 Traditional Server CloudWatch Logs OS Agent-based OS Agent-based Native
  157. 157. Advanced uses of CloudWatch – Logs CloudTrail S3 EC2 Traditional Server CloudWatch Logs OS Agent-based OS Agent-based Native Pull/Push Lambda??
  158. 158. Advanced uses of CloudWatch – Logs CloudTrail S3 EC2 Traditional Server CloudWatch Logs CloudFront OS Agent-based OS Agent-based Native Pull/Push Lambda?? Pull/Push Lam bda??
  159. 159. Advanced uses of CloudWatch – Logs CloudTrail S3 EC2 Traditional Server CloudWatch Logs CloudFront OS Agent-based OS Agent-based Native Pull/Push Lambda?? Pull/Push Lam bda?? Metrics filters:
  160. 160. Advanced uses of CloudWatch – Logs CloudTrail S3 EC2 Traditional Server CloudWatch Logs CloudFront OS Agent-based OS Agent-based Native Pull/Push Lambda?? Pull/Push Lam bda?? Metrics filters: • Literal Terms
  161. 161. Advanced uses of CloudWatch – Logs CloudTrail S3 EC2 Traditional Server CloudWatch Logs CloudFront OS Agent-based OS Agent-based Native Pull/Push Lambda?? Pull/Push Lam bda?? Metrics filters: • Literal Terms
  162. 162. Advanced uses of CloudWatch – Logs CloudTrail S3 EC2 Traditional Server CloudWatch Logs CloudFront OS Agent-based OS Agent-based Native Pull/Push Lambda?? Pull/Push Lam bda?? Metrics filters: • Literal Terms • Common Log Format
  163. 163. Advanced uses of CloudWatch – Logs CloudTrail S3 EC2 Traditional Server CloudWatch Logs CloudFront OS Agent-based OS Agent-based Native Pull/Push Lambda?? Pull/Push Lam bda?? Metrics filters: • Literal Terms • Common Log Format
  164. 164. Advanced uses of CloudWatch – Logs CloudTrail S3 EC2 Traditional Server CloudWatch Logs CloudFront OS Agent-based OS Agent-based Native Pull/Push Lambda?? Pull/Push Lam bda?? Metrics filters: • Literal Terms • Common Log Format • JSON
  165. 165. Lambda-powered custom resources EC2 instance Software pkgs, config, & dataCloudWatch alarms Your AWS CloudFormation stack // Implement custom logic here Look up an AMI ID Your AWS Lambda functions Look up VPC ID and Subnet ID Reverse an IP address Lambda-powered custom resources
  166. 166. Lambda-powered custom resources security group Auto Scaling group EC2 instance Elastic Load Balancing ElastiCache memcached cluster Software pkgs, config, & dataCloudWatch alarms Your AWS CloudFormation stack // Implement custom logic here Look up an AMI ID Your AWS Lambda functions Look up VPC ID and Subnet ID Reverse an IP address Lambda-powered custom resources
  167. 167. Demo
 Lambda & CloudFormation
  168. 168. Recent announcements of interest • AWS Lambda GA • Amazon EC2 Container Service GA • Amazon Machine Learning • Amazon Workspaces Application Manager • Amazon Elastic File System

×