Improving Infrastructure Governance on AWS - AWS June 2016 Webinar Series

2. https://secure.flickr.com/photos/mgifford/4525333972 Why are we here today?

3. Why are we here today? Using cloud based infrastructure changes how to think about governing our infrastructure:  Infrastructure can be provisioned in seconds.. and go away just as quickly!  Development teams expect a higher level of flexibility and self control in interfacing with their infrastructure needs  Being API driven means that the way people provision and manage infrastructure in the cloud has changed compared to on-premises

4. Why are we here today? That doesn’t mean that our basic governance needs change:  We still need to have some ability to drive best practices/patterns in our organizations  We need to make sure that we’re able to audit and track changes to our infrastructure for both regulation and security purposes  We need to make sure that we understand how resources are related and integrated

5. What can we do? There are a few areas to focus on that can help us accomplish both the freedom to rapidly provision, manage, and update our infrastructure while meeting our governance needs:  Policy as Code  Infrastructure standardization (via code!)  Self service environments  Logging/Auditing/Reacting to infrastructure change

6. Policy as Code builds off of infrastructure as code practices by allowing organizations to codify infrastructure and system configurations allowing them to monitor and enforce compliance dynamically and at scale.

7. Infrastructure as Code is a practice in which infrastructure is provisioned and managed using code and software development techniques, such as version control and continuous integration.

8. Infrastructure as Code “levels” AWS Resources Operating System and Host Configuration Application Configuration

9. Infrastructure as Code “levels” AWS Resources Operating System and Host Configuration Application Configuration allOfThis == $Code

10. Browse and launch AWS ConfigAWS CloudTrail Use and modify Users Admin Putting the AWS Management services together AWS Service Catalog Provision with Tags API calls Configuration checks and reactions to change Troubleshoot and Audit

11.  Create templates of your infrastructure  CloudFormation provisions AWS resources based on dependency needs  Version control/replicate/update templates like code  Integrates with development, CI/CD, management tools AWS CloudFormation

12. Template CloudFormation Stack JSON formatted file Parameter definition Resource creation Configuration actions Configured AWS resources Comprehensive service support Service event aware Customizable Framework Stack creation Stack updates Error detection and rollback CloudFormation – Components & Technology

13. Template File Defining Stack The entire infrastructure can be represented in an AWS CloudFormation template. Many Stacks & Environments from One Template

14. Template File Defining Stack The entire infrastructure can be represented in an AWS CloudFormation template. Use the version control system of your choice to store and track changes to this template Many Stacks & Environments from One Template Git Perforce SVN …

15. Template File Defining Stack Git Perforce SVN … Dev Test Prod The entire infrastructure can be represented in an AWS CloudFormation template. Use the version control system of your choice to store and track changes to this template Build out multiple environments, such as for Development, Test, Production and even DR using the same template Many Stacks & Environments from One Template

16. CloudFormation example use cases: Have “full stack” templates that can be used to stand up common application patterns inside your organization such as a 3-tier application template that:  uses Lambda custom resources to look up appropriate VPC information (VPC ID, Subnets, etc) based on tags  creates an Elastic Beanstalk environment that supports Multi-AZ, AutoScaling, CloudWatch Metrics, and Elastic Load Balancing  contains security controls such as AWS Identity and Access Management (IAM) roles, profiles, and policies, and Security Groups  allows the user to specify the language of their application  allows a user to specify which database they want (SQL or NoSQL) and then creates the appropriate resource

17. Using Parameters and Conditionals are two key ways to make a single template much more dynamic: "Parameters" : { "Database": { "Type" : "String", "Default" : "RDS", "AllowedValues" : ["RDS", "DynamoDB", "None"], "Description" : "Database to create. Select None if using an existing database.” } }, "Conditions" : { "CreateRDS" : {"Fn::Equals" : [{"Ref" : "Database"}, "RDS"]}, "CreateDynamoDB" : {"Fn::Equals" : [{"Ref" : "Database"}, "DynamoDB"]}, "CreateNone" : {"Fn::Equals" : [{"Ref" : "Database"}, "None"]} }, “Resources” : { ”RDSdb01" : { "Condition" : " CreateRDS ", "Type" : "AWS::RDS::Instance",

18. https://secure.flickr.com/photos/wscullin/3770015991 Now that we have these templates, what can we do to help simplify our developers’ lives further and increase our infrastructure standardization?

19.  Customized catalogs of products  Manage products centrally  Personalized, self-service portal  Integrate with existing systems AWS Service Catalog

20. What is AWS Service Catalog? AWS Service Catalog allows organizations to create and manage catalogs of IT services. It enables users to quickly deploy the approved IT services they need in a self-service manner. Organizations Developers Control Standardization Governance Agility Self-service Time to market

21. Creates portfolio Adds constraints and grant access 1 4 5 Administrator Portfolio Users Browse Products 6Launch ProductsAWS CloudFormation template Creates product3Authors template2 ProductX ProductY ProductZ 7 Deploys stacks Events Events 8 8 Service Catalog Create custom services and grant access Use a personalized portal to find and launch services

22. Service Catalog use cases: You can remove the need for developers to understand how all AWS services work. Treat infrastructure provisioning like buying components from a retail site:  provide standardized Service Catalog products around common internal application frameworks/architectural patterns  provide common application component products such as databases, queues, caches, worker tiers, etc  build logging, monitoring, metrics into these stacks  leverage service discovery tools when possible  build in the same best practices across development, staging, production environments with these provided products

23. We’ve helped solve some of our developer’s access and standardization issues, but how can we now go about auditing changes to our infrastructure? https://www.flickr.com/photos/atoach/7623237104

24. AWS CloudTrail  Records AWS API calls for your account  Delivers log files of API calls to S3  Delivery typically within 15 minutes of API call  Logs contain detailed information  Log files can be encrypted and have their integrity verified by you

25. AWS CloudTrail CloudTrail can help you achieve many tasks  Security analysis  Track changes to AWS resources, for example VPC security groups and NACLs  Compliance – log and understand AWS API call history  Prove that you did not:  Use the wrong region  Use services you don’t want  Troubleshoot operational issues – quickly identify the most recent changes to your environment

26. AWS CloudTrail logs can be delivered cross-account CloudTrail can help you achieve many tasks  Accounts can send their trails to a central account  Central account can then do analytics  Central account can:  Redistribute the trails  Grant access to the trails  Filter and reformat Trails (to meet privacy requirements)

27. CloudTrail

28. CloudTrail – Amazon CloudWatch Logs Integration

29.  Continuous recording  Inventory of AWS resources  New & deleted resources  Configuration change & compliance notifications AWS Config

30. Relationships Bi-directional map of dependencies automatically assigned Change to a resource propagates to create Configuration Items for related resources

31. Configuration Item  All configuration attributes  Normalized  Point in time  Captured on configuration change

32. Component Description Contains Metadata Information about this configuration item Version ID, Configuration item ID, Time when the configuration item was captured, State ID indicating the ordering of the configuration items of a resource, MD5Hash, etc. Common Attributes Resource attributes Resource ID, tags, Resource type. Amazon Resource Name (ARN) Availability Zone, etc. Relationships How the resource is related to other resources associated with the account EBS volume vol-1234567 is attached to an EC2 instance i- a1b2c3d4 Current Configuration Information returned through a call to the Describe or List API of the resource e.g. for EBS Volume State of DeleteOnTermination flag Type of volume. For example, gp2, io1, or standard Related Events The AWS CloudTrail events that are related to the current configuration of the resource AWS CloudTrail event ID Configuration Item

33. Sample Configuration Item "configurationItemVersion": "1.0", "configurationItemCaptureTime": "2014…", "configurationStateID": “….", "configurationItemStatus": "OK", "resourceId": "vol-ce676ccc", "arn": "arn:aws:us-west-………", "accountId": "12345678910", "availibilityZone": "us-west-2b", "resourceType": "AWS::EC2::Volume", "resourceCreationTime": "2014-02..", "tags": {}, "relationships": [ { "resourceId": "i-344c463d", "resourceType": "AWS::EC2::Instance", "name": "Attached to Instance" } ], "relatedEvents": [ "06c12a39-eb35-11de-ae07-db69edbb1e4", ], Metadata Common Attributes Relationships Related Events

34. Sample Configuration Item "configuration": { "volumeId": "vol-ce676ccc", "size": 1, "snapshotId": "", "availabilityZone": "us-west-2b", "state": "in-use", "createTime": "2014-02-……", "attachments": [ { "volumeId": "vol-ce676ccc", "instanceId": "i-344c463d", "device": "/dev/sdf", "state": "attached", "attachTime": "2014-03-", "deleteOnTermination": false } ], "tags": [ { "tagName": "environment", "tagValue": "PROD" Configuration

35. Having this data is one thing, but how can we react to change? https://www.flickr.com/photos/livenature/204420128/

36. CloudTrail – Amazon CloudWatch Logs Integration

37. CloudTrail – Amazon CloudWatch Logs Integration Trigger a CloudWatch Alarm based on API call!

38.  Check configuration changes  Pre-built rules provided by AWS  Custom rules using AWS Lambda  Continuous assessment  Dashboard  Compliance visualization  Identify offending changes  GitHub repo: Community sourced rules AWS Config Rules

39. NormalizeRecordChanging Resources AWS Config & Config Rules Deliver Stream Snapshot (ex. 2014-11-05) AWS Config APIs Store History Rules

40. Config Rules

41. Config Rules

42. Config Rules GitHub repository

43. AWS Management Tools Partners

44. FIN, ACK We’ve seen a quick run through today of the ways you can improve your governance on AWS:  Treat your infrastructure and host configuration as code!  This lends itself to being able to use services like Service Catalog to enable self-service in your organization  Track, trend, and alert on CloudTrail API logs to keep on top of access to your AWS resources  Use Config and Config Rules to understand the relationship between resources and react to policy violations  Putting all this together is what gives you Policy as Code!

45. Browse and launch AWS ConfigAWS CloudTrail Use and modify Users Admin Putting the AWS Management services together AWS Service Catalog Provision with Tags API calls Configuration checks and reactions to change Troubleshoot and Audit

46. But wait, there’s more! Resources to learn more:  More on DevOps: https://aws.amazon.com/devops/  AWS Management Services: https://aws.amazon.com/products/management/  AWS CloudFormation  https://aws.amazon.com/cloudformation/  AWS Service Catalog  https://aws.amazon.com/servicecatalog/  AWS CloudTrail  https://aws.amazon.com/cloudtrail/  AWS Config / Config Rules  https://aws.amazon.com/config/  GitHub repo: https://github.com/awslabs/aws-config-rules

47. Thank you! Happy Deploying!

Improving Infrastructure Governance on AWS - AWS June 2016 Webinar Series

Amazon Web Services

Improving Infrastructure Governance on AWS - AWS June 2016 Webinar Series