Dev/Test Environment Provisioning and Management on AWS
•
5 likes•1,979 views
This document discusses using AWS for development and test environments. It covers why AWS is well-suited for these use cases, the types of AWS services that can be used, and common patterns. Development and test environments on AWS are characterized as being disposable and numerous. AWS provides unlimited elastic capacity for development and test needs in the cloud at low cost since resources can be quickly provisioned as needed and discarded when no longer required.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (7)
Similar to Dev/Test Environment Provisioning and Management on AWS
Similar to Dev/Test Environment Provisioning and Management on AWS (20)
More from Shiva Narayanaswamy
More from Shiva Narayanaswamy (20)
Recently uploaded
Recently uploaded (16)
Dev/Test Environment Provisioning and Management on AWS
- 1. Environment Provisioning and Management on AWS Shiva N– Solution Architect
- 2. Platform Overview Why AWS for development & test? AWS services that can be employed Common dev-test patterns Security and Billing Agenda
- 3. Platform Overview Why AWS for development & test? AWS services that can be employed Common dev-test patterns Security and Billing Agenda
- 4. Support CertificationTrainingProfessional Services Technology Partners Consulting Partners AWS MarketplaceEcosystem Elastic Beanstalk for Java, Node.js, Python, Ruby, PHP and .Net OpsWorks CloudFormationContainers & Deployment (PaaS) Management & AdministrationIAM CloudWatchCloudTrail APIs and SDKsManagement ConsoleCloud HSM Command Line Interface Direct Connect Route 53VPC Networking Analytics Data PipelineRedshiftEMR Kinesis SWFSNS SQS CloudSearchSES AppStreamCloudFront Application Services WorkSpaces Regions Availability Zones Content Delivery POPs Storage GatewayS3 EBS Glacier Import/Export DynamoDB ElastiCache StorageCompute Databases RDS MySQL, PostgreSQL Oracle, SQL Server Elastic Load BalancerEC2 Auto Scaling Platform Logical View : Block Diagram
- 5. Regions Availability Zones Content Delivery POPs Storage GatewayS3 EBS Glacier Import/Export DynamoDB ElastiCache StorageCompute Databases RDS MySQL, PostgreSQL Oracle, SQL Server Elastic Load BalancerEC2 Auto Scaling
- 6. Regions Availability Zones Content Delivery POPs Storage GatewayS3 EBS Glacier Import/Export DynamoDB ElastiCache StorageCompute Databases RDS MySQL, PostgreSQL Oracle, SQL Server Elastic Load BalancerEC2 Auto Scaling PV/HVM x86 Hourly Pricing - On Demand - Reserved L|M|H - Spot - Dedicated 15 Families 37 Instance Types
- 7. Regions Availability Zones Content Delivery POPs Storage GatewayS3 EBS Glacier Import/Export DynamoDB ElastiCache StorageCompute Databases RDS MySQL, PostgreSQL Oracle, SQL Server Elastic Load BalancerEC2 Auto Scaling Auto Scaling Regional HA SSL Offloading Loose Coupling Health Checks Connection Mgt
- 8. Regions Availability Zones Content Delivery POPs Storage GatewayS3 EBS Glacier Import/Export DynamoDB ElastiCache StorageCompute Databases RDS MySQL, PostgreSQL Oracle, SQL Server Elastic Load BalancerEC2 Auto Scaling
- 9. Regions Availability Zones Content Delivery POPs Storage GatewayS3 EBS Glacier Import/Export DynamoDB ElastiCache StorageCompute Databases RDS MySQL, PostgreSQL Oracle, SQL Server Elastic Load BalancerEC2 Auto Scaling
- 10. Regions Availability Zones Content Delivery POPs Storage GatewayS3 EBS Glacier Import/Export DynamoDB ElastiCache StorageCompute Databases RDS MySQL, PostgreSQL Oracle, SQL Server Elastic Load BalancerEC2 Auto Scaling
- 11. Regions Availability Zones Content Delivery POPs Storage GatewayS3 EBS Glacier Import/Export DynamoDB ElastiCache StorageCompute Databases RDS MySQL, PostgreSQL Oracle, SQL Server Elastic Load BalancerEC2 Auto Scaling
- 12. Regions Availability Zones Content Delivery POPs Storage GatewayS3 EBS Glacier Import/Export DynamoDB ElastiCache StorageCompute Databases RDS MySQL, PostgreSQL Oracle, SQL Server Elastic Load BalancerEC2 Auto Scaling
- 13. Regions Availability Zones Content Delivery POPs Storage GatewayS3 EBS Glacier Import/Export DynamoDB ElastiCache StorageCompute Databases RDS MySQL, PostgreSQL Oracle, SQL Server Elastic Load BalancerEC2 Auto Scaling
- 14. Regions Availability Zones Content Delivery POPs Storage GatewayS3 EBS Glacier Import/Export DynamoDB ElastiCache StorageCompute Databases RDS MySQL, PostgreSQL Oracle, SQL Server Elastic Load BalancerEC2 Auto Scaling Yes SQL Amazon RDS In-Memory Amazon ElastiCache AWS Managed versus Customer Managed Database Options No SQL MPP DW Redshift Self Managed EC2
- 15. Direct Connect Route 53VPC Networking Regions Availability Zones Content Delivery POPs Storage GatewayS3 EBS Glacier Import/Export DynamoDB ElastiCache StorageCompute Databases RDS MySQL, PostgreSQL Oracle, SQL Server Elastic Load BalancerEC2 Auto Scaling
- 16. Direct Connect Route 53VPC Networking Analytics Data PipelineRedshiftEMR Kinesis SWFSNS SQS CloudSearchSES AppStreamCloudFront Application Services WorkSpaces Regions Availability Zones Content Delivery POPs Storage GatewayS3 EBS Glacier Import/Export DynamoDB ElastiCache StorageCompute Databases RDS MySQL, PostgreSQL Oracle, SQL Server Elastic Load BalancerEC2 Auto Scaling
- 17. Direct Connect Route 53VPC Networking Analytics Data PipelineRedshiftEMR Kinesis SWFSNS SQS CloudSearchSES AppStreamCloudFront Application Services WorkSpaces Regions Availability Zones Content Delivery POPs Storage GatewayS3 EBS Glacier Import/Export DynamoDB ElastiCache StorageCompute Databases RDS MySQL, PostgreSQL Oracle, SQL Server Elastic Load BalancerEC2 Auto Scaling
- 18. Direct Connect Route 53VPC Networking Analytics Data PipelineRedshiftEMR Kinesis SWFSNS SQS CloudSearchSES AppStreamCloudFront Application Services WorkSpaces Regions Availability Zones Content Delivery POPs Storage GatewayS3 EBS Glacier Import/Export DynamoDB ElastiCache StorageCompute Databases RDS MySQL, PostgreSQL Oracle, SQL Server Elastic Load BalancerEC2 Auto Scaling
- 19. Direct Connect Route 53VPC Networking Analytics Data PipelineRedshiftEMR Kinesis SWFSNS SQS CloudSearchSES AppStreamCloudFront Application Services WorkSpaces Regions Availability Zones Content Delivery POPs Storage GatewayS3 EBS Glacier Import/Export DynamoDB ElastiCache StorageCompute Databases RDS MySQL, PostgreSQL Oracle, SQL Server Elastic Load BalancerEC2 Auto Scaling
- 20. OpsWorks CloudFormation Management & AdministrationIAM CloudWatchCloudTrail APIs and SDKsManagement ConsoleCloud HSM Command Line Interface Direct Connect Route 53VPC Networking Analytics Data PipelineRedshiftEMR Kinesis SWFSNS SQS CloudSearchSES AppStreamCloudFront Application Services WorkSpaces Regions Availability Zones Content Delivery POPs Storage GatewayS3 EBS Glacier Import/Export DynamoDB ElastiCache StorageCompute Databases RDS MySQL, PostgreSQL Oracle, SQL Server Elastic Load BalancerEC2 Auto Scaling
- 21. Elastic Beanstalk for Java, Node.js, Python, Ruby, PHP and .Net OpsWorks CloudFormationContainers & Deployment (PaaS) Management & AdministrationIAM CloudWatchCloudTrail APIs and SDKsManagement ConsoleCloud HSM Command Line Interface Direct Connect Route 53VPC Networking Analytics Data PipelineRedshiftEMR Kinesis SWFSNS SQS CloudSearchSES AppStreamCloudFront Application Services WorkSpaces Regions Availability Zones Content Delivery POPs Storage GatewayS3 EBS Glacier Import/Export DynamoDB ElastiCache StorageCompute Databases RDS MySQL, PostgreSQL Oracle, SQL Server Elastic Load BalancerEC2 Auto Scaling
- 22. Technology Partners Consulting Partners AWS MarketplaceEcosystem Elastic Beanstalk for Java, Node.js, Python, Ruby, PHP and .Net OpsWorks CloudFormationContainers & Deployment (PaaS) Management & AdministrationIAM CloudWatchCloudTrail APIs and SDKsManagement ConsoleCloud HSM Command Line Interface Direct Connect Route 53VPC Networking Analytics Data PipelineRedshiftEMR Kinesis SWFSNS SQS CloudSearchSES AppStreamCloudFront Application Services WorkSpaces Regions Availability Zones Content Delivery POPs Storage GatewayS3 EBS Glacier Import/Export DynamoDB ElastiCache StorageCompute Databases RDS MySQL, PostgreSQL Oracle, SQL Server Elastic Load BalancerEC2 Auto Scaling
- 23. Support CertificationTrainingProfessional Services Technology Partners Consulting Partners AWS MarketplaceEcosystem Elastic Beanstalk for Java, Node.js, Python, Ruby, PHP and .Net OpsWorks CloudFormationContainers & Deployment (PaaS) Management & AdministrationIAM CloudWatchCloudTrail APIs and SDKsManagement ConsoleCloud HSM Command Line Interface Direct Connect Route 53VPC Networking Analytics Data PipelineRedshiftEMR Kinesis SWFSNS SQS CloudSearchSES AppStreamCloudFront Application Services WorkSpaces Regions Availability Zones Content Delivery POPs Storage GatewayS3 EBS Glacier Import/Export DynamoDB ElastiCache StorageCompute Databases RDS MySQL, PostgreSQL Oracle, SQL Server Elastic Load BalancerEC2 Auto Scaling
- 25. AWS CLI View Autocomplete : Service Methods Autocomplete : Service Names
- 26. AWS CLI View Inline Contextual Help Autocomplete : Method Parameters
- 27. SDK View
- 28. Platform Overview Why AWS for development & test? AWS services that can be employed Common dev-test patterns Security and Billing Agenda
- 30. Disposable Numerous Projects start and stop frequently Required for short period of time Golden masters preferred
- 31. Disposable Numerous Projects start and stop frequently Required for short period of time Golden masters preferred Many environments support an app Preservation of known configurations Overlapping development cycles
- 32. Projects start and stop frequently Required for short period of time Golden masters preferred Many environments support an app Preservation of known configurations Overlapping development cycles Development & test in the cloud
- 33. Development & test in the cloud Take lots of it when you need it Unlimited elastic capacity
- 34. Development & test in the cloud Take lots of it when you need it Throw it away when you don’t Unlimited elastic capacity Cost optimization
- 35. Development & test in the cloud Preserve it for future reference Take lots of it when you need it Throw it away when you don’t Unlimited elastic capacity Cost optimization Durable imaging & storage
- 36. Connect over dedicated links…
- 38. Platform Overview Why AWS for development & test? AWS services that can be employed Common dev-test patterns Security and Billing Agenda
- 39. VPC VPC Public Subnet VPC Private Subnet NAT Instance Public: 54.200.129.18 Private: 10.1.1.11 /24 Web Server Public: 54.200.129.29 Private: 10.1.1.12 /24 Database Server Private: 10.1.10.3 /24 Database Server Private: 10.1.10.4 /24 Database Server Private: 10.1.10.5 /24 Route Table Destination Target 10.1.0.0/16 local 0.0.0.0/0 igw AWS Public API Endpoints VPC 10.1.0.0/16 VPN or Direct Connect Route Table Destination Target 10.1.0.0/16 local 172.16.0.0/8 vgw 0.0.0.0/0 NAT IGW VGW CGW
- 40. Convenience Control Higher-level services Do it yourself AWS Elastic Beanstalk AWS OpsWorks AWS CloudFormation AWS CodeDeploy Deployment & Configuration
- 41. Higher-level Services Stack through defining layers Layers Load balancing HA Proxy installation in an availability zone Application container Static, Node.js, Rails, PHP Database layer MySQL Stack Layers ManagementApps Elastic Beanstalk OpsWorks Elastic Beanstalk for Java, Node.js, Python, Ruby, PHP and .Net OpsWorks CloudFormationContainers & Deployment (PaaS)
- 42. Amazon Route 53 Elastic Load Balancer S3 BucketCloudFront Distribution Web Servers Web Servers Web ASG Elastic Beanstalk App App Master Standby RR 1 RR 2 RR 3 RR 4 ElastiCache Cluster This is a stack Explicit Blue Printing with CloudFormation Elastic Beanstalk for Java, Node.js, Python, Ruby, PHP and .Net OpsWorks CloudFormationContainers & Deployment (PaaS)
- 43. This is a STACK. JavaScript Object Notation ( JSON ) A template of your datacenter / workload. Your infrastructure as code. Headers Parameters Mappings Resources Outputs Git Subversion Mercurial Dev Test Prod Elastic Beanstalk for Java, Node.js, Python, Ruby, PHP and .Net OpsWorks CloudFormationContainers & Deployment (PaaS)
- 44. Cloud Formation is a great Cookie Cutter Your infrastructure as code. Elastic Beanstalk for Java, Node.js, Python, Ruby, PHP and .Net OpsWorks CloudFormationContainers & Deployment (PaaS)
- 45. Cloud Formation is context aware Your infrastructure as code. Create: PROD dev.mysite.com test.mysite.com prod.mysite.com Create: TESTCreate: DEV Elastic Beanstalk for Java, Node.js, Python, Ruby, PHP and .Net OpsWorks CloudFormationContainers & Deployment (PaaS)
- 46. SOURCE CODE REPOSITORY DNS CONTINUOUS INTEGRATION SERVER PROJECT MANAGEMENT SERVER BUILDS Your infrastructure as code. Use Cloud Formation in Continuous Integration (CI)
- 47. Platform Overview Why AWS for development & test? AWS services that can be employed Common dev-test patterns Security and Billing Agenda
- 49. SOURCE CODE REPOSITORY PM SERVER CI DEVELOPER PICK TASKS SUBMIT CODE CHECKSTYLE BINARIES & PACKAGES & DOCS CONFIG MANAGEMENT SCRIPTS CLOUDFORMATION AMIS or CONTAINERS CODE COVERAGE STATIC SECURITY ANALYSIS STATIC INFRASTRUCTURE SECURITY ANALYSIS CONTINUOUS ASSURANCE Delivery to Deploy
- 50. Our deployment automation flow: Code Repository
- 53. Code Repository CI Infra CI SaaS Code Bundler Our deployment automation flow:
- 54. Code Repository CI Infra CI SaaS Code Bundler Deploy Object Amazon S3 Bucket Our deployment automation flow:
- 55. Code Repository CI Infra CI SaaS Code Bundler Deploy Object Amazon S3 Bucket Our deployment automation flow:
- 56. Code Repository CI Infra CI SaaS Code Bundler Deploy Object Amazon S3 Bucket AWS OpsWorks Our deployment automation flow:
- 57. Code Repository CI Infra CI SaaS Code Bundler Deploy Object Amazon S3 Bucket AWS OpsWorks Dev Web/App Servers Our deployment automation flow:
- 58. Code Repository CI Infra CI SaaS Code Bundler Deploy Object Amazon S3 Bucket AWS OpsWorks Dev Web/App Servers Dev/ QA Users Our deployment automation flow:
- 59. Create environments to support specific test types Testing at scale Unit & regression Scale up and parallel run unit and regression plans in a fraction of the time Load & performance Utilize spot market for generating load and test how applications perform with auto- scaling A/B Run A/B scenario testing with replica stacks Security Create sandboxes for aggressive security testing
- 60. 1 instance for 100 hours = 100 instances for 1 hour
- 61. 100 Small instances 1 hour In Sydney On demand = $10
- 63. Load & Performance Test
- 64. Bees with Machine Guns EC2 Instance Contents Instances EC2 Instance Contents Amazon CloudWatch #1 Spawns #3 Monitors My App github.com/newsapps/beeswithmachineguns
- 65. Creates portfolio Adds constraints and grant access 1 4 5 Administrator Portfolio Users Browse Products 6Launch ProductsAWS CloudFormation template Creates product3Authors template2 ProductX ProductY ProductZ 7 Deploys stacks Notifications Notifications 88 AWS Service Catalog
- 66. Platform Overview Why AWS for development & test? AWS services that can be employed Common dev-test patterns Security and Billing Agenda
- 67. Control access and segregate duties everywhere With AWS IAM you get to control who can do what in your AWS environment and from where Fine-grained control of your AWS cloud with two- factor authentication Integrated with your existing corporate directory using SAML 2.0 and single sign-on AWS account owner Network management Security management Server management Storage management
- 68. You are making API calls... On a growing set of services … CloudTrail is continuously recording API calls… And delivering log files to you CloudTrail
- 69. Division B admin@divisionB.com User2 Dev2 Admin2 IAM Tags: Own=Div Proj=P Tags: Own=Div Proj=Q Tags: Own=Div Proj=R Master Account aws.invoices@mycompany.com consolidated billing information Tags: (key- value) e.g Own=Div Proj=R Billing
- 70. Operating Co. A admin@opcoa.com User1 Dev1 Admin1 IAM Tags: Own=OpCo Proj=A Tags: Own=OpCo Proj=B Tags: Own=OpCo Proj=C Division B admin@divisionB.com User2 Dev2 Admin2 IAM Tags: Own=Div Proj=P Tags: Own=Div Proj=Q Tags: Own=Div Proj=R Business Unit C admin@busUnitC.com User3 Dev3 Admin3 IAM Tags: Own=BusC Proj=X Tags: Own=BusC Proj=Y Tags: Own=BusC Proj=Z Master Account aws.invoices@mycompany.com consolidated billing information Billing Alerts
- 71. The cloud makes development & test easy You can make extensive savings by leveraging elasticity Provides unique toolsets to help you create and manage environments Let’s you perform at scale and agility beyond traditional physical environments
- 72. discussion…
Editor's Notes
- Lots of projects that start and stop. Had ot reuse environments to get efficiencies for the investment in the hardware they run on. When you terminate an environment, you can persists the storage, but stop paying for compute. Differenent environment for every project, and get started immediately. No lead time. Repurpose dev environment for a new project. Change, re-gear - pay for the time for this to happen.
- Build websites that sleep at night. Build machines only live when you need it. Supercomput erin the hands of every dev.
- Customer Network on the upper right Internet on the upper left VPC below both of those Public Subnet and routing Private Subnet and Routing NAT to AWS APIs VPN Connection
- Discuss lightly pros/cons of each. Elastic Beanstalk is easiest to start with, but offers less control. Opsworks gives you more tools, with a bit more work on your part. CloudFormation is a template driven tool with its own language, so a bit of a learning curve, but very very powerful. Lastly you could do all this manually, but at scale its nearly impossible without a huge team.
- How does CloudFormation work? Let’s take a look at a system built in AWS. This entire system is considered the stack.
- CloudFormation is this stack distilled into a template file.
- Segregate roles and responsibilities to a fine-grained level that is probably in excess of what you can do in a physical environment User A can change firewalls tagged ‘development’ only User B can snapshot database storage volumes, but cannot access those volumes All of this can be federated back to the existing enterprise directory – you do not need to setup a new directory within AWS. Your users sign into the existing directory (using existing authentication and MFA solutions), then are granted a temporary role within AWS to perform whatever duty they have been allocated. This role exists for a configurable period of time. One even more awesome feature is that your EC2 instances themselves can have roles within identity and access management, to restrict what AWS APIs the instance can call. Thus, an attacker cannot use an EC2 instance to upload data to S3, if the role assigned to the instance does not include S3 API permissions. AWS IAM also includes full multi-factor authentication for users, using either hardware Gemalto tokens, or soft tokens running on all three major phone platforms.
- Segregate roles and responsibilities to a fine-grained level that is probably in excess of what you can do in a physical environment User A can change firewalls tagged ‘development’ only User B can snapshot database storage volumes, but cannot access those volumes All of this can be federated back to the existing enterprise directory – you do not need to setup a new directory within AWS. Your users sign into the existing directory (using existing authentication and MFA solutions), then are granted a temporary role within AWS to perform whatever duty they have been allocated. This role exists for a configurable period of time. One even more awesome feature is that your EC2 instances themselves can have roles within identity and access management, to restrict what AWS APIs the instance can call. Thus, an attacker cannot use an EC2 instance to upload data to S3, if the role assigned to the instance does not include S3 API permissions. AWS IAM also includes full multi-factor authentication for users, using either hardware Gemalto tokens, or soft tokens running on all three major phone platforms.
- Extension: SQS for queued builds
- Full deve ennv with source control and devel workstations.
- Vertical scaling on commodity hardware. Perfect for Hadoop.
- Vertical scaling on commodity hardware. Perfect for Hadoop. 100 instances in Syd for 1 hour is $10. Dexact copy of production, performance regression testing.
- Each autoscaling group uses a different set of AMI AMI-B contains the test code.
- Enterprises segregate important duties to reduce risk of accidental or malicious changes AWS allows fine-grained segregation across virtually all aspects of the service For example, you can segregate Who can change network configuration Who can change firewalls Who can change how the VPC connects to the Internet or back to your corporate premises Who can start and stop servers Who can snapshot and restore storage volumes AWS IAM offers a programatic level of control and granularity that would not be possible to implement in traditional on-premise environments
- CloudTrail is your eyes behind the scenes at AWS. It gives you insight into all of the API calls made which are associated with your account(s). It lets you understand the who did what from where, when. Just a few weeks ago, we added the ability for CloudTrail to record both successful and unsuccessful console logins from your AWS IAM accounts as well.