Successfully reported this slideshow.
Your SlideShare is downloading. ×

Dev/Test Environment Provisioning and Management on AWS

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Loading in …3
×

Check these out next

1 of 72 Ad
Advertisement

More Related Content

Slideshows for you (20)

Advertisement

Similar to Dev/Test Environment Provisioning and Management on AWS (20)

More from Shiva Narayanaswamy (18)

Advertisement

Recently uploaded (20)

Dev/Test Environment Provisioning and Management on AWS

  1. 1. Environment Provisioning and Management on AWS Shiva N– Solution Architect
  2. 2. Platform Overview Why AWS for development & test? AWS services that can be employed Common dev-test patterns Security and Billing Agenda
  3. 3. Platform Overview Why AWS for development & test? AWS services that can be employed Common dev-test patterns Security and Billing Agenda
  4. 4. Support CertificationTrainingProfessional Services Technology Partners Consulting Partners AWS MarketplaceEcosystem Elastic Beanstalk for Java, Node.js, Python, Ruby, PHP and .Net OpsWorks CloudFormationContainers & Deployment (PaaS) Management & AdministrationIAM CloudWatchCloudTrail APIs and SDKsManagement ConsoleCloud HSM Command Line Interface Direct Connect Route 53VPC Networking Analytics Data PipelineRedshiftEMR Kinesis SWFSNS SQS CloudSearchSES AppStreamCloudFront Application Services WorkSpaces Regions Availability Zones Content Delivery POPs Storage GatewayS3 EBS Glacier Import/Export DynamoDB ElastiCache StorageCompute Databases RDS MySQL, PostgreSQL Oracle, SQL Server Elastic Load BalancerEC2 Auto Scaling Platform Logical View : Block Diagram
  5. 5. Regions Availability Zones Content Delivery POPs Storage GatewayS3 EBS Glacier Import/Export DynamoDB ElastiCache StorageCompute Databases RDS MySQL, PostgreSQL Oracle, SQL Server Elastic Load BalancerEC2 Auto Scaling
  6. 6. Regions Availability Zones Content Delivery POPs Storage GatewayS3 EBS Glacier Import/Export DynamoDB ElastiCache StorageCompute Databases RDS MySQL, PostgreSQL Oracle, SQL Server Elastic Load BalancerEC2 Auto Scaling PV/HVM x86 Hourly Pricing - On Demand - Reserved L|M|H - Spot - Dedicated 15 Families 37 Instance Types
  7. 7. Regions Availability Zones Content Delivery POPs Storage GatewayS3 EBS Glacier Import/Export DynamoDB ElastiCache StorageCompute Databases RDS MySQL, PostgreSQL Oracle, SQL Server Elastic Load BalancerEC2 Auto Scaling Auto Scaling Regional HA SSL Offloading Loose Coupling Health Checks Connection Mgt
  8. 8. Regions Availability Zones Content Delivery POPs Storage GatewayS3 EBS Glacier Import/Export DynamoDB ElastiCache StorageCompute Databases RDS MySQL, PostgreSQL Oracle, SQL Server Elastic Load BalancerEC2 Auto Scaling
  9. 9. Regions Availability Zones Content Delivery POPs Storage GatewayS3 EBS Glacier Import/Export DynamoDB ElastiCache StorageCompute Databases RDS MySQL, PostgreSQL Oracle, SQL Server Elastic Load BalancerEC2 Auto Scaling
  10. 10. Regions Availability Zones Content Delivery POPs Storage GatewayS3 EBS Glacier Import/Export DynamoDB ElastiCache StorageCompute Databases RDS MySQL, PostgreSQL Oracle, SQL Server Elastic Load BalancerEC2 Auto Scaling
  11. 11. Regions Availability Zones Content Delivery POPs Storage GatewayS3 EBS Glacier Import/Export DynamoDB ElastiCache StorageCompute Databases RDS MySQL, PostgreSQL Oracle, SQL Server Elastic Load BalancerEC2 Auto Scaling
  12. 12. Regions Availability Zones Content Delivery POPs Storage GatewayS3 EBS Glacier Import/Export DynamoDB ElastiCache StorageCompute Databases RDS MySQL, PostgreSQL Oracle, SQL Server Elastic Load BalancerEC2 Auto Scaling
  13. 13. Regions Availability Zones Content Delivery POPs Storage GatewayS3 EBS Glacier Import/Export DynamoDB ElastiCache StorageCompute Databases RDS MySQL, PostgreSQL Oracle, SQL Server Elastic Load BalancerEC2 Auto Scaling
  14. 14. Regions Availability Zones Content Delivery POPs Storage GatewayS3 EBS Glacier Import/Export DynamoDB ElastiCache StorageCompute Databases RDS MySQL, PostgreSQL Oracle, SQL Server Elastic Load BalancerEC2 Auto Scaling Yes SQL Amazon RDS In-Memory Amazon ElastiCache AWS Managed versus Customer Managed Database Options No SQL MPP DW Redshift Self Managed EC2
  15. 15. Direct Connect Route 53VPC Networking Regions Availability Zones Content Delivery POPs Storage GatewayS3 EBS Glacier Import/Export DynamoDB ElastiCache StorageCompute Databases RDS MySQL, PostgreSQL Oracle, SQL Server Elastic Load BalancerEC2 Auto Scaling
  16. 16. Direct Connect Route 53VPC Networking Analytics Data PipelineRedshiftEMR Kinesis SWFSNS SQS CloudSearchSES AppStreamCloudFront Application Services WorkSpaces Regions Availability Zones Content Delivery POPs Storage GatewayS3 EBS Glacier Import/Export DynamoDB ElastiCache StorageCompute Databases RDS MySQL, PostgreSQL Oracle, SQL Server Elastic Load BalancerEC2 Auto Scaling
  17. 17. Direct Connect Route 53VPC Networking Analytics Data PipelineRedshiftEMR Kinesis SWFSNS SQS CloudSearchSES AppStreamCloudFront Application Services WorkSpaces Regions Availability Zones Content Delivery POPs Storage GatewayS3 EBS Glacier Import/Export DynamoDB ElastiCache StorageCompute Databases RDS MySQL, PostgreSQL Oracle, SQL Server Elastic Load BalancerEC2 Auto Scaling
  18. 18. Direct Connect Route 53VPC Networking Analytics Data PipelineRedshiftEMR Kinesis SWFSNS SQS CloudSearchSES AppStreamCloudFront Application Services WorkSpaces Regions Availability Zones Content Delivery POPs Storage GatewayS3 EBS Glacier Import/Export DynamoDB ElastiCache StorageCompute Databases RDS MySQL, PostgreSQL Oracle, SQL Server Elastic Load BalancerEC2 Auto Scaling
  19. 19. Direct Connect Route 53VPC Networking Analytics Data PipelineRedshiftEMR Kinesis SWFSNS SQS CloudSearchSES AppStreamCloudFront Application Services WorkSpaces Regions Availability Zones Content Delivery POPs Storage GatewayS3 EBS Glacier Import/Export DynamoDB ElastiCache StorageCompute Databases RDS MySQL, PostgreSQL Oracle, SQL Server Elastic Load BalancerEC2 Auto Scaling
  20. 20. OpsWorks CloudFormation Management & AdministrationIAM CloudWatchCloudTrail APIs and SDKsManagement ConsoleCloud HSM Command Line Interface Direct Connect Route 53VPC Networking Analytics Data PipelineRedshiftEMR Kinesis SWFSNS SQS CloudSearchSES AppStreamCloudFront Application Services WorkSpaces Regions Availability Zones Content Delivery POPs Storage GatewayS3 EBS Glacier Import/Export DynamoDB ElastiCache StorageCompute Databases RDS MySQL, PostgreSQL Oracle, SQL Server Elastic Load BalancerEC2 Auto Scaling
  21. 21. Elastic Beanstalk for Java, Node.js, Python, Ruby, PHP and .Net OpsWorks CloudFormationContainers & Deployment (PaaS) Management & AdministrationIAM CloudWatchCloudTrail APIs and SDKsManagement ConsoleCloud HSM Command Line Interface Direct Connect Route 53VPC Networking Analytics Data PipelineRedshiftEMR Kinesis SWFSNS SQS CloudSearchSES AppStreamCloudFront Application Services WorkSpaces Regions Availability Zones Content Delivery POPs Storage GatewayS3 EBS Glacier Import/Export DynamoDB ElastiCache StorageCompute Databases RDS MySQL, PostgreSQL Oracle, SQL Server Elastic Load BalancerEC2 Auto Scaling
  22. 22. Technology Partners Consulting Partners AWS MarketplaceEcosystem Elastic Beanstalk for Java, Node.js, Python, Ruby, PHP and .Net OpsWorks CloudFormationContainers & Deployment (PaaS) Management & AdministrationIAM CloudWatchCloudTrail APIs and SDKsManagement ConsoleCloud HSM Command Line Interface Direct Connect Route 53VPC Networking Analytics Data PipelineRedshiftEMR Kinesis SWFSNS SQS CloudSearchSES AppStreamCloudFront Application Services WorkSpaces Regions Availability Zones Content Delivery POPs Storage GatewayS3 EBS Glacier Import/Export DynamoDB ElastiCache StorageCompute Databases RDS MySQL, PostgreSQL Oracle, SQL Server Elastic Load BalancerEC2 Auto Scaling
  23. 23. Support CertificationTrainingProfessional Services Technology Partners Consulting Partners AWS MarketplaceEcosystem Elastic Beanstalk for Java, Node.js, Python, Ruby, PHP and .Net OpsWorks CloudFormationContainers & Deployment (PaaS) Management & AdministrationIAM CloudWatchCloudTrail APIs and SDKsManagement ConsoleCloud HSM Command Line Interface Direct Connect Route 53VPC Networking Analytics Data PipelineRedshiftEMR Kinesis SWFSNS SQS CloudSearchSES AppStreamCloudFront Application Services WorkSpaces Regions Availability Zones Content Delivery POPs Storage GatewayS3 EBS Glacier Import/Export DynamoDB ElastiCache StorageCompute Databases RDS MySQL, PostgreSQL Oracle, SQL Server Elastic Load BalancerEC2 Auto Scaling
  24. 24. Service Console View
  25. 25. AWS CLI View Autocomplete : Service Methods Autocomplete : Service Names
  26. 26. AWS CLI View Inline Contextual Help Autocomplete : Method Parameters
  27. 27. SDK View
  28. 28. Platform Overview Why AWS for development & test? AWS services that can be employed Common dev-test patterns Security and Billing Agenda
  29. 29. Common traits Disposable Numerous
  30. 30. Disposable Numerous Projects start and stop frequently Required for short period of time Golden masters preferred
  31. 31. Disposable Numerous Projects start and stop frequently Required for short period of time Golden masters preferred Many environments support an app Preservation of known configurations Overlapping development cycles
  32. 32. Projects start and stop frequently Required for short period of time Golden masters preferred Many environments support an app Preservation of known configurations Overlapping development cycles Development & test in the cloud
  33. 33. Development & test in the cloud Take lots of it when you need it Unlimited elastic capacity
  34. 34. Development & test in the cloud Take lots of it when you need it Throw it away when you don’t Unlimited elastic capacity Cost optimization
  35. 35. Development & test in the cloud Preserve it for future reference Take lots of it when you need it Throw it away when you don’t Unlimited elastic capacity Cost optimization Durable imaging & storage
  36. 36. Connect over dedicated links…
  37. 37. Platform Overview Why AWS for development & test? AWS services that can be employed Common dev-test patterns Security and Billing Agenda
  38. 38. VPC VPC Public Subnet VPC Private Subnet NAT Instance Public: 54.200.129.18 Private: 10.1.1.11 /24 Web Server Public: 54.200.129.29 Private: 10.1.1.12 /24 Database Server Private: 10.1.10.3 /24 Database Server Private: 10.1.10.4 /24 Database Server Private: 10.1.10.5 /24 Route Table Destination Target 10.1.0.0/16 local 0.0.0.0/0 igw AWS Public API Endpoints VPC 10.1.0.0/16 VPN or Direct Connect Route Table Destination Target 10.1.0.0/16 local 172.16.0.0/8 vgw 0.0.0.0/0 NAT IGW VGW CGW
  39. 39. Convenience Control Higher-level services Do it yourself AWS Elastic Beanstalk AWS OpsWorks AWS CloudFormation AWS CodeDeploy Deployment & Configuration
  40. 40. Higher-level Services Stack through defining layers Layers Load balancing HA Proxy installation in an availability zone Application container Static, Node.js, Rails, PHP Database layer MySQL Stack Layers ManagementApps Elastic Beanstalk OpsWorks Elastic Beanstalk for Java, Node.js, Python, Ruby, PHP and .Net OpsWorks CloudFormationContainers & Deployment (PaaS)
  41. 41. Amazon Route 53 Elastic Load Balancer S3 BucketCloudFront Distribution Web Servers Web Servers Web ASG Elastic Beanstalk App App Master Standby RR 1 RR 2 RR 3 RR 4 ElastiCache Cluster This is a stack Explicit Blue Printing with CloudFormation Elastic Beanstalk for Java, Node.js, Python, Ruby, PHP and .Net OpsWorks CloudFormationContainers & Deployment (PaaS)
  42. 42. This is a STACK. JavaScript Object Notation ( JSON ) A template of your datacenter / workload. Your infrastructure as code. Headers Parameters Mappings Resources Outputs Git Subversion Mercurial Dev Test Prod Elastic Beanstalk for Java, Node.js, Python, Ruby, PHP and .Net OpsWorks CloudFormationContainers & Deployment (PaaS)
  43. 43. Cloud Formation is a great Cookie Cutter Your infrastructure as code. Elastic Beanstalk for Java, Node.js, Python, Ruby, PHP and .Net OpsWorks CloudFormationContainers & Deployment (PaaS)
  44. 44. Cloud Formation is context aware Your infrastructure as code. Create: PROD dev.mysite.com test.mysite.com prod.mysite.com Create: TESTCreate: DEV Elastic Beanstalk for Java, Node.js, Python, Ruby, PHP and .Net OpsWorks CloudFormationContainers & Deployment (PaaS)
  45. 45. SOURCE CODE REPOSITORY DNS CONTINUOUS INTEGRATION SERVER PROJECT MANAGEMENT SERVER BUILDS Your infrastructure as code. Use Cloud Formation in Continuous Integration (CI)
  46. 46. Platform Overview Why AWS for development & test? AWS services that can be employed Common dev-test patterns Security and Billing Agenda
  47. 47. SOURCE CODE REPOSITORY PM SERVER CI DEVELOPER PICK TASKS SUBMIT CODE CHECKSTYLE BINARIES & PACKAGES & DOCS CONFIG MANAGEMENT SCRIPTS CLOUDFORMATION AMIS or CONTAINERS CODE COVERAGE STATIC SECURITY ANALYSIS STATIC INFRASTRUCTURE SECURITY ANALYSIS CONTINUOUS ASSURANCE Delivery to Deploy
  48. 48. Our deployment automation flow: Code Repository
  49. 49. Code Repository CI Infra CI SaaS Our deployment automation flow:
  50. 50. Code Repository CI Infra CI SaaS Our deployment automation flow:
  51. 51. Code Repository CI Infra CI SaaS Code Bundler Our deployment automation flow:
  52. 52. Code Repository CI Infra CI SaaS Code Bundler Deploy Object Amazon S3 Bucket Our deployment automation flow:
  53. 53. Code Repository CI Infra CI SaaS Code Bundler Deploy Object Amazon S3 Bucket Our deployment automation flow:
  54. 54. Code Repository CI Infra CI SaaS Code Bundler Deploy Object Amazon S3 Bucket AWS OpsWorks Our deployment automation flow:
  55. 55. Code Repository CI Infra CI SaaS Code Bundler Deploy Object Amazon S3 Bucket AWS OpsWorks Dev Web/App Servers Our deployment automation flow:
  56. 56. Code Repository CI Infra CI SaaS Code Bundler Deploy Object Amazon S3 Bucket AWS OpsWorks Dev Web/App Servers Dev/ QA Users Our deployment automation flow:
  57. 57. Create environments to support specific test types Testing at scale Unit & regression Scale up and parallel run unit and regression plans in a fraction of the time Load & performance Utilize spot market for generating load and test how applications perform with auto- scaling A/B Run A/B scenario testing with replica stacks Security Create sandboxes for aggressive security testing
  58. 58. 1 instance for 100 hours = 100 instances for 1 hour
  59. 59. 100 Small instances 1 hour In Sydney On demand = $10
  60. 60. Load & Performance Test
  61. 61. Bees with Machine Guns EC2 Instance Contents Instances EC2 Instance Contents Amazon CloudWatch #1 Spawns #3 Monitors My App github.com/newsapps/beeswithmachineguns
  62. 62. Creates portfolio Adds constraints and grant access 1 4 5 Administrator Portfolio Users Browse Products 6Launch ProductsAWS CloudFormation template Creates product3Authors template2 ProductX ProductY ProductZ 7 Deploys stacks Notifications Notifications 88 AWS Service Catalog
  63. 63. Platform Overview Why AWS for development & test? AWS services that can be employed Common dev-test patterns Security and Billing Agenda
  64. 64. Control access and segregate duties everywhere With AWS IAM you get to control who can do what in your AWS environment and from where Fine-grained control of your AWS cloud with two- factor authentication Integrated with your existing corporate directory using SAML 2.0 and single sign-on AWS account owner Network management Security management Server management Storage management
  65. 65. You are making API calls... On a growing set of services … CloudTrail is continuously recording API calls… And delivering log files to you CloudTrail
  66. 66. Division B admin@divisionB.com User2 Dev2 Admin2 IAM Tags: Own=Div Proj=P Tags: Own=Div Proj=Q Tags: Own=Div Proj=R Master Account aws.invoices@mycompany.com consolidated billing information Tags: (key- value) e.g Own=Div Proj=R Billing
  67. 67. Operating Co. A admin@opcoa.com User1 Dev1 Admin1 IAM Tags: Own=OpCo Proj=A Tags: Own=OpCo Proj=B Tags: Own=OpCo Proj=C Division B admin@divisionB.com User2 Dev2 Admin2 IAM Tags: Own=Div Proj=P Tags: Own=Div Proj=Q Tags: Own=Div Proj=R Business Unit C admin@busUnitC.com User3 Dev3 Admin3 IAM Tags: Own=BusC Proj=X Tags: Own=BusC Proj=Y Tags: Own=BusC Proj=Z Master Account aws.invoices@mycompany.com consolidated billing information Billing Alerts
  68. 68. The cloud makes development & test easy You can make extensive savings by leveraging elasticity Provides unique toolsets to help you create and manage environments Let’s you perform at scale and agility beyond traditional physical environments
  69. 69. discussion…

Editor's Notes

  • Lots of projects that start and stop. Had ot reuse environments to get efficiencies for the investment in the hardware they run on.
    When you terminate an environment, you can persists the storage, but stop paying for compute.
    Differenent environment for every project, and get started immediately. No lead time.
    Repurpose dev environment for a new project.
    Change, re-gear - pay for the time for this to happen.
  • Build websites that sleep at night. Build machines only live when you need it. Supercomput erin the hands of every dev.
  • Customer Network on the upper right
    Internet on the upper left
    VPC below both of those

    Public Subnet and routing
    Private Subnet and Routing
    NAT to AWS APIs
    VPN Connection
  • Discuss lightly pros/cons of each.

    Elastic Beanstalk is easiest to start with, but offers less control. Opsworks gives you more tools, with a bit more work on your part. CloudFormation is a template driven tool with its own language, so a bit of a learning curve, but very very powerful. Lastly you could do all this manually, but at scale its nearly impossible without a huge team.
  • How does CloudFormation work? Let’s take a look at a system built in AWS. This entire system is considered the stack.
  • CloudFormation is this stack distilled into a template file.
  • Segregate roles and responsibilities to a fine-grained level that is probably in excess of what you can do in a physical environment

    User A can change firewalls tagged ‘development’ only
    User B can snapshot database storage volumes, but cannot access those volumes

    All of this can be federated back to the existing enterprise directory – you do not need to setup a new directory within AWS. Your users sign into the existing directory (using existing authentication and MFA solutions), then are granted a temporary role within AWS to perform whatever duty they have been allocated. This role exists for a configurable period of time.

    One even more awesome feature is that your EC2 instances themselves can have roles within identity and access management, to restrict what AWS APIs the instance can call. Thus, an attacker cannot use an EC2 instance to upload data to S3, if the role assigned to the instance does not include S3 API permissions.

    AWS IAM also includes full multi-factor authentication for users, using either hardware Gemalto tokens, or soft tokens running on all three major phone platforms.
  • Segregate roles and responsibilities to a fine-grained level that is probably in excess of what you can do in a physical environment

    User A can change firewalls tagged ‘development’ only
    User B can snapshot database storage volumes, but cannot access those volumes

    All of this can be federated back to the existing enterprise directory – you do not need to setup a new directory within AWS. Your users sign into the existing directory (using existing authentication and MFA solutions), then are granted a temporary role within AWS to perform whatever duty they have been allocated. This role exists for a configurable period of time.

    One even more awesome feature is that your EC2 instances themselves can have roles within identity and access management, to restrict what AWS APIs the instance can call. Thus, an attacker cannot use an EC2 instance to upload data to S3, if the role assigned to the instance does not include S3 API permissions.

    AWS IAM also includes full multi-factor authentication for users, using either hardware Gemalto tokens, or soft tokens running on all three major phone platforms.
  • Extension: SQS for queued builds
  • Full deve ennv with source control and devel workstations.
  • Vertical scaling on commodity hardware. Perfect for Hadoop.
  • Vertical scaling on commodity hardware. Perfect for Hadoop. 100 instances in Syd for 1 hour is $10.
    Dexact copy of production, performance regression testing.
  • Each autoscaling group uses a different set of AMI
    AMI-B contains the test code.
  • Enterprises segregate important duties to reduce risk of accidental or malicious changes
    AWS allows fine-grained segregation across virtually all aspects of the service

    For example, you can segregate
    Who can change network configuration
    Who can change firewalls
    Who can change how the VPC connects to the Internet or back to your corporate premises
    Who can start and stop servers
    Who can snapshot and restore storage volumes

    AWS IAM offers a programatic level of control and granularity that would not be possible to implement in traditional on-premise environments
  • CloudTrail is your eyes behind the scenes at AWS. It gives you insight into all of the API calls made which are associated with your account(s). It lets you understand the who did what from where, when.

    Just a few weeks ago, we added the ability for CloudTrail to record both successful and unsuccessful console logins from your AWS IAM accounts as well.

×