Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Configuring Highly Scalable Compile Masters, Vasco Cardoso, AWS

32 views

Published on

Scaling Puppet Enterprise with Compile Masters requires you to provision new machines and manually configure them, as well as your Puppet Master server.
Learn how you can automatically provision and configure new Compile Master nodes for your AWS Opsworks for Puppet Enterprise server by leveraging AWS Systems Manager

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Configuring Highly Scalable Compile Masters, Vasco Cardoso, AWS

  1. 1. © 2020, Amazon Web Services, Inc. or its Affiliates. Vasco Cardoso, Solutions Architect, AWS 05.05.20 Configuring Highly Scalable Compile Masters with AWS OpsWorks for Puppet Enterprise
  2. 2. © 2020, Amazon Web Services, Inc. or its Affiliates. Puppet Scaling Vertical Scaling CPU Memory Max 4,000 nodes Horizontal Scaling Add Compile Masters 1.5k – 3k nodes / each
  3. 3. © 2020, Amazon Web Services, Inc. or its Affiliates. Install Compile Masters Infrastructure VM to run CM Load Balancer Configuration New Node needs to be classified as a Compile Master Communication needs to be routed correctly
  4. 4. © 2020, Amazon Web Services, Inc. or its Affiliates. Step by Step • Create Load Balancer • Configure load balancer to allow communication on ports 8140 and 8142 • Create new machine to serve as compile master • Configure load balancer to forward traffic to new compile master • SSH into future Compile master node and install the agent using the dns_alt_name flag • SSH into Master of Masters to sign the certificate • Log into the console and pin the compile master node to the PE Master node group • Run puppet on the Compile Master • Run puppet on Master of Masters • Configure the agent install requests to be sent to the load balancer • Run puppet on the compile masters • Run puppet on Master of Masters • Configure infrastructure agents to connect the orchestration agents to the Master of Masters • Run puppet on all nodes under the PE Infrastructure Group • Configure agents to connect orchestration agents to the load balancer • Run puppet on the Master of Masters • Run puppet on all the nodes, or re-install the agent Compile Master Infrastructure Master of Masters Configuration
  5. 5. © 2020, Amazon Web Services, Inc. or its Affiliates. Automation – Puppet Master AWS OpsWorks for Puppet Enterprise Fully managed Puppet Enterprise server Automatically: • Patched • Updated • Backed Up
  6. 6. © 2020, Amazon Web Services, Inc. or its Affiliates. Automation - Infrastructure • Load Balancer • Elastic Load Balancing– Network Load Balancer • Compile Master • Amazon EC2 Instance • Build it all • AWS CloudFormation
  7. 7. © 2020, Amazon Web Services, Inc. or its Affiliates. Step by Step – Now with (some) automation • Create Load Balancer • Configure load balancer to allow communication on ports 8140 and 8142 • Create new machine to serve as compile master • Configure load balancer to forward traffic to new compile master • SSH into future Compile master node and install the agent using the dns_alt_name flag • SSH into MoM to sign the certificate • Log into the console and pin the compile master node to the PE Master node group • Run puppet on the compile master • Run puppet on MoM • Configure the agent install requests to be sent to the load balancer • Run puppet on the compile masters • Run puppet on MoM • Configure infrastructure agents to connect the orchestration agents to the MoM • Run puppet on all nodes under the PE Infrastructure Group • Configure agents to connect orchestration agents to the load balancer • Run puppet on the master • Run puppet on all the nodes, or re-install the agent Compile Master Infrastructure Master of Masters Configuration
  8. 8. © 2020, Amazon Web Services, Inc. or its Affiliates. Automation - Configuration Master of Masters Configuration Node classifier service API Requires sharing of credentials Username/Password SSH Keys AWS Systems Manager Run commands without SSH
  9. 9. © 2020, Amazon Web Services, Inc. or its Affiliates. Automation – Step by Step • Create Load Balancer • Configure load balancer to allow communication on ports 8140 and 8142 • Create new machine to serve as compile master • Configure load balancer to forward traffic to new compile master • SSH into future Compile master node and install the agent using the dns_alt_name flag • SSH into MoM to sign the certificate • Log into the console and pin the compile master node to the PE Master node group • Run puppet on the compile master • Run puppet on MoM • Configure the agent install requests to be sent to the load balancer • Run puppet on the compile masters • Run puppet on MoM • Configure infrastructure agents to connect the orchestration agents to the MoM • Run puppet on all nodes under the PE Infrastructure Group • Configure agents to connect orchestration agents to the load balancer • Run puppet on the master • Run puppet on all the nodes, or re-install the agent Compile Master Infrastructure Master of Masters Configuration • Launch CloudFormation Stack
  10. 10. © 2020, Amazon Web Services, Inc. or its Affiliates. Automation Dive Deep – ELB • Dynamic registration of targets (Compile Masters) • Only allows traffic ports used by the Puppet and the Orchestration Agents • Health Checks • Managed service
  11. 11. © 2020, Amazon Web Services, Inc. or its Affiliates. Automation Deep Dive – Compile Master • Bootstrap and update configuration at launch • Puppet Agent installation includes dns_alt_names flag • Certificate signed automatically • Compile Master automatically pinned to the PE Master group
  12. 12. © 2020, Amazon Web Services, Inc. or its Affiliates. Automation Deep Dive – Autoscaling Group • Scales the number of Compile Masters in and out • Based on metrics • Via a dedicated trigger • Replaces unhealthy Compile Masters automatically • Automatic registration with Load Balancer
  13. 13. © 2020, Amazon Web Services, Inc. or its Affiliates. Automation Deep Dive – Master of Masters • Configure compile master pool address (load balancer address) • Validate change • Run Puppet • Configure PCP Broker for Infrastructure Agents • Validate change • Run Puppet • Configure PCP Broker for Agents • Validate change • Run Puppet
  14. 14. © 2020, Amazon Web Services, Inc. or its Affiliates. Benefits • Move away from manual processes • Built-in error detection and handling • Unattended scaling and replacement of Compile Masters • Replicate across environments
  15. 15. © 2020, Amazon Web Services, Inc. or its Affiliates. Demo
  16. 16. © 2020, Amazon Web Services, Inc. or its Affiliates.
  17. 17. © 2020, Amazon Web Services, Inc. or its Affiliates.
  18. 18. © 2020, Amazon Web Services, Inc. or its Affiliates.
  19. 19. © 2020, Amazon Web Services, Inc. or its Affiliates.
  20. 20. © 2020, Amazon Web Services, Inc. or its Affiliates.
  21. 21. © 2020, Amazon Web Services, Inc. or its Affiliates. Q&A
  22. 22. © 2020, Amazon Web Services, Inc. or its Affiliates. Thank you!

×