Getting Started with PoolParty and EC2

Getting Started with PoolParty & EC2

PoolParty

PoolParty • PoolParty makes it easy and simple to conﬁgure any cloud of computers

PoolParty • PoolParty makes it easy and simple to conﬁgure any cloud of computers • Written in Ruby and Erlang

PoolParty

PoolParty • Not Ruby or Rails speciﬁc

PoolParty • Not Ruby or Rails speciﬁc • Not even tied to Amazon EC2

sample conﬁg

Amazon

EC2

EC2 S3

EC2 S3 EBS

EC2 S3 EBS AMI

EC2 S3 Elastic IP EBS AMI

EC2 S3 Elastic IP Keypairs EBS AMI

EC2 S3 Elastic IP Keypairs Security Groups EBS AMI

EC2 S3 Elastic IP Keypairs CloudFront Security Groups EBS AMI

EC2 S3 ??? AMI Elastic IP Keypairs CloudFront Security Groups EBS

Amazon: EC2

Amazon: EC2 • Platform to launch machine instances

Amazon: EC2 • Platform to launch machine instances • ec2-* (git-style) commands such as:

Amazon: EC2 • Platform to launch machine instances • ec2-* (git-style) commands such as: • ec2-run-instances

Amazon: EC2 • Platform to launch machine instances • ec2-* (git-style) commands such as: • ec2-run-instances • ec2-terminate-instances

Amazon: EC2 • Platform to launch machine instances • ec2-* (git-style) commands such as: • ec2-run-instances • ec2-terminate-instances • ec2-allocate-address

Amazon: EC2 • Platform to launch machine instances • ec2-* (git-style) commands such as: • ec2-run-instances • ec2-terminate-instances • ec2-allocate-address • etc.

Amazon: EC2 (AMIs)

Amazon: EC2 (AMIs) • AMI - Amazon Machine Image

Amazon: EC2 (AMIs) • AMI - Amazon Machine Image • Conceptually similar to a Xen or VMware image

Amazon: EC2 (AMIs) • AMI - Amazon Machine Image • Conceptually similar to a Xen or VMware image • Base to build on

Amazon: EC2 (AMIs) • AMI - Amazon Machine Image • Conceptually similar to a Xen or VMware image • Base to build on • Hundreds pre-built

Amazon: EC2 (AMIs) • AMI - Amazon Machine Image • Conceptually similar to a Xen or VMware image • Base to build on • Hundreds pre-built • Snapshots

Amazon: EC2 (AMIs) • AMI - Amazon Machine Image • Conceptually similar to a Xen or VMware image • Base to build on • Hundreds pre-built • Snapshots • PoolParty default is the widely used - ec2ubuntu ami-1cd73375

Amazon: EC2

Amazon: EC2 • Primarily uses ephemeral storage

Amazon: EC2 • Primarily uses ephemeral storage • e.g. node crashes = data gone (sortof)

Amazon: EC2 • Primarily uses ephemeral storage • e.g. node crashes = data gone (sortof) • requires new way to think about servers

Amazon: EC2 • Primarily uses ephemeral storage • e.g. node crashes = data gone (sortof) • requires new way to think about servers • forces redundancy and clustering issues to the front

Amazon: S3

Amazon: S3 • Simple Storage Service (S3)

Amazon: S3 • Simple Storage Service (S3) • Persistent

Amazon: S3 • Simple Storage Service (S3) • Persistent • Data stored in “buckets”

Amazon: S3 • Simple Storage Service (S3) • Persistent • Data stored in “buckets” • EC2 does not use S3 *directly*

Amazon: EBS

Amazon: EBS • Elastic Block Store

Amazon: EBS • Elastic Block Store • Persistent disks for EC2 instances

Amazon: EBS • Elastic Block Store • Persistent disks for EC2 instances • Can store snapshots on S3

Amazon: EBS • Elastic Block Store • Persistent disks for EC2 instances • Can store snapshots on S3 • Fast

Amazon: EBS • Elastic Block Store • Persistent disks for EC2 instances • Can store snapshots on S3 • Fast • Mounts to only one instance at a time (not a SAN)

Amazon: Elastic IPs

Amazon: Elastic IPs • Instance IP addresses are dynamic

Amazon: Elastic IPs • Instance IP addresses are dynamic • Elastic IPs are static

Amazon: Elastic IPs • Instance IP addresses are dynamic • Elastic IPs are static • Free to use, but $0.01/hr if unassociated with an instance

EC2 Security Groups

EC2 Security Groups • Amazon has a strict ﬁrewall

EC2 Security Groups • Amazon has a strict ﬁrewall • ec2-authorize

EC2 Security Groups • Amazon has a strict ﬁrewall • ec2-authorize • default security group nodes have unlimited (network) access to each other

EC2 Security Groups • Amazon has a strict ﬁrewall • ec2-authorize • default security group nodes have unlimited (network) access to each other • everything else is turned off

EC2 Security Groups ec2-add-group web -d \"Web server group\" ec2-authorize -p 80 web ec2-authorize -p 22 web

EC2 Security Groups ec2-add-group web -d \"Web server group\" ec2-authorize -p 80 web ec2-authorize -p 22 web ec2-authorize -P icmp -t -1:-1 web

Amazon: CloudFront

Amazon: CloudFront • Amazon’s “self service” CDN

Amazon: CloudFront • Amazon’s “self service” CDN • Pay-as-you-go

Amazon: CloudFront • Amazon’s “self service” CDN • Pay-as-you-go • Upload to S3

Amazon: CloudFront • Amazon’s “self service” CDN • Pay-as-you-go • Upload to S3 • Content is distributed & cached on Amazon’s machines

PoolParty

PoolParty overview

PoolParty overview • binary tools

PoolParty overview • binary tools • conﬁguration ﬁles

PoolParty overview • binary tools • conﬁguration ﬁles • erlang-driven messenger

PoolParty: Binary Tools

PoolParty: Binary Tools • pool-* binaries

PoolParty: Binary Tools • pool-* binaries • server-* binaries

PoolParty: Binary Tools Shell Example

PoolParty Conﬁguration

PoolParty Conﬁguration Shell Example a look at clouds/plugins/poolparty-mysql-plugin/mysql.rb

how does this actually work?

Puppet

Puppet • Puppet is a system for automating system administration tasks.

Puppet • Puppet is a system for automating system administration tasks. • Being used at Google to manage all Mac desktops and laptops, expanding into Linux clients.

Puppet • Puppet is a system for automating system administration tasks. • Being used at Google to manage all Mac desktops and laptops, expanding into Linux clients. • Red Hat - In the process of moving legacy systems onto Puppet.

Puppet

Puppet • Resources: ﬁles, folders, permissions, cron jobs, mounted directories, packages, ssh keys, services, arbitrary commands, etc.

Puppet • Resources: ﬁles, folders, permissions, cron jobs, mounted directories, packages, ssh keys, services, arbitrary commands, etc. • Mini-language

Puppet • Resources: ﬁles, folders, permissions, cron jobs, mounted directories, packages, ssh keys, services, arbitrary commands, etc. • Mini-language • Dependencies

Puppet • Resources: ﬁles, folders, permissions, cron jobs, mounted directories, packages, ssh keys, services, arbitrary commands, etc. • Mini-language • Dependencies • If/unless conditions

Puppet • Resources: ﬁles, folders, permissions, cron jobs, mounted directories, packages, ssh keys, services, arbitrary commands, etc. • Mini-language • Dependencies • If/unless conditions • Variables

Puppet

Puppet • Client/Server architecture

Puppet • Client/Server architecture • puppetmasterd

Puppet • Client/Server architecture • puppetmasterd • puppetd

Puppet: Language Example

Puppet

Puppet • Puppet base conﬁg: /etc/puppet/puppet.conf

Puppet • Puppet base conﬁg: /etc/puppet/puppet.conf • PoolParty’s Puppet conﬁg: /etc/puppet/manifests/classes/poolparty.pp

Puppet Conﬁguration Shell Example a look at /etc/puppet/manifests/classes/poolparty.pp

Puppet • Cron jobs: * 1 * * * . /etc/profile && puppetmasterd --verbose */15 * * * * /usr/bin/puppetrunner

Puppet

Puppet • Builds dependency graph

Puppet • Builds dependency graph • Provisions the machines (e.g. runs the commands)

Puppet Security

Puppet Security • Uses SSL certiﬁcates as the required and only form of authentication

Puppet Security • Uses SSL certiﬁcates as the required and only form of authentication • Master authenticates the certiﬁcates

Puppet Security • Uses SSL certificates as the required and only form of authentication • Master authenticates the certificates • Test certificate from node: /usr/sbin/puppetd --waitforcert 60 \\ --server master --test --verbose

Puppet Security • Uses SSL certificates as the required and only form of authentication • Master authenticates the certificates • Test certificate from node: /usr/sbin/puppetd --waitforcert 60 \\ --server master --test --verbose • If certificate doesn’t match, probably cached on server. Try on master: puppetca --clean node1.compute-1.internal

Puppet Cache

Puppet Cache • Keeps a cache of the parsed conﬁguration in: /var/lib/puppet/localconfig.yaml

Troubleshooting Puppet

Troubleshooting Puppet • tail -f /var/log/syslog

Troubleshooting Puppet • tail -f /var/log/syslog • Can the nodes contact master?

Troubleshooting Puppet • tail -f /var/log/syslog • Can the nodes contact master? • Do the certiﬁcates match?

Troubleshooting Puppet • tail -f /var/log/syslog • Can the nodes contact master? • Do the certiﬁcates match? • Does /var/lib/puppet/localconfig.yaml contain the right information?

Troubleshooting Puppet • tail -f /var/log/syslog • Can the nodes contact master? • Do the certiﬁcates match? • Does /var/lib/puppet/localconfig.yaml contain the right information? • Does /etc/puppet/manifests/classes/ contain the right information? poolparty.pp

Troubleshooting Puppet • tail -f /var/log/syslog • Can the nodes contact master? • Do the certiﬁcates match? • Does /var/lib/puppet/localconfig.yaml contain the right information? • Does /etc/puppet/manifests/classes/ contain the right information? poolparty.pp • Does contain the /var/poolparty/poolparty.pp right information? (more on that later)

Troubleshooting Puppet

Troubleshooting Puppet • server-provision -n app -i 1 \\ --slave --verbose --debug

master

PoolParty DNS on master root@master:~# cat /etc/hosts 127.0.0.1 localhost.localdomain localhost 127.0.0.1 master 75.101.128.147 master puppet localhost 10.252.166.18 node1.app node1 10.252.167.47 master.app master

PoolParty DNS on master root@master:~# cat /etc/hosts 127.0.0.1 localhost.localdomain localhost 127.0.0.1 master 75.101.128.147 master puppet localhost 10.252.166.18 node1.app node1 10.252.167.47 master.app master on node1 root@node1:~# cat /etc/hosts 127.0.0.1 localhost.localdomain localhost 127.0.0.1 node1 75.101.128.147 puppet master 10.252.167.47 master.app master 10.252.166.18 node1.app node1

PoolParty DNS • /etc/hosts modiﬁed by PoolParty when you cloud provision

HAProxy

HAProxy • Extremely fast

HAProxy • Extremely fast • Reports of HAProxy saturating gigabit ﬁber (http://is.gd/4myI)

HAProxy • Extremely fast • Reports of HAProxy saturating gigabit ﬁber (http://is.gd/4myI) • Reputation of being reliable and secure

HAProxy • Extremely fast • Reports of HAProxy saturating gigabit ﬁber (http://is.gd/4myI) • Reputation of being reliable and secure • Installed and setup by PoolParty by default

HAProxy

HAProxy • Conﬁg kept in: /etc/haproxy.cfg

HAProxy • Conﬁg kept in: /etc/haproxy.cfg • Reload conﬁg by: /etc/init.d/haproxy reload

HAProxy • View stats by: http://ec2-your-ip/poolparty

PoolParty Messenger

PoolParty Messenger • Three parts:

PoolParty Messenger • Three parts: • Master

PoolParty Messenger • Three parts: • Master • Node

PoolParty Messenger • Three parts: • Master • Node • Client

PoolParty Messenger • Three parts: • Master • Node • Client • “Instance Glue”

PoolParty Messenger • Three parts: • Master • Node • Client • “Instance Glue” • Master provisions nodes (will change)

PoolParty Messenger

PoolParty Messenger • Erlang - excellent node-to-node monitoring

PoolParty Messenger • Erlang - excellent node-to-node monitoring • Gathers load data

PoolParty Messenger • Erlang - excellent node-to-node monitoring • Gathers load data • Decides when to launch new nodes

PoolParty Messenger

PoolParty Messenger • Changes happening in architecture

PoolParty Messenger • Changes happening in architecture • Will become neighborhood-based instead of master/node-based

PoolParty Messenger • Changes happening in architecture • Will become neighborhood-based instead of master/node-based • The brains of the operation when humans aren’t around

Using PoolParty

Using PoolParty • cloud start

Using PoolParty • cloud start • cloud configure --verbose --debug

Using PoolParty • cloud start • cloud configure --verbose --debug • cloud provision --verbose --debug

Using PoolParty: What Happens

Using PoolParty: What Happens • PoolParty reads clouds.pool

Using PoolParty: What Happens • PoolParty reads clouds.pool • Generates templates, puppet conﬁgs, etc in a local storage directory (/tmp/poolparty)

Using PoolParty: What Happens • PoolParty reads clouds.pool • Generates templates, puppet conﬁgs, etc in a local storage directory (/tmp/poolparty) • Files are rsync’d to /var/poolparty on master

Using PoolParty: What Happens • PoolParty reads clouds.pool • Generates templates, puppet conﬁgs, etc in a local storage directory (/tmp/poolparty) • Files are rsync’d to /var/poolparty on master • A gang of bootstrapping tasks are run (via Capistrano)

Using PoolParty: What Happens • PoolParty reads clouds.pool • Generates templates, puppet conﬁgs, etc in a local storage directory (/tmp/poolparty) • Files are rsync’d to /var/poolparty on master • A gang of bootstrapping tasks are run (via Capistrano) • Puppet conﬁg is cp’d from /var/poolparty to /etc/ puppet/manifests/classes/poolparty.pp

Using PoolParty: What Happens • Puppet is started

Using PoolParty: What Happens • Puppet is started • Puppet tries to provision everything

Using PoolParty: What Happens • Puppet is started • Puppet tries to provision everything • Puppet starts PP Messenger master

Using PoolParty: What Happens • PP Messenger master contacts Amazon and boots more instances if needed.

Using PoolParty: What Happens • PP Messenger master contacts Amazon and boots more instances if needed. • PP Messenger bootstraps the new node

Using PoolParty: What Happens • PP Messenger master contacts Amazon and boots more instances if needed. • PP Messenger bootstraps the new node • The bootstrap process starts PP Messenger Node

Using PoolParty: What Happens • PP Messenger master contacts Amazon and boots more instances if needed. • PP Messenger bootstraps the new node • The bootstrap process starts PP Messenger Node • PP Messenger initiates Puppet provisioning on the new node

Using PoolParty: What Happens • PP Messenger master contacts Amazon and boots more instances if needed. • PP Messenger bootstraps the new node • The bootstrap process starts PP Messenger Node • PP Messenger initiates Puppet provisioning on the new node • time passes

Using PoolParty: What Happens • Puppet runs on master and master discovers a new node has been started.

Using PoolParty: What Happens • Puppet runs on master and master discovers a new node has been started. • In the case of HAProxy/Apache, Puppet re- generates the HAProxy conﬁg to include the new node.

Using PoolParty: What Happens • Puppet runs on master and master discovers a new node has been started. • In the case of HAProxy/Apache, Puppet re- generates the HAProxy conﬁg to include the new node. • As speciﬁed in poolparty.pp, when haproxy.cnf gets updated, it asks HAProxy to reload.

Using PoolParty: What Happens • Puppet runs on master and master discovers a new node has been started. • In the case of HAProxy/Apache, Puppet re- generates the HAProxy conﬁg to include the new node. • As speciﬁed in poolparty.pp, when haproxy.cnf gets updated, it asks HAProxy to reload. • HAProxy now starts distributing the load to the new node

Using PoolParty: What Happens • Similar conﬁg changes happen throughout the system

Real World Misc.

Real World Misc. • tail -f /var/log/syslog is your friend

Real World Misc. • tail -f /var/log/syslog is your friend • ssh into master: cloud ssh

Real World Misc. • tail -f /var/log/syslog is your friend • ssh into master: cloud ssh • ssh into i’th node: cloud ssh -i 1

Real World Misc.

Real World Misc. • scp scp -i /Users/me/.ec2/id_rsa-clouds_app \\ root@ec2-75-101-191-10.compute-1.amazonaws.com:/ \\ etc/snmp/snmpd.conf .

Real World Misc. • scp scp -i /Users/me/.ec2/id_rsa-clouds_app \\ root@ec2-75-101-191-10.compute-1.amazonaws.com:/ \\ etc/snmp/snmpd.conf . • rsync rsync -av -e \"ssh -i /var/poolparty/id_rsa-clouds_app\" \\ /var/www/cpohunterfan.com/documents node1:/var/www/cpohunterfan.com/

Workshop

Agenda:

Agenda: • Launch, login, terminate an EC2 instance

Agenda: • Launch, login, terminate an EC2 instance • into our instances and look around cloud ssh

Agenda: • Launch, login, terminate an EC2 instance • into our instances and look around cloud ssh • Take a closer look at clouds.pool

Agenda: • Launch, login, terminate an EC2 instance • into our instances and look around cloud ssh • Take a closer look at clouds.pool • Examine a basic PoolParty plugin

Getting Started with PoolParty and EC2

0 comments

Notes on slide 1

3 Favorites