Getting Started with PoolParty and EC2

1. Getting Started with PoolParty & EC2

2. PoolParty

3. PoolParty • PoolParty makes it easy and simple to conﬁgure any cloud of computers

4. PoolParty • PoolParty makes it easy and simple to conﬁgure any cloud of computers • Written in Ruby and Erlang

5. PoolParty

6. PoolParty • Not Ruby or Rails speciﬁc

7. PoolParty • Not Ruby or Rails speciﬁc • Not even tied to Amazon EC2

8. sample conﬁg

9. Amazon

10. EC2

11. EC2 S3

12. EC2 S3 EBS

13. EC2 S3 EBS AMI

14. EC2 S3 Elastic IP EBS AMI

15. EC2 S3 Elastic IP Keypairs EBS AMI

16. EC2 S3 Elastic IP Keypairs Security Groups EBS AMI

17. EC2 S3 Elastic IP Keypairs CloudFront Security Groups EBS AMI

18. EC2 S3 ??? AMI Elastic IP Keypairs CloudFront Security Groups EBS

19. Amazon: EC2

20. Amazon: EC2 • Platform to launch machine instances

21. Amazon: EC2 • Platform to launch machine instances • ec2-* (git-style) commands such as:

22. Amazon: EC2 • Platform to launch machine instances • ec2-* (git-style) commands such as: • ec2-run-instances

23. Amazon: EC2 • Platform to launch machine instances • ec2-* (git-style) commands such as: • ec2-run-instances • ec2-terminate-instances

24. Amazon: EC2 • Platform to launch machine instances • ec2-* (git-style) commands such as: • ec2-run-instances • ec2-terminate-instances • ec2-allocate-address

25. Amazon: EC2 • Platform to launch machine instances • ec2-* (git-style) commands such as: • ec2-run-instances • ec2-terminate-instances • ec2-allocate-address • etc.

26. Amazon: EC2 (AMIs)

27. Amazon: EC2 (AMIs) • AMI - Amazon Machine Image

28. Amazon: EC2 (AMIs) • AMI - Amazon Machine Image • Conceptually similar to a Xen or VMware image

29. Amazon: EC2 (AMIs) • AMI - Amazon Machine Image • Conceptually similar to a Xen or VMware image • Base to build on

30. Amazon: EC2 (AMIs) • AMI - Amazon Machine Image • Conceptually similar to a Xen or VMware image • Base to build on • Hundreds pre-built

31. Amazon: EC2 (AMIs) • AMI - Amazon Machine Image • Conceptually similar to a Xen or VMware image • Base to build on • Hundreds pre-built • Snapshots

32. Amazon: EC2 (AMIs) • AMI - Amazon Machine Image • Conceptually similar to a Xen or VMware image • Base to build on • Hundreds pre-built • Snapshots • PoolParty default is the widely used - ec2ubuntu ami-1cd73375

33. Amazon: EC2

34. Amazon: EC2 • Primarily uses ephemeral storage

35. Amazon: EC2 • Primarily uses ephemeral storage • e.g. node crashes = data gone (sortof)

36. Amazon: EC2 • Primarily uses ephemeral storage • e.g. node crashes = data gone (sortof) • requires new way to think about servers

37. Amazon: EC2 • Primarily uses ephemeral storage • e.g. node crashes = data gone (sortof) • requires new way to think about servers • forces redundancy and clustering issues to the front

38. Amazon: S3

39. Amazon: S3 • Simple Storage Service (S3)

40. Amazon: S3 • Simple Storage Service (S3) • Persistent

41. Amazon: S3 • Simple Storage Service (S3) • Persistent • Data stored in “buckets”

42. Amazon: S3 • Simple Storage Service (S3) • Persistent • Data stored in “buckets” • EC2 does not use S3 *directly*

43. Amazon: EBS

44. Amazon: EBS • Elastic Block Store

45. Amazon: EBS • Elastic Block Store • Persistent disks for EC2 instances

46. Amazon: EBS • Elastic Block Store • Persistent disks for EC2 instances • Can store snapshots on S3

47. Amazon: EBS • Elastic Block Store • Persistent disks for EC2 instances • Can store snapshots on S3 • Fast

48. Amazon: EBS • Elastic Block Store • Persistent disks for EC2 instances • Can store snapshots on S3 • Fast • Mounts to only one instance at a time (not a SAN)

49. Amazon: Elastic IPs

50. Amazon: Elastic IPs • Instance IP addresses are dynamic

51. Amazon: Elastic IPs • Instance IP addresses are dynamic • Elastic IPs are static

52. Amazon: Elastic IPs • Instance IP addresses are dynamic • Elastic IPs are static • Free to use, but $0.01/hr if unassociated with an instance

53. EC2 Security Groups

54. EC2 Security Groups • Amazon has a strict ﬁrewall

55. EC2 Security Groups • Amazon has a strict ﬁrewall • ec2-authorize

56. EC2 Security Groups • Amazon has a strict ﬁrewall • ec2-authorize • default security group nodes have unlimited (network) access to each other

57. EC2 Security Groups • Amazon has a strict ﬁrewall • ec2-authorize • default security group nodes have unlimited (network) access to each other • everything else is turned off

58. EC2 Security Groups ec2-add-group web -d quot;Web server groupquot; ec2-authorize -p 80 web ec2-authorize -p 22 web

59. EC2 Security Groups ec2-add-group web -d quot;Web server groupquot; ec2-authorize -p 80 web ec2-authorize -p 22 web ec2-authorize -P icmp -t -1:-1 web

60. Amazon: CloudFront

61. Amazon: CloudFront • Amazon’s “self service” CDN

62. Amazon: CloudFront • Amazon’s “self service” CDN • Pay-as-you-go

63. Amazon: CloudFront • Amazon’s “self service” CDN • Pay-as-you-go • Upload to S3

64. Amazon: CloudFront • Amazon’s “self service” CDN • Pay-as-you-go • Upload to S3 • Content is distributed & cached on Amazon’s machines

65. PoolParty

66. PoolParty overview

67. PoolParty overview • binary tools

68. PoolParty overview • binary tools • conﬁguration ﬁles

69. PoolParty overview • binary tools • conﬁguration ﬁles • erlang-driven messenger

70. PoolParty: Binary Tools

71. PoolParty: Binary Tools • pool-* binaries

72. PoolParty: Binary Tools • pool-* binaries • server-* binaries

73. PoolParty: Binary Tools Shell Example

74. PoolParty Conﬁguration

78. PoolParty Conﬁguration Shell Example a look at clouds/plugins/poolparty-mysql-plugin/mysql.rb

79. how does this actually work?

80. Puppet

81. Puppet

82. Puppet • Puppet is a system for automating system administration tasks.

83. Puppet • Puppet is a system for automating system administration tasks. • Being used at Google to manage all Mac desktops and laptops, expanding into Linux clients.

84. Puppet • Puppet is a system for automating system administration tasks. • Being used at Google to manage all Mac desktops and laptops, expanding into Linux clients. • Red Hat - In the process of moving legacy systems onto Puppet.

85. Puppet

86. Puppet • Resources: ﬁles, folders, permissions, cron jobs, mounted directories, packages, ssh keys, services, arbitrary commands, etc.

87. Puppet • Resources: ﬁles, folders, permissions, cron jobs, mounted directories, packages, ssh keys, services, arbitrary commands, etc. • Mini-language

88. Puppet • Resources: ﬁles, folders, permissions, cron jobs, mounted directories, packages, ssh keys, services, arbitrary commands, etc. • Mini-language • Dependencies

89. Puppet • Resources: ﬁles, folders, permissions, cron jobs, mounted directories, packages, ssh keys, services, arbitrary commands, etc. • Mini-language • Dependencies • If/unless conditions

90. Puppet • Resources: ﬁles, folders, permissions, cron jobs, mounted directories, packages, ssh keys, services, arbitrary commands, etc. • Mini-language • Dependencies • If/unless conditions • Variables

91. Puppet

92. Puppet • Client/Server architecture

93. Puppet • Client/Server architecture • puppetmasterd

94. Puppet • Client/Server architecture • puppetmasterd • puppetd

95. Puppet: Language Example

100. Puppet

101. Puppet • Puppet base conﬁg: /etc/puppet/puppet.conf

102. Puppet • Puppet base conﬁg: /etc/puppet/puppet.conf • PoolParty’s Puppet conﬁg: /etc/puppet/manifests/classes/poolparty.pp

103. Puppet Conﬁguration Shell Example a look at /etc/puppet/manifests/classes/poolparty.pp

104. Puppet • Cron jobs: * 1 * * * . /etc/profile && puppetmasterd --verbose */15 * * * * /usr/bin/puppetrunner

105. Puppet

106. Puppet • Builds dependency graph

107. Puppet • Builds dependency graph • Provisions the machines (e.g. runs the commands)

108. Puppet Security

109. Puppet Security • Uses SSL certiﬁcates as the required and only form of authentication

110. Puppet Security • Uses SSL certiﬁcates as the required and only form of authentication • Master authenticates the certiﬁcates

111. Puppet Security • Uses SSL certificates as the required and only form of authentication • Master authenticates the certificates • Test certificate from node: /usr/sbin/puppetd --waitforcert 60 --server master --test --verbose

112. Puppet Security • Uses SSL certificates as the required and only form of authentication • Master authenticates the certificates • Test certificate from node: /usr/sbin/puppetd --waitforcert 60 --server master --test --verbose • If certificate doesn’t match, probably cached on server. Try on master: puppetca --clean node1.compute-1.internal

113. Puppet Cache

114. Puppet Cache • Keeps a cache of the parsed conﬁguration in: /var/lib/puppet/localconfig.yaml

115. Troubleshooting Puppet

116. Troubleshooting Puppet • tail -f /var/log/syslog

117. Troubleshooting Puppet • tail -f /var/log/syslog • Can the nodes contact master?

118. Troubleshooting Puppet • tail -f /var/log/syslog • Can the nodes contact master? • Do the certiﬁcates match?

119. Troubleshooting Puppet • tail -f /var/log/syslog • Can the nodes contact master? • Do the certiﬁcates match? • Does /var/lib/puppet/localconfig.yaml contain the right information?

120. Troubleshooting Puppet • tail -f /var/log/syslog • Can the nodes contact master? • Do the certiﬁcates match? • Does /var/lib/puppet/localconfig.yaml contain the right information? • Does /etc/puppet/manifests/classes/ contain the right information? poolparty.pp

121. Troubleshooting Puppet • tail -f /var/log/syslog • Can the nodes contact master? • Do the certiﬁcates match? • Does /var/lib/puppet/localconfig.yaml contain the right information? • Does /etc/puppet/manifests/classes/ contain the right information? poolparty.pp • Does contain the /var/poolparty/poolparty.pp right information? (more on that later)

122. Troubleshooting Puppet

123. Troubleshooting Puppet • server-provision -n app -i 1 --slave --verbose --debug

124. master

125. PoolParty DNS on master root@master:~# cat /etc/hosts 127.0.0.1 localhost.localdomain localhost 127.0.0.1 master 75.101.128.147 master puppet localhost 10.252.166.18 node1.app node1 10.252.167.47 master.app master

126. PoolParty DNS on master root@master:~# cat /etc/hosts 127.0.0.1 localhost.localdomain localhost 127.0.0.1 master 75.101.128.147 master puppet localhost 10.252.166.18 node1.app node1 10.252.167.47 master.app master on node1 root@node1:~# cat /etc/hosts 127.0.0.1 localhost.localdomain localhost 127.0.0.1 node1 75.101.128.147 puppet master 10.252.167.47 master.app master 10.252.166.18 node1.app node1

127. PoolParty DNS • /etc/hosts modiﬁed by PoolParty when you cloud provision

128. HAProxy

129. HAProxy

130. HAProxy • Extremely fast

131. HAProxy • Extremely fast • Reports of HAProxy saturating gigabit ﬁber (http://is.gd/4myI)

132. HAProxy • Extremely fast • Reports of HAProxy saturating gigabit ﬁber (http://is.gd/4myI) • Reputation of being reliable and secure

133. HAProxy • Extremely fast • Reports of HAProxy saturating gigabit ﬁber (http://is.gd/4myI) • Reputation of being reliable and secure • Installed and setup by PoolParty by default

134. HAProxy

135. HAProxy • Conﬁg kept in: /etc/haproxy.cfg

136. HAProxy • Conﬁg kept in: /etc/haproxy.cfg • Reload conﬁg by: /etc/init.d/haproxy reload

137. HAProxy • View stats by: http://ec2-your-ip/poolparty

138. HAProxy • View stats by: http://ec2-your-ip/poolparty

139. PoolParty Messenger

141. PoolParty Messenger • Three parts:

142. PoolParty Messenger • Three parts: • Master

143. PoolParty Messenger • Three parts: • Master • Node

144. PoolParty Messenger • Three parts: • Master • Node • Client

145. PoolParty Messenger • Three parts: • Master • Node • Client • “Instance Glue”

146. PoolParty Messenger • Three parts: • Master • Node • Client • “Instance Glue” • Master provisions nodes (will change)

148. PoolParty Messenger • Erlang - excellent node-to-node monitoring

149. PoolParty Messenger • Erlang - excellent node-to-node monitoring • Gathers load data

150. PoolParty Messenger • Erlang - excellent node-to-node monitoring • Gathers load data • Decides when to launch new nodes

152. PoolParty Messenger • Changes happening in architecture

153. PoolParty Messenger • Changes happening in architecture • Will become neighborhood-based instead of master/node-based

154. PoolParty Messenger • Changes happening in architecture • Will become neighborhood-based instead of master/node-based • The brains of the operation when humans aren’t around

155. Using PoolParty

156. Using PoolParty

157. Using PoolParty • cloud start

158. Using PoolParty • cloud start • cloud configure --verbose --debug

159. Using PoolParty • cloud start • cloud configure --verbose --debug • cloud provision --verbose --debug

160. Using PoolParty: What Happens

161. Using PoolParty: What Happens • PoolParty reads clouds.pool

162. Using PoolParty: What Happens • PoolParty reads clouds.pool • Generates templates, puppet conﬁgs, etc in a local storage directory (/tmp/poolparty)

163. Using PoolParty: What Happens • PoolParty reads clouds.pool • Generates templates, puppet conﬁgs, etc in a local storage directory (/tmp/poolparty) • Files are rsync’d to /var/poolparty on master

164. Using PoolParty: What Happens • PoolParty reads clouds.pool • Generates templates, puppet conﬁgs, etc in a local storage directory (/tmp/poolparty) • Files are rsync’d to /var/poolparty on master • A gang of bootstrapping tasks are run (via Capistrano)

165. Using PoolParty: What Happens • PoolParty reads clouds.pool • Generates templates, puppet conﬁgs, etc in a local storage directory (/tmp/poolparty) • Files are rsync’d to /var/poolparty on master • A gang of bootstrapping tasks are run (via Capistrano) • Puppet conﬁg is cp’d from /var/poolparty to /etc/ puppet/manifests/classes/poolparty.pp

167. Using PoolParty: What Happens • Puppet is started

168. Using PoolParty: What Happens • Puppet is started • Puppet tries to provision everything

169. Using PoolParty: What Happens • Puppet is started • Puppet tries to provision everything • Puppet starts PP Messenger master

171. Using PoolParty: What Happens • PP Messenger master contacts Amazon and boots more instances if needed.

172. Using PoolParty: What Happens • PP Messenger master contacts Amazon and boots more instances if needed. • PP Messenger bootstraps the new node

173. Using PoolParty: What Happens • PP Messenger master contacts Amazon and boots more instances if needed. • PP Messenger bootstraps the new node • The bootstrap process starts PP Messenger Node

174. Using PoolParty: What Happens • PP Messenger master contacts Amazon and boots more instances if needed. • PP Messenger bootstraps the new node • The bootstrap process starts PP Messenger Node • PP Messenger initiates Puppet provisioning on the new node

175. Using PoolParty: What Happens • PP Messenger master contacts Amazon and boots more instances if needed. • PP Messenger bootstraps the new node • The bootstrap process starts PP Messenger Node • PP Messenger initiates Puppet provisioning on the new node • time passes

177. Using PoolParty: What Happens • Puppet runs on master and master discovers a new node has been started.

178. Using PoolParty: What Happens • Puppet runs on master and master discovers a new node has been started. • In the case of HAProxy/Apache, Puppet re- generates the HAProxy conﬁg to include the new node.

179. Using PoolParty: What Happens • Puppet runs on master and master discovers a new node has been started. • In the case of HAProxy/Apache, Puppet re- generates the HAProxy conﬁg to include the new node. • As speciﬁed in poolparty.pp, when haproxy.cnf gets updated, it asks HAProxy to reload.

180. Using PoolParty: What Happens • Puppet runs on master and master discovers a new node has been started. • In the case of HAProxy/Apache, Puppet re- generates the HAProxy conﬁg to include the new node. • As speciﬁed in poolparty.pp, when haproxy.cnf gets updated, it asks HAProxy to reload. • HAProxy now starts distributing the load to the new node

182. Using PoolParty: What Happens • Similar conﬁg changes happen throughout the system

183. Real World Misc.

185. Real World Misc. • tail -f /var/log/syslog is your friend

186. Real World Misc. • tail -f /var/log/syslog is your friend • ssh into master: cloud ssh

187. Real World Misc. • tail -f /var/log/syslog is your friend • ssh into master: cloud ssh • ssh into i’th node: cloud ssh -i 1

189. Real World Misc. • scp scp -i /Users/me/.ec2/id_rsa-clouds_app root@ec2-75-101-191-10.compute-1.amazonaws.com:/ etc/snmp/snmpd.conf .

190. Real World Misc. • scp scp -i /Users/me/.ec2/id_rsa-clouds_app root@ec2-75-101-191-10.compute-1.amazonaws.com:/ etc/snmp/snmpd.conf . • rsync rsync -av -e quot;ssh -i /var/poolparty/id_rsa-clouds_appquot; /var/www/cpohunterfan.com/documents node1:/var/www/cpohunterfan.com/

191. Workshop

192. Agenda:

193. Agenda: • Launch, login, terminate an EC2 instance

194. Agenda: • Launch, login, terminate an EC2 instance • into our instances and look around cloud ssh

195. Agenda: • Launch, login, terminate an EC2 instance • into our instances and look around cloud ssh • Take a closer look at clouds.pool

196. Agenda: • Launch, login, terminate an EC2 instance • into our instances and look around cloud ssh • Take a closer look at clouds.pool • Examine a basic PoolParty plugin

Getting Started with PoolParty and EC2

Nate Murray

Getting Started with PoolParty and EC2