The document outlines a complex infrastructure setup involving load balancers, Apache web servers, MongoDB, and various security measures such as UFW configurations and SSH settings. It includes specific code snippets for server configuration, routine management tasks, and system updates across multiple servers. Additionally, it discusses logging practices and emergency management protocols for live server situations.

1. From zero to Puppet
http://www.rankpop.com/you-need-to-start-structuring-your-blog-posts-asap/

3. +15TB / mth
+1bn docs /mth
2-5k inserts/s @ 3ms
10K RPM @ 140ms

4. Servers
HTTP Load Balancer – 5x
Apache - 14x
Build - 2x
MongoDB - 19x data,
13x routing,
6x configuration,
6x arbiter,

5. HTTP Load Balancer
$globalIPs_array = split($globalIPs, ',')
<% globalIPs_array.each do |globalIP| ­%>
ListenHTTP
# primary public IP address
Address <%= globalIP %>
Port 80

6. Apache

7. Apache
file
{
'vhost':
path => "/etc/apache2/sites­enabled/${::siteDomain}",
ensure => file,
content => template('apache­php/vhost.erb'),
notify => Service['apache2'],
}

8. Apache
<VirtualHost *:80>
ServerName <%= siteDomain %>
DocumentRoot <%= documentRoot %>
...
ErrorLog /var/log/apache2/error­<%= siteDomain %>.log
<% if @requestLogging and requestLogging == "yes" %>
CustomLog /var/log/apache2/access­<%= siteDomain %>.log vhost_combined
<% end %>
<% if enableSSL == "yes" %>
<VirtualHost *:443>
SSLEngine On
SSLCertificateFile /var/www/ssl/<%= siteDomain %>.crt
SSLCertificateKeyFile /var/www/ssl/<%= siteDomain %>.key
...

9. $excess_bagage = [
"ppp",
"bind9­host",
"laptop­detect",
"open­iscsi",
"libnss3:i386"
]
package
{
$excess_bagage:
ensure => purged,
}

10. exec
{
'ufw allow is­1':
command => '/usr/sbin/ufw allow from 184.173.178.67',
unless => '/usr/sbin/ufw status verbose | grep "184.173.178.67"',
}
exec
{
'ufw enable':
command => '/usr/sbin/ufw enable',
unless => '/usr/sbin/ufw status | grep "Status: active"',
Require => [Exec['ufw allow is­1']],
notify => Exec['ufw reload'],
}
exec
{
'ufw reload':
command => '/usr/sbin/ufw reload',
require => Exec['ufw enable'],
refreshonly => true,
}

11. SSH
file
{
'sshd_config':
path => '/etc/ssh/sshd_config',
ensure => file,
content => template('sshd/sshd_config.erb'),
}
Port 22
Protocol 2
AllowUsers david pessoa chris rob wes
Compression yes
...
<% if @duoSecKey and @duoIntKey and @duoHost %>
ForceCommand /usr/sbin/login_duo
PermitTunnel no
AllowTcpForwarding no
<% end %>

12. System updates
CANARIES="
aws­prod­apac­singapore­exm­a1
a2.wdc.sl
exm1.wdc.sl
mtx­web1.wdc.sl
sdapp­web1.wdc.sl
sdapi­web1.wdc.sl
"
function canaries {
echo "REMOVE *­web1.wdc.sl FROM ROTATION!"
echo "press ENTER"
read
echo "have you REALLY removed *­web1.wdc.sl from rotation?"
echo "press ENTER"
read
for i in $CANARIES
do
echo $i
mco rpc puppetral ­I $i create type=exec
title="/bin/bash ­c 'apt­get dist­upgrade ­y'"
done
}

13. System updates
function reboot_canaries {
echo "REMOVE *­web1.wdc.sl FROM ROTATION!"
echo "press ENTER"
read
echo "have you REALLY removed *­web1.wdc.sl from rotation?"
echo "press ENTER"
read
for i in $CANARIES
do
echo $i
mco rpc puppetral ­I $i create type=exec
title="/bin/bash ­c 'reboot'"
echo "­­­­­­­­­­­­­­"
done
}

14. What to Reboot
mco rpc puppetral create type=exec
title="/bin/bash ­c 'ls /var/run/reboot­required'"
| grep ­B 1 "Resource was created"
| grep ".sl"
exm1.wdc.sl
Status: change from notrun to 0 failed:
/bin/bash ­c 'ls /var/run/reboot­required'
returned 2 instead of one of [0]
exm­md1a.wdc.sl
Status: Resource was created
Resource: {"tags"=>["exec"],
"title"=>"/bin/bash ­c 'ls /var/run/reboot­required'",
"type"=>"Exec",
"parameters"=>{:returns=>:notrun},
"exported"=>false}

15. Live Management
- Emergency
# Metrics
<%= metricsIP %> metrics­svc
# Infrastructure services
184.173.178.66 puppet
...

16. Live Management
- Emergency

17. Live Management
- Emergency

18. Live Management
- Emergency

19. Live Management
- Emergency

20. Pedro Pessoa
pessoa@serverdensity.com
www.serverdensity.com