The document outlines the Stacki server provisioning system led by David Peterson, detailing hardware requirements, ZFS integration, and customization capabilities. It emphasizes the ease of managing large server arrays, configuration specifics, and the integration of Chef for server provisioning. The vendor provides guidelines for handling SAN setups and detecting provisioning issues through various monitoring techniques.

1. Stacki – The 1600+ Server Journey
David Peterson
Lead Systems Engineer
david.peterson@salesforce.com

2. Agenda
▪ Why Stacki
▪ Hardware and provisioning requirements
▪ Stacki configuration with chef integration
▪ ZFS and data safe re-provisioning
▪ Detecting issues and adhoc reporting

3. Why Stacki

4. Why Stacki
▪ Managing thousands of servers is easy (csv)
▪ HP raid controller support
▪ Easy out of the box provisioning but deep customization
available
▪ Ability to re-provision without losing data
▪ Easy network/subnet configuration
▪ YUM repo support
▪ Command line, command line, command line
▪ Support

5. Hardware and Provisioning
Requirements

6. Hardware and Provisioning Requirements
Firewall/ACL Ports
▪ bootpc and bootps (dhcp): UDP 67 and 68
▪ tftp: UDP 69
▪ tftp ephermeral: UDP 32765-65535
▪ http/https: TCP 80 and 443

7. Hardware and Provisioning Requirements
Host RAID LUNs and Partition Setup
LUNs:
▪ 4x 2TB SATA disks => RAID 10
▪ 200GB LUN (sda)
▪ 7.7TB LUN (sdb)
▪ 2x 480GB SSD disks => RAID 0
▪ 960GB LUN (sdc)

8. Hardware and Provisioning Requirements
Host RAID LUNs and Partition Setup
Partitions:
▪ sda
▪ /boot, ext4 (sda) => 500MB
▪ Swap => 5GB
▪ /, ext4 => ~195GB
▪ sdb
▪ No partitions
▪ sdc
▪ sdc1 => 10GB, non-formatted
▪ sdc2 => 200GB, non-formatted

9. Hardware and Provisioning Requirements
Latest LT Kernel and ZFS
▪ Kernel LT => 3.10.95-1
▪ ZFS => 0.6.5.2

10. Hardware and Provisioning Requirements
Chef Integration
▪ End to end server provisioning with chef
▪ Chef configured on each server, host added to
chef server, and a chef-client run to apply base
roles

11. Stacki Configuration

12. Stacki Configuration
Concurrent kickstart limitation
▪ /export/stack/sbin/kickstart.cgi:L154
# Use a semaphore to restrict the number of concurrent kickstart
# file generators. The first time through we set the semaphore
# to the number of CPUs (not a great guess, but reasonable).
▪ semaphore = stack.lock.Semaphore('/var/tmp/kickstart.semaphore')
[root@stacki]# echo 200 > /var/tmp/kickstart.semaphore

13. Stacki Configuration
Custom RAID Controller Setup
▪ /export/stack/site-profiles/prod/2.0/nodes/replace-storage-controller-client.xml

14. /export/stack/site-profiles/prod/2.0/nodes/replace-storage-controller-client.xml
<?xml version="1.0" standalone="no"?>
<kickstart>
<pre>
if [ "&nukecontroller;" == "true" ]
then
/opt/stack/sbin/hpssacli ctrl slot=0 delete forced override
/opt/stack/sbin/hpssacli ctrl slot=0 create type=ld
drives=1I:1:1,1I:1:2,1I:1:3,1I:1:4 raid=1+0 size=200000
/opt/stack/sbin/hpssacli ctrl slot=0 create type=ld
drives=1I:1:1,1I:1:2,1I:1:3,1I:1:4 raid=1+0
/opt/stack/sbin/hpssacli ctrl slot=0 create type=ld
drives=2I:0:5,2I:0:6 raid=0
fi
</pre>
<!-- now reset the nukecontroller attribute to false -->
<pre>
<eval>
/opt/stack/bin/stack set host attr &hostname; attr=nukecontroller value=false
</eval>
</pre>
</kickstart>

15. /export/stack/site-profiles/prod/2.0/nodes/replace-storage-controller-client.xml
<pre cond="appliance in [’rabbitmq']">
if [ "&nukecontroller;" == "true" ]
then
/opt/stack/sbin/hpssacli ctrl slot=0 delete forced override
/opt/stack/sbin/hpssacli ctrl slot=0 create type=ld
drives=1I:1:1,1I:1:2,1I:1:3,1I:1:4 raid=1+0 size=500000
/opt/stack/sbin/hpssacli ctrl slot=0 create type=ld
drives=1I:1:1,1I:1:2,1I:1:3,1I:1:4 raid=1+0
/opt/stack/sbin/hpssacli ctrl slot=0 create type=ld
drives=2I:0:5,2I:0:6 raid=1
fi
</pre>

16. /export/stack/site-profiles/prod/2.0/nodes/extend-backend.xml
<boot order="post">
# Set HP RAID controller cache settings
/opt/stack/sbin/hpssacli ctrl slot=0 ld 1 modify arrayaccelerator=enable
/opt/stack/sbin/hpssacli ctrl slot=0 ld 2 modify arrayaccelerator=enable
/opt/stack/sbin/hpssacli ctrl slot=0 modify cacheratio=80/20
/opt/stack/sbin/hpssacli ctrl slot=0 array b modify ssdsmartpath=disable
/opt/stack/sbin/hpssacli ctrl slot=0 logicaldrive 3 modify arrayaccelerator=enable
echo "y" | /opt/stack/sbin/hpssacli ctrl slot=0 modify dwc=enable
</boot>

17. Stacki Configuration
Custom Partitions
▪ /export/stack/site-profiles/prod/2.0/nodes/extend-partition.xml

18. /export/stack/site-profiles/prod/2.0/nodes/extend-partition.xml
<?xml version="1.0" standalone="no"?>
<kickstart>
<post>
<![CDATA[
/sbin/fdisk /dev/sdc << EOF
d
w
EOF
/sbin/fdisk /dev/sdc << EOF
n
p
1
1
+10G
w
EOF
/sbin/fdisk /dev/sdc << EOF
n
p
2
+200G
w
EOF
]]>
</post>
</kickstart>

19. Stacki Configuration
Custom Appliance Types
▪ /export/stack/site-profiles/prod/2.0/nodes/extend-backend.xml
[root@stacki]# stack list appliance
APPLIANCE MEMBERSHIP PUBLIC
frontend: Frontend no
backend: Backend yes
rabbitmq: Rabbitmq yes
redis: Redis yes
mysql: Mysql yes
[root@stacki]# stack add appliance loadbalancer
[root@stacki]# stack set appliance attr loadbalancer attr=managed value=true
[root@stacki]# stack set appliance attr loadbalancer attr=kickstartable value=true
[root@stacki]# stack set appliance attr loadbalancer attr=node value=backend

20. /export/stack/site-profiles/prod/2.0/nodes/extend-backend.xml
<boot order="post" cond="appliance == ’rabbitmq' and (nukecontroller or nukedisks)">
/sbin/zfs create data/rabbitmq
/sbin/zfs set mountpoint=/var/lib/rabbitmq data/rabbitmq
</boot>
<boot order="post" cond="appliance == ’redis' and (nukecontroller or nukedisks)">
/sbin/zfs create data/redis
/sbin/zfs set mountpoint=/var/lib/redis data/redis
adduser -r redis -U
chown redis:redis /var/lib/redis
</boot>
<boot order="post" cond="appliance == 'mysql' and (nukecontroller or nukedisks)">
# Disabling THP
<![CDATA[
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag
sed -i 's/kernel.* console=ttyS0,19200n8$/& transparent_hugepage=never/' /boot/grub/grub.conf
sed -i 's/kernel.* crashkernel=auto$/& transparent_hugepage=never/' /boot/grub/grub.conf
]]>
/sbin/zfs create data/mysql
/sbin/zfs create data/mysql-log
/sbin/zfs create data/mysql-tmp
/sbin/zfs set recordsize=16K data/mysql
/sbin/zfs set mountpoint=/var/lib/mysql data/mysql
/sbin/zfs set mountpoint=/var/log/mysql data/mysql-log
/sbin/zfs set mountpoint=/var/lib/mysql/tmp data/mysql-tmp
adduser -r mysql -U
chown mysql:mysql /var/lib/mysql /var/log/mysql /var/lib/mysql/tmp
</boot>

21. Stacki Configuration
Custom Appliance Types

22. Stacki Configuration
Chef Cart
▪ /export/stack/carts/chef/nodes/cart-chef-backend.xml

23. /export/stack/carts/chef/nodes/cart-chef-backend.xml
<?xml version="1.0" standalone="no"?>
<kickstart>
<description>
chef cart backend appliance extensions
</description>
<package>chef</package>
<!-- shell code for post RPM installation -->
<post>
mkdir -p /etc/chef /var/log/chef /var/run/chef
</post>
<post cond="not ‘proxy’ in hostname">
<file name="/etc/chef/client.rb">
<![CDATA[
#
# Chef Client Config File
#
# Dynamically generated by Stacki
#
log_level :info
log_location STDOUT
chef_server_url "#CHEF_SERVER#"
validation_client_name ”chef-validator"
validation_key "/etc/chef/validation.pem"
client_key "/etc/chef/client.pem"
ssl_verify_mode :verify_none
http_proxy 'http://proxy1:3128'
https_proxy 'http://proxy2:3128'
no_proxy ’test1,localhost,127.0.0.1'
environment 'production'
# Using default node name (fqdn)
node_name "#HOSTNAME#”
Ohai::Config[:plugin_path] << '/etc/chef/ohai'
]]>
</file>
# Need to add the chef server and client hostname to the client.rb file
sed -i 's,#CHEF_SERVER#,&chef_server;,g' /etc/chef/client.rb
sed -i 's/#HOSTNAME#/&hostname;.&domainname;/g' /etc/chef/client.rb
</post>

24. /export/stack/carts/chef/nodes/cart-chef-backend.xml
<post>
<file name="/etc/chef/first-boot.json">
{
"run_list": [
"role[base_role]",
"role[dc_sfo]"
]
}
</file>
</post>
# If we are nuking disks we are assuming this is a new server
# or the chef client/node has been deleted out of the chef server if it existed.
<boot order="post" cond="nukedisks">
# Run chef-client for the first time
/usr/bin/chef-client -j /etc/chef/first-boot.json -L /var/log/chef/chef.log
# Make a backup of the chef private key in case we need to re-provision/upgrade a server
mkdir -p /data/chef-backup
chown root:root /data/chef-backup
chmod 700 /data/chef-backup
cp -a /etc/chef/* /data/chef-backup
</boot>
# If we are not nuking the disks we are assuming we are re-loading or upgrading
# the OS and need to keep the client.pem chef key so chef-client can run properly
<boot order="post" cond="not nukedisks">
cp /data/chef-backup/client.pem /etc/chef/
/usr/bin/chef-client -L /var/log/chef/chef.log
</boot>

25. Stacki Configuration
RCS Issues
▪ Stacki installs foundation-rcs package on provisioned servers
▪ Caused issues for our rsyslog daemon because of RCS config files being loaded.
Other daemons were affected as well.
▪ Let’s remove it and clean up all the RCS directories
/export/stack/site-profiles/prod/2.0/nodes/extend-backend.xml
<boot order="post">
# Remove rcs rpm and cleanup RCS directories
rpm -e foundation-rcs
find / -type d -name 'RCS' -print0 |xargs -0 rm –rf
</boot>

26. ZFS and Data Safe
Provisioning

27. ZFS and Data Safe Provisioning
What is ZFS?
▪ A combined file system and logical volume manager
▪ Data integrity
▪ Software raid
▪ Storage pools
▪ Sophisticated caching: ARC (RAM MFU/MRU), L2ARC
(SSDs), ZIL/SLOG
▪ Snapshots and Clones
▪ Compression

28. ZFS and Data Safe Provisioning
ZFS and Latest Kernel Installation
▪ YUM repos imported into Stacki
▪ http://elrepo.org/
▪ http://zfsonlinux.org/
<?xml version="1.0" standalone="no"?>
<kickstart>
<package>kernel-lt</package>
<package>kernel-lt-devel</package>
<package>kernel-lt-headers</package>
<package>zfs</package>
/export/stack/site-profiles/prod/2.0/nodes/extend-backend.xml

29. /export/stack/site-profiles/prod/2.0/nodes/extend-backend.xml
<post>
# Enable kernel 3.x
sed -i 's/^default=.*/default=1/g' /boot/grub/grub.conf
# Add zfs module config options
echo "options zfs zfs_arc_max=34359738368" >> /etc/modprobe.d/zfs.conf
echo "options zfs zfs_nocacheflush=1" >> /etc/modprobe.d/zfs.conf
echo "options zfs zfs_read_chunk_size=1310720" >> /etc/modprobe.d/zfs.conf
echo "options zfs zfs_prefetch_disable=1" >> /etc/modprobe.d/zfs.conf
echo "options zfs zil_slog_limit=104857600" >> /etc/modprobe.d/zfs.conf
</post>

30. /export/stack/site-profiles/prod/2.0/nodes/extend-backend.xml
<boot order="post" cond="nukedisks">
/sbin/modprobe zfs
/sbin/zpool create -f data sdb log sdc1 cache sdc2
/sbin/zfs set atime=off data
/sbin/zfs set compression=lz4 data
# Add /opt filesystem
/sbin/zfs create data/opt
/bin/mv /opt/* /data/opt/
/bin/rm -rf /opt
/sbin/zfs set mountpoint=/opt data/opt
# Add /var/log/httpd filesystem
/sbin/zfs create data/httpd-log
/sbin/zfs set mountpoint=/var/log/httpd data/httpd-log
chmod 700 /var/log/httpd
# Add /var/log/logstash filesystem
/sbin/zfs create data/logstash
/sbin/zfs set mountpoint=/var/log/logstash data/logstash
adduser -r logstash -U
chown logstash:logstash /var/log/logstash
echo "create zfs data pool..." > /tmp/zfs-create.log
</boot>

31. /export/stack/site-profiles/prod/2.0/nodes/extend-backend.xml
<boot order="post" cond="not nukedisks">
# We need to empty/move the data in /opt before we can import zfs
mkdir /tmp/opt
mv /opt/* /tmp/opt/
/sbin/modprobe zfs
/sbin/zpool import -d /dev/disk/by-path/ data
echo "Importing zfs data pool..." > /tmp/zfs-import.log
mv /tmp/opt/* /opt/
rm -rf /tmp/opt
</boot>

32. Detecting Issues and
AdHoc Reporting

33. Detecting Issues and AdHoc Reporting
What? We have Issues?
▪ Stacki is great at provisioning but getting the status of a provisioned
or currently being provisioned server is a little harder.
▪ A couple different ways but at various stages in the provisioning
process:
1. Tailing /var/log/messages for DHCP requests and acks
2. Watching the nukecontroller and nukedisks attributes
3. Tailing /var/log/httpd/access_log for rpm downloads
4. Watching the boot action flag
5. iftop
6. Chef node entry
▪ Note: Tailing log files for a couple servers is fine but when
provisioning hundreds of servers at a time, it is not viable.

34. Detecting Issues and AdHoc Reporting
What? We have Issues?
▪ Watching the nukecontroller and nukedisks attributes
[root@stacki]# stack list host attr chef1-1 |grep nuke
chef1-1: -------------------- nukecontroller true H
chef1-1: -------------------- nukedisks true H
192.168.10.50 - - [09/Feb/2016:20:39:52 -0700] "GET /install/sbin/public/setDbPartitions.cgi HTTP/1.1" 200 1
/var/log/httpd/ssl_access_log

35. Detecting Issues and AdHoc Reporting
What? We have Issues?
▪ Tailing /var/log/httpd/access_log for rpm downloads
192.168.10.50 - - [09/Feb/2016:17:09:32 -0700] "GET /install/distributions/prod/x86_64/RedHat/RPMS/gtk2-2.24.23-6.el6.x86_64.rpm HTTP/1.1" 200 3339880 "-" "-"
192.168.10.50 - - [09/Feb/2016:17:09:31 -0700] "GET /install/distributions/prod/x86_64/RedHat/RPMS/hdparm-9.43-4.el6.x86_64.rpm HTTP/1.1" 200 83060 "-" "-”
192.168.10.50 - - [09/Feb/2016:17:09:32 -0700] "GET /install/distributions/prod/x86_64/RedHat/RPMS/libXext-1.3.2-2.1.el6.x86_64.rpm HTTP/1.1" 200 35644 "-" "-"
192.168.10.50 - - [09/Feb/2016:17:09:32 -0700] "GET /install/distributions/prod/x86_64/RedHat/RPMS/filesystem-2.4.30-3.el6.x86_64.rpm HTTP/1.1" 200 1057228 "-" "-"
192.168.10.50 - - [09/Feb/2016:17:09:32 -0700] "GET /install/distributions/prod/x86_64/RedHat/RPMS/NetworkManager-0.8.1-99.el6.x86_64.rpm HTTP/1.1" 200 1185212 "-" "-"
192.168.10.50 - - [09/Feb/2016:17:09:32 -0700] "GET /install/distributions/prod/x86_64/RedHat/RPMS/libstoragemgmt-1.2.3-1.el6.x86_64.rpm HTTP/1.1" 200 211068 "-" "-"
tail –f /var/log/httpd/access_log | grep –E “192.168.10.50|192.168.10.51”

36. Detecting Issues and AdHoc Reporting
What? We have Issues?
▪ Watching the boot action flag
[root@stacki]# stack list host boot chef1-*
HOST ACTION
chef1-2: install
chef1-1: os
192.168.10.50 - - [09/Feb/2016:20:39:52 -0700] "GET /install/sbin/public/setPxeboot.cgi?params={"action":"os"} HTTP/1.1" 200 1
/var/log/httpd/ssl_access_log

37. Detecting Issues and AdHoc Reporting
What? We have Issues?
▪ Issues we encountered
• TORs ip helper-address not set properly
• ACL mis-match between racks causing DHCP/TFTP to be
blocked
• Mis-configured host networks causing the gateways to be wrong
which prevented DHCP/PXE from working properly
• Post boot zfs commands not running properly due to hardware
missing drives

38. Detecting Issues and AdHoc Reporting
AdHoc Reporting
▪ Find all hosts that still have the “install” flag and generate a report
for h in `stack list host boot |grep -w install|awk '{print $1}'|sed s/://`;
do for ip in `stack list host interface $h|grep eth0|awk '{print $5}'`;
do echo -e "Host: $hnChecking for IP: $ip"; echo "";
cat /var/log/messages /var/log/httpd/ssl_access_log /var/log/httpd/access_log|grep -iw $ip; echo "";
done; done > host_report.txt

39. Host: test1
Checking for IP: 192.168.10.50
Feb 9 19:32:12 stacki-host dhcpd: DHCPOFFER on 192.168.10.50 to ba:c2:3d:c3:ab:13 via 192.168.10.1
Feb 9 19:32:12 stacki-host dhcpd: DHCPOFFER on 192.168.10.50 to ba:c2:3d:c3:ab:13 via 192.168.10.1
Feb 9 19:32:16 stacki-host dhcpd: DHCPREQUEST for 192.168.10.50 (192.168.10.5) from ba:c2:3d:c3:ab:13 via 192.168.10.1
Feb 9 19:32:16 stacki-host dhcpd: DHCPACK on 192.168.10.50to ba:c2:3d:c3:ab:13 via 192.168.10.1
Feb 9 19:32:16 stacki-host dhcpd: DHCPREQUEST for 192.168.10.50 (192.168.10.5) from ba:c2:3d:c3:ab:13 via 192.168.10.1
Feb 9 19:32:16 stacki-host dhcpd: DHCPACK on 192.168.10.50 to ba:c2:3d:c3:ab:13 via 192.168.10.1
192.168.10.50 - - [09/Feb/2016:19:32:54 -0700] "GET /install/sbin/kickstart.cgi?arch=x86_64&np=40 HTTP/1.1" 200 96101
192.168.10.50 - - [09/Feb/2016:19:33:13 -0700] "GET /install/distributions/prod/x86_64/images/updates.img HTTP/1.1" 404 329 "-" "-”
192.168.10.50 - - [09/Feb/2016:19:33:33 -0700] "GET /install/distributions/prod/x86_64/images/product.img HTTP/1.1" 200 782336 "-" "-"
192.168.10.50 - - [09/Feb/2016:19:33:35 -0700] "GET /install/distributions/prod/x86_64/images/install.img HTTP/1.1" 200 236163072 "-" "-"
Host: test2
Checking for IP: 192.168.10.51
Host: test3
Checking for IP: 192.168.10.52

40. Detecting Issues and AdHoc Reporting
AdHoc Reporting
▪ Find the top racks with the most un-provisioned hosts. Helps us
identify racks with potential ACL issues.
[root@stacki]# stack list network|awk '{print $1}’
NETWORK
rack1-prod_vlan1:
rack2-prod_vlan2:
rack3-prod_vlan1:
rack4-prod_vlan2:
rack5-prod_vlan2:
[root@stacki]# for h in `stack list host boot |grep -w install|awk '{print $1}'|sed s/://`; do stack list host interface $h; done
|grep eth0|awk '{print $3}'|cut -d- -f 1|sort|uniq -c|sort -rn|head
40 rack2
9 rack3
7 rack5
6 rack1
6 rack4

41. Lessons Learned
▪ With thousands of servers, you need a standard naming convention for hosts, networks,
appliance types, etc.
▪ Standardized servers saves you time and headaches.
▪ Created custom scripts to augment stacki functionality and reduce human errors
• create-stack-appliances.sh: This script will look for appliance types in the extend-backend.xml file, check to
see if they already exist and if not, create them in Stacki.
• create-stack-networks.sh: This script will import a list of networks from a csv file you specify.
• stack-hosts.sh: This script enables or disables provisioning of hosts listed in a file and can optionally set the
nuke attributes.
▪ Stacki by default does not allow you to have a high number of concurrent kickstart
sessions.
▪ When making config changes, verify proper syntax and expected output by running:
stack list host profile <hostname> | less

42. thank y u