Talk given by David Petersen, Lead Systems Engineer at Salesforce, at Stacki Webinar on Nov 2016 Learn how we've integrated chef into Stacki provisioning system and how we've automated the process.

1. Stacki and Chef at Pardot
David Peterson
Lead Systems Engineer
david.peterson@salesforce.com

2. About Me
I joined Salesforce in 2014 and am currently a Lead Systems Engineer based out of
Atlanta, GA. I help lead the infrastructure design and scaling of Salesforce’s
product offering, Pardot Marketing Automation.
Prior to Salesforce, I worked at Perfect Sense Digital where I was the Lead
Engineer in architecting and implementing HGTV and Travel Channel's new
website infrastructure in AWS. I also helped design and implement the
infrastructure for digital media sites such as Food Network, Univision, Coca-Cola,
Walmart, Lionsgate, BET, Snagfilms, National Geographic, and others.
Prior to PSD, I worked at ChannelAdvisor, an e-commerce SaaS company, where I
helped design and manage the infrastructure responsible for processing over $3
billion in GMV.

3. Agenda
▪ Why Stacki (I’m assuming version 3.2)
▪ Stacki global configuration for multiple DCs
▪ Stacki chef cart setup
▪ Specific Chef configurations and how we handle server
re-provisions
▪ Proper ordering with multiple carts
▪ Questions

4. Why Stacki
▪ Managing thousands of servers is easy (csv)
▪ HP and Dell raid controller support
▪ Easy out of the box provisioning but deep customization available
▪ Ability to re-provision without losing data
▪ Easy network/subnet configuration and host assignment
▪ YUM repo support
▪ Command line, command line, command line
▪ Support

5. Stacki Global Configurations
[root@stacki]# stack list attr
SCOPE ATTR VALUE SOURCE
--------- Confirm_Private_PureRootPassword a G
--------- HttpConf /etc/httpd/conf G
--------- HttpConfigDirExt /etc/httpd/conf.d G
--------- HttpRoot /var/www/html G
--------- Info_CertificateCountry US G
--------- Info_CertificateLocality Solana Beach G
--------- Info_CertificateOrganization StackIQ G
--------- Info_CertificateState California G
--------- Info_ClusterContact ---------------------------------------------------------------------------- G
--------- Info_ClusterLatlong N32.87 W117.22 G
--------- Info_ClusterName Stacki G
--------- Info_ClusterURL http://stackihost.local/ G
--------- Info_FQDN stackihost.local G
--------- Kickstart_BoxDir /export/stack G
--------- Kickstart_DistroDir /export/stack G
--------- Kickstart_Keyboard us G
--------- Kickstart_Lang en_US G
--------- Kickstart_Langsupport en_US G
(continued)...

6. Stacki Global Configurations
[root@stacki]# stack add attr attr=datacenter value=DC-EAST
[root@stacki]# stack add attr attr=chef_server value=https://chef-dc-east.xyz.com:443/organizations/companyX
[root@stacki]# stack list attr
SCOPE ATTR VALUE SOURCE
--------- chef_server https://chef-dc-east.xyz.com:443/organizations/companyX G
--------- datacenter DC-EAST G

7. Chef Stacki Cart
[root@stacki]# stack add cart chef
[root@stacki]# stack list cart
NAME BOXES
chef: -----
[root@stacki]# tree /export/stack/carts/chef/
├── graph
│ └── cart-chef.xml
├── nodes
│ └── cart-chef-backend.xml
└── RPMS

8. Chef Config
/export/stack/carts/chef/nodes/cart-chef-backend.xml
<?xml version="1.0" standalone="no"?>
<kickstart>
<description>
chef cart backend appliance extensions
</description>
<package>chef</package>
<!-- shell code for post RPM installation -->
<post>
mkdir -p /etc/chef /var/log/chef /var/run/chef
</post>

9. Chef Config
XML tag Overview
● <post>
○ This tag allows the admin to run scripts after the package installation is done. The
scripts can be in any interpreted language present on the installed system.
● <boot order=”post”>
○ This section is a collection of shell scripts that are run on first-boot after all services
and daemons have started. This is typically used to run scripts that interact with
system daemons, or to finish and clean-up the installation process.

10. Chef Config
/export/stack/carts/chef/nodes/cart-chef-backend.xml
<post cond="not 'proxyout' in hostname">
############# DC-EAST ##########
if [ "&datacenter;" == "DC-EAST" ]
then
<file name="/etc/chef/client.rb">
<![CDATA[
# Chef Client Config File
# Dynamically generated by Stacki
log_level :info
log_location STDOUT
chef_server_url "#CHEF_SERVER#"
validation_client_name "prod-validator"
validation_key "/etc/chef/validation.pem"
client_key "/etc/chef/client.pem"
ssl_verify_mode :verify_none
http_proxy 'http://proxyout.dc-east.local.com:3128'
https_proxy 'http://proxyout.dc-east.local.com:3128'
environment 'production'
node_name "#HOSTNAME#"
]]>
</file>
# Need to add the chef server and client hostname to the client.rb file
sed -i 's,#CHEF_SERVER#,&chef_server;,g' /etc/chef/client.rb
sed -i 's/#HOSTNAME#/&hostname;.&domainname;/g' /etc/chef/client.rb
fi
</post>

11. Chef Config
/export/stack/carts/chef/nodes/cart-chef-backend.xml
<post cond="'proxyout' in hostname">
<file name="/etc/chef/client.rb">
<![CDATA[
# Chef Client Config File
# Dynamically generated by Stacki
log_level :info
log_location STDOUT
chef_server_url "#CHEF_SERVER#"
validation_client_name "prod-validator"
validation_key "/etc/chef/validation.pem"
client_key "/etc/chef/client.pem"
ssl_verify_mode :verify_none
environment 'production'
# Using default node name (fqdn)
node_name "#HOSTNAME#"
]]>
</file>
# Need to add the chef server and client hostname to the client.rb file
sed -i 's,#CHEF_SERVER#,&chef_server;,g' /etc/chef/client.rb
sed -i 's/#HOSTNAME#/&hostname;.&domainname;/g' /etc/chef/client.rb
</post>

12. Chef Config
/export/stack/carts/chef/nodes/cart-chef-backend.xml
<post>
if [ "&datacenter;" == "DC-EAST" ]
then
<file name="/etc/chef/first-boot.json">
{
"run_list": [
"role[base]",
"role[dc_east]"
]
}
</file>
<file name="/etc/chef/validation.pem" perms="0644">
-----BEGIN RSA PRIVATE KEY-----
asdfasdfasdfasdfasdfasdfasdfasdfasdfsadfasdfasdfasdfsadfasdfasdfasdfsadfsadfasdf
adsasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasfasdfasdfasdfasdfa
-----END RSA PRIVATE KEY-----
</file>
<file name="/etc/chef/encrypted_data_bag_secret" perms="0400">
asdfasdfasdfasdfasfdasfdasdfasfdasdfasfasdfasfdasdfasdfsafasfdasfdsadfasdfasdfasdf
adsfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasfdasdfasdfasdfasdfasdfa
</file>
fi
</post>
######### DC-EAST END ########

13. Chef Config: Server Provisioning
● Two attributes attached to a host control what happens to existing data on that host when it is configured
to be provisioned by setting the action attribute: “stack set host boot host1.local.com action=install”
○ nukecontroller
■ If set to “False”: Leaves the existing RAID controller setup as it is
■ If set to “True”: Removes all existing RAID groups and LUNS
○ nukedisks
■ If set to “False”: Only removes everything in the “/” and “/var/log” partition
■ If set to “True”: Will remove all partitions from all disks
[root@stacki]# stack list host attr host1.local.com |egrep 'nuke|HOST'
HOST SCOPE ATTR VALUE SOURCE
host1.local.com: --------- nukecontroller false H
host1.local.com: --------- nukedisks false H

14. Chef Config: Server Provisioning
/export/stack/carts/chef/nodes/cart-chef-backend.xml
# If we are nuking disks we are assuming this is a new server
# or the chef client/node has been deleted out of the chef server if it existed.
<boot order="post" cond="nukedisks">
# Run chef-client for the first time
/usr/bin/chef-client -j /etc/chef/first-boot.json -L /var/log/chef/chef.log
# Make a backup of the chef private key in case we need to re-provision/upgrade a server
mkdir -p /data/chef-backup
chown root:root /data/chef-backup
chmod 700 /data/chef-backup
cp -a /etc/chef/* /data/chef-backup
</boot>
[root@stacki]# stack set host attr host1.local.com attr=nukedisks value=true
[root@stacki]# stack list host attr host1.local.com |egrep 'nukedisks|HOST'
HOST SCOPE ATTR VALUE SOURCE
host1.local.com: --------- nukedisks true H

15. Chef Config: Server Provisioning
/export/stack/carts/chef/nodes/cart-chef-backend.xml
# If we are not nuking the disks we are assuming we are re-loading or upgrading
# the OS and need to keep the client.pem chef key so chef-client can run properly
<boot order="post" cond="not nukedisks">
cp /data/chef-backup/client.pem /etc/chef/
/usr/bin/chef-client -L /var/log/chef/chef.log
</boot>
</kickstart>

16. Chef Cart Config
● Put your chef RPM file in /export/stack/carts/chef/RPMS/ directory
● Assign your chef cart to a box
● Compile a cart when adding/removing RPMs to update the repo
[root@stacki]# stack enable cart chef box=default
[root@stacki]# stack compile cart chef
[root@stacki]# stack list cart chef
NAME BOXES
chef: default

17. Chef Cart Config
PRO TIP: When making xml config changes, always verify them by running the “stack list host
profile” command which does syntax checking and gives you the full generated kickstart file
[root@stacki]# stack list host profile host1.local.com|less
<?xml version="1.0" standalone="no"?>
<profile lang="kickstart">
<section name="kickstart">
<![CDATA[
#
# Node Traversal Order
#
# /export/stack/pallets/stacki/3.2/redhat/x86_64/nodes/partitions-save.xml (stacki)
# /export/stack/pallets/stacki/3.2/redhat/x86_64/nodes/pre-ssh-key.xml (stacki)
# /export/stack/pallets/stacki/3.2/redhat/x86_64/nodes/storage-controller-client.xml (stacki)
# /export/stack/pallets/stacki/3.2/redhat/x86_64/nodes/nukedisks.xml (stacki)
# /export/stack/pallets/stacki/3.2/redhat/x86_64/nodes/partition.xml (stacki)
(continued)...

18. Chef Cart Ordering
[root@stacki]# stack list cart
NAME BOXES
base: default
custom-rpms: default
chef: default
[root@stacki]# stack list host profile host1.local.com|less
<?xml version="1.0" standalone="no"?>
<profile lang="kickstart">
<section name="kickstart">
<![CDATA[
#
# Node Traversal Order
#
...
# /export/stack/pallets/stacki/3.2/redhat/x86_64/nodes/do-partition.xml (stacki)
# /export/stack/carts/chef/nodes/cart-chef-backend.xml (unknown)
# /export/stack/pallets/stacki/3.2/redhat/x86_64/nodes/base.xml (stacki)
# /export/stack/carts/base/nodes/cart-base-backend.xml (unknown)
# /export/stack/carts/custom-rpms/nodes/cart-custom-rpms-backend.xml (unknown)
(continued)...

19. Chef Cart Ordering
/export/stack/carts/chef/graph/cart-chef.xml
<?xml version="1.0" standalone="no"?>
<graph>
<description>
chef cart
</description>
<order head="backend" tail="cart-chef-backend"/>
<edge from="backend" to="cart-chef-backend"/>
</graph>
<?xml version="1.0" standalone="no"?>
<graph>
<description>
chef cart
</description>
<order head="backend" tail="cart-chef-backend"/>
<edge from="backend" to="cart-chef-backend"/>
<order head="cart-base-backend" tail="cart-chef-backend"/>
<edge from="cart-base-backend" to="cart-chef-backend"/>
</graph>

20. Chef Cart Ordering
[root@stacki]# stack list host profile host1.local.com|less
<?xml version="1.0" standalone="no"?>
<profile lang="kickstart">
<section name="kickstart">
<![CDATA[
#
# Node Traversal Order
#
...
# /export/stack/pallets/stacki/3.2/redhat/x86_64/nodes/do-partition.xml (stacki)
# /export/stack/carts/base/nodes/cart-base-backend.xml (unknown)
# /export/stack/carts/chef/nodes/cart-chef-backend.xml (unknown)
# /export/stack/pallets/stacki/3.2/redhat/x86_64/nodes/base.xml (stacki)
# /export/stack/carts/custom-rpms/nodes/cart-custom-rpms-backend.xml (unknown)
(continued)...

21. thank y u