An introduction to Stacki-the fastest bare metal Linux server provisioning tool from the Stacki Atlanta kickoff meetup on 2/23/16 at the Microsoft Innovation Center. Greg Bruno is the VP Engineering at StackIQ.

1. Introduction to Stacki
Greg Bruno, PhD
VP Engineering, StackIQ

2. Open Source Stack Installer
cki is a very fast and ultra reliable Linux server provisioning tool … at sca
h zero prerequisites for taking systems from bare metal to a ping and prom

3. PayPal

4. Hadoop @ PayPal
12 x 2TB SATA
data drives
48 nodes
each rack
1GBE-10GBE
NICs
24 x 900GB 6G SA
10K data drives
24 nodes
each rack
10GBE NIC
8 x 4TB NR-SAS
data drives
10 GBE NIC
Bay Area
Salt Lake City
Las Vegas
DATACENTERS
000 nodes and growing
+ initial server racks
terogeneous HW
oss multiple DCs
Science
structure Footprint
48 nodes
each rack

5. Stacki @ PayPal
Ambari HDP
Health Detectio
Integration
IPMI/iLOOS Disk Network
DHCP / DNS /
TFTP
Ansible
- Disk Array Controller Configuration
- Disk Partitioning Configuration
“Stacki + Ansible = Happiness. :D” – Stacki mailing list 8/11/15

6. Quick, Early Success
14 Minutes*To Fully Provision 6 Racks of Bare Metal (288 Servers)
s wiping all
en fully
ning & formatting
rives
And Now…
Upgrades all firmware
automatically
Executes Ansible
scripts on all hosts
Hadoop packages
installed
us hours with other hyperscale management tools, or days to weeks with traditional tools and proc

7. How We Solve the Problem

8. Stacki Positioning
DevOps / Configuration Tool
DHCP /
DNS / TFTP
NetworkDiskOS
In-house
developed
deployment
tools
- Disk Array Controller Configuration
- Disk Partitioning Configuration

9. Datacenter Architecture
Frontend
Network
Backend Backend Backend Backend
em1 em1
em1
em1
em1

10. Download and Boot the ISO
www.stacki.com and download the ISO
s 1.2 GB
“stacki” pallet plus stripped down CentOS 6.7
the ISO on the host that will be your frontend

17. Frontend Services
ces to build backend nodes
HCP
TP
amed (optional)
ces to access backend nodes
SH key management
rallel execution shell

18. Host Configuration Spreadsheet

19. Frontend
Network
Backend Backend Backend B
em1 em1
em1
em1
em1
Backend Installation
your Host Configuration spreadsheet as a CSV
t CSV on frontend
tack load hostfile file=hosts.csv”
ackend nodes to install on their next PXE boot
tack set host boot backend action=install”
boot all backend nodes

20. BitTorrent-Inspired Package Installation
Stacki

21. Customizing Your Hosts

22. Advanced Networking
ost Configuration spreadsheet, you can configure:
onded interfaces
ANs
idging
y combo of the above
ge hosts in multiple subnets
uild a single cluster from hosts in multiple subnets
anage hosts in multiple datacenters

23. Host Configuration Spreadsheet

24. Disk Controller Configuration Spreadsheet

25. Disk Partition Configuration Spreadsheet

26. Multiple Distributions
ntend houses a default distribution
ased on stripped down CentOS 6.7 or 7.2
sed to build backend nodes
dd any number of new distributions to a frontend
g., RHEL 6.x based distro, CentOS 6.5, etc.
n any backend node to any distro

27. Why is this hard and important?

28. The “Step 0” Problem
Check namenodes are
empty
Format/start HDFS
Create all directories
Create all metastores
Start services (Hbase, Hive,
Oozie, Sqoop, Impala, etc)
Deploy client configuration Configure database
Setup/assign monitors
(activity, services, and host)
Test database connections
Validate/resolve hostnamesConsistent host timezones
No bad kernel versions
running
(CDH) version consistency
Java version consistency
Daemons versions
consistency
Mgmt Agents versions
consistency
Host specification/SSH
ports
MUCH MORE …
DHCP Server/Client setup TFTP/PXE configuration
Server OS installation
Node OS Install
RAID configuration
Boot configuration
System/data disk
partitioning
Monitoring system setup
and config
Lights Out/IPMI setup
User accounts added and
synced
SSH keys on all hosts
Network node configuration
Config Mgmt install and
configuration
Route configurationOS upgrades/updates
Site specific software and
configuration
Host specification/SSH
ports
Security
Firewall setupCluster Mgmt utility Database install and config
Multiple network configPackage installation MUCH MORE …

29. Clusters are Different
ng new servers does require coordination
y added servers must:
Have same software stack as original
servers
Have same configuration as original
servers
Know about original servers
original servers must:
Know about new servers
lt: The management complexity added to the
ations staff is “exponential”

30. Exponential Complexity
Number of Servers
ManagementComplexity
General Data Center
Clusters

31. The Pain Curve
Number of Servers
ManagementComplexity
General Data Center
Clusters
PAIN

32. The Pain Threshold
pain threshold differs for every
nization
ion of:
cluster(s) size
number of people in Operations
Operations staff cluster expertise

33. Moore’s Law
50 1 2 3 4
8
1
2
3
4
5
6
7
Time (Years)
Density
18 month
doubling

34. Moore’s Law and Infrastructure Value

35. What it Means for You
50 1 2 3 4
100
0
10
20
30
40
50
60
70
80
90
Time (Years)
Value(%)
3 months
90% value
18 months
50% value

36. Time is Money
clock starts ticking when hosts land on your
ng dock
out your applications online, you have an
r weight that consumes power, cooling, and
agement’s attention

37. Try It Out

38. stacki.com
Download - www.stacki.com
Source & Docs - github.com/StackIQ/stacki/wiki
Discuss - groups.google.com/forum/#!forum/stacki

39. History
n Diego Supercomputer Center
1986 - National Science Foundation
Along with NCSA only two non-classified centers
Mission: serve computational scientists
cks
2000 - First cluster group inside SDSC
Version 1.0 released that November as open source
10k+ clusters world-wide
ckIQ
2006 - Commercial support for Rocks
2011 - Venture Backed
Focus on next generation clustered systems (Data, Cloud)
cki - 2015
June – released as open source
July – first hyper-scale user

40. Must Haves
it – Automatic
nk about it, test it. Deploy it.
ople don’t scale, software does. Free your people – allow ops guys to be ops/analysis guys, move them from single machine vie
bal machine view.
it – Repeatable
te of the environment is guaranteed. Does not require homogeneity of hardware or functionality. Make compute environments
mogenous on heterogeneous hardware and software.
ally, nothing is homogenous. Environment maybe, behavior of that environment on different machines while predictable will not be
me across all hardware. Stacki gets you flexibility and predictability.
it – Reliable
u always get what you want when you want it. You can make reasonable estimates of need because you’ve made the environmen
dictable and repeatable. Just like science!
it – Comprehensive
nage application layer(s) down to kernels and device configuration with one tool. Never hit the network unconfigured.
vide turn-key deployment with reasonable default settings and ability to customize / re-wire as desired.

41. PayPal’s Options
Bring what we used at former parent company eBay with us.
Build our own soups-to-nuts bespoke bare metal provisioning tool.
Find the perfect open source tool that we can use and grow with.
Not Possible
Not Optimal
Not Likely

42. Quick, Early Success
2 Weeks Instead of 2 Years
To Build a Scale-out Management Solution
stalled Stacki Frontend (base management server)
an test installations of backend servers
Single Server test
Full Rack test (48 nodes)
dated distribution (CentOS 6.6) to install additional
kages
grated IPMI information into Stacki
Can now ssh into all IPMI consoles from the Stacki
frontend host using <hostname>.ipmi
ran with PayPal kickstart changes/additions and was
e to image 6 racks in 14 minutes, including:
Nuking disks/partitions and running a full format of all
data drives
5. Updated the Stacki post-boot piece to do the follo
1. Upgrade firmware if host needs it
2. Runs PayPal Ansible playbook, which:
1. Installs additional packages
2. Creates user accounts
3. Disables unused services
4. Sets up resolver/ntp/syslog-ng/sudoers/lim
d/sysctl/etc.
5. Installs/configures Ambari agents
6. Checks data drive mounts, fstab
7. Prepares the rack to be added to a Hadoo
cluster
development with Stacki includes:

43. DevOps Agnostic
DevOps / Configuration Tool
DHCP /
DNS / TFTP
NetworkDiskOS
In-house
developed
deployment
tools
- Disk Array Controller Configuration
- Disk Partitioning Configuration

44. The “Step 0” Problem
k namenodes are
empty
Format/start HDFS
ate all directories
Create all metastores
Start services (Hbase, Hive,
Oozie, Sqoop, Impala, etc)
client configuration Configure database
Setup/assign monitors
(activity, services, and host)
Test database connections
Validate/resolve hostnamestent host timezones
ad kernel versions
running
version consistency
Java version consistency
Daemons versions
consistency
Mgmt Agents versions
consistency
Host specification/SSH
ports
MUCH MORE …
Server/Client setup TFTP/PXE configuration
er OS installation
Node OS Install
RAID configuration
ot configuration
System/data disk
partitioning
Monitoring system setup
and config
Lights Out/IPMI setup
User accounts added and
synced
keys on all hosts
k node configuration
g Mgmt install and
configuration
Route configurationOS upgrades/updates
Site specific software and
configuration
Host specification/SSH
ports
Security
Firewall setupster Mgmt utility Database install and config
Multiple network configkage installation MUCH MORE …
App Confi
Site Confi
HW Insta
System Performance
Validation
Bare Metal Installers
Hadoop Mgmt Tool
Upgrades/Patching
Disk Configuration
Monitoring Tool
Configuration Tool
Network/Site Config ToolsSystems Mgmt Tool
Others …
MANUAL
SEMI-AUTOMATED
TOOLCHAIN
(w/o StackIQ)
w/Stack
FULLY AUTO

45. StackIQ Boss

46. Configuration Database
Server appliance types (e.g. data, namenode, tomcat, …)
Number of CPUs
Disk partitioning
Hardware RAID config
PCI bus information
…
And other System Attributes

47. Attributes
Global
◦ stack set attr
Appliance
◦ stack set appliance attr
OS
◦ stack set os attr
Host
◦ stack set host attr

48. Kickstart Profiles

49. Zoom In

50. Starting from the Empty Set
{ }

51. { os }
© 2

52. { os, core }
© 2

53. { os, core, kernel }
© 2

54. { os, core, kernel, mapr }
© 2

55. Manage the Deltas
{os, core, kernel, mapr} {os, core, kernel, horton}
© 2

56. stacki.com
@masonkatz