Stacki Lab
Joe Kaiser
Director of Open Source Engineering
Open Source Stack Installer
Stacki is a very fast and ultra reliable Linux server provisioning tool … at scale.
With zero prerequisites for taking systems from bare metal to a ping and prompt.
Stuff it does
 Installs to bare metal or VMs that look like bare metal
 Kickstart based
 Parallel formatting of disk
 Parallel Sharing of RPMs
 CentOS/RHEL
 Networking
 Multiple subnets, vlaning, bonding.
 Storage
 Controller config
 Partitioning
Whatever you can do in Linux, you can do with Stacki only clustered
Stacki and Hortonworks Data Platform
Hortonworks – stacki-hdp-bridge pallet
 Add/enable/run
 Stacki creates ambari appliance
 Download software as pallets (isos)
 Add/enable HDP and Ambari
 Run gethdp script
 Or download (links in documentation)
 Assign a node to be Ambari deployment.
 Do partitions and preps backend nodes for Hadoop.
 Install all.
 Go to Ambari interface to deploy Hadoop.
 Current versions: (updated /export/HDP/hdp.cfg for new versions)
 distribution = 2.x
 os = centos7
 ambari = 2.4.2.0
 hdp = 2.5.3.0
Augment the “default” Box
1) Replace the “os” pallet with the “CentOS” and “CentOS-
Updates” pallets
2) Add the Hortonworks pallets: “HDP”, “HDP-UTILS”,
“Updates-ambari” or run /export/HDP/gethdp.py
3) Add a pallet to glue the two layers together: “stacki-
hdp-bridge”
Specify One Host as the “Ambari” appliance
“backend-0-0” will be automatically configured with Ambari
Install the node and then point your web browser at it
Reinstall All Backend Nodes
Wipe all hardware disk array configuration and rebuild all the LUNs:
◦ stack set host attr ambari backend attr=nukecontroller value=true
Remove all partitions then repartition and reformat the disks:
◦ stack set host attr ambari backend attr=nukedisks value=true
Instruct nodes to install on next PXE boot:
◦ stack set host boot ambari backend action=install
Try It
 Website
www.stacki.com
 Vagrant tire kick
https://github.com/rfkrocktk/vagrant-stacki
Source Code and docs
github.com/stackiq/stacki
github.com/StackIQ/stacki-hdp-bridge
Slack Channel (because everyone)
Google Groups
groups.google.com/forum/#!forum/stacki
Finis
Thanks

Building a Hadoop Cluster with Stacki

  • 1.
    Stacki Lab Joe Kaiser Directorof Open Source Engineering
  • 2.
    Open Source StackInstaller Stacki is a very fast and ultra reliable Linux server provisioning tool … at scale. With zero prerequisites for taking systems from bare metal to a ping and prompt.
  • 3.
    Stuff it does Installs to bare metal or VMs that look like bare metal  Kickstart based  Parallel formatting of disk  Parallel Sharing of RPMs  CentOS/RHEL  Networking  Multiple subnets, vlaning, bonding.  Storage  Controller config  Partitioning Whatever you can do in Linux, you can do with Stacki only clustered
  • 4.
  • 5.
    Hortonworks – stacki-hdp-bridgepallet  Add/enable/run  Stacki creates ambari appliance  Download software as pallets (isos)  Add/enable HDP and Ambari  Run gethdp script  Or download (links in documentation)  Assign a node to be Ambari deployment.  Do partitions and preps backend nodes for Hadoop.  Install all.  Go to Ambari interface to deploy Hadoop.  Current versions: (updated /export/HDP/hdp.cfg for new versions)  distribution = 2.x  os = centos7  ambari = 2.4.2.0  hdp = 2.5.3.0
  • 6.
    Augment the “default”Box 1) Replace the “os” pallet with the “CentOS” and “CentOS- Updates” pallets 2) Add the Hortonworks pallets: “HDP”, “HDP-UTILS”, “Updates-ambari” or run /export/HDP/gethdp.py 3) Add a pallet to glue the two layers together: “stacki- hdp-bridge”
  • 7.
    Specify One Hostas the “Ambari” appliance “backend-0-0” will be automatically configured with Ambari Install the node and then point your web browser at it
  • 8.
    Reinstall All BackendNodes Wipe all hardware disk array configuration and rebuild all the LUNs: ◦ stack set host attr ambari backend attr=nukecontroller value=true Remove all partitions then repartition and reformat the disks: ◦ stack set host attr ambari backend attr=nukedisks value=true Instruct nodes to install on next PXE boot: ◦ stack set host boot ambari backend action=install
  • 9.
    Try It  Website www.stacki.com Vagrant tire kick https://github.com/rfkrocktk/vagrant-stacki Source Code and docs github.com/stackiq/stacki github.com/StackIQ/stacki-hdp-bridge Slack Channel (because everyone) Google Groups groups.google.com/forum/#!forum/stacki
  • 10.

Editor's Notes

  • #3 Linux – Focused on RedHat-ish (Kickstart/Anaconda) Provisioning – Bare Metal (total stack control) Scale – solve 1000+ servers problem then scale down Ping and Prompt – Get machine up to known base OS fully configuration raid / disk / networking / ssh access on Nothing else … No agent left on the server