STANDARDISE DEVELOPMENT ENVIRONMENTS
AND MACHINE IMAGES WITH PACKER
Marcelo Pinheiro	

http://salizzar.net - @salizzar
SUMMARY
• Motivation	

• Installation	

• How it works	

• InsideTemplates	

• Some examples	

• Our experience	

• FAQ
MOTIVATION
• How to take control inside the
following environment issues?	

• Common workstation problems
(HD failure, dea...
MOTIVATION
• Vagrant or Docker. Period.	

• Embrace virtualisation	

• Each offers a way to pre-setup yourVM with necessar...
MOTIVATION
• It sounds good, but… how to take control over each applicationVM?	

• Sometimes your team needs to use some t...
MOTIVATION
• For sysadms / sysops:	

• How to export a new machine image to your
virtualisation server (KVM, Xen,VMWare, e...
MOTIVATION
• Packer for the rescue	

• Written in Go	

• Owner: Mitchell Hashimoto
(Vagrant, Serf)	

• http://www.packer.io
INSTALLATION
• http://www.packer.io/downloads.html	

• Download zipped binaries for your OS	

• OSX	

• Linux	

• Windows	...
INSTALLATION
• Move binaries to your /usr/local/bin, ~/bin,
whatever	

• It’s done.
HOW IT WORKS
• Packer recipes are JSON files	

• Validate template:	

• $ packer validate your_recipe.json	

• Run template...
INSIDETEMPLATES
• Packer templates have the following structure:	

• Variables	

• Builders	

• Provisioners	

• Post-proc...
INSIDETEMPLATES:	

VARIABLES
• User-defined variables to be used along template	

• Can be declared in a custom file
INSIDETEMPLATES:	

VARIABLES
"variables": {!
"box_ostype": "centos",!
"box_osversion": "6.5",!
"box_nick": "6.5",!
"box_ar...
INSIDETEMPLATES:	

VARIABLES
# my_variables.json!
{!
"type": "vmware-iso",!
"vm_name": "mybox-vmw“,!
"guest_os_type": “cen...
INSIDETEMPLATES:	

BUILDERS
• Create a machine image from scratch	

• Download a ISO from official OS mirror, select a
base...
INSIDETEMPLATES:	

BUILDERS
"builders": [!
{!
"type": "virtualbox-iso",!
"iso_url": "http://an.repository.com/an-image.iso...
INSIDETEMPLATES:	

BUILDERS
• How to automate setup mundane tasks?	

• Minimal set of packages	

• Disk partition	

• Netw...
INSIDETEMPLATES:	

BUILDERS
• For CentOS: Kickstart	

• For Debian: Preseed	

• For Windows:	

• Windows Automated Install...
INSIDETEMPLATES:	

BUILDERS
• Available builders:	

• QEMU - KVM and Xen
(experimental)	

• VMWare	

• Virtualbox	

• Dock...
INSIDETEMPLATES:	

BUILDERS /VMWARE &VIRTUALBOX
• VMWare:	

• vmware-iso: create from scratch	

• vmware-vmx: create from ...
INSIDETEMPLATES:	

BUILDERS / QEMU
• Create KVM / Xen images from scratch	

• Packer depends on qemu-system-x86_64, availa...
INSIDETEMPLATES:	

BUILDERS / DOCKER
• Creates a Docker image by pulling a existent,
starting a container, provision it an...
INSIDETEMPLATES:	

BUILDERS / OTHERS
• For other builders, you simply need to inform:	

• username / password,API key	

• ...
INSIDETEMPLATES:
PROVISIONERS
• After the setup of a machine image, it’s time to configure
it	

• Here is where magic happe...
INSIDETEMPLATES:
PROVISIONERS
• Available provisioners:	

• Shell Scripts	

• File Uploads	

• Ansible	

• Chef Solo	

• P...
INSIDETEMPLATES:	

PROVISIONERS / SHELL SCRIPTS
• Most simple way to setup machine	

• Run apt-get, yum and friends
INSIDETEMPLATES:	

PROVISIONERS / SHELL SCRIPTS
"provisioners": [!
{!
"type": "shell",!
"execute_command": "echo 'root' | ...
INSIDETEMPLATES:	

PROVISIONERS / FILE UPLOADS
• Need to set default configuration files or upload
some custom packages (.ta...
INSIDETEMPLATES:	

PROVISIONERS / FILE UPLOADS
"provisioners": [!
{!
"type": "shell",!
"execute_command": "echo 'root' | s...
INSIDETEMPLATES:	

PROVISIONERS / OTHERS
• The following provisioners requires installation before run:	

• Ansible	

• Pu...
INSIDETEMPLATES:	

PROVISIONERS / OTHERS
"provisioners": [!
{!
"type": "ansible-local",!
"playbook_file": "recipes/ansible...
INSIDETEMPLATES:	

POST-PROCESSORS
• After create / setup a machine image, you can:	

• Convert to aVagrant Custom Box	

•...
INSIDETEMPLATES:	

POST-PROCESSORS /VAGRANT
• Defines a box output name	

• You can attach aVagrantfile template and other
t...
INSIDETEMPLATES:	

POST-PROCESSORS /VAGRANT
"post-processors": [!
{!
"type": "vagrant",!
“output": "lamp-vagrant.box"!
}!
]
INSIDETEMPLATES:	

POST-PROCESSORS / DOCKER
• You can locally import a Docker image	

• You can push a Docker image to a r...
INSIDETEMPLATES:	

POST-PROCESSORS / DOCKER
"post-processors": [!
{!
"type": "docker-import",!
"repository": "salizzar/pac...
INSIDETEMPLATES:	

POST-PROCESSORS /VSPHERE
• Upload to a vSphere endpoint
INSIDETEMPLATES:	

POST-PROCESSORS /VSPHERE
"post-processors": [!
{!
"type": "vsphere",!
"host": "a-vsphere-host.com",!
"u...
SOME EXAMPLES
• It’s time to see some code!	

• All examples are available on:	

• https://github.com/salizzar/packer-exam...
OUR EXPERIENCE
Ivan IVVasilyevich (theTerrible)
OUR EXPERIENCE
• Prepare to argue (sometimes fight :)	

• It’s hard to change development tradition of
premature optimisati...
OUR EXPERIENCE
• Sometimes the better choice must be autocracy-based	

• Use OS package system ASAP (or backport / automat...
OUR EXPERIENCE
• Make all applications ready-to-setup-and-run with one command	

• Track all dependencies with Dockerfile o...
OUR EXPERIENCE
• Divide to conquer	

• Adopt a bottom-up strategy	

• Minor systems that are easy to setup	

• Minor teams...
OUR EXPERIENCE
• At this time, major systems in Locaweb PaaS areVagrant-ready	

• git clone, vagrant up, vagrant ssh	

• D...
FAQ
• Questions?	

• New recipes available on:	

• https://github.com/salizzar/packer-vmware
THANKYOU!	

:)
Upcoming SlideShare
Loading in …5
×

Standardise development environments and machine images with packer

2,877 views

Published on

A introduction to Packer and how this tool can provide you a way to standardise development environments.

Published in: Technology
0 Comments
9 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,877
On SlideShare
0
From Embeds
0
Number of Embeds
66
Actions
Shares
0
Downloads
52
Comments
0
Likes
9
Embeds 0
No embeds

No notes for slide

Standardise development environments and machine images with packer

  1. 1. STANDARDISE DEVELOPMENT ENVIRONMENTS AND MACHINE IMAGES WITH PACKER Marcelo Pinheiro http://salizzar.net - @salizzar
  2. 2. SUMMARY • Motivation • Installation • How it works • InsideTemplates • Some examples • Our experience • FAQ
  3. 3. MOTIVATION • How to take control inside the following environment issues? • Common workstation problems (HD failure, dead computer) • Different OS’s • Extra machine configuration to enable devs to work (programming languages, databases, plugins, etc), taking one or two days to be ready-to-code • “Works on my machine” syndrome
  4. 4. MOTIVATION • Vagrant or Docker. Period. • Embrace virtualisation • Each offers a way to pre-setup yourVM with necessary libraries, databases and so on • No more development databases in your pre-staging DB server, additional dependencies • A try to make development environment more similar to production
  5. 5. MOTIVATION • It sounds good, but… how to take control over each applicationVM? • Sometimes your team needs to use some tools that are not available in official package repository (or are too old), forcing to manual configuration after up aVM • Even running aVM, developer personal choices can contaminate the application (example: rspec add-ons, irb plugins) • Some developers don’t have knowledge about Chef / Puppet recipes • How to maintainVagrant Custom Boxes / Docker Custom Images when you need to add / change tools, repositories or configs?
  6. 6. MOTIVATION • For sysadms / sysops: • How to export a new machine image to your virtualisation server (KVM, Xen,VMWare, etc) when a new OS release is launched without “dist-upgrade”? • How to automate it?
  7. 7. MOTIVATION • Packer for the rescue • Written in Go • Owner: Mitchell Hashimoto (Vagrant, Serf) • http://www.packer.io
  8. 8. INSTALLATION • http://www.packer.io/downloads.html • Download zipped binaries for your OS • OSX • Linux • Windows • FreeBSD • OpenBSD
  9. 9. INSTALLATION • Move binaries to your /usr/local/bin, ~/bin, whatever • It’s done.
  10. 10. HOW IT WORKS • Packer recipes are JSON files • Validate template: • $ packer validate your_recipe.json • Run template: • $ packer run your_recipe.json
  11. 11. INSIDETEMPLATES • Packer templates have the following structure: • Variables • Builders • Provisioners • Post-processors
  12. 12. INSIDETEMPLATES: VARIABLES • User-defined variables to be used along template • Can be declared in a custom file
  13. 13. INSIDETEMPLATES: VARIABLES "variables": {! "box_ostype": "centos",! "box_osversion": "6.5",! "box_nick": "6.5",! "box_arch": "x64",! "box_type": "base",! "box_memory": "512",! "box_cpus": "1",! ! "iso_arch": "x86_64",! "iso_type": "netinstall",! "iso_md5": "939fd1d87c11ffe7795324438b85adfb",! ! "ssh_user": "vagrant",! "ssh_pass": "vagrant",! "hostname": "vagrant-centos-6.5",! "domain": "vagrantup.com"! }
  14. 14. INSIDETEMPLATES: VARIABLES # my_variables.json! {! "type": "vmware-iso",! "vm_name": "mybox-vmw“,! "guest_os_type": “centos",! "disk_size": “4096”,! ! (… other definitions here…)! }!
  15. 15. INSIDETEMPLATES: BUILDERS • Create a machine image from scratch • Download a ISO from official OS mirror, select a base image to start • Set CPU cores, memory size, disk size • See documentation for further details (a lot of options)
  16. 16. INSIDETEMPLATES: BUILDERS "builders": [! {! "type": "virtualbox-iso",! "iso_url": "http://an.repository.com/an-image.iso",! "iso_checksum": "an-checksum",! "iso_checksum_type": "md5",! "http_directory": "http",! "ssh_username": "root",! "ssh_password": "apassword",! "ssh_wait_timeout": "100000s",! "shutdown_command": "echo {{ user `ssh_user` }} | sudo halt -p",! ! "boot_command": [! "<esc> ",! "install ",! "auto “,! ! (… other definitions here …)! ! "<enter><wait>"! ]! }! ]
  17. 17. INSIDETEMPLATES: BUILDERS • How to automate setup mundane tasks? • Minimal set of packages • Disk partition • Network • Timezone
  18. 18. INSIDETEMPLATES: BUILDERS • For CentOS: Kickstart • For Debian: Preseed • For Windows: • Windows Automated Installation Kit (AIK) • Microsoft DeploymentToolkit (MDT)
  19. 19. INSIDETEMPLATES: BUILDERS • Available builders: • QEMU - KVM and Xen (experimental) • VMWare • Virtualbox • Docker ! • OpenStack • Google Compute Engine • Amazon EC2 • Digital Ocean
  20. 20. INSIDETEMPLATES: BUILDERS /VMWARE &VIRTUALBOX • VMWare: • vmware-iso: create from scratch • vmware-vmx: create from a baseVMX file • Virtualbox: • virtualbox-iso: create from scratch • virtualbox-ovf: create from a base OVF file
  21. 21. INSIDETEMPLATES: BUILDERS / QEMU • Create KVM / Xen images from scratch • Packer depends on qemu-system-x86_64, available only on Debian at this time as a binary • CentOS have qemu-kvm, but you need to manually override all Packer default options
  22. 22. INSIDETEMPLATES: BUILDERS / DOCKER • Creates a Docker image by pulling a existent, starting a container, provision it and exports a .tar file • Provision without Dockerfile
  23. 23. INSIDETEMPLATES: BUILDERS / OTHERS • For other builders, you simply need to inform: • username / password,API key • base image • zone and other related information • See Packer documentation
  24. 24. INSIDETEMPLATES: PROVISIONERS • After the setup of a machine image, it’s time to configure it • Here is where magic happens: • Add packages, useful scripts • Standardise config files • Apply existent recipes from a CM
  25. 25. INSIDETEMPLATES: PROVISIONERS • Available provisioners: • Shell Scripts • File Uploads • Ansible • Chef Solo • Puppet • Salt
  26. 26. INSIDETEMPLATES: PROVISIONERS / SHELL SCRIPTS • Most simple way to setup machine • Run apt-get, yum and friends
  27. 27. INSIDETEMPLATES: PROVISIONERS / SHELL SCRIPTS "provisioners": [! {! "type": "shell",! "execute_command": "echo 'root' | sh '{{ .Path }}'",! "scripts": [! "scripts/locale.sh",! "scripts/elrepo.sh"! ]! },! {! "type": "shell",! "pause_before": "30s",! "execute_command": "echo 'root' | sh '{{ .Path }}'",! "scripts": [! "scripts/vagrant.sh",! "scripts/sudoers.sh"! ]! }! ]
  28. 28. INSIDETEMPLATES: PROVISIONERS / FILE UPLOADS • Need to set default configuration files or upload some custom packages (.tar, .deb / .rpm) to be installed later? • Upload them and after process with a shell script or CM recipe
  29. 29. INSIDETEMPLATES: PROVISIONERS / FILE UPLOADS "provisioners": [! {! "type": "shell",! "execute_command": "echo 'root' | sh '{{ .Path }}'",! "scripts": [! "scripts/lamp/vagrant.sh",! "scripts/lamp/apache2.sh",! "scripts/lamp/php5.sh",! "scripts/lamp/mysql.sh"! ]! },! {! "type": "file",! "source": "files/lamp-vagrant/vhost",! "destination": "/etc/apache2/sites-available/lamp-php"! },! {! "type": "shell",! "execute_command": "echo 'root' | sh '{{ .Path }}'",! "script": "scripts/lamp/enable-vhost"! }! ]
  30. 30. INSIDETEMPLATES: PROVISIONERS / OTHERS • The following provisioners requires installation before run: • Ansible • Puppet • Salt • Chef Solo is installed by Packer if not present • At this time, all provisioners are executed in client mode (no remote server)
  31. 31. INSIDETEMPLATES: PROVISIONERS / OTHERS "provisioners": [! {! "type": "ansible-local",! "playbook_file": "recipes/ansible/lamp.yml"! },! {! "type": "chef-solo",! "cookbook_paths": [ "recipes/chef/lamp" ]! },! {! "type": "puppet-masterless",! "manifest_file": "recipes/puppet/lamp"! },! {! "type": "salt-masterless",! "local_state_tree": "recipes/salt/lamp"! }! ]
  32. 32. INSIDETEMPLATES: POST-PROCESSORS • After create / setup a machine image, you can: • Convert to aVagrant Custom Box • Locally add it as a Docker container • Publish in a Docker registry • Publish in a vSphere endpoint
  33. 33. INSIDETEMPLATES: POST-PROCESSORS /VAGRANT • Defines a box output name • You can attach aVagrantfile template and other template files (cookbooks) • Change compression rate if you want
  34. 34. INSIDETEMPLATES: POST-PROCESSORS /VAGRANT "post-processors": [! {! "type": "vagrant",! “output": "lamp-vagrant.box"! }! ]
  35. 35. INSIDETEMPLATES: POST-PROCESSORS / DOCKER • You can locally import a Docker image • You can push a Docker image to a registry • Needs manual login (automated soon) • Important: Docker pushes a completely new image, not incremental
  36. 36. INSIDETEMPLATES: POST-PROCESSORS / DOCKER "post-processors": [! {! "type": "docker-import",! "repository": "salizzar/packer",! "tag": "0.1"! },! "docker-push"! ]
  37. 37. INSIDETEMPLATES: POST-PROCESSORS /VSPHERE • Upload to a vSphere endpoint
  38. 38. INSIDETEMPLATES: POST-PROCESSORS /VSPHERE "post-processors": [! {! "type": "vsphere",! "host": "a-vsphere-host.com",! "username": "my_user",! "password": "my_password",! "cluster": "a-cluster",! “datacenter": "xyz",! "datastore": "zyx",! “resource_pool": "zyx",! "vm_folder": "images",! "vm_name": "lamp",! “vm_network": "staging"! },! ]
  39. 39. SOME EXAMPLES • It’s time to see some code! • All examples are available on: • https://github.com/salizzar/packer-examples
  40. 40. OUR EXPERIENCE Ivan IVVasilyevich (theTerrible)
  41. 41. OUR EXPERIENCE • Prepare to argue (sometimes fight :) • It’s hard to change development tradition of premature optimisation, ultra-high performance, personal choices,“nightly build” syndrome • Create a culture first
  42. 42. OUR EXPERIENCE • Sometimes the better choice must be autocracy-based • Use OS package system ASAP (or backport / automate installation if package not exists) • Introduce to developers a wisdom to use the same package of programming language / DB / whatever that runs in production (!) • If is old, upgrade your app to use a newer version • The same for tools that “vendorize" your app libraries (maven, bundler, etc)
  43. 43. OUR EXPERIENCE • Make all applications ready-to-setup-and-run with one command • Track all dependencies with Dockerfile orVagrant Shell Scripts • Bash scripts are more easy to setup than 3rd party CM tools at first time • Adopt a convention to make all applications more similar as possible about their structure • Code generators
  44. 44. OUR EXPERIENCE • Divide to conquer • Adopt a bottom-up strategy • Minor systems that are easy to setup • Minor teams • Start to apply with more systems and greater teams • Standardise ASAP
  45. 45. OUR EXPERIENCE • At this time, major systems in Locaweb PaaS areVagrant-ready • git clone, vagrant up, vagrant ssh • Docker in development • Internally created a gem to apply standardisation of Rails apps: • Packaging (Debian) • Vagrant • Packer recipes to createVagrant custom boxes, using our mirrors
  46. 46. FAQ • Questions? • New recipes available on: • https://github.com/salizzar/packer-vmware
  47. 47. THANKYOU! :)

×