Slides for the path that the Productivity Engineering team at Box took to move from bare-metal servers to a modern cloud platform, consisting of OpenStack and AWS. This was done on the back of two open-source tools by HashiCorp: Packer & Terraform. The Productivity Engineering exists to make other engineers at Box more productive.

1. Nadeem Ahmad
Productivity Engineering’s
Journey to the Cloud

2. • Inelastic
• Host divergence
• Dependence on other teams for changes
State of our infrastructure

3. G

4. Privat
e
Publi
c

5. Privat
e
Publi
c

9. Develop abstract system that
works across multiple cloud
platforms

10. Source: https://martinfowler.com/bliki/ConfigurationSynchronization.html

11. Source: https://martinfowler.com/bliki/ConfigurationSynchronization.html

12. Source: https://martinfowler.com/bliki/ImmutableServer.html

13. Source: https://martinfowler.com/bliki/ImmutableServer.html

14. Build
Image
s
Verify
Image
s
Deploy
Instance
s
Phase 1 Phase 2

15. • Easy-to-use
• Self-service
• Fully automated
• Built-in validation
Requirements

16. Phase 1: Build & Verify Images

17. https://www.packer.io/
Packer

18. Packer supports multiple cloud
platforms.

19. {
"provisioners": [
{
"type": "shell",
"inline": [
"yum install -y python33"
]
}
],
"builders": [
{
"name": "openstack",
"identity_endpoint": "http://openstack.host.com",
"username": "user",
"password": "secret",
"source_image": "centos7.qcow2",
"type": "openstack"
}
]
}
Sample packer.json

20. {
"provisioners": [
{
"type": "shell",
"inline": [
"yum install -y python33"
]
}
],
"builders": [
{
"name": "openstack",
"identity_endpoint": "http://openstack.host.com",
"username": "user",
"password": "secret",
"source_image": "centos7.qcow2",
"type": "openstack"
}
]
}
Sample packer.json

21. {
"provisioners": [
{
"type": "shell",
"inline": [
"yum install -y python33"
]
}
],
"builders": [
{
"name": "openstack",
"identity_endpoint": "http://openstack.host.com",
"username": "user",
"password": "secret",
"source_image": "centos7.qcow2",
"type": "openstack"
}
]
}
Sample packer.json

22. {
"provisioners": [
{
"type": "shell",
"inline": [
"yum install -y python33"
]
}
],
"builders": [
{
"name": "openstack",
"identity_endpoint": "http://openstack.host.com",
"username": "user",
"password": "secret",
"source_image": "centos7.qcow2",
"type": "openstack"
}
]
}
Sample packer.json

23. {
"provisioners": [
{
"type": "shell",
"inline": [
"yum install -y python33"
]
}
],
"builders": [
{
"name": "openstack",
"identity_endpoint": "http://openstack.host.com",
"username": "user",
"password": "secret",
"source_image": "centos7.qcow2",
"type": "openstack"
}
]
}
Sample packer.json

24. Solution: Image Pipeline

25. Bake
Source Image
Provisioner 1
Provisioner 2
Provisioner N
Baked
Image
Verified
Image
Verify

26. {
"provisioners": [
{
"type": "shell",
"inline": [
"yum install -y python33"
]
}
],
"builders": [
{
"name": "openstack",
"identity_endpoint": "http://openstack.host.com",
"username": "user",
"password": "secret",
"source_image": "centos7.qcow2",
"type": "openstack"
}
]
}
---
source_image: centos7.qcow2
provisioners: [install_python33.json]
verifiers: [verify_python33.json]
Comparison: Building python3.3 image
Packer Image Pipeline

27. ---
source_image: centos7.qcow2
provisioners: [install_python33.json]
verifiers: [verify_python33.json]
Sample image_pipeline.yaml file

28. ---
source_image: centos7.qcow2
provisioners: [install_python33.json]
verifiers: [verify_python33.json]
Sample image_pipeline.yaml file

29. ---
source_image: centos7.qcow2
provisioners: [install_python33.json]
verifiers: [verify_python33.json]
Sample image_pipeline.yaml file

30. {
"provisioners": [
{
"type": "shell",
"inline": [
"yum install -y python33"
]
}
],
"builders": [
{
"name": "openstack",
"identity_endpoint": "http://openstack.host.com",
"username": "user",
"password": "secret",
"source_image": "centos7.qcow2",
"type": "openstack"
}
]
}
Sample packer.json

31. ---
source_image: centos7.qcow2
provisioners: [install_python33.json]
verifiers: [verify_python33.json]
Sample image_pipeline.yaml file

32. Abstracted
Credentials
User
Friendly
Validation Reusability
Benefits of Image Pipeline

33. Build
Image
s
Verify
Image
s
Deploy
Instance
s
Phase 1 Phase 2

34. Build
Image
s
Verify
Image
s
Deploy
Instance
s
Phase 1: Image Pipeline Phase 2

35. Build
Image
s
Verify
Image
s
Deploy
Instance
s
Phase 1: Image Pipeline Phase 2

36. Phase 2: Deploying Instances

37. https://www.terraform.io/
Terraform

38. Image
Create
Instanc
e
Initializ
e
Instanc
e

39. Create
Instanc
e
Initializ
e
Instanc
e
AMI

40. provider "aws" {
access_key = "${var.aws_access_key}"
secret_key = "${var.aws_secret_key}"
region = "us-west-2"
}
resource "aws_instance" "jenkins_node" {
ami = "ami-id"
instance_type = "t2.large"
count = "100"
provisioner "local-exec" {
command = "add_jenkins_node --ip-address
${self.private_ip} --master 'jenkins.pod.box.net' --labels
'rosie'"
}
}
Sample terraform_config.tf in AWS

41. provider "aws" {
access_key = "${var.aws_access_key}"
secret_key = "${var.aws_secret_key}"
region = "us-west-2"
}
resource "aws_instance" "jenkins_node" {
ami = "ami-id"
instance_type = "t2.large"
count = "100"
provisioner "local-exec" {
command = "add_jenkins_node --ip-address
${self.private_ip} --master 'jenkins.pod.box.net' --labels
'rosie'"
}
}
Sample terraform_config.tf in AWS

42. provider "aws" {
access_key = "${var.aws_access_key}"
secret_key = "${var.aws_secret_key}"
region = "us-west-2"
}
resource "aws_instance" "jenkins_node" {
ami = "ami-id"
instance_type = "t2.large"
count = "100"
provisioner "local-exec" {
command = "add_jenkins_node --ip-address
${self.private_ip} --master 'jenkins.pod.box.net' --labels
'rosie'"
}
}
Sample terraform_config.tf in AWS

43. provider "aws" {
access_key = "${var.aws_access_key}"
secret_key = "${var.aws_secret_key}"
region = "us-west-2"
}
resource "aws_instance" "jenkins_node" {
ami = "ami-id"
instance_type = "t2.large"
count = "100"
provisioner "local-exec" {
command = "add_jenkins_node --ip-address
${self.private_ip} --master 'jenkins.pod.box.net' --labels
'rosie'"
}
}
Sample terraform_config.tf in AWS

44. ---
provisioner:
jenkins_node:
master: 'jenkins.pod.box.net'
labels: 'rosie'
providers:
aws:
us-west-2:
ami-id: 'ami-id'
count: 100
instance_type: 't2.large'
openstack
dsv31:
image_name: 'jenkins_node_rosie'
count: 50
flavor: 'm1.large'
Custom Terraform Definition

45. ---
provisioner:
jenkins_node:
master: 'jenkins.pod.box.net'
labels: 'rosie'
providers:
aws:
us-west-2:
ami-id: 'ami-id'
count: 100
instance_type: 't2.large'
openstack
dsv31:
image_name: 'jenkins_node_rosie'
count: 50
flavor: 'm1.large'
Custom Terraform Definition

46. ---
provisioner:
jenkins_node:
master: 'jenkins.pod.box.net'
labels: 'rosie'
providers:
aws:
us-west-2:
ami-id: 'ami-id'
count: 100
instance_type: 't2.large'
openstack
dsv31:
image_name: 'jenkins_node_rosie'
count: 50
flavor: 'm1.large'
Custom Terraform Definition

47. ---
provisioner:
jenkins_node:
master: 'jenkins.pod.box.net'
labels: 'rosie'
providers:
aws:
us-west-2:
ami-id: 'ami-id'
count: 100
instance_type: 't2.large'
openstack
dsv31:
image_name: 'jenkins_node_rosie'
count: 50
flavor: 'm1.large'
Custom Terraform Definition

48. Build
Image
s
Verify
Image
s
Deploy
Instance
s
Phase 1: Image Pipeline Phase 2

49. Deploy
Instance
s
Phase 2: Terraform

50. Build
Image
s
Verify
Image
s
Deploy
Instance
s
Phase 1: Image Pipeline Phase 2: Terraform

57. Push
Button

64. • Interact with infrastructure via API
• Make it easy to build & verify images
Lessons Learned

65. Auto Scaling Containers
A Look Ahead

66. Thanks!
productivity@box.com