Slides from a talk given at the Toronto HashiCorp User Group monthly meetup held on March 14, 2018 at Street Contxt. Abstract: Street Contxt is a global knowledge exchange for institutional finance, with their SaaS platform delivering smart, actionable insights customers all over the globe. They have been working with Packer, Consul, and Terraform for a year now in their rapidly changing production environment. In this "stories from the road" style talk, we'll take a look at the process of adopting these tools and how they've managed to safely begin to move away from a central monolith to a service-based architecture, while bringing about repeatability, testability, and audibility across their infrastructure via Infrastructure as Code and Service Discovery.

1. A Year in Production with the Hashistack*
Well… most of it anyway

2. Introductions
@redmind
Jason Harley
jharley@streetcontxt.com
@redmind
https://www.linkedin.com/in/jharley/
https://github.com/jharley

3. Introductions:
Street Contxt

4. Agenda
• Initial state
• Packer
• Containerization and Terraform
• Discovery and Visibility and Consul
• Iterations and Improvements
• Reflection
@redmind

5. Street Contxt: Jan ‘17
• AWS as VMWare
 Five separate AWS accounts, running manually created resources
 A brief exploration into CloudFormation
• Ubuntu 14.04 based environment
 Some upgraded from 12.04 by hand
• Lovingly, hand-tended “pets”
• Distributed Monolith
 Play<>Wildfly<>PostgreSQL
 Apache Solr
 AWS ElasticMapReduce (EMR)
 A few containers running back-office stuff on “service” boxes
• Desire to move to Immutable Infrastructure
@redmind

6. Packer and Ansible
• Ansible was already in use at Street Contxt
 Reasonably complicated, single-purpose (use?) playbooks using a static inventory to
manage machines
 Developing a practice of writing reusable, immutable roles was needed
• Started carving out a “base” Ansible role with help from test-kitchen and
InSpec
@redmind

7. Mission: critical path containers?!
• Data Science team was developing two Tensorflow-based services, which
were nicely packaged as Docker containers and exposed an HTTP interface
• We needed a way to deploy, manage and route traffic to these service
 … on the cheap
• ECS looked like a decent candidate for a Proof of Concept (PoC)
 Rolling upgrades weren’t going to be an issue, really
 The price was right
 You only pay for EC2, *LB, and network traffic
@redmind

8. Introducing Terraform (March ’17)
• Starting from scratch current
 We had just closed-out a decent year.. we’re at our highest usage as a company
• Terraform 0.8.8 had just been released
• You don’t need to import everything(!)
 This seems to be the biggest initial failing point for Terraform adoption
 You might not need to import anything…
• For each of our environments, we setup static variables for the critical
ARNs/IDs of VPCs, subnets, Availability Zones, and Route53 zones
@redmind

9. @redmind
$ cat example.tfvars
env = ”exp”
vpc_id = "vpc-82e2471f”
availability_zones = ["us-east-1c", "us-east-1f"]
subnet_ids = {
public = ["subnet-c9b969a7", "subnet-1acb5a77"]
private = ["subnet-96b969f8", "subnet-77a5bca1"]
}
internal_domain_name = “exp.scx-internal.net”
internal_domain_id = “Y2KDOD09BQCVUN”
external_domain_name = "streetcontxt.com”
external_domain_id = “A6PGBR0HLIEJNB”
$ cat initialize.tf
provider "aws" {
region = “us-east-1”
}
variable "env" {
type = “string”
}
variable "vpc_id" {
type = “string”
}
variable "subnet_ids" {
type = "map”
default = {
public = []
private = []
}
}
[…]

10. Introducing Terraform (March ’17)
• We decided on an environment-based directory structure
 Heavily influenced by a Charity Majors blog post about separating your Terraform
into per-environment state
 Workspaces (nee Environments) didn’t yet exist
 Released in Terraform 0.9.0
• We decided to store and share state in versioned S3 buckets, with an
encryption policy configured
• To save us from ourselves… we wrote a Makefile
 Commands were starting to look potentially complicated
 Had to have the right environment variable set
 Had to pass the correct vars file
 Had to make sure you were using the right version of Terraform
 We make use of tfenv
 Notably, there is no make destroy
@redmind

11. Introducing Terraform (March ’17)
@redmind
$ make init
[…]
$ make plan
[…]
$ make apply

12. Introducing Terraform (March ’17)
@redmind
• Setup environments
• Built a module to create an ECS cluster with
all related policies and resources
• AMI created with Packer using an
Ansible role and built off the base image
• Didn’t terraform import anything…

13. Terraform and CloudFormation(?!)
• We hand bombed the pair of
Tensorflow services into existence
• Quickly realized we had a “fleet
management” issue
• Terraform doesn’t do rolling updates
 CloudFormation does
• Discovered an example from
AWSLabs using Lambda, SNS and
Cloudformation
• Simple enough to refactor the module
to setup a CFN stack to manage the
AutoscalingGroup
@redmind
Source: https://github.com/awslabs/ecs-cid-sample

14. Discovery and Visibility and Consul
• Visibility into the health and location of services
 Especially these new Tensorflow services in ECS
 talk of wanting an easy KV-store for a few projects
• Wrote a new Ansible role with the help of test-kitchen and InSpec
• Built an AMI with Packer and Ansible
• Wrote a Terraform consul_cluster module
 three node autoscaling group
 solved Consul bootstrapping via userdata and a Route53 record
 called it a “soft lock”
• Ansible role made use of EC2 tag-based discovery, configured dnsmasq to
redirect “*.consul” lookups to the Consul agent
• Successfully launched a cluster with a make apply(!)
 … and was quickly reminded we had zero Consul clients
@redmind

15.  Our Ansible role was written to
support clients and servers
 Quick and dirty script to add
the Consul client security group
to EC2 instances
 Rolled out the Consul agents
with ansible-playbook and
watched everyone report in
@redmind
Discovery and Visibility and Consul

16. Discovery and Visibility and Consul
• Now… we needed Dockerized-services to register themselves as Consul
Services
• Came across a great article from ZenDesk Engineering on using
Registrator from Gliderlabs
• Registrator automatically registers and deregisters services for any Docker
container by inspecting containers as they come online.
 SERVICE_NAME
 SERVICE_CHECK_HTTPS
 SERVICE_CHECK_INTERVAL
• Back into test-kitchen with our ECS role
 Added a registrator systemd unit that started with the Docker unit
@redmind

17. Discovery and Visibility and Consul
• Packer brought us a new AMI and with a make plan and make apply cycle
our registrator-enabled container hosts were in the wild
• Quickly added some ”SERVICE_” environment variables to our ECS Task
definitions, and updated the ECS Service to see registration of services into
Consul
@redmind

18. Consul in the critical path: ElasticSearch
• Up until now, Consul was telling us things... and while the data was useful
the conversation was fairly one sided
• We had a single instance Apache Solr service that needed to become more
critical.. and we decided that SolrCloud wasn't for us
• With our past success with Consul and ECS we dove back into test-kitchen
with a new Ansible role
• With a role we trusted, we could build an AMI, and then a Terraform module
• Instead of using a load balancer (which, many folks seem to use with ES)
Consul service discovery via DNS became the norm
@redmind

19. @redmind
• ElasticSearch clients began accessing
ElasticSearch as
“elasticsearch.service.consul”
• May 23, 2017 Consul became part of
the critical customer path for all
searches

20. Six months in…
@redmind
Ansible
Role
Packer
Image
Terraform
Module

21. Six months in…
• In that time, more containerized services have been written and are ready to
head out the door
• Two new modules to round out our growing library
 ecs_task
 ecs_service
• Quickly went from the initial 2 Tensorflow services
• 19 services and 4 batch-style tasks today
• More interestingly: these new services are being written using AWS services
• S3, SQS, Kinesis, KMS, Lambda
• We’d shaken off the inertia of the distributed monolith!
• We converted our Ansible inventory to a dynamic inventory driven by Terraform-
managed EC2 Tags in late July
@redmind

22. The latter half
• Started to bring parts of the legacy systems into Terraform and Consul
 Backend admin tool containerized and migrated to ECS in late June
 Elastic MapReduce (EMR) taken under Terraform control in August
 Our Wildfly cluster was turned into a Terraform module in November
 We actually imported things here 
 Our frontend UI moved to ECS in early January ‘18
@redmind

23. Reflection: Imports?
• We still don’t have everything managed by Terraform
• Our “legacy resource” variables for VPC and subnets are still variables
 No great urgency or business need to deal with importing them
 Data Providers make this a non-issue
 We’ll likely go “full Terraform” by the end of 2018
@redmind

24. Reflection: Outages? Uh-ohs?
• We’ve been really fortunate
 diligent about running plans and paying attention to the output
• We did lose search one day…
 Consul agents didn’t startup on the ElasticSearch instances after a maintenance
script ran
 Not Consul’s fault: operator error
@redmind

25. Reflection: Do overs?
• Wish I’d known about Molecule sooner
 We’ve yet to move a bunch of test-kitchen projects to it
• Our ecs_task module’s definition of the container’s environment is brittle
 We have plans in the works to move to envconsul as an entrypoint to address this
issue
• We love the our environment model (it makes us feel safe :D)... but a single
statefile per-environment is starting to get slow
 Plan to breakup this state by the fall
• We wish we already had dynamic secrets...
 currently doing some hacky magic with encrypted S3 objects and KMS to deal with
getting secrets into containers
@redmind

26. Questions?
@redmind
Jason Harley
jharley@streetcontxt.com
@redmind
https://www.linkedin.com/in/jharley/
https://github.com/jharley

27. Addendum: external links
• Building Immutable Machine Images with Packer and Ansible
 https://www.slideshare.net/JasonHarley3/building-immutable-machine-images-with-
packer-and-ansible/
• charity.wtf: TERRAFORM, VPC, AND WHY YOU WANT A TFSTATE FILE
PER ENV
 https://charity.wtf/2016/03/30/terraform-vpc-and-why-you-want-a-tfstate-file-per-
env/
• tfenv: Terraform version manager inspired by rbenv
 https://github.com/kamatama41/tfenv
• AWS Samples: ECS Container draining
 https://github.com/awslabs/ecs-cid-sample
@redmind