This document summarizes a presentation given by the New York Times' Delivery Engineering Team about how they adopted and manage Hashicorp Vault at the New York Times. Some key points: - They were having problems managing secrets before and Vault helped simplify this. - They leverage Vault's authentication methods, ACLs, auditing, and "break glass" capabilities. - To better isolate access, they created namespaces in Vault for teams and integrated this with Terraform and a tool called Goblin. - They also integrated Vault with their Google Kubernetes Engine cluster by having init containers fetch signed JWTs and write secrets to shared memory volumes. - This approach removes secrets

1. Vault @ NYT
NYT’s Delivery Engineering Team
Hashicorp User Group - April ‘18
New York City
THE NEW YORK TIMES
Managing and integrating Vault at the
New York Times

2. 1. Who are we?
2. Problems in Secretsville
3. Adopting and Managing Vault
4. Namespaces
5. Integration with other Systems

3. Scott
Who are we?
Shawn Prashanth

4. 1. Who are we?
2. Problems in Secretsville
3. Adopting and Managing Vault
4. Namespaces
5. Integration with other Systems

5. Our secrets management was
more difficult than it had to be.

6. 1. Who are we?
2. Problems in Secretsville
3. Adopting and Managing Vault
4. Namespaces
5. Integration with other Systems

7. Operator Keysharing
The key sharing capabilities of Vault
ensure that no single operator can
perform any “root” operations and
that when these operations must
occur, all (or most) operators are
aware.
Variety of Auth Methods
Vault supports a wide array of
authentication methods which makes
developer and automated
authentication easy.
Granular ACLs
Vault’s granular policy system
allows a single Vault cluster to be
used comfortably by multiple teams
in the organization. It also ensures
proper isolation of secrets.
Audited
The Vault audit log is invaluable for
incident response and figuring out
exactly what happened and when.
Every action must be logged so
there is always a trail to follow.
Break Glass
Benefits of Adopting Vault
And more...
- Simple secret sharing
- Leases / Revocation
- Dynamic secrets
- HTTP API
- Open source
- Public testing API
- Constantly improving
In case of emergency Vault can be
sealed and secret access can be
temporarily shut down quickly and
easily.

8. Image from: https://registry.terraform.io/modules/hashicorp/vault/google/0.0.3

9. 1. Who are we?
2. Problems in Secretsville
3. Adopting and Managing Vault
4. Namespaces
5. Integration with other Systems

10. Vault is great out of the box but
to get the isolation we wanted
we started creating
namespaces.

11. To manage namespaces we use
terraform and a home-grown
tool called goblin.

13. variable "name" {
description = "The name of the team on Github."
}
variable "org" {
description = "The name of the org the team belongs to on Github."
}
variable "ensured_policies" {
type = "list"
default = []
description = "The list of ensured policies for the team."
}
Team Namespace Module: Variables

14. resource "vault_mount" “secret” {
path = "${var.org}/teams/${var.name}/secret"
type = "generic"
}
resource "vault_mount" “transit” {
path = "${var.org}/teams/${var.name}/transit"
type = "transit"
}
Team Namespace Module: Mounts

15. # The member policy is given to all members of the team
resource "vault_policy" “member” {
name = "${var.org}/teams/${var.name}/member"
policy = <<EOT
path “{var.org}/teams/${var.name}/secret/*” {
capabilities = [“create”, “delete”, “list”, “read”, “update”]
}
… # more paths below
EOT
}
Team Namespace Module: Policies

16. # Ensure certain policies are always mapped onto the team
resource "null_resource" “ensure-policies” {
triggers {
policies = "${join(“,”, var.ensured_policies)}"
}
provisioner "local_exec" {
command = “goblin ensure -team=${var.name}
-policies=${join(“,”, var.ensured_policies)}”
}
}
Team Namespace Module: Ensuring Policies

17. module "delivery-engineering" {
source = "../modules/base_team"
org = "nytm"
name = "delivery-engineering"
ensured_policies = [
"nytm/teams/delivery-engineering/member",
"operator",
]
}
Using the Team Namespace Module

18. module "my-repo" {
source = "../modules/base_repo"
org = "nytm"
name = "my-repo"
}
Using the Repository Namespace Module

19. Syncing Repository Policies
What repositories
are active?

20. Syncing Repository Policies
my-repo,a-repo,
another-repo

21. Syncing Repository Policies
Which teams have
access to these
repositories?

22. Syncing Repository Policies
A-Team has read
access to my-repo
and admin access
to that-repo.

23. Syncing Repository Policies
Does A-Team have
any ensured
policies?

24. Syncing Repository Policies
a-team-member

25. Syncing Repository Policies
Map the policies
a-team-member,
my-repo-read, and
that-repo-adminto
the A-team.

26. Syncing Repository Policies
Done!

27. Uses extensible
templates.
Creating new namespaces
and adding to existing
namespaces are both
easy and fast.
Main Benefits
Provides some
self-service policy
management.
Teams really love being
able to do things
themselves.
Enforces good Github
practices.
Part of onboarding is
reviewing Github teams,
membership, and
permissions.

28. 1. Who are we?
2. Problems in Secretsville
3. Adopting and Managing Vault
4. Namespaces
5. Integration with other Systems

29. Connecting GKE to Vault
● Enable GCE auth backend
● Create init container in deployment
○ Mount shared memory volume
○ Get signed JWT
○ Login to Vault
○ Write secrets to shared memory volume

30. Connecting GKE to Vault
initContainers:
- name: {{.app}}-init
image: {{.int_image}}
env:
- name: VAULT_ROLE
value: "{{.vault_role}}"
- name: GOOGLE_SERVICE_ACCOUNT
value: "{{.google_service_account}}"
- name: VAULT_ADDR
value: "{{.vault_addr}}"
- name: VAULT_PATH
value: "{{.vault_path}}"
volumeMounts:
- mountPath: /secrets
name: secrets
VAULT_ROLE
(required )
The Role in vault to authenticate with
GOOGLE_SERVICE_
ACCOUNT (required)
The google service account that will be used to authenticate
with vault. Typically this will be default
VAULT_ADDR
(required)
The URL of the vault server to connect to.
VAULT_PATH This is a vault path (ie nytm/goblin/secret/something). If
this variable is specified the path will be recursed and all
secrets will be written to memory.
VAULT_SECRETS This is a comma delimited list of vault secrets. Each secret
will have every key written to memory
TOKEN_ONLY This is a boolean variable. If true a short live vault token
will be written to /secrets/vaultToken.

31. Connecting GKE to Vault
● Secrets are no longer in our CI/CD pipeline
● Secrets only stored in memory
○ Not unencrypted in GKE
○ Not leaking in environment variables

32. We’re hiring
nyti.ms/technology
@NYTDevs | developers.nytimes.com

33. Stay updated
open.blogs.nytimes.com
@NYTDevs | developers.nytimes.com