The document details the process of securing Prometheus exporters using HashiCorp Vault, providing installation instructions for both Prometheus and Node Exporter. It covers secret management, configuring PKI, and automating certificate lifecycle management. The setup emphasizes security practices for scraping metrics with Vault-issued certificates and includes examples of configuration files and commands.

1. Securing Prometheus Exporters
using HashiCorp Vault
Bram Vogelaar
@attachmentgenie

2. $ whoami
• Used to be a Molecular Biologist
• Then became a Dev
• Now an Ops
• Currently Cloud Engineer @ The Factory

3. Metrics
l Open-Source tool to do metrics collection and storage
l Since 2012/2015, by SoundCloud
l CNCF graduated project
l HTTP pull model
l PromQL DSL
https://prometheus.io/

4. Prometheus Installation
wget https://github.com/prometheus/prometheus/releases/download/v2.33.3/prometheus-
2.33.3.linux-amd64.tar.gz
tar xvf prometheus-2.33.3.linux-amd64.tar.gz
cp prometheus-2.33.3.linux-amd64/prometheus /usr/local/bin/prometheus
cp /vagrant_data/common/prometheus.service /etc/systemd/system/prometheus.service
mkdir -p /etc/prometheus/
cp /vagrant_data/insecure/prometheus.yml /etc/prometheus/prometheus.yml
systemctl enable prometheus
systemctl restart prometheus

5. Prometheus Config
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: "node_exporter"
static_configs:
- targets: ["node.pki.vagrant:9100"]

6. Node Exporter Installation
wget https://github.com/prometheus/node_exporter/releases/download/v1.3.1/node_exporter-
1.3.1.linux-amd64.tar.gz
tar xvf node_exporter-1.3.1.linux-amd64.tar.gz
cp node_exporter-1.3.1.linux-amd64/node_exporter /usr/local/bin/node_exporter
cp /vagrant_data/common/node_exporter.service /etc/systemd/system/node_exporter.service
mkdir -p /etc/node_exporter/
cp /vagrant_data/tls/node_exporter.yml /etc/node_exporter/node_exporter.yml
systemctl enable node_exporter
systemctl restart node_exporter

7. Node Exporter Metrics

8. Prometheus UI

9. Access Security?!?

10. Secrets
l Open-Source tool to do secrets management
l Secure, store and tightly control access to tokens, passwords,
encryption keys for protecting secrets and other sensitive data using a UI, CLI, or HTTP API.
l Password rotation (e.g. SSH, MySQL)
l Certificate management via PKI secret engine
https://www.vaultproject.io/

11. Vault installation
yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
yum -y install vault
cp /vagrant_data/insecure/vault.hcl /etc/vault.d/vault.hcl
systemctl enable vault
systemctl restart vault
export VAULT_ADDR=http://localhost:8200
vault operator init -key-shares=1 -key-threshold=1
vault operator unseal ${VAULT_KEY}
* Wildly inappropriate setup, for demo purpose only

12. Vault Config
ui = true
disable_mlock = true
storage "file" {
path = "/opt/vault/data"
}
listener "tcp" {
address = "0.0.0.0:8200"
tls_disable = "true"
}
* Wildly insecure config, for demo purpose only

13. Bootstrap Certificate Authority
export VAULT_ADDR=http://localhost:8200
vault secrets enable pki
vault secrets tune -max-lease-ttl=87600h pki
vault write -field=certificate pki/root/generate/internal
common_name="pki.vagrant"
ttl=87600h > CA_cert.crt
vault write pki/config/urls
issuing_certificates="$VAULT_ADDR/v1/pki/ca"
crl_distribution_points="$VAULT_ADDR/v1/pki/crl"

14. Generate intermediate CA
vault secrets enable -path=pki_int pki
vault secrets tune -max-lease-ttl=43800h pki_int
vault write -format=json pki_int/intermediate/generate/internal
common_name="pki.vagrant Intermediate Authority"
| jq -r '.data.csr' > pki_intermediate.csr
vault write -format=json pki/root/sign-intermediate csr=@pki_intermediate.csr
format=pem_bundle ttl="43800h"
| jq -r '.data.certificate' > intermediate.cert.pem
vault write pki_int/intermediate/set-signed certificate=@intermediate.cert.pem

15. Add Certificate Strategy
vault write pki_int/roles/pki-dot-vagrant
allowed_domains="pki.vagrant"
allow_subdomains=true
max_ttl="720h”

16. Securing Node Exporter
$ vault write pki_int/issue/ pki.vagrant common_name=”node.pki.vagrant" ttl="24h"
tls_server_config:
cert_file: /etc/node_exporter/ssl/node.pki.vagrant.cert
key_file: /etc/node_exporter/ssl/node.pki.vagrant.key
client_auth_type: RequireAndVerifyClientCert
client_ca_file: /etc/node_exporter/ssl/ca.cert

17. Secured Node Exporter
curl -k https://tls.pki.vagrant:9100/metrics -v
–key tls.pki.vagrant.key
–cacert ca.cert
–cert tls.pki.vagrant.cert

18. Update Prometheus Connection
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: "tls"
scheme: https
static_configs: - targets: [”node.pki.vagrant:9100"]
tls_config:
cert_file: /etc/prometheus/ssl/prometheus.pki.vagrant.cert
key_file: /etc/prometheus/ssl/prometheus.pki.vagrant.key
ca_file: /etc/prometheus/ssl/ca.cert

19. Build In certificate expiration

20. Vault PKI Approle
vault auth enable approle
vault policy write pki_policy /vagrant_data/tls/pki_policy.hcl
path "pki_int*" { capabilities = ["create", "read", "update", "delete", "list", "sudo"]}
* Wildly insecure config, for demo purpose only

21. Vault Agent Config
vault {
address = "http://prometheus.pki.vagrant:8200"
}
auto_auth {
method "approle" {
config = {
role_id_file_path = "/vagrant_data/tmp/role_id"
secret_id_file_path = "/vagrant_data/tmp/secret_id"
remove_secret_id_file_after_reading = false
}
}

22. Vault Agent Config Cont.
template {
contents="{{ with secret "pki_int/issue/pki-dot-vagrant" "ttl=3m"
"common_name=prometheus.pki.vagrant" }}{{ .Data.certificate }}{{ end }}"
destination="/etc/prometheus/ssl/prometheus.pki.vagrant.cert"
command = "killall -HUP prometheus"
}
…
{{ .Data.issuing_ca }} => "/etc/prometheus/ssl/ca.cert”
{{.Data.private_key }} => "/etc/prometheus/ssl/prometheus.pki.vagrant.key"

23. Vault Agent Service
[Unit]
Description="Vault Agent to serve Tokens"
Wants=network-online.target
After=network-online.target
[Service]
Type=simple
WorkingDirectory=/etc/vault.d
ExecStart=/usr/bin/vault agent -config=/etc/vault.d/certs.hcl
[Install]
WantedBy=multi-user.target

24. Q.E.D?
• Deployed Prometheus and node_exporter
• Secured Metrics Scraping using Vault issued certificates
• Automated certificate Life Cycle Management using Vault Agent

25. Contact
bram@attachmentgenie.com
@attachmentgenie
https://www.slideshare.net/attachmentgenie
https://github.com/attachmentgenie/prom-vault-pki

26. The Floor is yours…
Questions ?