My previous experience was working for a global training provider where I responsible for developing lab infrastructure for training courses. A lot of my early IaaS experience comes from the operations and automation side of deploying vApps in a vCloud Director environment.
OpenStack was one of many open source alternatives to vCloud and I was involved in several different Proof of Concept projects to utilize the stack where possible to save licensing costs and prevent lock-in.
My main design philosophy. Having worked on small, highly diverse operations teams over the years, this philosophy always produces products with fewer bugs, which is easier to diagnose when there is an issue and easier to understand documentation and add features. Only caveat, don’t paint yourself into a corner trying to make things too simple.
It’s not as bad as this picture makes it seem. OpenStack is actually just a microservice based virtualization manager built for scale. You only have to deploy the services you need. This still presents a challenge for small teams to operate efficiently at scale. Hence the need for service providers like OnRamp.
OnRamp is a managed service provider headquartered out of Austin, Texas. We have datacenters in Austin as well as on the East Coast in Raleigh, North Carolina. All of our datacenters are world class facilities with fully redundant network and power paths.
We offer managed hosting and colocation services with a focus on security and compliance. We also have acquired HIPPA and HiTRUST certification for our security practices.
Last year we released our Virtual Private Cloud product which is built on OpenStack.
In addition to using the OpenStack API, we provide our customers pre-hardened Linux and Windows images, and provide usage-based billing to allow for burstable cloud workloads.
So, why should you encrypt? * Protect Against Leaks - PHI - PCI - IP
Compliance Requirement: - HIPPA requires PHI data to be encrypted at rest and in-transit - Credit card data also falls under strict guidelines for the handling of credit card data. - Both: Shared environments must be kept private
Perhaps a better question is: What is a key manager? Key managers are basically the equivalent of password managers you might use with your web browser. The only difference is instead of user passwords, they store service passwords or keys.
Barbican is an Open Source secret storage service written specifically for OpenStack. It was designed by the developers at RackSpace and was originally introduced with the Icehouse release.
Thanks to some fantastic work done by Johns Hopkins Applied Physics Laboratory and others…
Encrypt your volumes with barbican open stack 2018
Encrypt Your Volumes with Barbican
• About OnRamp
• Why Encrypt?
• Key Managers
• How it Works
• Our Encryption Story
• Gotchas & Limitations
Meet the Speaker
• Previously Built and Managed Lab
Environments for VMware
Certification Training (VCP5/6)
• Experience with OpenStack:
- Started with Icehouse on Ubuntu
- Juno on Canonical MaaS + Juju
- Liberty on CentOS / RDO
- Newton on Platform9
• OpenStack Engineer for OnRamp
• Deployed Barbican and support for
Encrypted Volumes for our OpenStack
Based Public Cloud
Who Is OnRamp?
OnRamp is a HITRUST-certified data center services company specializing in
high security and compliant hybrid hosting.
OnRamp’s Virtual Private Cloud
■ Built on OpenStack®
■ Control costs with capped maximum resource usage
■ Open-source APIs enable simple migrations and eliminate vendor lock-in
■ Hardened Linux and Windows Images
■ Retain full control of your private networks, virtual machines, and storage
Offers the ease-of-use of a public cloud, with the security of a private cloud.
• Protect Data against Leaks
- Personal Health Information (PHI)
- Credit Card Payment Data (PCI)
- Intellectual Property
• Compliance Requirement
- HIPAA requires encryption for data at rest
and in-transit for PHI
- PCI also requires encryption for data at rest
and in-transit for cardholder data
- In shared hosting environments, each Tenant
must only have access to their own stuff
Per-Tenant or Per-Volume encryption keys
Why Use a Key Manager?
• Security Best Practice
- You don’t leave keys in your car
- You shouldn’t leave your keys next to your encrypted data
• Compliance Requirement
- PCI-DSS Requirements:
Store keys in the fewest possible locations
Store secret and private keys used to encrypt/decrypt cardholder data
using a key encryption key at least as strong as the encryption key
- HIPAA- HITRUST Requirements:
Keys shall be stored separately from encrypted data
Key manager systems should be physically protected by fewest
number of custodians necessary
What is Barbican?
• ReST API for Secrets Management
• Pluggable Backends
- Simple Crypto
- PKCS#11 and KMIP (HSM)
• Integration with Nova, Cinder, and
Swift, Neutron, Heat, and many
other OS projects
• Integration with KeyStone for Auth
• Built to Scale
Does Not Provide:
• Graphical User Interface
• Key Splitting for Secure
Import/Export Plain Text Keys using
multiple Key Custodians
• Generation of X.509 Certificates
• Volume Encryption
What about Volume Encryption?
• LUKS - Linux Unified Key Setup
- Allows for multiple user keys or passwords per volume
Master Key always stays the same
- Supports CPU Hardware Acceleration (it’s fast!)
- Uses CryptSetup and DM-CRYPT
Decrypts full volume to a Local Block Device
Protects iSCSI attached volumes
Can also protect ephemeral storage if using LVM
• Queens = QEMU Native LUKS support
- QEMU 2.6 and LibVirt 2.2 introduce native LUKS support
Transparent Encryption in Nova and Cinder
Thanks to some fantastic work done by Johns Hopkins Applied
Physics Laboratory and others…
• Nova and Cinder Integration exists with Barbican
• Volume Decryption Happens on the Hypervisor instead of
within the Guest OS
- No Agent Required
- Works with Any Operating System
- Works with Bootable Volumes
- Protects Data at Rest and In-Transit to your hypervisor
- Every volume is protected by it’s own unique key
How it Works: Creating an Encrypted Volume
1. User Gets a token from Keystone
2. User Asks Cinder to create volume
3. Cinder verifies the user token from
4. Cinder then asks Barbican for a key
5. Barbican checks the Cinder Token
6. Barbican creates the secret and
returns secret HREF to Cinder
7. Cinder stores the returned secret
HREF into the volume metadata
How it Works: Mounting an Encrypted Volume
Nova Keystone Cinder Barbican
1. Nova receives request to mount
volume with token
2. Nova Validates User Token
3. Nova Fetches the Secret HREF
4. Cinder Validates Nova Token
5. Cinder returns Secret HREF to
6. Nova Fetches Secret from
7. Barbican Validates Nova Token
8. Barbican Returns Secret
9. Nova Mounts Volume using
OnRamp Encryption Story
• Followed Documentation
- Wait, these docs are terrible!
Thankfully, Barbican Devs on IRC are very helpful!
- Added Endpoints to KeyStone
- Added Barbican Service User
- Added Creator Role
• Installed the Barbican-API:
- Configured Barbican’s database server
- Configured Barbican
- Barbican CLI Access Works!
OnRamp Encryption Story
• Cinder Integration Issues:
- Blank Encrypted Volumes were created successfully
- But they could not be created from an image
• Nova Integration Issues:
- Nova was unable to attach any encrypted volumes
- It wasn’t even trying to talk to Barbican
Issue 1: Key orders not getting to Barbican
SSH VPN barbican
Secret HREF: null
for key 0000…
href = all zeros!
no key 0000…
Issue 1 Fix: Key orders not getting to Barbican
SSH VPN barbican
Secret HREF: valid
Cloud-Hosted On-Premise On-Premise
Secret HREFSecret HREF
valid href request secretreturn secret
Issue 2: Nova Not Talking to Barbican
• Nova error was mentioning a Fixed Key not being defined
• ConfKeyManager (Nova Default for Volume Encryption)
• Single fixed key for all volumes
• Used for testing to substitute in for a real key manager,
such as Barbican! Not for production.
• Setting the api_class in nova.conf would always use
ConfKeyManager and ignore the setting to use barbican
• Submitted Bug:
• Fixed in Pike or newer!
• Manual Fix for older releases is to comment out lines 27-31
Some Gotchas using Encrypted Volumes
• Live Migration does not work and is dangerous
- Volume mounts without decryption on new host which will cause
corruption if this is an active file system
- Fixed in Queens!
Previous releases use symlinks on hypervisor
New Method uses QEMU Native LUKS support
• Barbican Doesn’t Start after Reboot (CentOS Specific)
- RDO Packaging Issue
- Create /etc/tmpfiles.d/barbican.conf:
d /var/run/barbican 0755 barbican barbican –
- Bug report submitted…
Some Limitations using Encrypted Volumes
• No mechanism to rotate volume encryption keys
- Manual process of creating a new volume and copying over the
- LUKs supports multiple user keys, so the capability is there
• No UI for key management in horizon
- for securely exporting and importing split-keys
- for managing key ACL’s
- for managing key expiration and revocation
Tips for Running in Production
• Make sure your key manager and database are secured in a
locked cabinet with limited physical access
• Use a private barbican instance not accessible to tenants
• Automated database backups
• Use a highly available database cluster such as Galera
• Use multiple barbican-api nodes behind a load balancer
• Use SSL to protect key requests in-transit to hypervisors