Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The Messy Underlay Dilemma - automating PKI at Defragcon

534 views

Published on

Deep discussion of why IT infrastructure is hard to automate with very specific examples building and rotating an internal PKI infrastructure.

Published in: Internet
  • Login to see the comments

The Messy Underlay Dilemma - automating PKI at Defragcon

  1. 1. The Messy Underlay Dilemma Lessons Learned Securing K8s Rob Hirschfeld, @zehicle
  2. 2. Hang on tight! We’re going deep. To automated live encryption key rotation
  3. 3. Is Operating Kubernetes HARD? DF No. But underlay is hard. From http://www.slideshare.net/rhirschfeld/ Containers, Orchestration and Security, Oh My!
  4. 4. Underlay vs Overlay Platform Overlay Infrastructure Underlay Ready State 0 Ready Prerequisites 1 Prereq Cluster API & Control Services 2 Control Worker Nodes 3 Nodes Cluster Add-ons 4 Add-Ons User Applications 5 Apps Application Overlay Underlay = Crust Overlay = Filling App = Topping Underlay components are the operational integrations and prerequisites that go into building a system to before we can install a platform.
  5. 5. Why is Underlay Hard? It’s Sequential, Multi-node & Environment Specific Unlike development environments, production cannot overlook integration points HA/LB PKI DNS SDN IPAM BMC RAID BIOS Highly Available & Load Balanced Public Key Infrastructure Domain Name Servers Software Defined Networks IP Address Management Out of Band Management Drive Arrays Firmware Even in cloud only deployments, these critical components for production platforms and applications require a level of different systems thinking. Strong underlay builds an IT foundation.
  6. 6. Platform & Infrastructure Underlay Ready State 0 Ready Prerequisites 1 Prereq Cluster API & Control Services 2 Control Worker Nodes 3 Nodes Cluster Add-ons 4 Add-Ons User Applications 5 Apps Application Overlay DevOps Is Struggling Developers don’t want do this infrastructure specific stuff Companies are turning to containers and application platforms (like Kubernetes) to abstract the messy underlay. While platforms hide complexity from developers, the issues still need to be addressed by Ops.
  7. 7. What makes underlay hard? Let’s look at Internal PKI
  8. 8. Protection via Tunnel Level Security (TLS) This is pretty complex stuff…. At a very basic level: 1. Send public key to client 2. Client encrypts token with public key 3. Client returns encrypted package 4. Server decrypts token with private key 5. Server uses token to encrypt tunnel Server Private Key Client Public 1 2 Token 4 Token 3 5 T L S
  9. 9. Trusted 3rd Party Trust Anchor Trust Chain of Trust in Public Key Infrastructure (PKI) PKI is doing something amazing! It establishes asynchronous trust By relying on strong encryption And Trust Anchors. Server Private Key Cert Auth Client Public T L S Digital Signature Root
  10. 10. Half of all Internet Traffic is encrypted! HTTPS > 50% That’s great for public traffic where trust is anchored / embedded into clients What about the internal traffic? We want a “narrow trust domain” so there’s no embedded trust mechanism and we maintain full control. We also want to protect both sides. Public End Points East-West Traffic North-SouthTraffic Back End Services End User Clients
  11. 11. Server Client TrustTrust Shared Root in Public Key Infrastructure (PKI) Self-signing keys is not considered secure. Internal PKI uses a shared root strategy The private root of the CA must not be known to in the trust relationship. Members of the Trust Domain rely on the CA to verify membership and identity. External Trust Anchors are not desirable because we want an exclusive Trust Domain. Private Key Public T L S D. SigRoot Private Key D. Sig Public Root Site CA
  12. 12. Master Node 1 Worker Nodes Worker Nodes Master Node 2+ “Narrow” Trust Domain Limits Shared Roots etcd Kublet Proxy API Server Controller Controller etcd API Server Scheduler Scheduler User! Illustration from Slideshare Rob Hirschfeld
  13. 13. Master Node 1 Root Worker Nodes Worker Nodes Master Node 2+ RootRoot Root Shared Roots Create Trust Zones etcd Kublet Proxy API Server Controller Controller etcd API Server Scheduler Scheduler User!
  14. 14. Services Files 1 2 Config App How do we automate this? Mix of Service and Configuration 1. Run a Root CA Service 2. Create a unique Root 3. Generate Key Pair Certificate for Server 4. Generate Digital Signature with Public Key for Client(s) 5. Configure Server with Certificate 6. Configure Client with Signature Private Key Digital Sig Public Site CA Root Server Certificate Client Certificate 3 4 5 6 Public
  15. 15. Old Root New Root Root Rotation protects Trust Zone - Do it daily?! By design, root rotation breaks cluster communications! Like an in-place upgrade, rotation can break APIs. We need to change the keys without breaking communication between components. Previous Cluster Member Cluster Trust Zone Old New New New Old New
  16. 16. Step 1: Root Rotation without Downtime Relies on Client to using support multiple digital signatures for the server. Create a new root and propagate new certificates in the cluster. Update the client configurations to use either signature. Old Root New Root Previous Cluster Member Cluster Trust Zone Old Old New Old Old Old New New
  17. 17. Step 2: Root Rotation without Downtime Ensure all the desired clients have new signature. Replace the server private key with the new value. Old keys will no longer work. In a daily rotation, leave both old and new signatures in place. Old Root New Root Previous Cluster Member Cluster Trust Zone Old Old New Old Old New New New
  18. 18. Happily, this is a repeatable pattern for underlay automation.
  19. 19. Questions? Rob Hirschfeld @zehicle RackN.com Rebar.Digital Private Key Site CA Root Public

×