So you’ve scaled your app up to 1000 instances. Do they all share the same credentials for access to stateful resources? Then the attack surface for your stateful resources just got scaled up too. Automated secret management lets you focus on scaling up your app, not your risk of data compromise. This talk aims to introduce some important considerations in attack surface management at scale, and provide some patterns and tips on integrating secret management workflows into Continuous Deployment infrastructure.

1. Mary Racter
ScaleConf Cape Town
March 2018

2. About
● mary@praekelt.org
● medium.com/@racter
● Security Engineer at Praekelt.org
http://www.supermegacomics.com

3. Contents
● Secrets and Secret Management
● Secret Management at Scale
● Automating Secret Management in Continuous
Deployment

4. Secrets
and
Secret Management

5. A secret is some knowledge, or piece of data,
whose access should be restricted.
Knowledge of a secret is commonly used to validate
the holder’s identity (ie. Authentication).

6. Secrets grant access to resources, and are often
generated by the resources they grant access to.
Substrate: Short-hand for “resource that a secret
grants access to”.
Examples of secrets in computing:
● Passwords
● RSA Private Keys
● Encryption Keys
● API Tokens

7. A policy is applied after authentication with a secret.
If a secret maps to an identity on a resource, then a
policy maps to the privileges that identity has on
that resource, (ie. Authorisation).
Examples of policies in computing:
● File Permissions
● Access Control Lists (ACLs)
● Role-Based Access Control
(RBAC) Policies

8. Attack Surface Map

9. 1. Spatial Attack
Surface
2. Temporal Attack
Surface
3. Algorithmic
Attack Surface

10. Secret Management Goals
● Prevent unauthorised discovery of secrets
● Prevent misuse of discovered secrets / elevation of
privilege

11. Secret Management Tasks
● Creation
○ Randomly-generated keys of sufficient length
● Storage
○ Encrypted at rest
● Distribution
○ Encrypted transmission
○ Secrets are only held by their rightful consumer(s)
● Revocation / Renewal
○ Secrets can be revoked and renewed

12. General Principles in Secret Management
● DON’T roll your own crypto or
secret generation algorithms
● DON’T reinvent the wheel if
you don’t have to

13. General Principles in Secret Management
● DO have a central source of
truth for your secrets and who
has what kind of access to
them
● DO have an audit trail for
secrets requests

14. Secret Management Services
● eg. HashiCorp’s Vault, Square’s KeyWhiz
● Benefits
○ Modular, dedicated secret storage and creation mechanisms
○ Support for centralised ledger of secrets and audit logging of secret requests
○ Useful primitives and abstractions for secret management at scale (see later)
○ Unified, standard API for accessing and managing all secrets
○ Optimised for cloud-based architectures
● Trade-offs
○ Single point of leakage for a large number of credentials
○ Critical internal target for Denial of Service (DoS)
This talk assumes usage of a Secret Management Service.

15. A Secret Management Service maps an
identity to the resources they should
have access to, and the level of access
the identity has on that resource.
It also brokers secrets on the resource’s
behalf.

16. Secret Management Tasks
● Creation
○ Randomly-generated keys of sufficient length ✓
● Storage
○ Encrypted at rest ✓
● Distribution
○ Encrypted transmission ✓
○ Secrets are only held by their rightful consumer(s)
● Revocation / Renewal
○ Secrets can be revoked and renewed ✓
+1 Secret
Management Service

17. Secret Management
Work lows at Scale

18. Assumptions
● You’re containerising your applications
● You’re using a container orchestration platform
● You’re using a secret management service

19. Secret Management Tasks
● Creation
○ Randomly-generated keys of sufficient length ✓
● Storage
○ Encrypted at rest ✓
● Distribution
○ Encrypted Transmission ✓
○ Secrets are only held by their rightful consumer(s)
● Revocation / Renewal
○ Secrets can be revoked and renewed ✓

21. Distributing Secrets
• If we can securely get the initial secret granting the
container access to the Secret Management service, then
the container can securely fetch all subsequent secrets
from the service.
• But how do we fetch this first secret?

22. A Secure Introduction Service is coupled with the
cluster scheduler, and maintains a mapping of
container properties to access policies on the
Secret Management service.
Jeff Mitchell, Secure Introduction At Scale: Think Like A Vault Developer,
ContainerDays NYC 2016 Talk: https://www.youtube.com/watch?v=R-jJXm3QGLQ

30. Secret Management Tasks
● Creation
○ Randomly-generated keys of sufficient length ✓
● Storage
○ Encrypted at rest ✓
● Distribution
○ Encrypted Transmission ✓
○ Secrets are only held by their rightful consumer(s) ✓
● Revocation / Renewal
○ Secrets can be revoked and renewed ✓

31. Secrets Scaling Poorly

32. Secrets Scaling Poorly 1

33. Secrets Scaling Poorly 1
Secret re-use for different applications using the same resource
○ n-fold increase in spatial attack surface for n applications on which
secret is re-used
○ Increases temporal attack surface to the combined lifetime of all apps
○ Potentially violates Principle of Least Privilege

34. Secrets Scaling Poorly 2

35. Secret re-use for different resource instances
○ n-fold increase in spatial attack surface for n resource instances on
which secret is re-used
Secrets Scaling Poorly 2

36. Secrets Scaling Poorly 3

37. Secrets Scaling Poorly 3
Secret re-use for all instances of an application
○ Increases temporal attack surface to the combined lifetime of all app
instances

38. Secret Management Issues Happen At Scale
Because Manual Secret Management is Tedious
• Manually creating secrets at scale ⇒ Tedious
• Manually distributing secrets at scale ⇒ Tedious
• Manually revoking/renewing secrets at scale ⇒ Tedious

39. Ideal Scenario
● Unique secrets for each application on a resource
○ And each application instance on a resource
● Upholding the Principle of Least Privilege for every
application instance
○ On the resource
○ On the Secret Management service

41. Tools for Managing Secrets at Scale
● Secure Introduction
● Dynamic Secrets
● Leases

42. Secure Introduction
● Manual distribution of secrets assume the developer is trusted to use
those secrets for their app
● Secure Introduction creates a pipeline for that trust to percolate to
the application itself
● SI authorises an application to fetch its own secrets from the Secret
Management Service

43. Dynamic Secrets
• Supported by HashiCorp Vault
• Dynamic secrets are lazily generated on-demand from one “master”
secret
• Supports automated renewal and rotation of secrets
• Scales well for unique passwords in 1:∞ resource:client scenarios
• Requires instructions to Secret Management Service on how to
generate and revoke new credentials (eg. CREATE USER
'{{name}}'@'%' IDENTIFIED BY '{{password}}'; GRANT ALL ON db1 TO
'{{name}}'@'%'; )

44. Leases
• Supported by HashiCorp Vault
• Metadata on dynamic secrets that give them a
validity period
● The consumer must renew the lease by checking
in with Secret Management service
● Dynamic secrets that are no longer used are
revoked automatically with the lease expiry
mechanism

45. Secure Introduction
+
Dynamic Secrets
+
Leases
=
Better Secret Management Practices At Scale

46. Integrating Secret
Management at Scale
with
Continuous Deployment

47. Continuous Deployment Platforms
Infrastructure platform that ensures software can be reliably deployed at any
time.
Configure once ⇒ Deploy infinite times
Automation, instrumentation, set-up/tear-down, and QA phases are incorporated
into continuous development pipelines.
Reducing lead time on deployment of new code by automating the steps leading
up to deployment.

48. Unfortunately, secret management best practices
often don’t integrate seamlessly with Continuous
Deployment setups.
Configure once ⇒ Static secret for
all app instances?
Configure once ⇒ One extremely
permissive policy for the static
secret?

49. Pattern 1: Assume the resource exists and the secret is in the target deployment environment.
Existing Secret Management Patterns in CD

50. Existing Secret Management Patterns in CD
Pattern 2: Bake secrets into application images and/or binaries

51. Existing Secret Management Patterns in CD
Pattern 3: Developer configures secret store with existing secret, CD pipeline injects during
deployment.

52. Existing Secret Management Patterns in CD
Pattern 4: Existing secret is packaged and injected during deployment

53. In all 4...
● Resources are manually created outside of the
pipeline
● Secrets are manually created outside of the
pipeline
● Low manual action in assigning secrets, but other
secret management actions (creation, distribution,
revocation/renewal) is manual
● Baking secrets into deployment images increases
attack surface of secrets to image and any
repositories the image may reside in

54. Reminder: Manual administration of
secret management does not scale
well proportionally to the number of
consumers, nor does it scale well to
the number of secrets.

55. In-band vs. Out-of-band Configuration
● In-band configuration
○ Happens in sequence within deployment pipeline
● Out-of-band configuration
○ Happens outside of sequence of deployment pipeline
● Out-of-band configuration is most commonly done manually
○ If it’s automated, there is more overhead in making it compatible with
the CD workflow
Reducing manual action == Reducing out-of-band
configurations

56. Ideal Scenario
● Dynamic Secrets and Secure Introduction is incorporated into the app
deployment workflow and abstracted away from the developer
● (Almost) all configuration is carried out in Continuous Deployment pipeline

57. Manual Configuration
● Deployment Target
● Deployment Parameters
● Network
● Container Settings
● Environment Variables
● Required Resources and
Access Modes

58. Resource Creation
● Resource creation requires additional action to integrate newly-created
resource
● Ownership of Secret Management service registration and resource lifecycle
management should reside with the resource management module

59. Configuring Application Access to SM service
● The Secure Introduction service is the module that maps an application to its permissions on the
Secret Management service
● Application access configured by updating the SI service with the required mappings for the
application to fetch its required credentials. These are derived from the deployment parameters
● Ownership with CD platform or resource manager

60. Remaining Out-of-Band Configuration Per
Deploy
● Storing the master secrets for resource creation on the
Secret Management service and configuring the resource
management module to read from the Secret
Management service
○ Once per resource host/fleet

61. Possible Challenges
● Updating app mappings on Secure Introduction Service
before knowing what the app name will be
○ Generate app name with a known prefix and use a wildcard mapping on the SI service
● Mapping access modes on the resource to its
corresponding policy on the Secret Management Service
○ Use clearly-established naming conventions for Secret Management service policies
● Managing secret life cycles
○ Wrap the application in a supervising process that can fetch, renew, and revoke secrets

62. Conclusion
● Newer Secret Management services are built to be used at scale -
use them
● Secret Management is most useful when integrated into Continuous
Deployment
○ Automate fully by reducing out-of-band configurations for secret
management
○ Unify Resource and Secret Management APIs with a broker architecture
● Automating Secret Management frees up your team to do great
things

63. Thank You