The document discusses policy-based resource placement across hybrid-cloud federations of Kubernetes clusters, emphasizing the need for context-aware and programmable placement strategies. It describes user stories highlighting the importance of compliance with regulations, resource management during maintenance, and the use of a policy engine to enhance control. The conclusion stresses that Kubernetes federation supports diverse deployments while addressing complex business requirements through policy solutions.

1. Policy-based Resource Placement
...across Hybrid-Cloud Federations of Kubernetes Clusters
Irfan UR Rehman (Huawei)
@irfanurrehman
Torin Sandall (Styra)
@sometorin

2. Federation: Overview
federation-apiserver
federation-controller
app app
Federation Control Plane
Federated
Deployment
etcd
Federated
Clusters

3. Federation: Placement
federation-apiserver
federation-controller
kind: ReplicaSet
metadata:
annotations:
replica-set-preferences: |
{“clusters”: {
“us-west-1”: {“weight”:1},
“asia-se-2”: {“weight”:1}
}}
● Placement can be controlled
per-resource via annotations
● 2 annotations supported:
federation.kubernetes.io/replica-set-preferences
federation.alpha.kubernetes.io/cluster-selector
● federation-controller evaluates
annotations to produce final
placement
ReplicaSet
created
1
2
3
4

4. User Stories
● As a federation user, I want my workload to be placed in clusters
conforming to particular types/regulations/jurisdiction.
○ Lot of properties - annotations, labels, regions, zones,..., that are available or can be applied
on clusters.
● As a federation admin, I want placement to account for context such as
maintenance windows.
○ Clusters often need to be brought down for maintenance in DCs.
○ The window might be small but, the workload re-distribution is largely manual.
● As a federation admin I want to enforce field-level access control.
○ K8s today has RBAC, which is object level access control.
○ A policy engine can fill the gap.

5. Policy-based Placement
federation-apiserver
federation-controller
● Resource placement is a “policy-rich”
problem space
○ Legal regulation, cost, technical constraints, internal
conventions, etc.
● Decouple dev policy from admin policy
○ Avoid duplication
○ Prevent (and detect) violations
● Give admins greater control and flexibility
○ Automated
○ Programmable
○ Expressive
○ Context-aware
opa

6. Architecture
federation-apiserver
federation-controller
opaadmission controller
kind: ReplicaSet
metadata:
name: my-app
annotations:
customer: acmecorp
● Admission Controller inside federation-apiserver
queries Policy Engine when resources are
created or updated
● Admission Controller implements “fail-closed”
model in case query fails.
POST /v1/data/k8s/placement
input:
resource:
metadata:
name: my-app
annotations:
customer: acmecorp
...
HTTP/1.1 200 OK
result:
annotations:
replica-set-preferences:
clusters:
us-west-1: {weight: 1}
asia-se-3: {weight: .5}
Example Query

7. Architecture
host cluster
pod
federation-apiserver
federation-controller opa
admission controller
etcd
sidecar
external state
policy definition

8. Demo
● Apps labelled with customer name
● Customers associated with a jurisdiction (e.g, EU, US)
● Apps may be labelled with criticality
○ If “low” then public cloud clusters may be used
○ Otherwise on-premise clusters must be used

9. Visibility & Remediation
federation-apiserveropa sidecar
which of customer
x’s apps are
deployed in EU?
which customers are
affected if cluster X is
no longer PCI
certified?
policy
clusters, deployments,
services, ...

10. Conflicts
● Devs could specify conflicting intent
(result: empty set)
● Devs could request clusters that are not
allowed (result: error)
● Resolve conflicts within policy engine
whenever possible
○ Policy is the only place where all intent is known
kind: ReplicaSet
metadata:
annotations:
customer: acmetel-US
pci-compliance: false
replica-set-preferences:
clusters:
- us-west-2
- eu-central-1
...
not_allowed[cluster] {
requested_clusters[cluster]
not allowed_clusters[cluster]
}
errors[“invalid cluster(s)”] { not_allowed != set() }

11. Future Work
● Improve policy management
○ Current: policies stored as ConfigMaps in the
federation-apiserver
○ Future: policies represented as first-class API objects
○ Cleaner mechanism for reporting policy enforcement
status
■ Installed, errors, etc.
● Demonstrate new use cases
○ Cost-based policies
■ Replicate external data representing resource
pricing (e.g., cpu, memory, etc.)
■ Pick clusters based on pricing data
■ Cluster inter-connect may be expensive
federation-apiserver
federation-controller
opa

12. ● Kubernetes Federation enables hybrid-cloud
deployments for a variety of use cases
● Resource placement is a policy-rich problem
that must address important business
requirements
● Policy solution should empower admins with
greater control and flexibility
Conclusion
federation-apiserver
federation-controller
opa

13. Thank You!
SIG-Federation
Open Policy Agent (OPA)
github.com/open-policy-agent/opa