The document discusses policy-based resource placement in hybrid-cloud federations of Kubernetes clusters, emphasizing the need for a flexible, automated, and context-aware approach to manage deployments. It outlines how the federation control plane, along with a policy engine, evaluates resource annotations and controls placement across different clusters while considering legal, cost, and technical constraints. The future work includes improving policy management and introducing cost-based policies to optimize resource allocation.

1. Policy-based Resource Placement
...across Hybrid-Cloud Federations of Kubernetes Clusters
Irfan UR Rehman (Huawei)
@irfanurrehman
Torin Sandall (Styra)
@sometorin

2. Federation: Overview
federation-apiserver
federation-controller
app app
Federation Control Plane
Federated
Deployment
etcd
Federated
Clusters

3. Federation: Placement
federation-apiserver
federation-controller
kind: ReplicaSet
metadata:
annotations:
replica-set-preferences: |
{“clusters”: {
“us-west-1”: {“weight”:1},
“asia-se-2”: {“weight”:1}
}}
● Placement can be controlled
per-resource via annotations
● 2 annotations supported:
federation.kubernetes.io/replica-set-preferences
federation.alpha.kubernetes.io/cluster-selector
● federation-controller evaluates
annotations to produce final
placement
ReplicaSet
created
1
2
3
4

4. Policy-based Placement
federation-apiserver
federation-controller
● Resource placement is a “policy-rich” problem space
○ Legal regulation, cost, technical constraints, internal
conventions, etc.
● Goal: give admins greater control and flexibility
○ Automated & programmable
○ Expressiveness
○ Leverage context
○ Ease management
● Decouple developer intent from admin policy
○ Avoid duplication
○ Prevent (and detect) violations
○ Abstract policy implementation
● Policy Engine decides which clusters app runs on
○ Pluggable
○ Simple interface
opa

5. Architecture
federation-apiserver
federation-controller
opaadmission controller
kind: ReplicaSet
metadata:
name: my-app
annotations:
customer: acmecorp
● Admission Controller inside federation-apiserver
queries Policy Engine when resources are
created or updated
● Admission Controller implements “fail-closed”
model in case query fails.
POST /v1/data/k8s/placement
input:
resource:
metadata:
name: my-app
annotations:
customer: acmecorp
...
HTTP/1.1 200 OK
result:
annotations:
replica-set-preferences:
clusters:
us-west-1: {weight: 1}
asia-se-3: {weight: .5}
Example Query

6. Architecture
host cluster
pod
federation-apiserver
federation-controller opa
admission controller
etcd
sidecar
external state
policy definition

7. Example
Apps labelled with customer name. Customers are associated with a
jurisdiction. Furthermore, apps may be labelled with criticality. If low then public
cloud clusters may be used, otherwise, on-prem clusters must be used.

8. Visibility & Remediation
federation-apiserveropa sidecar
which of customer
x’s apps are
deployed in EU?
which customers are
affected if cluster X is
no longer PCI
certified?
policy
clusters, deployments,
services, ...

9. Conflicts
● Developers could accidentally specify
conflicting intent (result: empty set)
● Developers could explicitly request invalid
clusters (result: error)
● Resolve conflicts within policy engine
whenever possible
○ Policy is the only place where all intent is known
kind: ReplicaSet
metadata:
annotations:
customer: acmetel-US
criticality: low
replica-set-preferences:
clusters:
- us-west-2
- eu-central-1
...
not_allowed[cluster] {
requested_clusters[cluster]
not allowed_clusters[cluster]
}
errors[“invalid cluster(s)”] { not_allowed != set() }

10. Future Work
● Improve policy management
○ Current: policies stored as ConfigMaps in the
federation-apiserver
○ Future: policies represented as first-class API objects
○ Cleaner mechanism for reporting policy enforcement
status
■ Installed, errors, etc.
● Demonstrate new use cases
○ Cost-based policies
■ Replicate external data representing resource
pricing (e.g., cpu, memory, etc.)
■ Pick clusters based on pricing data
■ Cluster inter-connect may be expensive
federation-apiserver
federation-controller
opa

11. ● Kubernetes Federation enables hybrid-cloud
deployments for a variety of use cases
● Resource placement is a policy-rich problem
that must address important business
requirements
● Policy solution should empower admins with
greater control and flexibility
Conclusion
federation-apiserver
federation-controller
opa

12. Thank You!
SIG-Federation
Open Policy Agent (OPA)
github.com/open-policy-agent/opa