Securing & Monitoring Your K8s Cluster with RBAC and Prometheus”.

1
Securing and monitoring your K8s cluster
(RBAC + Prometheus)

3
What’s new in 1.6? – Alpha features
•Per pod eviction
•Pod Injection Policy
•Custom Metrics support in the
Horizontal Pod Autoscaler

4
RBAC in Kubernetes 1.6
RBAC – Role Based Access Control
• Allows fine-grained authorization configuration
• Configurable via API
• Permissions are purely additive, there are no deny
permissions
• Uses concepts of Role and Role Bindings to create assign
and enforce permissions
• Permissions can be scoped to:
• Namespaces
• Nodes
• Control plane
• Enabled by starting the API server with the flag
• --authorization-mode=RBAC

5
RBAC - Attributes
A request has the following attributes that can be considered for
authorization:
• user (the user-string which a user was authenticated as).
• group (the list of group names the authenticated user is a
member of).
• “extra” (a map of arbitrary string keys to string values,
provided by the authentication layer)
• whether the request is for an API resource.
• the request path.
• allows authorizing access to miscellaneous non-resource
endpoints like /api or /healthz (see kubectl).

6
RBAC - Attributes
• The request verb.
• API
verbs get, list, create, update, patch, watch, proxy, redirect,
delete, and deletecollection are used for resource
requests
• HTTP verbs get, post, put, and delete are used for non-
resource requests
• What resource is being accessed (for resource requests only)
• What subresource is being accessed (for resource requests
only)
• The namespace of the object being accessed (for namespaced
resource requests only)
• The API group being accessed (for resource requests only); an
empty string designates the core API Group

7
RBAC – API Overview
Declares 4 top level types that can be interacted with via the API
or kubectl
• Role
• Cluster Role
• RoleBinding
• ClusterRoleBinding

8
RBAC – Role
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
namespace: default
name: pod-reader
rules:
- apiGroups: [""] # "" indicates the core API group
resources: ["pods"]
verbs: ["get", "watch", "list"]

9
RBAC – RoleBinding to Role
# This role binding allows "jane" to read pods in
the "default" namespace.
kind: RoleBinding
metadata:
name: read-pods
namespace: default
subjects:
- kind: User
name: jane
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: pod-reader

10
RBAC – ClusterRole
kind: ClusterRole
metadata:
# "namespace" omitted since ClusterRoles are
not namespaced
name: secret-reader
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "watch", "list"]

11
RBAC – RoleBinding to ClusterRole
# This role binding allows "dave" to read secrets in
the "development" namespace.
kind: RoleBinding
metadata:
name: read-secrets
namespace: development # This only grants
permissions within the "development" namespace.
subjects:
- kind: User
name: dave
roleRef:
kind: ClusterRole
name: secret-reader

12
RBAC – ClusterRoleBinding
# This cluster role binding allows anyone in the
"manager" group to read secrets in any namespace.
kind: ClusterRoleBinding
metadata:
name: read-secrets-global
subjects:
- kind: Group
name: manager
roleRef:
kind: ClusterRole
name: secret-reader

13
RBAC – Use Cases
• Limit user access to specific namespaces
• Limit user permissions relative to their Role
• Group users based on their Roles and assign permissions to the
group
• Pre-assign special hardware nodes to certain user groups and
workloads
• Segment your continuous deployment pipeline with
permissions to deploy in to only specific namespaces.

14
Hierarchy of reliability - Monitoring

15
Prometheus
What is Prometheus?
• Monitoring system and time series database
• Inspired by BorgMon
• Not BorgMon
• Opensource – developed at SoundCloud initially
• How does it work?
• Collects metrics at scale via HTTP
• Easy scaling
• Can handle thousands of targets, millions of timeseries

16
Prometheus - Features
• Multi-dimensional data model
• Flexible query language to leverage dimensionality
• No reliance on distributed storage, autonomous single server
nodes
• Collection happens via a pull model over HTTP
• Push is also supported via an intermediary gateway
• Targets can be discovered statically or via service discovery
• Good graphing and dashboarding support

18
Running Prometheus in Kubernetes
• What we need
• Pod specification defining how to run Prometheus
• Load and manage configuration
• Service specification to access Prometheus on stable IP
• Options
• Write own pod+service+petset+... manifests
• Kubernetes Helm chart in the making
https://github.com/kubernetes/charts/pull/151
• CoreOS wrote an Operator managing Prometheus and its
configuration
https://github.com/coreos/kube-prometheus

19
Prometheus Operator by CoreOS
• What is an Operator?
• Builds on basic Kubernetes resource and controller
concepts
• Included application domain knowledge and best practices
• Frees you from deployment and lifecycle management
details
• Prometheus Operator by CoreOS
• Single command install
• Configure manage instances using a single declarative
configuration
• Configuration drives the creation, configuration and
management of Prometheus instances

20
Prometheus Operator - Features
• Create/Destroy
• Simple Configuration
• Target Services via Labels

21
Prometheus Operator – How does it work?
The Operator defines 2 Third Party Resources(TPR)
• Prometheus Resource
• Data Retention
• Persistent Volume Claims
• Number of replicas
• Version
• Alert Managers
• Service Monitor

22
Prometheus Operator – How does it work?

23
Prometheus Operator - Deployment
Deploy the operator
$ kubectl create -f
https://coreos.com/operators/prometheus/latest/prometheus
-operator.yaml
deployment "prometheus-operator" created
Create the Prometheus TPR
apiVersion: monitoring.coreos.com/v1alpha1
kind: Prometheus
metadata:
name: prometheus-k8s
labels: prometheus: k8s
spec: version: v1.3.0
$ kubectl create -f
-k8s.yaml

24
Prometheus Operator – Cluster Monitoring
Deploy exporters providing metrics on cluster nodes and
Kubernetes business logic
$ kubectl create -f
https://coreos.com/operators/prometheus/latest/exporters.
yaml deployment
"kube-state-metrics"
created service "kube-state-metrics"
created daemonset "node-exporter"
created service "node-exporter" created
Create the ConfigMap containing the Prometheus
configuration
$ kubectl apply -f
-k8s-cm.yaml
configmap "prometheus-k8s" configured

25
Prometheus Operator – Cluster Monitoring

26
Prometheus Operator – Service Monitoring
Define how a service should be monitored
kind: ServiceMonitor
metadata:
name: frontend
labels:
tier: frontend
spec:
selector:
matchLabels:
tier: frontend
endpoints:
- port: web # works for different port numbers as long as
the name matches
interval: 10s # scrape the endpoint every 10 seconds

27
Create a Prometheus instance that includes this
ServiceMonitor
kind: Prometheus
metadata:
name: prometheus-frontend
labels: prometheus: frontend
spec: version: v1.3.0 # Define that all ServiceMonitor
TPRs with the label `tier = frontend` should be included
# into the server's configuration.
serviceMonitors:
- selector:
matchLabels: tier: frontend

28
Create the Service monitor and Prometheus Objects
$ kubectl create -f
https://coreos.com/operators/prometheus/latest/servicemon
itor-frontend.yaml
servicemonitor "frontend" created
$ kubectl create -f
-frontend.yaml
prometheus "prometheus-frontend"
created service "prometheus-frontend" created

29
Deploy an application that matches the label tier =
frontend
$ kubectl create –f ./example-app.yaml

Securing & Monitoring Your K8s Cluster with RBAC and Prometheus”.

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Securing & Monitoring Your K8s Cluster with RBAC and Prometheus”.

Similar to Securing & Monitoring Your K8s Cluster with RBAC and Prometheus”. (20)

Recently uploaded

Recently uploaded (20)

Securing & Monitoring Your K8s Cluster with RBAC and Prometheus”.