Expert Tips for Successful Kubernetes Deployments on AWS

© 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Mitch Beaumont
Solutions Architect, Amazon Web Services
Expert tips for successful Kubernetes
deployments on AWS

We Give You The Power To Choose:
ECS EK
S
EC2 Fargate EC2 Fargate
1. Choose your orchestration
tool
2. Choose your launch type

Kubernetes 101
• Started at Google
• Influenced by Google Borg
• Container orchestrator
• Contributed to the CNCF
Cloud Native Computing Foundation

Operations as Code
Annotated Documentation
Frequent, small, reversible changes
Refine operation procedures frequently
Anticipate failure
Learn from operational failures
What does operational
excellence look like on AWS?

$ kubectl get tips –n aws-summit

GET api/v1/namespaces/aws-summit/tips/{1}
Never build a Kubernetes Cluster the hard way!

Options for Kubernetes cluster setup
Community
• Kops – kubernetes-aws.io
• Kubeadm – toolkit for bootstrapping a cluster
• eksctl – simple cli for creating cluster on EKS
Enterprise
• Elastic Container Service for Kubernetes (EKS)
• Red Hat OpenShift
Other options
• CloudFormation, Terraform (EKS Module), Ansible and Puppet

Kops – Kubernetes Operations
Community supported
• SIG AWS
• Kops office hours and Slack channel
Generate CloudFormation or Terraform scripts

kops create cluster
--name cluster1.kubernetes-aws.io
--zones ap-southeast-2a, ap-southeast-2b, ap-southeast-2c
--master-count 3
--master-size m4.large
--node-count 5
--state s3://kubernetes-aws-io
--yes

Operational
Excellence
Security
ReliabilityPerformance
Efficiency
Cost
Optimisation
Master
s
etcd

Elastic Container Service for Kubernetes
mycluster.eks.amazonaws.com
Availability
Zone 1
Availability
Zone 2
Availability
Zone 3
Kubectl

Elastic Container Service for Kubernetes
Availability
Zone 1
Availability
Zone 2
Availability
Zone 3
etcd
Master
etcd
Master
etcd
Master
AWS Managed
Customer Managed

aws eks create-cluster --cluster-name summit-cluster
--role-arn arn:aws:iam::1123581321:role/eks-vpc-
EksServiceRole-21345589144
--vpc-id vpc-21345589144
--region us-west-2

Pop Quiz – Deploying Clusters
What are some of the community tools you
can use for deploying Kubernetes clusters
on AWS?
Kops, kubeadm and eksctl, kubicorn help
remove some of the effort.

What does Kops do?
Provisions infrastructure, as well as the
Kubernetes cluster components.
It can also generate CloudFormation
templates and Terraform Manifests which
can be used as a baseline for a cluster.

Amazon EKS provides a fully managed
____?
“Control Plane”. This includes highly
available Kubernetes master and etcd
nodes.

GET api/v1/namespaces/aws-summit /tips/{2}
Consider your networking options

“Every pod should have its own IP
address, and all pods should be able
to talk to one and other”
Node Node
Pod Pod
Networking With Kubernetes

What Is CNI
Network
Plugin
Runtime
Network
• A way for Kubernetes to tell an
underlying SDN that it wants to
connect a container to a network.
• Standards based pluggable
architecture for container networking.
• API for writing plugins to configure
network interfaces for containers.
• CNCF Project

Popular Solutions For Kubernetes
Networking
StarsonGithub

ENI
VPC Subnet – 172.16.18.0/24
Bridge
Pod CIDR
10.244.10.0/24
Destination Via
10.244.10.0/24 172.16.18.101
10.244.11.0/24 172.16.18.102
… …
Node IP
172.16.18.101
AWS Route Table
Networking With Kubenet

Overlay Networks
• Can't get enough IP space? (subnets
not sized correctly)
• Your existing network cannot handle
the number of routes required (VPC
route tables have a limit of 50 routes).
• You want to tap in to additional
capabilities that a specific overlay
network provides – network policies

Networking With Flannel
src: node
dst: node
VPC Subnet – 10.0.0.0/24
Instance 2
Bridge Bridge
Flannel0
src: pod
dst: pod
Flannel0
Flannel Pod CIDR
10.244.0.0/16
Flannel Pod CIDR
10.244.0.0/16
Instance 1
ENIENI

Do I Need An Overlay Network?

Networking With Amazon VPC CNI
Nginx Pod
Java Pod
ENI
Secondary IPs:
10.0.0.1
10.0.0.2
Veth IP: 10.0.0.1
Veth IP: 10.0.0.2
Nginx Pod
Java Pod
ENI
Veth IP: 10.0.0.20
Veth IP: 10.0.0.22
Secondary IPs:
10.0.0.20
10.0.0.22
ec2.associateaddress()
VPC Subnet – 10.0.0.0/24
Instance 1 Instance 2

https://github.com/aws/amazon-vpc-cni-k8s/

Pop Quiz - Networking
What do all pods in a Kubernetes cluster
need?
All pods need a real IP and must be able to
communicate with each other.

When you deploy a Kubernetes cluster
using Kops, what is the default network
provider?
Kops defaults to the Kubenet network
provider.

Why might I choose one CNI network
plugins over another?
Some CNI network plugins require a backing
data store, others need an overlay network
and other offer additional functionality.

Ensure Role Based Access Control (RBAC)
is enabled on your cluster

IAM Authentication + Kubectl
K8s action allowed/denied
Authorises AWS Identity with RBAC
K8s API
Passes AWS Identity
Verifies AWS Identity
AWS Auth
1
2
3
4
Kubectl

Binding IAM Roles to Kubernetes RBAC
---
apiVersion: v1
kind: ConfigMap
metadata:
name: aws-auth
namespace: kube-system
data:
mapRoles: |
- rolearn: arn:aws:iam::000000000000:role/KubernetesAdmin
username: admin-mitch:{{SessionName}}
groups:
- mitch:editors
mapUsers: |
- userarn: arn:aws:iam::000000000000:user/mitch
username: mitch
• Create an IAM role.
• Attach IAM Role to
IAM Group.
• Apply the aws-auth
config map.
• Create a cluster role
binding to the k8s role
for those users who
assume the IAM role.

RBAC - Namespace
get
list
Pod
summit-view
RoleBinding
summit-view
RoleUser
view
ClusterRole

RBAC – Cluster-Wide
get
list
Cluster
summit-view
ClusterRoleBinding
summit-view
ClusterRoleUser
view
ClusterRole

Pop Quiz – RBAC
Why should you enable RBAC on your
Kubernetes cluster?
Implementing fine-grained control over how
users access resources is an important
property of any well architected system.

Pop Quiz – RBAC
What is one of the best ways to authenticate
requests to a Kubernetes cluster deployed
on AWS?
Using the AWS IAM Authenticator. A tool that
lets you use IAM credentials to authenticate
to a Kubernetes cluster.

Pop Quiz – RBAC
Can you use an IAM group to control access
to resources within a Kubernetes cluster?
Yes. By binding an IAM role to a Kubernetes
role using the aws-auth config map you can
control the specific actions IAM users are
able to perform within your cluster.

Observe all the things!

Observability in Your Kubernetes Cluster
TracingAlertsEventsMetricsLogs
ApplicationContainerNodeCluster

Building A Log Aggregator
An open source data collector providing a unified logging layer,
supported by 500+ plugins connecting to many types of systems.
A distributed, RESTful search and analytics engine.
(Amazon Elasticsearch)
Lets you visualise your Elasticsearch data.

WorkerWorkerMaster
WorkerWorkerMaster
ASG
AZ1
Region
AZ2
ASG
Amazon
CloudWatch
Logs
Amazon
Elasticsearch
Service
Kibana
Fluentd
DaemonSet
Kubectl logs
Elasticsearch (index),
Fluentd (store), and
Kibana (visualise)
Logs

Tracing with AWS X-Ray
https://aws.amazon.com/de/blogs/compute/application-tracing-on-kubernetes-with-aws-x-ray/
• Configure IAM to allow
pods running on
nodes to send traces
to X-Ray.
• Deploy the AWS X-Ray
Demon.
• Integrate AWS X-Ray
SDK in to your
application.

Pop Quiz – Observability
What are some of the key traits required by
tool used to monitor Kubernetes?
They should compliment the dynamic nature
of containerised environments.

Why is it important that we capture
application and cluster log data?
This data is invaluable in helping us learn
from operational failures and evolve our
application architectures.

What are some of the tools we can use to
capture, store and analyse log information in
a Kubernetes cluster?
Open source tools like Elasticsearch,
FluentD and Kibana, as well as great
solutions from partners.

Build, Ship, Run …

Application Deployment to Kubernetes
Developer
AWS CodePipeline
AWS CodeCommit AWS CodeBuild AWS Lambda
Amazon ECR Kubernetes

Developer
AWS CodePipeline
• Code is committed to AWS CodeCommit.
• PR created for review of changes.
• Changes merged to master branch.
• AWS CodePipeline detects changes and
starts moving changes through the
pipeline.

Developer
AWS CodePipeline
• AWS CodeBuild packages code
changes and dependencies and
builds a Docker image.
• Other pipeline stages can be
included to test code and the
package, also using AWS
CodeBuild.
• The Docker Image is pushed to
Amazon ECR.

Developer
AWS CodePipeline
• AWS CodePipeline invokes an
AWS Lambda function which
updates the Kubernetes
deployment file with the image
tag.
• AWS Lambda invokes Kubernetes
API (Python SDK) to update
application deployment.
• A rolling update is performed of
the pods to match the Docker
image that was created.

Pop Quiz – Application Deployment
Why is it important that we deploy
frequently, and deploy small, reversible
changes?
Changes can be easily reversed and if they
do fail, it aids in the identification and
resolution of issues.

What are some of the built-in capabilities of
Kubernetes that can help improve the
reliability of application deployments?
The Kubernetes deployment resource type
supports roll-out histories and liveness and
readiness probes.

What are some of the tools that can help me
deploy software in to my Kubernetes
cluster?
In addition to the AWS Code suite of
services, many great open source tools exist
including Helm, Jenkins, GitKube and
Skaffold.

https://github.com/aws-samples/aws-microservices-deploy-options

GET api/v1/namespaces/aws-
summit/tips/{summary}
• Understand what your networking
requirements are.
• There are lots of options available
for deploying clusters.
• It’s hard to know where you’re going
without knowing where you’ve come
from, logging and monitoring are critical.
• Strive for Operational Excellence.

Some Stuff For You …
Kubernetes on AWS Workshop
https://github.com/aws-samples/kubernetes-aws-workshop
Networking with Amazon EKS
https://aws.amazon.com/blogs/opensource/networking-foundation-eks-
aws-cni-calico/

Thank you!
beaumonm@amazon.com
@mitchybgood

Expert Tips for Successful Kubernetes Deployments on AWS

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Expert Tips for Successful Kubernetes Deployments on AWS

Similar to Expert Tips for Successful Kubernetes Deployments on AWS (20)

More from Amazon Web Services

More from Amazon Web Services (20)

Expert Tips for Successful Kubernetes Deployments on AWS