This document summarizes Chen Fisher's presentation on Kubernetes at nanit. The key points are: 1) Nanit uses Kubernetes for service orchestration, taking advantage of its built-in service discovery, high availability features, and port management capabilities. 2) Kubernetes is preferred over Amazon ECS due to ECS lacking service discovery and only supporting basic health checks. 3) Nanit runs two Kubernetes clusters for staging and production, with over 100 pods across more than 20 instances in production. 4) Nanit uses Kubernetes for deploying microservices from development to production, with Jenkins doing automated testing and deployment via Slack notifications.

1. Kubernetes @ nanit
Chen Fisher, VP of SW

2. nanit
Smart baby monitor

4. API
Clipper
Focus
Gateway
Grant
Mailman
RTMP
RabbitMQ
S3
Stampa
State
Department
Timeline
Taper
Algorithms
Communication

5. Microservices
Docker

6. docker-compose up -d

7. Orchestration

8. Orchestration
- Service Discovery
- High Availability
- Resource Management
- Port Management
API
Clipper
Focus
Gateway
Grant
Mailman
RTMP
RabbitMQ
S3
Stampa
State
Department
Timeline
Taper
Algorithms

9. ECS vs Kubernetes

10. https://railsadventures.wordpress.com/2015/12/06/why-we-chose-kubernetes-over-ecs/
-- Erez Rabih

11. Service Discovery / ECS
- Offers no service discovery
- One can use external tools like Consul (https://aws.amazon.com/blogs/compute/service-discovery-
via-consul-with-amazon-ecs/)

12. Service Discovery / Kubernetes
- Has built-in service discovery (http://kubernetes.io/docs/user-guide/services/)
- One can simply refer to a service by its name
- Can group and isolate services by namespaces

13. High Availability / ECS
- Uses ELB health checks
- Limited to HTTP/TCP health checks
- If not using ELB then only recovers on container exit
- Must create ELB for every service to get health checks

14. High Availability / Kubernetes
- Offers exec health check in addition to http
- One can perform application level health check

15. Port Management / ECS
- Maps an ELB to a service which exposes a container port on host
- Cannot have more than one container with same exposed port on same host
- Must manually manage container ports

16. - Assigns random port for every container
- Maps ELB to the container’s random port
Port Management / Kubernetes

17. Kubernetes
- AWS
- Kube 1.2.2
- Two clusters: Staging & Production
- 20+ m4-large instances on production (pre-launch)
- 100+ pods
- RDS, DynamoDB outside of cluster

18. From Dev to Prod

19. Dev and test locally
clojure, ruby, go, c/c++, elixir

20. Push to Github
pull requests

21. Jenkins picks up and tests
marks commit on Github

22. Manual merge to master

23. Deployment via Slack

25. Deployment / directory structure
● nanit/
o api/
 app/
 Dockerfile
 code...
 kube/
 dep.yml
 Makefile

26. apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: {{APP_NAME}}
spec:
replicas: {{REPLICAS}}
template:
metadata:
labels:
app: {{APP_NAME}}
spec:
volumes:
- name: some-secret
secret:
secretName: some-secret
containers:
- name: {{APP_NAME}}
image: {{IMAGE_NAME}}
...
Deployment / kube yaml file

27. Deployment / Makefile
APP_NAME=api
IMAGE_TAG=$(shell git log -n 1 --pretty=format:%h app)
IMAGE_NAME=nanit/$(APP_NAME):$(IMAGE_TAG)
define generate-dep
sed -e 's/{{IMAGE_NAME}}/$(IMAGE_NAME)/g' kube/dep.yml
endef
docker:
sudo docker pull $(IMAGE_NAME) || (sudo docker build -t $(IMAGE_NAME) app && sudo docker push $(IMAGE_NAME))
deploy: docker
$(call generate-dep) | kubectl apply -f -
migrate:
…
test:
...

28. Deployment / config
from Makefile:
REPLICAS?=$(shell curl -s config/$(NANIT_ENV)/api/replicas)
from kube yaml file:
spec:
replicas: {{REPLICAS}}

29. Post deployment
(Monitoring & Alerts)

30. Post deployment (Monitoring & Alerting)
- Currently using built in Kibana & Grafana
- Alerting to Slack using proprietary service

31. Thank You!