One of the most underrated features of Kubernetes is namespaces. In the market, instead of using this feature, people are still stuck with having different clusters for their environments. This talk will try to break this approach, and will introduce how we end up using ephemeral namespaces within our CI/CD pipeline. It will cover the architecture of our system for running the user acceptance tests on isolated ephemeral namespaces with every bits and pieces running within pods. While doing this, we will set up our CI/CD pipeline on top of TravisCI, GoCD, and Selenium that is controlled by Nightwatch.js. Sched Link: http://sched.co/6Bcb

1. LEVERAGINGLEVERAGING
EPHEMERAL NAMESPACESEPHEMERAL NAMESPACES
IN A CI/CD PIPELINEIN A CI/CD PIPELINE
Can Yücel (@canthefason)
Senior Software Engineer
KubeCon EU 2016

2. HighlightsHighlights
Fundamentals of namespaces
Breaking the idea of having separate clusters
Ephemeral namespaces
Talk about some Kubernetes early stage features
Running every single piece as Kubernetes components

3. NamespacesNamespaces
“ A namespace is a mechanism to partition resources
created by users into a logically named group.
~ Kubernetes Docs

4. Isolation on Different LevelsIsolation on Different Levels
Network level isolation
Access policies
Resource control

5. Network Level IsolationNetwork Level Isolation
Leveraging subdomainsLeveraging subdomains

6. Access PoliciesAccess Policies
{"user":"admin"}
{"user":"scheduler", "readonly": true, "resource": "pods"}
{"user":"scheduler", "resource": "bindings"}
{"user":"proxy", "resource": "services"}
{"user":"proxy", "resource": "endpoints"}
{"user":"kubelet", "resource": "pods"}
{"user":"kubelet", "resource": "nodes"}
{"user":"kubelet", "readonly": true, "resource": "services"}
{"user":"kubelet", "readonly": true, "resource": "endpoints"}
{"user":"kubelet", "resource": "events"}
{"user":"bob", "readonly": true, "namespace": "prod"}
{"user":"alice", "namespace": "prod"}
policy.jsonl
ABAC provides much more granularity on policy
management

7. Resource ControlResource Control
apiVersion: v1
kind: ResourceQuota
metadata:
name: quota
spec:
- hard:
memory: "1Gi"
cpu: 20
pods: 15
services: 5
replicationcontrollers: 10
resourcequotas: 1
Cluster:
32 GB RAM, and 16 cores
Team A:
20 GB RAM, and 10 cores
Team B:
10 GB RAM, and 4 cores

8. How to Partition?How to Partition?
Environment based partitioningEnvironment based partitioning
qa, stage, production...
System / team based partitioningSystem / team based partitioning
kube-system, devops, bots
Project based partitioningProject based partitioning
example.com, better-example.com

9. A Day of a CI/CD PipelineA Day of a CI/CD Pipeline
Provision separate machines for every build
Run your tests on isolated clusters
When all tests are successful tear down the cluster
If it fails keep the cluster up for a while for debugging
Ephemeral Namespaces!Ephemeral Namespaces!
namespaces
namespaces
namespace

10. Ephemeral Namespaces AreEphemeral Namespaces Are
Isolated environments that are running different versions
of services on top of it
The environments where we run our integrations/e2e tests,
and gets dumped when we get the end results

11. Namespaces with Benefits!Namespaces with Benefits!
Time effective provisioning
Efficient resource utilization
In a CI/CD pipeline, namespaces provide:In a CI/CD pipeline, namespaces provide:

12. Time Effective ProvisioningTime Effective Provisioning
It takes only a couple of seconds to create all
services

13. Efficient Resource UtilizationEfficient Resource Utilization
Let your scheduler decide on which
host you will run your test
instances

14. Deployment ProcessDeployment Process
1. Run your unit tests
2. Build Docker Image
3. Deploy to sandbox
4. Provision services that you will run your
tests against
5. Run your integration/e2e tests
6. Delete namespace
7. Deploy updated services to staging/prod
Happy Path!

15. Provisioning Test EnvironmentsProvisioning Test Environments
Identical environments with different versions!

16. Pods From Different NamespacesPods From Different Namespaces
➜ kubectl get po --namespace=e2e-1
NAME READY STATUS RESTARTS AGE
mongo-oij3f 1/1 Running 0 10m
nginx-44k6p 1/1 Running 0 10m
selenium-9bcfc 1/1 Running 0 10m
todo-service-phgrb 1/1 Running 0 10m
todo-service-rbrjl 1/1 Running 0 10m
➜ kubectl get po --namespace=e2e-2
NAME READY STATUS RESTARTS AGE
mongo-p6g8c 1/1 Running 0 5m
nginx-mgdzz 1/1 Running 0 5m
selenium-9l81p 1/1 Running 0 5m
todo-service-mt9gh 1/1 Running 0 5m
todo-service-yxo9v 1/1 Running 0 5m
➜ kubectl get po --namespace=e2e-3
NAME READY STATUS RESTARTS AGE
mongo-llm3x 1/1 Running 0 1m
nginx-vvov6 1/1 Running 0 1m
nightwatch 1/1 Running 0 34s
selenium-g2g1i 1/1 Running 0 1m
todo-service-1k8vc 1/1 Running 0 1m
todo-service-ddfjw 1/1 Running 0 1m

17. Adding E2E Components as PodsAdding E2E Components as Pods
Selenium server
Nightwatch.js scripts

18. All Tests PassedAll Tests Passed ✓✓
$ kubectl delete namespace e2e-10
It will dump every Kubernetes component
within that namespace!

19. Test Gets FailedTest Gets Failed 😞
Find a way to connect to the Selenium Server for
debugging
Expose VNC Port 5900​
kubectl port-forward selenium :5900 --namespace=e2e-1

20. Live In ActionLive In Action

21. How We Use GoCDHow We Use GoCD
Idempotent pipeline stages
Dependency management is handled with fan-in
resolution

22. Evaluating DependenciesEvaluating Dependencies
Text
GO_DEPENDENCY_LABEL_E2E=5.2eedd92
GO_DEPENDENCY_LOCATOR_E2E=e2e-tests/5/buildImage/1
GO_DEPENDENCY_LABEL_TODO=35.86ca86c
GO_DEPENDENCY_LOCATOR_TODO=todo-service/35/deployK8s/1
GO_DEPENDENCY_LABEL_NGINX=12.4288a7c
GO_DEPENDENCY_LOCATOR_NGINX=nginx/12/deployK8s/1
Each GO_DEPENDENCY variable has
dependant pipeline information
For Provisioning Test Environments: Create all dependencies
For Deployment: Compare versions and call create/rolling
update

23. Running Every Piece in PodsRunning Every Piece in Pods
Nightwatch scripts
kubectl run -i -tty nightwatch --image=canthefason/e2e-tests:$E2E_IMAGE_TAG
--restart=Never --namespace=e2e-$GO_PIPELINE_LABEL
state=$(kubectl get -o template po nightwatch $kubeargs
--template={{.status.phase}})
while [ "$state" == "Running" ]; do
sleep 5
echo "waiting for the state"
state=$(kubectl get -o template po nightwatch $kubeargs
--template={{.status.phase}})
done
echo "State: $state"
if [ "$state" == "Failed" ]; then
exit 1
fi

24. Running Every Piece in PodsRunning Every Piece in Pods
Selenium manifest
apiVersion: v1
kind: ReplicationController
metadata:
name: selenium
spec:
replicas: 1
selector:
app: selenium
template:
metadata:
name: selenium
labels:
app: selenium
spec:
volumes:
- name: shm
hostPath:
path: /dev/shm
containers:
- name: selenium
image: selenium/standalone-chrome-debug:2.52.0
ports:
- containerPort: 4444
- containerPort: 5900
imagePullPolicy: Always
volumeMounts:
- name: shm
mountPath: /dev/shm

25. Health CheckersHealth Checkers
Text
curl -k --retry 10 --retry-delay 5 -v
https://$KUBE_HOST/api/v1/proxy/namespaces/sandbox/services/todo/ping
curl -k --silent --output /dev/stderr --write-out "%{http_code}" -v
https://$KUBE_HOST/api/v1/proxy/namespaces/sandbox/services/todo/ping
if [ "$STATUSCODE" -ne "200" ]; then
if [ "$rcExist" != "ReplicationController" ]; then
kubectl delete -f scripts/rc.yml $kubeargs
fi
exit 1
fi

26. Future WorkFuture Work
Scale down the pods when the namespace is idle
Automatically delete namespaces that are older
than certain age
Build a Selenium Grid infrastructure and utilize
Selenium Agents among the namespaces

27. TakeawaysTakeaways
Never ever expose your Apiserver 8080 port!
Think twice before defining your ssh keys as
secrets!
Make sure that you properly setup kubelet
garbage collectors
--maximum-dead-containers=100
--maximum-dead-containers-per-container=2
--minimum-container-ttl-duration=1m0s

28. LinksLinks
http://github.com/canthefason/kubecon
https://github.com/kubernetes/contrib

29. Thanks ToThanks To
Kubernetes Team
LaunchPad Central
Quest Henkart
UK Consulate in NY...

30. Q & AQ & A
Twitter: @canthefason
GitHub: /canthefason