Kuberneteswith
privatedocker
registry
hung-weichiu

WHOAMI
Hung-Wei Chiu ( )
hwchiu.com
Experience
○ Software Engineer at Linker Networks
(now)
○ Co-organizer of SDNDS-TW
○ Co-organizer of CUTNG
Fields
○ Linux Kernel, Network Stack
○ Networking/Kubernetes

outline
Introduction to Docker Registry
K8S & Docker Registry
○ Scenario
○ What’s problem
○ How to solve

DockerRegistry
A stateless, highly scalable server side
application that stores and lets you
distribute Docker images.

Thebasicdockerusage
docker pull nginx:yyy
○ Official Repositories + tag
docker pull hwchiu/xxxx:yyy
○ Username + Image name + tag

Thebasicdockerusage
docker pull nginx:yyy
○ Official Repositories + tag
docker pull hwchiu/xxxx:yyy
○ Username + Image name + tag
That’s for Docker Hub.
You don’t need to worry about the
location of the registry

Docker Hub
Docker pull nginx
Host1
Docker pull
hwchiu/aaa
Host2
Docker knows that you want to pull image from
the docker hub (Implicitly)

Privateregistry
Setup the registry for yourself
For externally-accessible
○ You need to handle the network by
yourself
○ IP address or hostname
○ Use HTTPS by default
For localhost
○ Only accessed from localhost
○ Always trust (HTTP/HTTPS)

My Registry
Docker pull
192.168.2.3/image1
Host1
Docker pull
192.168.2.3/image2
Host2
Docker knows that you want to pull image from
the private registry
192.168.2.3

Kubernetes
Pull the docker images and run

ImageAscenario
I want to use the kubernetes to do the
CI/CD of my application.
I want to build the docker image based
on my application in the kubernetes
pod.
I want to run the docker image I build
before

workloads
Run a kubernetes pod(DockerHelper)
○ Build the docker image
○ Push the docker image to private registry
Run another kubernetes pod
○ Based on your own application

In the network view
Make sure hosts can connect to registry
via IP/Hostname

My Registry
Host1
Host2
a.b.c.d
Kubernetes cluster
Docker Helper
Build/Push my
own docker
image
We must make sure the host1 can
connect to a.b.c.d

My Registry
Host1
Host2
a.b.c.d
Kubernetes cluster
Docker Helper
Build/Push my
own docker
image
Run my own
docker image
We must make sure the host2 can
connect to a.b.c.d

In the (in)security view
We need to handle the HTTPS

If you’re rich, buy the
certificate and skip this.

My Registry
Host1
Host2
IP:$REGISTRY_SERVER
Kubernetes cluster
Run my own
docker image
Start the docker daemon with
–insecure-registry=$REGISTRY_SERVER option
For insecure solution
Docker Helper
Build/Push my
own docker
image

My Registry
Host1
Host2
IP:$REGISTRY_SERVER
Kubernetes cluster
Run my own
docker image
Put the cert into
/etc/docker/certs.d/$REGISTER_SERVER/ca.crt
For secure solution
Docker Helper
Build/Push my
own docker
image

Let’s start to design the architecture

Privateregistrylocation
Where do we setup the docker registry?
In the cluster/Out of the cluster ?

Inthecluster
Run as a Pod in the cluster
Pros:
○ DH pod can access it by hostname
■ Kubernetes service
○ K8S guarantee the running instance of
registry container
Cons:
○ Need to handle the data sync within all
nodes
■ If the new registry runs on different node.
○ The k8s node can’t access it via hostname.

Kubernetes cluster
K8S master
K8S nodeK8S node K8S node
Magic Network
Registry Server
1. Deploy the Register
Server with k8s service.

Kubernetes cluster
K8S master
K8S nodeK8S node K8S node
Magic Network
Registry Server
1. Deploy the Register
Server with k8s service.
2. Docker-Helper (Pod)
can use registry.default
to talk (easy)
3. K8s node can’t use
registry.default (not
easy)
Docker Helper
Build/Push my
own docker
image
Registry.default

Kubernetes cluster
K8S master
K8S nodeK8S node K8S node
Magic Network
Registry Server
1. Deploy the Register
Server with k8s service.
2. Docker-Helper (Pod)
can use registry.default
to talk (easy)
3. K8s node can’t use
registry.default (not
easy)
Run own docker
image (can’t use the
registry.default)

Kubernetes cluster
K8S master
K8S nodeK8S node K8S node
Magic Network
Registry Server
1. Sync the data between
all K8S node
Ø Mount external volume
Ø Sync the filesystem

Outofthecluster
Run as a standalone server out of the
cluster
Pros:
○ Node can access it by ip/hostname(DNS)
■ Should consider the SSL
Cons:
○ DH Pod should knows the location of
registry server
■ Use DNS, need dns server (not easy)
■ Use IP, DH Pod show know the IP once server
changes IP.

Kubernetes cluster
K8S master
K8S nodeK8S node K8S node
Magic Network
1. The admin should know
the IP/DNS of the
Register Server
K8S masterRegistry Server

Kubernetes cluster
K8S master
K8S nodeK8S node K8S node
Magic Network
1. The admin should know
the IP/DNS of the
Register Server
2. DH push the image to
Registry Server
K8S masterRegistry Server
Docker Helper
Build/Push my
own docker
image

Kubernetes cluster
K8S master
K8S nodeK8S node K8S node
Magic Network
1. The admin should know
the IP/DNS of the
Register Server
2. DH push the image to
Registry Server
3. Pull the image from the
Registry Server and run
as Pod.
K8S masterRegistry Server
Run own docker
image

Problems
Certificated (we’re not rich)
○ We need to modify the docker config for
each node
○ We also need to modify the docker config
for DH pod
Network
○ Only the out of cluster + DNS seems good
■ I don’t want to handle any DNS by myself

How to solve those problems ????

Finally, I thought a solution

Fornetworking
We run the registry as Pod on K8S
We also run nginx (daemonSet) on
each k8s node.
○ Act as a proxy
○ Forward the packet to registry.default
Assume the docker image is
○ localhost:5566/myapp:master

Registry Server Nginx-ProxyNginx-Proxy
Docker daemon
localhost:5566
iptables
Pod_IP:80
Registry.default:5566
Overlay
network
K8s node IP: 5566
iptables
Pod_IP:5566
K8S cluster
NODE view

FortheDockerHelperPod
We runs the pod with
hostnetwork=true
The DH Pod will use the host network
stack.
The DH must build the docker image
name as the form localhost:5566/…

Forsecurity
The docker trust the localhost
We don’t need to handle all
certificated problems now.

Thanks!