Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Integration kubernetes with docker private registry

218 views

Published on

What's the problem when we want to use the private registry in the kubernetes.
We also want to run a Docker-In-Docker Pod to push the private image to that private registry and the kubernetes node will pull the private image to run

Published in: Software
  • Be the first to comment

  • Be the first to like this

Integration kubernetes with docker private registry

  1. 1. Kuberneteswith privatedocker registry hung-weichiu
  2. 2. WHOAMI Hung-Wei Chiu ( ) hwchiu.com Experience ○ Software Engineer at Linker Networks (now) ○ Co-organizer of SDNDS-TW ○ Co-organizer of CUTNG Fields ○ Linux Kernel, Network Stack ○ Networking/Kubernetes
  3. 3. outline Introduction to Docker Registry K8S & Docker Registry ○ Scenario ○ What’s problem ○ How to solve
  4. 4. DockerRegistry A stateless, highly scalable server side application that stores and lets you distribute Docker images.
  5. 5. Thebasicdockerusage docker pull nginx:yyy ○ Official Repositories + tag docker pull hwchiu/xxxx:yyy ○ Username + Image name + tag
  6. 6. Thebasicdockerusage docker pull nginx:yyy ○ Official Repositories + tag docker pull hwchiu/xxxx:yyy ○ Username + Image name + tag That’s for Docker Hub. You don’t need to worry about the location of the registry
  7. 7. Docker Hub Docker pull nginx Host1 Docker pull hwchiu/aaa Host2 Docker knows that you want to pull image from the docker hub (Implicitly)
  8. 8. Privateregistry Setup the registry for yourself For externally-accessible ○ You need to handle the network by yourself ○ IP address or hostname ○ Use HTTPS by default For localhost ○ Only accessed from localhost ○ Always trust (HTTP/HTTPS)
  9. 9. My Registry Docker pull 192.168.2.3/image1 Host1 Docker pull 192.168.2.3/image2 Host2 Docker knows that you want to pull image from the private registry 192.168.2.3
  10. 10. Kubernetes Pull the docker images and run
  11. 11. ImageAscenario I want to use the kubernetes to do the CI/CD of my application. I want to build the docker image based on my application in the kubernetes pod. I want to run the docker image I build before
  12. 12. workloads Run a kubernetes pod(DockerHelper) ○ Build the docker image ○ Push the docker image to private registry Run another kubernetes pod ○ Based on your own application
  13. 13. In the network view Make sure hosts can connect to registry via IP/Hostname
  14. 14. My Registry Host1 Host2 a.b.c.d Kubernetes cluster Docker Helper Build/Push my own docker image We must make sure the host1 can connect to a.b.c.d
  15. 15. My Registry Host1 Host2 a.b.c.d Kubernetes cluster Docker Helper Build/Push my own docker image Run my own docker image We must make sure the host2 can connect to a.b.c.d
  16. 16. In the (in)security view We need to handle the HTTPS
  17. 17. If you’re rich, buy the certificate and skip this.
  18. 18. My Registry Host1 Host2 IP:$REGISTRY_SERVER Kubernetes cluster Run my own docker image Start the docker daemon with –insecure-registry=$REGISTRY_SERVER option For insecure solution Docker Helper Build/Push my own docker image
  19. 19. My Registry Host1 Host2 IP:$REGISTRY_SERVER Kubernetes cluster Run my own docker image Put the cert into /etc/docker/certs.d/$REGISTER_SERVER/ca.crt For secure solution Docker Helper Build/Push my own docker image
  20. 20. Let’s start to design the architecture
  21. 21. Privateregistrylocation Where do we setup the docker registry? In the cluster/Out of the cluster ?
  22. 22. Inthecluster Run as a Pod in the cluster Pros: ○ DH pod can access it by hostname ■ Kubernetes service ○ K8S guarantee the running instance of registry container Cons: ○ Need to handle the data sync within all nodes ■ If the new registry runs on different node. ○ The k8s node can’t access it via hostname.
  23. 23. Kubernetes cluster K8S master K8S nodeK8S node K8S node Magic Network Registry Server 1. Deploy the Register Server with k8s service.
  24. 24. Kubernetes cluster K8S master K8S nodeK8S node K8S node Magic Network Registry Server 1. Deploy the Register Server with k8s service. 2. Docker-Helper (Pod) can use registry.default to talk (easy) 3. K8s node can’t use registry.default (not easy) Docker Helper Build/Push my own docker image Registry.default
  25. 25. Kubernetes cluster K8S master K8S nodeK8S node K8S node Magic Network Registry Server 1. Deploy the Register Server with k8s service. 2. Docker-Helper (Pod) can use registry.default to talk (easy) 3. K8s node can’t use registry.default (not easy) Run own docker image (can’t use the registry.default)
  26. 26. Kubernetes cluster K8S master K8S nodeK8S node K8S node Magic Network Registry Server 1. Sync the data between all K8S node Ø Mount external volume Ø Sync the filesystem
  27. 27. Outofthecluster Run as a standalone server out of the cluster Pros: ○ Node can access it by ip/hostname(DNS) ■ Should consider the SSL Cons: ○ DH Pod should knows the location of registry server ■ Use DNS, need dns server (not easy) ■ Use IP, DH Pod show know the IP once server changes IP.
  28. 28. Kubernetes cluster K8S master K8S nodeK8S node K8S node Magic Network 1. The admin should know the IP/DNS of the Register Server K8S masterRegistry Server
  29. 29. Kubernetes cluster K8S master K8S nodeK8S node K8S node Magic Network 1. The admin should know the IP/DNS of the Register Server 2. DH push the image to Registry Server K8S masterRegistry Server Docker Helper Build/Push my own docker image
  30. 30. Kubernetes cluster K8S master K8S nodeK8S node K8S node Magic Network 1. The admin should know the IP/DNS of the Register Server 2. DH push the image to Registry Server 3. Pull the image from the Registry Server and run as Pod. K8S masterRegistry Server Run own docker image
  31. 31. Problems Certificated (we’re not rich) ○ We need to modify the docker config for each node ○ We also need to modify the docker config for DH pod Network ○ Only the out of cluster + DNS seems good ■ I don’t want to handle any DNS by myself
  32. 32. How to solve those problems ????
  33. 33. Finally, I thought a solution
  34. 34. Fornetworking We run the registry as Pod on K8S We also run nginx (daemonSet) on each k8s node. ○ Act as a proxy ○ Forward the packet to registry.default Assume the docker image is ○ localhost:5566/myapp:master
  35. 35. Registry Server Nginx-ProxyNginx-Proxy Docker daemon localhost:5566 iptables Pod_IP:80 Registry.default:5566 Overlay network K8s node IP: 5566 iptables Pod_IP:5566 K8S cluster NODE view
  36. 36. FortheDockerHelperPod We runs the pod with hostnetwork=true The DH Pod will use the host network stack. The DH must build the docker image name as the form localhost:5566/…
  37. 37. Forsecurity The docker trust the localhost We don’t need to handle all certificated problems now.
  38. 38. Thanks!

×