Download as PDF, PPTX
Uploaded byNico Meisenzahl
Effiziente CI/CD-Pipelines – mit den richtigen Tools klappt das
The document discusses the implementation of efficient CI/CD pipelines using various tools and technologies, including GitLab CI/CD and Kubernetes. It highlights the importance of containerization for scalability, immutability, and dependency management, while providing insights into building images with Kaniko and managing cloud-native pipelines with Tekton. Additionally, it addresses recent Docker hub rate limits and offers solutions for managing them.
More Related Content
Similar to Effiziente CI/CD-Pipelines – mit den richtigen Tools klappt das
Effiziente CI/CD-Pipelines – mit den richtigen Tools klappt das
- 1. Effiziente CI/CD-Pipelines –mit den richtigen Tools klappt das Continuous Delivery Day, November 2020
- 2. Nico Meisenzahl • SeniorCloud & DevOps Consultant at white duck • GitLab Hero, Microsoft MVP & Docker Community Leader • Container, Kubernetes, Cloud-Native & DevOps © white duck GmbH 2020 Phone: +49 8031 230159 0 Email: nico.meisenzahl@whiteduck.de Twitter: @nmeisenzahl LinkedIn: https://www.linkedin.com/in/nicomeisenzahl Blog: https://meisenzahl.org
- 3. Agenda • containerized pipeline– why? • pipeline workload on Kubernetes with GitLab CI/CD • container image builds on Kubernetes with Kaniko • cloud-native pipelines with Tekton • Docker Hub rate-limiting © white duck GmbH 2020
- 4. Containerized Pipelines –Why? • for the same reasons why you should use containers • isolation • dependencies • scalability • immutability • example: your new project needs version X all others still require Y • you can include any kind of build / deploy dependency • NodeJS, .NET Core, Go, Terraform, Ansible… you name it © white duck GmbH 2020
- 5. docker run • runningDocker on build hosts is a familiar pattern • but: docker run pipeline jobs have some issues • containers are immutable • you will end up managing caching and artifacts on your own • (and we're not even talking about building containers) © white duck GmbH 2020
- 6. GitLab Runner Kubernetesexecutor • a Gitlab Runner that is aware of Kubernetes • integrates your CI/CD with Kubernetes • runs itself within your Kubernetes Cluster (as pod) • schedules pipeline jobs as pods • allows you to share your compute and scale your pipelines • easy deployment • one-click • production-ready Helm deployment • GitLab Operator (OpenShift) © white duck GmbH 2020
- 7. Cloning, Caching andArtifacts • GitLab Runner Kubernetes executor runs service containers within the pipeline job pod to handle cloning, caching and artifacts • every pipeline job contains out of these stages • prepare → creates pod with build and service containers • pre-build → clones repo, restore cache, download artifacts • build → user build steps • post-build → creates caches and upload artifacts © white duck GmbH 2020
- 8. Advanced topics • supportsadvanced configurations like • resource limits for build jobs • CI services for build dependencies • node selectors for runners • … • GitLab Kubernetes Agent • supports GitOps pattern • not yet available on SaaS • premium only © white duck GmbH 2020
- 9. Pipeline job image •contains everything a single pipeline job needs • binaries, libraries, tools, ... • provides all external dependencies • uses a pipeline to build/rebuild it periodically (security fixes!) • you should define fix versions for your dependencies • uses caching to speed up your builds © white duck GmbH 2020
- 10. Pipeline job image ©white duck GmbH 2020
- 11. Demo • containerized deploypipeline • https://gitlab.com/containerized-cicd/containerized-pipelines © white duck GmbH 2020
- 12. Container Image buildson Kubernetes • you may use Docker-in-Docker to build containers in containers • DinD has some issues • exposing Docker socket • mounting /var/lib/docker • privileged mode • those might be acceptable on a build host but not in a Kubernetes cluster © white duck GmbH 2020
- 13. Image builds withKaniko • image builds without the need of any privileges or dependencies • needs to run as root most of the time L • runs in a container (gcr.io/kaniko-project/executor) • easy to use • you should think about caching to speed up your pipeline • layer caching (layers get pushed to a registry) • base image caching (GitLab cache) © white duck GmbH 2020
- 14. Image builds onKubernetes with Kaniko © white duck GmbH 2020
- 15. Demo • containerized imagebuilds on Kubernetes with Kaniko • https://gitlab.com/containerized-cicd/containerized-jobimage © white duck GmbH 2020
- 16. Cloud-native pipelines withTekton • Tekton Pipelines emerged out of the Knative Build project • founding Continuous Delivery Fundation project • „CI/CD framework for Kubernetes” • Jenkins X pipelines are based on Tekton • contributions by Google, IBM, RedHat, Cloudbees, TriggerMesh, … © white duck GmbH 2020
- 17. Tekton Pipelines are… • … Cloud Native • run on Kubernetes • use containers as their building blocks • managed with Controllers • … decoupled • a pipeline can be reused in multiple clusters • tasks can be isolated and are reusable • resources can be swapped between runs © white duck GmbH 2020
- 18. Triggers, CLI &Dashboard • Tekton Triggers can be used to call a pipeline • push, issue, webhook, …. • Tekton also provides a CLI and Dashboard © white duck GmbH 2020
- 19. Demo • cloud-native pipelineswith Tekton Pipelines • https://gitlab.com/containerized-cicd/tekton-sample © white duck GmbH 2020
- 20. Docker rate-limiting (sinceNov 2nd) • Free plan • anonymous users: 100 pulls per 6 hours • authenticated users: 200 pulls per 6 hours • Pro/Team plan – unlimited • free opt-in for OSS projects (unlimited pulls) • was introduced slowly • starting with 6000 pull per 6 hours • final limits are active since Nov 18 © white duck GmbH 2020
- 21. Solutions • authenticate oropt-in for Pro/Team • Docker config, imagePullSecrets, … • configure a registry mirror • run your own • https://docs.docker.com/registry/recipes/mirror/ • use mirror.gcr.io • https://cloud.google.com/container-registry/docs/pulling-cached-images © white duck GmbH 2020
- 22. GitLab, Kaniko, Tekton •GitLab CI Runner • SaaS: is using mirror.gcr.io • Self-hosted: configure a registry mirror • Kaniko • use --registry-mirror to define registry mirror • Tekton • configure imagePullSecrets • configure a registry mirror © white duck GmbH 2020
- 23. Questions? Slides: https://www.slideshare.net/nmeisenzahl Demos: https://gitlab.com/containerized-cicd NicoMeisenzahl (Senior Cloud & DevOps Consultant) Phone: +49 8031 230159 0 Email: nico.meisenzahl@whiteduck.de Twitter: @nmeisenzahl LinkedIn: https://www.linkedin.com/in/nicomeisenzahl Blog: https://meisenzahl.org © white duck GmbH 2020