Presented at CNCF Toronto Q2 meetup on June 22, 2023. You’ve seen the light of GitOps and are excited to streamline the release process of your development teams and platform engineers. Ready to proceed down this path, the realities of Day 2 operations are settling in. Does a perfect repository, folder/branching structure, and templating process exist? Who must approve which pull requests? How involved will the CI pipeline be? Oops!-Do you have a rollback process in place? All of these requirements must be met while still ensuring security and compliance is kept in check. This talk will run over several different GitOps recipes and discuss relevant security and compliance considerations. You will leave this talk with your eyes once again glittering with excitement to travel down the GitOps path armed with an swiss army knife of recipes and how they will fit into your organization, all while still maintaining a high security and compliance posture.

1. GitOps ⚙
Recipes 🍱 With a
Spice 🌶 of
Security 🔐
Julian Mazzitelli - @thejmazz, CIO
BioBox(.io)

2. What is GitOps?
https://www.weave.works/blog/automate-kubernetes-with-gitops
https://www.weave.works/technologies/gitops/

3. How ArgoCD Performs the GitOps Reconciliation Loop

4. Monorepo or Polyrepo?
● Simplicity vs. Scale
● 🌶 Branch Protections and PR Approvals 🔜

5. Image Updater
● FluxCD has this built in
● argocd-image-updater is v0.12.2, works for
Kustomize and Helm
● Changes image tags in manifests when new
images are pushed
● Last resort: write your own git bot script! Easy to
replace values in yaml with dasel

6. Image Updates at BioBox
● Monorepo
● Pushes build images, write back from a bot into
Git, updating images in bx.application.yaml
● PRs with “env” label, kubectl apply current
bx.application.yaml for dev/QA environments
● Prod release updates/hotfixes are made manually
via PR on deployments repo
● Beyond images: other configuration values,
database migration targets

7. Branches or Directories?
(for environments - base vs dev vs staging vs prod)

8. Branches or Directories?
(for environments - base vs dev vs staging vs prod)
Directories.

9. Branches or Directories?
(for environments - base vs dev vs staging vs prod)
Directories. Nobody likes purple ketchup.

10. Branches or Directories?
(for environments - base vs dev vs staging vs prod)
Directories. Nobody likes purple ketchup, especially Git.

11. App of Apps vs. Giant Application
● Application: A single repo+path+revision watched by the
GitOps controller
● A single application may start off simple, but will grow
complex quickly!
○ Every resource must be checked each sync (though ArgoCD has an
option to only sync out-of-sync resources)
○ Overwhelming amount of resources in dashboard, combining
unrelated resources
○ Limits use of hooks (PreSync, PostSync)
● So, app-of-apps? “Deployments” repo consists purely of
Application.argoproj.io CRDs, who reference the
“templates” repo
● 🚧 Beware of multi-cluster complications
○ E.g. “templates” render out Applications
○ Central GitOps -> app-of-apps needs to have children destination
in-cluster (cannot mix ad-hoc K8s resources with children apps!)
○ GitOps-per-cluster -> Could mix applications with resources

12. Render Templates - Ahead of Time or Live?
● Should your repo contain un-rendered Helm/Kustomize/x templates, or should
you render everything into yaml?
Ahead Live
✅ No surprises, review resources as they will end
up
✅ 100% Declarative
❌ More copy-pasting, management of your
templates (Kustomize overlays can help, so much)
✅🌶 Ability to run thorough CI checks (kubelinter -
e.g. enforce no root, Security is 😁)
🔸 Can make monorepo tamable?
(name.resource.namespace.yaml)
✅ Better diffing (can sort keys deterministically)
✅ Ability to provide “last-mile” configurations (e.g.
requests, replicas)
❌ Only “semi” declarative
✅ Simpler, up and running faster
❌ Limited CI (unless you render out in CI?)
● Use both techniques! Depending on situation: semi-declarative on dev (+flexibility), full
declarative on prod. Platform, database fully declarative, stateless apps semi-declarative

13. 🌶 Compliance
● Polyrepo
○ Assign specific teams to specific repositories
○ Can probably get away with branch protections requiring one review
○ ✅ Simpler change management controls
○ 🔐 Map repository teams directly to Kubernetes RBAC
○ ❌ Polyrepo management and complexity
● Monorepo
○ Developers and Operations belong to the same repo - how to avoid stepping on each other’s
toes?
○ Write a CI script that checks for reviews from specific individuals based on contents of change
■ No ingress, platform, storage class changes? Allow developer to approve, otherwise
operations must approve.
🔏 “Provide the list of users who can view/edit/delete the
in-scope production applications”

14. 🚧 Multi-Git Source Applications!? 🔧

15. Thank you!
Julian Mazzitelli
@thejmazz
BioBox.io