Reto Bollinger talks about opportunities to create more security through #GitOps. A deployment that is defined by a yaml file allows the administrator to restrict debugging access to read-only. Fixes are not done manually through SSH but with an adaption of the yaml file which is pushed to Git. GitOps means, everything is declared, we do not depend on external states but declare these states and we change only the declaration of instances but do not work on actual instances. Yaml files are used to declare the deployment, the CI/CD pipelines and the monitoring system. Any changes should be incremental and tested. Public repositories should be mirrored if you require long lasting states. From the perspective of confidentiality, it is still possible to accidentally misconfigure i.e. permissions. But it is documented and traceable. Using Git, merge requests can be configured to require approval (4 eyes principle) and they can be checked for suspicious patterns. Integrity can’t be broken since everything is declarative. From the perspective of availability it either runs or it doesn’t. If it doesn’t revert to its previous state, rolling out quick-fixes definitively takes longer but changes and fixes are of better quality.

1. GitOps and security
Zürich, 2020-03-12

2. Zürich,Reto Bollinger / public 2019-02-05
iso@nine:~$
Name: Reto Bollinger
Job Title: Chief Information Security Officer @ Nine Internet Solutions AG
Tasks: maintain ISMS (ISO27001) and QMS (ISO9001), good cop/bad cop in one
Attributes: slightly paranoid ;) and a bit reclusive but overly communicative...
iso@nine:~$ wiso@nine:~$ whiso@nine:~$ whoiso@nine:~$ whoaiso@nine:~$ whoamiso@nine:~$ whoami

3. Zürich,
C I A
What is information security?
Confidentiality Integrity Availability

4. Zürich,
Ops / DevOps / DevSecOps
and confidentiality
Hey, I need to quickly access that data
Sure, * I will adjust permissions quickly and revert them after
*) add here one of the following:
a) <nothing>
b) let me just disable config mgmt and
c) let me just get approval for disabling config mgmt and disable config mgmt and

5. Zürich,
Ops / DevOps / DevSecOps
and confidentiality

6. Zürich,
Ops / DevOps / DevSecOps
and confidentiality
… 12 months later:
Hey! Why is that accessible to the world?
Who the f*** messed with the permissions? And why?
Let’s check logs!
Damn! Logs only go back 6 months, must have happened before...

7. Zürich,
Ops / DevOps / DevSecOps
and integrity
Hey I quickly need to check these log files
What ugly format is that? Let me strip unnecessary stuff
<insert some fancy in-place sed/awk/grep command here
recursively going through subdirs>
Oops the config files were affected too?
Wait, what do you mean they were not revision controlled?!?

8. Zürich,
Ops / DevOps / DevSecOps
and availability
Let’s roll out that brand new version of the app
Oh man, so many dependencies? Luckily package manager resolves it all
Oops it breaks everything!
Wait, what do you mean we can’t roll back because of dependencies?

9. Zürich,
Sounds familiar?
Who has experienced one or the other of those scenarios or something similar?

10. Zürich,
And now for something completely different!

11. Zürich,
Firmware
A story from my former life...
App
Libs
OS
Drivers
Buildsystem
(CI/CD) Config
Packages
OS
Builds loaded onto
Binary Repo Sourcecode Repo 1 Sourcecode Repo 2
Assume you have the following (slightly simplified):
How likely are you able to build a 1:1 identical firmware in 5 years from now?

12. Zürich,
GitOps to the rescue!
● We have lots of dependencies
● We have multiple repositories with their own histories and revisions
● We have additional configurations
● We probably even have data that relies on specific revision of SW (databases)
-> We have to track the whole state of all these.
Describe the state of each repository, configuration and whatever and track this
combination in a repository as well

13. Zürich,
Deployment
Let’s get back to the cloud
Service 1 Service 2 Service 3
yaml
Defined by

14. Zürich,
Deployment
Service 2
Dealing with service disruptions/changes
Service 1 Service 3
Monitoring
DISRUPTION!
ssh
Ooops it failed!
Let’s SSH into it and “fix” it
(e.g. chmod a+rw *.*)
oh and silence that alert

15. Zürich,
Deployment
Service 2
Dealing with service disruptions/changes
Service 1 Service 3
Monitoring
Service 2 is now insecure.
Alerting is completely silenced (for that case)
Any update to Service 2 will make the problem re-occur or break even more

16. Zürich,
Deployment
Let’s go the GitOps way
CI/CD
Service 1 Service 2 Service 3
MonitoringBuilds
yaml
Defined by

17. Zürich,
Deployment
Service 2
Let’s go the GitOps way
CI/CD
Service 1 Service 3
MonitoringBuilds
yaml
Defined by
DISRUPTION!
ssh
Ooops it failed!
Let’s SSH into it and “fix” it
No No!
No touchy touchy!
Only looky looky!
With GitOps you
should have
“read-only” access to
your instances
X

18. Zürich,
Deployment
Service 2
Let’s go the GitOps way
CI/CD
Service 1 Service 3
MonitoringBuilds
Defined by
DISRUPTION!
vi
git
Ooops it failed!
Let’s SSH into it and “fix” it
Let’s fix the declaration and push it
yaml

19. Zürich,
Deployment
Let’s go the GitOps way
CI/CD
Service 1 Service 2 Service 3
MonitoringBuilds
yaml
Defined by
What GitOps means:
● Declare EVERYTHING
● Do not depend on external states but declare these states
● Work ONLY on declaration but not on actual instances (looking is allowed)

20. Zürich,
Deployment
Let’s go the GitOps way
CI/CD
Service 1 Service 2 Service 3
Configuration
MonitoringBuilds
yaml
Defined by
yaml

21. Zürich,
Deployment
CI/CD
Service 1
Configuration
Service 2 Service 3
Configuration
Monitoring
yaml
Defined by
yaml
yaml
Builds
Let’s go GitOps all the way
● Declare everything
● Think about how to bootstrap
● Test incremental changes as well as full rebuilds
● Keep a mirror of public repositories
● Use unique identifiers (tags most often are not necessarily unique)

22. Zürich,
So now:
Why is GitOps good from security perspective?
Confidentiality: One can still accidentially misconfigure permissions
BUT: its documented and traceable
merge requests require approval (4 eyes principle)
merge requests could be checked for suspicious patterns
Direct access to instances can be restricted
Integrity: One can no longer create inconsistency: it’s all declarative
Availability: Either it runs or it doesn’t: if it doesn’t revert to previous state
Rolling out quick-fixes definitely takes longer
...but changes/fixes are of better quality

23. Zürich,
What makes a good GitOps project
from security perspective?
● No (or read-only) “backend” access to running instances
● Automated testing: test everything: new way to break it? -> new test!
● Automated testing: incremental (update) as well as bootstrap (from scratch)
● Automated testing: find edge cases and robustify with chaos engineering*
● Merge requests (4+ eyes principle)
● Have a well defined scope: what is declared? Where are your boundaries?
(not only “outwards” also “inwards”: customers realm)
*) Dan Acristini held a very good talk about it: https://www.nine.ch/de/blog/gcp-meetup-ein-rueckblick-3

24. Zürich, 2019-11-21Intern / Reto Bollinger / ISM-QMS-intro
“Tha ’ al folk !”

25. Zürich, 2019-11-21Intern / Reto Bollinger / ISM-QMS-intro
“An Question ?”