How to continuously improve security in software development and software operations by proactive collaboration, robust processes and readily available tooling to make sure the "paved path" (the path of least resistance) for developers is the correct/secure/supported path. Talk held at the Security Chat on Mar 25th 2019 in Zürich, Switzerland

1. VSHN - The DevOps Company
Continuous Security
improvement in the
DevOps process
Aarno Aukia, CTO @ VSHN - The DevOps Company

2. VSHN - The DevOps Company
● About Aarno & VSHN.ch
● From Ops to DevOps
● DevOps/DevSecOps/SecOps?
● Automating Operations to include security
○ Build
○ Test
○ Deployment
○ Ops
● IT Governance benefits
22
Agenda

3. VSHN - The DevOps Company
@aarnoaukia http://about.me/aarno aarno.aukia@vshn.ch
ETH → Google → Atrila → VSHN
VSHN - The DevOps Company
Since 2014, currently 35 VSHNeers in Zürich, Switzerland
Helping Developers run applications on any infrastructure making both visitors
happy with stability and developers happy with agility
33
About Aarno & VSHN.ch

4. VSHN - The DevOps Company 4
OPS = Firefighting-as-a-Service ?
4

5. VSHN - The DevOps Company
Capability Maturity Model Integration (CMMI)
55
Operations
2014
How to get to
this level?

6. VSHN - The DevOps Company
DevOps: CMMI Level 5:
People, Processes & Tools
66

7. VSHN - The DevOps Company
● Developer education, requirements engineering, design review -> AppSec
● Software Build/Deployment/Operations -> DevSecOps
● Incident detection & management -> SecOps
77
Areas of security improvement

8. VSHN - The DevOps Company
● Developer education, requirements engineering, design review -> AppSec
● Software Build/Deployment/Operations -> DevSecOps
● Incident detection & management -> SecOps
88
Areas of security improvement

9. VSHN - The DevOps Company
DevSecOps principles
99

10. VSHN - The DevOps Company
● static code analysis automatically for each commit
● Dependency Management
● (base) container image scanning
1010
Build

11. VSHN - The DevOps Company
Code analysis: sonarqube
1111

12. VSHN - The DevOps Company 1212
Dependency updates: https://dependabot.com

13. VSHN - The DevOps Company
Container scanning: aquasec
1313

14. VSHN - The DevOps Company
● smoke tests
● test envs “à discretion”
1414
Test

15. VSHN - The DevOps Company
● atomic container deployment
● every deployment (and rollback) is a “normal deployment”
● deployment automation removes need for (all) devs root prod access and/or
waiting for ops to deploy new dev version
1515
Deployment

16. VSHN - The DevOps Company
● standardization on (minimal, hardened) OS and container orchestrator
● immutable (application) infrastructure using containers
● process/storage/network separation of applications/environments
● detect/prevent configuration drift between dev/test/stage/prod envs
● documentation & automatic backup of all volumes
● documentation & monitoring of routes/loadbalancers/ingresspoints with
enforcing SSL/TLS
● AAI for admin & application
● key & secrets management
● audit logging of control & application planes
1616
Ops

17. VSHN - The DevOps Company
Container isolation
1717
● Kernel namespacing (process & network)
● Control groups (resource quota to prevent DoS)
● SELinux (additional syscall filter)
● prevent running as root inside container, no user-provided privileged
containers (enforce best practice)
● readonly container filesystem (harder to persist exploit at runtime)

18. VSHN - The DevOps Company
AAI: Keycloak
1818
● Identity & Access Management
● Single sign in/out
● Identity brokering:
○ OpenID Connect (OAuth2, FB/Twitter/Github etc.)
○ SAML2.0
○ Kerberos
● User federation: LDAP, AD, etc
● 2FA: TOTP/HOTP
● Managing the Authorization groups

19. VSHN - The DevOps Company
Logs: ELK/EFK/Greylog
1919
● Logging all access and changes through the control plane
● Logging all access to the application and correlate with application logs
● Index, view, filter, aggregate KPI → monitoring
● Store outside of application scope

20. VSHN - The DevOps Company
● Prometheus
○ time series database
○ open source / CNCF-project
○ well-integrated in docker/kubernetes stats
● NewRelic APM
○ application-level profiling
○ performance tracking
○ exception tracking (backend & frontend)
○ available as SaaS
2020
Metrics: Prometheus / NewRelic

21. VSHN - The DevOps Company
● “Full Stack Audit”
● Review design document
● Every layer was custom built
○ physical hardware
○ handcrafted servers
○ manual application deployment
● Review each layer
● Review each layer again next year...
2121
Traditional IT governance

22. VSHN - The DevOps Company
● Standardized components
○ already audited, some even externally certified
○ re-used, economies of scale, CMMI level 5
○ tech controls (AAI, RBAC, logs/SIEM) implemented once
○ financial controls implemented once
● Infrastructure: private/public cloud
● Ops: Container orchestration platform
● Review design document & platform
configuration
2222
Cloud native IT governance

23. VSHN - The DevOps Company
Docker
Kubernetes
2323
Layers of abstraction
Hardware
Operating System
Service discovery & Load
balancing
Application Server
Application
Cloud/Onprem

24. VSHN - The DevOps Company
● Red Hat OpenShift
● Rancher RKE
● Canonical
● Docker Datacenter Enterprise
● IBM cloud private
● EKS, AKS, GKE
● APPUiO.ch
See also https://thenewstack.io/find-perfect-kubernetes-distribution/
2424
Kubernetes Distributions

25. VSHN - The DevOps Company
● Free & open standard
● Adopted by all major vendors (Google, AWS, MS, Redhat, Suse, IBM, etc)
● available as managed service both on-premises and (private) cloud based
● Provides integration in infrastructure (compute, storage, networking)
● Provides optional integration in plattform (e.g. DBaaS, S3) services
● Infrastructure as code, automation, tools for DevOps processes
● Large ecosystem of auxiliary tooling & integration available
● Is being adopted as standard runtime by ISVs (Avaloq, Finnova, Abacus,
Adcubum, Ergon, etc)
2525
Benefits of Kubernetes as abstraction

26. VSHN - The DevOps Company
● prevent configuration drift
○ immutable (application) infrastructure using containers
○ deploy dev/test/stage/prod envs from CI/CD
● prevent manual errors
○ validate configuration in CI/CD before deployment
○ standardization on (minimal, hardened) OS and container orchestrator
○ deployment automation removes need for (most) root prod access
● security by default
○ image scanning, dependency vulnerability management
○ process/storage/network separation of applications/environments
○ volumes & ingresspoints best practice (documentation, monitoring, backup, SSL/TLS/WAF)
○ AAI for admin & application, audit trail logging of CI/CD, control & application planes
○ key & secrets management
● 2626
IT governance controls in container platforms

27. VSHN - The DevOps Company
● Please do get in touch with feedback
● Twitter: @aarnoaukia
● Linkedin: https://www.linkedin.com/in/aukia/
● Email: aarno.aukia@vshn.ch
2727
Thank you

28. VSHN - The DevOps Company
The CNCF Landscape
2828

29. VSHN - The DevOps Company
Next Event
May 9, 2019 from 6.00pm
https://www.meetup.com/Cloud-Native-Computing-Switzerland
Please volunteer for Sponsoring & Talks
https://cnc-meetup.ch
2929
Cloud Native Computing

30. Come visit us for a coffee!
VSHN AG - Neugasse 10 - CH-8005 Zürich - +41 44 545 53 00 - https://vshn.ch/ - info@vshn.ch
https://vshn.ch/kontakt/
Follow us on Twitter!
@vshn_ch
30