Continuous Delivery helps to keep your software and Docker images updated and deploy new versions in production easily. Microservices are great at reducing the attack vector and limiting the privileges or credentials access of each piece of your application. Containers provide an opportunity to implement better security, small, immutable, single process and purpose. In this session, we will discover real use case examples on how to make your CI/CD pipeline interact with Docker security tools. But security doesn’t stop where your deployment pipeline ends. How can we prepare for 0-days and policy violations that happen at run-time? Can we make it part of the CI/CD process?

1. CI / CD / CS

2. About me

3. Continuous Integration.

5. Continuous Delivery.

6. Tools to deploy automatically apps into Kubernetes:

8. Continuous Security.

9. - New infrastructure, new layers, new risks
- But we have seen them before:
- DDoS, isolation break-out, injections
- Fast pipeline: skip security?
- This is an opportunity
- More steps: more security onion layers
https://sysdig.com/blog/7-docker-security-vulnerabilities/
Are containers secure?.

10. How to do security?.
- Establish trust boundaries (dev vs prod)
- Identify, minimise, and harden attack
surfaces
- Reduce scope and access
- Layer protections and defenses (secure and
updates)
- Traceability and test

11. Continuous Security.
- DevOps: agile and faster
- Security team: less incidents
DevSecOps!
- Modify process to bring security, agile
- Security as Code
- Failure: open an issue vs break the build

12. Security pipeline.
- Build
- Shipment
- Run-time

13. Security pipeline: Build.
- Code analysis
- specific vulnerabilities
- licensing
- style
- branch policies: PR, check, merge
- Test Driven Security (TDS)
- OWASP ZAP Scanning

14. Security pipeline: Build.
- Container build
- Trusted base image
- Restrict functionality
- Restrict libraries / dependencies
- Multi-stage builds
- Restrict privileges
- root, privileged, host, mounts
- Dockerfile: USER

15. Security pipeline: Build.
- Container scan
- At the CI -> registry step
- Find known vulnerabilities: inventory
- CoreOS Red Hat Clair
- Anchore (integrated in Sysdig Secure soon)
- Red Hat OpenScap
- Vuls.io
- Other commercial vendors

17. What we are looking for?.
- Package lists
- Software installed manually (pip, rake, …)
- Static binaries
- Hashes of known vulnerabilities
- Lost credentials
- Docker image layers

18. Ubuntu: 14.04
Apache: 2.2
Wordpress: 4.6 PHP: 7.0

19. Ubuntu: 14.04
Apache: 2.2
Wordpress: 4.6 PHP: 7.0

20. Ubuntu: 14.04
Apache: 2.2
Wordpress: 4.6 PHP: 7.0

21. How to bring this into the pipeline?.
- CI, build the image
- Scan:
- https://github.com/optiopay/klar
- https://gitlab.com/gitlab-org/clair-scanner
- https://wiki.jenkins.io/display/JENKINS/Anchore+C
ontainer+Image+Scanner+Plugin
- https://github.com/jenkinsci/anchore-container-sc
anner-plugin
- Push into the registry

22. Container registries.
- Often they already include scanning:
- DockerHub
- Quay.io
- GCR
- ECR
- ACR
- many!

23. Security pipeline.
- Build
- Shipment
- Run-time

24. Security pipeline: Shipment.
- Trust
- enable image signing
- DOCKER_CONTENT_TRUST=1
- Restrict
- registry auth and CI/CD tools
- Kubernetes ValidatingAdmissionWebhook
- Google Grafeas

26. Security pipeline.
- Build
- Shipment
- Run-time

27. Security pipeline: Run-time.
- Infrastructure security configuration:
- Host security
- Docker Engine security
- Kubernetes security
- Docker CIS benchmark: docker-bench
- Kubernetes CIS benchmark: kube-bench

28. Security pipeline: Kubernetes.
- RBAC
- namespaces
- Subjects: users and serviceAccounts
- resources
- Role and ClusterRole
- verbs: LIST, WATCH, GET, UPDATE, PATCH, DELETE
- RoleBindings and ClusterRoleBindings
https://sysdig.com/blog/kubernetes-security-rbac-tls/

29. kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: default
name: pod-reader
rules:
- apiGroups: [""] # "" indicates the core API group
resources: ["pods"]
verbs: ["get", "watch", "list"]
# This role binding allows "jane" to read pods in the "default" namespace.
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: read-pods
namespace: default
subjects:
- kind: User
name: jane
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: pod-reader
apiGroup: rbac.authorization.k8s.io

30. Security pipeline: Kubernetes.
- Admission controllers: PodSecurityPolicy
- Privileged/hostPID/hostIPC/hostNetwork/
hostPorts
- runAsUser
- volumes/allowedHostPaths/ReadOnlyRootFi
lesystem
- Capabilities
- SELinux/AppArmor/seccomp
https://sysdig.com/blog/kubernetes-security-psp-network-policy/

31. Security pipeline: Kubernetes.
- Admission controllers
- DenyEscalatingExec
- NodeRestriction
- PodSecurityPolicy
- ValidatingAdmissionWebhooks

32. Security pipeline: Kubernetes.
- Resource management
resources:
requests:
memory: 512Mi
limits:
memory: 700Mi
- Network Policies: Cilium, Calico, iptables
- Audit system
- TLS everywhere
https://sysdig.com/blog/kubernetes-security-harden-kube-system/

33. Security pipeline: Run-time scanning.
- Threat detection
- Network inspection
- Privilege escalation
- Post-mortem analysis and forensics
- Attacks are multiple steps
- Successful attack those are unknown to us
- But we just need to recognize one IoC

34. Seccomp.
- Application syscall sandboxing
- Create filter (BPF) with allowed syscalls
- Failures-> log message, error return,
and/or kill process
- Docker runs containerized process under a
seccomp profile
- Notable disallowed syscalls:
- clone (creating new namespaces)
- reboot (reboot the host)
- setns (change namespaces)

35. MAC: SELinux / AppArmor.
- Kernel-level interception/filtering
- features++ && complexity++
- Higher level:
- Actors (process)
- Actions (read/write on files/sockets)
- Targets (files, IPs, ports)
https://sysdig.com/blog/selinux-seccomp-falco-technical-discussion/

36. - An open-source behavioral activity monitor
- Detects suspicious activity defined by a
set of rules
- Uses Sysdig’s flexible and powerful
filtering expressions
- With full support for
containers/orchestration
sysdig.com/opensource/falco/

38. - macro: proc_is_new
condition: proc.duration <= 5000000000
- rule: Read secret file after startup
desc: >
an attempt to read any secret file (e.g. files containing user/password/authentication
information) Processes might read these files at startup, but not afterwards.
condition: fd.name startswith /etc/secrets and open_read and not proc_is_new
output: >
Sensitive file opened for reading after startup (user=%user.name
command=%proc.cmdline file=%fd.name)
priority: WARNING
Falco real rule example.

39. Incident response.

40. Post-mortem and forensics.
What?
Where?
Who?
Why?
logs? SSH into prod? and start messing around?

41. How we did this in the past?.

42. System call tracing

43. System calls for forensics?

44. Sysdig Inspect
https://github.com/draios/sysdig-inspect (integrated in Sysdig Secure too)

45. CI/CD/CS, closing the security gap
- Build
- Shipment
- Run-time
- Forensics

46. Thank you!