The document discusses security within Cisco Container Platform. It provides an overview of the security model and features, including platform hardening through the Cisco Secure Development Lifecycle process, role-based access control for Kubernetes, and secure multi-tenancy capabilities in Kubernetes clusters. It also covers container and Kubernetes security best practices like encryption, authentication, and network policies that are supported in Cisco Container Platform. The presentation concludes with a demo of secure multi-tenancy in Kubernetes clusters.

1. #CLUS

2. #CLUS
Sanjeev Rampal
Principal Engineer, Cloud Platforms BU
BRKCLD-2011
A comprehensive look
at security within the
Cisco Container
Platform

3. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Agenda
• Introduction to Cisco Container Platform
• Security Model, Agile delivery, Sample Topology
• Platform Hardening & Cisco Secure Development
• Kubernetes & Container Security
• Kubernetes Secure Multi-tenancy
• Demo
BRKCLD-2011 3

4. WEBEX TEAMS
DOCUMENTS
SPEAKER 2
SPEAKER 1
cs.co/ciscolivebot#
Questions?
Use Cisco Webex Teams to chat
with the speaker after the session
Find this session in the Cisco Live Mobile App
Click “Join the Discussion”
Install Webex Teams or go directly to the team space
Enter messages/questions in the team space
How
Webex Teams will be moderated
by the speaker until June 16, 2019.
1
2
3
4
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Cisco Webex Teams
BRKCLD-2011
4

5. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Can a look at security ever be “comprehensive” ?
BRKCLD-2011 5

6. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Control Plane Data Plane
VM VM
Control Plane Kubernetes
Automation
Orchestration
Operations
HX Connect Cluster/
Machine
Controllers
VM VM VM
Cluster 1 Kubernetes
Cluster1
Workloads
Cluster1
Ops
Pod
Pod
Pod
VM VM VM
Cluster 2 Kubernetes
Cluster2
Workloads
Cluster2
Ops
Pod
Pod
Pod
Kubernetes Fluentd Prometheus Kibana Hyperflex Contiv
Storage (HyperFlex / VMware)
Networking (Nexus 9K)
Compute Hardware (UCS)
Hypervisor Layer (HyperFlex / VMware)
Cisco Container Platform Architecture
VM VM
Istio
BRKCLD-2011 6

7. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Ops Personas & Logical production layout
Tenant cluster 1
Devops admin/ Dev
K8s api, RBAC
K8s data plane
Tenant cluster 2
Devops Admin/ Dev
K8s api, RBAC
K8s data plane
CCP Admin (IT Ops)
CCP api, RBAC
Full cluster & services
life-cycle mgmt
“Immutable” infra
Ubuntu
K8s
Add-ons
Ubuntu
K8s
Add-onsUbuntu
K8s
CCP app
CCP admin
Web based
Installer VM
BRKCLD-2011 7

8. Security Model,
Agile delivery,
Sample Topology

9. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Software Layering & CCP Security Scope
Hypervisor, Virtualization infra
e.g. vSphere
VMs, Instances, Node OS
Kubernetes, Docker, Container
infra plugins
CCP Application Addons
Physical Compute, Network,
Storage
Hypervisor, Virtualization infra
e.g. vSphere
VMs, Instances, Node OS
Kubernetes, Docker, Container
infra plugins
End-user
Applications
Addons
Physical Compute, Network,
Storage
CCP
packaging &
Security
responsibility
Physical Infra
separate
setup +
responsibility
End-user Application
responsibility
Control cluster Tenant cluster
BRKCLD-2011 9

10. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Agile Delivery & Immutable infra based model
• Immutable Infrastructure
• Integrated provisioning and full lifecycle management of of infrastructure (VMs, Node OS etc)
along with Kubernetes, container infra
• No additional software patching or maintenance needed for Node OS
• Centralized upgrades + patching of combined infra => No configuration drift or snowflakes
• Continuous Release and Delivery
• Bi-weekly internal releases, Monthly external releases, patch releases asap when needed
=> Improved overall product security, predictability & quick
turnaround of security patches
BRKCLD-2011 10

11. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Sample secure deployment: Private stub network
11BRKCLD-2011
K8S cluster
K8S
Pod IPs
192.168.0.0/16
K8S
Service IPs
10.96.0.0/12
Cluster
Node IPs
Exposed k8s api
and application
IPs/ VIPs
vSphere
cluster
CCP
control
cluster
IP gateways
External
Routed n/w
Inbound
Proxy
Outbound
Proxy (optional)
Non-
containerized
Oracle DB (for
example)
Firewall
Private stub network
w/ RFC 1918 addressing
SNAT
IPAM

12. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Multi-cloud deployment: Cisco CP + AWS EKS
BRKCLD-2011 12

13. Platform
Hardening & Cisco
Secure
Development

14. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
CCP Platform Hardening incl CSDL
• Cisco Container Platform is developed using the comprehensive
security requirements defined in the Cisco Secure Development
Lifecycle (CSDL) process
• Curated Ubuntu OS from Canonical
• Cisco performs additional hardening of containers (internally
developed for CCP application as well as sourced from upstream)
• Frequent internal vulnerability scanning & fixing of every CCP
release using a mix of external vendor container security tools as
well as internal tooling
BRKCLD-2011 14

15. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Internal DevSecOps: Secure Development Pipeline
Validate IPC
Developed code
BOM created,
reviewed and
approved
BOM to IPC
(CCP Github)
Releases into
CCP CI file repo
To CCO
On-demand test deployment
Static registry
scan
Run-time test
Requirements
Input
+
Vulnerability alert
feeds
Release built
CCO
Ubuntu, K8S …
Container artifacts
Ex. Prometheus,
NGINX etc
CCP CI registry
Vulnerability
Scanning tools
BRKCLD-2011 15

16. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Additional platform hardening features
• TLS communication for http traffic (Encrypted data in motion
internal and external)
• Support for TLS 1.3 on CCP API/Dashboard
• Strong ciphers for internal encrypted data at rest
• ecdsa and ed25119 keys for ssh into cluster nodes
• Continuous monitoring of NVD and industry standard vulnerability
intelligence streams and rapid turnaround of patch releases
• Recent industry CVEs fixed & delivered rapidly on CCP
• Example: Critical k8s patch delivered in 2 weeks … CVE-2018-1002105: proxy
request handling in kube-apiserver can leave vulnerable TCP connections
BRKCLD-2011 16

17. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
CCP App Security: Role Based Access Control
BRKCLD-2011 17

18. Kubernetes &
Container Security

19. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Kubernetes Security is a Journey
From: J Jalava: Kubernetes Security Journey
BRKCLD-2011 19

20. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Tenant Kubernetes & Docker Security in CCP
K8S Security related features on CCP Kubernetes clusters
• K8S dashboard protection
• K8S Authentication
• K8S Authorization
• K8S Cert manager
• K8S Encrypted secrets
• Kubernetes Ingress with TLS/ https
• Istio Ingress gateway + Service mesh
• K8S Network policy
• Secure Multi-tenancy, Admission controllers, Pod Security Policies, AppArmor, Kata (future)
BRKCLD-2011 20

21. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
K8S dashboard protection
Kubernetes dashboard locked in CCP
"The hackers had infiltrated Tesla's Kubernetes console
which was not password protected," - ArsTechnica
BRKCLD-2011 21

22. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
K8S AuthN, AuthZ, Admission Control flow
BRKCLD-2011 22

23. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
K8S AuthN, AuthZ on CCP K8S tenant clusters
• K8S Authentication options:
• X.509 Client certificates <Suggested for simple deployments only>
• If team has AWS account, can use AWS IAM with on-prem CCP K8S
• Integrate 3rd party identity solutions e.g. Tremolo
• Direct Kubernetes OIDC-LDAP integration (future)
• K8S Authorization options:
• ABAC: Disabled on CCP K8S
• RBAC: Role Based Access Control; enabled by default on CCP K8S
• Authorization webhooks (Tech preview; full support in upcoming release)
• Open Policy Agent (Tech preview; full support in upcoming release)
BRKCLD-2011 23

24. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
AWS IAM Authentication for On-prem CCP K8S
BRKCLD-2011 24

25. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Kubernetes cert-manager & encrypted secrets
• Kubernetes Cert-manager:
• Kubernetes project to automate generation of X.509 certificates
• Used in CCP to generate certs for internal communication & external API
• Kubernetes Encrypted secrets:
• Kubernetes state in etcd can be encrypted using Kubernetes encrypted
secrets feature
• Note, this feature must be enabled via CCP api, not exposed to GUI yet
• Set etcd_encrypted=True to enable this capability per tenant k8s cluster
BRKCLD-2011 25

26. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Admission Controllers & Secure Multi-tenancy
• New feature (Tech preview in CCP v3.2)
• Setting secure_multitenancy_enabled to True enables
1. Multiple built-in k8s admission controllers on the new cluster:
• PodSecurityPolicy
• LimitRanger
• ResourceQuota
• ValidatingAdmissionWebhook
• MutatingAdmissionWebhook
2. Privileged & restricted pod security policies and associated
PodSecurity and RBAC policies and bindings (with AppArmor and
Seccomp based tenant and container isolation)
3. Privileged-tenant & restricted-tenant as sample tenants
BRKCLD-2011 26

27. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Kubernetes Network Policies Support
• Supported on CCP tenant clusters
for all 3 CNI options
• Network microsegmentation tool
within Application + across teams
• L3, L4 CNI network policies (ingress
and egress)
• Extra network policy options when
using ACI CNI
• L7 policies on K8S Ingress (Nginx)
and Istio (Envoy)
• E-W http traffic encryption w/ Istio
App-front
App-db
App-core1
App-front
App-metrics
BRKCLD-2011 27

28. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Technical Description
• Network policies of Kubernetes supported using standard
upstream format but enforced through OpFlex / OVS using
APIC Host Protection Profiles
• Kubernetes app configurations can be moved without
modification to/from ACI and non-ACI environments
• Standard K8S Container Network policies + (optional)
enhanced ACI container network policies
Kubernetes Network Policies on CCP+ACI CNI
Node
OpFlex OVS
Kubernetes
ACI Policies
Network Policy
Node
OpFlex OVS
BRKCLD-2011 28

29. Demo: Secure Multi-tenancy in CCP K8S

30. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
More Information
• https://www.cisco.com/c/en/us/products/cloud-systems-
management/container-platform/index.html
• https://www.cisco.com/c/en/us/support/cloud-systems-
management/container-platform/tsd-products-support-series-
home.html
• http://www.cisco.com/go/multicloud
• Webex space for this session cs.co.ciscolivebot#BRKCLD-2011
• Or contact/ follow: srampal@cisco.com @sr2357
BRKCLD-2011 30

31. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Conclusion: Cisco Container Platform
.. so that you won’t need to be this guy
BRKCLD-2011 31

32. Complete your
online session
evaluation
• Please complete your session survey
after each session. Your feedback
is very important.
• Complete a minimum of 4 session
surveys and the Overall Conference
survey (starting on Thursday) to
receive your Cisco Live water bottle.
• All surveys can be taken in the Cisco Live
Mobile App or by logging in to the Session
Catalog on ciscolive.cisco.com/us.
Cisco Live sessions will be available for viewing
on demand after the event at ciscolive.cisco.com.
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS BRKCLD-2011 32

33. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Continue your education
Related sessions
Walk-in labs
Demos in the
Cisco campus
Meet the engineer
1:1 meetings
BRKCLD-2011 33

34. Thank you
#CLUS

35. #CLUS

36. Backup content

37. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
K8S Security: Some key standards & initiatives
• CIS Docker Benchmark https://www.cisecurity.org/benchmark/docker/
• CIS Kubernetes Benchmark https://www.cisecurity.org/benchmark/kubernetes/
• NIST SP 800-190 https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-190.pdf
• AppArmor http://wiki.apparmor.net/index.php/Main_Page
• SELinux https://selinuxproject.org/page/Main_Page
• CRI-O https://github.com/cri-o/cri-o
• Kata containers https://katacontainers.io/
• And more …
• These are in addition to common infrastructure security & compliance related standards such as
Common Criteria, FIPS, PCI-DSS, GDPR, HIPAA
BRKCLD-2011 37

38. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Preconfigured Pod Security Policy: “restricted”
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name:restricted
annotations:
seccomp.security.alpha.kubernetes.io/allowedProfileNa
mes:'docker/default’
apparmor.security.beta.kubernetes.io/allowedProfileNam
es:'runtime/default’
seccomp.security.alpha.kubernetes.io/defaultProfileNam
e:'docker/default’
apparmor.security.beta.kubernetes.io/defaultProfileNam
e:'runtime/default’
spec:
privileged:false
allowPrivilegeEscalation:false
requiredDropCapabilities:
-ALL
volumes:
-'configMap’
-'emptyDir’
-'projected’
-'secret’
-'downwardAPI’
-'persistentVolumeClaim'
hostNetwork:false
hostIPC:false
hostPID:false
runAsUser:
rule:'MustRunAsNonRoot’
seLinux:
rule:'RunAsAny’
supplementalGroups:
rule:'MustRunAs’
ranges:
- min:1
max:65535
fsGroup:
rule:'MustRunAs’
ranges:
- min:1
max: 65535
readOnlyRootFilesystem:false
BRKCLD-2011 38

39. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Calico/ Contiv Overlay Container Networking
K8S master
nodes/ VMs 1..3
K8S compute
nodes/ VMs 1..M
VMWare VM Port group 100
Physical L3 gateways
Contiv
VXLAN overlays Non-contiv
VLAN traffic
K8S compute
nodes/ VMs 1..M
BRKCLD-2011 39

40. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
HX vSphere
Cluster CCP – CP1
K8S-Red
K8S-Blue
vCenter
PG10
PG20
PG30
10.1.1.0/28
10.1.2.0/24
10.1.3.0/24
ASR1K or any L3 GW
Leaf e.g. N93xx
Spine e.g. N95xx 100.1.x.x
DHCP server*
(for pre-3.0
releases)
Secure On-Premises Deployment Topology
BRKCLD-2011 40