Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Deploying Cloud Native Red Team Infrastructure with Kubernetes, Istio and Envoy


Published on

Deploying Cloud-Native Red Team Infrastructure with Kubernetes, Istio, and Envoy

Published in: Software
  • Be the first to comment

  • Be the first to like this

Deploying Cloud Native Red Team Infrastructure with Kubernetes, Istio and Envoy

  1. 1. Deploying Cloud Native Red Team Infrastructure with Kubernetes, Istio and Envoy
  2. 2. Who we are ◎Larry Suto - Larry is an independent security consultant based out of Oakland, CA. He spends a lot of time researching using cloud infrastructure for all types of security testing. He spends some time on Windows security as well. Twitter: @larrysuto 2
  3. 3. Who we are ◎Jeff Holden – Works at the largest college system in the united states. CISO by day, hacker by night. Currently living in an RV with 3 dogs, 1 wife, and 1 kid. ◎@jeffh
  4. 4. Goals ◎Working in AWS, GCE, Azure ◎Automated ○ Set domains, access keys, etc ◎Portable ◎Scriptable
  5. 5. Why Kubernetes? ◎Infrastructure as code ◎Portable ◎Speed ◎Efficiency
  6. 6. What is this Docker thing?
  7. 7. Docker
  8. 8. Kubernetes Open source container-orchestration system for automating application deployment, scaling, and management.
  9. 9. Creating a Kubernetes Deployment ◎Configmap/Secret ◎Deployment yaml ◎Service yaml ◎Helm chart (optional)
  10. 10. What is this Kops thing? • Takes care of the infrastructure grunt work • Creates Network • Creates Base VMs • Creates load balancer • Creates Access list • Supports • AWS • GCE
  11. 11. Kops ◎kops create cluster --zones us-west-2a,us- west-2b --topology private ◎--networking calico --master-size t3.micro - -master-count 3 --node-size ◎ t3.large --name <kube cluster name> ◎--state=s3://<Your S3 Bucket Name> --yes
  12. 12. External DNS ◎Support all major cloud services ◎ Pod adds and updates DNS entries to cloud providers DNS ○ Need to use cloud provider specific tool to create hosted zone ○ Can automate domain registration 12
  13. 13. SSL Cert Manager 13
  14. 14. SSL Cert Manager ◎ ◎Automates LetsEncrypt Certificate Management ◎TLS Passthrough 14
  15. 15. Background ◎Placing Apache mod_rewrite in front of C2 infrastructure has been common place for many years now ◎Many advances have been made in redirection technology recently driven by the move to container-based cloud systems such as Kubernetes ◎Adoption of Docker by the red team is common but we have seen little in the way of container management systems and service mesh technology which provide sophisticated mechanisms for ingress and egress manipulation ◎Service mesh technology holds a lot of promise for sophisticated c2 redirection and traffic manipulation and can leverage multiple clouds with many simultaneous ingress points ◎With Kubernetes its quite straight forward to introduce new redirection point dynamically
  16. 16. Recipes for Containerizing C2 ◎Candidates ○ Cobalt Strike ○ Merlin (http/2) ○ Many others (Empire, Faction, silver, Covenant,..)
  17. 17. Dockerizing Cobalt Strike docker build --build-arg cskey="$(cat license)" -t cobalt/ub18:1.0
  18. 18. FROM ubuntu:18.04 as base MAINTAINER kubered LABEL version=”1.0" LABEL description="CobaltStrike." ARG cskey ENV cs_key ${cskey} SHELL ["/bin/bash", "-c"] RUN apt-get update && apt-get install -y wget curl net-tools sudo software-properties-common apt-utils --no-install-recommends && apt install -y openjdk-11-jdk && update-java-alternatives -s java-1.11.0-openjdk-amd64 && rm -rf /var/lib/apt/lists/* RUN var=$(curl '' -XPOST -H 'Referer:' -H 'Content-Type: application/x-www-form-urlencoded' -H 'Origin:' -H 'Host:' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H 'Connection: keep-alive' -H 'Accept-Language: en-us' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_1) AppleWebKit/604.3.5 (KHTML, like Gecko) Version/11.0.1 Safari/604.3.5' --data "dlkey=$cs_key" | sed -n 's/.*href="([^"]*).*/1/p' | grep /downloads/ | cut -d '.' -f 1) && cd /opt && wget$var.tgz && tar xvf cobaltstrike-trial.tgz && cd cobaltstrike && echo $cs_key > ~/.cobaltstrike.license && ./update RUN apt-get -y clean && apt-get -y autoremove COPY cobalt-kube.profile /opt/cobaltstrike/profiles/ # set entry point WORKDIR "/opt/cobaltstrike" ENTRYPOINT ["./teamserver"]
  19. 19. Dockerizing Merlin FROM golang:stretch MAINTAINER @audibleblink RUN apt-get update && apt-get install -y git make RUN go get WORKDIR $GOPATH/src/ VOLUME ["data/temp"] EXPOSE 443 CMD ["go", "run", "cmd/merlinserver/main.go", "-i", ""]
  20. 20. Configmap and Secret kubectl create –f cobalt-config.yaml --- apiVersion: v1 kind: Secret metadata: name: cobalt-password data: password: cbs76654A Configmaps allow you modify application behavior without respinning an image apiVersion: v1 kind: ConfigMap metadata: name: {{ .Release.Name }}-{{ .Values.malConfig }} data: randomized.profile: |- {{ .Files.Get "randomized.profile" | indent 4 }}
  21. 21. Deployment yaml (Helm template) apiVersion: apps/v1 kind: Deployment metadata: name: {{ .Release.Name }}-deployment namespace: {{ .Values.default }} spec: selector: matchLabels: app: {{ .Release.Name }} replicas: 1 template: metadata: labels: app: {{ .Release.Name }} spec: securityContext: runAsUser: 0 imagePullSecrets: - name: {{ .Values.pullSecret }} containers: - name: {{ .Release.Name }} image: {{ .Values.image }}:{{ .Values.tag }} imagePullPolicy: IfNotPresent volumeMounts: - name: {{ .Release.Name }}-{{ .Values.malConfig }} mountPath: {{ .Values.malc2path }} subPath: {{ .Values.subpath }}
  22. 22. Deployment yaml cont ports: - containerPort: 80 - containerPort: 50050 - containerPort 443 env: - name: COBALT_PASSWD value: {{ .Values.password | quote }} - name: MY_POD_IP valueFrom: fieldRef: fieldPath: status.podIP - name: MAL_C2_PATH value: {{ .Values.malc2path | quote }} args: ["$(MY_POD_IP)", "$(COBALT_PASSWD)", $(MAL_C2_PATH)] volumes: - name: {{ .Release.Name }}-{{ .Values.malConfig }} configMap: name: {{ .Release.Name }}-{{ .Values.malConfig }}
  23. 23. Service Definition (teamserver mgmt internal) apiVersion: v1 kind: Service metadata: name: cobalt-console annotations: spec: ports: - port: 50050 protocol: TCP targetPort: 50050 selector: app: teamserver1-deployment type: LoadBalancer status: loadBalancer: ingress: - hostname:
  24. 24. Service Definition (Cobalt Strike Listener) apiVersion: v1 kind: Service metadata: name: cobalt-listener labels: app: team-server spec: ports: - port: 80 protocol: TCP selector: app: team-server
  25. 25. Mounting SSL Certificates to Containers pt 1 apiVersion: v1 data: merlin.key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2QUlCQURBTkJ na3Foa2lHOXcwQkFRRUZBQVNDQktZd2dnU2lBZ0VBQW9JQkFRQ1lrL 2hMaEMzalh2Y3kKUHY1VDdNcU1OMWR5STlQNVM5MlpUUllNT1VZb2 JiUXREeE1KbWxMd3g4c0owQURlWjVzTWRSQkYwWjJzNVBrMApHL3V 2d2c2c2JpSTFCaXVqaVBzdnRwWVpIaC9nZVdJUG5zS.... merlin.crt: S0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURHRENDQWdBQ 0NRRHJDajdxWHFhR1VqQU5CZ2txaGtpRzl3MEJBUXN…. kind: Secret metadata: name: merlin-ssl type: Opaque $ cat ssl.secret
  26. 26. Mounting SSL Certificates to Containers pt 2 spec: containers: - image: merlin name: merlin volumeMounts: - mountPath: "/opt/merlin/data/x509" name: merlin-ssl readOnly: true ports: - containerPort: 443 volumes: - name: merlin-ssl secret: secretName: merlin-ssl
  27. 27. Merlin Deployment (Helm)
  28. 28. Merlin C2 Console (Helm Deployment)
  29. 29. Silenttrinity Deployment (Helm)
  30. 30. Silenttrinity Console
  31. 31. Envoy Proxy ◎It can proxy any TCP protocol. ◎It can do SSL. Either direction. ◎Full support for HTTP/2 and can translate between HTTP/2 and HTTP/1.1 (either direction). ◎It has good flexibility around discovery and load balancing. ◎It’s a sidecar process, so it’s completely agnostic to you service implementation language
  32. 32. Envoy xDS API ◎Listener Discovery Service (LDS) - an API that allows Envoy to query what listeners should be exposed on this proxy ◎Route Discovery Service (RDS) - a part of the configuration for listeners that specifies which routes to use; this is a subset of LDS for when static and dynamic configuration should be used ◎Cluster Discovery Service (CDS) - an API that allows Envoy to discover what clusters and respective configuration for each cluster this proxy should have ◎Endpoint Discovery Service (EDS) - a part of the configuration for clusters that specifies which endpoints to use for a specific cluster; this is a subset of CDS ◎Secret Discovery Service (SDS) - an API used to distribute certificates ◎Aggregate Discovery Service (ADS) - a serialized stream of all the changes to the rest of the APIs; you can use this single API to get all of the changes in order
  33. 33. Envoy Docker Container and httpbin App
  34. 34. Envoy Sample Config
  35. 35. Running Envoy
  36. 36. Envoy Headers
  37. 37. Envoy Stats
  38. 38. Envoy Based Ingress Controllers ◎Gloo ◎Istio ◎We also will start providing examples for Traefik as it is a very simple and easy to deploy edge proxy written in golang.
  39. 39. Envoy based Gateways 39 ◎Gateway describes a load balancer operating at the edge of the mesh receiving incoming or outgoing HTTP/TCP connections.
  40. 40. Envoy based Virtual Services ◎A Virtual Service defines a set of traffic routing rules to apply when a host is addressed. A routing rule defines matching criteria for traffic of a specific protocol. ◎If the traffic is matched, then it is sent to a named destination service (or subset/version of it) defined in the registry. ◎Supports many protocols : http, https, http/2
  41. 41. gloo Envoy Gateway ◎Ingress Gateway (Envoy) ◎Autodiscovers Kubernetes services as Upstreams
  42. 42. gloo Routing Rules
  43. 43. gloo Gateway ◎Installs with Helm: ○ helm repo add gloo helm ○ helm install gloo/gloo --namespace my-namespace ◎Installs with default ports 80 and 443 but this can be customized with Helm ◎Or kubectl edit gateway-proxy-v2 -n gloo-system ◎Also can be configured as a pure TCP Proxy ○
  44. 44. gloo Upstreams
  45. 45. gloo VirtualService
  46. 46. gloo Setting up Server TLS kubectl create secret tls gateway-tls --key tls.key --cert tls.crt --namespace default glooctl create secret tls gateway-tls --certchain $CERT --privatekey $KEY glooctl edit virtualservice --name default --namespace gloo-system --ssl-secret-name gateway-tls --ssl-secret-namespace default
  47. 47. gloo Multiple TLS VirtualServices with SNI kubectl create secret tls teamserver2-cert --key tls.key --cert tls.crt --namespace default openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/" glooctl create virtualservice --name teamserver2 --domains glooctl edit virtualservice --name teamserver2 --namespace gloo-system --ssl-secret-name teamserver2-cert --ssl-secret-namespace default --ssl-sni-domains You will get errors because the upstream listener is serving https
  48. 48. gloo Setting up Client TLS (mTLS) glooctl edit upstream --name default-teamserver2-server-443 --namespace gloo- system --ssl-secret-name teamserver2-upstream-tls --ssl-secret-namespace default kubectl create secret tls teamserver2-upstream-tls --key key.pem --cert cert.pem --namespace default kubectl get upstream -n gloo-system default-teamserver2-server-443 - o yaml Create Upstream Secret (or use the one for the gateway) Update the Upstream with Secret Dump the Updated Upstream
  49. 49. gloo Updated Upstream with TLS
  50. 50. Istio Service Mesh Mixer Istio-AuthPilot Envoy Envoy SvcA SvcB Envoy Istio-Ingress (Gateway) Service A (C2) Service B http, https, http/2 http, https, http/2 http, https, http/2
  51. 51. Installing Istio ◎kubectl create namespace istio-system ◎Install CRD with kubectl apply ◎helm install install/kubernetes/helm/istio --name istio --namespace istio-system --values install/kubernetes/helm/istio/values-istio-demo.yaml ◎Inject the Istio/Envoy sidecar manually or using injection webhook ○ istioctl kube-inject -f samples/sleep/sleep.yaml | kubectl apply -f - ○ kubectl label namespace default istio-injection=enabled –overwrite ◉ Injection occurs when the pod restarts
  52. 52. Istio Ingress ◎kubectl get svc istio-ingressgateway -n istio-system ◎Handles http or tcp ingress ◎Overcomes weaknesses of Kubernetes Ingress ○ Kubernetes Ingress mostly focused on http/https ◎Generally auto-provisions a load balancer 52
  53. 53. Multiple Ingress (Redirectors) C2 Services Kubernetes cluster istioingressC glooGateway istiongressA istioingressB IstioegressA C2 Services
  54. 54. Modifying the Ingress Gateway 54 istio-ingressgateway: enabled: true … gateway. image: node-agent-k8s … labels: app: istio-ingressgateway istio: ingressgateway … ports: - port: 15020 targetPort: 15020 name: status-port - port: 80 targetPort: 80 name: http2 nodePort: 31380 - port: 443 name: https nodePort: 31390 - port: 50050 name: tcp nodePort: 31400 This file can be modified to add multiple ingress points: install/kubernetes/helm/istio/charts/gateways/values.yaml
  55. 55. Deploy your own Ingress Gateway 55 apiVersion: extensions/v1beta1 kind: Deployment metadata: name: istio-ingressgateway namespace: istio-system spec: replicas: 1 template: metadata: labels: app: istio-ingressgateway istio: ingressgateway visibility: internal # put a custom label here. annotations: "false" "" spec: ...
  56. 56. Istio Ingress Private Load Balancer kubectl get svc istio-ingressgateway -n istio-system -o yaml > private-ingressgateway.yaml • Add annotation:“true” • Change the name to anything: IE from istio-ingressgateway to private-ingressgateway • Change the app label to anything: IE istio-ingressgateway to private-ingressgateway • Change the istio label to anything: IE ingressgateway to private- ingressgateway • Change selector configuration section and update the app and istio label to match the values you defined in the metadata section • Remove all the nodePort values from the ports configuration so new ports can be allocated automatically
  57. 57. Istio Ingress Private Loadbalancer(cont)
  58. 58. Exposing Services via Istio-ingeress gateway ◎To expose a service using ingressgateway you have to create at least 2 objects ○ Gateway ○ Virtual Service
  59. 59. Istio Gateway apiVersion: kind: Gateway metadata: name: teamserver-gateway spec: selector: istio: ingressgateway # use istio default ingress gateway servers: - port: number: 80 name: http-system protocol: HTTP hosts: - "*" - port: number: 50050 # exposes teamserver admin port name: tcp-admin protocol: TCP hosts: - "*" - port: number: 443 name: https protocol: HTTPS tls: mode: PASSTHROUGH hosts: - "*"
  60. 60. Istio Virtual Service Example apiVersion: kind: VirtualService metadata: name: teamserver1-basic spec: hosts: - gateways: - teamserver-gateway tcp: - match: - port: 80 route: - destination: host: teamserver1-service port: number: 80
  61. 61. Virtual Service -Multiple Conditions 61 gateways: - teamserver-gateway http: - match: - headers: user-agent: regex: "Trident/7.0;srv:11.0" uri: prefix: "/zC" route: - destination: host: teamserver1-service - route: - destination: host:
  62. 62. StringMatch for HTTP Headers Field Type Description exact string (oneof ) exact string match prefix string (oneof ) prefix-based match regex string (oneof ) ECMAscript style regex-based match Case sensitive
  63. 63. Istio ingressgateway Tips ◎A selector is used by istio to select the ingressgateway. This is important when there are multiple ingressgateways ◎istio uses port naming in some routing logic ◎Port names are of the form protocol-suffix with grpc, http, http2, https, mongo, redis, tcp, tls or udp as the protocol.
  64. 64. Routing to External Destinations ◎Envoy passthrough to external services ○ On by default ○ global.outboundTrafficPolicy.mode option set to ALLOW_ANY ○ kubectl get configmap istio -n istio-system -o yaml | grep -o "mode: ALLOW_ANY"
  65. 65. Notes on Ingress Capabilities ◎A mesh can have any number of gateways, and multiple different implementations of the gateway can coexist ◎The Kubernetes Ingress API cannot express the routing needs of Istio. Kubernetes Ingress looks for a common intersection between different HTTP proxies. It only supports the most basic of HTTP routing. ◎Kubernetes Ingress itself does not support the TCP protocol. Kubernetes Ingress cannot be set up to configure an NGINX Ingress Controller for TCP load balancing – requires a special configmap ◎Istio Gateway has overcome the above shortcomings of Ingress by separating the L4-L6 configuration from the L7 configuration ◎Gateway is only used to configure L4-L6 functions ○ exposed ports, TLS configuration ◎A VirtualService is bound a Gateway in order to control inbound TCP and HTTP traffic
  66. 66. Istio Ingress Multiple SNI Hosts (Cobalt Strike) Beacon Teamserver1-service Istio Ingress
  67. 67. Istio Ingress Multiple SNI Hosts (cont)
  68. 68. TLS Ingress Gateway Non-Passthrough Create a Kubernetes secret to hold the server’s certificate and private key. Use kubectl to create the secret istio-ingressgateway-certs in namespace istio- system . The Istio gateway will load the secret automatically. create -n istio-system secret tls istio-ingressgateway-certs –key key.pem --cert cert.pem Create a Gateway and a VirtualService with the desired routing rules
  69. 69. Istio Gateway
  70. 70. Istio TLS VirtualService
  71. 71. Istio TLS Ingress Gateway Multiple Hosts Create the secret for the additional backend kubectl create -n istio-system secret tls istio-ingressgateway-teamserver2service-certs --key key.pem --cert .cert.pem To include a volume mounted from the new created secret, patch the istio-ingressgateway deployment: patch.json kubectl -n istio-system patch --type=json deploy istio-ingressgateway -p "$(cat patch.json)"
  72. 72. Traefik
  73. 73. Installing Traefik ◎ guides/crd-acme/ ◎Deploy Cluster Resources – CRDs ○ IngressRoute and the Middleware kinds ○ RBAC definitions ◎Traefik Service ◎Traefik Deployment ◎Traefik Ingress Routes
  74. 74. Traefik Services
  75. 75. Traefik Deployment
  76. 76. Traefik Entrypoints
  77. 77. Drone ◎Drone is a modern CI/CD platform built with containers as first class citizens. Pipelines are configured using a yaml file that you check-in to your git repository. ◎Can be deployed as a Docker container to a VM or a Kubernetes cluster ◎Integrates with git repository and can deploy pipelines to a Kubernetes cluster
  78. 78. Using Drone 78 pipeline: build: image: gradle:4.5.1-jdk9 commands: gradle --help docker: image: plugins/docker secrets: [ docker_username, docker_password ] repo: komljen/test-image tags: ${DRONE_COMMIT_SHA} kubectl: image: komljen/drone-kubectl-helm secrets: [ kubernetes_server, kubernetes_cert, kubernetes_token ] kubectl: "get pods" helm: image: komljen/drone-kubectl-helm secrets: [ kubernetes_server, kubernetes_cert, kubernetes_token ] helm: list notify: image: plugins/slack channel: drone-notification secrets: [ slack_webhook ]
  79. 79. Monitoring C2 with Prometheus ◎Open source metrics based monitoring system ◎Can be used to instrument application ◎Applications without instrumentation support can be monitored using exporters and other legacy methods ○ extract whitebox monitoring data from application logs for collection in a timeseries database ○ Google mtail ○ Grok Exporter ○ JMX (Cobalt Strike teamserver is a Java application so theoretically it could be supported) ◎C2 authors and developers can instrument code to provide fine grained monitoring support
  80. 80. Google mtail ◎Can be used to create timeseries data from standard logs ○ ◎Prometheus exporter is available ◎Cobalt Strike teamserver logs are in the Cobalt Strike working directory: ○ Ie /opt/cobaltstrike/logs ○ Weblogs and beacon session logs are available ◎Use mtail to create counters by matching events in log with regular expressions
  81. 81. Grok Exporter ◎ ◎Can be used to convert arbitrary logs into Prometheus compatible time series data ◎An example would be the Cobalt Strike beacon log checkin entry ○ This interval can be collected as a metric to monitor beacon health ◎Grok Exporter uses same language as
  82. 82. Prometheus Visuals (with Sysdig)
  83. 83. References and Thanks ◎ ◎ management/ 83
  84. 84. Questions? ◎ 84