Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Securing Microservices Continuous
Delivery using Grafeas and Kritis
2
Vishal Banthia
Software Engineer,
Microservices Platform
3
● What is a secure software supply chain?
● Grafeas approach
● How Microservices Platform is using Grafeas and Kritis to...
4
In general, a secured software supply chain means that only
authorized softwares are executed at each supply chain stage...
5
Some policy examples:
● Only vulnerability scanned and passed docker images are allowed
● Only QA verified software arti...
Code DeployBuild Test
Compute
Engine
Kubernete
s
Engine
App
Engine
AWS
EC2
AWS
EC2
Software Supply Chain
7
In software supply chain stages, every stage consume some artifact
and produce a new one. “By authorized at each stage”,...
8
● Only commits from authorized authors are allowed for build stage
● Only unit / integrations tests passed artifacts are...
9
Traditionally, it is achieved by creating a process workflow or
orchestration where each stage will run only when previo...
Code DeployBuild Test
Compute
Engine
Kubernete
s
Engine
App
Engine
AWS
EC2
AWS
EC2
Software Supply Chain
tests will only t...
Software Supply Chain Governance
12
Governance in software supply chain means that all the software
artifacts deployed in production can be tracked. CTO/CI...
15
CTO / CIO need to make this big decision
● Do not make any change and just hope that we are running good
code 🤞
● Make ...
16
Take Away
● Now we understand what is a secure software supply chain
● Why centralized governance is required in softwa...
17
● Growing and fragmented toolsets
● Microservices Architecture
● Open-Source software adoption
What are the current pro...
18
Grafeas approach
“An open artifact metadata API to audit and govern your software
supply chain”
Basically, instead of j...
20
Grafeas Components
Metadata Server
metadata server which store all the metadata information generated during various
su...
build stage uses grafeas API
to check if commit is allowed
to build or not
deploy stage uses grafeas API
to check if artif...
22
Take away!
● Using grafeas, we can store metadata of all supply chain stages
in one centralised database
● Using policy...
How microservices platform team is using
Grafeas and Kritis to secure microservices
Continuous Delivery?
24
Microservices Platform Architecture
● We use GCP
● Each microservice has its own dedicated GCP project and they
are fre...
25
GCP project for GKE
Centralized cluster
Namespace: Service A
Namespace: Service B
IAM: Platform Team IAM: Team A
IAM: T...
PM will confirm if feature is
ready to release and ask
developers to create tag
QA will notify PM regarding
QA results
QA ...
PM will confirm if feature is
ready to release and ask
developers to create tag
QA will notify PM regarding
QA results
QA ...
For secure microservices Continuous Delivery, we want to make sure that
only authorized docker images get deployed to our ...
29
What is authorized docker image for us?
● Image is built by service owner
● Image has been verified by security team fo...
30
● We use GCP Container Analysis API which is implementation of
Grafeas
● We use Kritis fork which is basically Policy E...
Security Team
Signer Service
deploy new docker image in
echo-namespace with digest
(sha256:abcd…)
Cloud
Build
GCR
docker i...
32
How it works?
1. Cloudbuild build docker image and push it to GCR. It also create a
build metadata with information suc...
33
6. Kritis calls container analysis API and get metadata for that docker
image
7. Depending on the policy, Kritis valida...
34
Signer Service
Signer Services run independently and sign images based on their job.
For example, security signer servi...
35
Why FORK?
● Official kritis is still at very early stage and does not have feature
which we wanted such as:
○ Validate ...
36
Ending
Using Grafeas and Kritis, we have introduced a new supply chain
governance strategy in our eco-system. Our exper...
Thank You!
https://grafeas.io/blog/introducing-grafeas
https://github.com/grafeas/kritis/blob/master/docs/binary-authoriza...
Securing microservices continuous delivery using grafeas and kritis
Securing microservices continuous delivery using grafeas and kritis
Securing microservices continuous delivery using grafeas and kritis
Upcoming SlideShare
Loading in …5
×

Securing microservices continuous delivery using grafeas and kritis

4,354 views

Published on

This talk is about how Microservices Platform team at Mercari is using Grafeas and Kritis to make Microservices Continuous Delivery Secure

Published in: Engineering
  • Be the first to comment

Securing microservices continuous delivery using grafeas and kritis

  1. 1. Securing Microservices Continuous Delivery using Grafeas and Kritis
  2. 2. 2 Vishal Banthia Software Engineer, Microservices Platform
  3. 3. 3 ● What is a secure software supply chain? ● Grafeas approach ● How Microservices Platform is using Grafeas and Kritis to secure microservices Continuous Delivery? Agenda
  4. 4. 4 In general, a secured software supply chain means that only authorized softwares are executed at each supply chain stage resulting in only authorized software getting deployed in production environment. Definition of “authorized” varies from organization to organization depending on security policies and their strictness. What is a Secure Software Supply Chain?
  5. 5. 5 Some policy examples: ● Only vulnerability scanned and passed docker images are allowed ● Only QA verified software artifacts are allowed ● Only PM signed off software artifacts are allowed ● Only software artifacts which are passed during canary stage are allowed
  6. 6. Code DeployBuild Test Compute Engine Kubernete s Engine App Engine AWS EC2 AWS EC2 Software Supply Chain
  7. 7. 7 In software supply chain stages, every stage consume some artifact and produce a new one. “By authorized at each stage”, it means that each stage is consuming a verified artifact based on organization policy. What does authorized at each stage means?
  8. 8. 8 ● Only commits from authorized authors are allowed for build stage ● Only unit / integrations tests passed artifacts are allowed for QA or end to end testing ● Only unit / integrations tests passed artifacts are allowed for security scanning Some examples:
  9. 9. 9 Traditionally, it is achieved by creating a process workflow or orchestration where each stage will run only when previous stage is passed. This can be done in various CI/CD tools or bots. One of the problem with this approach is that it is hard to govern whole supply chain from bird eye point of view. Metadata created by one stage are lost after next stage has consumed it How to achieve secure software supply chain?
  10. 10. Code DeployBuild Test Compute Engine Kubernete s Engine App Engine AWS EC2 AWS EC2 Software Supply Chain tests will only trigger when build is passed deploy will only trigger when tests are passed
  11. 11. Software Supply Chain Governance
  12. 12. 12 Governance in software supply chain means that all the software artifacts deployed in production can be tracked. CTO/CIO has full visibility and can make policy changes. Software supply chain should be designed in such a way that new security policies can be added easily in whole organisation Security is something which should be forced from top-down IMO What does governance means in software supply chain?
  13. 13. 15 CTO / CIO need to make this big decision ● Do not make any change and just hope that we are running good code 🤞 ● Make the change and let performance degrade by 40% Without data it is very difficult to make these kind of decisions. Software supply chain should be designed in a way that we can do these kind of analysis and make better action plans Untrusted Code? 🤔
  14. 14. 16 Take Away ● Now we understand what is a secure software supply chain ● Why centralized governance is required in software supply chain
  15. 15. 17 ● Growing and fragmented toolsets ● Microservices Architecture ● Open-Source software adoption What are the current problems?
  16. 16. 18 Grafeas approach “An open artifact metadata API to audit and govern your software supply chain” Basically, instead of just using supply chain result metadata for next stage, store them in a metadata server so that, that information is not lost and can be used anytime in any stage.
  17. 17. 20 Grafeas Components Metadata Server metadata server which store all the metadata information generated during various supply chain stages Policy enforcement tool tool which uses grafeas metadata using API and make decision based on configured policy. It is not exactly a grafeas component but without this supply chain will not be complete
  18. 18. build stage uses grafeas API to check if commit is allowed to build or not deploy stage uses grafeas API to check if artifact is QA verified and have not vulnerability before deploying
  19. 19. 22 Take away! ● Using grafeas, we can store metadata of all supply chain stages in one centralised database ● Using policy enforcement tool, each stage can verify if incoming artifact is authorized or not
  20. 20. How microservices platform team is using Grafeas and Kritis to secure microservices Continuous Delivery?
  21. 21. 24 Microservices Platform Architecture ● We use GCP ● Each microservice has its own dedicated GCP project and they are free to choose any service such as cloudsql, pub-sub etc ● We have a centralised GCP project which is managed by platform team and we run GKE there ● Each microservice has its own namespace and microservice owner only have access to their namespace ● We use Spinnaker for Continuous Delivery
  22. 22. 25 GCP project for GKE Centralized cluster Namespace: Service A Namespace: Service B IAM: Platform Team IAM: Team A IAM: Team B Service A Service B RBAC: Team A RBAC: Team B
  23. 23. PM will confirm if feature is ready to release and ask developers to create tag QA will notify PM regarding QA results QA get notified and will do QA if necessary in dev environment if image-tag has some version, it will trigger deploy image to production cluster if image-tag has master-* prefix, it will deploy image to development cluster spinnaker start deployment pipeline based on image tag docker image is pushed to GCR cloudbuild triggers if tests passes and docker image is built developers push code to github (branch, master, tag) Software Supply Chain for Microservice circleci is triggered and it runs unit tests
  24. 24. PM will confirm if feature is ready to release and ask developers to create tag QA will notify PM regarding QA results QA get notified and will do QA if necessary in dev environment if image-tag has some version, it will trigger deploy image to production cluster if image-tag has master-* prefix, it will deploy image to development cluster spinnaker start deployment pipeline based on image tag docker image is pushed to GCR cloudbuild triggers if tests passes and docker image is built developers push code to github (branch, master, tag) Software Supply Chain for Microservice circleci is triggered and it runs unit tests
  25. 25. For secure microservices Continuous Delivery, we want to make sure that only authorized docker images get deployed to our production cluster only authorized docker images
  26. 26. 29 What is authorized docker image for us? ● Image is built by service owner ● Image has been verified by security team for vulnerabilities ● Image has been signed by QA ● Image has been signed by PM ● …
  27. 27. 30 ● We use GCP Container Analysis API which is implementation of Grafeas ● We use Kritis fork which is basically Policy Enforcement Tool for Kubernetes ○ Kritis is basically a Kubernetes admission webhook which get triggered whenever a new pod is created. Based on policy written in `ImageSecurityPolicy` CRD kirtis make judgement of allowing or denying that pod. Kritis uses container analysis API to get image metadata How we do this?
  28. 28. Security Team Signer Service deploy new docker image in echo-namespace with digest (sha256:abcd…) Cloud Build GCR docker image Container Analysis API BUILD (Occurrence) VULNERABILITY (Occurrence) ATTESTATION (Occurrence) microservice GCP Project GKE Cluster GCP Project Admission Controller kritis-namespace echo-namespace kritis validation webhook ImageSecurity Policy echo pod OK to deploy 1 2 6 5 4 8 7 3 QA Team Signer Service
  29. 29. 32 How it works? 1. Cloudbuild build docker image and push it to GCR. It also create a build metadata with information such as which project was used to build image 2. GCR triggers spinnaker 3. Spinnaker apply kubernetes manifest file with new docker image tag. This goes to kubernetes admission controller 4. Admission controller call kritis validation webhook with new pod information 5. Kritis admission controller check `ImageSecurityPolicy` CRD from echo-namespace and get current policy
  30. 30. 33 6. Kritis calls container analysis API and get metadata for that docker image 7. Depending on the policy, Kritis validate if image is allowed or not and inform admission controller 8. Depending on Kritis results, admission controller allow to create new pod or not How it works?...
  31. 31. 34 Signer Service Signer Services run independently and sign images based on their job. For example, security signer service confirm all vulnerabilities or check what base image is being used and sign it based on their policy. Similarly, QA signer service can also be made which sign images once their job is done.
  32. 32. 35 Why FORK? ● Official kritis is still at very early stage and does not have feature which we wanted such as: ○ Validate GCPProjectId where docker image is built ○ Custom attestation check ○ Clusterwide image whitelist https://github.com/mercari/kritis
  33. 33. 36 Ending Using Grafeas and Kritis, we have introduced a new supply chain governance strategy in our eco-system. Our experience is good so far. We want to write more signer services and add grafeas in more supply chain stages and add full observability for our software supply chain
  34. 34. Thank You! https://grafeas.io/blog/introducing-grafeas https://github.com/grafeas/kritis/blob/master/docs/binary-authorization.md https://kubernetes.io/blog/2017/11/securing-software-supply-chain-grafeas/ https://github.com/mercari/kritis https://codelabs.developers.google.com/codelabs/cloud-binauthz-intro

×