Successfully reported this slideshow.
Your SlideShare is downloading. ×

Containerization

Advertisement

More Related Content

Advertisement

Related Books

Free with a 30 day trial from Scribd

See all

Containerization

  1. 1. Containerization*
  2. 2. Primary Goals Dreams ● Improvements in reliability among different compute environment. Prod == QA ==Dev ● Better CI /CD ○ Deployment process aligned with our branching policy ○ A common pipeline to catch mistakes ○ Flexibility to bring new environments ○ RBAC ○ Deploy anything, anytime ● Improvements in AWS resource usage ● Scalability - Faster boot times
  3. 3. New Changes ● Config file Management - Encrypt all sensitive datas and No AWS keys ● Dependency Management - Composer, Maven and go modules ● Prometheus operators / Push-gateways ● Kubernetes native Jenkins containers ● Certificate management ● Granular IAM Roles for security ● Port Management ● Egress controls for security
  4. 4. New Changes ● Canary deployments ● Easier A/B Testing using Service mesh ● Smart routing using ingress controllers ● Continuous Integration using Jenkins ● Continuous Delivery ● Get rid of Rsyslog ● 90 % workloads is on Spot instances ● Better RBAC on exotel infra
  5. 5. New Changes ● Automatic Network retries ● Rate limiting ● Production traffic mirroring ● Zipkin request tracing ● Controlled CPU and RAM for all services This is what I can remember as of now :)
  6. 6. We did a Mistake ● We named the project wrongly ● This is not just about containers and it’s orchestration ● This is a complete exotel platform revamp
  7. 7. Gitops - our philosophy ● A Git repo is the single source of truth for the desired state of whole system ● All changes to the desired state are git commits ● In case of divergence, Kubernetes will try to sync according to git repo ● Rollback is “Convergence to an earlier desired state” ● There can’t be one more cron machine or Erix server or exoconsole
  8. 8. Repository Structure Changes ● Each service will have it’s own repository ● All service config files will be present in a separate repository called “Exotel_Configs” ● All Kubernetes deployment files will be present in one more repository called “Exotel_Deployments” Service Repo + Exotel_configs => Exotel_Deployments => Git-Ops => Exotel
  9. 9. Exotel Configs ● Single repository to host all config files. It’s a monorepo ● Only sensitive information in the configs are encrypted using PGP or AWS KMS ● Only certain members in the team will have KMS access to encrypt/decrypt passwords ● This will be the home for Kubernetes helm configs as well ● Configs are encrypted during runtime when the pod is created
  10. 10. Exotel Deployments ● Single repository that contains all the deployment files (For every environment) of Kubernetes. ● Kubernetes controllers like argo watches this repo and any new change can be synced to the cluster ● Wanna Rollback to previous versions ? Just do a git revert of the deployment file.
  11. 11. SOPS - Envelope Encryption ● Mozilla/sops is used to manage the encryption and decryption.
  12. 12. Envelope Decryption
  13. 13. Steps to containerize ● Add Dockerfile & docker-compose file in the service repo ● Add Jenkinsfile in the service repo ● Push all your logs to stdout / stderr ● Use versions for all your dependencies ● Move config files to config repo ● Add Helm files in config repo ● Commit to PU branch
  14. 14. CI Pipeline - Jenkins ● Works based on PU and Next branches ● A commit pushed to PU/Next branch triggers an CI Jenkins job and an image is built and pushed to ECR ● Helm deployment files are created and pushed to “exotel_deployments” repository
  15. 15. CD Pipeline - Argo ● We don’t do continuous deployment ● There is just 1 Jenkins for all environment but different argocd tool for each environment
  16. 16. Logging Pipeline ● Fluentd daemonset runs on all machines that collects all logs that are pushed to stdout/stderr of the containers ● All logs are shipped to doglump ● Doglump will die in near future. ● We are ready for the futrure - Fluentd already enriches the logs so that it can be consumed by an Elastic cluster
  17. 17. Collecting Metrics ● We got rid of Rsyslog ● Jellibabix was using 500Mb of RAM but rsyslog required 2.5 Gb of RAM to ship metrics ● Fluentd-metrics daemonset runs in every machine to collect the metrics from containers and forwards it to kafka ● Services can no longer just push to localhost / 127.0.0.1
  18. 18. Kubernetes Monitoring ● Prometheus operators are used ● Any divergence in the declarative config is raised as an alert ● Grafana is present to visualise server / container metrics ● Kubelet daemon running on every machine sends container metrics to prometheus ● You can just expose an metrics endpoint on your service and configure prometheus-operator to scrape data
  19. 19. Feedback / Questions ?

×