Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
© 2016 VMware Inc. All rights reserved.© 2016 VMware Inc. All rights reserved.
Kubernetes in PROD
VMware SAAS (Cloud Manag...
VMware SAAS
2
1 K8S Usage
2 Dynamic Provisioning
3 Monitoring & Upgrade
VMware SAAS in the current context refers to CMBU ...
VMware SAAS: Overview
• MCM solution combining the capabilities of Application orchestration, monitoring, management,
cost...
Types of Application
• Green Field (New) Apps
– Xenon based Micro-services
– Distributed
– Soft states often categorized t...
Deployments
• Core business logic are modelled as stateless containers
– Java & Scala applications
– Load based replica se...
Stateful Sets
• Persistence layer & distributed applications
• Mongo DB
– Containerized, replicated
– Entry point Initiali...
Best Practices
• Cloud agnostic
– Avoid AWS specific services
• Prefer stateless
• Service Discovery
– Internal communicat...
K8S in PROD
8
1 K8S Usage
2 Dynamic Provisioning
3 Monitoring & Upgrade
Dynamic Provisioning
• Use cases
– Single instance of N-tier application/distributed application supports “X” tenants
– Us...
Dynamic Provisioning
• Control plane
– Build using Xenon
– K8S REST spec client in Java
– Business logic to scale based on...
K8S in PROD
11
1 K8S Usage
2 Dynamic Provisioning
3 Monitoring & Upgrade
Monitoring
• Tools Used
– VMWare vRealize Log Insight for log monitoring (Kibana is good too)
– Data dog (Grafana is good ...
Monitoring
• Application
– Pod contains Data dog agent & Log insight agent
– Captures application metrics
• Dropwizard met...
Upgrade
• Side by side Upgrade
– Preferred option for soft state (deployment) applications
– Pauses user request momentari...
Questions ???
15
Upcoming SlideShare
Loading in …5
×

K8S in prod

1,113 views

Published on

Usage of Kubernetes container Orchestration in Production

Published in: Technology
  • Be the first to comment

  • Be the first to like this

K8S in prod

  1. 1. © 2016 VMware Inc. All rights reserved.© 2016 VMware Inc. All rights reserved. Kubernetes in PROD VMware SAAS (Cloud Management BU) Mageshwaran R Kumar Gaurav
  2. 2. VMware SAAS 2 1 K8S Usage 2 Dynamic Provisioning 3 Monitoring & Upgrade VMware SAAS in the current context refers to CMBU initiative alone
  3. 3. VMware SAAS: Overview • MCM solution combining the capabilities of Application orchestration, monitoring, management, costing etc • SaaS first, platform not a product • Enables DevOps engineers and Developers to be more agile and leverage resources from any cloud. • Built using variety of tech Stack – Distributed Micro-services – Java, Scala, Spring, Guice, Xenon etc., – Relational databases, Key value stores, Document stores etc., VMware SAAS in the current context refers to CMBU initiative alone 3
  4. 4. Types of Application • Green Field (New) Apps – Xenon based Micro-services – Distributed – Soft states often categorized to Stateless apps • Brown Field (Existing) Apps – Existing applications follows n-tier architecture – Containerization – Leverages K8S Pod – Separate Stateful & Stateless apps 4
  5. 5. Deployments • Core business logic are modelled as stateless containers – Java & Scala applications – Load based replica sets, auto scale • Soft state applications – Xenon services & distributed task processing – Distributed applications uses remote nodes to store the data than disk – Spread pod across nodes (anti-affinity) • Configure Liveness Probe – Health Checks API 5
  6. 6. Stateful Sets • Persistence layer & distributed applications • Mongo DB – Containerized, replicated – Entry point Initializes and configures replica set – Customized Storage class : aws-ebs (type, iops, zone) • Postgres DB – Containerized, Active-stand by – Streaming replication – Pgpool 6
  7. 7. Best Practices • Cloud agnostic – Avoid AWS specific services • Prefer stateless • Service Discovery – Internal communication • Uses FQDN of K8S service name • HTTP, TCP – External Service communication • NGINX/ HA proxy • CI/CD Pipeline using vRealize code stream 7
  8. 8. K8S in PROD 8 1 K8S Usage 2 Dynamic Provisioning 3 Monitoring & Upgrade
  9. 9. Dynamic Provisioning • Use cases – Single instance of N-tier application/distributed application supports “X” tenants – Use case demands due to compliance/ data isolation requirements etc., – Horizontal scaling of application (collection of heterogeneous K8S resources) • Onboarding Service – Runs as a deployment inside cluster – Use JSON templates to create K8S resources – Lazy while provisioning and avoid shuffling – Orchestrates updates 9
  10. 10. Dynamic Provisioning • Control plane – Build using Xenon – K8S REST spec client in Java – Business logic to scale based on the usage & load – Wraps bootstrap logic using containers • Challenges – K8S Version upgrade – Scale down – Environment abstractions 10
  11. 11. K8S in PROD 11 1 K8S Usage 2 Dynamic Provisioning 3 Monitoring & Upgrade
  12. 12. Monitoring • Tools Used – VMWare vRealize Log Insight for log monitoring (Kibana is good too) – Data dog (Grafana is good too) • Support for all standard processes like JVM, RDBMS, Mongo etc., • Docker & Kubernetes • Support for posting custom metrics – Pingdom • Users perspective – Pager duty – Status.io • Communicate health to stakeholders 12
  13. 13. Monitoring • Application – Pod contains Data dog agent & Log insight agent – Captures application metrics • Dropwizard metrics & Xenon Stats • Publishes to data dog – Custom monitoring dashboards in Data dog • Dev Ops UI – Uses Kubernetes auth – Separate from application authn & authz • Fault Injection – Simulates Pod failures – Simulates CPU Usage & network delays 13
  14. 14. Upgrade • Side by side Upgrade – Preferred option for soft state (deployment) applications – Pauses user request momentarily to minimize downtime • Rolling upgrade – Preferred option for Stateful sets – Avoid transformations and provide backward compatibility • Backup & Restore – Soft states are backed up to S3. – EBS are periodically snapshotted to S3 – Periodic restore to Staging – DR scenarios 14
  15. 15. Questions ??? 15

×