1. The Reality of DIY
Kubernetes vs PKS
October 7–10, 2019
Austin Convention Center
Kendrick Coleman
Open Source Technical Product Manager
github.com/kacole2
@kendrickcoleman

2. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
2
This is Not a Conversation of “VMs
or Containers?”
Containers have proven to be greatly beneficial to
the application development lifecycle. The
decision going forward is
”how should we run our container infrastructure?”

3. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
3
Kubernetes
The focus is on orchestration
Kubernetes functions as a master/worker
relationship. Master roles require availability of it’s
services to keep quorum and schedule containers
on worker nodes. Kubernetes master components
are critical to the overall health of the cluster.

4. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
4
Kubernetes
Linux Distro with Kubernetes Installed
Container-Centric
Management
Agile Application
Creation
Automated
Scaling
Dev and Ops
Separation
Environmental
consistency
Loosely Coupled
Micro services
Update Software
at Scale
A Single Platform for
All Applications
Tightly Coupled
Network Virtualization
Abstracted Storage
Interoperability
Comprehensive High
Availability
Automated Data
Center Operations
VMware ESXi
Combined with the VMware Software-Defined Data Center
Added Security
Layers

5. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
“Cool! Let’s Build Something”
- every engineer

6. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
6
Kubernetes is an abstraction layer that allows freedom for customization and
tailoring for any platform. Run it in on your local machine, hosted in the cloud,
turnkey cloud options, on-premises turnkey solutions, or completely custom
to “do it yourself” (DIY).
Choices
And choices and choices and choices

7. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
7
From The Kubernetes Documentation:
Best-practice “fast paths” for creating a minimum viable cluster.
Installing various nice-to-have addons, monitoring solutions, and cloud-
specific addons, is not in scope.
Instead, we expect higher-level and more tailored tooling to be built on top
of kubeadm.
Inference
Kubernetes is hard. Kubeadm is a tool that builds and upgrades clusters
very well. The ability to operationalize Kubernetes beyond installation is
dependent upon maturity of the organization.
Kubeadm
Built by the community for the community

8. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
8
From the beginning…
It all starts with the machine
Step 1.
Choose your distro:
• Ubuntu, CentOS, CoreOS, SUSE, RancherOS, Talos Etc
Step 2.
Make your distro configurable as a template/AMI/etc.
• Countless blogs with outdated methods?
• Will the template work with your environment? (ie Cloud Init vs Static IP)
• Process for updating templates
• Install Docker and Kubernetes components? Automate?
• Sizing requirements?

9. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
9
Installing Kubernetes
Or is there more research to do?
DOCS, BLOGS, AND MORE DOCS!
What container runtime do you want to use?
• Docker? Rkt (no more)? Or some other CRI compatible offering?
• Lessons learned from running Docker in production?
• What container runtime version is supported with the version of
Kubernetes? What happens after yum install docker-ce?
Architecturally, single or multi-master? Stacked or separate etcd?
• Different paths in the docs
• Load Balancers, DNS, certificate sharing, and more to investigate

10. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
10
More research to weigh out the differences:
• Is there an advantage to have overlay L2 vs L3?
• kubeadm init requires special instructions for each solution.
• What tools are available for troubleshooting?
• How to manage at scale?
• What is the scaling point?
Networking
More choices? Yes more!

11. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
11
Ready to Install?
Not quite yet
Persistent Storage
• Native in-tree driver to provide persistent
backing for Kubernetes applications.
• Continually evolving (API, process, plugin
model)
• Cloud/Provider Specific Configs and
Initialization
• Few examples of how to locate or
properly configure flags on kubelet and
manifests after cluster initialization
• Few examples with kubeadm exist (even
for major cloud providers)

12. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
12
The best install guide that’s always a work in progress
1. Creating an Ubuntu 18.04 LTS cloud image for cloning on
VMware
• Reliant on DHCP
2. Setting up K8s and the vSphere Cloud Provider using kubeadm
• All the little details that seem to have not made it into the
actual docs. For both kubeadm and vSphere
• Uses Flannel as the networking layer
• Learn to automate from here
3. Using the vSphere Cloud Provider for K8s to dynamically deploy
volumes
• Storage Classes, apps, and the k8s dashboard.
Automated Steps to Install Kubernetes on CentOS7 with
Kubeadm and vSphere
• https://bit.ly/indyk8s
Now can we install?
Sure, Good Luck!
Credit to @myleasgray

13. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
13
DEMO TIME!
Want to see it in action?
You’re really selling it here…

14. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
14
Flowchart of DIY Kubeadm
DIY
Create
Master VM
1. Disable Swap for installation
2. Implement IP Table rules or disable firewall for
Kubernetes communications
3. Install Docker or CRI of choice
4. Add Kubernetes Repo to get binaries
5. Configure SELinux
6. Enable kubelet service
7. Evaluate and decide on a CNI solution for overlay
network connectivity
8. kubeadm init to initialize the cluster
9. Apply CNI solution to implement networking
10.Create custom vsphere.conf file for your environment
and protect passwords using Secrets or SAML
11.Add flags to kubelet config, controller manifest, and
API server manifest for vSphere integration
12.Restart all services and apply a default storage class
to use persistent storage
Create
Worker VM
1. Disable Swap for installation
2. Implement IP Table rules or disable firewall
Kubernetes communications
3. Install Docker or CRI of choice
4. Add Kubernetes Repo to get binaries
5. Configure SELinux
6. Enable kubelet service
7. kubeadm to join existing cluster
8. Add flags to kubelet config for vSphere
configuration
9. Restart kubelet services for vSphere
integration
10.Repeat for every worker node
But what about Day 2+?

15. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
15
Kubernetes Worker
Move to Production?
There’s more than installation
Operationalize the entire stack
• Logging
• Monitoring
• Dependency Management
• Security
• Upgrades
• Automated Repeatability
• Regression Testing
• Support
App App
Kubernetes Control Plane
Kubernetes Worker
App App
Infrastructure

16. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
16
Core Principals
Integration and Regression Testing
3 month cycle
Azure*
Over every cloud you are planning to support
Networking &
Security
OS + K8s +
Docker +
Automation
Day 2
Operations
and Support
Storage and
Persistence

17. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
17
Multi-Cloud Kubernetes with choice of consumption and services
VMware Kubernetes Portfolio
17
Private & Public Clouds
Cloud AssemblyNSX Service Mesh
Turnkey NativeModular
VMware Essential PKS
BUILD
VMware Enterprise PKS VMware Project Pacific
OPERATE CONSUME
VMware Cloud Services and Partner Solutions
Choose a solution based on
experience and ability
Quicker ROI
Verified Updates
Deterministic capabilities
Support becomes common
instead of custom
Common components create
tighter integrations
Manufactured repeatability
A better user experience
Tanzu Mission Control

18. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
18
DEMO #2
Enterprise PKS
Let’s go!

19. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
19
Enterprise PKS Turnkey Solution
Image Registry
Kubernetes Software Lifecycle Management
Security and Networking
Persistence
Virtual Infrastructure
Physical Infrastructure
Monitoring,Logging,Analytics
Cluster Health Monitoring, Healing and
Lifecycle Management
Scheduling,
Orchestration, Service Creation
VCP
vSphere | GCP | AWS | NSX | VSAN
Physical Infrastructure
NSX
BOSH
Kubernetes
Harbor
vRealizeSuite
P
K
S

20. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Harbor
An open source enterprise-class registry server.
• CNCF Incubated Project
• User management & access control
• RBAC: admin, developer, guest
• AD/LDAP integration
• Policy based image replication
• Notary
• Vulnerability Scanning
• Web UI
• Audit and logs
• Restful API for integration
• Lightweight and easy deployment
https://goharbor.io/
https://github.com/goharbor/harbor

21. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
21Confidential │ ©2018 VMware, Inc.
What’s my organization’s level of
maturity?
Can my organization investment time
for research, testing, and
integration to operationalize day
2+?
Can my team provide additional value
beyond a standard cluster?
Multiple VMware solutions that cover
validated architectures and pre-
packaged solutions to drive
maximum ROI with proven
repeatability.
Tightly integrated components provide
reliability for day 2+ operations.
Global support for the entire stack
DIY VMware PKS Family

22. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Get Started

23. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
23Confidential │ ©2018 VMware, Inc.

24. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
24
Kubernetes is complicated – use a pre-packaged solution that does this for you.
We recommend kubeadm for installation of kubernetes.
Stand up a cluster with kubeadm for free here: https://training.play-with-
kubernetes.com/kubernetes-workshop/
Click to edit optional subtitle; delete or type a space if not needed
Build your own vSphere Cluster

25. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
25
The best install guide that’s always a work in progress
1. Creating an Ubuntu 18.04 LTS cloud image for cloning on
VMware
• Reliant on DHCP
2. Setting up K8s and the vSphere Cloud Provider using
kubeadm
• All the little details that seem to have not made it into
the actual docs. For both kubeadm and vSphere
• Uses Flannel as the networking layer
• Learn to automate from here
3. Using the vSphere Cloud Provider for K8s to dynamically
deploy volumes
• Storage Classes, apps, and the k8s dashboard.
Automated Steps to Install Kubernetes on CentOS7 with
Kubeadm and vSphere
• https://bit.ly/indyk8s
Build your own vSphere Cluster

26. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
26
Learn about the features of VMware Enterprise PKS
- https://www.vmware.com/try-vmware/pks-hol-labs.html
Try out our VMware PKS Hands On Lab
FREE VMware HOL

27. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Kenny Coleman
Open Source Technical Product Manager / CNABU
Twitter: @kendrickcoleman
GitHub: kacole2
Thank You