This is my noob recap of KubeCon 2019, which I transformed into a kubernetes bootcamp. I walked away with a bunch of learnings, so here they are for you :)

1. OR: HOW I TURNED KUBECON 2019 INTO A KUBERNETES BOOTCAMP

2. ABOUT ME
• Husband & father of 2
• English major, almost-lawyer
• Cloud, CICD, resiliency, open-source
• Also ’s boxing, jiu-jitsu, dendrology
• Richmond AWS User Group
• RVA Tech Talks
Zero Kubernetes credibility 😊

4. DAY -1 | PREP
Archiecture
how does it
look?
“Deep dive”
how does it
work?
Biz use case
what does it
solve?
“Kubernetes Deep-Dive”
by Nigel Poulton (A Cloud Guru)
Blogs & KubeCon 2018

5. DAY 0 | AWS CONTAINER DAY
• I already knew the ecosystem & tools
• Really good docs & awesome workshops
• AM - Kubeflow w/ EKS
• PM - Service mesh w/ AppMesh
• Felt like a quick win
• Went to chill on an aircrab carrier
DAY 1 | PUTTING IT TOGETHER
• Context (opening keynote)
• Overview (CNCF projects & updates)
• More context (closing keynotes)

6. Party on the
U.S.S. Midway

7. DAY 2 | SWIMMING IN DEEP WATERS
• Keynotes (from hotel room)
• Meet the maintainers (floor)
• How to contribute (session)
• Keynotes (from hotel room)
DAY 3 | WHAT’S MISSING?
• “Rails moment” (closing keynote)
• Use cases (Reddit & Tinder)
• Random topics (security & edge)
• Cool down (holy moly)

8. MY STRATEGY | BE UBER-PRESENT
• Zero pride (honest w/ myself & others)
• Took a break when I needed it (lots of running)
• Zero social commitments (to focus on learning)
• Didn’t hang on details (e.g. during a talk)
TAKEAWAYS | ENLIGHTENMENT!
• Kubernetes is very different
• non-intuitive paradigm shift w/ tentacles
• Tech moves very fast (stupid questions galore)
• Kubernetes is big on-prem (lots of running)
• Interesting edge (to focus on learning)

9. borg
@ google omega
@ google
Kuber-
netes!
1.0 &
CNCF
pokemon,
helm, kops
😍
service
mesh &
stability
🦕

14. nodes*
cluster
*e.g. namespaces on 1 or more EC2 instances

15. nodes*
cluster
node
pod
pod
pod
your app!
aws
your stack!

16. cluster
myapp-container
$ kubectl create –f app.yaml $ kubectl apply –f app.yaml
replicas: n
app: myapp
app: myapp
This is called a
DEPLOYMENT
It is an object
in the API.
app.yaml

17. myapp-container
kubeletproxy
api-server what should exist
“This is who I am!”
“Make it so!”
pod
scheduler
x 1,000+
x 1
controller-manager

18. a “normal” view

19. another “normal” view

20. Let’s get weird.

21. CarlLarry Val
Ford Seth Concho

22. control plane
scheduler
server
controller
fry pod
grill pod
salad pod
x2
node

23. pod network 10.0.0.0/16
10.0.1.0/24 10.0.2.0/24 10.0.3.0/24
10.0.1.x 10.0.2.x 10.0.3.x 10.0.3.y
node network 192.168.0.0/16
eth0 eth0 eth0
192.168.1.1 192.168.1.2 192.168.1.3
svc
172.11.250.5K8s cluster
Your cloud provider will
scale nodes to add pods
Kubernetes will scale
pods to add containers
192.168.n.n
Watch KubeCon 18 talk

24. OR: HOW I TURNED KUBECON 2019 INTO A KUBERNETES BOOTCAMP

25. LIGHTNING TALKS
• They were great.
• Youtube playlist

26. HIDING IN THE DARK
• His 5 y/o son was hiding from zombies
• Strategy: digging into the dirt and staying very still
• He hung out w/ a 7 y/o & learned to build a table, then use it to build weapons
• Enterprises are like 5 year-olds, K8S is the table, community is 7 y/o

27. HOW CNCF
PROJECTS
WORK
Sandbox (beta)
Credibility + guidance
+ access to CNCF
warchest (e.g.
organization,
marketing, staff)
Incubation (alpha) Early adopters
Graduation (v1)
Mission critical usage
Maintainer diversity
Steady health

30. graduated

31. incubating

32. sandbox

33. The K8S
landscape
😮
incubating
graduated
important

34. https://landscape.cncf.io
CNCF projects (all stages)

35. WHO PAYS FOR
THIS STUFF?
• Kubernetes saves money.
• Corporaoons love it.
• Why did Google open-source it?
• 90% of developers are being paid to
work on CNCF projects*
• CNCF is kinda like a country club

36. TREES &
CLOUDS
• Podocarpus, jacaranda, carrotwood, big figLiterally!
• aws, azure, gke, etc.Providers
• managed as part of K8S core
• This is how it started
“In-tree”
cloud provider
• Has its own release cycle
• e.g. AWS manages it’s own set of integraoons
“Out-of-tree”
cloud provider
• Kubernetes is truly cloud-agnostic with no
native integrations for any cloud provider!Milestone!

37. PROJECT
UPDATES
•MySQL
•Slack @ 100%
•Crazy portability (adios Amazon)
Vitess (graduated)
•Bring policies all over the place
•Admission Control in K8S
•Web Assembly is coming…
Open Policy Agent (incubaong)
•Cloud-native messaging (services + streams)
•Digital signing (not PPK’s)
NATS (incubating)
•1.16
•Better storage!
•Windows support!
•Better debugging! (ephemeral containers)
•More reliability (affinity)
•All cloud providers moved “out of tree”
Kubernetes

38. RANDOM COOL
THING:
REBUILDING THE
HYPERVISOR
Rust – language of
hypervisors. Very weird.
Rebuilding KVM to be a truly
“cloud native hypervisor”
• Small VM’s for firecrackers
• Big VM’s for legacy or ML workloads
Guy from intel

39. How do we
manage????
empire-building vendor lock-in
😵systems are
complicated!

40. instrumentation
telemetry
open tracing
open census
tracing back-end
ui front-end
data mining platform
jaegar
client SDK’s (all major languages)
new & better ways to trace

42. Live demo!
• Slow Fibbonaci sequence in Go
• Imported & added Open Telemetry
• Saw where exactly slowness was occurring
• Fixed it

44. link to pdf
Open Policy Agent (OPA)
deep-dive

45. • OPA = the best
admission controller
• mutation + verification
This is what happens every time an
API call is made on Kubernetes
• E.g.
What is OPA?

46. • OPA = the best
admission controller
• mutation + verification
This is what happens every time an
API call is made on Kubernetes
• E.g.

48. myth of the monocluster
matt silverlock, google
TL;DR
• 1 massive cluster = bad
• Lots of small clusters = good?
• Some general rules:
• No pets
• Map risk domains
• Dedicated platform team

49. • Common sense: 1 massive cluster
• K8S supports this kind of thing, right?
• Easy to think about & interact with, right?
• Bad news:
• It’s always DNS (especially when you’re syncing w/ external infra)
• You have just 1 apiserver & etcd (your control plame), which can scale vertically until…
• A single workload can and will impact your entire cluster operations (e.g. eating IO)
• More bad news:
• You will inevitably move at the pace of your most risk-averse team
• Inertia against security upgrades, small patches, and new features
• Teams don’t really care about each other
• 1 cluster / team?
• Conway’s Law doesn’t work for Kubernetes
• You get lots of needy pets 🐕 🐕 🐕 🐕 🐕 🐕 🐕 🐕 🐕 (poorly-trained & dangerous)
• General rules:
• No pets! (use cloud providers toolkits + standardized OPA & CI + sane defaults)
• Map out “risk domains” along fault lines, and isolate high-risk services
• Staff a dedicated platform team to: 1) own platform, and 2) help customize clusters
• Getting started:
• Start small
• Don’t expose native K8S API’s
• Don’t wait for requirements
• Start small
Link to
pdf

50. you have 1
control planeneighbors are noisy
DNS is
inflexible

51. Avoid “general
purpose”
…but that’s hard.

52. OR: HOW I TURNED KUBECON 2019 INTO A KUBERNETES BOOTCAMP

53. 2005
DHH
Ruby + Rails
“Whoops!
…this is NOT Kubernetes in 2019
Front-end Optimizer Back-End
Ruby Rails Websites!
C++/Julia/Haskel LLVM Machine
Our app KubernetesYAML?????“Senior Engineer of Defaults”
How might we capture
the essence of Rails
without diminishing the
power?

55. Extra slide for Ed & others who love Ruby on Rails
DHH
used
TextMates
“let’s copy
DHH!”
devs use
Macs
Resurgence
of text
editors
modern text
editors
modern
IDE’s
Macs &
modern
IDE’s
Devs have
free time!
Homebrew
(written in
Ruby)
More free time
to think outside
the box!
“Forget web
requests”Rails ecosystem
rethinking
Sinatra
Heroku
rethinking
PaaS
git push
heroku:master
thingsHelm!

56. 👀 defenders 👀attackers
kubectl auth can-i
systems
reflect
attitudes
vision
organization
beliefs
fears
K8S = open, trusong, ❤ ❤ ❤
Not always good!
also:
silos are for grain!
Sec peeps, the
other SIGs
need you!

58. Resource attacks (CPU)
Network attacks (latency)
scaling
dependenciesCluster + Namespace
***Not your actual
environments!***
“We should do chaos testing!”
“NO!”
“What if it was 100% safe?
“GAME DAY!”
🙂 1x/quarter
😄 1x/month
😍 On-demand
😈 All the time!
😱 OKR’s
🤖 ML insights!

61. X x 11,000 stores
265 million
customers/week
K8s cluster
azure
satellites
slow!
data
availability
simplify
consumption
observability
sec &
management
control resiliency
kafka
custom
orchestrator
Prometheus
+ FluentD
Vault, Sloop,
& CA Store
federated
control plane
client failover
w/ Istio &
Envoy sidecar
How?

62. DAY IN THE LIFE
• What’s a SIG?
• Special Interest Group
• <list of SIG’s>
• Every Fall, 859 core people vote on the Steering Committee
• How do they work under the hood?
• Each is unique but follows its own norms
• 100% remote
• Community Meetings: streamed weekly on YouTube (e.g. Docs is Th, AWS is Friday)
• PR’s reviewed by members (this is biggest pain point)
sig-list.md

63. HOW-TO GET STARTED
• Getting started:
• Pick a SIG that’s 1) easy, or 2) that you’re interested in
• Attend the meetings (& offer to take notes!)
• Join Slack channel and introduce yourself
• Take a “first pass” at PR’s
• Eventually…there’s a “contributor ladder” to climb
• Wait…I have a life!
• Core contributors are paid, but don’t make this your goal (that’s weird)
• Per-hour productivity increases over time
• Careful w/ commitments! Your reputation matters.
• You don’t need loads of time to get started (focus on step 1)
• Don’t focus on everything
productivity
age

64. 3 SUPER COOL
“K8S JOURNEY”
TALKS
• Tinder
• Reddit
• DoD

65. REDDIT’S
K8S
JOURNEY
Link to PPT

66. In 2018, Reddit built a
K8S platform called
InfraRed to empower
service owners to do
all most of the things.
KubeCon 2018 deck

67. 2019:
TALES FROM
PRODUCTION
• Reduced blast radius + saved costs
Single-AZ is better than multi-AZ?
• OPA calls overwhelmed control plane
Cluster policies (RBAC à OPA) caused death spiral
• Helm generates “Mega Charts” that are confusing, resulting in
config drift
• Baseplate.py auto-generates service definitions (“now, SRE’s at
least know what’s in production”)
YAML drudgery for service owners
• MiniKube à remote clusters
• Skaffold à Tilt upgrade
• Helm Mega-Charts à Baseplate service generator
Local dev sucks
2020:
• Self-service onboarding
• Refine Dev Env story
• Build out SRE org
2018:
• 1 service every 2 wks
• mostly worked!
• “When can my team onboard?

68. TINDER’S
K8S
JOURNEY

69. LOOP DO; FEEDBACK; END
• EC2 auto-scaling group
• Code pushed to NFS mount on EC2 & triggered service restart
Pre-2018
• “Builder container” to standardize image creaoon
• VPC peering (AWS VPC’s to K8S VPC’s) + Route 53 balancing load
Step 1: create a new K8S environment
• ARP – several hours downtime (ARP cache exhaustion) – increase limit!
• 250k/second - # of DNS requests to 1000 Core DNS Pods – redeployed w/ Daemon Set to cut down on HTTP requests
Step 2: work out the DNS bugs
• Some pods hot, even though new pods added
• Enjoy sidecars used (diagram)
• “Everything else is dropped”?
Step 3: work out Load Balancing bugs

70. YouTube:
link
DOD’S K8S
JOURNEY

71. THE U.S. GOVERNMENT USES K8S, TOO!
• CSO of Air Force
• 3-10 y SDLC, 8 mo. Procurement
• Be lucky you work at Capital One
• “DoD DevSecOps Initiative”
• Entire stack is FOSS
• Zero Trust w/ Istio
• 2 teams à platform & devops
• Hard to innovate on high-side

72. Unexpected enthusiasm for K8S.

73. ENOUGH STICKERS TO
PATCH A BATTLESHIP HULL

74. How it be when Kelsey Hightower comes back.

76. Forget the tech for a
second…
Why are we here?
To solve problems.
To work together.
To empower each other.
To show up & take risks.
And why not learn
something?
Why not include as many
people as possible?

77. Forget the competitiveness for a second…
Forget your title & what company you work for…
We’re all just pace setters.
Just do your own teeny tiny part.
And step aside to make room for the next person.
…in your own unique way.

78. NON-MEN IN TECH:
• Ian Coldwater
• Liz Rice
• Liz Fong-Jones
• #womenintech
& non-white people
NON-WHITES IN TECH:
• Brian Liles
• Kelsey Hightower
• Stephen Augustus
• #BlackTechTwitter

79. • Are you sleeping? Are you eating healthy?
Do you have someone who can let you
know?
• Arrogance is dangerous. You are probably
average.
• “Letting people know” à so they can hold
you accountable.
Work-life balance:
• Mental health is stigmatized. Misdiagnosed,
too.
• “Drugs make me feel normal”
• “Burn out” is very very bad.
Mental health in tech:

80. A SUPER
INCLUSIVE
CONFERENCE
*EXCEPT FOR IF YOU’RE NEW
TO K8S J
Paw Therapy
Yoga & mediation rooms
Diversity Scholarship (plus low rates for
individuals & academics)
Diversity Hack Lunch
Daycare
Vegan/GF/Kosher/Halal options at all meals

81. noteworthy: genuine attempts at coolness

82. 1. Kubernetes is a very good thing
2. Thank you & your welcome
3. KubeCon 2020 is in Boston!

83. LINKS:
In Search of the Kubernetes Rails Moment – Brian Liles
•https://youtu.be/ZqQTEdHVaCw?t=308
Reflections on Kubernetes - Kelsey Hightower, Google
•https://www.youtube.com/watch?v=jiaLsxjBeOQ
Mental Health in Tech - Dr. Jennifer Akullian, Growth Coaching Institute
•https://youtu.be/G-SdeRBHc9M
An Introducoon to Helm - Ma| Farina, Samsung SDS & Josh Dolitsky, Blood Orange
•https://youtu.be/Zzwq9FmZdsU
Cloud Provider Subproject AWS / User Group AWS - Nishi Davidson, Pulumi & Justin SB, Google
•https://youtu.be/z6LlhFfFGQM
Panel: State of the Kubernetes Union - Steering Committee Discussion
•https://youtu.be/0Su1kKlr9q0
Panel: Improving and Managing Kubernetes at Scale - Xiang Li, Alibaba; Corin Dwyer, Netflix:
•https://youtu.be/BetxFccSpxQ
How the Department of Defense Moved to Kubernetes and Isoo - Nicolas Chaillan
•https://youtu.be/YjZ4AZ7hRM0
Kubernetes Deconstructed: Understanding Kubernetes by Breaking It Down - Carson Anderson, DOMO
•https://youtu.be/90kZRyPcRZw