This document discusses using K3s (a lightweight Kubernetes distribution) to deploy globalized edge workloads on bare metal servers located at infrastructure edge locations. It provides examples of edge architectures using K3s for database replication, function-as-a-service workloads, and content delivery. Packet Labs enhances these patterns by providing bare metal servers across multiple core and edge sites with network automation and anycast IP addressing to easily deploy and manage K3s clusters globally. The document demonstrates deploying a location-aware application on K3s using orchestration tools to provision a standardized cluster pool across Packet infrastructure.

1. Globalized Edge Workloads with K3s
August 13, 2019
Joseph D. Marhee of Packet Labs

2. Packet
Labs Overview

3. MISSION
At Packet Labs, we craft solutions
to uncommon problems that live
at the intersection of hardware,
software and networks.

4. Meet Joseph D. Marhee
Ecosystem Engineer @ Packet Labs
● Customer-first solutions obsessive.
● Long time IaaS systems solutions engineer.
● Has worked with Kubernetes in the provider
space since 2015 and has a lot of opinions
about it.
● @joseph.packet on the Packet Community
Slack

5. Labs Team
My D. Truong
Senior Director
Scott Anderson
VP, Outreach
Ed Vielmetti
Special Projects Director
Joseph Marhee
Ecosystem Engineer
Carl Perry
Ecosystem Engineer
Scott Burns
Senior Director, R & D
➔49 contributing team
members on Github
➔15,000 users around
the world

6. Programming

7. Overview
Why Edge Compute? Why K3s? Why Bare metal? Why anything?
● Infrastructure Edge, Mobile Edge
● Edge Architectures
● K3s Powered Applications & Emerging Uses
● Orchestration
● Containers on Bare Metal
● How does Packet enhance these patterns?
● Demonstration of Packet K3s

8. Quick Recap of Traditional
K8s Architecture Concepts
“The network is the computer”
● Kubernetes is, conceptually, a scheduler and a
reconciliation loop.
● APIServer takes instructions to scheduler
● Scheduler informs node agent (Kubelet)
● Kubelet reports to control plane on state,
metrics, provision success/failure.
● Define -> Deploy > loop(Check, Act)

9. What’s the Infrastructure
Edge?
● Globally-distributed points of presence of, for
example, an IaaS endpoint.
● These can be local to where IoT edge devices
are located.
● Architecturally, this is a simple concept, but
in practice, the tools may not be intuitive,
easily globalized and standardized.

10. Edge Architecture Example: Replicated workloads
- Database engines with global replication
support (think something like FaunaDB)
- Ingests and validates data on one instance,
shards across other clusters globally in a
mesh of facility clusters
- The pattern would not be unlike a “Kubernetes
Underlay” in the absence of robust federation.

11. Edge Architecture Example: FaaS
Users are load-balanced between sites, in
this example, and when a request comes
into the LB, a facility is selected. This pattern
is ideal for FaaS-based workloads:
- Tools like Fission.io or OpenFaaS on
each endpoint
- Can be a local POP for representing--
at the edge- data store in your core
site persistent storage.
- Apply this to mobile edge to
infrastructure edge patterns.

12. Practical Use Case: Content Delivery
Source
If familiar with a traditional Content
Delivery Network (CDN) model, this
is one such pragmatic use case for
K3s:
- Core site running a full cluster
- Edge sites with disposable
deployments, rapid re-
deployment, serving a cached
data.

13. Orchestration
The ability to roll out and manage a fleet
of high number of comparatively low-
powered on-prem/mobile devices, and
more robust instances in a local facility is
an important one for edge compute.
Traditional configuration management
aside, tools like:
- Terraform (only as opinionated as
your provider/platform, on-prem or
in DC)
- Wrappers like Gitops
- Image builders like Packer
Can speed up CI/CD of these resources
to your edge sites (mobile and
infrastructure).

14. Why Containers on Bare Metal?
“Stop killing kittens and melting the ice caps: Run containers on bare metal already.” - Bryan Cantrill
Virtualization can be an inefficient* abstraction for edge architectures:
● Low-cost hardware for small deployments available (removes the costly virtualization layer--
particularly at the edge)
● Container security less of a question (see: the Unikernel v. Container-in-VM debate circa 2015)
● For edge: maximizing benefits of the hardware platform features for single-tenancy.
● K3s enables common patterns for CI/CD, deployment, and rapid architecture rollout.
* if you are not in the business of providing VMs, and if you are only servicing containers for your entire environment

15. How does Packet enable Kubernetes?
● The speed and developer friendliness of cloud provider tools.
● Unopinionated platform that enables:
○ Compatibility with nearly all community solutions
○ Partners to target Packet effectively
● Network automation suitable for:
○ LoadBalancer type Services
○ Cluster networking (and between clusters and facilites)
○ Location-aware routing with Anycast IP addresses

16. Packet and the Edge
● Packet offers multiple core sites and edge locations-- a point of presence in most areas.
● Layer 2 & 3 network automation, Anycast IP addressing, and diverse NICs across a broad set of
specs means that these features can be leveraged for K3s.
● Network volume storage in core sites, attached storage on all host types.
● Building an Infrastructure Edge-facing architecture becomes highly declarative using common
tooling like Terraform, Ansible, Packer, to automate provisioning, and continuous integration
of an edge site.

17. Packet K3s Demo
Deploying a global, location-aware application on K3s. Using
orchestration and provisioning tooling to manage a standardized cluster
pool.
Technologies used:
- Anycast IPs/Global IPv4
- K3s
- Packet Terraform provider to provision hosts, network, and
bootstrap cluster.

18. I want to believe! What’s next?
Running a K3s-powered edge strategy on Packet.
Packet Labs Projects for Kubernetes in Core
and Edge site models on Packet:
● Kubernetes w/BGP
● Multi-architecture Kubernetes
● Packet K3s
Some of the tools and software that can target
an edge strategy discussed today:
● Fission.io
● OpenFaaS
● chick-fil-a/gitops
● FaunaDB

19. Conclusion
● Any questions?
● Want to get involved with improving the Kubernetes & K3s experiences on Packet?