This document describes a hybrid approach to local microservice development against an EKS cluster. The key points are: - Developing locally but against an EKS cluster is challenging due to differences in environments. - The proposed solution is to run all services in the EKS cluster except the one being developed locally, replacing it with a proxy that redirects traffic to the local machine. - This allows using local development tools while still interacting with the full production environment in the cluster. It can be done through exclusion, scaling the local service to 0, or interception using an nginx proxy to redirect traffic. - A case study is provided of how this approach is implemented at NTWRK using Tail

1. Local Microservice
Development in EKS
David Kirk

2. About Me
- 10 years as a Dev, 8 of which in DevOps
- Staff Engineer at NTWRK
- AWS & Kubernetes have been main focuses for 6 years
- d@vidkirk.com or endlessbreadsticks@thankgoodnessitisifridays.com

3. What’re we talkin’ about?
- Describe the problem - how do we develop against EKS?
- Attempted & Potential Solutions
- Hybrid Environments - the solution we’re using at NTWRK
- Case-study of how it’s implemented at NTWRK
- Suggestions of quick wins w/ this technique

4. I’ve done a talk like this before
- I’ve learned lessons since then about how to do this better
- That talk proposed something that took 6+ months to fully build
- The goal here is to describe the solution and give practical wins you
can bring home

5. The Problem - Developing against EKS
- Most development happens locally
- Tools are designed with this in mind
- Debuggers, profilers, build tools, etc
- Developers are used to this mental framework
- EKS is not running locally
- But it’s where production is running
- How do we bridge this gap?
- Ideal solution is one that allows us to use EKS and local tools

6. Why is this worth solving?
- Reduce cognitive overhead
- There’s a barrier to entry here
- Worth it in the long run - fewer variations to consider
- Minimize internal tooling & configuration complexity
- The fewer environmental variations, the less work to align them
- Devs get Ops experience
- Hey look, it’s DevOps!
- If devs gain k8s experience while dev-ing, they know better how to work
with k8s in prod

7. Summary of Goals
- Easy to use & implement
- Useful
- Devs can tie their local machine into EKS
- Ideally vice-versa as well

8. High Level Solution - Redirect Traffic to Local
- This is the basis for all proposed solutions
- Run the thing you’re developing locally
- Run everything else somewhere else
- Send traffic that would go to the thing you’re developing to localhost
Everything Else Runs Here Your Laptop
Service
A
Service
B
Dummy
Service
C
Actual
Service
C
- Dummy service replaces yours in the cluster
- Receives traffic like normal
- Routes all traffic to your localhost
Network
Traffic

9. Potential Solution: docker-compose
- Pro: Simple to configure
- It’s entry-level deployment yaml, IMO
- Very readable, and in one file
- Con: It’s not kubernetes
- Duplicating configuration - effort to maintain
- Devs aren’t exposed to kubernetes
- Run everything not being actively developed in docker-compose
- Substitute the thing you’re developing w/ an nginx container
- Forward all traffic it gets to `host.docker.internal`
- Run the thing you’re developing locally

10. Potential Solution: Minikube
- Pro: It’s kubernetes!
- You get some kubernetes experience
- But also you don’t get multi-node kubernetes
- Con: Configuration complexity
- All of your charts/definitions need to support both minikube & EKS
- That’s gonna be some complex charts
- Run everything not being actively developed in Minikube
- Substitute the thing you’re developing w/ an nginx container
- Redirect its traffic to `host.minikube.internal`
- Run the thing you’re developing locally

11. Running Everything Locally Has Limits
- Things will get slow
- You’re better at creating stuff for your CPU to do than Intel & Apple are
at making CPUs that can do stuff fast
- Even beefy machines will slow down
- This slows down everything else around development
- You can deploy subsets of services
- Again introducing configuration complexity
- Hard to maintain, prone to issues

12. Proposed Solution - Hybrid EKS Development
- Run everything in EKS except what you’re developing
- Substitute the thing you’re developing w/ an nginx container
- Forward all traffic it gets to your IP address
- Run the thing you’re developing locally
- Use a VPN to access the cluster
- Reach into cluster w/ it - use k8s DNS for your cluster’s CIDR
- Reach out of the cluster w/ it - use VPN IP addresses to talk to dev
machines

13. Exclusion & Interception
- Two modes of network traffic interaction
- These cover all of the use cases we’ve seen internally
- Exclusion
- Scale the deployment you’re developing to 0
- Run it locally, and use k8s DNS through the VPN to interact w/ cluster
- Used if it doesn’t receive requests - e.g. Kafka Consumers & Cronjobs
- Interception
- Replace the pod you’re developing w/ nginx
- Redirect that traffic to your laptop’s VPN IP address
- Run it locally, again use k8s DNS through the VPN

14. Exclusion & Interception Examples
EKS Cluster Your Laptop
Kafka Broker
Service A Service B
nginx
Service C
Actual
Service C
Network
Traffic over
VPN
Consumer
D
- Exclude D
- It doesn’t receive requests - it consumes from Kafka
- Intercept C
- Capture requests & send to local machine

15. Case Study: Using this at NTWRK
- We have “personal environments”
- Single centralized helm repo defines everything that’s deployed
- `make build-dev` builds all containers
- `make deploy-dev` takes helm and deploys it into personal namespace
- Intercept & Exclude at deploy time
- `make deploy-dev-tailscale INTERCEPT=[chart_name]`
- `make deploy-dev-tailscale EXCLUDE=[chart_name]`
- Can intercept & exclude in one deployment

16. Interception Details
- How do we route that traffic to the VPN IP address?
- We use Tailscale as our VPN
- All examples will reference those specifics, but concepts can be tweaked
based upon your needs
- Replace containers in intercepted pod
- nginx routes traffic to VPN IP address - set at deploy time
- Tailscale side car gives nginx access to VPN over pod’s localhost

17. Interception Details: Actual Code Edition
Use this template if
intercepting
Default
deployment
configuration

18. Interception Details: Actual Code Edition pt 2
Internal Traffic
External Traffic
Send it to local
machine’s VPN IP

19. Interception Details: Actual Code Edition pt 3
nginx container
nginx configuration
Tailscale sidecar
grants VPN access

20. Potential Quick Wins
- You don’t need personal environments
- Run command locally to intercept static environment
- Could be staging or UAT
- Be careful about collisions
- Team-specific long-lived environments
- Less concern about collisions

21. That’s it! Questions