David Kirk Local Microservice Development in EKS AWS Communtity Day Midwest 2023

1. Local Microservice
Development in EKS
David Kirk

2. About Me
- 10 years as a Dev, 8 of which in DevOps
- Staff Engineer at NTWRK
- AWS & Kubernetes have been main focuses for 6 years
- d@vidkirk.com or endlessbreadsticks@thankgoodnessitisifridays.com

3. What’re we talkin’ about?
- Describe the problem - how do we develop against EKS?
- Attempted & Potential Solutions
- Hybrid Environments - the solution we’re using at NTWRK
- Case-study of how it’s implemented at NTWRK
- Suggestions of quick wins w/ this technique

4. I’ve done a talk like this before
- I’ve learned lessons since then about how to do this better
- That talk proposed something that took 6+ months to fully build
- The goal here is to describe the solution and give practical wins you
can bring home

5. The Problem - Developing against EKS
- Most development happens locally
- Tools are designed with this in mind
- Debuggers, profilers, build tools, etc
- Developers are used to this mental framework
- EKS is not running locally
- But it’s where production is running
- How do we bridge this gap?
- Ideal solution is one that allows us to use EKS and local tools

6. Why is this worth solving?
- Reduce cognitive overhead
- There’s a barrier to entry here
- Worth it in the long run - fewer variations to consider
- Minimize internal tooling & configuration complexity
- The fewer environmental variations, the less work to align them
- Devs get Ops experience
- Hey look, it’s DevOps!
- If devs gain k8s experience while dev-ing, they know better how to work
with k8s in prod

7. Summary of Goals
- Easy to use & implement
- Useful
- Devs can tie their local machine into EKS
- Ideally vice-versa as well

8. High Level Solution - Redirect Traffic to Local
- This is the basis for all proposed solutions
- Run the thing you’re developing locally
- Run everything else somewhere else
- Send traffic that would go to the thing you’re developing to localhost
Everything Else Runs Here Your Laptop
Service
A
Service
B
Dummy
Service
C
Actual
Service
C
- Dummy service replaces yours in the cluster
- Receives traffic like normal
- Routes all traffic to your localhost
Network
Traffic

9. Potential Solution: docker-compose
- Pro: Simple to configure
- It’s entry-level deployment yaml, IMO
- Very readable, and in one file
- Con: It’s not kubernetes
- Duplicating configuration - effort to maintain
- Devs aren’t exposed to kubernetes
- Run everything not being actively developed in docker-compose
- Substitute the thing you’re developing w/ an nginx container
- Forward all traffic it gets to `host.docker.internal`
- Run the thing you’re developing locally

10. Potential Solution: Minikube
- Pro: It’s kubernetes!
- You get some kubernetes experience
- But also you don’t get multi-node kubernetes
- Con: Configuration complexity
- All of your charts/definitions need to support both minikube & EKS
- That’s gonna be some complex charts
- Run everything not being actively developed in Minikube
- Substitute the thing you’re developing w/ an nginx container
- Redirect its traffic to `host.minikube.internal`
- Run the thing you’re developing locally

11. Running Everything Locally Has Limits
- Things will get slow
- You’re better at creating stuff for your CPU to do than Intel & Apple are
at making CPUs that can do stuff fast
- Even beefy machines will slow down
- This slows down everything else around development
- You can deploy subsets of services
- Again introducing configuration complexity
- Hard to maintain, prone to issues

12. Proposed Solution - Hybrid EKS Development
- Run everything in EKS except what you’re developing
- Substitute the thing you’re developing w/ an nginx container
- Forward all traffic it gets to your IP address
- Run the thing you’re developing locally
- Use a VPN to access the cluster
- Reach into cluster w/ it - use k8s DNS for your cluster’s CIDR
- Reach out of the cluster w/ it - use VPN IP addresses to talk to dev
machines

13. Exclusion & Interception
- Two modes of network traffic interaction
- These cover all of the use cases we’ve seen internally
- Exclusion
- Scale the deployment you’re developing to 0
- Run it locally, and use k8s DNS through the VPN to interact w/ cluster
- Used if it doesn’t receive requests - e.g. Kafka Consumers & Cronjobs
- Interception
- Replace the pod you’re developing w/ nginx
- Redirect that traffic to your laptop’s VPN IP address
- Run it locally, again use k8s DNS through the VPN

14. Exclusion & Interception Examples
EKS Cluster Your Laptop
Kafka Broker
Service A Service B
nginx
Service C
Actual
Service C
Network
Traffic over
VPN
Consumer
D
- Exclude D
- It doesn’t receive requests - it consumes from Kafka
- Intercept C
- Capture requests & send to local machine

15. Case Study: Using this at NTWRK
- We have “personal environments”
- Single centralized helm repo defines everything that’s deployed
- `make build-dev` builds all containers
- `make deploy-dev` takes helm and deploys it into personal namespace
- Intercept & Exclude at deploy time
- `make deploy-dev-tailscale INTERCEPT=[chart_name]`
- `make deploy-dev-tailscale EXCLUDE=[chart_name]`
- Can intercept & exclude in one deployment

16. Interception Details
- How do we route that traffic to the VPN IP address?
- We use Tailscale as our VPN
- All examples will reference those specifics, but concepts can be tweaked
based upon your needs
- Replace containers in intercepted pod
- nginx routes traffic to VPN IP address - set at deploy time
- Tailscale side car gives nginx access to VPN over pod’s localhost

17. Interception Details: Actual Code Edition
Use this template if
intercepting
Default
deployment
configuration

18. Interception Details: Actual Code Edition pt 2
Internal Traffic
External Traffic
Send it to local
machine’s VPN IP

19. Interception Details: Actual Code Edition pt 3
nginx container
nginx configuration
Tailscale sidecar
grants VPN access

20. Potential Quick Wins
- You don’t need personal environments
- Run command locally to intercept static environment
- Could be staging or UAT
- Be careful about collisions
- Team-specific long-lived environments
- Less concern about collisions

21. That’s it! Questions