Complimentary Live Webinar Sponsored by StormForge Analyzing the performance and behavior of applications run on Kubernetes is often challenging, making the need to optimize prior to production something that you must have. However, a problem has reared its head in the form of a question: How do you get an accurate measurement of application performance or other behavior without accurate testing or an accurate representation of how it will run in production? In this webinar, we will present and discuss a new fully Open Source tool for creating the needed tests with which to accurately measure your applications. We hope you will join us to learn more about this tool, and find out how you can help contribute. This webinar is sponsored by StormForge and hosted by The Linux Foundation. Speaker Noah Abrahams, Open Source Advocate Noah is an Open Source Advocate for StormForge, merging Open Source Strategy with Developer Advocacy. He has been involved in cloud for over 12 years, has been contributing to the Kubernetes ecosystem for 5 years, and has been up and down the business stack from DevOps and Architecture to Sales, Enablement, and Education. You will find him running meetups in Las Vegas and attending conferences, once those are both happening again.

1. Your Testing is Flawed:
Noah Abrahams
Open Source Advocate
February 10, 2021
Introducing a New Open Source Tool for
Accurate Kubernetes Application Testing

2. Agenda ● Who are we, and why do we care about
testing?
● Current testing models are flawed, so what
can we do about it?
● The project itself
● How to get started and/or involved with the
project
2

3. Who Are We? If you want to know
more about us:
● Website:
www.stormforge.io
● Public Slack:
stormforge.slack.com
● Bi-weekly Office Hours:
Thursdays at 10am PST, 1pm
EST, 6pm GMT
www.stormforge.io/event/stor
mforge-office-hours/
● StormForge (formerly Carbon Relay)
● A market leader in Performance
Testing and K8s App Optimization
○ Apply Machine Learning
○ Cloud Native
○ Open Source

4. Optimization / Desired Behavior
● Optimization is getting your application to behave how you want it
○ Could be Cost, Performance, Resiliency, # of Sandwiches, whatever
● How do you measure “Desired Behavior”?
○ The first step of measurement is knowing your tool
○ Even harder in a world of Kubernetes and Distributed Systems
● We have to stimulate behavior
○ Without a load you cannot see what the behavior response is

5. Current Load Testing
Either you don’t know at all:
● Hitting a particular testing endpoint
● Hitting just the application root
● How many 200’s per second?
● Ratio of connections to errors
● Grasping at straws
Or you’re making some quality guesses:
● Measuring the traffic
● Analyzing the flow and patterns
● Recreating those patterns
● Putting a lot of work in
● Requires engineering time and effort
Inaccurate/Irrelevant Load Tests Handcrafted Artisanal Load Tests

6. The Problem of Accuracy
Re-Creation:
● Putting in lots of effort to take real data,
and analyze it
● Using those analyses to approximate the
real data you had in the first place
● Better analysis = Better approximation
● Better analysis = More work
Estimation:
● “Just hit the root web page” -or-
● “Here’s a test endpoint”
● “Did that work?”
● “Did it break?”
● “It’s not on fire”
● “Looks good to me. Ship it!”
© NIST
© public domain
Analogy:
1 rod = “The length of 16
random peoples’ left feet”
Analogy:
1 meter = “The distance
light travels in a vacuum
in 1/299,792,458 of a
second”

7. Testing in Prod

8. Testing in Prod
● Testing outside of prod doesn’t reflect reality, but why not?
● Is it differences in infrastructure?
○ That’s easily solved
● Is it differences in operational care and feeding?
○ Should be no difference, but still easily solved
● Mostly, it’s differences in usage.
○ “Virtual Reality is an Oxymoron”

9. So, Current Testing is Inherently Flawed...

10. To Solve The Problem Of Testing In Prod,
Introducing: “VHS”

11. Aspects of “VHS”
Sidecar Record/Replay Traffic Storage
© Bob Killen © public domain
© fir0002/flagstaffotos under GFDL 1.2
ALL PLUG-INS

12. Where Could This Fit in Your Lifecycle?
Kubernetes Applications
Receiving Web Traffic
On Your Clusters
VHS
Sidecar
Storage Of
Your Choice
(S3, GCS, Local, etc)
HAR
File
Your Typical CI/CD Pipeline
VHS VHS VHS
Pipeline Knows
The Bucket Info

13. Potential Directions
● Other Storage Mediums
● Encryption, Data Sanitization, or
Parsing of Output
● Playback Session Management
● Other Traffic Types
(and Attach to Backend)
PLUGINS
● Manual Traffic Shaping
(add delays, replay at 10x
volume, etc)
● Apply ML to the Stored Traffic
● “Replace all PCI data with a
test string”
STORED TRAFFIC
MANIPULATION
● Built in Metrics
● Testing of the Kubernetes
platform, not just the apps
● Non-sidecar versions
CORE

14. The Danger of a Swiss Army Knife
© Victorinox © Victorinox © Victorinox

15. The Project and its Governance
We Do Have:
● Weekly Meetings
● Mailing Lists
● Contributor’s Guide
● Code of Conduct
● GitHub Org
We Don’t Yet Have:
● Formal Maintainer
Promotion Process
○ owners.md
● You
● A New Name

16. 1. “VHS” is unsurprisingly difficult
to Google
2. It’s already taken, almost
everywhere
3. We want something more
unique/catchy/cloud native
How?
1. Open suggestions during
meetings + shared suggestion
document
2. Hold a CIVS poll
3. Formally rename around
beginning of Q2
Why?
The Renaming Effort

17. How to Get Started/Involved
● GitHub
○ github.com/rename-this/vhs
● Mailing List
○ groups.google.com/g/vhs-pre-rename-launch
● Public Slack
○ stormforge.slack.com in #vhs-to-be-renamed
● Weekly Public Meetings
○ Mondays at 9am PST, noon EST, 5pm GMT
○ Join the mailing list for the Zoom invite
○ Public Minutes/Agenda in Google Documents

18. We Hope To See You Soon!