From APIs to Agents: Rethinking Testing for Dynamic Systems

Editor's Notes

• #1 SLIDE 1 — Title (0:00 – 1:00) [Walk to center stage before you speak. Let the room settle. Don't start until it's quiet — silence is a tool, use it.] Good morning, everyone. I want to start with a claim, and I want you to sit with it for a second before I explain it: every API in this room, right now, today — passes its test suite. Green across the board. Contracts honored, schemas valid, status codes correct. And that fact has almost nothing to do with whether the agent calling that API tomorrow will actually get your user what they came for. [pause] That gap — between "the API is correct" and "the system succeeded" — is the entire talk. I'm going to walk you through why that gap exists, four specific ways it breaks things in production, why the testing you already trust can't see it, and then — because I don't like talks that just diagnose a problem and walk off stage — a concrete, practical way to close it. Something you can start building next sprint, not next year. Twenty-five minutes. Let's go.
• #2 SLIDE 2 — Divider: The Shift (1:00 – 1:30) For a decade, "calling an API" meant one thing: a developer sat down, wrote code that called endpoint A, then B, then C, in an order they chose, and shipped it. Boring. Predictable. Beautifully, wonderfully boring. That's over. And I mean that as a compliment to the old world, not the new one.
• #3 SLIDE 3 — The Shift content (1:30 – 3:00) Look at the left card first. This is the caller every testing strategy in this industry was designed around. One caller, one path, one shape in, one shape out. If it breaks, it breaks loudly, at the call site, in a stack trace with your name on it. Now look at the right card. An agent doesn't run code you wrote — it reasons about which API to call. Same user request, and depending on the day, depending on what the model felt like doing, it might call three APIs or five. It might call them in a different order tomorrow than it did this morning, using the exact same prompt. [gesture at the on-call line] That's why your on-call rotation started crying. Softly, at first. Because the failure mode isn't "the server is down." It's "the server is up, healthy, answering every request correctly — and the agent is still making bad decisions with perfectly good tools." Here's the sentence I want you to leave this section with: your APIs didn't change. The caller did. Everything downstream of that sentence is what the rest of this talk is about.
• #4 SLIDE 4 — Divider: The Illusion of "It Works" (3:00 – 3:30) So if the caller changed, why hasn't our idea of "it works" changed with it? Because green checkmarks feel like proof. They're not. They're the participation trophies of software testing — everyone gets one, and they tell you almost nothing about who actually won.
• #5 SLIDE 5 — Illusion content (3:30 – 7:00) Let me make this concrete with our first running example: a travel booking agent. Somebody types, "Book me a hotel in Goa next weekend, use my usual discount code." Simple ask. Watch the pipeline. Search Availability — 200 OK, valid response. Check Policy — 200 OK. Apply Discount — 200 OK. Confirm Booking — 200 OK. Four calls. Four green checkmarks. If you're watching a dashboard of API health right now, everything is glowing green. Somebody on the platform team could screenshot this and put it in a status report and nobody would blink. [pause, point to the outcome card] And the booking that actually got created? Wrong dates. Wrong price — the discount got applied to a date range the user never asked for, because the agent called "Apply Discount" before "Check Policy" had actually finished resolving the trip dates. The support inbox is now on fire, and somewhere, a customer is refreshing their booking confirmation trying to figure out why they're checking into a hotel in Goa on a Tuesday when they asked for a Saturday. Every single API call in that chain was correct. The contract was never violated, not once. That's not a bug in contract testing — and this is the important part — contract testing did exactly what it was built to do. It just was never built to answer the question that actually mattered here, which is: was this the right sequence of decisions for this specific user's specific goal? Hot take, and I mean this one: contract testing didn't fail on this booking. It was never trying to succeed at this in the first place. We've just been treating "the contract held" and "the system worked" as the same sentence for so long that we forgot they're not. If your test suite only checks the four boxes on the left of this slide, it will confidently, cheerfully, ship the box on the right. Every time.
• #6 SLIDE 6 — Divider: Failure Patterns (7:00 – 7:30) So let's get specific. Not "agents sometimes make mistakes" — that's not useful to anyone in this room. Let's talk about the four specific, repeatable ways agent-API systems break, quietly, in production, while every dashboard stays green.
• #7 SLIDE 7 — Failure patterns grid (7:30 – 11:30) API Selection. The agent doesn't call a wrong endpoint — it calls a plausible one. Somewhere in your API catalog you probably have `getRefundStatus` and `getOrderStatus` sitting near each other, described in similarly-worded docstrings. A user asks "where's my order," and the agent, reasoning over tool descriptions rather than your actual business logic, picks the one that sounds closest. Confidently. It doesn't know it's wrong. It will never tell you it's not sure. Sequencing. The agent calls the right APIs — every one of them — in the wrong order, because nothing in an OpenAPI spec tells it "you must confirm inventory before you capture payment." That dependency lived in a developer's head, in a comment, maybe in a wiki page nobody's opened since 2023. The agent has never read that wiki page. Payment gets captured first. Ask me how I know. Orchestration. This is what happens the moment you have more than one agent, or parallel branches of the same agent, touching shared state. Two sub-agents, working the same user request, both decide independently that applying the loyalty coupon is a good idea. Neither one knows the other exists. Both apply it. Both are, in whatever sense an agent can be pleased with itself, very pleased with themselves. Context Drift. This one's my favorite because it's the most human. Long session, lots of back-and-forth. Three turns ago, the user said, clearly, "no seafood, I have an allergy." By turn eight, that constraint has been summarized, and re-summarized, and truncated out of the active context window — and the agent orders lobster. Confidently. It seems, if I can anthropomorphize for a second, almost proud of the recommendation. [pause, let that land, it usually gets a laugh] Here's the thread connecting all four of these: every single one of them produces valid, schema-correct, contract-respecting API traffic. None of them — not one — is a contract violation. Your monitoring never sees it. Your CI never sees it. The only place it shows up is in the outcome the user actually experienced.
• #8 SLIDE 8 — Hot take slide (11:30 – 13:00) I want to leave you with one uncomfortable sentence before we move on, because I think it's the most important one in this whole talk. [slow down deliberately here, this is the moment to actually pause] If your agent's test suite has never — not once — made you nervous... you're not testing the agent. [beat] You're testing your own optimism. [let it sit for a real three or four seconds before continuing — do not fill the silence] Screenshot that one if you want. Bring it to your next retro. I'll wait right here while you do it. [small smile, then move]
• #9 SLIDE 9 — Divider: Limitations of Current Testing (13:00 – 13:30) Okay. So why can't the tools we already trust — the ones half this room has spent years building — catch any of the four patterns I just described?
• #10 SLIDE 10 — Testing's blind spots table (13:30 – 16:00) Read the third column with me, because that's the actual content of this slide, not the second one. Contract testing confirms the shape of a request and response. It has no concept of whether calling that endpoint, right now, in this position in the flow, was the correct decision. It would have happily passed every call in the booking agent's Goa disaster from three slides ago. Schema validation confirms fields and types match what's expected. It cannot tell you whether the data returned actually satisfies what the user was trying to accomplish. A perfectly-typed wrong answer sails right through. Traditional automation scripts one expected route through the system, end to end, and checks that route works. But an agent doesn't take "the" route. It takes a route, and it might be a different one tomorrow with the identical input. A single-path test is checking a path the agent may never walk again. Translation, and I mean this plainly: all three of these ask "was this well-formed?" None of them ask "was this right?" We've been pricing those two questions as if they were the same question for years, because for the deterministic callers on the left side of that earlier slide, they basically were the same question. They are not the same question anymore. That gap — "well-formed" versus "right" — is exactly what the rest of this talk closes.
• #11 SLIDE 11 — Divider: Behavioral API Testing (16:00 – 16:30) Here's where I stop complaining and start being useful. Four moves. In order. Each one is a direct answer to one of the gaps we just named — I'll call that out explicitly as we go, so you're not just collecting techniques, you're watching a puzzle get solved.
• #12 SLIDE 12 — Step 1 & 2: Golden datasets + multi-pass (16:30 – 19:30) Move one: curated evaluation datasets. People call them golden datasets. This is the direct fix for "no validation of intent" from the table we just left. A golden dataset is not a list of request payloads — that's what a Postman collection is for, and you already have one. A golden dataset is a list of real user intents, and — this is the part people skip — each case carries an expected outcome, not an expected schema. "User wants a Goa hotel for next Saturday under 8,000 rupees, using their loyalty discount" is a golden case. "GET /hotels returns 200" is not. Good golden datasets are deliberately uncomfortable. They include the ambiguous asks, the conflicting constraints, the missing data — because that's where agents actually fail, not on the happy path everybody writes their demo for. And critically: this dataset needs to be owned like a product spec. Reviewed, versioned, argued about in meetings. Not generated once by an intern and frozen in a repo from eighteen months ago. Move two: multi-pass execution. Here's the thing nobody wants to hear: because the agent is non-deterministic, running a test case once tells you almost nothing. It tells you what happened on one roll of the dice. Look at the diagram — same input, three independent runs. Run one, clean. Run two, it takes a detour at step three that runs one and three never took. Run three, clean again. [gesture at the flagged dot] One agent. Same prompt, word for word. Three different moods. That's not a flaky test you should quarantine and ignore — that variance is the finding. A behavior that only shows up two times out of ten is exactly the kind of failure a single-path test can never generate, because a single-path test only ever rolls the dice once. Run every golden case five, ten, twenty times, and look at the spread, not just the average.
• #13 SLIDE 13 — Step 3: Outcome-based scoring (19:30 – 22:00) Running the case is only half the job. Now you have to grade what happened — and "pass or fail" isn't a rich enough vocabulary for that anymore. We use three dimensions. Relevance — did the agent call APIs that actually served this specific intent, not just tools that sounded plausible? This is the direct catch for the API-selection failure from earlier — `getRefundStatus` masquerading as the right answer gets caught right here. Faithfulness — does the agent's final response to the user accurately reflect what the APIs actually returned, with nothing invented, nothing smoothed over, nothing summarized into something that wasn't true? This catches a failure that gets more common the more capable these models get: the agent confidently narrating an outcome that its own tool calls don't actually support. Completeness — were all the necessary steps carried out, in a state the user would actually recognize as "done"? Not "the last call in the chain returned 200" — actually done, from the user's chair. And to be very clear about what these are not: not "is this a valid API," not "is the JSON well-formed," not "did every call return 200." Those questions are already answered. These three ask the question underneath them. This needs an LLM-as-judge, or a rubric-based scorer, checked against the golden dataset's expected outcome for that case. A boolean assertion cannot do this job. I want to be honest about that up front, because somebody in this room is already thinking "can't I just write an if-statement for this," and the answer is no, and that's actually the interesting part, not a limitation to apologize for.
• #14 SLIDE 14 — Step 4: Continuous evaluation harness (22:00 – 24:00) Last piece, and it's the one that turns everything I just described from a research exercise a couple of engineers ran once into an actual engineering practice. Golden dataset feeds the agent under test. Multi-pass runner executes every case N times. Outcome scorer grades every single transcript on those three dimensions. Results land on a regression dashboard that tracks trend, not just today's snapshot. And — look at the arrow looping back — failures you find in production get written back into the golden dataset as new cases. The dataset should grow with the system. It does not get to stay frozen from day one. This harness is API-driven on purpose. It calls your agent exactly the way a real caller would, scores the transcript, and fails the build on a drop in relevance, faithfulness, or completeness — the same discipline you already apply to a performance gate or a security gate. Runs on every deploy. Wired into CI. Not a quarterly audit that somebody schedules and nobody reads. No vibes-based sign-off. That's the whole idea.
• #15 SLIDE 15 — Implications for engineering teams (24:00 – 24:45) Fast, because we're almost out of time and this section earns brisk, not rushed. Strategy — leadership has to accept a trend line instead of a binary gate. That's a real cultural shift for anyone raised on green-means-ship, red-means-stop. Tooling — this is additive. Contract and schema testing keep doing exactly what they're good at. This sits alongside them, catching the thing they were structurally never built to see. Ownership — and this is the one people push back on in the room afterward — golden datasets are not a QA-only artifact. They need the product owner, the domain expert, whoever actually knows what "done" looks like for the user. QA can't write intent nobody told them. Yes, this is more work. You're welcome — it's cheaper than the incident review.
• #16 SLIDE 16 — Summary mapping (24:45 – 25:15) [speak quickly and rhythmically here, almost like a drumroll — three beats, three answers] Structural validation only → outcome-based evaluation: relevance, faithfulness, completeness. Single-path testing → multi-pass execution: same input, many independent runs. No validation of intent → curated golden datasets: real intents, real expected outcomes, owned like a spec. That's the whole talk, compressed into three lines.
• #17 SLIDE 17 — Closing (25:15 – 25:45) [slow all the way back down for this — this is the line people quote afterward] The agent will always pass your unit tests. The question is whether it passes your users. [pause] Thank you. And yes — I did just spend twenty-five minutes telling you your agent lies to you. I stand by it. Come argue with me about it, I'll be right there at the speaker table. [walk off before the applause fully lands — don't linger]