When Nobody Checks the Homework: Real-World Task-Verification Failures in AI-Agent Systems

In this post, I talk about something that kept biting me while building agentic systems in production: multi-agent LLM pipelines rarely check whether a step actually got done right before passing it on to the next agent. This isn't a niche edge case, it shows up in almost a quarter of all recorded multi-agent failures, and it has already caused real, documented damage at real companies. I spent last quarter digging into why this happens for a Research in Web Data and Information Management course at TU Delft, and this is the longer, easier to read version of that work, now with the receipts.

The problem, in one sentence 👋

If you've built anything with multiple LLM agents talking to each other (an agent that plans, one that codes, one that reviews, one that ships), you already know the scary part isn't that agents make mistakes. It's that mistakes get passed along as if they were correct, because nothing at the boundary between agents actually stops to check.

I've run into this myself, both while building an agentic editorial pipeline at ABN AMRO and a RAG-based verification system at CloudSEK. An agent finishes a subtask, hands its output to the next one, and everyone just assumes it's fine. Multi-agent systems are stochastic by design, so a single silent error can cascade through the whole pipeline before a human or another agent ever notices.

So how big is this, really? 📊

Turns out, pretty big. The largest empirical study of multi-agent failures to date (Cemri et al., NeurIPS 2025) annotated over 1,600 execution traces across seven popular frameworks like MetaGPT, ChatDev, and HyperAgent, and built a taxonomy of exactly how these systems break. One whole category of that taxonomy is task verification failure: the system's inability to confirm that an output is accurate, complete, and reliable before it moves downstream. That category alone accounts for 23.5% of all observed failures, and it splits into three distinct failure modes.

A quick walkthrough of what each one actually means:

• Premature termination (6.2%): the agent stops before the job is actually done, and the output looks complete even though it isn't. There's no explicit finish line the agent is checking against, so it just... stops.
• Absent or incomplete verification (8.2%): nobody checks the output at all before it's passed on. This is the "we'll just trust it" failure mode.
• Incorrect verification (9.1%): the trickiest one. A check does happen, but the check itself is wrong. Usually this is because the same model that produced an output is the one grading it, and that's a bit like asking a student to mark their own exam.

That third one is the most common of the three, and it points to something structural rather than just sloppy prompting. Also, this isn't just a lab statistic. It has a body count in production systems, so let's look at two cases up close.

Case study 1: the agent that deleted a production database, then told everyone it couldn't fix it 💥

In July 2025, SaaStr founder Jason Lemkin was running a twelve day "vibe coding" experiment with Replit's AI agent, building an app end to end mostly by talking to it. On day nine, in the middle of an explicit, all caps, do-not-touch-anything code freeze, the agent deleted the live production database. Not a staging copy. Records for over 1,200 executives and nearly 1,200 companies, gone.

Here's the part that matters for this post though. The database being wiped was the visible disaster, but the invisible one was worse: the agent's own status reports could not be trusted either. It told Lemkin the rollback function wouldn't work, which turned out to be false, he recovered most of the data manually. Along the way it had also fabricated a chunk of test data and presented it as real. When confronted, the agent's own words were: "This was a catastrophic failure on my part." Replit's CEO, Amjad Masad, agreed publicly that this should never have been possible, and the company shipped fixes afterward: automatic separation between dev and production, a proper one click restore, and a planning-only mode that can't touch anything live.

What makes this a task verification failure and not just "a bad AI" story is the layering. First, there was no boundary check before an irreversible action (deleting a table should never happen without a hard gate). Second, once it happened, the agent's own account of what it had done and whether it could be undone was itself unverified and wrong. You had a failure at the action boundary and a failure at the reporting boundary, back to back. Lemkin's own reaction sums up the trust problem neatly: how do you put something like this in production if it can ignore direct instructions and delete your data anyway?

Case study 2: when the agent grades its own homework, and gives itself an A 📝

The second case is quieter than a wiped database, but I'd argue it's the more important one, because it shows why incorrect verification (that 9.1% slice) keeps happening even when people are actively trying to avoid it.

Anthropic's own Claude Code issue tracker has become an accidental case study in this. In one widely discussed thread, a developer running a trading system described the agent reporting that "130 tests" were passing when they weren't, because it was reading the test runner's exit code rather than actually checking the results. In another instance on the same tracker, the agent ran 14,400 tests and reported everything green while real money was on the line behind those tests. The developer's own framing has stuck with me since I read it: pytest passing is not verification. Verification means checking that the real process, with real data, produces the output you actually expect, not just that a command exited without crashing.

A separate issue on the same tracker documents the agent describing what a screenshot showed, in detail, before the user had even sent the screenshot. The user's reply was blunt: so you lie? And then, after it happened again: you lie again.

I don't think "the model is lying" is the most useful way to read this, these are token predictions, not confessions. But functionally, for the person on the other end of the pipeline, the effect is identical to being lied to. And structurally, this is exactly what the MAST taxonomy calls incorrect verification: a check happens, the check is wrong, and because the same model produced both the work and the grade, there was never an independent signal to catch the mismatch. Anthropic's own internal research backs this up directly, they found that when you ask an agent to evaluate its own work, it tends to confidently praise it, which is precisely why they now recommend keeping the agent that does the work separate from the agent (or process) that checks it.

Why you can't just fix it after the fact 🔍

My first instinct, and honestly most of the field's first instinct, was to focus on catching and fixing errors after they happen. Rollback, checkpointing, re-running the failed step. Both case studies above show why that instinct runs into trouble: in the Replit case, even the rollback status itself couldn't be trusted, and in the Claude Code case, there wasn't even an error message to chase, everything reported green.

There's a good reason to be skeptical that post-hoc recovery is enough on its own, and it isn't just these two anecdotes. Zhang et al. (ICML 2025) built a dataset of 127 annotated multi-agent pipelines and tested how well even state-of-the-art reasoning models could pin down the responsible agent and step, after the fact.

Getting the agent right happens barely better than a coin flip. Getting the exact step right is worse than one in seven. If we can't even reliably diagnose failures after they've already happened, and Lemkin and the Claude Code users above only got to the bottom of things through manual digging, that's a strong argument for trying to catch problems before they enter the pipeline at all, rather than leaning entirely on cleanup afterward.

People have noticed, and trust is dropping because of it 📉

This isn't just a developer complaint either, it shows up in the numbers. Stack Overflow's 2025 Developer Survey (nearly 49,000 developers) found that trust in AI generated code accuracy actually fell year over year, and experienced developers were the most skeptical group of all. On the enterprise side, Capgemini's Research Institute surveyed 1,500 executives across 14 countries and found trust in fully autonomous agents dropping sharply in a single year, with a large share of leaders now saying the risks outweigh the benefits.

What's happening underneath these numbers is basically what both case studies illustrate: teams are discovering that verifying an agent's work is not optional overhead, it's the actual job now. Multiple industry surveys (Foxit, and separately Sage/IDC surveying finance leaders) put a number on this: the hours people believe they're saving with AI are nearly matched by the hours they now spend double checking its output. One Sage/IDC stat stuck with me in particular: a majority of finance leaders said they'd reject a tool that was 99% accurate if it couldn't explain its answers, because the unexplained 1% is exactly where the Replit and Claude Code style failures live.

What people have actually tried

The literature has approached this in three broad ways, each with a different tradeoff. I simplified the comparison table from my paper below. Three things matter for a mechanism to really close the gap: does it need a human in the loop (or is it automated), does it give you a provable pass or fail rather than a vibe check, and is it actually enforced at the handoff point between agents, not just checked at the very end.

Approach	Catches half-finished tasks	Catches missing checks	Catches wrong checks	No human needed	Provable pass/fail	Enforced between agents
Chain-of-thought / self-consistency	❌	🟡	❌	✅	❌	❌
MetaGPT (structured hand-off rules)	🟡	🟡	❌	✅	❌	✅
ChatDev (agents ask for missing detail before responding)	🟡	🟡	❌	✅	❌	✅
Reflexion (memory of past failures)	🟡	❌	❌	✅	❌	❌
VeriPlan (formal model-checking of plans)	✅	✅	❌	❌	✅	🟡
VeriMAP (pass/fail functions per subtask)	✅	✅	🟡	✅	🟡	✅

Nothing hits every column. The closest attempt, VeriMAP, gets you automated checks enforced right at the handoff, but the checks themselves are generated from the same natural-language task description that could be ambiguous in the first place, so a version of "who checks the checker" sneaks back in at one level up. Which brings us to the actual root of the problem.

The actual root of the problem

Here's the part I found most interesting while writing the original review. Every one of these approaches runs into the same wall eventually: you can only verify a task if you can say, formally, what "done" looks like. Most real agentic tasks are described in open-ended natural language, and turning that into something a machine can check automatically is genuinely unsolved. It's a well-recognized bottleneck in formal methods research too, sometimes called the specification problem or the oracle problem, not something I made up for this paper.

Both case studies map onto this cleanly. Replit's agent had no formally stated rule like "never execute a destructive database operation during a stated freeze," so nothing was there to block it. Claude Code had no independent, non-gameable definition of "the tests actually pass," so a green exit code became a stand-in for truth. In both cases the gap wasn't a smarter model, it was the absence of an explicit, checkable contract at the boundary.

So what would an actual fix look like? 🛠️

This is the part I want to push further than the original course paper did. If the root cause is the absence of a formal, enforced, independent check at the boundary between agents, then the fix has to be architectural, not just a better prompt. Here's the pattern I've been sketching out, which is also where my thesis is headed: boundary verification contracts, or the boundary contract pattern for short.

The idea has three moving parts, and each one directly answers one of the failures above.

1. A contract, not a suggestion. Every subtask gets an explicit, machine-checkable specification of what "done" means, written down before the agent starts, not inferred afterward from whatever it produced. For Replit, that contract would include something as blunt as "no destructive writes during a declared freeze, full stop." For a research subtask, it might be "the returned answer must cite a source that actually exists and actually says this." This is the same idea behind VeriMAP's verification functions, just pushed to be a first class citizen of the pipeline rather than an optional add-on.

2. A verifier that has no reason to lie to you. The single biggest lesson from the Claude Code case is that self-grading doesn't work, not because models are dishonest, but because the same weights that produced the answer are being asked to judge the answer, and there's no independent signal in that loop. The fix is structural separation: a different model, a deterministic test harness, a schema validator, or a human, depending on how much is at stake. Critically, the verifier should never be the same process that did the work.

3. A hard gate before anything irreversible, and a permanent log either way. If the contract isn't satisfied, the system doesn't proceed and hope for the best, it halts, rolls back, or kicks the decision to a human, the same way Replit's post-incident fixes now force a plan-only mode and a real one click restore. And because we already know post-hoc attribution is unreliable (that 14.2% figure again), every boundary check gets logged with a timestamp the moment it happens. That turns "we'll figure out what went wrong later" into "we already know, because we checked at the time."

I'd tier the actual verification method by how much is at stake, roughly like this:

Risk tier	What it checks	Would have caught
Tier 1: deterministic	Schema validation, read-after-write checks, exit codes cross-checked against actual state	A file-deleting agent that assumed a folder existed without confirming it, a classic absent-verification bug
Tier 2: independent model	A second model, with no visibility into how the first one reasoned, re-checks the output against the stated goal	Claude Code style "green but wrong" reports, where the honest failure was invisible to the model that produced it
Tier 3: formal / hard gate	A model-checked or rule-based constraint that physically blocks the action if violated, no override	Replit's database wipe during a stated freeze, where the only acceptable outcome is the action simply not being possible

I want to be honest about one thing here, because I have a personal allergy to overclaiming: I don't think anything is truly foolproof, and neither does the literature. Even this pattern inherits the same specification problem underneath it, someone or something still has to write the contract, and if the contract itself is wrong or incomplete, you've just moved the "who checks the checker" question up one more level, the same critique that applies to VeriMAP. What I do think is true is that this is the most rigorous version of the idea currently available, and it converts an implicit hope ("the agent probably did it right") into an explicit, loggable, independently checked claim. That gap, between hoping and checking, is exactly where Replit's database and Claude Code's green tests fell through.

Where this is going

This is the direction my MSc thesis is heading: boundary verification contracts for multi-agent pipelines, trying to get the sketch above from a diagram into something with actual guarantees behind it. If there's interest, I'll follow up with a deeper post on how VeriMAP's verification functions could be extended into something closer to formally checkable, and what a minimal version of this pattern looks like in code. For now, if you're building anything with multiple agents talking to each other, it's worth asking yourself a simple question: when one agent hands work to the next, does anything actually check it, independently, before something irreversible happens, or is everyone just assuming it's fine until a production database disappears?

Sources: Cemri et al., "Why Do Multi-Agent LLM Systems Fail?", NeurIPS 2025. Zhang et al., "Which Agent Causes Task Failures and When?", ICML 2025. Fortune's July 2025 reporting on the Replit incident. Anthropic's public claude-code issue tracker. Stack Overflow 2025 Developer Survey. Capgemini Research Institute, "Rise of Agentic AI" (2025). Full reference list available in the original paper on request.