Supervision Debt

AI was supposed to remove work. So why does using it sometimes feel like another job?

I’ve had the intoxicating experience of setting several coding agents loose at once. One investigates a bug while another writes tests and a third explores an architectural change. Terminals scroll. Files change. Work is visibly happening at a speed I could never match.

Then I return an hour later and realize I have no idea what I’m looking at.

In one case, an agent encountered ambiguity about where a piece of authorization logic belonged. It made a reasonable decision, propagated that decision through the implementation, updated the tests to reflect it, and returned a clean, internally consistent result. The problem was that I disagreed with the original decision. Understanding that took much longer than spotting a broken test would have. I had to reconstruct the assumption, trace how it spread, and separate the useful mechanical work from the architectural direction I wanted to undo.

The machine had produced a lot of output. It had not produced an equivalent amount of progress.

This is more than review. Review asks whether the output is correct; supervision debt is the cost of reconstructing the decisions that produced it before you can judge what "correct" even means. As AI makes execution cheap and abundant, supervision debt may become one of the central design problems of agentic products.

Historically, execution was expensive.

Writing code took time. Producing ten design alternatives took time. Investigating five possible explanations took time. Product design therefore focused heavily on reducing the effort required to execute.

AI changes those economics. An agent can attempt several implementations, run experiments, inspect results, ask other agents for opinions, and revise its approach without getting tired. Generation is fast, inexpensive, and parallelizable.

Human attention does not scale the same way.

The human still provides goals, taste, contextual knowledge, accountability, and the ability to notice that something is subtly wrong even when every local step appears reasonable. Those capabilities become more valuable - not less - as the amount of machine-produced work increases.

One useful, if intentionally provocative, mental model is that an agent has access to many tools. Some are browsers, terminals, databases, and other AI agents. One is the human.

The human is an unusually capable tool, but an inconsistent one. Their attention fluctuates. They get tired and interrupted. Sometimes they want to deeply understand a problem; sometimes they just want it handled. Their willingness to participate can change between tasks or halfway through the same task.

Of course, products do not exist to optimize people as system resources. The entire system exists to serve human purposes. That is exactly why human attention should not be treated as free infrastructure.

Every clarification, approval, review, and handoff spends some of it. Great AI products should spend it where it changes the outcome.

When an agent works autonomously, it does not only create code. It also makes assumptions, rejects alternatives, interprets ambiguity, chooses abstractions, encounters failures, and revises its plan.

If I was not present for that process, I incur a re-entry cost before I can exercise useful judgment.

That cost is not proportional to the amount of output. A thousand-line migration may be easy to review if it is mechanical and familiar. A ten-line change may demand intense scrutiny if it quietly establishes a new product rule.

This makes common measures of agent productivity misleading. Tokens generated, tasks completed, files changed, and agents running in parallel measure machine activity. They do not tell us how much usable progress occurred - or how much supervision debt was created along the way.

If five agents generate five hours of review work in one hour, the system has not necessarily saved four hours. It may have converted execution time into a queue of unresolved human judgment.

That does not mean autonomy is bad. Like technical debt, supervision debt can be a rational trade. I am happy to reconstruct some context later if an agent can safely complete a large amount of routine work now. The problem is that current products rarely make the trade visible.

A better agent would use AI to compress its own work before returning it to me:

• I changed twelve files, but only two decisions materially affect future behavior.

• This assumption is weak and worth confirming now.

• The migration is routine; the authorization change deserves close review.

• I considered three approaches and rejected two for reasons that still apply.

• Everything is verified except this edge case, where the desired behavior remains ambiguous.

That is different from a chronological transcript or a giant diff. It organizes the handoff around decisions, uncertainty, and the places where human judgment has unusually high value.

Software development is often presented as a choice between two modes.

• At one end, the developer remains close to the code. They make the important decisions, write much of the implementation, and use AI for suggestions or small transformations. They retain a rich mental model but sacrifice much of the speed AI can provide.

• At the other end, the developer describes an outcome and lets an agent work. For a contained migration, a familiar refactor, or a mechanical test expansion, this can be exactly right. The developer does not need to participate in every edit to receive a good result.

The interesting product space is not the midpoint. It is the ability to move between these modes during the work.

I might want to explore a data model closely, delegate its implementation, and return for a careful review of the user-facing behavior. I might begin with a vague problem that requires collaboration, then reach a point where the remaining work is routine and should proceed autonomously. On a low-energy afternoon, I may prefer safe defaults and fewer questions. The next morning, I may want to unpack every architectural tradeoff.

This is how I think about flow in an agentic product: continuity between my intention, my understanding of what is happening, and the forward momentum of the work.

AI can break that continuity in two directions. It can require so much mechanical participation that it provides little leverage. Or it can run so far ahead that I lose the context needed to steer it.

In the first case, I remain informed but become a needless bottleneck. In the second, the work moves forward while my understanding stands still. When the agent returns, I am no longer continuing a thought; I am beginning an investigation.

The right balance varies by person, task, and moment. Autonomy should therefore be a changing relationship, not a permanent setting.

Today, users are largely responsible for calibrating that relationship. We tell an agent whether to plan first, ask questions, make changes, avoid changes, explain itself, keep going, or stop at a decision.

More capable systems should take on more of that calibration.

An AI product could expose an involvement level ranging from “take this and handle it” to “work through this closely with me.” An explicit control would be useful, but it should be only one input. The system could also adapt from the user’s behavior.

A person answering every question with one line may be asking for more delegation. Someone writing several paragraphs, correcting details, comparing tradeoffs, or asking conceptual questions is probably willing to invest more thought. Repeatedly accepting defaults may mean the system should stop presenting low-value choices. Drilling into implementation details may signal that the user is trying to build a durable mental model.

These signals are ambiguous. A short response could mean confidence, distraction, frustration, or exhaustion. They should be treated as evidence, not truth. The useful question is narrow: How much participation does this user appear to want right now, and where would asking for it improve the outcome?

That could lead to several concrete interaction patterns.

• An agent could distinguish routine choices from high-value checkpoints. Naming a local variable rarely deserves an interruption. Choosing whether a feature belongs inside an existing service or establishes a new boundary might. Thirty seconds of human input at the right moment can prevent an hour of work in the wrong direction.

• The product could weigh the value of an interruption against its cost. How consequential is the decision? How reversible is it? How uncertain is the agent? Does the user appear engaged or depleted? Can the decision be safely deferred?

• Handoffs could lead with what changed conceptually, what remains uncertain, and what the user should inspect if they only have five minutes. Evidence, rejected alternatives, implementation details, and the full transcript could remain available through progressive disclosure.

• The system could also preserve the user’s mental model over time: which parts of the architecture they understand, which decisions they participated in, which areas they deliberately delegated, and which unresolved tensions are likely to matter later.

That is more than project memory. It is continuity of understanding.

If the human is the bottleneck, why not eliminate the bottleneck?

Where an AI system can reliably define success, observe the consequences of its actions, correct its mistakes, and close the loop on its own, it should. Requiring ceremonial approval from a person who adds no information is not meaningful supervision.

But removing unnecessary human work is different from making the remaining human work cognitively impossible.

Humans still supply goals, taste, accountability, and judgment under ambiguity. An agent may be able to prove that a change passes every test without being able to decide whether the product should behave that way.

As long as that judgment remains necessary, forcing the human to keep pace with unlimited machine output is poor product design.

The answer is not to slow AI to human speed. The product must metabolize machine speed into a form the human can actually use: closing the loop independently when it can, asking one sharp question before making a consequential choice, and compressing an hour of execution into the few decisions the user needs to understand.

The first generation of AI interfaces largely taught humans to accommodate models. We learned to write the right prompt, supply context, review output, catch mistakes, restate forgotten constraints, and decide at every step how much to delegate.

As systems become more capable, that responsibility should invert.

The AI should increasingly understand when I want to participate, when my judgment is needed, how much context I can absorb, and what I must know to remain an effective supervisor. It should preserve my mental model when that model matters and spare me detail when it does not.

It should be able to run far ahead without leaving me far behind.

I do not think this reduces to one perfect interface or a tidy framework. It is a design stance: human attention is not free simply because the user is present.

Agentic systems will keep generating more code, running more experiments, and coordinating more work than any person can follow directly. The most useful products will not be the ones that display the most machine activity. They will be the ones that convert that activity into a manageable number of moments where a person can still think clearly and matter.

The future of agentic product design is not about making the human move as fast as the machine. It is about using machine intelligence to make human attention go further.