The Review Queue Is the AI Benchmark

The AI industry’s preferred unit of progress is the thing produced: tokens, pull requests, prototypes, maybe a little dashboard with an upward-sloping arrow. Engineering’s unit is the thing survived: the service that stayed correct under load, the patch that did not become a midnight archaeology expedition, the contributor who can explain why a change exists. Those are not the same metric, which is awkward for anyone selling autocomplete as industrial policy.

Josh Collinsworth’s survey of the productivity literature and his own agent-assisted work makes the distinction plainly. He describes getting more done in the visible sense—refactors, upgrades, features, tests—while sometimes gaining neither durable user value nor understanding. The problem is not that a model can never help. It is that a generation event is being confused with completed work. As he puts it, “The more you take a big-picture, holistic view of productivity, the more gains from LLM usage shrink, or even go negative.”

Open source has encountered this accounting reality first because its review capacity is volunteered and finite. Otto Kekäläinen reports that projects and platforms have begun restricting or banning AI-generated material, not because maintainers have mistaken themselves for eighteenth-century Luddites, but because untrusted output externalizes verification onto whoever still cares about the codebase. That is the actual production function: cheap text in, expensive judgment out. Kekäläinen’s account cites Greg Kroah-Hartman’s blunt assessment that “even with the best of the current and next generation tools, at least 1/3 of the results they generate are flat out wrong or harmful.”

The enduring bottleneck is therefore not code emission. It is epistemic ownership: knowing what changed, whether it is right, and who is on call when reality files its bug report. Models can make the first step absurdly cheap. They have not made the others disappear; they have merely put them in a smaller font beneath the demo.

Debian’s answer to generative AI is almost offensively adult. Its membership selected a position called Responsible Use of Generative AI: neither a project-wide endorsement nor a ban, but permission bounded by ordinary standards. The project “neither endorses nor prohibits the use of generative AI tools”, while requiring contributed work to meet the same expectations for quality, correctness, maintainability and legal compliance.

That may sound like policy oatmeal until one notices what it refuses to subsidize: magical thinking. The adopted resolution says “The use of a generative AI tool does not diminish the contributor's responsibility for the work they submit.” In other words, the package maintainer is not required to accept a defect merely because it was conceived by an expensive stochastic parrot with a tasteful IDE integration.

This is a useful model for organizations rushing to write either sweeping AI mandates or performative prohibitions. Tool provenance can matter—for legal, security, and community reasons—but the operational question is simpler: can the submitter understand, test, explain and maintain the result? If not, the organization has not automated engineering. It has automated the creation of review tickets.

The policy disagreement in open source is real, and the supplied reporting does not establish that every AI-assisted contribution is bad. But the divergence itself is informative. Senior maintainers are pricing a cost that executive scorecards often omit: every questionable patch consumes attention from people capable of finding its non-obvious failure modes. Attention, unlike tokens, does not scale because a vendor has renamed a chat box an agent.

Samsung’s LPDDR5X processing-in-memory design offers a more honest version of the same story: moving compute closer to data can unlock impressive local capability, but capability promptly arrives with a suitcase full of systems problems. The design described by Chips and Cheese puts a PIM block in each of 16 DRAM banks, exploiting internal bandwidth that the external memory bus cannot expose. The article gives 614 GB/s across those banks, compared with 76.8 GB/s for regular accesses under its stated comparison. That is a meaningful architectural proposition.

Then software enters the chat, immediately followed by cache coherence, speculation, prefetching, multitasking and interrupts. Samsung’s approach uses reserved rows and standard LPDDR5X commands to change what reads and writes mean. PIM reads can trigger computation and alter vector-register state; data produced inside memory can be invisible to ordinary caches. The article reports Samsung’s recommendation that PIM memory be mapped uncacheable. Congratulations: the RAM has become a peripheral, which is less a victory lap than a request to meet the operating-system team.

There is real elegance in preserving compatibility with a standard memory controller, and there is real power in avoiding data movement. But the PIM blocks are constrained SIMD engines: broadcast setup, identical operations across banks, and no direct bank-to-bank exchange. The host still has to move data through normal DRAM operations when banks need to communicate. Bandwidth is not abolished; it has simply been promoted into a coordination problem.

That should sound familiar to anyone deploying coding agents. In both cases, the headline number measures a local capability—TOPS near memory, code generated per minute—while the hard work migrates upward into orchestration and correctness. Hardware is merely more candid about it: once an ordinary load can have side effects, nobody gets to pretend the runtime is a detail.

There is, however, a constructive path between banning the tools and treating them as colleagues with unusually good typing speed. UCL and the African Institute of Mathematical Sciences have released an adaptable university-level AI Research Foundations Toolkit that joins instruction on building and fine-tuning LLMs with sociology, philosophy, ethics and locally grounded case studies. It includes more than 50 hours of material and was shaped through workshops involving educators, students and researchers across Africa. The project is explicitly designed for adaptation rather than export-by-PDF.

That design choice matters. “Responsible AI” is often the corporate equivalent of placing a fern beside a data centre: soothing, greenish, and not especially connected to the power supply. A curriculum that treats technical design and social consequence as one problem is closer to the real work. As one project leader says, “This toolkit provides educators a means of teaching the technical and the social together”. Another insists that “teachers didn’t want another rigid textbook imposed on them.” Sensible people, apparently, still exist.

For CTOs and engineering leaders, the equivalent is not an ethics slide at kickoff. It is training and process that make people better at judging outputs: evaluation against real workloads, explicit ownership, review time, rollback paths, and measures that capture rework and reliability rather than just production volume. Collinsworth’s warning is worth keeping nearby: “The less you understand, the more you trust AI. But the more you trust AI, the less you understand.”

The industry’s next maturity test is not whether it can make models generate more. It plainly can. It is whether institutions can preserve the human ability to know when generated work is wrong—whether that work appears in a Debian patch, a production repository, or a DRAM bank doing maths behind the cache controller’s back. The future may be agentic. The pager will remain emphatically biological.