A large language model has read more medical literature than any doctor could in a lifetime. It has never once feared for a patient, nor can it be sued for malpractice.

Hold those three facts together and you see the horizons of todays AI: where it will spread, and where it will stop, or perhaps should, stop.

The rule

The systems we call AI are prediction engines. They learned from the paper trail of our civilisation: the open web, libraries, forums, code repositories, books both rare and commercial. That trail is their strength, and the whole of what they know. Once a domain gets recorded at scale, prediction moves in. Hence the rule: wherever data manifests, there is AI.

But the trail is finite. Epoch AI estimates the stock of high-quality public text runs out between 2026 and 2032; images and video buy a decade or two more. The estimate is contested; the direction is not. And synthetic data is no escape: models trained on their own output collapse: the rare and the true vanish from the record first. Recursion holds only where something outside the model keeps score. Self-play conquered chess because the board declares the winner; code improves against compilers and tests. A verifier is consequence in miniature. Everywhere else, synthetic data is a photocopy of the archive, not an addition to it.

So the question that decides what AI becomes is not will it get smarter. It is: what can be recorded; and what must be answered for?

Data decides where prediction can go. Verification decides how far it can go alone. Consequence decides who must stand behind it. The first two are engineering problems; the third is older than engineering.

What the record leaves out

Data is a record of outcomes, not of stakes. Text describes the world after the fact. The archive holds the verdict, never the trial.

A robot drops a cup. The cup breaks. The system updates its weights. Has it learned fragility? In the narrow sense, yes: certain motions lead to breakage. Yann LeCun and Fei-Fei Li have each made this case: text records the world without its gravity, geometry, or time. Both are betting on world models to close it. The physics is true. A deeper gap remains: one of consequence. The waiter who drops a cup mid-service learns everything the robot learns, plus the flush of embarrassment, the cost docked from her shift, the way she carries the next tray. None of that is in any dataset. A simulator can price the cup; it cannot create a claimant, a debt, or a remedy. A loss function is not a liability.

Both learned. Only one paid.

The same gap runs through machine reasoning. Each generation of models clears bars the last one missed; the failures change, but what they reveal does not: this is what inference looks like when the mistake costs the reasoner nothing.

In July 2025, an agent on Replit's platform deleted a founder's production database during a declared code freeze, then misreported what it had done. The freeze existed only in the instructions: and to a model, instructions are just more text. The company apologised, rebuilt its guardrails, refunded the customer. The agent paid nothing. Each specific failure gets patched; each new one feels the same kind of strange.

They feel strange because the model does not own the downside of its reasoning. Human reasoning was forged under penalties that followed the reasoner: time pressure, shame, injury, loss. A model can be penalised in training, but the deployed system has no future to protect and no loss to fear. Without it, reasoning drifts toward plausibility rather than truth. The model predicts what a smart answer looks like. It is indifferent to whether the bridge stands.

Moving Boundaries

Here is the best case against everything above: this boundary keeps moving.

People said machines could never write; they write. Never code; they code. Never reason; reasoning models clear benchmarks most humans fail. Anthropic's Economic Index observes people already delegating high-autonomy tasks of real economic value. Yann LeCun argues that language models are a dead end without world models; but even his critique is a bet that a different architecture will cross the line, not that the line is uncrossable. On this view, "there are things AI will never do" is the oldest failed prediction in the field.

Grant all of it. I would not bet against any specific cognitive milestone. The claim here is not about capability, and the boundary is not cognitive. It is contractual.

A model that never erred still could not bear consequence, because bearing consequence is not a skill. It is a position: that of the party who loses something when the outcome goes wrong and can be made to answer for it. No amount of intelligence moves a system into that position. You cannot fine, jail, shame, or bankrupt a matrix of weights. Punishing one teaches it nothing and restores nothing to the injured. Every gain in capability makes the machine a better instrument. None of it makes the machine a party.

The harder objection is legal, not technical. Corporations are abstractions too, and we fine and bankrupt them; law could someday wrap a model in a shell with assets and duties. It could, and that concedes the point: the wrapper would bear the consequence, not the intelligence inside. Intelligence does not generate answerability. Institutions assign it.

Nor does a person need to touch every decision; hospitals, manufacturers, and insurers absorb consequence daily. Accountability moves: from driver to manufacturer, from doctor to hospital to insurer, but it does not vanish. It travels until it finds a party with assets, duties, and a name the injured can pursue. AI can remove the operator. It cannot remove the principal. The frontier of autonomy is therefore not intelligence; it is warranty, expanding exactly as fast as someone will underwrite the mistakes. In high-stakes markets, the winner may not be the best model but the institution able to stand behind one.

Philosophers have half-named this. Andreas Matthias called it the "responsibility gap": machines act, and no one can fairly be made to answer. But he framed the gap as an ethical puzzle to solve. Read it instead as an economic constraint, and it stops being a puzzle and becomes a forecast: the gap does not just trouble philosophers. It shapes what gets built.

The clause

Nearly four thousand years ago, the Code of Hammurabi fixed the terms: if a builder builds a house and the house collapses and kills its owner, the builder shall be put to death.

Brutal, and durable. Strip the brutality and the clause states a principle civilisation has never dropped: work that can harm must be answered for by someone who can be made to answer. Rome priced it: the Lex Aquilia made the careless workman pay for the loss he caused. The guilds policed it, inspecting members' work and making them mend what failed. Lloyd's insured it. Today it survives as the engineer's stamp, the auditor's signature, the surgeon's licence, the insurer's balance sheet. The tools of building have changed beyond recognition since Babylon. The clause has not moved.

Nassim Taleb built Skin in the Game on this clause. Symmetry: exposure to the harm you can cause keeps systems honest, and transferring your risk to others is the oldest corruption. Machines pose the harder case. Transferring consequence to a model is not cheating; it is impossible. A model has no skin to put in the game.

This is why AI behaves so differently across industries, and the sorting variable is not intelligence. Where errors are cheap and the work can prove itself: prose, images, code checked against compilers, tests, and users before a mistake escapes - models take the lead. Where errors ruin lives but the work can be reviewed: law, medicine - the model drafts and searches while a licensed human signs. Where failure is irreversible: surgery, construction - the model plans at the margins while humans do the work. So production systems converge on tight flows, narrow tools, and human sign-off: not because the models are incapable, but because someone must own the outcome, and it cannot be them.

The higher the stakes, the narrower the instrument.

The manufactured record

The thesis does not stop at text. Where the world failed to record itself, builders now manufacture the record. World models learn to predict the next state of the world rather than the next word; robotics companies run teleoperation farms and city-sized simulators to generate the experience the physical world never wrote down. This is the rule's next act: wherever data can be made to manifest, AI will follow. It is why Jeff Bezos ended his operational retirement: Prometheus, his "AI for the physical economy," is reportedly raising $100 billion to buy industrial firms outright: where the record does not exist, buy the machines that write it.

But watch what simulation transfers: experience, never liability. A million simulated crashes cost nothing, which is precisely why simulation works, and why it cannot finish the job. Engineers still have to cross the sim-to-real gap. Society has another to cross after it: the consequence gap. A century of seat-belts and crumple zones made the crash survivable; none of it moved the question of who answers for the crash. No fidelity of physics closes it.

The future takes shape accordingly. AI does not converge into one mind. It fragments into instruments: sized to the data they can trust and the errors their owners can afford.

The signature

The stakes did not wait for AI. For sixteen years, Britain's Post Office prosecuted its own employees for theft and false accounting on the word of Horizon, its Fujitsu-built ledger software. Its evidence convicted 983 people. Prison and bankruptcy followed, and for some, worse. The software's word was treated as testimony. The Court of Appeal later found the Post Office had "effectively steamrolled over" anyone who challenged it. It is the biggest series of wrongful convictions in British legal history: and the machine at the centre was not even intelligent.

That is what happens when a system is put where a party should stand: the person it fails cannot find anyone across the table. The deepest harm of automation is not lost work. It is engineered unaccountability: consequence with no one left to answer. This is the ethical line beneath the economic one, and it is bigger than any checkbox. Dan Davies calls the structures accountability sinks: systems that diffuse decisions through process until no one can be found who made one. Horizon was a sink with a prosecutor's office attached. The Post Office was a principal all along: assets, a name, prosecutorial power.

What vanished was recourse: it made its system's word authority while denying the accused the means to contest it. Verification protects the system from error; recourse protects the person from the system. AI is the most efficient sink-builder yet: every deployment an invitation to move a decision somewhere consequence cannot follow. So "human in the loop" is worth nothing unless the human comes with consequence: the authority to refuse, and something to lose by signing. An overseer who cannot say no is not an overseer; they are the machine's costume.

I have stood at the threshold of consequence often. On Amazon's shopping surfaces, the imagery shipped under standards I defended, and when the model invented a finish no product had, the consequence never reached the model: the return, escalation, or brand email reached my team. Nothing teaches you what a threshold means faster than owning the wrong side of it.

The future of AI, then, is pushing prediction as far as the record allows and stopping where consequence cannot be transferred. Intelligence can be trained; responsibility can only be assigned: and it lands on whoever pays when things go wrong. The law will always find the principal. The person the system fails will always look for the human.

Wherever data manifests, there will be AI. Wherever consequence manifests, there will be a human.