A few months ago, a Chinese provincial government issued a public notice announcing certain policy adjustments. The notice cited, as legal authority, a specific statutory provision. The provision, it turned out, did not exist. Not a misnumbered citation, not a typo—the statute itself was a fabrication. Someone in the office had drafted the notice with AI assistance, the AI had invented a plausible-sounding legal reference, and no one had checked.
The notice circulated for some time before the error was discovered. It was eventually withdrawn and reissued with a corrected reference, but by then it had been widely cited, screenshotted, and forwarded. The corrected version exists. The original version also exists, in archives and on social media and in the memories of people who saw it. The information ecosystem now contains both.
This is not the first incident of its kind. American courts have been quietly issuing reprimands to lawyers who file briefs with fabricated citations. Judges in various jurisdictions have themselves been caught using AI to draft portions of opinions, occasionally with fabricated authority slipping through. The pattern is widening.
I want to write about this, because it is treated in most coverage as a technology problem—a quirk of AI models that will be solved by better training. I do not think that’s the right frame. It is an institutional trust problem, and institutional trust takes generations to build and months to lose.
The category that was supposed to be safe
For most of modern legal practice, there have been categories of documents that we treated as ground truth. Statutes, regulations, published cases, court orders, official government notices. These documents were not infallible, but they had a particular quality: they had been produced by an institutional process designed to filter out the kind of casual error that appears in private correspondence or commentary.
The trust didn’t come from the documents being right in every detail. It came from the process behind them. A statute had been drafted by a legislative staff, reviewed by lawyers, voted on by representatives, codified by an editorial process. A court opinion had been drafted by a judge, reviewed by clerks, filed with the court, and added to official reports. Each step provided a small but real check against error.
When you cited a statute in a brief, you didn’t have to verify that the statute existed. The institutional process had already done that work. You could spend your effort arguing about what the statute meant, not whether it was real.
This is the property that AI-assisted drafting is quietly destroying.
When a government office uses AI to draft a notice, and no one in the institutional chain actually verifies the citations, the institutional check has been removed without anyone announcing it. The document still looks the same. The official seal is still on it. The signature is still real. But the underlying process has been hollowed out. The document might be real, or it might contain plausible-sounding fictions. There is no way to tell from the document itself.
Why this is worse than ordinary error
I want to be precise about why fabricated legal references inside official documents are different, structurally, from ordinary errors.
An ordinary error is correctable. If a government office cites the wrong section of a real statute, anyone can look it up, identify the error, and request a correction. The correction is straightforward because the underlying authority exists; the citation was just wrong.
A fabricated reference is harder to detect and harder to correct. When the AI invents a statutory section that doesn’t exist, the document is internally coherent. The reference looks like a real reference. Anyone trying to verify it has to look up a section that doesn’t exist, fail to find it, and then determine whether they are looking in the wrong place or whether the reference is fake. This takes orders of magnitude more time than verifying a real reference. Most people don’t do it. The error propagates.
The error pollutes downstream materials. Once a fabricated reference appears in an official document, it gets cited in commentary, in legal databases, in articles, in subsequent official documents. The fake reference acquires a kind of secondary existence. It exists in the network of citations even though it doesn’t exist in the actual law. Future searchers find it referenced. Some assume it must be real because so many sources mention it. The hallucination becomes part of the apparent legal landscape.
Trust in the source category degrades. Once it is publicly known that a class of official documents can contain fabricated authority, the implicit trust that justified treating them as ground truth begins to erode. Lawyers who used to rely on official documents start having to verify them. Verification costs add up. The functional value of the institutional product drops.
This is the deepest damage, and it is the hardest to repair. The trust that made the institutional documents useful was a cumulative property, built over many decades of careful institutional practice. Once eroded, it is not restored by a press release. It is restored by another generation of careful practice, and only if no further incidents occur in the meantime.
The legal-system version of supply-chain contamination
There is an analogy I have been turning over.
The food supply chain is built on layered trust. The farmer trusts the seed supplier. The processor trusts the farmer. The retailer trusts the processor. The consumer trusts the retailer. Each layer adds a small institutional check; no consumer has to test their food for arsenic, because the layered system has done that work.
When a single contamination event occurs—say, a salmonella outbreak in spinach—the immediate damage is to the people directly affected. But the deeper damage is to the trust structure of the entire supply chain. Consumers who used to grab spinach without thinking now hesitate. Retailers run additional inspections. Processors demand more testing from farmers. The cost of producing safe food rises across the system. The convenience that the trust structure provided is eroded for everyone, not just for the people directly poisoned.
AI hallucinations in official documents are doing the same thing to the legal information supply chain. Each individual incident is, on its face, fixable: withdraw the document, issue a correction, sanction the responsible party. But the cumulative effect is to introduce uncertainty into the entire system. Lawyers can no longer rely on official sources the way they used to. The cost of verification rises for everyone. The convenience that institutional trust provided is gradually eroding for the whole profession.
And unlike a salmonella outbreak, this contamination is invisible. There is no test for “this document contains a fabricated citation” short of manually verifying every reference. The contamination spreads quietly. By the time we recognize how widespread it has become, it will already be everywhere.
What lawyers need to start doing
I want to draw out three implications for legal practice, because I think most lawyers are not yet adapting their workflows to the world we are now in.
First, verify citations in materials you used to trust. Court orders, regulations, government notices, even commentary from established sources—all of these can now contain AI-introduced errors. The verification step that used to be unnecessary is now necessary. This adds time to legal work, but the alternative is filing briefs that rely on fabricated authority.
Second, build internal protocols against your own AI-introduced errors. Most lawyers using AI are not deliberately introducing fake citations into their own work. They are inheriting them from AI tools that did. The protocol I have settled on: any citation that came from AI gets verified against the primary source before it appears in any document I sign. No exceptions. This sounds obvious but is, in practice, not what most AI-using lawyers do.
Third, treat AI-introduced errors in opposing counsel’s work as opportunities. When the other side is using AI carelessly, their briefs may contain fabrications. Verifying their citations is now a high-leverage activity. A motion to strike a brief that relies on fabricated authority is a powerful procedural weapon. As more lawyers use AI carelessly, this opportunity will appear more often.
The longer view
I don’t know how this resolves. The most optimistic version is that institutions adapt: AI is used carefully, with explicit verification protocols, and the integrity of official documents is restored to something close to its prior level. This is possible but it requires deliberate effort, and most institutions are not yet making that effort.
The more realistic version is that the trust in official documents continues to erode for some period—five years, ten years, perhaps longer—until the cumulative cost becomes high enough that institutions are forced to invest in serious verification practices. In the meantime, individual lawyers and parties absorb the cost of verifying what used to be ground truth.
The pessimistic version is that we never recover the prior level of trust. The information ecosystem becomes permanently more uncertain. Legal practice becomes more expensive across the board because every reference has to be independently checked. Some categories of legal work become uneconomic at current rates.
I think the realistic version is what we get. The trust degrades, the costs rise, the profession adapts, but the adaptation never quite restores the prior equilibrium. We will look back in fifteen years on the early 2020s as the period when one of legal practice’s quiet foundations—the reliability of official sources—was permanently weakened.
The provincial government notice I started this piece with was a small incident. Most of the people who read it have already forgotten it. But the lessons it carried, and the pattern it represents, are going to define legal practice for the rest of our careers.
Part of an ongoing series on the institutional consequences of AI in legal practice. Related: why a generation of associates is becoming AI’s servants.
If you’ve seen institutional AI errors that haven’t been widely reported, email [email protected]. I’m tracking them.