AI is rapidly becoming part of everyday legal practice. From legal research and drafting to document review, AI has the potential to transform how lawyers work. But alongside those benefits is a growing problem: AI hallucinations.

The latest example comes from the Crown Prosecution Service (CPS), which has admitted submitting two completely fictitious legal authorities to the High Court after they were apparently generated by AI. While the errors were caught before the hearing and had no impact on the outcome of the case, the incident is another reminder that hallucinations are becoming one of the biggest practical risks facing the legal profession.

Even the CPS?

The case arose during appeals against extradition to Romania. In its written submissions, the CPS cited two legal authorities that did not exist.

The false authorities first appeared in the CPS’s grounds of opposition before being repeated in further submissions prepared by junior counsel. They were not identified during the court’s permission stage and remained in the case papers until questions were raised shortly before the hearing.

The CPS accepted that the fake cases were likely generated through the use of generative AI. But rather than blaming the technology, the CPS told the court that the real failure was human. The authorities had not been properly checked before formal submissions were filed.

Following the discovery, the CPS apologised to the court, carried out an internal review, examined almost 80 other cases handled by the same lawyer and reinforced its verification processes. The review found no evidence of similar issues elsewhere.

A growing pattern

The significance of this case extends beyond the CPS.

Over the past two years, courts around the world have seen a steady increase in cases involving AI-generated hallucinations. Fictional cases, invented quotations and fabricated legal authorities have appeared in court filings submitted by lawyers, litigants and, now, public prosecutors.

The CPS incident demonstrates just how easily these errors can spread through the legal process.

The hallucinated authorities were copied from one document into another, survived multiple stages of review and were before the High Court in more than one filing. As legal commentators have noted, once incorrect information appears repeatedly, there is a natural tendency for later reviewers to assume someone else has already verified it. The mistake quickly becomes embedded, making it far harder to detect.

This cascading effect is one of the greatest dangers posed by generative AI. A single hallucination can easily become part of the official record if proper safeguards are not in place.

The judge’s warning for the legal profession

Mr Justice Sweeting accepted the CPS’s apology and confirmed that there had been no attempt to mislead the court. Because the errors were identified before the hearing, they had no effect on the legal arguments or the court’s decision.

But he noted that the episode should serve as a warning.

He observed that it would be “naive” to assume AI will not become increasingly common in legal work and acknowledged that its use may be both necessary and beneficial. At the same time, he stressed that the case highlighted the risks of relying on AI without appropriate human oversight, particularly for legal research.

His comments reinforce what courts have been emphasising, that lawyers remain personally responsible for every submission they place before the court, regardless of whether AI assisted in producing it.

AI isn’t the problem. Unchecked AI is

The CPS itself made perhaps the most important point arising from the case.

Although AI may have generated the fictitious cases, the technology was not considered the root cause. The real failure was the lack of verification before the documents were submitted.

That is significant because generative AI can significantly improve productivity and assist lawyers with drafting, summarising and research. But it is not a reliable source of legal authority. Every citation, quotation and legal proposition must still be independently verified against authoritative sources.

The legal duty has not changed just because AI has entered the workflow.

What should law firms do?

For law firms, this case is another reminder that AI governance is no longer simply an IT issue. It is a professional responsibility issue.

As AI becomes embedded in everyday legal practice, firms need clear policies governing how AI can be used, mandatory verification procedures for AI-assisted work, appropriate staff training and effective supervision. Lawyers also need to understand where AI is most prone to hallucinate and when additional scrutiny is required.

The courts have consistently shown that they are not opposed to the use of AI. What they expect is that lawyers continue to exercise independent professional judgment.

The CPS incident indicates that even experienced legal professionals working within one of the UK’s largest legal organisations can be caught out if proper safeguards fail.

Hallucinations are no longer unusual headlines. They are becoming a recurring feature of legal practice. The firms that embrace AI need to ensure effective human oversight into every stage of its use.