Court decisions are heavily dependent on the pleadings, briefs, arguments etc., summitted by the lawyers under the common law framework. Therefore, if the lawyer or self-represented litigant relies on any Artificial Intelligence (AI) tools particularly generative AI i.e., Large Language Models (LLMs) and modern agentic AI at some stage in the preparation of the case, there is substantial risk that hallucinated citation of information may find its way into the decision without the judge ever typing a prompt.
The Supreme Court (SC) of the US state of Georgia recently found that the impugned order of the trial court in Payne vs. State case contains several legal citations that either didn’t exist or didn’t actually support the arguments being made. Later, it revealed that the draft of the proposed order was prepared by the prosecutor with the aid of LLM tool. She didn’t cross check the citations before submitting to the judge. The trial judge also signed it blindly in good faith without further verification. Consequently, the SC of Georgia admonished the prosecutor and suspended her for six-month from practicing before the state’s apex court and ordered for academic training on ethics, brief writing and the proper use of AI. The state justices urged trial judges to examine proposed court orders “with the understanding that artificial intelligence software, with all of its potential risks and benefits, may have been used”.
It is noteworthy in the case that the order of the trial court was not structurally defective or unusual. The only unusual thing was that the law cited in the order did not actually exist. This contamination questions the sanity of the process. It is also worth noting that the Payne case is not the first time that such an incident has occurred in the world. In March 2026, in another case involving a pet custody case ( In re Domestic Partnership of Torres Campos and Munoz), a California appellate court upheld a trial court order relying upon non-existent cases as the appellant had relinquished his right to challenge the decision. Although the appellate court fined the attorney drafted the proposed order. There too, the trial court judge signed the proposed order containing fictitious citations. Additionally, in last year, in a divorce case (Shahid v. Esaam) also in Georgia, a trial court similarly signed an order drafted by a lawyer; where two fake or non-existent cases were cited. The Georgia Court of Appeals later vacated that order.
In the illustrated cases, only the lawyers were punished as it is a non-delegable duty where attorneys are personally responsible for the accuracy of all parts of their filing, whether or not they use technology to create it. Nonetheless, the trial judge knew all along that the proposed order needed to be reviewed. Although in practice, this process of subsequent scrutiny may seem like a burden for a court due to several cognitive biases arising out of unusual workload and various other institutional limitations.
Apart from the above illustrations, there are a long list of LLM hallucinated contents in judicial proceedings. Perhaps the journey started with Mata v. Avianca where the attorney presented court filings citing fake legal precedents generated by ChatGPT in 2023 in the Southern District of New York. A database tracks 19 court decisions until 7 June containing LLMs generated hallucinated citations.
Why is this trend not stopping?
One might wonder why the opposing lawyer didn’t bring such contamination before the appellate court. Usually competing lawyers identify hallucinated citations and seek redress against that. Nonetheless, a legal instrument drafted using LLM technology may appear refined, legally cited, credible, organized, sound in principle and satisfactory; yet the draft may contain laws, precedents, or citations that do not actually exist. Despite having professional obligation, repeated caution against AI hallucination and numerous sanctions for citing fake cases, legal professionals keep citing non-existent cases and references due to misplaced trust on AI systems.
Human cognitive tendency such as heuristic, automation bias, confirmation bias, anchoring, framing effect, cognitive offloading and so on also contributes to this on-going tendency. In addition, AI hallucinations do not look like ordinary human mistakes. AI outputs look highly plausible, confident, and convincing that motivate human psychology to trust them. As a result, there are high risks that the opposing lawyer and judge may not suspect each and every citation. Moreover, opposing counsel and judge are not immune to the same cognitive biases as the filing lawyer. The professional duty to verify only triggers when the lawyer or judge knows or suspects an error, whereas AI hallucinations are designed not to trigger suspicion which make it much harder to recognize. That emerges as a great challenge in the legal profession.
Why this practice is concerning for the British legal system
Firstly, unlike civil law systems where judges actively investigate facts, Britain’s adversarial common law system depends on lawyers’ written pleadings, briefs, and skeleton arguments to prepare the judgment. Secondly, court backlog are at record level in Britain now while the Ministry of Justice is actively encouraging AI adoption in legal proceedings to clear them. In the environment of such backlog and institutional pressure for quick disposal, lawyers may be tempted to use LLM systems to draft submissions faster and may have less time to verify every citation. Similarly, judges managing voluminous dockets cannot be expected to audit every citation in every submission. Ironically, the push to reduce backlogs may accelerate the very contamination it seeks to remedy.
The risk of AI hallucinations becoming legal authority
When a lawyer files a document with non-existent case information, that undoubtedly creates a serious problem in the court proceeding. But when a judge directly imports information from lawyers’ submission and signs a court order without careful verification, the problem becomes even more serious. That hallucinated contents become the court’s statement and carry legal authority.
Again, a lawyer’s error can be detected by an opposing party or by the superior court and subsequently remedied by the appellate court. Although the same cannot be done when a superior or appellate court with no further recourse for appeal also relied on lawyers’ presentation. The higher the court, the greater the risk that an uncorrectable LLM‑induced error will embed itself into the fabric of English law. Such a scenario would not merely be a technical failure; it would constitute a profound institutional crisis, capable of eroding public confidence from the judiciary and challenging the legitimacy of the process.
Nevertheless, the dependence on this new and emerging technology is increasing. Irresponsible and uncritical use of these tools could be detrimental as it may invisibly undermine due process and fairness in the court outcomes. It is clear that the danger of issuing a corrupt verdict does not decrease in the age of AI albeit judges rely on their own conscience and even do not employ a single prompt to these scientific tools personally. Hence, this is high time to think critically as to how we can achieve efficiency in the judiciary without weakening fundamental principles of justice.
Published on Bonik Barta as Sub-editorial on 28 May 2026.
Published in the Law and Our Rights page of The Daily Star on 12 August 2026.
