In the wake of the explosive rise of artificial intelligence across industries, one of the most consequential arenas where its advocates now see promise is the U.S. judiciary. A recent article in Judicature, a law journal associated with Duke University’s Bolch Judicial Institute, argues that generative AI tools might help judges manage heavy workloads, sift through reams of evidence, and summarize complex case materials. The piece acknowledges potential pitfalls — particularly confidentiality concerns — but ultimately suggests that, properly managed, AI could be a useful assistive technology for judges.
Yet this tempered optimism belies a deeper truth: the technology is simply too immature, too opaque, and too risky for routine judicial use. Below the surface of the Judicature article’s case studies and hypothetical efficiencies lies a host of legal, ethical, and practical dangers that — if ignored or underestimated — threaten core principles of justice, fairness, and due process.
1. AI Is Not Reliable Enough for Legally Consequential Judgments
One of the central claims in favor of judicial AI is that it could help judges locate documents or summarize testimony in long, document-intensive cases. The Judicature author reports experimenting with a third-party e-discovery tool (Merlin) to extract relevant testimony and assist in drafting orders — and found that AI could produce summaries quickly.
But speed isn’t accuracy.
Generative AI systems — including the large language models (LLMs) that underpin tools like ChatGPT — are notorious for producing hallucinations: plausible-sounding but entirely fabricated information. A recent study found that leading AI legal research tools hallucinated between 17 % and 33 % of the time, even when built for the legal market. Another benchmark found that general AI models hallucinate on legal queries in a startling percentage of cases (up to 58 % or more).
These are not trivial errors. In court — where lives, liberties, and billions of dollars can hinge on precise legal analysis — false or misleading summaries, incorrect citations, misclassified evidence, or invented legal principles are unacceptable. Unlike clerical or administrative tasks, judicial decision-making demands an unwavering commitment to accuracy. Relying on a system that systematically produces nonsense — even occasionally — is a risk no judge should take.
2. Judges Are Already Struggling With Less Complex Technologies
Before we even talk about generative AI, many courts have adopted automatic speech recognition (ASR) systems to transcribe hearings and trials. These tools are supposed to produce verbatim records of proceedings, yet studies and court experiences show they don’t. Misheard words, incorrect names, and garbled technical terms are the norm rather than the exception with current ASR systems.
If judges are already encountering problems with ASR — which only attempts to turn spoken words into text — it strains credulity to assume that far more complex generative AI tools are ready to assist in interpretation, summarization, or research without producing significant errors.
The Judicature article’s vision assumes a level of baseline technological competence and accuracy in courts that does not yet exist. Before using AI for summaries or analysis, judicial systems must first ensure that the technologies they already rely on are delivering truly reliable, verbatim results — and right now, they aren’t.
3. AI’s Black-Box Nature Threatens Transparency and Accountability
One of the most profound problems with current AI technology is its opacity. Unlike a human law clerk whose reasoning can be examined and critiqued, AI models — particularly commercial, proprietary ones — are black boxes. Their training data is controlled by a handful of corporations, and the decision processes behind their outputs cannot be audited or explained.
In essence, we would be asking judges to incorporate into the judicial process a tool whose inner workings are largely unknowable. What if an AI’s output reflects biases learned from its training data? What if it relies on outdated, wrong, or discriminatory information? Simply telling judges to “double-check” the output doesn’t solve the core problem — if the judge doesn’t understand how the AI arrived at a conclusion, how can they reliably validate it?
This opacity also extends to data retention and privacy. When confidential legal information is fed into an AI system, that information is not “used and forgotten.” Instead, it becomes part of a cloud-based dataset that may be stored indefinitely and used in future AI training or outputs. Legal data — including privileged client communications, sealed evidence, or sensitive personal information — could inadvertently become part of the broader AI ecosystem. Once this happens, there is often no effective way to retract it.
These risks are profound not just for individual litigants, but for the legitimacy of the entire judicial system. Judges must enforce procedural fairness and protect confidentiality; they should not be opening the courthouse doors to a technology that permanently captures and repurposes privileged and sensitive information.
4. The Legal Profession Is Already Struggling With AI Hallucinations
Even outside the courtroom, the legal community is facing a rising tide of problems caused by generative AI’s errors. Lawyers and even litigants have submitted briefs with fictitious case law and false legal citations generated by AI, forcing courts to sanction them — including fines and reprimands.
These aren’t isolated anecdotes. Analysts have documented more than 120 such incidents involving hallucinated legal authorities, and judges around the world have publicly warned against relying on AI for research without careful verification. The underlying issue is clear: these tools are not databases of vetted legal authority but probabilistic generators that can whip up convincing, but entirely false, references.
If trained attorneys cannot safely use AI for legal research without creating risk, the notion that judges should begin to integrate generative AI into their work — even in a limited support role — seems premature at best and reckless at worst.
5. Only a Few Corporations Control the Tech and the Data
Another issue largely overlooked in optimistic commentary about judicial AI is who controls the underlying models and data. A handful of tech companies dominate the development and deployment of large language models and the datasets they rely on. These corporations decide what information is included, how it is curated, and how the AI “learns.”
This concentration of control poses both ethical and democratic concerns when such systems enter the judiciary. Judges, as neutral arbiters, should not be dependent on opaque systems built by private companies with unknown priorities and profit motives. The law demands transparency and accountability, not reliance on commercial entities whose interests may not align with the principles of justice.
6. Ethical and Procedural Questions Remain Unanswered
Even proponents of AI in courts acknowledge the need for caution. Many legal ethics bodies have issued guidance requiring lawyers to verify AI outputs and ensure client confidentiality — but these guidelines are inconsistent across jurisdictions and far from settled. The fact that the profession itself does not yet have uniform standards for lawyers using AI should give pause before we extend any AI use to judges, whose decisions have far greater impact.
Moreover, ethical rules governing ex parte communications, confidentiality, and impartiality may be implicated if judges consult an AI system whose outputs are influenced by undisclosed training data or communications that exclude the opposing side.
7. Judges Are Not Replacing Machines — But They Are Already Too Close
Some argue that generative AI will never truly replace a judge’s human value judgment — that it will only supplement or accelerate clerical tasks. That may be true in principle, but in practice, the line between clerical assistance and influence on judicial reasoning is slippery.
Cognitive psychology has shown that even minimal exposure to suggestions — even when known to be flawed — can influence human decision-making. If a judge receives a summary, a draft finding, or a suggested line of reasoning from an AI tool, it almost inevitably anchors them — skewing their thinking even if they consciously try to correct it.
In contexts where fairness and accuracy are non-negotiable, that risk alone should be enough to argue for restraint.
8. The Judiciary Should Wait — Not Leap
There is no question that generative AI holds promise in certain domains — from legal research and administrative tasks to possibly improving access to justice for unrepresented litigants in the distant future. But courts are not laboratories for beta-testing emerging technology.
Efficiency is an admirable goal, but justice is not measured in seconds saved or paragraphs drafted. It is measured in correct outcomes, procedural fairness, and public confidence in the rule of law.
Given the technology’s persistent hallucination problems, lack of transparency, data privacy risks, ethical uncertainties, and the fact that even trained legal professionals struggle with AI’s flaws, the judiciary should adopt a moratorium on integrating generative AI into substantive judicial functions until the technology matures and robust safeguards are in place.
This is not an argument against innovation — but an appeal for prudence. The stakes are too high, the technology too immature, and the potential harms too grave to rush into a future where computers quietly shape justice.
Judges should lead with caution, not curiosity, when it comes to artificial intelligence.
Disclaimer
Disclaimer: This article reflects the author’s analysis and opinions on emerging courtroom technologies. It is not intended as legal advice. All references to artificial intelligence systems are general in nature and do not allege misconduct by any specific company, court, or individual. Readers should independently evaluate evolving technologies and applicable laws before drawing conclusions or implementing any practices.
STENOIMPERIUM 