FOLLOW US

Case Insights

• Despite judicial warnings and sanctions dating back to 2023 in Mata v. Avianca, attorneys continue submitting AI-generated documents with fabricated citations to courts nationwide
• Recent cases show AI misinformation has spread beyond legal briefs to expert testimony, with even state Attorney General offices falling victim to hallucinated citations
• California's adoption of new AI rules for judges and court staff signals an urgent need for comprehensive AI compliance training across the legal profession

By Samuel Lopez – USA Herald

The legal profession finds itself at a critical juncture as mounting cases of AI-generated hallucinations continue to plague courtrooms across the nation, despite clear judicial precedents and escalating sanctions. What began as isolated incidents of technological mishaps has evolved into a systemic crisis threatening the integrity of legal proceedings and the credibility of practitioners nationwide.

The Persistent Pattern of AI Negligence

The landmark case of Mata v. Avianca, Inc., 678 F. Supp. 3d 443, 466 (S.D.N.Y. 2023), served as what many hoped would be a definitive wake-up call for the legal profession. The court's sanctioning of an attorney for submitting fake, AI-generated legal citations should have been the shot heard round the legal world. Yet, despite widespread publication and commentary, the pattern continues with disturbing regularity.

Recent cases demonstrate that the problem has not only persisted but expanded in scope. In Park v. Kim, 91 F.4th 610, 614–16 (2d Cir. 2023), another attorney faced potential disciplinary action for including fabricated AI-generated citations. The Missouri Court of Appeals went further in Kruse v. Karlan, 692 S.W.3d 43, 53 (Mo. Ct. App. 2024), dismissing an entire appeal due to multiple fake citations generated by artificial intelligence.

Expert Testimony Under Fire

Perhaps most alarming is the migration of AI hallucinations from legal briefs to expert evidence. The recent case of Kohls v. Ellison, 2025 U.S. Dist. LEXIS 4928 (D. Minn. Jan. 25, 2025), involved the Minnesota Attorney General's office submitting expert testimony containing fabricated citations. Despite the Attorney General's sincere apology and assertion that the office had no knowledge of the fake citations, the incident underscores how even experienced legal institutions can fall victim to AI-generated misinformation.

Similarly, In re Matter of Weber, 2024 NY Slip Op 24258 (N.Y. Surrogates Ct. Oct. 10, 2024), demonstrates that expert reports and affidavits are increasingly contaminated with AI-generated false information, creating a crisis of credibility that extends far beyond individual practitioners.

The Competency Crisis

Legal experts emphasizes that lawyers are ethically obligated to understand the technology they use, as mandated by Comment 8 of ABA Model Rule 1.1 and its state counterparts. Yet when it comes to generative AI, many practitioners fail to grasp both its capabilities and limitations.

"Lawyers need to step up their game, and get with the A.I. program" stated one analyst “It’s imperative that our field uphold its ethical and professional duties when integrating AI. The standards are well-defined, and the fixes are simple.”

The fundamental issue isn't intentional misconduct but rather a profound misunderstanding of how generative AI functions. "I submit that lawyers and other legal professionals who make this mistake simply do not understand how generative AI functions and what is required to use it properly and effectively."

Understanding AI Hallucinations

Generative AI's creative capabilities come with inherent risks that legal professionals must comprehend. These systems can produce output that appears realistic but contains misleading or entirely fabricated information. Citations may reference real sources that don't contain the purported language, or they may create entirely fictional cases and legal authorities.

Professional guidance from California, Florida, New York, and other jurisdictions has consistently cautioned lawyers to become proficient in generative AI before implementation. This requires understanding both the tools themselves and the necessity of verifying all AI-generated output before court submission.

The Verification Imperative

The solution, according to experts, lies in treating AI output with the same scrutiny applied to work from law clerks, associates, or co-counsel. The only way to ensure that a cited case or authority is accurate is to check it carefully and verify the existence, the content, and whether it is still valid authority and not overturned or superseded.

This human-in-the-loop approach ensures that AI functions as "co-intelligence rather than a replacement," allowing legal professionals to maximize efficiency without compromising ethical standards. Lawyers must maintain control, providing human oversight to ensure accuracy, context, and ethical compliance.

California's Regulatory Response

California's recent adoption of new rules governing AI use by judges and court staff signals recognition of the technology's growing influence in judicial proceedings. These regulations suggest a continued need for ongoing training, compliance materials, and specialized tools to help legal professionals navigate the evolving AI-legal landscape effectively.

The regulatory framework acknowledges that as AI becomes increasingly integrated into legal practice, comprehensive oversight mechanisms become essential to maintain judicial integrity and public trust.

Moving Forward: Education and Adaptation

The legal profession stands at a crossroads. The time for learning curves has passed, and practitioners must now fully embrace their ethical and professional obligations regarding AI use. This requires three fundamental elements: sufficient knowledge of AI capabilities, understanding of specific tools, and careful attention to detail.

These requirements align perfectly with existing professional responsibilities that all lawyers should possess. The challenge lies not in developing new skills but in applying traditional legal diligence to emerging technology.

As the profession grapples with these challenges, one thing remains clear: the integrity of the judicial system depends on legal professionals' commitment to accuracy, verification, and ethical AI implementation. The mounting cost of AI negligence extends beyond individual sanctions to threaten the very foundation of legal practice.

The path forward requires immediate action from practitioners, continuing education providers, and regulatory bodies to ensure that artificial intelligence enhances rather than undermines the administration of justice.

Resources:

For legal professionals seeking comprehensive AI compliance solutions and training materials to prevent hallucination incidents in their practice, specialized resources are now available that combine advanced AI knowledge with decades of legal expertise.