FOLLOW US

When Arkansas automated its disability assessment system in 2016, it triggered a crisis that exposed one of the most dangerous vulnerabilities in modern government: what happens when a machine makes consequential decisions about human lives, and no one can explain why.

The Arkansas Department of Health Services contracted a software company to build an algorithm that would streamline the process of evaluating disabled patients and determining the level of home care they required. The results were disastrous. In one widely cited case, the algorithm slashed the amount of home care provided to an amputee simply because the patient had no recorded "foot problems." The system failed to grasp that the absence of a foot was itself the problem. Across the board, the program negatively affected nearly half of all Arkansas Medicaid recipients, leaving vulnerable people without the care they depended on.

That case is no longer an isolated incident. It is a preview.

The Federal Government Is Doubling Down on AI

Artificial intelligence is expanding rapidly throughout the federal government. According to a recent report from the U.S. Government Accountability Office, the number of AI use cases across federal agencies doubled from 2023 to 2024. For generative AI specifically, the increase was even more dramatic, growing ninefold in a single year. States including Idaho, Texas, Michigan, and Wisconsin have already encountered their own versions of the Arkansas situation, and similar failures have surfaced at the federal level within the Department of Homeland Security.

The Trump administration has reportedly signaled its intention to integrate AI directly into the federal rulemaking process, which governs how agencies develop and implement regulations covering everything from environmental standards to food safety to workplace protections. The exact scope of this ambition remains unclear, but the direction is not. AI is being positioned as a tool for governing, not just a tool for administration.

This raises a set of urgent legal questions that the American administrative law system is not yet equipped to answer.

The Law Requires Reasoned Decisions. AI Cannot Always Explain Itself.

At the heart of American administrative law sits the Administrative Procedure Act, a federal statute that sets the rules for how government agencies may act. One of its central requirements is that agency decisions cannot be "arbitrary and capricious." In plain language, this means that when a government agency makes a ruling, it must be able to explain its reasoning. Courts are empowered to strike down decisions that lack that rational foundation.

For decades, legal scholars have wrestled with how courts should apply this standard to human decision-makers. But they have largely assumed that the decision-makers in question are, in fact, human. AI changes that assumption in ways that may overwhelm existing legal doctrine.

The core problem is what technologists call the "black box" phenomenon. Many AI systems, particularly large language models and advanced machine learning tools, arrive at conclusions through processes that even their designers cannot fully trace or explain. You can ask an AI to explain why it made a recommendation, but the explanation it produces may bear no relationship to the actual computational process that generated the output. The model can rationalize. It cannot necessarily reveal.

This creates a profound dilemma for judicial review. Under the Administrative Procedure Act, judges are supposed to examine an agency's reasoning and determine whether it holds up. But if the agency's reasoning was shaped by an AI system whose internal logic is opaque, the judge may have no meaningful basis for that review. The stated rationale and the actual rationale may not be the same thing.

Two Supreme Court Principles That AI May Violate by Default

The Supreme Court has established two specific guideposts for evaluating whether agency decisions are arbitrary and capricious. First, agencies may not rely on "improper factors" in their decision-making, meaning considerations that Congress never intended the agency to weigh. Second, agencies cannot "entirely fail to consider an important aspect" of the problem before them. Both principles are designed to ensure that government decisions are grounded in law and reason rather than irrelevant or prohibited considerations.

AI creates serious tension with both of these standards, and here is why: an agency using an AI model to inform its decisions has no reliable way to know what factors that model actually considered. If the model weighted a factor that Congress prohibited the agency from considering, the agency's decision will have incorporated that improper factor, regardless of what the agency believed it was doing.

Amazon confronted this problem directly when it built an AI system to screen job applicants from submitted resumes. The model began penalizing resumes that included the word "women's," a reflection of bias embedded in the training data, which consisted disproportionately of male applicants. Amazon's engineers tried to correct the bias but eventually abandoned the project entirely because they could not guarantee that other forms of bias had not also embedded themselves in ways they could not detect. The lesson was not merely that this particular model was flawed. It was that the team could not be certain any version of the model would be clean.

Now apply that scenario to federal regulatory decisions. When the Food and Drug Administration uses an AI system to evaluate a new drug application, or when an environmental agency uses one to set emissions standards, the agency cannot know which variables the model is treating as significant or how much weight it is assigning to each. The decision may appear reasonable on its face while resting on a foundation that no human reviewed and no law authorized.

The Training Data Problem: A Risk Hiding in Plain Sight

Most existing scholarship on AI and administrative law has focused on the black box problem as it exists within a single model's architecture. But there is a larger and less examined issue: the integrity of the data those models are trained on in the first place.

The majority of federal agencies that use AI are not building it from scratch. According to the Government Accountability Office, more than half of selected federal agencies are primarily acquiring AI tools from commercial vendors rather than developing them internally. That means the AI shaping federal decisions is often built on training data that the government did not collect, does not control, and may not be able to inspect. Commercial licensing agreements frequently shield both the model's architecture and its training data from public or government scrutiny, leaving those assets in the hands of the vendor.

This matters enormously because of what that training data often contains. Modern large language models are trained on massive datasets drawn heavily from the internet, including forum posts, comment sections, social media content, and other forms of user-generated material. ChatGPT, for instance, is trained in part on data sourced from Reddit. These are enormous datasets, and their scale makes them valuable. But their content is unfiltered, and the internet contains vast quantities of material that no human decision-maker would ever be permitted to rely on when making a government ruling.

Racist language, sexist assumptions, discriminatory framing, and culturally embedded biases are widespread across the web. Because these patterns appear frequently enough in the training data, they become statistically reinforced during the model's training process. The model does not "believe" these things the way a person might, but it learns that certain patterns correlate with certain outcomes, and that learning shapes its outputs. When a federal agency then uses that model to inform a regulatory decision, it is, in a meaningful sense, considering those improper factors, even if no human in the process was aware of it.

This is categorically different from the standard black box concern. The worry is no longer merely that a model might infer improper conclusions by drawing on proxy variables. The worry is that the model may be directly incorporating concrete, impermissible information obtained from external data sources into the reasoning that shapes government action. Those factors could include constitutionally protected characteristics such as race or gender, raising the possibility not only of violations of the Administrative Procedure Act but also of the Fourteenth Amendment's Equal Protection Clause.

Can Procedural Safeguards Fill the Gap?

Courts have long grappled with the possibility that human agency officials might construct legally acceptable rationales for decisions that were actually driven by improper considerations. Their response has been to focus on policing the procedures agencies follow as a proxy for ensuring that the underlying reasoning is sound. If an agency followed the right steps, compiled the right record, and addressed the relevant factors on the record, courts will often give it the benefit of the doubt even if the true internal reasoning cannot be fully verified.

Some have suggested applying the same functional fix to AI-assisted decision-making. But there are important reasons to be skeptical of that approach.

The procedural workaround for human decision-makers exists because there is no alternative. Human cognition is imperfect, and no legal system can fully audit what happens inside a person's mind before they reach a conclusion. Courts accept that limitation and work around it. But AI is not inherently exempt from scrutiny in the same way. The black box problem with AI is not a fixed feature of the universe. It is an engineering and transparency challenge, one that can potentially be addressed through better design, mandatory explainability requirements, or restrictions on the types of models agencies may use.

Beyond that, human agency officials operate within a web of institutional constraints designed to promote honest and lawful conduct. Federal employees swear an oath of office. Civil service protections are meant to insulate career staff from political pressure and allow them to document concerns without fear of retaliation. Whistleblower laws provide additional protection for those who expose misconduct. The Freedom of Information Act creates a degree of public transparency over how agencies develop their work. None of these safeguards apply to an AI model.

An Unresolved Legal Frontier

The questions raised by AI in federal decision-making do not have clean answers under current law, but they are not entirely without analogies. Scholars in law and political science have long examined what happens when government agencies outsource core functions to private contractors, and some of the accountability problems that arise in that context parallel the challenges created by AI. Delegating government authority to a third party does not eliminate the government's legal obligations, but it can make those obligations much harder to enforce.

Whether courts will extend that reasoning to AI, require agencies to disclose the training data and architecture of commercial models they rely on, or demand a new statutory framework is an open question. What is no longer open is whether these questions need to be answered. As AI becomes a standard instrument of federal governance, the gap between what administrative law requires and what AI makes possible will only grow wider.

The Arkansas amputee who lost home care because of a foot problem he did not have was not a glitch. He was a warning.