Skip to main content
Ayla
Send us a role

How government agencies test AI before they rely on it

By Javier Dutan, founder of Ayla · published

The short answer

Federal agencies must test high-impact AI before they deploy it and keep monitoring it after, under OMB memo M-25-21. Most organize that work with the NIST AI Risk Management Framework, which sorts it into four functions: Govern, Map, Measure and Manage. A test is independent when the people running it neither built nor sell the system.

open federal roles
19,373
open state roles
24,871

What the federal rules ask

OMB memo M-25-21, issued April 3, 2025, sets how federal agencies govern their use of AI. It names a category, high-impact AI: AI whose output is a principal basis for a decision or action with a legal, material, binding or similarly significant effect on people's rights or safety.

For high-impact AI, the memo lists minimum risk management practices. The ones that call for testing:

  • Pre-deployment testing that reflects expected real-world outcomes. Where the agency cannot see the source code, model or data, it must test another way, such as querying the system and checking what comes back.
  • An AI impact assessment, written before deployment, that includes the results of an independent review by someone in the agency who was not involved in building the system.
  • Ongoing monitoring once it is in use, to catch a system whose performance drifts or whose risks change.
  • Human oversight, so trained people can intervene, and a way for those affected to seek a remedy.

Agencies had until April 3, 2026 to put the practices in place. A high-impact use that does not meet them must be safely stopped, unless the Chief AI Officer grants a written waiver for that system, reported to OMB within 30 days. National security systems follow their own policy.

For hiring tools, OPM’s memo of August 27, 2026 says drafting a job announcement, and screening support where the deciding official reviews the record independently, are generally not high-impact. Before deployment the agency still checks veterans’ preference, accessibility, Privacy Act protections, auditability and a way to ask for reconsideration.

When an agency buys AI, OMB memo M-25-22 asks it to test proposed solutions during source selection where practicable, and to write ongoing testing into the contract, on the agency’s own test data the vendor cannot see, with access and time for independent evaluation.

The NIST framework most agencies use

The NIST AI Risk Management Framework (AI RMF 1.0, NIST AI 100-1, January 2023) is voluntary, but agencies and their vendors use it as the common structure for AI risk work. Its four functions:

  • Govern: who is accountable, and the policies and culture around AI risk.
  • Map: the system’s context, its intended use and who it affects.
  • Measure: testing, evaluation, verification and validation (TEVV). In NIST’s words, systems should be tested before deployment and regularly while in operation.
  • Manage: deciding what to do about each risk found, and tracking it.

NIST is revising the framework under the July 2025 AI Action Plan; version 1.0 is the one in force.

For chatbots and other generative AI, NIST’s Generative AI Profile (NIST AI 600-1, July 2024) applies the framework to the risks particular to them, such as confabulation, data leaks and harmful content. For attacks on AI, NIST’s adversarial machine learning taxonomy (NIST AI 100-2) names the kinds of attack red teams test for, prompt injection among them.

Four kinds of test

Each of these answers a different question.

  • Independent verification and validation (IV&V): does the system do what its vendor claims, on the agency’s own tasks? A team that neither built nor sells it reruns the claims and the mission’s edge cases.
  • Red teaming: how does it fail when someone tries to make it fail? Testers attempt prompt injection, data leaks, jailbreaks and, for agents, tools or permissions used where they should not be.
  • Deployment testing: does it still work where it will run? An open-weight model moved onto an agency’s own servers, or onto a network with no internet connection, is measured again there, compression included.
  • Provenance review: where did the weights, the training data and the software dependencies come from, and can each be traced?

Ask for a report that lists every test, every finding with the steps to reproduce it, and what could not be tested. A finding nobody can reproduce cannot be fixed or checked again.

Defense, states and cities

Defense and intelligence systems follow their own policies.

  • Autonomous weapon systems: DoD Directive 3000.09 (January 2023) requires hardware and software verification and validation and realistic test and evaluation, repeated after changes such as new machine learning, with senior review before development and again before fielding.
  • Other defense AI: test and evaluation follows the program’s acquisition pathway and its program office, so ask which requirements apply.
  • Classified work: testing runs inside the accredited environment, under the program’s security rules, by people who hold the clearance the work needs.

States write their own rules. Since February 1, 2024, a Connecticut state agency may not implement an AI system without an impact assessment (Public Act 23-16). Since September 1, 2025, a Texas agency using a heightened-scrutiny AI system must assess its risks and limits, under standards consistent with the NIST framework (SB 1964). Elsewhere, start with the state CIO’s AI policy or the city’s IT office.

Police and public safety systems that touch criminal justice information also fall under the FBI’s CJIS Security Policy for anyone with access, testers included.

How to buy independent testing

Testing is a service, so it is bought like one. A federal office can buy up to $15,000 on a purchase card, up to $350,000 with simplified procedures, and above that through a schedule, a task order or a subcontract under a prime. Defense prototypes can also use other transactions under 10 U.S.C. 4022.

Two federal resources are built for it. The Chief Digital and AI Office’s Test and Evaluation Services BPA, run with Army Contracting Command–Rock Island, is open for ordering by every federal agency for AI test and evaluation. GSA’s USAi lets agencies compare generative AI models side by side before they buy.

Before work starts, agree in writing what will be tested, with which data, where it runs, who sees the results, and that the tester does not build or sell the system under test.

Where Ayla fits

Ayla tests AI systems for government with US-citizen evaluators, on tests built from its own government data, and recruits evaluators for teams that hire their own.

AI evaluation covers the four kinds of test, AI evaluators the people, and the independence policy the conflict rules.

Common questions.

Do federal agencies have to test AI before deploying it?
For high-impact AI, yes: OMB memo M-25-21 makes testing before deployment one of seven minimum risk management practices, in place since April 3, 2026. National security systems follow their own policy.
What is TEVV?
Test, evaluation, verification and validation: the measurements that show how an AI system performs and fails. The NIST AI Risk Management Framework places it under its Measure function.
What is the difference between IV&V and red teaming?
IV&V checks that a system does what it is meant to, on the buyer’s own tasks, by a team that did not build it. Red teaming tries to make it fail on purpose, to find what an attacker or a careless user could do.