Can We Trust AI-Generated Tests?

AI-generated tests can create an illusion of coverage.

Neither the number of tests nor code coverage alone proves that a test suite can detect defects.

Trust requires answering three different questions:

1. Does the test validate the correct requirement?
2. Can it detect an incorrect implementation?
3. Can it run reliably in a controlled environment?

Mutation testing helps answer the second question.

A mutation-testing tool makes small changes to production code and runs the relevant tests against each modified version.

Each modified version is called a mutant.

When a test fails, the mutant is killed: the suite detected the change. When all relevant tests still pass, the mutant survived: the suite failed to detect it. A mutant is marked no coverage when no test executes the mutated code.

Mutation testing therefore does not test the application directly. It evaluates how effectively the existing tests respond to changes in the implementation.

Mutation tools such as Stryker begin with a dry run against the unchanged application. This confirms that the tests pass, discovers the available tests, and can collect per-test mutation coverage before executing mutants.

The value is not the number of mutants alone. The useful result is that a relatively small set of focused tests measurably improved the suite’s ability to detect changes in critical business logic.

Stryker’s default reporting thresholds classify scores of at least 80% as high and scores below 60% as low. However, these are configurable reporting thresholds — not universal quality standards for every project. At Treegress, we believe the mutation score for the selected critical scope should not be lower than 90%.

Surviving mutants need engineering analysis.

An equivalent mutant changes the code without changing its observable behavior. No useful test may be able to kill it, and mutation tools cannot always identify equivalent mutants automatically.

This is why teams should not pursue a 100% mutation score at any cost. Mutation score is a diagnostic signal, not a target that should be optimized blindly.

Mutation testing is generally most effective with fast, deterministic unit, component, service, contract, and API-level tests.

A broad mutation run using browser-based end-to-end tests can become expensive because many code modifications must be evaluated separately, potentially requiring repeated application setup, data preparation, and browser execution.

For larger repositories, a practical approach is to run mutation testing against changed or high-risk code during pull requests and schedule broader runs separately. Stryker’s incremental mode can reuse previous results and execute only the mutants affected by code or test changes.

End-to-end mutation testing should be selective and focused on critical workflows where lower-level tests cannot provide sufficient confidence.

Mutation testing can show that a test detects certain implementation changes.

It cannot prove that the test validates the correct business behavior.

An AI agent could generate a strong test for the wrong expected result. That test might kill many mutants while still misrepresenting what the product is supposed to do.

Agents therefore need more than source code. Their context should include:

Tests generated only from code tend to confirm what the implementation currently does. Tests grounded in requirements can verify what the product should do.

AI agents can help teams analyze changes, identify coverage gaps, generate candidate tests, investigate surviving mutants, and recommend stronger assertions.

But generated tests should still be reviewed before they are merged. Release validation should remain reproducible through a stable testing framework, controlled data, consistent environments, and deterministic CI/CD execution.

Teams should not trust AI-generated tests merely because they compile, pass, or increase coverage.