AI Coding Agent Says Tests Passed? What MCP Logs Actually Prove (2026)
The tests passed. But which version of the code did they test?
When an AI coding agent finishes a change and announces that the tests passed, what exactly has been established?
There are at least three distinct claims: the agent says it ran tests, an instrumented runner observed a successful test command, and that success applies to the code and configuration about to be submitted. Those are not interchangeable.
In an independent Ubuntu experiment, we watched a real Codex CLI session fix a small Python bug and call an MCP test tool. All four tests passed, and the tool recorded a real process exit code of zero. We then deliberately changed the project after testing. A checker that compared only Python files missed a change to pytest’s discovery configuration. A broader file manifest detected it.
This is not a report of Codex deceiving its operator. The original agent result was accurate. The later failures were researcher-injected cases designed to test the evidence boundary.
What an MCP test record can actually establish
The Model Context Protocol (MCP) exposes tools through calls such as tools/call. A tool result can indicate an MCP-level error, but its isError indicator should not be confused with the operating-system exit code of a test process. To evaluate pytest, our tool explicitly recorded the subprocess return code and output. Official documentation defines the MCP tool result format and the distinction between protocol/tool errors and application results: MCP Tools specification.
A useful record includes the exact command, the tests collected and passed, the process exit code, start/end timestamps, and the relevant inputs being tested. Even then, the record applies to what the runner observed. It does not prove that every agent execution path was monitored, that the tests covered every requirement, or that the final repository state remained unchanged.
The controlled experiment
We used the following lab configuration:
The environment was Ubuntu 24.04, with Codex CLI 0.167.0, official MCP Python SDK 2.3.0 (MCPServer), protocol revision 2026-07-28, and pytest 8.2.2. The agent and evidence server ran in separate containers; the independent rerun used a separate, read-only Docker environment with network access disabled.
The project intentionally began with a faulty clamp function. The code used the upper bound in both positions, so two of four unit tests failed.
# Faulty expression
return min(high, max(value, high))
# Corrected expression
return min(high, max(value, low))Codex changed the function and then called the connected MCP tool with scope set to full. The actual test runner returned four passing tests and exit code zero. The evidence file was written outside the agent container’s mounts, so the agent container could not directly edit that particular log file.
The recorded evidence included these values:
{
"scope": "full",
"exit_code": 0,
"start_utc": "2026-11-10T13:43:52.157250+00:00",
"end_utc": "2026-11-10T13:43:53.158796+00:00",
"source_sha256_start": "1ebb44ae5bcc62dbd0ca859f55bcdb0e7c79630096cac5f246f86fd04a88ac1a",
"source_sha256_end": "1ebb44ae5bcc62dbd0ca859f55bcdb0e7c79630096cac5f246f86fd04a88ac1a"
}This is an excerpt from the real MCP tool event, not invented output. The first checksum intentionally covered Python source files only. That narrow scope is crucial to what happened next.
We checked out the corrected Git revision in a separate Docker test environment without network access and ran pytest independently. It also returned four passes and exit code zero. This was a local CI-style rerun, not a GitHub Actions run or an enforced pull-request status check.
When a successful record becomes stale
After the actual Codex session had finished, we performed two controlled post-test modifications. Neither change was performed autonomously by the agent.
The three recorded revisions produced distinct results:
- Corrected revision: The original source-only evidence checker reported PASS_LOG_ONLY. A fresh independent run returned 4 passed, exit code 0.
- Regression after testing, with file modification time restored: The checker reported STALE. A fresh run returned 2 failed, 2 passed, exit code 1.
- Test discovery configuration changed, with Python files untouched: The source-only checker still reported PASS_LOG_ONLY, a false approval. A fresh run collected zero tests, exit code 5.
The first alteration reintroduced the bug and restored the original filesystem modification time. A content hash detected the change even though the timestamp was unchanged. Timestamps alone would have been a weaker signal.
The second alteration was more subtle. We did not change the Python files. We changed only the pytest discovery setting:
[pytest]
python_files = no_such_tests_*.pyAt the altered revision, the same unfiltered pytest command no longer discovered any tests. Its output was:
no tests ran in 0.00sThe actual process exit code was 5, which pytest documents as no tests collected. Yet the original checker compared only Python file content and accepted the old success record. It was a false approval of the final revision, not a fabricated original test result.
A successful log is therefore meaningful only when its coverage of the tested state is explicit.
Improving the evidence boundary
We built a second, narrowly scoped MCP tool. Instead of hashing only Python source files, it constructed a deterministic digest from the names and contents of repository files relevant to this small project, excluding transient directories such as the Git database, Python caches, and the virtual environment.
The tool recorded the test command, return code, passing test count, timestamps, and file-manifest hashes immediately before and after the run. A separate verifier compared the final workspace manifest with the saved post-test manifest.
Without a subsequent change, the verifier reported PASS. After the same pytest.ini modification, it reported STALE, whereas the original Python-only checker still reported PASS_LOG_ONLY.
That addresses the specific gap we measured. It does not certify the software or create tamper-proof evidence: a host administrator or Docker operator can still change the records, and a file manifest cannot automatically capture every environmental dependency, external API, generated artifact, tool version, or test-coverage shortcoming.
A practical validation checklist
Before accepting an AI agent’s claim, ask six questions:
- What exact command was run? Distinguish the full suite from a filtered test selection.
- What was the underlying process exit code? An MCP call that completed is not necessarily a passing test process.
- How many tests were collected and passed? Treat a zero-test run as a failure of the validation requirement, not as a green result.
- What revision and test configuration were evaluated? Include changes to test files, discovery settings, scripts, and dependency declarations.
- What changed after the run? A relevant edit can invalidate the earlier record even when the recorded command genuinely passed.
- Can the submitted revision be checked independently? An external CI run at the actual commit provides a distinct trust boundary.
A complete policy must specify what happens when evidence is missing. Lack of an observed command can mean the command did not run, but it can also mean the observer lacked coverage. Unknown evidence is not proof that an action never occurred.
On GitHub, required checks have their own configuration traps. Official documentation notes that a skipped workflow may leave a required check pending, whereas a conditionally skipped job can report success. GitHub merge queues also need the merge_group event when using GitHub Actions. We did not test those GitHub behaviors in this lab; see the official required-check troubleshooting guide.
What this test cannot prove
This experiment involved one small Python fixture and one real Codex run. It did not measure how often coding agents report incorrect test results. Codex’s own final report agreed with the successful observed test. The false approval came from our deliberately limited checker after a researcher-created configuration change.
The MCP server and agent ran in separate containers, with the evidence directory unavailable inside the agent container. This restricted one direct tampering path, not every possible path. The normal Codex host sandbox was unavailable because of local user-namespace restrictions; the agent ran inside a disposable, resource-limited Docker container with its CLI sandbox/approval bypassed only within that disposable container. Do not copy that arrangement into a live development workflow as a security recommendation.
The independent test was a Docker rerun, not production CI or a protected GitHub merge check. We did not install, evaluate, or benchmark Code Telemetry or another commercial grader.
Bottom line
MCP logs can provide concrete evidence that a particular test command ran and returned a particular result. By themselves, they do not prove that the result still applies to the final code, test configuration, and runtime inputs.
In our test, four real passing tests became stale evidence after a configuration-only change caused the submitted revision to collect zero tests. Expanding the file boundary and rerunning tests independently made that mismatch visible.
For teams using coding agents, the useful question is not simply “Did the agent say green?” It is “Which checks ran, against which inputs, and what independent evidence applies to the exact revision we intend to accept?”
Primary references and related reading
- MCP Tools, protocol revision 2026-07-28
- Official MCP Python SDK v2
- pytest exit codes
- GitHub required status checks
- Agentic Pulse: How to Sandbox AI Coding Agents on Linux
- Agentic Pulse: Building a Custom MCP Server. This earlier guide covers a different SDK generation.
Editorial disclosure: This topic was suggested in an unsolicited email by Elliot Gardiner, who identified himself as the founder of Code Telemetry, an agent-observability vendor. We received no payment, did not use vendor software or a vendor-supplied workflow, and conducted the described experiments independently. The company did not direct, review, or approve the findings.
Experiment conducted November 10, 2026. Technical review November 11, 2026.