All Tests Passed. Why AI Code Can Still Be Wrong

The tests are green. The patch can still miss the requirement that matters. InfraDev studies coding-agent changes to AI infrastructure. This short explainer shows why checking the expected output is different from checking memory, state, or communication—and how additional probes can expose a missing promise. In the paper’s evaluation, 102 submissions passed every official test; 33 of them did not pass the added probes. That is approximately 32% of that subset, not a universal failure rate for AI-generated code. CHAPTERS 00:00 Why passing tests can miss the promise 00:19 What InfraDev evaluates 00:31 Three different promises 00:42 Hidden checks, clear requirements 00:57 The 102 / 33 result 01:11 Same output, different transfer 01:25 What to measure PRIMARY SOURCE Deng et al., “InfraDev: Can Coding Agents Develop AI Infrastructure, and How Can We Tell?” arXiv:2610.36589v1, 27 October 2026. https://arxiv.org/abs/2610.36589 The transfer animations are original simplified illustrations. Their toy block counts are not experimental measurements. This video does not reproduce the multi-GPU benchmark or show a real agent run. Created with AI-assisted narration and original explanatory graphics. Original minimal sound design. What would your tests miss if the output stayed correct? #AICoding #SoftwareTesting #InfraDev