AI wrote the tests: 7 signs they check nothing
Checkcraft is a desktop app for checking APIs and logs without code, with a free edition. Download, no sign-up →
The agent wrote twenty tests and all are green. That does not mean they check anything: a green test that cannot fail is worse than no test, because it builds confidence. Here are seven signs of an empty test and a way to check one in five minutes.
Seven signs of an empty test
1. Only status 200 is checked. 2. The expected value was copied from the actual response, not from the requirements. 3. The test checks that a field exists, not that its value is right. 4. A try/except swallows the error, or the assertion sits inside a condition that never runs. 5. The test checks a mock instead of the real service. 6. After a failure the agent "fixed" the test by changing the expectation. 7. The test has never failed: nobody ran it on wrong data.
A five-minute check
Take any test and feed it a deliberately wrong response: a different total, an extra field, a wrong order status. If the test stays green, it checks nothing. The practice API, where every response is 200 and the bugs are planted on purpose, is made for this: a proper total test fails on order 5001, because 3170 − 300 = 2870 while the response says 2970.
# An empty test: passes on any response with status 200
def test_order():
r = requests.get(URL)
assert r.status_code == 200
# A test that checks something: the total by the rule from the requirements
def test_order_total():
order = requests.get(URL).json()
assert order["total"] == order["subtotal"] - order["discount"]
# Try it on the practice order: the second test fails (2970 instead of 2870)
URL = "https://checkcraft.ru/practice-api/v1/orders/5001.json"How to avoid empty tests
Take expected values from the requirements and write them down before you see the response. Run every new test at least once on wrong data. And split the roles: the agent writes the checks, and an app runs them from saved rules. Then the agent has nowhere to bend the expectation, and every change is visible. That is how Checkcraft works with AI agents over MCP.
Repeat this on your own API
Checkcraft is a desktop app that saves API response checks without code and shows what did not match. Start with a sample, then use your own sanitized data. Free runs on Windows and Linux with no time limit.
Download FreeDownload and first runNo sign-up or email needed, the archive is ready in a couple of minutes. See the edition limits.
Start with the linked example and compare its result with your own test requirement.