230 green tests and a system that didn't work
The first milestone of the system closed with 230 green tests, a green CI and the production image building. I brought everything up, logged in through the browser, and nothing but the login worked. Every screen with clinic data returned 401 or 500. One session of actual use turned up eleven failures, and one of them let anyone on the internet create an account in the system.
The detail that matters: this system is 100% vibe-coded. It’s a system for medical clinics, and I didn’t touch the code. AI agents wrote all of it: the code, the tests, the migrations, the CI. My job was to decide what to build and to use what came back.
No real data had gone through it yet, it was all test data. All eleven failures were fixed, and each one got a test that keeps it from coming back. Still, the question stuck with me: how did 230 tests miss all of this?
Where the tests went wrong
What caught my attention was that almost none of the failures came from missing tests. Every piece had coverage. What failed was the way the tests talked to the system, almost always through a path no user ever takes.
I think this has a lot to do with who wrote them. The agent writes the code and the test with the same head. If it misunderstood how something works, the test is born with the same misunderstanding, and the two agree with each other. The suite is green because it confirms what the agent believed. Whether that matches what the system does, nobody checked.
I grouped what I found into seven patterns. You can check any of them in your own suite today, written by people or by AI.
1. The test uses a database with more power than the application
The authorization suite connected to Postgres as a superuser. Superusers bypass Row Level Security. The role and permission tables had per-clinic policies, and the code read those tables outside the clinic context. In the test, everything came back. Running for real, nothing came back, and the user got in with no permissions at all.
Once the test connection became the same one the application uses (NOBYPASSRLS), two tests that used to pass started failing.
What to check: does the test use the same database user, with the same permissions, that the application uses in production?
2. The test builds by hand what the application builds on its own
Each test built its DataSource with an explicit list of entities. The application loads them by glob (*.entity.ts). Two files were born plural, authz.entities.ts and mfa.entities.ts, and fell outside the glob. The application started, the healthcheck returned 200, and the first authorization query blew up with No metadata for Membership.
Every test passed, because none of them exercised the real loading.
What to check: is what the test builds by hand the same thing the application builds on its own?
3. The test confirms the comment instead of the behavior
This was the worst one. There was a test making sure every route declares what it requires. It read the whole file looking for the word @Public(). If the word showed up anywhere, the whole controller passed.
The @Public() was on the class of the authentication controller. The @RequirePermission on the method didn’t undo it, because in Nest they are different metadata keys. The result: the invite route answered without authentication and returned the invite token in the response body. With that token, anyone could set a password and end up with an account.
The comment in the code said the route was protected. The test agreed with the comment.
Now the test reads decorator by decorator, and a second test checks the real metadata and calls the route without a token.
What to check: does the test verify the effect, or does it verify that someone wrote down the right intention?
4. The test data is cleaner than real data
The patient search test used “jose antonio” against “josé antônio pereira”. It passes. Nobody types like that. At the front desk people type “maria”, and “maria” against “maria aparecida souza” scored 0.27 similarity, below the 0.3 threshold of pg_trgm. The search said the patient didn’t exist.
That’s how duplicate records are born. And merging two medical records later is close to irreversible.
What to check: does the test data look like what people type, or like what’s convenient for the test to pass?
5. No test waits for time to pass
The bug that bothered me most only showed up 15 minutes after login, when the access token expired. The refresh ran during rendering, where Next doesn’t let you set cookies. But the rotation had already happened in the API. The new pair got lost, the old token came back on the next request, and the API treated it as session theft: it killed every session the person had and fired a critical security event.
No test waited for anything to expire. To reproduce it, I ran it with a 90-second token.
What to check: is there anything in the system that changes behavior over time? Is there any test that waits for it to happen?
6. Only one side of an optional field is tested
A field in the lab order accepted the exam’s catalog code, and the code was silently dropped when it didn’t exist. Every test sent only the free-text name, without a code, which is also a valid path and the most common one. The path with a code never ran, and on it the link to the catalog got lost without any signal.
An optional field creates two paths. Testing only the empty one is testing half.
What to check: for every optional field that changes behavior when filled in, is there a test with it filled in?
7. The checks don’t look where the problem lives
The project’s typecheck only covered src. Not a single test file was type-checked. A service signature changed, the typecheck passed, and the error only showed up when that test ran.
When the check started including the tests, 34 accumulated errors showed up. Among them, tests reading a field the type doesn’t guarantee. They passed because the test double never produced the other case.
What to check: do typecheck, lint and coverage look at all of the code, or only at the part someone remembered to include?
What changed in how I work with AI
Closing a delivery now includes using what was built. No formal QA step: open the system and click around like the person who’s going to use it.
One hour of use found what 230 tests didn’t, and the worst failures showed up in the first ten minutes. I repeated it at every milestone after that. The second one had 7 failures and the third had 8, always with a green suite. The system reached the end of the MVP with more than a thousand tests, and a good part of them came from a failure someone only saw by using it.
AI writes tests very well and very fast. But a test written by the same head that wrote the code confirms what that head believes. Someone has to look from the outside, and the cheapest way I’ve found to do that is to use the system.
One rule came along with it: what the system doesn’t understand, it rejects at the door. Three of the failures were data accepted and silently dropped. Whoever made the call thinks it was saved, the system stored a null, nothing fails, and the consequence shows up months later, when some other feature needs a link that never existed.