iTranslated by AI
Week 06: Why AI Ignores Rules — The Perils of High-Speed Wrong Hypotheses and Human Oversight
Introduction
Previously, I wrote an article organizing the strengths and weaknesses of AI. Coding is fast. Problem diagnosis is weak. I established rules for when to use what and documented the investigation procedure in MEMORY.
A week later, the same type of failure recurred.
While investigating a permission-denied error, the AI jumped to the conclusion that it was a "write rule issue" and failed three times. It ignored my instruction to "verify before implementing" and began writing code without confirmation. The rule I wrote in MEMORY, "enumerate all candidates before eliminating them," was ignored.
Last time, I concluded that we need to "identify AI's strengths and weaknesses and use them accordingly." What I saw this time was a problem one step beyond that. Even if you create rules for usage, the AI doesn't necessarily follow them. And I couldn't stop the AI when it was breaking those rules.
What happens during AI investigations
It's not that AI lacks investigative capabilities. It's fast at reading code and good at picking up hints from logs.
However, things get tricky when it makes an incorrect initial hypothesis.
In this case, the moment the AI saw the "permission-denied during save" error, it formed a hypothesis: "a write rule issue." It read the security rules, checked the write conditions, and reported, "there is no problem."
It should have switched to, "If there's no problem with write, maybe it's read." Instead, the AI re-investigated in the same direction. It read the write rules again, checking for any oversights. The second time, it also said, "no problem." Only on the third attempt did it realize the cause was in a read query within the save process.
If the hypothesis is correct, the AI's speed is a weapon. If the hypothesis is wrong, it proceeds quickly in the wrong direction. And when the hypothesis is off, it doesn't shift gears easily. Instead of switching to, "Write was fine, so what should I suspect next?", it says, "Let me re-check the write rules."
Three cases that occurred this time
Determined it was a write issue for permission-denied
The save process consisted of three steps:
- Batch write items (write) → Success
- Get related posts (read) → Failed here
- Update photo information for posts (write) → Not reached because 2 failed
It's not that the AI was unaware of the existence of step 2. However, because the hypothesis "save error = write issue" was already established, it postponed the possibility of read. When a hypothesis is strong, other candidates become difficult to see.
Ignored "Verify before implementing"
I told it many times, "Check in PLAN mode before implementing." When the AI determines that it "knows the answer," it skips the verification step and begins writing code.
In the case of the permission-denied error, it omitted verification because it was convinced it was a "write rule issue." One would think verification is most necessary when there is conviction, but the more conviction there is, the more likely it is to skip verification.
Structural oversight in notification processing
A similar thing happened with the issue where comment reply notifications were not being sent.
There was a guard in the notification processing: "if operator === poster, do not notify." This is the correct logic to prevent a notification from being sent when you comment on your own post. However, because this guard was at the beginning of the function, all notifications were skipped even in cases where the poster was replying to someone else's comment.
This is a problem that could be noticed by looking at the control flow of the entire function. The AI has the ability to read code. However, when the judgment "this guard is correct" came first, the confirmation did not reach its scope of influence.
Even if you write rules, they aren't followed
Based on the lessons learned from last time, I wrote the following in MEMORY:
- "Do not guess the cause based on intuition"
- "List all candidates before eliminating them"
- "List all Firestore operations before investigating"
This week, the same type of error occurred. The rules were not followed.
My guess is that when the AI determines "I am correct," it prioritizes its own judgment over the rules. If it is confident in the judgment that "it's a save error, so it's a write issue," the rule to "list all operations" is treated as "not applicable this time."
Rules are referred to "when in doubt." When not in doubt—when confident in a hypothesis—the rules are not referred to. The rules fail at the exact moment they are needed most.
This also happens to humans. Even if we make a checklist, we skip it thinking "it's obvious this time." However, in the case of humans, as we experience painful setbacks, we gradually acquire the sense that "one should be cautious precisely when it seems obvious." AI does not have this. Confidence in a hypothesis and skipping procedures are directly linked every single time.
Where the real problem lay
Writing this far might make it sound like "controlling AI is difficult," but looking back, the problem actually lay a step earlier.
In the investigation of the permission-denied error, the AI reported that it was a "write rule issue." I couldn't doubt it. My intuition that "error during save = write issue" was the same. I didn't even know that a Firestore read query could be rejected due to insufficient filtering.
If I had been able to ask "Did you check the read side?", I might have reached the cause in the first investigation. But that question didn't come up.
When the AI is moving in the right direction, the human's ability to detect errors is not tested. What is tested is when the AI is moving in the wrong direction. And that judgment requires a certain level of basic knowledge.
If you are an engineer, you might naturally be able to switch to "if there is no problem with write, it must be read." However, when a non-engineer is developing with AI—a common situation in vibecoding—this switch is difficult. That was exactly my situation.
Things that can be done
Adding rules to the AI alone is not enough. I thought about mechanisms that we humans can supplement.
Create investigation templates
For each type of error, write down the investigation steps that should be taken first. In this case, it would be:
permission-denied investigation template:
1. List all Firestore operations of the target function (both read / write)
2. Compare rules corresponding to each operation
3. Check if the read query filter covers the rule conditions
If there is a template, I can instruct the AI to "list all operations first" before it jumps to a hypothesis. This forces it to follow the procedure without relying on the AI's own judgment.
Doubt the AI when it is confident
When the AI says, "This is clearly an issue with XX," I will reply, "I think it's clear, but just in case, list other possibilities as well." I want to make this a habit. It is precisely when there is confidence that the verification steps are skipped.
Use a second opinion
When I cannot judge the AI's diagnosis—in a state where I don't know if this is correct—I bring in a different perspective. If I send a request to an asynchronous review service like Codex, saying "I want you to investigate possibilities other than write," there is a possibility that a different hypothesis will emerge from a state without bias.
Record "why it was delayed" after incident response
After the problem is solved, I reflect on "why it took time to identify the cause." This reflection leads to the next template. This permission-denied incident itself became an investigation template.
Conclusion
When an AI has an incorrect hypothesis, it moves fast in a plausible direction, making it hard to notice that it is taking a detour. Last time, I wrote about "identifying the AI's strengths and weaknesses and using them selectively." This time, I learned that just creating rules for selective use is not enough.
There is a limit to asking the AI side for countermeasures. Even if you write rules, it won't follow them. Even if you increase the number of rules, it will judge them as "unnecessary this time."
The remaining means is to build a mechanism for detection and judgment on the human side. Structure the procedure with investigation templates. Have the habit of doubting the AI's confidence. Get a second opinion when you can't judge. They are all humble, but I feel they have more practical effectiveness than adding rules.
Technical details of this permission-denied incident (the relationship between Firestore queries and rules) are here.
The weekly development log is here.
Discussion