iTranslated by AI
Designing Log File Rotation as 'Selecting the Next Destination' Instead of 'Rename'
If you leave it to RotatingFileHandler or logrotate, you usually won't have any problems. I think that's fine.
The problem arises when issues appear as logs being missing at the boundary of dates or sizes, failures in renaming, or tail output being truncated.
If this sounds familiar, it may not be an accident, but a design issue of moving a file that is currently being written to.
Do you really need to rename a log file while it is being written to?
Here, I am reconsidering log file switching, not as "moving the current file," but as "selecting the next destination in advance."
A log file being written to holds state
app.log looks like a path.
The log file being written to, as referred to here, is the file the logger is currently using as the write destination. "Moving it later" refers to renaming or truncating that file.
However, from the logger's perspective, it has a handle, a buffer, and state on the writing side. It might also be touched by other processes tracking the output, collection infrastructure, backups, or antivirus software.
Therefore, the moment you rename or truncate a file that is being written to, it is not just a simple filename change.
It becomes an operation that moves the target being written to from the outside.
Differences between OSs bleed into the design
On Windows, renaming can fail depending on how the handle is held.
On POSIX, renaming is more likely to succeed. In return, the writing side may continue to write to the old file descriptor.
Even if both are correct behaviors, it is troublesome for logger design.
An operation success on the filesystem and the next log record heading to the intended new path are not the same thing.
Choosing to switch the write destination first
Therefore, I consider a direction that does not move the file being written to.
At the boundary of dates or sizes, instead of renaming the current file, I choose the next output destination.
logs/MyApp_20260403.log
logs/MyApp_20260404.log
Instead of moving the April 3rd file on April 4th, I write to the April 4th file starting from the log record for April 4th.
Rotation usually involves writing to the file being written to, then moving the file afterward. Switching output destinations means deciding which file to write the next log record to in advance. The latter is what I wanted to achieve.
This difference may seem small, but it is significant in the sense that you don't tamper with a file that is currently being written to later.
Case Study: D-SafeLogger
In D-SafeLogger, this concept is centered as append-only routing.
from dsafelogger import ConfigureLogger, GetLogger
ConfigureLogger(
log_path="./logs",
pg_name="MyApp",
routing_mode="daily",
)
logger = GetLogger(__name__)
logger.info("Application started")
logger.info(...) on the app side is similar to typical log output. What has changed is where the lifecycle of the file is handled.
What this judgment discarded
By focusing on append-only routing, I am distanced somewhat from the existing culture of external rotation.
Of course, there are sites where a configuration that links with external rotation tools is necessary. D-SafeLogger also has compatible paths like routing_mode="none" and ReopenLogFiles().
However, I do not place it at the center of the design.
What I place at the center is the boundary that, if D-SafeLogger owns the switching of output destinations, it will not rename or truncate the file currently being written to.
I wouldn't call it "safe"
This design is not a silver bullet.
OS failures, power outages, forced terminations, or unstable filesystems cannot be solved by the logger alone. You cannot change the very nature of NFS, SMB, or cloud-synced folders.
What I want here is not absolute safety, but to reduce avoidable uncertainty by one.
Eliminating the boundary where files are moved after being written to is meaningful for processes that run for a long time.
Conclusion
Log design tends to lean toward output formats and speed.
However, if you are writing to a local file, you cannot avoid the lifecycle of the file being written to.
A log file being written to has state, not just a path. Even if an operation on the filesystem succeeds, it does not guarantee that the next log record will head to the intended new path.
Therefore, I distinguish between rotation that moves files later and routing that selects the next write destination first.
By having the option not to move a file being written to in the first place, before deciding how to rotate, the perspective on design changes slightly.
Who decides "when to switch the file" and where leads to the next topic of viewing log settings as an operational contract.
Related Articles
Zenn Series:
- #1 Design log file switching as "selecting the next write destination" instead of renaming (This article)
- #2 Design log settings as an operational contract, not initialization code
- #3 Design log loss as something "explainable," not something to prevent
- #4 Logs that can be handled by stdout and logs that should remain in local files (Coming soon)
Related Qiita installments:
- #1 Is it really safe to rename the active file during log rotation?
- #2 I want to change log operations without rewriting logger.info(...)
- #3 Things to consider before letting child processes open the same log file
- #4 Don't read log library benchmarks in "fastest order" (Coming soon)
Discussion