Disk full error makes logging fatal to the application

[guillomovitch]

Currentl, any write error in Mojo::Log append() method raises an exception:

$handle->print(encode('UTF-8', $msg)) or croak "Can't write to log: $!";

As a consequence, a disk-full error, which is an unfortunate but possible issue on any kind of web-facing application, makes logging fatal to the application, unless each call to log() is wrapped into an eval, try/catch, or any other exception-handling block. While perfectly possible, this seems a bit overkill. An attribute making write error non-fatal, while retaining current default behaviour, would make the issue easier to handle:

# croak "Can't write to log: $!" when disk is full
Mojo::Log->new(file => 'file.log')->info("message"); 
# warn "Can't write to log: $!" when disk is full
Mojo::Log->new(file => 'file.log', resilient => 1)->info("message");

[kraih]

An interesting question, should the logger not die when out of disk space. 🤔 And i wonder what the next thing to die would be once the logger survives, temporary files for larger requests?

[christopherraa]

I might be biased here since I usually focus on operational aspects, but my preference is that applications fail fast and fail hard. My assumption is that an application that is deployed has monitoring of such things as available disc space, applications running, applications being available / responding to requests on correct ports etc. Given those assumptions having an application crash eg. when the disk is full is super valuable since that tells me that the monitoring of disk space is not actually functioning as intended.

So what I am saying is that if you have responsible ops then have a disk be filled to the point where logging can't succeed should not be able to happen. At least in my mind that makes this proposal kind of redundant, and potentially gives a new user more rope they can shoot themselves in the feet with.

I would much rather see a future where proper monitoring is the norm, rather that features that are introduced as a consequence of not having monitoring. Especially since monitoring is so easy to implement today. If you don't go the full Loki/Prometheus/Grafana route (since that can be kind of complex) there are good alternatives that a super easy, like Beszel and similar tools.

[guillomovitch]

Monitoring doesn't prevent accidents, it just allows to detect them earlier. And when the issue happens out of business hours, it doesn't change much anyway. While an hard failure may be better, and easier to identify than a soft failure in most situations, they are also other use cases where this may be discussable. Apache web server, for instance, still works as a reverse proxy even when unable to write access logs, and I don't think this is considered a reason not to use it. Hence my point to make the logger behaviour configurable, not to change it for everyone.
At the very least, current behaviour ought to be documented: unless I'm wrong, nowhere it is stated than a failure to write a message will raise an exception, and potentially prevent to complete handling current request, which was rather counter-intuitive for me.

[kraih]

Making the logger behavior configurable might give users the wrong impression. If they expect the app server to stay fully functional without being able to write to disk, they would be very surprised when it fails randomly for some requests, and then they have no logs to investigate why it failed.

Maybe worth mentioning, the logger is already replaceable, so you could just use your own that does not die.