Possibility to do JSON access logs to stdout

[ukarlsson]

Is your feature request related to a problem? Please describe.
We are deploying vespa on k8s with GKE, and it is using a statefulset for logging. The issue is that these logs seem to grow indefinitely and we can run out of storage, and in addition it hard for us to search them. For other services that we use, all logs go to stdout and are picked up by GCP Cloud Logging where we can inspect them.

We have looked at the accesslog directive, but it is unclear if you can make it just write logs to /dev/stdout or if that would fail, and there does not seem to be any directive to just print the logs instead of storing them in files.

Describe the solution you'd like
Allow writing logs to stdout instead of on file on disk

Describe alternatives you've considered
Implementing manual log cleanup with a script. But it seems unnecessary since we actually don't want logs on files like this and we use Cloud Logging for everything else.

Additional context
N/A

[aressem]

The vespa.log can be streamed to standard out by setting VESPA_LOG_STDOUT=true. This is not supported for the access log. Both logs are stored as files in /opt/vespa/var/logs/vespa/*. They should be rotated by logd in the container and not cause the filesystem to fill up. Which logs are filling up your diskspace ? How big are they ?

One suggestion would be to mount a volume into the container at /opt/vespa/logs to avoid filling up the container filesystem and then have some process to clean up that mounted volume.

[ukarlsson]

Thanks for the suggestions.

We have a separate filesystem for logs and it is 5 GB and it had filled up. The issue was with the access logs, and they were compressed during rotation, but they were never deleted so disk eventually filled up. The docs state that access logs are never deleted which corresponds to our observations:

Log retention
Access logs are rotated, but not deleted by Vespa processes. It is up to the application owner to take care of archiving of access logs.

If we have all this infrastructure for log rotation etc in Vespa it would be good if we could prevent the disks from filling up as well.

Would it be possible to add STDOUT logging for access logs like for the normal logs?

[dainiusjocas]

If access logs are not needed you can disable them https://docs.vespa.ai/en/reference/services-container.html#accesslog

  <container id="default" version="1.0">
    ...
    <!--    Don't waste disk space -->
    <accesslog type="disabled"/>
    ...
  </container>