google-cloud-platformlogginggoogle-cloud-storagegoogle-cloud-logging

How can I configure the lifecycle management for automatic deletion of logs in Google Cloud Platform's logging storage?


I have a few Google Cloud projects that accumulate a substantial amount of logs, around 200GB per month, and this is expected to nearly double in the next period. Initially, I assumed it would be straightforward to manage this by adjusting the retention settings: Edit the log bucket, set the retention period in days, and then update the bucket.

I configured this more than six months ago, setting it to one day for a non-production project. However, it seems that nothing gets deleted until the beginning of the next month. The logs continue to accumulate daily and only reset on the first day of the following month.

After reviewing the documentation and various Stack Overflow questions, I discovered that retention isn't the period for log deletion; it's actually the duration to preserve logs. To address this, we need to establish a lifecycle for the objects within the bucket. Unfortunately, the bucket in question (_Default logging bucket) is a system bucket, and I can't configure it.

Even when I delete logs using gcloud logging logs delete, the storage remains unchanged and doesn't reset until the next month. Is there a way to set the logs' lifecycle to be less than a month?

Logs storage

Retention

Edit: Actually, retention should delete the logs, but I don't think this is happening. My logs for all projects.

retentionDays   integer

Logs will be retained by default for this amount of time, after which 
they will automatically be deleted. The minimum retention period is 1 
day. If this value is set to zero at bucket creation time, the 
default time of 30 days will be used.

https://cloud.google.com/logging/docs/reference/v2/rest/v2/locations.buckets


Solution

  • I believe I solved my own question; it is a misunderstanding of how Cloud Logging pricing works.

    ingestion

    That's why it says ingestion, not storage.

    Logging charges for the pre-indexed volume of logs data that is stored in
     the _Default log bucket and in user-defined log buckets, when the total
     volume exceeds the free monthly allotment. Every write of a log entry to
     the _Default log bucket or to a user-defined log bucket counts toward your
     storage allotment. For example, if you have sinks that route a log entry to
     three log buckets, then that log entry is stored three times.
    

    retantion

    And the billable storage is 0 because retention was set to 1 day.

    Logging charges retention costs when the logs are retained longer than the
     default retention period. You can't configure the retention period for the
     _Required log bucket. There are no retention costs when logs are stored
     only for the default retention period of the log bucket.
    

    https://cloud.google.com/stackdriver/pricing#storage-pricing