varnish

Varnish: Can I "ban" based on TTL < 0?

The Varnish documentation on ban is not exactly clear to me. It suggests only a bunch of variables and operators are allowed, but also hints TTL could be used.

What I would like to write is in the line of

ban obj.ttl<0

Which obviously only makes sense because I use a long grace period.

Also, why am I doing this? RAM issues.

Please answer the question without judgement on why, I'm inheriting stuff, I just need help on that very specific idea of mine.

Thanks

Solution

  • Using the right ban expression

    The ban expression that you're looking for is the following:

    obj.ttl < 0s

    If you run it without the s suffix, you can get an error like this:

    varnish> ban obj.ttl < 0
106
expected duration <n.nn>[ms|s|m|h|d|w|y] got "0"

    Simulating the behavior

    I ran an experiment using the following VCL snippet:

    vcl 4.1;

backend default {
    .host = "localhost";
    .port = "8080";
}

sub vcl_backend_response {
    set beresp.ttl = 10s;
    set beresp.grace = 1d;
}

sub vcl_deliver {
    set resp.http.X-TTL = obj.ttl;
}

    The first run I did using curl -I localhost, resulted in the following output:

    HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Content-Length: 634
ETag: W/"27a-gP8zhkV0zZ2u/E8a5CI/uFwXGKI"
Date: Mon, 29 Jul 2024 08:48:39 GMT
X-Varnish: 15 32779
Age: 4
Via: 1.1 varnish (Varnish/7.5)
Accept-Ranges: bytes
X-TTL: 5.891
Connection: keep-alive

    The X-TTL header still returns a positive number for the remaining TTL. The value of the Age header shows how long the object has been in the cache.

    When you wait long enough and run it again, you'll get the following output:

    HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Content-Length: 634
ETag: W/"27a-gP8zhkV0zZ2u/E8a5CI/uFwXGKI"
Date: Mon, 29 Jul 2024 08:48:39 GMT
X-Varnish: 98308 32779
Age: 10
Via: 1.1 varnish (Varnish/7.5)
Accept-Ranges: bytes
X-TTL: -0.590
Connection: keep-alive

    The X-TTL value is now negative, so I an async backend fetch will take place because there's still grace left.

    But if you wait long enough without refreshing the page, the remaining TTL will drop below zero and these objects can be invalided by running the following command on the CLI:

    varnishadm ban "obj.ttl < 0s"

    When you run varnishadm ban.list, you'll see the ban you issued:

    Present bans:
1722243253.915465     0 -  obj.ttl < 0s
1722242677.672075     1 C

    The default value for the ban_lurker_age is 60 seconds. This means the ban lurker will only actively remove the objects from the cache based on the ban expression after 60 seconds.

    You can use varnishstat -f MAIN.n_object -1 to measure the impact of the ban expression to see if the object count drops.

    Requests for the banned content before the ban lurker removes them, also executes the ban expression.

    You can spot them by running a targeted varnishlog command like varnishlog -g request -c -i requrl,expban

    This will be the output you'll get:

    *   << Request  >> 98311
-   ReqURL         /
-   ExpBan         27 banned lookup

    This means the ban lurker didn't trigger the object removal, but the request did.

    Conclusion

    Running varnishadm ban "obj.ttl <0s" will remove all the expired objects from the cache that still have plenty of grace (or keep) left.

    Either the ban lurker will process this ban expression asynchronously after about 60 seconds, but meanwhile incoming requests for matching objects can also trigger the removal of those objects.