Varnish cache for ESI component

I'm using Varnish for caching. I created a ESI component and use it in my code

    <esi:include src="/esi/my-component" />

It works nice, except one thing. It's not in cache. On the backend side I'm setting response headers

header('Content-Type: text/html');
header('Vary: Accept-Encoding');
header('Cache-Control: public, max-age=300, stale-while-revalidate=600, stale-if-error=600');
header('Surrogate-Control: max-age=300, stale-while-revalidate=600, stale-if-error=600');
header('Pragma: cache');

I see that cache works when I'm making curl requests (I have HIT in response), but when I change content of response my browser immediately fetched it, which means that cache not working for browser. I have a theory that this is because of cookies - so should I just remove it in varnish config for esi requests ? Is there a way to achieve cache without unsetting cookies ?

Cookies definitely play a big role in this story.

As described in the built-in VCL, Varnish will not cache if it sees a Cookie or Authorization header in the request.

That's something you need to mitigate in your VCL in the same way you're doing this for top-level requests:

• Either bypass the cache for uncacheable routes and strip off the cookie for all other requests
• Or selectively remove cookies, keeping only the ones you need in your application

Stripping off the cookies for your ESI component

Even after you strip off the tracking cookies, there might still be some cookies left, like for example a session cookie. It's up to you to decide whether your ESI component depends on cookie values.

If you don't need the cookies, here's what you can do in VCL to strip them off only for the ESI component:

sub vcl_recv {
    if(req.url ~ "^/esi/component") {
        unset req.http.Cookie;
    }
}

This will ensure cookies are stripped off from requests for /esi/component, making the request cacheable, while keeping the cookies for other requests that might need them.

Debugging

While we're implying that cookies are responsible for your cache misses, there's only one way to really find out: debugging the requests using varnishlog.

You can run the following command to inspect requests for /esi/component:

sudo varnishlog -g request -q "ReqUrl ~ '^/esi/component'"

Don't hesitate to attach the logging output as preformatted text to your original question. I can help you debug.

I'd also like to refer to the following section of our troubleshooting tutorial, which helps you figure out what Varnish is not serving a specific item from the cache.