Varnish and ESI HTTP AUTH

I'm very lost on this problem, and I don't know where could be the problem, so, I hope that you could help me.

I have an HTTP BASIC authentification with symfony, and I'm trying to reach an url which is protected by this auth, with an tag in a Drupal page. Every requests are send to a Varnish

I give username and password in the url like that :

<esi:include src="http://admin:adminpass@api.dev:8081/app.php/next"/>

In my varnish configuration file, I have only that lines for auth.http:

if (req.http.Authorization) {
  return (pass);
}

My backend for Symfony is working well without http authentification, and the http authentification is working well when there's not Varnish and esi tag.

If anyone have an idea of the problem, please, tell me, even if it's wrong =)

Solution

ESI in varnish doesn't work like an iframe or link tag in a browser in that it doesn't connect to whatever url you give it. ESI just starts a new request within varnish and goes through the workflow (vcl_recv, etc).

You are expecting varnish to act like an http client, parsing the url, setting the authorization header, setting a host header to api.dev:8081 and initiating a new http connection/request which it will not. In this case, my guess is it starts a new req with req.url set to /app.php/next inheriting the headers from the request for the parent resource (containing the esi tag) or possibly just ignores the esi tag completely.

The way to accomplish what you want to do is (in vcl_recv):

if (req.esi_level > 0 && req.url == "/app.php/next") {
     set req.http.Authorization = "BASIC [base64 encoded admin:adminpass]"
     return (pass);
}

and then the esi tag should look like <esi:include src="/app.php/next" />

If you need the ESI request to hit a different backend server, you need to add that server as a different named backend:

backend authorization_needed {
   .host = "api.dev";
   .port = "8081";
}

and in vcl_recv, tell varnish to use it for esi requests:

if (req.esi_level > 0 && req.url == "/app.php/next") {
   set req.http.Authorization = "BASIC [base64 encoded admin:adminpass]"
   set req.backend = authorization_needed;
   return (pass);
}

you may also need to set req.http.Host in that if block if the backend responds to a different virtual host than "api.dev".

Update:

Since basic authorization is coming from the client, and you are calling return (pass) when req.http.Authorization is present, varnish will not ESI process those pages. You must explicitly enable esi in vcl_fetch() which is not called when you pass.

So to pass authorization for the ESI fragments but not for the parent page, change in vcl_rev:

if (req.http.Authorization && req.esi_level == 0) {
    set req.http.X-Esi-Authorization = req.http.Authorization;
    unset req.http.Authorization;
}
else if (req.http.X-Esi-Authorization && req.esi_level > 0 ) {
    set req.http.Authorization = req.http.X-Esi-Authorization;
    return (pass);
}

And add to vcl_fetch:

if (req.http.X-Esi-Authorization) {
    set beresp.do_esi = true;
}

The net effect is the parent response is cacheable and will process esi, the esi fragments themselves will always be passed to the backend with the client's authorization header.