I have a legacy system that still serves customers and needs to be behind a fastly cache. However, our new system needs to stay in sync with our legacy system, so we make updates via the rest api. However, since fastly caches the results, this can cause our synchronization to start getting bad data.
I know it's possible to run a PURGE against the endpoint, and that seems to work a lot of the time, but not always (I'm not sure if our fastly servers are behind a load balancer. seems weird to me, but I'm not really a fastly person, so ¯\_(ツ)_/¯).
So. My question is, does anyone know if there is some sort of header I can send along with the request to force fastly to get me the latest info? I've tried Cache-Control: no-cache and no-store, however, these are not working.
If there is a setting on fastly itself to make it so these do work, that might be a solution, but our system folks are under high load and it can take a while for me to get things like this done sometimes. if I can provide them with the answer ahead of time, it might help.
Fastly's CDN product uses Varnish to cache content. You can find out more about Varnish and its configuration language (VCL) here: https://developer.fastly.com/learning/vcl/
Varnish supports HTTP caching, which means your origin server (e.g. your legacy service) should be sending the appropriate HTTP response header(s) to indicate how that resource should be cached (if at all) to any caching servers in front of it (e.g. a CDN like Fastly).
I would suggest reading through the following Fastly 'core concepts' reference which explains Fastly's caching behaviours in more detail: https://developer.fastly.com/learning/concepts/cache-freshness/
The following resource is also worth reading as it explains both the Cache-Control and Surrogate-Control HTTP headers (and their values) in more detail:
https://www.integralist.co.uk/posts/http-caching/
With regard to your specific question, there isn't anything built-in to a CDN that would allow a client (e.g. the device making a request for an endpoint) to skip the CDN caching layer because if there were it would mean a malicious user (aka bad actor) could use that to side-step the cache as well and overload your origin server.
That said, one way to get fresh content would be to 'cache-bust' the endpoint you're requesting by appending a query string. For example:
https://www.example.com/foo?cachebust=1
A CDN will cache content using a 'key', and that key is typically a hash of the URL hostname and path/querystring (e.g. example.com and /foo). So by adding a query string (e.g. ?cachebust=1) you'll cause the cache lookup to fail and a request to be sent to the origin.
Another approach might be to set a custom HTTP request header assigned a secret key and then have the CDN configured with the same key. This would allow you to write some VCL that would execute something like:
sub vcl_recv {
if (req.http.My-Secret == "123") {
return(pass); // skip cache lookup
}
return(lookup); // lookup endpoint in the cache
}
Here is a working example using Fastly's "Fiddle" tool: https://fiddle.fastly.dev/fiddle/588f3c06