I'm trying to figure out how to purge a set of URLs without purging one by one (which is inefficient and buggy).
I'm also trying to figure out how to do this without purging content that we don't want purged.
Essentially, when I push updated files to the S3 bucket that my CDN points to, I want to purge any files that have changed -- but not purge files that have stayed the same.
I'm trying to figure out the difference between setting cache headers on CDN vs setting cache headers (the x-amz-meta-surrogate-key specifically I think?).
Could I somehow configure the metadata for the changed objects (when I push them to the s3 bucket) such that those files get purged and not the others?
(for what its worth, I'm using Fastly for CDN service).
I'm trying to figure out how to purge a set of urls without purging one by one
This is typically done by setting a Surrogate-Key on your origin's response. You can set the same 'key' on multiple different pages to support purging all of those pieces of content at the same time from one purge request.
For example: you could have www.example.com/abc sending Surrogate-Key: red blue while www.example.com/xyz sending Surrogate-Key: green yellow red.
So with Fastly you can issue a 'purge by key' request and that means you can purge the /abc page using the blue key, as it's unique to that page (although in that case you might as well just 'purge by url') but you can purge both /abc and /xyz by issuing a 'purge by key' request using the key red as that key is set on the response for both pages.
As far as coupling this to AWS S3, there is a Fastly documentation page that might help...
You can mark content with a surrogate key and use it to purge groups of specific URLs at once without purging everything, or purging each URL singularly. On the Amazon S3 side, you can use the x-amz-meta-surrogate-key header to mark your content as you see fit, and then on the Fastly side set up a Header configuration to translate the S3 information into the header we look for. -- https://docs.fastly.com/en/guides/setting-surrogate-key-headers-for-amazon-s3-origins
Some other Fastly material that might help you here: