One feature on my site allows registered users to create calendars for their organization. We provide a dynamically-generated iCal feed for these calendars through a URL with query-string parameters. Anyone can subscribe to these feeds by entering the provided URL into Google Calendar, Outlook, iPhone, etc...
This has been working well enough for a few years, but we now have a problem with stale or deleted calendars. If a registered user significantly alters or deletes their account, the calendar will no longer exist and the feed is useless. We currently return a "404 - Not Found" error for those requests (recently changed from "400 - Bad Request").
My question is, other than returning the 404, is there any way to get subscribers to stop requesting a bad feed? This is a similar question, where the accepted answer suggests returning 404 or 410 and hoping the clients will see the error and manually remove the subscription.
That doesn't seem to be working so far. We get ~ 100k feed requests an hour and a full 30% of those are for deleted calendars.
Do Google, Apple, et al not give up when they repeatedly get a 404 for a feed? How have others handled this issue?
If this was just a problem with log pollution I wouldn't worry too much about it. However, since the feeds are dynamically generated, each request hits the backend db. The processing is trivial and doesn't appear to be affecting performance, but the situation can only get worse.
Apologies if this belongs on ServerFault. While the issue affects my servers, I believe the solution is programmatic.
I don't believe there is an easy answer - I think it's been asked before.
It's like having to deal with all the traffic when some hackers use your site for target practise on logins or xmlrpc or just looking for vulnerabilities. Or the spammers trying a scatter gun approach sending emails. Or when a web spider decides to excessively crawl your site. You have to size for all that non useful traffic.
You could possibly generate and keep up to date a list outside of the database of the bad ics URLs and have a script check and bounce the request before it gets near the database ? Basically try to deal as efficiently as possible with the problem.
You could also in account deletion try adding a step that requests the user go to their calendar programs and delete the feed before continuing. However that might cause bad vibes and probably would not totally fix it anyway.