I am doing an application that receives some data, treats it then creates a prediction-ai batch job. After the predictions are done, I am supposed to take them all and merge them with a previous file. The batch prediction writes to a bucket, let's call is gs://predictions.
At the moment, I have a cloud function that gets triggered whenever gs://predictions is written to. However, the batch prediction job streams data into the file, updating such file many times when there are lots of predictions to be made. This means that my cloud function gets triggered many times when I just wanted it to be called when the job was finished.
To overcome this, right now the cloud function gets called and then checks whether the job is finished or not. If it is, process the file; if not, let it slide. This, of course, brings a lot of unnecessary processing (and unnecessary code!) :-(
What would really help me here: can the batch job somehow write to Pub/Sub when it is done? Or even better, can it use a webhook so that when it is finished itself calls my cloud function? I tried to look into the docs, but could not find anything.
Is there any other proposed solution for this?
You can create a log sink into PubSub and filter the logs on this custom filter:
resource.type="ml_job"
textPayload="Job completed successfully."
Then, when a batch job completed, a log trace is printed and a message is posted into the PubSub topic.