I'm attempting to shuttle data between buckets in different projects using Google Cloud's Storage Transfer Service based on an event that's being emitted. This event points to a subdirectory of a bucket belonging to project A that I want to copy to a destination bucket in project B, and so naturally STS looked like a perfect fit.
I've successfully crafted the request, including the prefix expressions, and I believe that I'm just dealing with a permissions error now because when I use my code to copy between buckets in the same project it creates and starts the transfer as expected. When I use my service account to being the transfer across buckets, however, I get the following error:
Error creating the transfer job:
Failed to obtain the location of the destination Google Cloud Storage (GCS) bucket due to insufficient permissions.
Please verify that the necessary permissions have been granted.
So from here I've looked into the permissions of my service account. I've manually added that service account as a project editor (and then owner) of the second project, and I've added the account to the ACLs for the destination bucket I'm trying to transfer into.
Are there any other avenues that I'm missing to look down? I haven't had much success googling around for others that have hit my situation.
One problem that you may be running into is you may be granting access to the wrong service account. One service account is created for each project. The service account can be found using the get googleServiceAccounts command documented here. It should look like
storage-transfer-{UID}@partnercontent.gserviceaccount.com
The UID will be different for each project. Try verifying that the service account with the UID specific to the projectId you specify in the transferJobs create request has READ/LIST access in source bucket and READ/WRITE/LIST access in sink bucket.
If the number of transfers you need to set up is relatively small, you might also try using the console, which is a GUI designed to simplify the process of seting up a transfer. Instructions for using the console can be found here.