Unauthorized access error from google bucket while accessing getObject API

Since past few days, our system is experiencing one strange issue. We have one ruby application in which has a sidkiq worker(a scheduled job). This worker is accessing google cloud bucket in frequent interval.

This worker is able to perform getObject operation from bucket, copy from one location to another location in same bucket and delete the object as well. However for some of the files(random files at random time) we are getting Unauthorized 401 error in the code.

This is pure random, there is no sequence or pattern.

Since, Most of the operations are working as expected by this worker, some of them are failing with the below error.

“Google::Apis::AuthorizationError Unauthorized“.

error_backtrace:

/home/app/webapp/vendor/bundle/ruby/3.0.0/gems/google-api-client-0.49.0/lib/google/apis/core/http_command.rb:222:in `check_status', 
/home/app/webapp/vendor/bundle/ruby/3.0.0/gems/google-api-client-0.49.0/lib/google/apis/core/api_command.rb:121:in `check_status', 
/home/app/webapp/vendor/bundle/ruby/3.0.0/gems/google-api-client-0.49.0/lib/google/apis/core/download.rb:109:in `execute_once', 
/home/app/webapp/vendor/bundle/ruby/3.0.0/gems/google-api-client-0.49.0/lib/google/apis/core/http_command.rb:113:in `block (2 levels) in execute', 
/home/app/webapp/vendor/bundle/ruby/3.0.0/gems/retriable-3.1.2/lib/retriable.rb:61:in `block in retriable', 
/home/app/webapp/vendor/bundle/ruby/3.0.0/gems/retriable-3.1.2/lib/retriable.rb:56:in `times', 
/home/app/webapp/vendor/bundle/ruby/3.0.0/gems/retriable-3.1.2/lib/retriable.rb:56:in `retriable', 
/home/app/webapp/vendor/bundle/ruby/3.0.0/gems/google-api-client-0.49.0/lib/google/apis/core/http_command.rb:110:in `block in execute', 
/home/app/webapp/vendor/bundle/ruby/3.0.0/gems/retriable-3.1.2/lib/retriable.rb:61:in `block in retriable', 
/home/app/webapp/vendor/bundle/ruby/3.0.0/gems/retriable-3.1.2/lib/retriable.rb:56:in `times', 
/home/app/webapp/vendor/bundle/ruby/3.0.0/gems/retriable-3.1.2/lib/retriable.rb:56:in `retriable', 
/home/app/webapp/vendor/bundle/ruby/3.0.0/gems/google-api-client-0.49.0/lib/google/apis/core/http_command.rb:102:in `execute', 
/home/app/webapp/vendor/bundle/ruby/3.0.0/gems/google-api-client-0.49.0/lib/google/apis/core/base_service.rb:366:in `execute_or_queue_command', 
/home/app/webapp/vendor/bundle/ruby/3.0.0/gems/google-api-client-0.49.0/generated/google/apis/storage_v1/service.rb:1914:in `get_object'

We are using gem google-api-client-0.49.0 in our application and the service-account which we are using to make REST API calls has the admin level privileges. We are not suspecting this as access level issue because application is able to perform get/copy/delete operations for most of the files here.

Any help would be really appreciated to identify what could be the root cause for this issue.

Sidekiq runs on threads which means the authentication context for your application is shared by all the threads. What's most likely happening is:

1. A Sidekiq job makes an initial request which fetches a refresh token
2. The refresh token is used to acquire an access token for use with the actual request
3. The access token is used successfully for a period of time while it is valid
4. The access token eventually expires and the next request made with it returns 401 Unauthorized
5. The access token is killed which makes it unavailable across all threads and future invocations of the worker
6. The next request after the access token is killed sees that there is no access token and uses the refresh token to acquire a new access token for its request
7. Repeat steps 3 through 6

Depending on how much concurrency you have there may be a race condition when the token expires where multiple threads attempting to use it may result in multiple 401 errors.

You can configure retries on error with the Google API client according to your logic.

Alternatively, you can configure retries on error in Sidekiq so that the job re-processes according to your logic.