I'm using google storage public bucket for serving binary resources on the web right now.
As the docs mentioned https://cloud.google.com/storage/docs/cross-origin
The authenticated browser download endpoint storage.cloud.google.com does not allow CORS requests. Note that the Google Cloud console provides this endpoint for each object's public URL link.
So in order to allow CORS, I need to put my proxy server in front.
I'm not sure there is a way I don't know to enable me to fetch resources directly from Cloud Storage with CORS but without having a relay server.
It looks like you're trying to use the URLs for objects that you're getting from the console. Those URLs use the domain storage.cloud.google.com. Because that domain allows callers to use their Google account cookies for authentication, its capabilities are intentionally kept very limited, and one such limitation is blocking cross-origin requests.
Instead, configure CORS using the standard Cloud Storage tools, then link your users to a URL like https://storage.googleapis.com/bucket_name/object_name, which will work fine with CORS. That will only work for publicly readable objects, but that sounds like what you've got.