prestoobject-storageibm-cloud-storagetrino

How to query different S3-compatible object storage by Prestosql


Background

prestosql works great with data on S3 and S3 compatible object storage (e.g., IBM cloud object storage) when using the URI prefix s3a:// with S3 configuration with single HMAC key pair via hive.s3.aws-access-key and hive.s3.aws-secret-key by following the prestosql guide Amazon S3 Configuration - Hive Connector.

Question

When data is served in two different buckets across two cloud accounts, it means a client has to use two different HMAC key pairs to access objects. does it mean it has to configure two catalogs via hive connector in prestosql?

This is common case when using IBM cloud where object storage services are managed as instances for different cloud accounts.


Solution

  • Yes, you need to configure two separate hive catalogs.

    Alternatively, you could use client-provided extra credentials (this is supported for GCS now, but can be easily extended to S3-compatible).