pythonboto3

How to list objects based on prefixes with wildcard using Python Boto3?


I need to find all the files with a specific prefix. For example:

raw/client/Hist/2017/*/*/Tracking_*.zip

I tried this line of code but it does not work:

    import boto3
    client = boto3.client("s3", aws_access_key_id=aws_access_key_id, aws_secret_access_key=aws_secret_access_key)
    client.list_objects(Bucket="myBucket", Prefix="raw/client/Hist/2017/*/*/Tracking_*.zip")

Solution

  • You won't be able to do this using boto3 without first selecting a superset of objects and then reducing it further to the subset you need via looping. However, you could use Amazon's data wrangler library and the list_objects method, which supports wildcards, to return a list of the S3 keys you need:

    import awswrangler as wr
    objects = wr.s3.list_objects('s3://myBucket/raw/client/Hist/2017/*/*/Tracking_*.zip')