aws-glue

Glue crawler exclude patterns

I have an s3 bucket that I'm trying to crawl and catalog. The format is something like this, where the SQL files are DDL queries (CREATE TABLE statements) that match the schema of the different data files, i.e. data1, data2, etc.) 

s3://my-bucket/somedata/20180101/data1/stuff.txt.gz
s3://my-bucket/somedata/20180101/data2/stuff.txt.gz
s3://my-bucket/somedata/20180101/data1.sql
s3://my-bucket/somedata/20180101/data2.sql  
s3://my-bucket/somedata/20180102/data1/stuff.txt.gz
s3://my-bucket/somedata/20180102/data2/stuff.txt.gz
...

I just want to catalog data1, so I am trying to use the exclude patterns in the Glue Crawler - see below - i.e. *.sql and data2/*.

Unfortunately the crawler is still classifying everything within the root path of s3://my-bucket/somedata/. I can live with having data2 cataloged; I'm most concerned/annoyed by the sql files.

Anyone have experience with exclude patterns or able to point out what is wrong here?

Solution

  • The * in the exclude pattern does not cross directories, but the ** does span across directories.

    To exclude all .sql files you can use: **.sql

    The fullpath of your data2/* exclusion is s3://my-bucket/somedata/data2/*, but its missing your date partition folders. This is remedied by adding a * in front.

    To exclude the data2/ directories use: */data2/*