azure-data-factoryhive-partitionszero-padding

Zero padding a number in dynamic file name with dynamic expression


I am building a pipeline in Azure Data Factory. The file name must be build with Hive Partitioning in the format year=YYYY/month=mm/day=DD.

For month= I need to pad the month number to 2 digits.

Current code:

@concat(
    '/year=', formatDateTime(utcNow(), 'yyyy'),
    '/month=', formatDateTime(utcNow(), 'MM'),
    '/day=', formatDateTime(utcNow(), 'dd'),
    '/FILENAME.parquet'
)

Current output for september:

/year=2024/month=9/day=30/FILENAME.parquet

Needed output for september:

/year=2024/month=09/day=30/FILENAME.parquet

How can I do this in Azure Data Factory? There is no padding function available in the useable functions described in:


Solution

  • There is no inbuilt function to get padded numbers. T resolve this uo need to make custom dynamic expression like below to add zero before month or date if length is 1 or it is less than 10 like below:

    --for month
    if(equals(length(formatDateTime(utcNow(), 'MM')),1),concat('0',formatDateTime(utcNow(), 'MM')),formatDateTime(utcNow(), 'MM'))
    --for day
    if(equals(length(formatDateTime(utcNow(), 'dd')),1),concat('0',formatDateTime(utcNow(), 'dd')),formatDateTime(utcNow(), 'dd'))
    

    Full query will look like below:

    @concat(
    '/year=', formatDateTime(utcNow(), 'yyyy'),
    '/month=', if(equals(length(formatDateTime(utcNow(), 'MM')),1),concat('0',formatDateTime(utcNow(), 'MM')),formatDateTime(utcNow(), 'MM')),
    '/day=', if(equals(length(formatDateTime(utcNow(), 'dd')),1),concat('0',formatDateTime(utcNow(), 'dd')),formatDateTime(utcNow(), 'dd')),
    '/FILENAME.parquet'
    )
    
    --for month
    @if(less(formatDateTime(utcNow(), 'MM'),10),concat('0',formatDateTime(utcNow(), 'MM')),formatDateTime(utcNow(), 'MM'))
    --for day
    @if(less(formatDateTime(utcNow(), 'dd'),10),concat('0',formatDateTime(utcNow(), 'dd')),formatDateTime(utcNow(), 'dd'))
    

    Full query will look like below:

    @concat(
    '/year=', formatDateTime(utcNow(), 'yyyy'),
    '/month=', @if(less(formatDateTime(utcNow(), 'MM'),10),concat('0',formatDateTime(utcNow(), 'MM')),formatDateTime(utcNow(), 'MM')),
    '/day=', @if(less(formatDateTime(utcNow(), 'dd'),10),concat('0',formatDateTime(utcNow(), 'dd')),formatDateTime(utcNow(), 'dd')),
    '/FILENAME.parquet'
    )