I am building a pipeline in Azure Data Factory.
The file name must be build with Hive Partitioning in the format year=YYYY/month=mm/day=DD
.
For month=
I need to pad the month number to 2 digits.
Current code:
@concat(
'/year=', formatDateTime(utcNow(), 'yyyy'),
'/month=', formatDateTime(utcNow(), 'MM'),
'/day=', formatDateTime(utcNow(), 'dd'),
'/FILENAME.parquet'
)
Current output for september:
/year=2024/month=9/day=30/FILENAME.parquet
Needed output for september:
/year=2024/month=09/day=30/FILENAME.parquet
How can I do this in Azure Data Factory? There is no padding function available in the useable functions described in:
There is no inbuilt function to get padded numbers. T resolve this uo need to make custom dynamic expression like below to add zero before month or date if length is 1 or it is less than 10 like below:
--for month
if(equals(length(formatDateTime(utcNow(), 'MM')),1),concat('0',formatDateTime(utcNow(), 'MM')),formatDateTime(utcNow(), 'MM'))
--for day
if(equals(length(formatDateTime(utcNow(), 'dd')),1),concat('0',formatDateTime(utcNow(), 'dd')),formatDateTime(utcNow(), 'dd'))
Full query will look like below:
@concat(
'/year=', formatDateTime(utcNow(), 'yyyy'),
'/month=', if(equals(length(formatDateTime(utcNow(), 'MM')),1),concat('0',formatDateTime(utcNow(), 'MM')),formatDateTime(utcNow(), 'MM')),
'/day=', if(equals(length(formatDateTime(utcNow(), 'dd')),1),concat('0',formatDateTime(utcNow(), 'dd')),formatDateTime(utcNow(), 'dd')),
'/FILENAME.parquet'
)
--for month
@if(less(formatDateTime(utcNow(), 'MM'),10),concat('0',formatDateTime(utcNow(), 'MM')),formatDateTime(utcNow(), 'MM'))
--for day
@if(less(formatDateTime(utcNow(), 'dd'),10),concat('0',formatDateTime(utcNow(), 'dd')),formatDateTime(utcNow(), 'dd'))
Full query will look like below:
@concat(
'/year=', formatDateTime(utcNow(), 'yyyy'),
'/month=', @if(less(formatDateTime(utcNow(), 'MM'),10),concat('0',formatDateTime(utcNow(), 'MM')),formatDateTime(utcNow(), 'MM')),
'/day=', @if(less(formatDateTime(utcNow(), 'dd'),10),concat('0',formatDateTime(utcNow(), 'dd')),formatDateTime(utcNow(), 'dd')),
'/FILENAME.parquet'
)