azureazure-pipelinesazure-data-factory

Issue with ADF removing the '\' Character while File-Conversion from .Tab to Parquet File type


During a certain File type conversion from The Raw Tab file That we fetch to convert to Parquet Type, one of the rows is always missing \ in the columns values,

so the expected value for example is :'GH\334' But the Parquet File only has 'GH334' not sure why its skipping the '' especially since we are not doing any sort of data transform, just running a simple Copy Activity.

Design is: Source: CSV file type Sink: Parquet File type Mapping: @json(item().Mapping) (This mapping is provided from a lookup activity which is fetched from the SQL server) no other settings or such, Has anyone dealt with such issues before? would appreciate your inputs on this; Thank you.

Also to mention, this row is being converted to string type for further data transformation...I was wondering if that had any relation with it, But I would need some Ideas on How I can move forward with this Again, the source CSV/.tab file does indeed have the row with '' Character, its just during the copy Activity, that its getting removed

Frankly its such a simple copy Activity, I am sort of out of ideas on what to try, I would appreciate it, if someone could give me an idea on what Can I try

Sample Input Data: GH/DFC\3456

Sample Expected Output Data: GH/DFC\3456

Sample Actual Output Data: GH/DFC3456


Solution

  • @mohammed rifat khan, posting our discussion as an answer for the community.

    I took the below data as sample which is similar to your data.

    Table2Id,Marks,name,age
    1,94,"GH/DFC\3456",25
    3,100,"Rakesh/DFC\3456",24
    5,98,"GH/Laddu\3456",23
    

    To not skip the \ character, you need to ensure that you have given the \ character as an escape character in the source dataset configurations.

    enter image description here

    Now, it will give the desired output which you can cross-check from the source dataset data preview.

    enter image description here