azure-data-factoryapache-synapseazure-notebooks

ADF Web Activity Error: The request content is not valid and could not be deserialized: 'After parsing a value an unexpected character was encountered

I am getting the error message in web Activity of ADF:

The request content is not valid and could not be deserialized: 'After parsing a value an unexpected character was encountered:

Further looking into this error, I found that getting an error from notebook which is:

---------------------------------------------------------------------------\nValueError                                Traceback (most recent call last)\n/tmp/ipykernel_10465/628309876.py in <module>\n      7 \n      8 \n----> 9 split_df.columns = ['dot1', 'dot2', 'dot3', 'dot4', 'dot5']\n     10 print(split_df)\n\n~/cluster-env/env/lib/python3.8/site-packages/pandas/core/generic.py in __setattr__(self, name, value)\n   5476         try:\n   5477             object.__getattribute__(self, name)\n-> 5478             return object.__setattr__(self, name, value)\n   5479         except AttributeError:\n   5480             pass\n\npandas/_libs/properties.pyx in pandas._libs.properties.AxisProperty.__set__()\n\n~/cluster-env/env/lib/python3.8/site-packages/pandas/core/generic.py in _set_axis(self, axis, labels)\n    668     def _set_axis(self, axis: int, labels: Index) -> None:\n    669         labels = ensure_index(labels)\n--> 670         self._mgr.set_axis(axis, labels)\n    671         self._clear_item_cache()\n    672 \n\n~/cluster-env/env/lib/python3.8/site-packages/pandas/core/internals/managers.py in set_axis(self, axis, new_labels)\n    218 \n    219         if new_len != old_len:\n--> 220             raise ValueError(\n    221                 f\"Length mismatch: Expected axis has {old_len} elements, new \"\n    222                 f\"values have {new_len} elements\"\n\nValueError: Length mismatch: Expected axis has 9 elements, new values have 5 elements"

and when I am sending this error to web activity to send email, I am getting the above web activity error.

Solution Tried:

So, to overcome this error, I created a variable "ErrorMessage" using expression which will hold the error message first:

@activity('Notebook1').error.Message

and then another variable variables('CleanErrorMessage') which will hold clean message replace the \n to '', using

@replace(variables('ErrorMessage'),'\n','')

but it is not working as expected

Can you help me to resolve this and remove this special symbols and line break from the notebook error message so that I can pass the error to web activity in json format and get the result.

Web Activity Json:

{
    "Pipeline":"@{pipeline().Pipeline}",
    "ErrorMessage": "@{variables('CleanErrorMessage')}",
    "DataFactoryName":"@{pipeline().DataFactory}",
    "Runid":"@{pipeline().RunId}"
}
Solution

  • You can do the following things to get it worked.

    @string(replace(activity('Notebook1').error.Message, '"',''''))