pythonapache-sparkpipazure-synapsesentence-transformers

Install Sentence-Transformers as a Workspace package on Azure Synapse Analytics

I'm trying to install sentence-transformers python package as a workspace package for Azure Synapse Analytics.(To be used on Apache Spark Pool)

Tried installing it through magic command %pip install sentence-transformers through a notebook and it works.

Following the steps for uploading the sentence-transformers-2.2.2.tar.gz to Workspace and configuration throws the following error mentioned in the screenshot.

I tested with couple of other python packages and they work fine.

Any workarounds?

enter image description here

Solution

  • I got same error when I tried to upload the sentence-transformers-2.2.2.tar.gz from workspace packages to spark pool.

    enter image description here

    I am able to install the above package using the below workaround.

    First I have created a requirements.txt file with package and version like below.

    sentence-transformers==2.2.2

    enter image description here

    I got this error when trying to import before installing the package.

    enter image description here

    Upload the requirements.txt file to spark pool packages like below.

    enter image description here

    Wait for the Job to get succeed.

    enter image description here

    Now, you can see I am able to import it.

    enter image description here