azureazure-data-lakemicrosoft-fabric

How can I write to my Fabric Lakehouse via external app?

I have a react application for user management and I need to write data directly (well via an azure function) to my fabric Lakehouse on a frequent basis.

Ideally I'd be able to use my service principal to authenticate

Any suggestions or help would be much appreciated. Preference would be using node/js but python would also be fine

Thanks

Solution

  • You can create an ADLS Gen2 Storage account and a data lake container. Then you can link the container to your lakehouse so it can be accessed as a folder, using a shortcut:

    https://learn.microsoft.com/en-us/fabric/data-engineering/get-started-shortcuts

    With this setup, you have many options for writing to the storage, including JavaScript:

    https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-directory-file-acl-javascript

    You can use Entra ID authentication with a service pricipal, as explained in that article.

    The files you write into ADLS will automatically appear in the lakehouse and you can work with them as any other file.

    PS: I made a library to help with processing data in Fabric lakehouses: https://pypi.org/project/bifabrik/