pythondatabricksazure-databricksdatabricks-connect

Databricks streaming "A file referenced in the transaction log cannot be found"


So whenever i am trying to read from a source with stream i get this error "A file referenced in the transaction log cannot be found" and it points to a file that does not exist.

I have tried:

Is there anything else i could do?

Thanks in advance guys n girls!


Solution

  • So! I had another stream that was running and it had the same parent directory as this stream.. this seems to have been a issue.

    First stream was looking in: .start("/mnt/dev_stream/first_stream") Second stream was looking in: .start("/mnt/dev_stream/second_stream")

    Editing the second stream to look in .start("/mnt/new_dev_stream/new_second_stream") fixed this issue!