I am using kedro together with the databricks extension for vscode to access databricks server on Azure. Everything works pretty well but I don't see any output when executing the file locally. The only output I receive is:
31/03/2023, 15:09:32 - Synchronizing code to /Repos/<>@<>.ide ...
31/03/2023, 15:09:32 - Running kedro-hello-world.py ...
[03/31/23 13:09:37] INFO Kedro project dp-kedro session.py:355
31/03/2023, 15:09:43 - Done (took 13914ms)
If i look at the logs i see the following:
{
"level": "debug",
"logger": "SDK",
"loggingFunction": "CommandExecutionService.commandStatus",
"message": "cluster",
"operationId": "id",
"operationName": "CommandExecutionService.commandStatus",
"request": {"method": "GET"},
"response": {
"id": "id",
"results": {
"data": "[03/31/23 12:41:43] INFO Kedro project dp-kedro session.py:355",
"resultType": "text",
},
"status": "Finished",
},
"timestamp": 1680266509470,
}
So it seems that the rest of the messages arent coming back to vscode? Executing the file directly on databricks works and I can see all outputs of Kedro.
I haven't had a chance to use the new databricks extension myself, perhaps the verbose mode here is useful?
As discovered in the thread below -
I've finally found it, because I am working through vscode -> databricks -> kedro, the logging (which is using rich) is messed up, removing rich from the config and replace it by console works... so the initial tip of looking into logging works :) thx
Please set your logging.yml
to plain console logging and remove the RichHandler
root:
- handlers: [rich, info_file_handler, error_file_handler]
+ handlers: [console, info_file_handler, error_file_handler]