Snowflake Notebook fails with NotebookSqlException: 000710 (02000): Result for query <id> has expired

We are currently evaluating the new Notebook feature for our BI needs. So far, we are loving it, but we experience some issues. Maybe the Snowflake community can help!

I have an ETL task in a Notebook "PRE_REPORT_PROVISION". This Notebook is part of a task graph, like this:

CREATE OR REPLACE TASK PRE_REPORT_PROVISION_TSK 
WAREHOUSE = BI_ELT_WH
AFTER DWH_DONE_TSK 
AS EXECUTE NOTEBOOK "BI_ELT"."INTEGRATIONS"."PRE_REPORT_PROVISION"();

When I run this task manually everything seems to be working as expected.

However, when this task runs as part of the scheduled pipeline, the task sometimes fails and the run history show this error:

ERROR: Python Interpreter Error: Exception: NotebookSqlException: 000710 (02000): Result for query 01b6d952-0303-00bf-0001-b882007a614e has expired:

I have tried to find this ID in TASK_HISTORY(), but no joy.

SELECT *
FROM TABLE(BI_ELT.INFORMATION_SCHEMA.TASK_HISTORY())
WHERE QUERY_ID = '01b6d952-0303-00bf-0001-b882007a614e'
; -- Query produced no results

Neither in the QUERY_HISTORY()

SELECT  *
FROM TABLE(BI_ELT.INFORMATION_SCHEMA.QUERY_HISTORY())
WHERE QUERY_ID = '01b6d952-0303-00bf-0001-b882007a614e'
; -- Query produced no results

What makes this hard to debug: This happens often, but not always. After a failed scheduled run, for instance, I would open the notebook and run it, and it would fail with the same error (different id). This is very weird: I am literally running a simple SQL statement:

But then I run it again, and it works.

So my questions:

1. Has anybody seen this before and maybe knows what the issue might be?

2. Why can't I find the ID in the error report with the queries above? Do I need to look somewhere else?

Any hints are greatly appreciated!

Apparently there is a bug in the latest version of the Snowpark Python package. The solution is to manually set "snowflake-snowpark-python" to 1.20.0, the version before "latest".

Alternatively, add the following to the environment.yaml:

dependencies:
    - snowflake-snowpark-python=1.20.0