python-3.xdataframesnowflake-cloud-data-platformdataloader

pandas.DataFrame.to_sql intermittently loading data partially to snowflake/database


Intermittently it happens that pandas.DataFrame.to_sql partially loads data into snowflake. Example: DF has 25000 rows, buy the function loads only 15000 to snowflake. Has anyone faced this issue and resolved it. I tried playing with function parameters but the issue persists.

Example: DF has 25000 rows, buy the function loads only 15000 to snowflake. Has anyone faced this issue and resolved it. I tried playing with function parameters but the issue persists.


Solution

  • Changed the chunksize to 500 from 16000 and no issues seen after that for almost a month now.