I am doing a database insertion using Pandas to_sql to move millions of rows into sqlalchemy. I've created a small test csv with only 4 rows so that I know exactly what data is in the file.
Here is the csv format
column_one,column_two,column_three,column_four
0001-1234,db38ad21b3,https://example.com,2
0034-1201,38db21adb3,https://example-two.com,3
My database table is defined with the exact same column names.
df = pd.read_csv("test_repositories.csv",
header=0,
sep=',',
quotechar='"',
dtype={'column_one': str,
'column_two': str,
'column_three': str,
'column_four': int},
error_bad_lines=False)
df = df.where(pd.notnull(df), None)
df.to_sql(self.staging_table, db.engine, self.chunksize, method='multi')
This seems like it should work, however I keep getting the following TypeError saying that the operation schema + "." + name can't support str + int
File "/ingest/utils.py", line 59, in copy_csv_to_temp_table
df.to_sql(self.staging_table, db.engine, self.chunksize, method='multi')
File "/venv/lib/python3.8/site-packages/pandas/core/generic.py", line 2776, in to_sql
sql.to_sql(
File "/venv/lib/python3.8/site-packages/pandas/io/sql.py", line 590, in to_sql
pandas_sql.to_sql(
File "/venv/lib/python3.8/site-packages/pandas/io/sql.py", line 1382, in to_sql
table = SQLTable(
File "/venv/lib/python3.8/site-packages/pandas/io/sql.py", line 700, in __init__
self.table = self._create_table_setup()
File "/venv/lib/python3.8/site-packages/pandas/io/sql.py", line 966, in _create_table_setup
return Table(self.name, meta, *columns, schema=schema)
File "<string>", line 2, in __new__
File "/venv/lib/python3.8/site-packages/sqlalchemy/util/deprecations.py", line 139, in warned
return fn(*args, **kwargs)
File "/venv/lib/python3.8/site-packages/sqlalchemy/sql/schema.py", line 537, in __new__
key = _get_table_key(name, schema)
File "/venv/lib/python3.8/site-packages/sqlalchemy/sql/schema.py", line 77, in _get_table_key
return schema + "." + name
TypeError: unsupported operand type(s) for +: 'int' and 'str'
I understand what this error means. However, I don't understand why schema or name would cause a problem as all the column names are clearly strings. Any help is appreciated.
The function signature is:
DataFrame.to_sql(name, con, schema=None, if_exists='fail', index=True, index_label=None, chunksize=None, dtype=None, method=None)
Note that schema
is the 3rd positional argument defaulted to None
.
Therefore by using:
df.to_sql(self.staging_table, db.engine, self.chunksize, method='multi')
What you think is the chunksize is being interpreted as the schema
argument, so change your chunksize to be explicitly named, eg:
df.to_sql(self.staging_table, db.engine, chunksize=self.chunksize, method='multi')