[SOLVED] TypeError: expected bytes, str found [while running 'Writing to DB/ParDo(_WriteToRelationalDBFn) while writing to db from using beam-nuggets

TypeError: expected bytes, str found [while running 'Writing to DB/ParDo(_WriteToRelationalDBFn) while writing to db from using beam-nuggets

@mohaseeb

I am trying below example to write data from pub\sub to postgresql.Getting below error while writing pub\sub data into postgresql. "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/result.py", line 545, in _colnames_from_description colname = description_decoder(colname) TypeError: expected bytes, str found [while running 'Writing to DB/ParDo(_WriteToRelationalDBFn)-ptransform-74904']

could you please help to fix it.

    months = p | "Reading month records" >> beam.Create([
        {'name': 'Jan', 'num': 1},
        {'name': 'Feb', 'num': 2},
    ])
    months | 'Writing to Sqlite table' >> relational_db.Write(
        source_config=relational_db.SourceConfiguration(
            drivername='sqlite',
            database='/tmp/months_db.sqlite',
            create_if_missing=True
        ),
        table_config=relational_db.TableConfiguration(
            name='months',
            create_if_missing=True```

Solution

This seems to be related to the version of pg8000. Try pip install pg8000==1.16.5. It looks like a breaking change was introduced in 1.16.6. More details can be found here: https://github.com/sqlalchemy/sqlalchemy/issues/5645