pythonmongodbapache-sparkpyspark

MongoDB-PySpark: StringType has no matching BsonValue

I'm trying to write data to Mongodb, and I'm having the following error:

StringType has no matching BsonValue. Error: For input string: "457133333333672364913"

while saving the dataframe to mongodb using pyspark.

To save the dataframe I'm using:

(df
   .write
   .format("mongodb")
   .mode("append")
   .option("convertJson", "any")
   .option("database", "test")
   .option("collection", "mongo_collection")
   .save())

The column giving the error is an ArrayType(StringType())

Using Python3.9 and pyspark 3.4.0

I've searched online but I'm not finding anything related.

Solution

  • I've fixed the error removing the option:

    .option("convertJson", "any")

    that was causing the error in the conversion in Bson.