I am trying to read a CSV file and convert into a Beam dataframe using apache_beam.dataframe.
I am getting the below error whenever I use the beam dataframe library.
Error: AttributeError: type object 'Series' has no attribute 'append'. Did you mean: '_append
import csv
import apache_beam as beam
from apache_beam.dataframe import convert
input_csv = 'sample.csv'
with open('{0}'.format(input_csv)) as f:
events = [ dict(row) for row in CSV.DictReader(f)]
print(events)
Please discard the above indentation issue and I did not add Apache beam dataframe CSV.reader logic.Issue is that whenever I import apache beam dataframe module I am getting an error. Please let me know what I am I missing
I am using Apache beam version 2.50.0 in python
As mentioned in this GitHub link, this error is because the function Series.append has been removed in Pandas 2.0.0 (refer to this document). Apache Beam doesn't support Pandas 2.x since Pandas 2.x removed append from Series and Dataframe. This issue is being tracked in this GitHub link.
As mentioned by @Adhish, as a workaround, you can read a CSV file using the Beam core API and convert it to Dataframe.