python-3.xpandasdataframetypesnan

Handling NaN entries in a dataframe created from CSV

I am trying to compare two dataframes. The first dataframe is created from a dictionary. In places the values may be empty, like so:

'Room #': ''

The second dataframe is created from the corresponding exported csv. Where this key value is created in the dataframe has a NaN.

My comparison is using the

df.equals()

method

because there are the NaN items in the second dataframe, this is (correctly) returning a false.

What is the best method, please, of changing the NaN values to an appropriate type for comparison to those in the dictionary dataframe? Or otherwise disregarding the no entry/NaN key:values? (I read that fillna is deprecated).

Solution

  • The documentation for fillna states, in not the most clear way, that method="backfill" is deprecated (use method="bfill" instead).

    This works if you prefer '' over NaN.

    df_read = pd.read_csv('...').fillna('')
is_eq = df_original.equals(df_read)